release: update go.mod for v1.16.0

* feat: render predastore templates with the cluster and S3 planes split

The templates set one bind_addr per host, which put raft and blob wherever
S3 was published. A host now binds the cluster plane on the address peers
dial and the gate carries its own bind_addr for the S3 API, so the installer
can hand S3 the WAN without exposing replication with it.

The single-node template also drops to one node of each role. Extra blob
nodes only hash-partition keys across more badger indexes on the same drive,
and extra meta replicas add fsyncs for a quorum that dies with the machine.

* docs: regenerate the AWS model operation coverage table

* fix: repin predastore to the merged dev commit

README.md

@@ -12,7 +12,7 @@
   <a href="#what-is-spinifex">What is Spinifex?</a> ·
   <a href="#the-platform">Platform</a> ·
   <a href="#three-ways-to-deploy">Deploy</a> ·
-  <a href="#aws-compatibility">AWS services</a> ·
+  <a href="#aws-compatibility">AWS Compatibility</a> ·
   <a href="#core-components">Components</a> ·
   <a href="#architecture-at-a-glance">Architecture</a> ·
   <a href="#installation">Installation</a> ·
@@ -48,7 +48,7 @@ You change *where* your software runs, not *what* it is.
 
 From commodity hardware up to unmodified AWS tooling, every layer is replaceable and yours to own.
 
-## Three ways to deploy
+## Three Ways To Deploy
 
 One AWS-compatible surface, three deployment shapes. Pick the one that matches your reality; the platform on top is the same.
 
@@ -64,21 +64,21 @@ Bring your own hardware and host Spinifex in your own data centre. A real multi-
 
 Cloud where the cloud can't reach. Air-gapped sites, vehicles, vessels, factories, and clinics. Compute next to the data, running through disconnection, on hardware you own. The same AWS APIs your software already uses.
 
-## AWS compatibility
+## AWS Compatibility
 
 Speak the AWS API surface, natively. The AWS SDKs, AWS CLI, and Terraform: everything you deploy on AWS deploys on Spinifex unchanged. At the edge, on-premise, or on a partner Neocloud.
 
 | Service | What it is | Status |
 | --- | --- | --- |
 | **EC2** | Compute | Available |
-| **EBS** | Block storage | Available |
-| **S3** | Object storage | Available |
+| **EBS** | Block Storage | Available |
+| **S3** | Object Storage | Available |
 | **VPC** | Networking | Available |
-| **IAM** | Identity & auth | Available |
-| **ALB / NLB** | Load balancers | Available |
+| **IAM** | Identity & Auth | Available |
+| **ALB / NLB** | Load Balancers | Available |
 | **EKS** | Kubernetes | Available |
-| **ECR** | Container registry | Available |
-| **ECS** | Container service | Available |
+| **ECR** | Container Registry | Available |
+| **ECS** | Container Service | Available |
 | **RDS** | Databases | Available |
 | **Bedrock** | AI Deployment | Q3 2026 |
 
@@ -112,6 +112,14 @@ Spinifex is a minimal VM orchestration layer built on top of QEMU, exposing APIs
 - Multipart uploads, streaming reads/writes
 - Data redundancy with Reed-Solomon encoding
 
+### Northstar (Authoritative DNS)
+
+[Northstar](https://github.com/mulgadc/northstar) is a lightweight authoritative DNS server that handles both internal service discovery and public-facing DNS for a Spinifex cluster. Zones are human-readable TOML files, served from the local filesystem or synced from Predastore, keeping DNS configuration in the same object storage as the rest of your infrastructure.
+
+- Authoritative DNS over UDP, TCP, and DNS-over-TLS
+- Service discovery for cluster components via SRV records
+- Zone files stored locally or in S3-compatible storage
+
 ## Architecture at a Glance
 
 <p align="center">
@@ -131,7 +139,7 @@ Every AWS API call is authenticated at the gateway, published to a NATS subject,
 
 ## Installation
 
-Installation requires an Ubuntu / Debian system. See the detailed documentation at [docs.mulgadc.com](https://docs.mulgadc.com) for maintaining and installing Spinifex.
+Installation requires an Ubuntu 26.04 or Debian 13 system. See the detailed documentation at [docs.mulgadc.com](https://docs.mulgadc.com) for installing Spinifex.
 
 ### Bare Metal ISO
 
@@ -141,16 +149,11 @@ The recommended installation is a [bootable x86 installer](https://iso.mulgadc.c
 curl -fLO https://iso.mulgadc.com/spinifex.iso
 ```
 
-Follow the [USB install guide](https://docs.mulgadc.com/docs/install-usb) to write the ISO to USB and install on your hardware. The install guide walks through the full process.
-
+Follow the [USB install guide](https://docs.mulgadc.com/docs/install-usb) to write the ISO to USB and install on your hardware.
 
 ### Single Node Install
 
-The installation is straightforward to set up and running on a single node for testing purposes. Debian 13 is currently supported, additional Linux distributions are on the immediate roadmap.
-
->*Prerequisite:* Linux bridge for networking.
-
-Spinifex requires a Linux bridge configured on the host for VM networking. See the [single-node install guide](https://docs.mulgadc.com/docs/install#prerequisites) prerequisites for setup details.
+>*Prerequisite:* Spinifex requires a Linux bridge configured on the host for VM networking. See the [single-node install guide](https://docs.mulgadc.com/docs/install#prerequisites) for setup details.
 
 ```bash
 curl -fsSL https://install.mulgadc.com | bash
@@ -168,7 +171,7 @@ aws ec2 describe-instance-types
 
 ### Development Setup
 
-For a complete development environment see the [Source Install](https://docs.mulgadc.com/docs/install-source) documentation
+For a complete development environment see the [Source Install](https://docs.mulgadc.com/docs/install-source) documentation.
 
 ### Component Repositories
 
@@ -176,6 +179,7 @@ Spinifex coordinates these independent components:
 
 - **[Predastore](https://github.com/mulgadc/predastore)** - S3-compatible object storage
 - **[Viperblock](https://github.com/mulgadc/viperblock)** - EBS-compatible block storage
+- **[Northstar](https://github.com/mulgadc/northstar)** - Authoritative DNS server
 
 Each component can be developed independently. See component-specific documentation for focused development guides.
 
@@ -189,10 +193,6 @@ Spinifex ships with a built-in web console — an optional alternative to the AW
 
 The console is served by each node on port `3000` over TLS, and becomes available as soon as `spinifex.target` is up:
 
-```bash
-open https://YOUR_NODE_IP:3000
-```
-
 - **Same API, different surface.** Every action in the UI is the same AWS SigV4 call the CLI makes — so RBAC, audit trails, and IAM policies apply uniformly.
 - **Single sign-on against your AWS credentials.** Log in with the access keys from `~/.aws/credentials` on the node where Spinifex is installed — no separate user database.
 - **Self-hosted, works offline.** The UI is embedded in the Spinifex binary and served from the node itself. No external CDN, no analytics calls, no cloud dependency.

cmd/installer/install/blockdev.go

@@ -45,6 +45,7 @@ var (
 	sysClassBlockDir = "/sys/class/block"
 	devByIDDir       = "/dev/disk/by-id"
 	procMountsPath   = "/proc/mounts"
+	procSwapsPath    = "/proc/swaps"
 )
 
 // ListDisks enumerates install-candidate disks from /sys/block. It reads sysfs
@@ -132,6 +133,144 @@ func partitionPath(disk string, n int) string {
 	return fmt.Sprintf("%s%d", disk, n)
 }
 
+// readKernelPartitions returns the partition device names the kernel currently
+// publishes for a disk, sorted.
+//
+// It reads sysfs rather than /dev deliberately. A partition device node
+// outlives the table that created it, so a stale node is indistinguishable
+// from a fresh one by existence alone — which is how a disk whose old table
+// the kernel never dropped passes every check and then gets formatted at the
+// previous layout's offsets.
+func readKernelPartitions(disk string) ([]string, error) {
+	name := filepath.Base(disk)
+	entries, err := os.ReadDir(filepath.Join(sysBlockDir, name))
+	if err != nil {
+		return nil, fmt.Errorf("read the kernel's partition list for %s: %w", disk, err)
+	}
+	var out []string
+	for _, e := range entries {
+		// Partitions are the child directories carrying a "partition" attribute.
+		// "queue", "device" and the plain attribute files sit alongside them.
+		if _, err := os.Stat(filepath.Join(sysBlockDir, name, e.Name(), "partition")); err != nil {
+			continue
+		}
+		out = append(out, e.Name())
+	}
+	slices.Sort(out)
+	return out, nil
+}
+
+// kernelPartitions is indirected so tests can drive the partition-table
+// assertions without a real block device.
+var kernelPartitions = readKernelPartitions
+
+// partitionDevice maps a kernel partition name back to its device node,
+// resolved alongside the disk it belongs to rather than assuming /dev.
+func partitionDevice(disk, part string) string {
+	return filepath.Join(filepath.Dir(disk), part)
+}
+
+// diskHolders names the md, LVM or device-mapper devices claiming this disk or
+// one of its partitions. A holder is the usual reason the kernel refuses to
+// re-read a partition table, so it is what the operator needs told when the
+// installer gives up on a disk.
+func diskHolders(disk string) []string {
+	name := filepath.Base(disk)
+	dirs := []string{filepath.Join(sysBlockDir, name, "holders")}
+	parts, _ := kernelPartitions(disk)
+	for _, p := range parts {
+		dirs = append(dirs, filepath.Join(sysBlockDir, name, p, "holders"))
+	}
+
+	seen := map[string]bool{}
+	var out []string
+	for _, dir := range dirs {
+		entries, err := os.ReadDir(dir)
+		if err != nil {
+			continue
+		}
+		for _, e := range entries {
+			if seen[e.Name()] {
+				continue
+			}
+			seen[e.Name()] = true
+			out = append(out, e.Name())
+		}
+	}
+	slices.Sort(out)
+	return out
+}
+
+// diskMount is one filesystem mounted from a disk the installer is about to
+// erase.
+type diskMount struct {
+	Device     string
+	Mountpoint string
+}
+
+// devicesOnDisk is the set of device paths belonging to a disk: the whole disk
+// and every partition the kernel currently publishes. Matching /proc tables
+// against a set rather than a string prefix keeps /dev/nvme0n1 from claiming
+// /dev/nvme0n11.
+func devicesOnDisk(disk string) map[string]bool {
+	out := map[string]bool{disk: true}
+	parts, _ := kernelPartitions(disk)
+	for _, p := range parts {
+		out[partitionDevice(disk, p)] = true
+	}
+	return out
+}
+
+// mountsOnDisk returns the filesystems mounted from a disk, deepest mountpoint
+// first so a nested mount is released before the one it sits inside.
+func mountsOnDisk(disk string) []diskMount {
+	devs := devicesOnDisk(disk)
+	var out []diskMount
+	for _, fields := range procTable(procMountsPath, 0) {
+		if len(fields) >= 2 && devs[fields[0]] {
+			out = append(out, diskMount{Device: fields[0], Mountpoint: fields[1]})
+		}
+	}
+	slices.SortFunc(out, func(a, b diskMount) int {
+		return strings.Count(b.Mountpoint, "/") - strings.Count(a.Mountpoint, "/")
+	})
+	return out
+}
+
+// swapsOnDisk returns the active swap areas backed by a disk.
+func swapsOnDisk(disk string) []string {
+	devs := devicesOnDisk(disk)
+	var out []string
+	// The first line of /proc/swaps is a column header, not an entry.
+	for _, fields := range procTable(procSwapsPath, 1) {
+		if len(fields) >= 1 && devs[fields[0]] {
+			out = append(out, fields[0])
+		}
+	}
+	return out
+}
+
+// procTable splits a whitespace-delimited /proc table into fields per line,
+// dropping the first skip lines and any blank ones.
+func procTable(path string, skip int) [][]string {
+	data, err := os.ReadFile(path)
+	if err != nil {
+		return nil
+	}
+	var out [][]string
+	n := 0
+	for line := range strings.SplitSeq(string(data), "\n") {
+		n++
+		if n <= skip {
+			continue
+		}
+		if fields := strings.Fields(line); len(fields) > 0 {
+			out = append(out, fields)
+		}
+	}
+	return out
+}
+
 // byIDPreference orders the /dev/disk/by-id/ namespaces we prefer to name a
 // pool member by. wwn- ids are last: they are stable but opaque, so a degraded
 // pool reports a number nobody can map to a drive bay.

cmd/installer/install/blockdev_test.go

@@ -5,6 +5,7 @@ import (
 	"path/filepath"
 	"slices"
 	"strconv"
+	"strings"
 	"testing"
 )
 
@@ -40,6 +41,7 @@ func newFakeSysfs(t *testing.T, disks ...fakeDisk) *fakeSysfs {
 	swap(&sysClassBlockDir, filepath.Join(f.root, "sys/class/block"))
 	swap(&devByIDDir, filepath.Join(f.root, "dev/disk/by-id"))
 	swap(&procMountsPath, filepath.Join(f.root, "proc/mounts"))
+	swap(&procSwapsPath, filepath.Join(f.root, "proc/swaps"))
 
 	f.mkdirAll(sysBlockDir)
 	f.mkdirAll(sysClassBlockDir)
@@ -113,6 +115,18 @@ func (f *fakeSysfs) setMounts(content string) {
 	f.write(procMountsPath, content)
 }
 
+func (f *fakeSysfs) setSwaps(content string) {
+	f.t.Helper()
+	f.write(procSwapsPath, content)
+}
+
+// addHolder marks a partition as claimed by md, LVM or device-mapper — the
+// state that makes the kernel refuse to re-read the disk's partition table.
+func (f *fakeSysfs) addHolder(disk, part, holder string) {
+	f.t.Helper()
+	f.mkdirAll(filepath.Join(sysBlockDir, disk, part, "holders", holder))
+}
+
 func boolAttr(b bool) string {
 	if b {
 		return "1\n"
@@ -352,3 +366,119 @@ func TestHasAnyPrefix(t *testing.T) {
 		t.Error("sda should not match")
 	}
 }
+
+func TestReadKernelPartitionsListsOnlyPartitions(t *testing.T) {
+	newFakeSysfs(t, fakeDisk{
+		name: "nvme0n1", sectors: 1 << 22,
+		partitions: []string{"nvme0n1p3", "nvme0n1p1", "nvme0n1p2"},
+	})
+
+	got, err := readKernelPartitions("/dev/nvme0n1")
+	if err != nil {
+		t.Fatalf("readKernelPartitions: %v", err)
+	}
+	// Sorted, and "queue" and "device" alongside them are not partitions.
+	want := []string{"nvme0n1p1", "nvme0n1p2", "nvme0n1p3"}
+	if !slices.Equal(got, want) {
+		t.Errorf("got %v, want %v", got, want)
+	}
+}
+
+func TestReadKernelPartitionsIsEmptyForAnUnpartitionedDisk(t *testing.T) {
+	newFakeSysfs(t, fakeDisk{name: "nvme2n1", sectors: 1 << 22})
+
+	got, err := readKernelPartitions("/dev/nvme2n1")
+	if err != nil {
+		t.Fatalf("readKernelPartitions: %v", err)
+	}
+	if len(got) != 0 {
+		t.Errorf("got %v, want none", got)
+	}
+}
+
+func TestReadKernelPartitionsFailsOnAnUnknownDisk(t *testing.T) {
+	newFakeSysfs(t)
+
+	if _, err := readKernelPartitions("/dev/nvme9n1"); err == nil {
+		t.Fatal("expected an error for a disk with no sysfs entry")
+	}
+}
+
+func TestDiskHoldersNamesEveryClaimant(t *testing.T) {
+	f := newFakeSysfs(t, fakeDisk{
+		name: "sda", sectors: 1 << 22, partitions: []string{"sda1", "sda2"},
+	})
+	f.addHolder("sda", "sda1", "md0")
+	f.addHolder("sda", "sda2", "dm-0")
+
+	if got, want := diskHolders("/dev/sda"), []string{"dm-0", "md0"}; !slices.Equal(got, want) {
+		t.Errorf("got %v, want %v", got, want)
+	}
+}
+
+func TestDiskHoldersIsEmptyWhenNothingClaimsTheDisk(t *testing.T) {
+	newFakeSysfs(t, fakeDisk{name: "sda", sectors: 1 << 22, partitions: []string{"sda1"}})
+
+	if got := diskHolders("/dev/sda"); len(got) != 0 {
+		t.Errorf("got %v, want none", got)
+	}
+}
+
+func TestMountsOnDiskFindsOnlyThisDiskDeepestFirst(t *testing.T) {
+	f := newFakeSysfs(t,
+		fakeDisk{name: "sda", sectors: 1 << 22, partitions: []string{"sda1", "sda2"}},
+		fakeDisk{name: "sdb", sectors: 1 << 22, partitions: []string{"sdb1"}},
+	)
+	f.setMounts(strings.Join([]string{
+		"/dev/sda1 /oldroot ext4 rw 0 0",
+		"/dev/sda2 /oldroot/boot/efi vfat rw 0 0",
+		"/dev/sdb1 /other ext4 rw 0 0",
+		"proc /proc proc rw 0 0",
+	}, "\n") + "\n")
+
+	got := mountsOnDisk("/dev/sda")
+	want := []diskMount{
+		{Device: "/dev/sda2", Mountpoint: "/oldroot/boot/efi"},
+		{Device: "/dev/sda1", Mountpoint: "/oldroot"},
+	}
+	if !slices.Equal(got, want) {
+		t.Errorf("got %v, want %v", got, want)
+	}
+}
+
+// /dev/nvme0n1 must not claim /dev/nvme0n11, which a string-prefix match would.
+func TestMountsOnDiskDoesNotMatchASiblingNamespace(t *testing.T) {
+	f := newFakeSysfs(t,
+		fakeDisk{name: "nvme0n1", sectors: 1 << 22, partitions: []string{"nvme0n1p1"}},
+		fakeDisk{name: "nvme0n11", sectors: 1 << 22, partitions: []string{"nvme0n11p1"}},
+	)
+	f.setMounts("/dev/nvme0n11p1 /elsewhere ext4 rw 0 0\n")
+
+	if got := mountsOnDisk("/dev/nvme0n1"); len(got) != 0 {
+		t.Errorf("got %v, want none", got)
+	}
+}
+
+func TestSwapsOnDiskSkipsTheHeaderAndOtherDisks(t *testing.T) {
+	f := newFakeSysfs(t,
+		fakeDisk{name: "sda", sectors: 1 << 22, partitions: []string{"sda1"}},
+		fakeDisk{name: "sdb", sectors: 1 << 22, partitions: []string{"sdb1"}},
+	)
+	f.setSwaps(strings.Join([]string{
+		"Filename\t\t\t\tType\t\tSize\tUsed\tPriority",
+		"/dev/sda1 partition 8388604 0 -2",
+		"/dev/sdb1 partition 8388604 0 -3",
+	}, "\n") + "\n")
+
+	if got, want := swapsOnDisk("/dev/sda"), []string{"/dev/sda1"}; !slices.Equal(got, want) {
+		t.Errorf("got %v, want %v", got, want)
+	}
+}
+
+func TestSwapsOnDiskIsEmptyWithoutASwapsTable(t *testing.T) {
+	newFakeSysfs(t, fakeDisk{name: "sda", sectors: 1 << 22, partitions: []string{"sda1"}})
+
+	if got := swapsOnDisk("/dev/sda"); len(got) != 0 {
+		t.Errorf("got %v, want none", got)
+	}
+}

cmd/installer/install/disk.go

@@ -4,6 +4,9 @@ import (
 	"fmt"
 	"log/slog"
 	"os"
+	"path/filepath"
+	"slices"
+	"strings"
 	"time"
 )
 
@@ -47,6 +50,95 @@ const (
 	tailReserveMiB = 1024
 )
 
+// partitionSettleTimeout bounds the wait for the kernel and udev to publish a
+// disk's new partition table. A var so tests do not pay it.
+var partitionSettleTimeout = 15 * time.Second
+
+// releaseDisk detaches whatever the live environment has attached to a disk
+// before it is erased. A mounted filesystem or an active swap area makes
+// BLKRRPART return EBUSY, and the kernel then keeps serving the disk's old
+// partition table while sgdisk writes the new one and reports success.
+func releaseDisk(d Disk) {
+	for _, m := range mountsOnDisk(d.Path) {
+		err := runQuiet("umount", m.Device)
+		if err != nil {
+			// Lazy detach: a busy mount must not stall an install that is about to
+			// erase the filesystem underneath it anyway.
+			err = runQuiet("umount", "-l", m.Device)
+		}
+		if err != nil {
+			slog.Warn("could not unmount before erase", "device", m.Device, "mountpoint", m.Mountpoint, "err", err)
+			continue
+		}
+		slog.Info("released mount before erase", "device", m.Device, "mountpoint", m.Mountpoint)
+	}
+	for _, s := range swapsOnDisk(d.Path) {
+		if err := runQuiet("swapoff", s); err != nil {
+			slog.Warn("could not disable swap before erase", "device", s, "err", err)
+			continue
+		}
+		slog.Info("disabled swap before erase", "device", s)
+	}
+}
+
+// settlePartitions forces the kernel to re-read a disk's partition table and
+// blocks until its view matches want, naming the partitions by number.
+//
+// An empty want asserts the disk has no partitions at all. That is the only
+// state no leftover can satisfy: a disk whose previous table had the same
+// partition numbers as the one being written — an old OS install is p1, p2, p3,
+// exactly what the boot layout writes — passes an "are the expected partitions
+// there" check on stale nodes alone.
+func settlePartitions(d Disk, want ...int) error {
+	if err := runQuiet("partprobe", d.Path); err != nil {
+		// partprobe rescans the whole disk and fails if it cannot open any part of
+		// it; the ioctl on its own sometimes still gets through.
+		if err := runQuiet("blockdev", "--rereadpt", d.Path); err != nil {
+			slog.Warn("could not force a partition table re-read", "disk", d.Path, "err", err)
+		}
+	}
+	_ = runQuiet("udevadm", "settle", "--timeout=10")
+
+	wanted := make([]string, 0, len(want))
+	for _, n := range want {
+		wanted = append(wanted, filepath.Base(d.PartitionPath(n)))
+	}
+	slices.Sort(wanted)
+
+	deadline := time.Now().Add(partitionSettleTimeout)
+	for {
+		got, err := kernelPartitions(d.Path)
+		if err != nil {
+			return err
+		}
+		if slices.Equal(got, wanted) {
+			return nil
+		}
+		if time.Now().After(deadline) {
+			return staleTableError(d, wanted, got)
+		}
+		time.Sleep(200 * time.Millisecond)
+	}
+}
+
+// staleTableError explains a kernel partition view that never caught up. The
+// erase case gets its own wording because it is the dangerous one — proceeding
+// would format the previous layout — and because the holder is usually the
+// whole diagnosis.
+func staleTableError(d Disk, wanted, got []string) error {
+	if len(wanted) == 0 {
+		msg := fmt.Sprintf("%s still shows partitions [%s] after its table was erased — the kernel is using the old table, so formatting would write at the previous layout's offsets",
+			d.Path, strings.Join(got, ", "))
+		if h := diskHolders(d.Path); len(h) > 0 {
+			return fmt.Errorf("%s; it is held by %s, which must be stopped before this disk can be installed to",
+				msg, strings.Join(h, ", "))
+		}
+		return fmt.Errorf("%s; something in the live environment is holding it open", msg)
+	}
+	return fmt.Errorf("%s: the kernel shows partitions [%s] but the new table has [%s] — the table was written but never picked up",
+		d.Path, strings.Join(got, ", "), strings.Join(wanted, ", "))
+}
+
 // espSizeMiB picks the ESP size for a disk.
 func espSizeMiB(d Disk) int {
 	if d.Bytes >= espLargeDiskGiB<<30 {
@@ -98,7 +190,7 @@ func partitionDataDisk(rm RoleMount) error {
 	if err := run("sgdisk", "-G", rm.Disk.Path); err != nil {
 		slog.Warn("could not randomise GPT GUIDs", "disk", rm.Disk.Path, "err", err)
 	}
-	return nil
+	return settlePartitions(rm.Disk, dataPartNum)
 }
 
 // partitionOne writes the GPT for a single disk.
@@ -129,25 +221,15 @@ func partitionOne(d Disk, rootType string) error {
 	if err := run("sgdisk", "-G", d.Path); err != nil {
 		slog.Warn("could not randomise GPT GUIDs", "disk", d.Path, "err", err)
 	}
-	return nil
+	return settlePartitions(d, biosPartNum, espPartNum, rootPartNum)
 }
 
 // waitForPartitions ensures every partition device node the installer is about
-// to use exists. Trixie's udev is slow enough after a table rewrite that mkfs
-// races it and fails with ENOENT on a device the kernel has already accepted.
+// to use exists. The kernel's own view was already asserted per disk as each
+// table was written; udev publishes the /dev nodes and by-id symlinks after
+// that, and Trixie is slow enough about it that mkfs races them.
 func waitForPartitions(cfg DiskConfig) error {
-	for _, d := range cfg.Disks {
-		// Best-effort: partprobe failing is not fatal, since udev may still act
-		// on the BLKRRPART ioctl sgdisk issued itself.
-		if err := run("partprobe", d.Path); err != nil {
-			slog.Warn("partprobe failed, continuing", "disk", d.Path, "err", err)
-		}
-	}
-	if err := run("udevadm", "settle", "--timeout=10"); err != nil {
-		slog.Warn("udevadm settle failed, continuing", "err", err)
-	}
-
-	deadline := time.Now().Add(15 * time.Second)
+	deadline := time.Now().Add(partitionSettleTimeout)
 	for _, d := range cfg.bootDisks() {
 		wanted := []string{d.PartitionPath(espPartNum), d.PartitionPath(rootPartNum)}
 		// The pool is built from by-id paths, so their symlinks have to exist

cmd/installer/install/steps_test.go

@@ -5,6 +5,8 @@ import (
 	"fmt"
 	"os"
 	"path/filepath"
+	"slices"
+	"strconv"
 	"strings"
 	"testing"
 	"time"
@@ -16,17 +18,69 @@ import (
 type fakeExec struct {
 	calls []string
 	fail  func(name string, args []string) error
+
+	// parts is the kernel's partition view per disk. Seeded by the test and then
+	// moved by the sgdisk calls the installer makes, so the partition-table
+	// assertions can run without a block device.
+	parts map[string][]string
+
+	// stale marks disks whose view never moves however many tables are written
+	// to them: something holds the disk open, BLKRRPART returns EBUSY, and the
+	// kernel keeps serving the old table while sgdisk reports success.
+	stale map[string]bool
+}
+
+// holdOpen makes a disk keep its current partition view forever.
+func (f *fakeExec) holdOpen(disk string) { f.stale[disk] = true }
+
+// seedPartitions gives a disk a pre-existing partition table — the state a
+// drive is in when the node has been installed on before.
+func (f *fakeExec) seedPartitions(disk string, nums ...int) {
+	for _, n := range nums {
+		f.parts[disk] = append(f.parts[disk], filepath.Base(partitionPath(disk, n)))
+	}
+}
+
+// applySgdisk moves the fake kernel view in step with a table rewrite, the way
+// a kernel that honoured every BLKRRPART would: -Z empties it, each -n adds the
+// partition it creates.
+func (f *fakeExec) applySgdisk(args []string) {
+	if len(args) == 0 {
+		return
+	}
+	disk := args[len(args)-1]
+	if f.stale[disk] {
+		return
+	}
+	for i, a := range args {
+		switch {
+		case a == "-Z":
+			f.parts[disk] = nil
+		case a == "-n" && i+1 < len(args):
+			num, _, _ := strings.Cut(args[i+1], ":")
+			n, err := strconv.Atoi(num)
+			if err != nil {
+				continue
+			}
+			f.parts[disk] = append(f.parts[disk], filepath.Base(partitionPath(disk, n)))
+		}
+	}
 }
 
 func fakeCommands(t *testing.T) *fakeExec {
 	t.Helper()
-	f := &fakeExec{}
-	prevRun, prevQuiet, prevEnv := run, runQuiet, runEnv
+	f := &fakeExec{parts: map[string][]string{}, stale: map[string]bool{}}
+	prevRun, prevQuiet, prevEnv, prevParts := run, runQuiet, runEnv, kernelPartitions
 
 	record := func(name string, args ...string) error {
 		f.calls = append(f.calls, strings.TrimSpace(name+" "+strings.Join(args, " ")))
 		if f.fail != nil {
-			return f.fail(name, args)
+			if err := f.fail(name, args); err != nil {
+				return err
+			}
+		}
+		if name == "sgdisk" {
+			f.applySgdisk(args)
 		}
 		return nil
 	}
@@ -35,8 +89,21 @@ func fakeCommands(t *testing.T) *fakeExec {
 	runEnv = func(env []string, name string, args ...string) error {
 		return record(name, append(env, args...)...)
 	}
+	kernelPartitions = func(disk string) ([]string, error) {
+		got := slices.Clone(f.parts[disk])
+		slices.Sort(got)
+		return got, nil
+	}
 
-	t.Cleanup(func() { run, runQuiet, runEnv = prevRun, prevQuiet, prevEnv })
+	// A short settle so a test asserting the stale-table failure does not wait
+	// out the install's real budget.
+	prevTimeout := partitionSettleTimeout
+	partitionSettleTimeout = 10 * time.Millisecond
+
+	t.Cleanup(func() {
+		run, runQuiet, runEnv, kernelPartitions = prevRun, prevQuiet, prevEnv, prevParts
+		partitionSettleTimeout = prevTimeout
+	})
 	return f
 }
 
@@ -114,6 +181,24 @@ func targetSkeleton(t *testing.T, root string) {
 
 // tempDisks builds disks whose device nodes are real files, so the steps that
 // wait for partition nodes to appear can complete.
+// fakeProcTables points the mount and swap readers at fixture tables for the
+// duration of a test.
+func fakeProcTables(t *testing.T, mounts, swaps string) {
+	t.Helper()
+	dir := t.TempDir()
+	point := func(dst *string, name, content string) {
+		prev := *dst
+		p := filepath.Join(dir, name)
+		if err := os.WriteFile(p, []byte(content), 0o644); err != nil {
+			t.Fatalf("write %s: %v", p, err)
+		}
+		*dst = p
+		t.Cleanup(func() { *dst = prev })
+	}
+	point(&procMountsPath, "mounts", mounts)
+	point(&procSwapsPath, "swaps", swaps)
+}
+
 func tempDisks(t *testing.T, n int, gib int64) []Disk {
 	t.Helper()
 	dir := t.TempDir()
@@ -237,21 +322,154 @@ func TestEspSizeMiB(t *testing.T) {
 	}
 }
 
-func TestClearDiskOnlyLabelclearsAnExistingPartition(t *testing.T) {
+// A drive arriving with a previous install's layout must have every partition
+// cleared, not just the one the root filesystem used to be on: a data drive's
+// partition is p1, so a p3-only labelclear leaves a former pool member's ZFS
+// label in place at both ends of the device.
+func TestClearDiskWipesEveryPartitionItFinds(t *testing.T) {
 	f := fakeCommands(t)
-	disks := tempDisks(t, 1, 40)
-	if err := clearDisk(disks[0]); err != nil {
+	d := tempDisks(t, 1, 40)[0]
+	f.seedPartitions(d.Path, biosPartNum, espPartNum, rootPartNum)
+
+	if err := clearDisk(d); err != nil {
 		t.Fatalf("clearDisk: %v", err)
 	}
-	f.mustRun(t, "zpool labelclear -f "+disks[0].PartitionPath(rootPartNum))
+	for _, n := range []int{biosPartNum, espPartNum, rootPartNum} {
+		f.mustRun(t, "zpool labelclear -f "+d.PartitionPath(n))
+		f.mustRun(t, "wipefs -a "+d.PartitionPath(n))
+	}
+	f.mustRun(t, "wipefs -a "+d.Path)
+	f.mustRun(t, "sgdisk -Z "+d.Path)
+}
 
-	f2 := fakeCommands(t)
+func TestClearDiskProbesNoPartitionsOnABlankDrive(t *testing.T) {
+	f := fakeCommands(t)
 	blank := Disk{Path: filepath.Join(t.TempDir(), "sdb")}
+
 	if err := clearDisk(blank); err != nil {
 		t.Fatalf("clearDisk: %v", err)
 	}
-	f2.mustNotRun(t, "labelclear")
-	f2.mustRun(t, "wipefs -a "+blank.Path)
+	f.mustNotRun(t, "labelclear")
+	f.mustRun(t, "wipefs -a "+blank.Path)
+}
+
+// The reported field failure: a drive whose old table the kernel never dropped.
+// Proceeding would format the previous layout's offsets, so the install must
+// stop, and the message must say the table is stale rather than blaming the
+// device node that did not appear.
+func TestClearDiskFailsWhenTheKernelKeepsTheOldTable(t *testing.T) {
+	f := fakeCommands(t)
+	d := tempDisks(t, 1, 40)[0]
+	f.seedPartitions(d.Path, biosPartNum, espPartNum, rootPartNum)
+	// sgdisk writes to the platters and exits 0, but BLKRRPART returned EBUSY,
+	// so the kernel's view never changes.
+	f.holdOpen(d.Path)
+
+	err := clearDisk(d)
+	if err == nil {
+		t.Fatal("clearDisk succeeded against a stale partition table")
+	}
+	for _, want := range []string{d.Path, "still shows partitions", "previous layout"} {
+		if !strings.Contains(err.Error(), want) {
+			t.Errorf("err = %q, want it to mention %q", err, want)
+		}
+	}
+}
+
+func TestClearDiskReleasesMountsAndSwapFirst(t *testing.T) {
+	f := fakeCommands(t)
+	d := tempDisks(t, 1, 40)[0]
+	f.seedPartitions(d.Path, biosPartNum, espPartNum, rootPartNum)
+
+	root, esp := d.PartitionPath(rootPartNum), d.PartitionPath(espPartNum)
+	swapDev := d.PartitionPath(biosPartNum)
+	fakeProcTables(t,
+		fmt.Sprintf("%s /oldroot ext4 rw 0 0\n%s /oldroot/boot/efi vfat rw 0 0\n", root, esp),
+		fmt.Sprintf("Filename\t\t\t\tType\t\tSize\tUsed\tPriority\n%s partition 1000 0 -2\n", swapDev),
+	)
+
+	if err := clearDisk(d); err != nil {
+		t.Fatalf("clearDisk: %v", err)
+	}
+	f.mustRun(t, "swapoff "+swapDev)
+
+	// Deepest mountpoint first: unmounting /oldroot while /oldroot/boot/efi is
+	// still attached fails with EBUSY and leaves the disk held.
+	var order []string
+	for _, c := range f.calls {
+		if c == "umount "+root || c == "umount "+esp {
+			order = append(order, c)
+		}
+	}
+	if want := []string{"umount " + esp, "umount " + root}; !slices.Equal(order, want) {
+		t.Errorf("unmount order:\n got %v\nwant %v", order, want)
+	}
+}
+
+// A drive held by md or LVM cannot be released — the ISO ships neither tool —
+// so the operator has to be told what is holding it.
+func TestStaleTableErrorNamesTheHolder(t *testing.T) {
+	f := newFakeSysfs(t, fakeDisk{name: "sda", sectors: 1 << 20, partitions: []string{"sda1"}})
+	f.addHolder("sda", "sda1", "md0")
+
+	prev := kernelPartitions
+	kernelPartitions = readKernelPartitions
+	t.Cleanup(func() { kernelPartitions = prev })
+
+	err := staleTableError(Disk{Path: "/dev/sda"}, nil, []string{"sda1"})
+	if err == nil || !strings.Contains(err.Error(), "held by md0") {
+		t.Fatalf("err = %v, want it to name the md0 holder", err)
+	}
+}
+
+// The reported field failure as a unit test: two drives arrive carrying a
+// previous install's partitions and the third has never been partitioned. All
+// three must be taken over, and the blank one must not be the only one to get
+// a new table.
+func TestPartitionDisksTakesOverDrivesThatAlreadyHavePartitions(t *testing.T) {
+	f := fakeCommands(t)
+	nodes := tempDisks(t, 3, 200)
+	f.seedPartitions(nodes[0].Path, biosPartNum, espPartNum, rootPartNum) // held an OS
+	f.seedPartitions(nodes[1].Path, dataPartNum)                          // held data
+	// nodes[2] arrives blank.
+
+	cfg := DiskConfig{FS: FSExt4}.WithRoles([]RoleMount{
+		{Role: RoleOS, Disk: nodes[0]},
+		{Role: RoleSpinifex, Disk: nodes[1]},
+		{Role: RolePredastore, Disk: nodes[2]},
+	})
+	if err := partitionDisks(cfg); err != nil {
+		t.Fatalf("partitionDisks: %v", err)
+	}
+	for _, d := range nodes {
+		f.mustRun(t, "sgdisk -Z "+d.Path)
+	}
+	f.mustRun(t, "sgdisk", nodes[0].Path, "-n 3:0:-1024M")
+	for _, d := range nodes[1:] {
+		f.mustRun(t, "sgdisk", d.Path, "-n 1:1M:-1024M", "-t 1:8300")
+	}
+}
+
+// A previous OS install leaves p1, p2, p3 — exactly the numbering the boot
+// layout writes. Asserting the expected partitions exist would pass on those
+// stale nodes and the install would format the old offsets, so the erase-stage
+// assertion on an empty table is what has to catch it.
+func TestPartitionOneStopsWhenAnOldTableHasTheSameLayout(t *testing.T) {
+	f := fakeCommands(t)
+	d := tempDisks(t, 1, 200)[0]
+	f.seedPartitions(d.Path, biosPartNum, espPartNum, rootPartNum)
+	f.holdOpen(d.Path)
+
+	err := partitionOne(d, typeLinuxFS)
+	if err == nil {
+		t.Fatal("partitionOne succeeded against a stale partition table")
+	}
+	// Specifically the erase-stage message: reaching the post-write check would
+	// mean the stale nodes had already been accepted.
+	if !strings.Contains(err.Error(), "still shows partitions") {
+		t.Errorf("err = %q, want the erase-stage stale-table message", err)
+	}
+	f.mustNotRun(t, "sgdisk -n")
 }
 
 func TestFormatESPsLabelsEveryMember(t *testing.T) {

cmd/installer/install/zfs.go

@@ -340,23 +340,35 @@ func exportPool() {
 	}
 }
 
-// clearDisk removes any prior filesystem, partition table and ZFS label from a
-// disk so a retry after a partial install is not rejected for stale metadata —
-// a failure that otherwise reads as mysterious rather than as leftover debris.
+// clearDisk takes a disk over unconditionally, whatever it holds: everything
+// attached to it is released, every signature on it is wiped, its partition
+// table is erased, and the kernel is made to agree the disk is now empty.
+//
+// The last part is the point. sgdisk writes to the platters and exits 0 even
+// when the kernel refuses to drop the old table, so without the assertion an
+// install proceeds against partition nodes describing the previous layout.
 //
 // The probes are quiet because a blank disk makes all of them fail, and their
 // stderr on the install console reads like the install is going wrong.
 func clearDisk(d Disk) error {
-	// ZFS labels live at both ends of the device and wipefs does not know about
-	// them, so an old pool member has to be cleared before its partition goes.
-	if old := d.PartitionPath(rootPartNum); exists(old) {
-		_ = runQuiet("zpool", "labelclear", "-f", old)
-	}
-	_ = runQuiet("wipefs", "-a", d.Path)
-	return runQuiet("sgdisk", "-Z", d.Path)
-}
+	releaseDisk(d)
 
-func exists(path string) bool {
-	_, err := os.Stat(path)
-	return err == nil
+	// Every partition, not just the root slice. ZFS labels live at both ends of
+	// a device and wipefs does not know about them, so a data drive that was
+	// once a pool member keeps its label and fails a later zpool create.
+	parts, err := kernelPartitions(d.Path)
+	if err != nil {
+		return err
+	}
+	for _, p := range parts {
+		dev := partitionDevice(d.Path, p)
+		_ = runQuiet("zpool", "labelclear", "-f", dev)
+		_ = runQuiet("wipefs", "-a", dev)
+	}
+
+	_ = runQuiet("wipefs", "-a", d.Path)
+	if err := runQuiet("sgdisk", "-Z", d.Path); err != nil {
+		return fmt.Errorf("erase partition table: %w", err)
+	}
+	return settlePartitions(d)
 }

cmd/spinifex/cmd/admin.go

@@ -1524,8 +1524,10 @@ func runAdminInit(cmd *cobra.Command, args []string) {
 	// is node-local, so it is derived here from this node's own config dirs.
 	northstarConfigPath := filepath.Join(dirs.Northstar, "northstar.toml")
 
-	// Parse multi-node predastore configuration (legacy flag-based approach for single-node)
-	var predastoreHostID int
+	// Parse multi-node predastore configuration. A single-node install is host
+	// 1, the only [[host]] its predastore.toml declares; the multi-node path
+	// below replaces this with the host matching this machine.
+	predastoreHostID := 1
 	if predastoreNodesStr != "" {
 		ips := strings.Split(predastoreNodesStr, ",")
 		if len(ips) < 2 {
@@ -1842,8 +1844,10 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 	// is node-local, so it is derived here from this node's own config dirs.
 	northstarConfigPath := filepath.Join(dirs.Northstar, "northstar.toml")
 
-	// Generate multi-node predastore config
-	var predastoreHostID int
+	// Generate multi-node predastore config. A single-node install is host 1,
+	// the only [[host]] its predastore.toml declares; the multi-node path below
+	// replaces this with the host matching this machine.
+	predastoreHostID := 1
 	hasPredastoreConfig := len(predastoreNodes) >= 2
 	if hasPredastoreConfig {
 		predastoreContent, err := admin.GenerateMultiNodePredastoreConfig(predastoreMultiNodeTemplate, predastoreNodes, accessKey, secretKey, region, natsToken, configDir, spxRoot, bindIP, compactionInterval, northstarCreds)
@@ -1858,7 +1862,14 @@ func runAdminInitMultiNode(cmd *cobra.Command, accessKey, secretKey, accountID,
 			os.Exit(1)
 		}
 
+		// One machine is one predastore host, so the node list entry matching
+		// this bind IP is this node's host ID. Without one the service has no
+		// [[host]] to run and refuses to start.
 		predastoreHostID = admin.FindNodeIDByIP(predastoreNodes, bindIP)
+		if predastoreHostID == 0 {
+			fmt.Fprintf(os.Stderr, "❌ Error: bind IP %s not found in the predastore node list\n", bindIP)
+			os.Exit(1)
+		}
 		fmt.Printf("✅ Created: multi-node predastore.toml (host ID: %d)\n", predastoreHostID)
 	}
 
@@ -2392,8 +2403,10 @@ func runAdminJoin(cmd *cobra.Command, args []string) {
 
 	northstarCreds, northstarConfigPath := northstarFromFormation(creds, dirs)
 
-	// Generate multi-node predastore config
-	var predastoreHostID int
+	// Generate multi-node predastore config. A single-node install is host 1,
+	// the only [[host]] its predastore.toml declares; the multi-node path below
+	// replaces this with the host matching this machine.
+	predastoreHostID := 1
 	hasPredastoreConfig := len(predastoreNodes) >= 2
 
 	if hasPredastoreConfig {

cmd/spinifex/cmd/admin_test.go

@@ -768,8 +768,7 @@ func predastoreMultinodeNodes() []admin.PredastoreNodeConfig {
 type predastoreBucketAuth struct {
 	Buckets []struct {
 		Name string `toml:"name"`
-		Type string `toml:"type"`
-	} `toml:"buckets"`
+	} `toml:"bucket"`
 	Auth []struct {
 		AccessKeyID     string `toml:"access_key_id"`
 		SecretAccessKey string `toml:"secret_access_key"`
@@ -821,16 +820,10 @@ func TestPredastoreMultinodeTemplate_NorthstarProvisioned(t *testing.T) {
 	var cfg predastoreBucketAuth
 	require.NoError(t, toml.Unmarshal([]byte(content), &cfg))
 
+	// The zone bucket is declared once for the whole cluster: every host's gate
+	// reads the same config, so a resolver on any node can serve it.
 	assert.True(t, cfg.hasBucket(admin.NorthstarBucketName), "northstar zone bucket not rendered")
 
-	// The zone bucket must be readable from every node's local predastore, so
-	// a resolver on any node can serve it.
-	for _, b := range cfg.Buckets {
-		if b.Name == admin.NorthstarBucketName {
-			assert.Equal(t, "distributed", b.Type)
-		}
-	}
-
 	// BootstrapBaseZone writes zone files with the system key, not the
 	// resolver's. Without this grant, seeding fails on every node.
 	systemActions, ok := cfg.policyFor("AK", admin.NorthstarBucketName)
@@ -1055,15 +1048,15 @@ func TestSpinifexTomlTemplate_PeerNorthstarStanzaIsPathOnly(t *testing.T) {
 }
 
 // Unset knob must leave production config byte-identical: no [compaction] block
-// and the original trailing bytes unchanged (single-node has no trailing
-// newline, multinode keeps one).
+// and the original trailing bytes unchanged — the [iam] table both templates
+// end on, and the newline after it.
 func TestPredastoreTemplates_UnsetCompactionEmitsNoBlock(t *testing.T) {
 	single := renderSingleNodePredastore(t, admin.ConfigSettings{
 		Region: "ap-southeast-2", BindIP: "0.0.0.0", ConfigDir: "/cfg",
 		AccessKey: "AK", SecretKey: "SK", NatsToken: "tok",
 	})
 	assert.NotContains(t, single, "[compaction]")
-	assert.True(t, strings.HasSuffix(single, `access_keys_bucket = "spinifex-iam-access-keys"`),
+	assert.True(t, strings.HasSuffix(single, "access_keys_bucket = \"spinifex-iam-access-keys\"\n"),
 		"unset single-node tail changed: %q", single[len(single)-40:])
 
 	multi, err := admin.GenerateMultiNodePredastoreConfig(

cmd/spinifex/cmd/admin_upgrade.go

@@ -8,10 +8,6 @@ import (
 
 	"github.com/mulgadc/spinifex/spinifex/migrate"
 
-	// Config migrations that need more than the framework register themselves
-	// here rather than in migrate, which every handler imports.
-	_ "github.com/mulgadc/spinifex/spinifex/migrate/predastoretopology"
-
 	"github.com/spf13/cobra"
 )
 

cmd/spinifex/cmd/admin_upgrade_test.go

@@ -13,17 +13,21 @@ import (
 // TestUpgradeSeesEveryConfigTarget pins the registrations `spx admin upgrade`
 // depends on. A migration whose package is no longer linked here would leave
 // the command reporting nothing pending on an install that needs migrating.
+//
+// spinifex.toml is the only config target: predastore.toml is not migrated at
+// all, so a predastore config on disk must contribute neither a version nor a
+// pending step.
 func TestUpgradeSeesEveryConfigTarget(t *testing.T) {
 	configDir := t.TempDir()
 
-	// Both targets must exist on disk to be reported at all.
+	// A target must exist on disk to be reported at all.
 	require.NoError(t, os.MkdirAll(filepath.Join(configDir, "predastore"), 0750))
 	require.NoError(t, os.WriteFile(filepath.Join(configDir, "spinifex.toml"), []byte("version = \"3\"\n"), 0640))
-	require.NoError(t, os.WriteFile(filepath.Join(configDir, "predastore", "predastore.toml"), []byte("version = \"4\"\n"), 0640))
+	require.NoError(t, os.WriteFile(filepath.Join(configDir, "predastore", "predastore.toml"), []byte("version = 1\n"), 0640))
 
 	versions, err := migrate.DefaultRegistry.ConfigVersions(configDir)
 	require.NoError(t, err)
-	assert.Equal(t, map[string]int{"spinifex.toml": 3, "predastore.toml": 4}, versions)
+	assert.Equal(t, map[string]int{"spinifex.toml": 3}, versions)
 
 	pending, err := migrate.DefaultRegistry.PendingConfig(configDir)
 	require.NoError(t, err)
@@ -32,8 +36,5 @@ func TestUpgradeSeesEveryConfigTarget(t *testing.T) {
 	for _, p := range pending {
 		targets[p.Target] = [2]int{p.FromVersion, p.ToVersion}
 	}
-	assert.Equal(t, map[string][2]int{
-		"spinifex.toml":   {3, 4},
-		"predastore.toml": {4, 5},
-	}, targets)
+	assert.Equal(t, map[string][2]int{"spinifex.toml": {3, 4}}, targets)
 }

cmd/spinifex/cmd/predastore_bind_test.go

@@ -58,11 +58,11 @@ host_id = 3
 	assert.Equal(t, 3, bind.HostID)
 }
 
-// TestDerivePredastoreBind_HostIDAbsentRunsWholeTopology covers the
-// single-node rule: those spinifex.toml templates omit host_id entirely,
-// pinning every node of the topology to one host, which only a
-// whole-topology process can serve.
-func TestDerivePredastoreBind_HostIDAbsentRunsWholeTopology(t *testing.T) {
+// TestDerivePredastoreBind_HostIDAbsentDerivesZero covers a spinifex.toml
+// that omits host_id: the derivation substitutes nothing, so the zero it
+// returns names no [[host]] and the start command rejects it rather than
+// guessing which host of the topology this process runs.
+func TestDerivePredastoreBind_HostIDAbsentDerivesZero(t *testing.T) {
 	resetGlobalViper(t)
 
 	path := writeSpinifexToml(t, `

cmd/spinifex/cmd/predastore_planes_test.go

@@ -0,0 +1,136 @@
+package cmd
+
+import (
+	"os"
+	"path/filepath"
+	"testing"
+
+	pds "github.com/mulgadc/predastore"
+	"github.com/mulgadc/spinifex/spinifex/admin"
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// loadRenderedPredastore parses a rendered template with predastore's own
+// loader. Asserting on the TOML text would only prove the template renders;
+// this proves the cluster it describes is one predastore will start.
+func loadRenderedPredastore(t *testing.T, content string) *pds.Config {
+	t.Helper()
+	path := filepath.Join(t.TempDir(), "predastore.toml")
+	require.NoError(t, os.WriteFile(path, []byte(content), 0o600))
+	cfg, err := pds.LoadConfig(path)
+	require.NoError(t, err, "the installer must not write a config predastore rejects")
+	return cfg
+}
+
+// singleNodeSettings is what a real install renders with: the paths predastore
+// validates are absolute, which the shared fixture leaves empty because the
+// tests around it only read the rendered text.
+func singleNodeSettings() admin.ConfigSettings {
+	settings := northstarSettings()
+	settings.ConfigDir = "/etc/spinifex"
+	settings.PredastoreDataDir = admin.PredastoreDataDir("/var/lib/spinifex")
+	return settings
+}
+
+func gateNode(t *testing.T, host pds.HostConfig) pds.NodeConfig {
+	t.Helper()
+	for _, n := range host.Nodes {
+		if n.Role == pds.RoleGate {
+			return n
+		}
+	}
+	t.Fatalf("host %d runs no gate", host.ID)
+	return pds.NodeConfig{}
+}
+
+// A single machine holds every shard of an object whatever the erasure code
+// is, so parity there costs a split, an encode and a second write for a copy
+// the same disk failure takes with it. Redundancy belongs to the pool under
+// the data directory.
+func TestPredastoreSingleNodeTemplateUsesNoParity(t *testing.T) {
+	t.Parallel()
+
+	cfg := loadRenderedPredastore(t, renderTemplate(t, predastoreTomlTemplate, singleNodeSettings()))
+
+	assert.Equal(t, 1, cfg.RS.Data, "a single node splits nothing")
+	assert.Equal(t, 0, cfg.RS.Parity, "a single node has nowhere to put parity")
+}
+
+// A single node is one server: one of each role. A second blob node would only
+// hash-partition the keys across another index on the same drive, and a second
+// meta node would add an fsync to every commit for a quorum sharing the disk it
+// exists to survive.
+func TestPredastoreSingleNodeTemplateRunsOneOfEachRole(t *testing.T) {
+	t.Parallel()
+
+	cfg := loadRenderedPredastore(t, renderTemplate(t, predastoreTomlTemplate, singleNodeSettings()))
+
+	require.Len(t, cfg.Hosts, 1, "a single node is one host")
+
+	byRole := make(map[pds.Role]int)
+	for _, n := range cfg.Hosts[0].Nodes {
+		byRole[n.Role]++
+	}
+	assert.Equal(t, map[pds.Role]int{pds.RoleGate: 1, pds.RoleBlob: 1, pds.RoleMeta: 1}, byRole)
+}
+
+// S3 is a public service and replication is not. The gate binds every
+// interface; raft and blob traffic bind the host's own address and stay there.
+func TestPredastoreSingleNodeTemplateSplitsThePlanes(t *testing.T) {
+	t.Parallel()
+
+	cfg := loadRenderedPredastore(t, renderTemplate(t, predastoreTomlTemplate, singleNodeSettings()))
+
+	require.Len(t, cfg.Hosts, 1)
+	assert.Equal(t, "0.0.0.0", gateNode(t, cfg.Hosts[0]).BindAddr, "S3 must answer on every interface")
+	assert.NotEqual(t, "0.0.0.0", cfg.Hosts[0].BindAddr, "the cluster plane must not be a wildcard")
+}
+
+func TestPredastoreMultiNodeTemplateSplitsThePlanes(t *testing.T) {
+	t.Parallel()
+
+	content, err := admin.GenerateMultiNodePredastoreConfig(
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok",
+		"/cfg", "/var/lib/spinifex", "10.0.0.1", 0, admin.NorthstarCredentials{},
+	)
+	require.NoError(t, err)
+
+	cfg := loadRenderedPredastore(t, content)
+
+	require.Len(t, cfg.Hosts, 3)
+	for _, host := range cfg.Hosts {
+		// Every machine's replication plane is the address its peers dial, so
+		// a second interface on any of them carries no cluster traffic.
+		assert.Equal(t, host.Addr, host.BindAddr, "host %d binds the cluster plane off its dial address", host.ID)
+		assert.Equal(t, "0.0.0.0", gateNode(t, host).BindAddr, "host %d must serve public S3", host.ID)
+	}
+}
+
+// The bind_addr under [[host.node]] is the gate's alone: no other role has a
+// listener to point anywhere, and predastore rejects one that tries.
+func TestPredastoreTemplatesBindOnlyTheGate(t *testing.T) {
+	t.Parallel()
+
+	multinode, err := admin.GenerateMultiNodePredastoreConfig(
+		predastoreMultiNodeTemplate, predastoreMultinodeNodes(), "AK", "SK", "ap-southeast-2", "tok",
+		"/cfg", "/var/lib/spinifex", "10.0.0.1", 0, admin.NorthstarCredentials{},
+	)
+	require.NoError(t, err)
+
+	for name, content := range map[string]string{
+		"single-node": renderTemplate(t, predastoreTomlTemplate, singleNodeSettings()),
+		"multi-node":  multinode,
+	} {
+		t.Run(name, func(t *testing.T) {
+			t.Parallel()
+			for _, host := range loadRenderedPredastore(t, content).Hosts {
+				for _, n := range host.Nodes {
+					if n.Role != pds.RoleGate {
+						assert.Empty(t, n.BindAddr, "node %d has role %q and must bind nothing of its own", n.ID, n.Role)
+					}
+				}
+			}
+		})
+	}
+}

cmd/spinifex/cmd/service.go

@@ -104,9 +104,9 @@ type predastoreBind struct {
 // normalization for callers that DIAL predastore, not for the address
 // predastore itself binds to.
 //
-// host_id defaults to 0 when spinifex.toml omits the key, which runs the whole
-// predastore topology in this one process — the single-node deployment. Only a
-// multi-node config names a host (>= 1), selecting just that host's nodes.
+// host_id resolves to 0 when spinifex.toml omits the key. That names no
+// [[host]] of the predastore topology, so the start command rejects it rather
+// than substituting one.
 func derivePredastoreBind(clusterConfig *config.ClusterConfig) (predastoreBind, error) {
 	node := clusterConfig.Node
 	bindKey := "nodes." + node + ".predastore.host"
@@ -138,7 +138,6 @@ var predastoreStartCmd = &cobra.Command{
 		port := viper.GetInt("predastore-port")
 		host := viper.GetString("predastore-host")
 		basePath := viper.GetString("predastore-base-path")
-		debug := viper.GetBool("predastore-debug")
 		hostID := viper.GetInt("predastore-host-id")
 
 		// Derive bind host/port/host-id from spinifex.toml when its path is
@@ -166,9 +165,10 @@ var predastoreStartCmd = &cobra.Command{
 			}
 		}
 
-		// Required, no default
-		if basePath == "" {
-			fmt.Println("Base path is not set")
+		// Required, no default: the host id selects the [[host]] of the
+		// predastore topology this process runs, and nothing can guess it.
+		if hostID <= 0 {
+			fmt.Println("Host ID is not set (--host-id, SPINIFEX_PREDASTORE_HOST_ID or host_id in spinifex.toml)")
 			return
 		}
 
@@ -200,26 +200,19 @@ var predastoreStartCmd = &cobra.Command{
 			return
 		}
 
-		pprofEnabled := viper.GetBool("pprof")
-		pprofOutput := viper.GetString("pprof-output")
-
-		defer initTelemetry("predastore", debug)()
+		defer initTelemetry("predastore", false)()
 
 		service, err := service.New("predastore", &predastore.Config{
 			Port:       port,
 			Host:       host,
 			BasePath:   basePath,
 			ConfigPath: configPath,
-			Debug:      debug,
 			TlsCert:    tlsCert,
 			TlsKey:     tlsKey,
 
 			EncryptionKeyFile: encryptionKeyFile,
 
 			HostID: hostID,
-
-			PprofEnabled:    pprofEnabled,
-			PprofOutputPath: pprofOutput,
 		})
 
 		if err != nil {
@@ -242,7 +235,11 @@ var predastoreStopCmd = &cobra.Command{
 	Run: func(cmd *cobra.Command, args []string) {
 		fmt.Println("Stopping predastore service...")
 
-		service, err := service.New("predastore", &predastore.Config{})
+		// Only the pid directory matters here: Stop finds the running process
+		// through it, and start wrote the file there.
+		service, err := service.New("predastore", &predastore.Config{
+			BasePath: viper.GetString("predastore-base-path"),
+		})
 
 		if err != nil {
 			fmt.Println("Error stopping predastore service:", err)
@@ -1156,7 +1153,7 @@ var qmpCollectorStatusCmd = &cobra.Command{
 // init()); pflag panics on a redefined flag.
 func registerPredastoreNamespacedFlags() {
 	predastoreCmd.PersistentFlags().String("host", "0.0.0.0", "Predastore (S3) host")
-	predastoreCmd.PersistentFlags().String("base-path", "", "Predastore (S3) base path")
+	predastoreCmd.PersistentFlags().String("base-path", "", "Directory holding the predastore service pid file")
 	predastoreCmd.PersistentFlags().String("config-path", "", "Predastore (S3) config path")
 	bindPredastoreNamespacedEnv()
 }
@@ -1175,16 +1172,13 @@ func bindPredastoreNamespacedEnv() {
 	viper.BindPFlag("predastore-config-path", predastoreCmd.PersistentFlags().Lookup("config-path"))
 }
 
-// bindPredastoreCollisionEnv namespaces predastore's port, debug, tls-cert,
-// tls-key, encryption-key-file and host-id keys, which nats and awsgw also
-// bind bare. Each derived env name now matches its own BindEnv target.
+// bindPredastoreCollisionEnv namespaces predastore's port, tls-cert, tls-key,
+// encryption-key-file and host-id keys, which nats and awsgw also bind bare.
+// Each derived env name now matches its own BindEnv target.
 func bindPredastoreCollisionEnv() {
 	viper.BindEnv("predastore-port", "SPINIFEX_PREDASTORE_PORT")
 	viper.BindPFlag("predastore-port", predastoreCmd.PersistentFlags().Lookup("port"))
 
-	viper.BindEnv("predastore-debug", "SPINIFEX_PREDASTORE_DEBUG")
-	viper.BindPFlag("predastore-debug", predastoreCmd.PersistentFlags().Lookup("debug"))
-
 	viper.BindEnv("predastore-tls-cert", "SPINIFEX_PREDASTORE_TLS_CERT")
 	viper.BindPFlag("predastore-tls-cert", predastoreCmd.PersistentFlags().Lookup("tls-cert"))
 
@@ -1259,13 +1253,10 @@ func init() {
 	// Predastore Port
 	predastoreCmd.PersistentFlags().Int("port", 8443, "Predastore (S3) port")
 
-	// Predastore host, base-path, config-path (namespaced viper keys —
+	// Predastore host, config-path (namespaced viper keys —
 	// see registerPredastoreNamespacedFlags doc comment)
 	registerPredastoreNamespacedFlags()
 
-	// Predastore Debug
-	predastoreCmd.PersistentFlags().Bool("debug", false, "Predastore (S3) debug")
-
 	// Predastore TLS Cert
 	predastoreCmd.PersistentFlags().String("tls-cert", "", "Predastore (S3) TLS certificate")
 
@@ -1276,25 +1267,15 @@ func init() {
 	predastoreCmd.PersistentFlags().String("encryption-key-file", "", "Path to this node's 32-byte AES-256 master key file (required)")
 
 	// Predastore host ID: which [[host]] of the predastore topology this
-	// process is. Default 0 runs every node of the topology in this process,
-	// which is the single-node deployment. Multi-node deployments set the
-	// host's real ID (>= 1) via SPINIFEX_PREDASTORE_HOST_ID or --host-id.
-	predastoreCmd.PersistentFlags().Int("host-id", 0, "Predastore cluster host ID (0 = run the whole topology in this process, >= 1 = this host only)")
+	// process runs. Required, and it must name a host the predastore config
+	// declares; the default 0 names none. Set it via spinifex.toml's host_id,
+	// SPINIFEX_PREDASTORE_HOST_ID or --host-id.
+	predastoreCmd.PersistentFlags().Int("host-id", 0, "Predastore cluster host ID, naming a [[host]] in the predastore config (required)")
 
-	// Namespaced viper keys for port/debug/tls-cert/tls-key/encryption-key-file/host-id
+	// Namespaced viper keys for port/tls-cert/tls-key/encryption-key-file/host-id
 	// (see bindPredastoreCollisionEnv doc comment)
 	bindPredastoreCollisionEnv()
 
-	// Predastore CPU Profiling
-	predastoreCmd.PersistentFlags().Bool("pprof", false, "Enable CPU profiling (also via PPROF_ENABLED=1)")
-	viper.BindEnv("pprof", "PPROF_ENABLED")
-	viper.BindPFlag("pprof", predastoreCmd.PersistentFlags().Lookup("pprof"))
-
-	// Predastore CPU Profile Output Path
-	predastoreCmd.PersistentFlags().String("pprof-output", "/tmp/predastore-cpu.prof", "CPU profile output path")
-	viper.BindEnv("pprof-output", "PPROF_OUTPUT")
-	viper.BindPFlag("pprof-output", predastoreCmd.PersistentFlags().Lookup("pprof-output"))
-
 	predastoreCmd.AddCommand(predastoreStartCmd)
 	predastoreCmd.AddCommand(predastoreStopCmd)
 	predastoreCmd.AddCommand(predastoreStatusCmd)

cmd/spinifex/cmd/service_bind_collision_test.go

@@ -21,19 +21,18 @@ func bindAllServiceCollisionEnv(t *testing.T) {
 	bindAwsgwCollisionEnv()
 }
 
-// TestSubcommandDebugKeysResolveIndependently proves predastore, nats and
-// awsgw each resolve their own SPINIFEX_<SERVICE>_DEBUG value instead of
-// sharing one last-registration-wins "debug" viper key.
+// TestSubcommandDebugKeysResolveIndependently proves nats and awsgw each
+// resolve their own SPINIFEX_<SERVICE>_DEBUG value instead of sharing one
+// last-registration-wins "debug" viper key. Predastore has no debug flag: it
+// logs at whatever level the predastore library is built to.
 func TestSubcommandDebugKeysResolveIndependently(t *testing.T) {
 	t.Cleanup(func() { viper.Reset() })
 	viper.Reset()
 	bindAllServiceCollisionEnv(t)
 
-	t.Setenv("SPINIFEX_PREDASTORE_DEBUG", "true")
 	t.Setenv("SPINIFEX_NATS_DEBUG", "false")
 	t.Setenv("SPINIFEX_AWSGW_DEBUG", "true")
 
-	assert.True(t, viper.GetBool("predastore-debug"))
 	assert.False(t, viper.GetBool("nats-debug"))
 	assert.True(t, viper.GetBool("awsgw-debug"))
 }
@@ -99,7 +98,6 @@ func TestGenericSpinifexEnvDoesNotShadowNamespacedKeys(t *testing.T) {
 	t.Setenv("SPINIFEX_HOST", "255.255.255.255")
 	t.Setenv("SPINIFEX_PORT", "1")
 
-	t.Setenv("SPINIFEX_PREDASTORE_DEBUG", "false")
 	t.Setenv("SPINIFEX_NATS_DEBUG", "false")
 	t.Setenv("SPINIFEX_AWSGW_DEBUG", "false")
 	t.Setenv("SPINIFEX_NATS_HOST", "10.0.0.9")
@@ -107,7 +105,6 @@ func TestGenericSpinifexEnvDoesNotShadowNamespacedKeys(t *testing.T) {
 	t.Setenv("SPINIFEX_PREDASTORE_PORT", "8443")
 	t.Setenv("SPINIFEX_NATS_PORT", "4222")
 
-	assert.False(t, viper.GetBool("predastore-debug"), "generic SPINIFEX_DEBUG must not shadow predastore-debug")
 	assert.False(t, viper.GetBool("nats-debug"), "generic SPINIFEX_DEBUG must not shadow nats-debug")
 	assert.False(t, viper.GetBool("awsgw-debug"), "generic SPINIFEX_DEBUG must not shadow awsgw-debug")
 	assert.Equal(t, "10.0.0.9", viper.GetString("nats-service-host"), "generic SPINIFEX_HOST must not shadow nats-service-host")

cmd/spinifex/cmd/templates/predastore-multinode.toml

@@ -1,13 +1,10 @@
 # Predastore S3-compatible storage configuration file in TOML format.
 # Generated for multi-node cluster
-version = "5"
+version = 1
 region = "{{.Region}}"
 
-# The gateway listen address comes from the daemon CLI options: a top-level
-# host key would collide with the [[host]] topology tables below.
-
-debug = false
-
+# Reed-Solomon erasure code. data + parity must not exceed the number of blob
+# nodes in the cluster, and each machine contributes exactly one.
 [rs]
 {{- if eq (len .Nodes) 2}}
 data = 1
@@ -17,46 +14,57 @@ data = 2
 parity = 1
 {{- end}}
 
-# One host per machine: a predastore process owning a socket and a data
-# directory. bind_addr binds every interface; public_addr is what the other
-# hosts dial.
-{{- range .Nodes}}
+# One host per machine: a predastore process owning a data directory, a TLS
+# identity and the nodes pinned to it. addr is what the other hosts dial, so it
+# carries no port and names one machine.
+#
+# bind_addr is the cluster plane: raft consensus and blob shard traffic listen
+# there and nowhere else. It is the address peers dial rather than a wildcard,
+# so on a multi-homed machine replication never reaches the public interface.
+# The gate below binds the public plane separately.
+#
+# Each machine runs a gate for the S3 API, a blob node for erasure-coded object
+# shards and a meta node for the Raft quorum over global state. Ports are
+# unique within a host, so every machine uses the same three.
+{{- range $host := .Nodes}}
 
 [[host]]
-id = {{.ID}}
-bind_addr = "0.0.0.0:6660"
-public_addr = "{{.Host}}:6660"
+id = {{$host.ID}}
+bind_addr = "{{$host.Host}}"
+addr = "{{$host.Host}}"
 data_dir = "{{$.PredastoreDataDir}}"
-{{- end}}
+tls_cert = "{{$.ConfigDir}}/server.pem"
+tls_key = "{{$.ConfigDir}}/server.key"
+encryption_key = "{{$.ConfigDir}}/predastore/encryption.key"
+{{- range $.ClusterNodes}}
+{{- if eq .HostID $host.ID}}
 
-# Nodes are roles pinned to a host: each machine carries one shard-storage
-# node for erasure-coded object shards and one state replica for the Raft
-# quorum over global state.
-{{- range .ClusterNodes}}
-
-[[node]]
+[[host.node]]
 id = {{.ID}}
-host_id = {{.HostID}}
 role = "{{.Role}}"
+port = {{.Port}}
+{{- if eq .Role "gate"}}
+# S3 is the public plane and answers on every interface; the host's bind_addr
+# above keeps raft and blob traffic off them. Only a gate may set this.
+bind_addr = "0.0.0.0"
+{{- end}}
+{{- end}}
+{{- end}}
 {{- end}}
 
-[[buckets]]
+[[bucket]]
 name = "predastore"
 region = "{{.Region}}"
-type = "distributed"
 public = false
-encryption = ""
 account_id = "000000000000"
 
 {{- if .NorthstarAccessKey}}
 # Northstar DNS zone storage. Read-only for the northstar service; the control
 # plane writes zone files with the system credentials.
-[[buckets]]
+[[bucket]]
 name = "{{.NorthstarBucket}}"
 region = "{{.Region}}"
-type = "distributed"
 public = false
-encryption = ""
 account_id = "000000000000"
 {{- end}}
 
@@ -101,7 +109,6 @@ burst = 200
 [iam]
 nats_url = "nats://{{if eq .BindIP "0.0.0.0"}}localhost{{else}}{{.BindIP}}{{end}}:4222"
 nats_token = "{{.NatsToken}}"
-nats_ca_cert = "{{.ConfigDir}}/ca.pem"
 master_key_path = "{{.ConfigDir}}/master.key"
 access_keys_bucket = "spinifex-iam-access-keys"
 {{- if .CompactionIntervalSeconds}}

cmd/spinifex/cmd/templates/predastore.toml

@@ -1,86 +1,77 @@
 # Predastore S3-compatible storage configuration file in TOML format.
-# MULTI NODE used for testing
-version = "5"
+# Single node: one host running every node in one process.
+version = 1
 region = "{{.Region}}"
 
-# The gateway listen address comes from the daemon CLI options: a top-level
-# host key would collide with the [[host]] topology table below.
-
-# Enable verbose debugging
-debug = false
-
-# Optionally configure base-path if full directory path not defined for "fs" buckets
-# base_path = "/home/predastore/data/"
-
-# Specify reed solomon encoding - 
-# RS(2,1), 2 data, 1 parity. Minimum
-# RS(3,2), 3 data, 2 parity. Recommended for production
+# Reed-Solomon erasure code. data + parity must not exceed the number of blob
+# nodes in the cluster.
+#
+# RS(1,0) is one shard on one node: no split, no parity encode and no fan-out.
+# Every shard of a wider code would land on this one machine anyway, so parity
+# buys nothing a second copy of the same disk failure would not take with it.
+# Redundancy here is the pool's job — ZFS raidz or a mirror under the data
+# directory. Multi-node installs get a real erasure code from the multi-node
+# template: RS(2,1) minimum, RS(3,2) recommended.
 [rs]
-data = 2
-parity = 1
+data = 1
+parity = 0
 
-# A host is one predastore process: it owns a socket and a data directory,
-# and runs the nodes pinned to it. A single-node deployment declares one host
-# and colocates every node in it, so all intra-cluster traffic goes over the
-# in-process pipe and no socket is bound. bind_addr/public_addr are unused
-# while everything is colocated; they only matter once nodes are split across
-# processes.
+# A host is one predastore process: it owns a data directory, a TLS identity
+# and the nodes pinned to it. Everything is colocated here, so the nodes talk
+# over the in-process pipe and only the gate binds a socket.
+#
+# bind_addr is the cluster plane — raft and blob traffic. A single-host install
+# opens no socket for it at all, but it is set to the loopback address rather
+# than left wide, so adding a second host later cannot quietly publish
+# replication to every interface.
 [[host]]
 id = 1
-bind_addr = "127.0.0.1:6660"
-public_addr = "127.0.0.1:6660"
+bind_addr = "127.0.0.1"
+addr = "127.0.0.1"
 data_dir = "{{.PredastoreDataDir}}"
+tls_cert = "{{.ConfigDir}}/server.pem"
+tls_key = "{{.ConfigDir}}/server.key"
+encryption_key = "{{.ConfigDir}}/predastore/encryption.key"
 
-# Shard nodes: the distributed hash ring places object shards across these.
-# RS(2,1) needs 3.
-[[node]]
+# The gate serves the S3 API. Without one the host answers no S3 request.
+# S3 is the public plane, so it binds every interface while the host's
+# bind_addr above keeps everything else private. Only a gate may set this.
+[[host.node]]
 id = 1
-host_id = 1
-role = "shard-storage"
+role = "gate"
+port = 8443
+bind_addr = "0.0.0.0"
 
-[[node]]
+# One blob node holds every object. Extra blob nodes would only hash-partition
+# the keys across more segment files and more badger indexes on the same drive,
+# which costs memory and compaction work and buys no redundancy. Colocated
+# nodes bind no socket, but each still needs a port unique within its host.
+[[host.node]]
 id = 2
-host_id = 1
-role = "shard-storage"
+role = "blob"
+port = 6660
 
-[[node]]
+# One meta node is the whole Raft quorum. Replicas here would share the disk
+# they are meant to survive, so each commit would pay their fsyncs for a quorum
+# that fails together.
+[[host.node]]
 id = 3
-host_id = 1
-role = "shard-storage"
+role = "meta"
+port = 7660
 
-# State replicas: the Raft quorum over global state (buckets, object index).
-[[node]]
-id = 4
-host_id = 1
-role = "state-replica"
-
-[[node]]
-id = 5
-host_id = 1
-role = "state-replica"
-
-[[node]]
-id = 6
-host_id = 1
-role = "state-replica"
-
-[[buckets]]
+[[bucket]]
 name = "predastore"
 region = "{{.Region}}"
-type = "distributed"
 public = false
-encryption = ""
 account_id = "000000000000"
 
 {{- if .NorthstarAccessKey}}
 # Northstar DNS zone storage. Read-only for the northstar service; the control
 # plane writes zone files with the system credentials.
-[[buckets]]
+[[bucket]]
 name = "{{.NorthstarBucket}}"
 region = "{{.Region}}"
-type = "distributed"
 public = false
-encryption = ""
 account_id = "000000000000"
 {{- end}}
 
@@ -125,11 +116,10 @@ burst = 200
 [iam]
 nats_url = "nats://{{if eq .BindIP "0.0.0.0"}}localhost{{else}}{{.BindIP}}{{end}}:4222"
 nats_token = "{{.NatsToken}}"
-nats_ca_cert = "{{.ConfigDir}}/ca.pem"
 master_key_path = "{{.ConfigDir}}/master.key"
 access_keys_bucket = "spinifex-iam-access-keys"
 {{- if .CompactionIntervalSeconds}}
 
 [compaction]
 interval_seconds = {{.CompactionIntervalSeconds}}
-{{- end}}
+{{- end}}

cmd/spinifex/cmd/templates/spinifex.toml

@@ -127,10 +127,9 @@ bridge_mode = "{{.BridgeMode}}"
 
 [nodes.{{.Node}}.predastore]
 # Wildcard rather than the bind IP: S3 is a public service and has to answer on
-# the wan plane, and predastore-start.sh scrapes this line into
-# SPINIFEX_PREDASTORE_HOST, which outranks the host set in predastore.toml. Only
-# the S3 API moves — the [[db]] and [[nodes]] peer addresses in
-# predastore-multinode.toml stay on the bind address.
+# the wan plane. This settles the gate node's own bind address and port, and
+# nothing else: raft and blob traffic bind [[host]].bind_addr from
+# predastore.toml, which is the private plane and stays there.
 #
 # This doubles as the local dial target, and 0.0.0.0 is not one: config loading
 # rewrites it to 127.0.0.1 for this node, and admin.DialTarget does the same for

cmd/spinifex/cmd/viperblock_region_test.go

@@ -36,7 +36,7 @@ func TestPredastoreTemplateBucketsUseConfiguredRegion(t *testing.T) {
 		Buckets []struct {
 			Name   string `toml:"name"`
 			Region string `toml:"region"`
-		} `toml:"buckets"`
+		} `toml:"bucket"`
 	}
 	require.NoError(t, toml.Unmarshal([]byte(content), &parsed))
 

docs/COMMANDS.md

@@ -10,7 +10,7 @@ Service lifecycle commands for starting, stopping, and checking status of all Sp
 
 | Command | Flags | Description |
 |---------|-------|-------------|
-| `spx service predastore start` | `--port` (default: 8443), `--host` (default: 0.0.0.0), `--base-path`, `--config-path`, `--debug`, `--tls-cert`, `--tls-key`, `--encryption-key-file`, `--host-id` (default: 0 = run the whole topology in this process), `--pprof`, `--pprof-output` | Creates predastore service instance with S3-compatible storage backend → starts service. When `--config`/`SPINIFEX_CONFIG_PATH` points at a cluster spinifex.toml, `--host`/`--port`/`--host-id` default to that node's `[nodes.<node>.predastore]` section instead of the flag defaults above; an explicit flag or `SPINIFEX_PREDASTORE_*` env var still overrides it. `host_id` absent from spinifex.toml resolves to 0, running the whole topology in this process. |
+| `spx service predastore start` | `--port` (default: 8443, the host's S3 gate port), `--host` (default: 0.0.0.0, the address the host's sockets bind), `--base-path` (directory holding the service pid file), `--config-path` (required), `--tls-cert` (required), `--tls-key` (required), `--encryption-key-file` (required), `--host-id` (required; names a `[[host]]` in the predastore config) | Creates predastore service instance with S3-compatible storage backend → starts service, serving that host's nodes and the S3 gate among them until the process is signalled. When `--config`/`SPINIFEX_CONFIG_PATH` points at a cluster spinifex.toml, `--host`/`--port`/`--host-id` default to that node's `[nodes.<node>.predastore]` section instead of the flag defaults above; an explicit flag or `SPINIFEX_PREDASTORE_*` env var still overrides it. `host_id` absent from spinifex.toml resolves to 0, which names no host and is rejected. |
 | `spx service predastore stop` | — | Stops the predastore service |
 | `spx service predastore status` | — | Reports predastore service status |
 | `spx service viperblock start` | `--s3-host` (default: 0.0.0.0:8443), `--s3-bucket` (default: predastore), `--s3-region` (default: ap-southeast-2), `--plugin-path` (auto-detected via `nbdkit --dump-config plugindir`; typically `/usr/lib/x86_64-linux-gnu/nbdkit/plugins/nbdkit-viperblock-plugin.so` on amd64, overridable via `SPINIFEX_VIPERBLOCK_PLUGIN_PATH` in `/etc/spinifex/systemd.env`) | Loads cluster config → connects to NATS and Predastore → starts viperblock block storage service with NBD plugin |

docs/DESIGN.md

@@ -238,6 +238,22 @@ Object storage used for:
 - SSH public key storage (`/bucket/ec2/{key-name}.pub`)
 - Volume metadata and configuration
 
+Spinifex runs Predastore in-process: `spx service predastore start` loads `/etc/spinifex/predastore/predastore.toml`, selects the `[[host]]` naming this machine, and serves it until the process is signalled. The topology is written by `spx admin init` / `join` from the servers that form the cluster.
+
+One `[[host]]` is one process. The nodes pinned to it under `[[host.node]]` carry the roles:
+
+| Role | Port | Responsibility |
+|------|------|----------------|
+| `gate` | TCP 8443 | The S3 API. Holds no cluster state; dials the meta and blob nodes to serve a request. |
+| `blob` | UDP 6660 | Erasure-coded object shards, encrypted at rest under this host's master key. |
+| `meta` | UDP 7660 | Raft replica of global state — buckets and the object index. |
+
+Ports are unique within a host but repeat across the cluster, so a multi-node cluster puts one node of each role on every machine and every machine uses the same three ports. Nodes sharing a host talk over an in-process pipe and bind no socket; nodes on other hosts are reached over QUIC, authenticated by the cluster CA.
+
+A single-node install is the same arrangement with one host: seven nodes — a gate, three blob and three meta — colocated in one process, needing seven distinct ports (8443, 6660–6662, 7660–7662) to stay unique within the host even though only the gate opens a listener. It has no code path of its own.
+
+The daemon reports the topology on `spinifex.storage.config` by parsing the config file. It does not connect to Predastore to do so, and no Predastore node exposes a status endpoint.
+
 ## Configuration
 
 Cluster configuration (`spinifex/config/config.go`):

docs/admin/update/README.md

@@ -32,6 +32,9 @@ Updating Spinifex is the same command used to install it. The installer detects
 
 For operators who want to review migrations before they are applied, a manual upgrade path is also supported.
 
+> [!WARNING]
+> **Object storage is exempt when upgrading from v1.15.0 or earlier to v1.16.0.** Predastore's configuration schema and on-disk layout changed with the object storage cutover in v1.16.0, and no migration converts an installation from before it. `spx admin upgrade` will not report anything pending for `predastore.toml`, and updating the binary over such an installation leaves Predastore unable to start. Those clusters have to be re-initialised from scratch, which discards their stored objects — export anything you need first. Upgrades between v1.16.0 and later releases are unaffected.
+
 ## Instructions
 
 ## Step 1. Re-run the Installer

docs/aws-model-operation-coverage.md

@@ -12,7 +12,7 @@ Implemented means a modelled operation is registered to a real handler. Stub and
 | ecs | 2014-11-13 | 56 | 39 | 31 | 5 | 0 | 20 | 3 |
 | elasticloadbalancingv2 | 2015-12-01 | 46 | 37 | 33 | 0 | 0 | 13 | 4 |
 | iam | 2010-05-08 | 159 | 75 | 75 | 0 | 0 | 84 | 0 |
-| rds | 2014-10-31 | 162 | 40 | 24 | 0 | 12 | 126 | 4 |
+| rds | 2014-10-31 | 162 | 41 | 24 | 0 | 12 | 126 | 5 |
 | s3 | 2006-03-01 | 99 | — | — | — | — | — | — |
 | sts | 2011-06-15 | 8 | 4 | 4 | 0 | 0 | 4 | 0 |
 
@@ -527,9 +527,9 @@ None.
 
 </details>
 
-<details><summary>Registered outside the pinned model (4)</summary>
+<details><summary>Registered outside the pinned model (5)</summary>
 
-`GetDBBootstrapConfig`, `PollDBCommands`, `RegisterDBInstance`, `SubmitDBStateChange`
+`AcknowledgeDBBootstrap`, `GetDBBootstrapConfig`, `PollDBCommands`, `RegisterDBInstance`, `SubmitDBStateChange`
 
 
 </details>

docs/identity/iam-groups/README.md

@@ -1,6 +1,6 @@
 ---
-title: "IAM Groups"
-description: "Organise IAM users into groups and manage permissions once for the whole team."
+title: "AWS IAM Groups for Shared User Permissions"
+description: "Organise AWS IAM users into groups, manage membership, and attach managed or inline policies once for the whole team instead of granting them user by user."
 category: "Identity"
 tags:
   - iam
@@ -15,9 +15,9 @@ resources:
     url: "https://docs.aws.amazon.com/IAM/latest/UserGuide/id_groups.html"
 ---
 
-# IAM Groups
+# AWS IAM Groups for Shared User Permissions
 
-> Organise IAM users into groups and manage permissions once for the whole team.
+> Organise AWS IAM users into groups, manage membership, and attach managed or inline policies once for the whole team instead of granting them user by user.
 
 ## Table of Contents
 
@@ -33,11 +33,11 @@ resources:
 
 ## Overview
 
-A group is a collection of IAM users. Policies attached to a group apply to every member, so instead of attaching the same policy to each developer individually, you attach it once to a `developers` group and manage membership.
+A group is a collection of [IAM users](/docs/iam-users-and-policies). Policies attached to a group apply to every member, so instead of attaching the same policy to each developer individually, you attach it once to a `developers` group and manage membership.
 
 Members inherit both the group's **attached managed policies** and its **inline policies**, combined with any policies on the user itself. Policy evaluation is the same as everywhere else: an explicit `Deny` in any applicable policy wins, and anything not allowed is denied.
 
-Groups cannot be nested, and a group is not a principal — it cannot sign requests or own access keys. Like all IAM resources, groups are scoped to the account that creates them.
+Groups cannot be nested, and a group is not a principal — it cannot sign requests or own access keys. The identity you assume to get credentials is an [IAM role](/docs/iam-roles-and-instance-profiles), not a group. Like all IAM resources, groups are scoped to the account that creates them.
 
 ## Prerequisites
 

docs/identity/iam-roles-and-instance-profiles/README.md

@@ -1,6 +1,6 @@
 ---
-title: "IAM Roles and Instance Profiles"
-description: "Create IAM roles with trust policies, wrap them in instance profiles, and grant EC2 instances credentials without static keys."
+title: "AWS IAM Roles and Instance Profiles for EC2"
+description: "Create AWS IAM roles with trust policies, wrap them in instance profiles, and launch EC2 instances that receive credentials with no static access keys at all."
 category: "Identity"
 tags:
   - iam
@@ -18,9 +18,9 @@ resources:
     url: "https://docs.aws.amazon.com/IAM/latest/UserGuide/id_roles_use_switch-role-ec2_instance-profiles.html"
 ---
 
-# IAM Roles and Instance Profiles
+# AWS IAM Roles and Instance Profiles for EC2
 
-> Create IAM roles with trust policies, wrap them in instance profiles, and grant EC2 instances credentials without static keys.
+> Create AWS IAM roles with trust policies, wrap them in instance profiles, and launch EC2 instances that receive credentials with no static access keys at all.
 
 ## Table of Contents
 
@@ -39,7 +39,7 @@ resources:
 
 ## Overview
 
-A role is an IAM identity with permissions but no long-lived credentials. Instead of an access key pair, a role has a **trust policy** declaring who may assume it; whoever assumes the role receives short-lived, auto-rotating STS credentials.
+A role is an IAM identity with permissions but no long-lived credentials. Unlike an [IAM group](/docs/iam-groups), which only bundles users together, a role is a principal in its own right: instead of an access key pair it has a **trust policy** declaring who may assume it, and whoever assumes the role receives short-lived, auto-rotating STS credentials.
 
 An **instance profile** is the container that binds a role to an EC2 instance. An instance launched with a profile gets the role's credentials delivered through [IMDS](/docs/imds) — no static keys baked into the image, no `~/.aws/credentials` inside the guest.
 

docs/identity/iam-users-and-policies/README.md

@@ -1,6 +1,6 @@
 ---
-title: "IAM Users and Policies"
-description: "Create IAM users, manage access keys, and control permissions with policies."
+title: "AWS IAM Users, Policies, and Access Keys"
+description: "Create AWS IAM users, issue and rotate access keys, and write and attach JSON policies that control exactly what each user can do in a Spinifex account."
 category: "Identity"
 tags:
   - iam
@@ -17,9 +17,9 @@ resources:
     url: "https://docs.aws.amazon.com/IAM/latest/UserGuide/reference_policies_elements.html"
 ---
 
-# IAM Users and Policies
+# AWS IAM Users, Policies, and Access Keys
 
-> Create IAM users, manage access keys, and control permissions with policies.
+> Create AWS IAM users, issue and rotate access keys, and write and attach JSON policies that control exactly what each user can do in a Spinifex account.
 
 ## Table of Contents
 
@@ -119,7 +119,7 @@ aws iam delete-user --user-name alice
 
 ## Access Keys
 
-Access keys are how users authenticate with the AWS CLI. Each user can have up to **2 access keys** at a time, allowing key rotation without downtime.
+Access keys are how users authenticate with the AWS CLI. Each user can have up to **2 access keys** at a time, allowing key rotation without downtime. For credentials that expire on their own instead of a long-lived key pair, see [STS and Temporary Credentials](/docs/sts). Workloads running inside an EC2 instance should not carry a user's access key at all — they pick up role credentials automatically through [IMDS](/docs/imds).
 
 ### Create an Access Key
 

docs/identity/imds/README.md

@@ -1,6 +1,6 @@
 ---
-title: "IMDS (Instance Metadata Service)"
-description: "Access instance metadata, user data, and IAM role credentials from inside a guest VM with IMDSv2."
+title: "AWS IMDS: Instance Metadata Service (IMDSv2)"
+description: "Query AWS instance metadata, read user data, and fetch short-lived IAM role credentials from inside a guest VM using IMDSv2 session tokens on Spinifex."
 category: "Identity"
 tags:
   - imds
@@ -18,9 +18,9 @@ resources:
     url: "https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/configuring-instance-metadata-service.html"
 ---
 
-# Instance Metadata Service (IMDS)
+# AWS IMDS: Instance Metadata Service (IMDSv2)
 
-> Access instance metadata, user data, and IAM role credentials from inside a guest VM with IMDSv2.
+> Query AWS instance metadata, read user data, and fetch short-lived IAM role credentials from inside a guest VM using IMDSv2 session tokens on Spinifex.
 
 ## Table of Contents
 
@@ -153,7 +153,7 @@ Returns the decoded user data, or 404 if the instance was launched without any.
 
 ## IAM Role Credentials
 
-Instances launched with an IAM instance profile get short-lived, auto-rotating credentials through IMDS — no static keys baked into the image. Creating roles and instance profiles is covered in [IAM Roles and Instance Profiles](/docs/iam-roles-and-instance-profiles).
+Instances launched with an IAM instance profile get short-lived, auto-rotating credentials through IMDS — no static keys baked into the image. Creating roles and instance profiles is covered in [IAM Roles and Instance Profiles](/docs/iam-roles-and-instance-profiles). These are the same temporary credentials [STS](/docs/sts) issues, delivered to the guest automatically instead of through an explicit `assume-role` call.
 
 ```bash
 # Discover the role name

docs/identity/sts/README.md

@@ -1,6 +1,6 @@
 ---
-title: "STS and Temporary Credentials"
-description: "Mint short-lived credentials with STS — assume roles, issue session tokens, and federate Kubernetes workloads with OIDC."
+title: "AWS STS Temporary Credentials and AssumeRole"
+description: "Use AWS STS to issue temporary credentials: assume an IAM role, request a session token, or federate Kubernetes workloads with OIDC web identity on Spinifex."
 category: "Identity"
 tags:
   - sts
@@ -18,9 +18,9 @@ resources:
     url: "https://docs.aws.amazon.com/IAM/latest/UserGuide/id_credentials_temp.html"
 ---
 
-# STS and Temporary Credentials
+# AWS STS Temporary Credentials and AssumeRole
 
-> Mint short-lived credentials with STS — assume roles, issue session tokens, and federate Kubernetes workloads with OIDC.
+> Use AWS STS to issue temporary credentials: assume an IAM role, request a session token, or federate Kubernetes workloads with OIDC web identity on Spinifex.
 
 ## Table of Contents
 
@@ -43,7 +43,7 @@ STS (Security Token Service) issues **temporary credentials**: an `ASIA`-prefixe
 Spinifex supports the three main ways to obtain them:
 
 - **`assume-role`** — exchange your IAM user credentials for a [role's](/docs/iam-roles-and-instance-profiles) permissions, gated by the role's trust policy.
-- **`get-session-token`** — get a time-boxed copy of your own user's permissions.
+- **`get-session-token`** — get a time-boxed copy of your own [IAM user's](/docs/iam-users-and-policies) permissions.
 - **`assume-role-with-web-identity`** — exchange an OIDC ID token (a Kubernetes ServiceAccount token) for role credentials, with no IAM credentials at all.
 
 EC2 instances get role credentials a fourth way, automatically through [IMDS](/docs/imds) — no STS call needed in the guest.

docs/install/install-multi-node/README.md

@@ -26,6 +26,7 @@ resources:
 ## Table of Contents
 
 - [Overview](#overview)
+- [Object Storage Layout](#object-storage-layout)
 - [Prerequisites](#prerequisites)
 - [Instructions](#instructions)
 - [Converting ISO-Installed Nodes](#converting-iso-installed-nodes)
@@ -53,6 +54,28 @@ Servers beyond the third join as compute nodes and do not run a database, which
 - UDP port 6081 open between hosts (Geneve tunnels)
 - TCP ports 4222, 4248, 6641, 6642 open between hosts (NATS, OVN)
 - TCP ports 6643, 6644 open between database servers (OVN database clustering)
+- TCP port 8443 and UDP ports 6660, 7660 open between hosts (Predastore S3, object shards, metadata consensus)
+
+Predastore uses the same three ports on every server, so the surface does not widen as the cluster grows. See [Object Storage Layout](#object-storage-layout) below.
+
+## Object Storage Layout
+
+Predastore is configured for the whole cluster in `/etc/spinifex/predastore/predastore.toml`. Each server is one `[[host]]` — a single Predastore process owning that machine's data directory and TLS identity — carrying three nodes under `[[host.node]]`:
+
+| Role | Port | Purpose |
+|------|------|---------|
+| `gate` | TCP 8443 | Serves the S3 API. Every server runs one, so any of them answers an S3 request. |
+| `blob` | UDP 6660 | Holds erasure-coded object shards. One per machine. |
+| `meta` | UDP 7660 | Member of the Raft quorum over global state — buckets and the object index. |
+
+Ports have to be unique within a host but not across the cluster, so every machine uses the same three. Blob and meta traffic between hosts runs over QUIC, authenticated by the cluster CA; nodes on the same machine talk over an in-process pipe and open no socket, which is why a single-server install listens on 8443 alone.
+
+Reed-Solomon parameters are chosen from the cluster size, since each machine contributes exactly one blob node: two servers get `RS(1,1)`, three or more get `RS(2,1)`. `RS(2,1)` survives the loss of any one server's shards.
+
+You do not configure any of this by hand. `spx admin init` and `spx admin join` build the topology from the servers that actually form the cluster in Step 4, and each machine gets the same file with its own host ID recorded in `spinifex.toml`.
+
+> [!WARNING]
+> **There is no upgrade path onto this layout.** A cluster installed before the object storage cutover cannot be migrated — the on-disk directory structure and the configuration schema both changed, and no migration rewrites either. Such a cluster has to be re-initialised from scratch, which discards its stored objects. Export anything you need first.
 
 ## Prerequisites
 

docs/reference-architectures/RTX-Pro-6000-EKS/README.md

@@ -137,19 +137,19 @@ lspci -d 10de: -nn
 
 **4. Attach Predastore storage**
 
-The X14 has four NVMe drives: two 1.5 TB SSDs (one carries the OS) and two ~880 GB SSDs. The OS occupies its own dedicated NVMe; the remaining three drives are pre-formatted and already mounted at `/mnt/nvme-1`, `/mnt/nvme-2`, and `/mnt/nvme-3`. Predastore is distributed across these three drives — one storage node per physical drive, with Reed–Solomon redundancy so a single drive failure is recoverable.
+The X14 has four NVMe drives: two 1.5 TB SSDs (one carries the OS) and two ~880 GB SSDs. The OS occupies its own dedicated NVMe; the remaining three drives are pre-formatted and already mounted at `/mnt/nvme-1`, `/mnt/nvme-2`, and `/mnt/nvme-3`. Predastore spreads object shards across these three drives — one blob node per physical drive, with Reed–Solomon redundancy so a single drive failure is recoverable.
 
-Relocate the Predastore data directories onto the mounted drives:
+A single-node install is one Predastore host running seven nodes, each with a directory named for its node ID under `/var/lib/spinifex/predastore/cluster`. The three blob nodes are `node-2`, `node-3` and `node-4`, so `nvme-$i` takes `node-$((i + 1))`. The gate keeps no data at all, and the three meta nodes hold only the Raft log for buckets and the object index, which is small enough to leave on the OS drive.
+
+Relocate the blob node directories onto the mounted drives:
 
 ```bash
 sudo systemctl stop spinifex.target
 
 for i in 1 2 3; do
-  sudo mkdir -p /mnt/nvme-$i/nodes /mnt/nvme-$i/db
-  sudo mv /var/lib/spinifex/predastore/distributed/nodes/node-$i /mnt/nvme-$i/nodes/node-$i
-  sudo mv /var/lib/spinifex/predastore/distributed/db/node-$i    /mnt/nvme-$i/db/node-$i
-  sudo ln -s /mnt/nvme-$i/nodes/node-$i /var/lib/spinifex/predastore/distributed/nodes/node-$i
-  sudo ln -s /mnt/nvme-$i/db/node-$i    /var/lib/spinifex/predastore/distributed/db/node-$i
+  node="node-$((i + 1))"
+  sudo mv /var/lib/spinifex/predastore/cluster/$node /mnt/nvme-$i/$node
+  sudo ln -s /mnt/nvme-$i/$node /var/lib/spinifex/predastore/cluster/$node
 done
 
 sudo systemctl start spinifex.target

docs/reference-architectures/SMCI-MIG-orchestration/README.md

@@ -138,7 +138,9 @@ sudo systemctl status spinifex.target
 
 ### 4. Attach Predastore storage
 
-Spinifex distributes the Predastore object storage volume across multiple nodes using Reed–Solomon encoding for data redundancy. This chassis has four NVMe drives — one for the OS, three dedicated to data — so we back each Predastore storage node with its own physical drive, giving us fault-tolerant distributed storage on a single machine — conceptually similar to RAID 5, where data and parity are spread across drives so a single drive failure is recoverable.
+Spinifex spreads Predastore's object shards across multiple blob nodes using Reed–Solomon encoding for data redundancy. This chassis has four NVMe drives — one for the OS, three dedicated to data — so we back each blob node with its own physical drive, giving us fault-tolerant storage on a single machine — conceptually similar to RAID 5, where data and parity are spread across drives so a single drive failure is recoverable.
+
+A single-node install is one Predastore host running seven nodes, each with a directory named for its node ID under `/var/lib/spinifex/predastore/cluster`. The three blob nodes are `node-2`, `node-3` and `node-4`, and they are the ones worth their own drive: the gate keeps no data, and the three meta nodes hold only the Raft log for buckets and the object index.
 
 Confirm drive assignments with `lsblk` before proceeding, as device names vary between systems.
 
@@ -148,20 +150,17 @@ lsblk   # identify the OS drive and the three data drives
 # Stop services so Predastore isn't writing while we relocate its data directories
 sudo systemctl stop spinifex.target
 
-# --- Repeat the block below for each data drive (node-1/nvme-1, node-2/nvme-2, node-3/nvme-3) ---
+# --- Repeat the block below for each data drive (node-2/nvme-1, node-3/nvme-2, node-4/nvme-3) ---
 
 # Mount the physical drive at a stable path
 sudo mkdir -p /mnt/nvme-1
 sudo mount /dev/nvme1n1 /mnt/nvme-1
-sudo mkdir -p /mnt/nvme-1/nodes /mnt/nvme-1/db
 
-# Move Predastore's node data and metadata off the OS drive onto the physical NVMe
-sudo mv /var/lib/spinifex/predastore/distributed/nodes/node-1 /mnt/nvme-1/nodes/node-1
-sudo mv /var/lib/spinifex/predastore/distributed/db/node-1    /mnt/nvme-1/db/node-1
+# Move the blob node's data off the OS drive onto the physical NVMe
+sudo mv /var/lib/spinifex/predastore/cluster/node-2 /mnt/nvme-1/node-2
 
-# Symlink the original paths back so Predastore finds its data unchanged
-sudo ln -s /mnt/nvme-1/nodes/node-1 /var/lib/spinifex/predastore/distributed/nodes/node-1
-sudo ln -s /mnt/nvme-1/db/node-1    /var/lib/spinifex/predastore/distributed/db/node-1
+# Symlink the original path back so Predastore finds its data unchanged
+sudo ln -s /mnt/nvme-1/node-2 /var/lib/spinifex/predastore/cluster/node-2
 
 # Persist the mount across reboots
 echo "/dev/nvme1n1  /mnt/nvme-1  auto  defaults  0  2" | sudo tee -a /etc/fstab

docs/security/malware-protection/README.md

@@ -80,7 +80,7 @@ All Spinifex nodes must include the following in their malware protection scope.
 
 | Path | Contents |
 |------|----------|
-| `/etc/spinifex/` | Cluster config directory: `spinifex.toml`, `predastore.toml`, `nats.conf`. |
+| `/etc/spinifex/` | Cluster config directory: `spinifex.toml`, `awsgw.toml`, `nats.conf`, and `predastore/predastore.toml`. |
 | `/etc/spinifex/ca.pem`, `/etc/spinifex/ca.key` | Cluster root CA certificate and private key (leader only). |
 | `/etc/spinifex/server.pem`, `/etc/spinifex/server.key` | Per-node TLS certificate and key. |
 | `/etc/spinifex/master.key` | Master encryption key that wraps per-volume DEKs and IAM secrets. **Highest-value artifact on the node.** |

docs/security/network-connections/README.md

@@ -85,9 +85,9 @@ The inventory in [§1](#1-inbound-listeners)–[§2](#2-outbound-connections) sa
 | 4432 | Formation server | HTTPS | External (bootstrap only) | Cluster join coordination; active only while a join token is valid. See *Formation port lifecycle* below. | Short-lived bearer token + TLS¹ |
 | 4222 | spinifex-nats (client) | NATS + TLS | Cluster | Internal service bus for EC2/EBS/VPC/S3 handlers | Token + mutual TLS (cluster CA) |
 | 4248 | spinifex-nats (cluster) | NATS + TLS | Cluster | Inter-node NATS federation | Token + mutual TLS (cluster CA) |
-| 8443 | spinifex-predastore | HTTPS | Cluster | S3-compatible object storage (AMIs, snapshots, user objects) | AWS SigV4 + TLS |
-| 6660–6662 | predastore (Raft) | TCP | Cluster | Metadata consensus (3 nodes) | Cluster network only |
-| 9991–9993 | predastore (data shards) | TCP | Cluster | Erasure-coded data shard transport | Cluster network only |
+| 8443 | spinifex-predastore (gate) | HTTPS | Cluster | S3-compatible object storage (AMIs, snapshots, user objects) | AWS SigV4 + TLS |
+| 6660 | predastore (blob node) | QUIC / UDP | Cluster | Erasure-coded object shard transport between hosts. Multi-node clusters only — see *Predastore ports* below. | TLS 1.3, server certificate verified against the cluster CA |
+| 7660 | predastore (meta node) | QUIC / UDP | Cluster | Raft consensus over global state — buckets and the object index — between hosts. Multi-node clusters only. | TLS 1.3, server certificate verified against the cluster CA |
 | 6641 | OVN Northbound DB (client) | OVSDB/TCP | Cluster | Logical network topology consumed by vpcd | Cluster network only; TLS planned |
 | 6642 | OVN Southbound DB (client) | OVSDB/TCP | Cluster | Chassis / port / MAC binding state | Cluster network only; TLS planned |
 | 6643 | OVN Northbound DB (RAFT) | OVSDB/TCP | Cluster | NB database RAFT replication between the 3 quorum nodes | Cluster network only; TLS planned |
@@ -97,6 +97,10 @@ The inventory in [§1](#1-inbound-listeners)–[§2](#2-outbound-connections) sa
 
 ¹ **Formation port lifecycle.** 4432 opens during `spx admin init` / `spx admin join` while a bootstrap token is outstanding and closes once the cluster is formed (token TTL default 30 min, `--token-ttl`). The server presents an ephemeral self-signed cert that pre-dates trust bootstrap, so the joining node does not verify the certificate chain for this single dial. Authenticity rests on the operator supplying the leader address out-of-band plus possession of the bearer token. Document in the security plan so reviewers do not flag 4432 as a persistent open port.
 
+**Predastore ports.** A Predastore cluster is described in `/etc/spinifex/predastore/predastore.toml` as a set of `[[host]]` blocks — one per machine, each running a single process — with the nodes pinned to it declared under `[[host.node]]`. There are three roles: a `gate` serving the S3 API, a `blob` node holding erasure-coded object shards, and a `meta` node in the Raft quorum over global state. Ports must be unique within a host but are not unique across the cluster, so **every machine uses the same three ports** — 8443, 6660 and 7660. It is three fixed ports per machine, not a range, and adding machines does not widen it.
+
+Nodes on the same host talk over an in-process pipe and bind no socket at all, so a single-node install opens only 8443; 6660 and 7660 appear only once a second host exists. The gate is dialled by S3 clients but never by peer nodes, so it binds no QUIC socket of its own — it takes an ephemeral UDP port to dial out with.
+
 **Development-only listeners.** When `dev_networking=true`, QEMU opens arbitrary host TCP ports for SSH port-forwarding into guest VMs. Production installs (the `/etc/spinifex` layout) do not enable this; it must not appear on compliance nodes.
 
 ## 2. Outbound Connections
@@ -114,7 +118,7 @@ Spinifex nodes initiate a small, fixed set of outbound connections.
 | `https://d2yp8ipz5jfqcw.cloudfront.net` | Alpine image for managed HAProxy load-balancer | HTTPS | TLS + checksum verification |
 | `https://install.mulgadc.com/install` | One-shot install telemetry POST on `spx admin init` / `join`. | HTTPS | TLS |
 
-**To peer nodes (cluster-internal):** NATS federation (4248), Predastore S3 (8443), OVN NB/SB (6641/6642) — see [§3](#3-cross-node-internal-connections) for encryption and verification of each. The daemon also probes local Predastore Raft status at `<bind IP>:6660/status` (TLS, cluster CA) and local NATS monitoring at `127.0.0.1:8222/varz` (loopback HTTP).
+**To peer nodes (cluster-internal):** NATS federation (4248), Predastore S3 (8443), OVN NB/SB (6641/6642) — see [§3](#3-cross-node-internal-connections) for encryption and verification of each. The daemon also polls local NATS monitoring at `127.0.0.1:8222/varz` (loopback HTTP). It opens no connection to Predastore for status: the storage topology it reports comes from reading `predastore.toml`, and no Predastore node serves a status endpoint.
 
 **Update checks and metadata.** Spinifex does not check for updates and does not consume a cloud metadata service (`169.254.169.254` is served *by* the cluster to guest VMs). Node software updates come from the operator's OS package channel. The install-telemetry endpoint above is the only vendor-operated destination contacted by a node; closed-egress deployments should disable it and record the opt-out in the security plan.
 
@@ -127,14 +131,14 @@ Control-plane and data-plane traffic between Spinifex nodes, for completeness an
 | Connection | Port(s) | Encryption / Auth | Notes |
 |-----------|---------|-------------------|-------|
 | NATS cluster routes | 4248 | Mutual TLS + cluster token | Full mesh between NATS servers |
-| Predastore S3 | 8443 | TLS + AWS SigV4 | Cross-node object reads/writes |
-| Predastore Raft | 6660–6662 | Cluster network only | Metadata consensus |
-| Predastore shards | 9991–9993 | Cluster network only | Erasure-coded data shards |
+| Predastore S3 (gate) | TCP 8443 | TLS + AWS SigV4 | Cross-node object reads/writes |
+| Predastore blob | UDP 6660 | QUIC with TLS 1.3; server certificate verified against the cluster CA | Erasure-coded object shards. Same port on every machine. |
+| Predastore meta | UDP 7660 | QUIC with TLS 1.3; server certificate verified against the cluster CA | Raft consensus over buckets and the object index. Same port on every machine. |
 | OVN NB/SB (client) | 6641 / 6642 | Cluster network only (TLS planned) | Network control plane; vpcd and ovn-controller dial the quorum |
 | OVN NB/SB (RAFT) | 6643 / 6644 | Cluster network only (TLS planned) | NB/SB database replication across the 3 quorum nodes |
 | OVN tunnels (Geneve) | UDP 6081 | None | Tenant traffic overlay between chassis, inside the cluster subnet |
 
-Nodes **must** sit on a network segment that is not routed to tenant/guest VMs or to the internet. Predastore Raft/shards and OVN DBs are cluster-internal and must not be reachable from anywhere else.
+Nodes **must** sit on a network segment that is not routed to tenant/guest VMs or to the internet. The Predastore blob and meta transports and the OVN DBs are cluster-internal and must not be reachable from anywhere else.
 
 ## 4. Limiting Controls
 
@@ -145,8 +149,8 @@ Default external surface is three listeners — **9999** (AWS API), **3000** (UI
 tcp dport { 22, 9999, 3000 } accept
 
 # Cluster-only: replace 10.0.1.0/24 with your cluster CIDR
-ip saddr 10.0.1.0/24 tcp dport { 4222, 4248, 8443, 6641-6644, 6660-6662, 9991-9993 } accept
-ip saddr 10.0.1.0/24 udp dport 6081 accept
+ip saddr 10.0.1.0/24 tcp dport { 4222, 4248, 8443, 6641-6644 } accept
+ip saddr 10.0.1.0/24 udp dport { 6081, 6660, 7660 } accept
 
 # Default deny
 tcp dport 0-65535 drop
@@ -163,7 +167,7 @@ Every listener and outbound destination is controlled by one of these files. Cha
 |------|------|----------|
 | `/etc/spinifex/spinifex.toml` | `nodes.<node>.{awsgw,nats,predastore,daemon}.host`, `nodes.<node>.vpcd.ovn_{nb,sb}_addr`, `nodes.<node>.daemon.dev_networking` | Per-service bind addresses/ports; dev-mode QEMU port forwarding. |
 | `/etc/spinifex/nats.conf` | `listen`, `cluster.listen`, `cluster.routes`, `http`, `tls`, `cluster.authorization` | NATS client/cluster/monitoring listeners, peer routes, TLS, cluster token. |
-| `/etc/spinifex/predastore.toml` | `[[host]].bind_addr`, `[[host]].public_addr`, `[[node]].role`, `tls.*` | Predastore intra-cluster listener (one socket per host, carrying every node pinned to it), node placement, TLS certs. The S3 listener itself is set by the service's `--host`/`--port`. |
+| `/etc/spinifex/predastore/predastore.toml` | `[[host]].bind_addr`, `[[host]].addr`, `[[host]].tls_cert`, `[[host]].tls_key`, `[[host.node]].role`, `[[host.node]].port` | Predastore host and node layout: `bind_addr` is the address the host's sockets bind, `addr` is the address peer hosts dial it on, and both carry no port — the nodes pinned to the host supply their own. The service's `--host`/`--port` override the bind address and the gate's S3 port. |
 | OVN packages (`ovn-central`, `ovn-host`) | `ovn-nb-db`, `ovn-sb-db` (via `ovs-vsctl set open_vswitch …`) | OVN DB bind addresses. |
 | Spinifex UI service | Built-in defaults: `host = "0.0.0.0"`, `port = 3000`. No `spinifex.toml` block today. | UI listener. |
 | `spx admin init` / `spx admin join` | `--port`, `--token-ttl`, `--no-telemetry` (or `SPX_NO_TELEMETRY=1`) | Formation port, token TTL, telemetry opt-out. |

go.mod

@@ -21,9 +21,9 @@ require (
 	github.com/insomniacslk/dhcp v0.0.0-20260407060928-11b94ed970f2
 	github.com/klauspost/cpuid/v2 v2.4.0
 	github.com/miekg/dns v1.1.72
-	github.com/mulgadc/northstar v1.15.1-0.20260810233215-04fade67fb2c
-	github.com/mulgadc/predastore v1.15.0
-	github.com/mulgadc/viperblock v1.15.0
+	github.com/mulgadc/northstar v1.16.0
+	github.com/mulgadc/predastore v1.16.0
+	github.com/mulgadc/viperblock v1.16.0
 	github.com/nats-io/nats-server/v2 v2.14.4
 	github.com/nats-io/nats.go v1.52.0
 	github.com/opencontainers/runtime-spec v1.3.0

go.sum

@@ -805,7 +805,6 @@ github.com/fatih/color v1.13.0/go.mod h1:kLAiJbzzSOZDVNGyDpeOxJ47H46qBXwg5ILebYF
 github.com/fatih/color v1.15.0/go.mod h1:0h5ZqXfHYED7Bhv2ZJamyIOUej9KtShiJESRwBDUSsw=
 github.com/fatih/color v1.19.0 h1:Zp3PiM21/9Ld6FzSKyL5c/BULoe/ONr9KlbYVOfG8+w=
 github.com/fatih/color v1.19.0/go.mod h1:zNk67I0ZUT1bEGsSGyCZYZNrHuTkJJB+r6Q9VuMi0LE=
-github.com/felixge/httpsnoop v1.0.4 h1:NFTV2Zj1bL4mc9sqWACXbQFVBBg2W3GPvqp8/ESS2Wg=
 github.com/felixge/httpsnoop v1.0.4/go.mod h1:m8KPJKqk1gH5J9DgRY2ASl2lWCfGKXixSwevea8zH2U=
 github.com/felixge/httpsnoop v1.1.0 h1:3YtUj32ZZkqZtt3sZZsClsymw/QDuVfpNhoA31zeORc=
 github.com/felixge/httpsnoop v1.1.0/go.mod h1:Zqxgdd+1Rkcz8euOqdr7lqgCRJztwr5hp9vDSi5UZCE=
@@ -865,7 +864,6 @@ github.com/go-logr/logr v1.2.2/go.mod h1:jdQByPbusPIv2/zmleS9BjJVeZ6kBagPoEUsqbV
 github.com/go-logr/logr v1.3.0/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
 github.com/go-logr/logr v1.4.1/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
 github.com/go-logr/logr v1.4.2/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
-github.com/go-logr/logr v1.4.3 h1:CjnDlHq8ikf6E492q6eKboGOC0T8CDaOvkHCIg8idEI=
 github.com/go-logr/logr v1.4.3/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
 github.com/go-logr/logr v1.4.4 h1:tG4xh9yMsRCAiodLVTxyrkzSZ9+o0L1Kg/+cPVcbP/8=
 github.com/go-logr/logr v1.4.4/go.mod h1:9T104GzyrTigFIr8wt5mBrctHMim0Nb2HLGrmQ40KvY=
@@ -1321,14 +1319,12 @@ github.com/muesli/cancelreader v0.2.2 h1:3I4Kt4BQjOR54NavqnDogx/MIoWBFa0StPA8ELU
 github.com/muesli/cancelreader v0.2.2/go.mod h1:3XuTXfFS2VjM+HTLZY9Ak0l6eUKfijIfMUZ4EgX0QYo=
 github.com/muesli/termenv v0.16.0 h1:S5AlUN9dENB57rsbnkPyfdGuWIlkmzJjbFf0Tf5FWUc=
 github.com/muesli/termenv v0.16.0/go.mod h1:ZRfOIKPFDYQoDFF4Olj7/QJbW60Ol/kL1pU3VfY/Cnk=
-github.com/mulgadc/northstar v1.15.0 h1:DbJE/w3FLk19PNwgaC+Rnu/gP0E5GNMK9EGwR4TPqqU=
-github.com/mulgadc/northstar v1.15.0/go.mod h1:QbP2ydcCbGHidIdp61O2HOcJANFgGetS5zw60mbjxfQ=
-github.com/mulgadc/northstar v1.15.1-0.20260810233215-04fade67fb2c h1:Ia73hhrUvRRV7iEdCczHMhzrXtOJlQRotMN7rcMGFuA=
-github.com/mulgadc/northstar v1.15.1-0.20260810233215-04fade67fb2c/go.mod h1:1tRJZAKsXvsnMaAHAt5za+sDsRpkaoDGH0s8LST2Xyg=
-github.com/mulgadc/predastore v1.15.0 h1:IGQX8bbuCrwU5mC4JyjxgdL8UXGfTA1Cu9g9Kuu8Irw=
-github.com/mulgadc/predastore v1.15.0/go.mod h1:vjad49Pql/NtQLGfAC43SSDBcPLD6ijbqMbddbt0vA4=
-github.com/mulgadc/viperblock v1.15.0 h1:nZ8d8q8lSgMucd43GYgvwpmnCWXN379z6H2MxUHGo9g=
-github.com/mulgadc/viperblock v1.15.0/go.mod h1:/pkP/liHmG+yzojZ3+xSvAcqMHb/zfiEQVsHeK9EPYM=
+github.com/mulgadc/northstar v1.16.0 h1:uX+A7PUwC51MRK6Q8dsEx40U2ZL0S7qz2x7lPTaAykc=
+github.com/mulgadc/northstar v1.16.0/go.mod h1:1tRJZAKsXvsnMaAHAt5za+sDsRpkaoDGH0s8LST2Xyg=
+github.com/mulgadc/predastore v1.16.0 h1:czQBOcl9NiB/V1A3kD5zgOsPePh2FHK94rAE3yHy9eE=
+github.com/mulgadc/predastore v1.16.0/go.mod h1:ph+eASnkMogVZ0cMiO/6aUw1S1+Gl3+NnVtlqCNdBo8=
+github.com/mulgadc/viperblock v1.16.0 h1:S1kT8fQ+Gbmj0I0+CCLXKwf6dMS7GpDZck86iGvLR+A=
+github.com/mulgadc/viperblock v1.16.0/go.mod h1:pyacEYyZvoxYhZbw5wMpYTW2DnDGpLs9Jackgh/CNgk=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822 h1:C3w9PqII01/Oq1c1nUAm88MOHcQC9l5mIlSMApZMrHA=
 github.com/munnerz/goautoneg v0.0.0-20191010083416-a7dc8b61c822/go.mod h1:+n7T8mK8HuQTcFwEeznm/DIxMOiR9yIdICNftLE1DvQ=
 github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
@@ -1631,8 +1627,6 @@ go.opencensus.io v0.24.0/go.mod h1:vNK8G9p7aAivkbmorf4v+7Hgx+Zs0yY+0fOtgBfjQKo=
 go.opentelemetry.io/auto/sdk v1.1.0/go.mod h1:3wSPjt5PWp2RhlCcmmOial7AvC4DQqZb7a7wCow3W8A=
 go.opentelemetry.io/auto/sdk v1.2.1 h1:jXsnJ4Lmnqd11kwkBV2LgLoFMZKizbCi5fNZ/ipaZ64=
 go.opentelemetry.io/auto/sdk v1.2.1/go.mod h1:KRTj+aOaElaLi+wW1kO/DZRXwkF4C5xPbEe3ZiIhN7Y=
-go.opentelemetry.io/contrib/bridges/otelslog v0.19.0 h1:5RgvxieNq9tS3ewrV1vnODvbHPfKUIJcYtF9Cvz+6aQ=
-go.opentelemetry.io/contrib/bridges/otelslog v0.19.0/go.mod h1:iTBIdNwx/xmUhfgJs6+84S4dIK059811cO1eUBjKcHY=
 go.opentelemetry.io/contrib/bridges/otelslog v0.20.0 h1:oEl2Pw/i4OQwhAuda2pAHFAcOMivA+Xa+iTccBfab/g=
 go.opentelemetry.io/contrib/bridges/otelslog v0.20.0/go.mod h1:yMSQaiiq5dpfrSJCYLBcqFeJkFFI67seT4ngvx6jfVo=
 go.opentelemetry.io/contrib/detectors/gcp v1.28.0/go.mod h1:9BIqH22qyHWAiZxQh0whuJygro59z+nbMVuc7ciiGug=
@@ -1670,12 +1664,8 @@ go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.59.0/go.mod h1:
 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.60.0/go.mod h1:69uWxva0WgAA/4bu2Yy70SLDBwZXuQ6PbBpbsa5iZrQ=
 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.61.0/go.mod h1:UHB22Z8QsdRDrnAtX4PntOl36ajSxcdUMt1sF7Y6E7Q=
 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.67.0/go.mod h1:C2NGBr+kAB4bk3xtMXfZ94gqFDtg/GkI7e9zqGh5Beg=
-go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.69.0 h1:8tvICD4vSTOOsNrsI4Ljf6C+6UKvpTEH5XY3JMoyPoo=
-go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.69.0/go.mod h1:z9+yiacE0IHRqM4qFfkbt/JYlmYXgss8GY/jXoNuPJI=
 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0 h1:LMuyCAyfalSjDyjdC65nK6N0zoTT63+E/u95X0JovZI=
 go.opentelemetry.io/contrib/instrumentation/net/http/otelhttp v0.70.0/go.mod h1:085m8qbm4hgc8rZWGDEa4vmyyo2c3nPxUslYUKUIU04=
-go.opentelemetry.io/contrib/instrumentation/runtime v0.69.0 h1:MtkMsuRo3zEXTTMALfyrszwCDZTkB6wolyPjbwFAdq0=
-go.opentelemetry.io/contrib/instrumentation/runtime v0.69.0/go.mod h1:FYTxnpsm+UPD0erZNq20GvnM8T2YQHiHtT2vokdpoac=
 go.opentelemetry.io/contrib/instrumentation/runtime v0.70.0 h1:1+WLVYezXA9tkuVzKQri8zgB1cEIVYKUSoYIRjsBiMU=
 go.opentelemetry.io/contrib/instrumentation/runtime v0.70.0/go.mod h1:rbAXUUXqQDMxpSnmof4VtcZ+7YpZQEtjXSCIfdvR0Go=
 go.opentelemetry.io/otel v0.20.0/go.mod h1:Y3ugLH2oa81t5QO+Lty+zXf8zC9L26ax4Nzoxm/dooo=
@@ -1699,24 +1689,15 @@ go.opentelemetry.io/otel v1.39.0/go.mod h1:kLlFTywNWrFyEdH0oj2xK0bFYZtHRYUdv1Nkl
 go.opentelemetry.io/otel v1.40.0/go.mod h1:IMb+uXZUKkMXdPddhwAHm6UfOwJyh4ct1ybIlV14J0g=
 go.opentelemetry.io/otel v1.42.0/go.mod h1:lJNsdRMxCUIWuMlVJWzecSMuNjE7dOYyWlqOXWkdqCc=
 go.opentelemetry.io/otel v1.43.0/go.mod h1:JuG+u74mvjvcm8vj8pI5XiHy1zDeoCS2LB1spIq7Ay0=
-go.opentelemetry.io/otel v1.44.0 h1:JjwHmHpA4iZ3wBxluu2fbbE7j4kqlE8jXyAyPXH7HqU=
 go.opentelemetry.io/otel v1.44.0/go.mod h1:BMgjTHL9WPRlRjL2oZCBTL4whCGtXch2H4BhOPIAyYc=
 go.opentelemetry.io/otel v1.45.0 h1:pdrWmLHofpubmArBv1LgFSv1Z0Ie/ppdZzu+kUN5EeU=
 go.opentelemetry.io/otel v1.45.0/go.mod h1:XZxIqPapzEYnhNSScF5DIqXhm/rYi0FzCe2XddAwZfQ=
-go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.20.0 h1:rydZ9sxbcFdm/oWrVyfLTjHIygMgv0bEeMd+3B/BvoM=
-go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.20.0/go.mod h1:earQ25dooT0Hhspq59DZ8YCC50jWfOlFEeWoxy/P444=
 go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.21.0 h1:WseeVYf5dJZTsyPiyW5L14k5qsSibqXAMTSiFEDiWr0=
 go.opentelemetry.io/otel/exporters/otlp/otlplog/otlploggrpc v0.21.0/go.mod h1:SiLZnQS6Qk2eCpvr2CH/XMAOa64TWGXxEZJZCpD2Lmc=
-go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.44.0 h1:SUplec5dp06reu1zaXmOXdvqH398taqrDXqUl99jxSc=
-go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.44.0/go.mod h1:ho2g4N+ane+swq5I/VBkKWnRDY4kUINH3FuqyZqX/Ug=
 go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.45.0 h1:klTViGcsvLCd1xN3rZzfZ12NslC/OimbmR+k+A006RI=
 go.opentelemetry.io/otel/exporters/otlp/otlpmetric/otlpmetricgrpc v1.45.0/go.mod h1:jRsK04CWmXuY8A0O+wMpSf+t90RHZ53o5Qmxn2PQPfk=
-go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.44.0 h1:4YsVu3B8+3qtWYYrsUYgn0OG78pN0rnNPRGX4SbokQI=
-go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.44.0/go.mod h1:+wnlSn0mD1ADVMe3v9Z/WIaiz6q6gL2J/ejaAmdmv80=
 go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.45.0 h1:QRefszxJmfPdjXUUm3j6iDzY03mTPXMjqErFqQ67vUg=
 go.opentelemetry.io/otel/exporters/otlp/otlptrace v1.45.0/go.mod h1:Tiz03lTBVBrm7eWZBOidzEaYaJa8tjwGUGv6d8mlTyk=
-go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.44.0 h1:qazEJlUOQzhCpzQpFETGby7EdqjI1wsd0W+6Gg1SCTU=
-go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.44.0/go.mod h1:fOD2Yefuxixkx3ahVNf0O/PERb6r4OlbxfATVnYvzCo=
 go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.45.0 h1:fG5MCxGz8+2VtrN/WgqSpJFctVz24gpxj8CxkKmc8Ww=
 go.opentelemetry.io/otel/exporters/otlp/otlptrace/otlptracegrpc v1.45.0/go.mod h1:BmAYTn+3ysbRe+IU2msxmf5Rx3g6DHvex+tWI3LdhYI=
 go.opentelemetry.io/otel/exporters/prometheus v0.57.0/go.mod h1:QpFWz1QxqevfjwzYdbMb4Y1NnlJvqSGwyuU0B4iuc9c=
@@ -1724,8 +1705,6 @@ go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.29.0/go.mod h1:BLbf7zb
 go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.35.0/go.mod h1:U2R3XyVPzn0WX7wOIypPuptulsMcPDPs/oiSVOMVnHY=
 go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.36.0/go.mod h1:dowW6UsM9MKbJq5JTz2AMVp3/5iW5I/TStsk8S+CfHw=
 go.opentelemetry.io/otel/exporters/stdout/stdoutmetric v1.42.0/go.mod h1:so9ounLcuoRDu033MW/E0AD4hhUjVqswrMF5FoZlBcw=
-go.opentelemetry.io/otel/log v0.20.0 h1:/5i0vuHxCLWUfChWG41K9wkM0jafruPw9NU1/RCJirs=
-go.opentelemetry.io/otel/log v0.20.0/go.mod h1:wOcMcjsZpG8x7Bak7IhSi/lg8wscV2C1VdrKCLPlt0E=
 go.opentelemetry.io/otel/log v0.21.0 h1:SLsVDGmtyBrdw8/a2Z0bOIxou/+bN4z56GebH7T0LvA=
 go.opentelemetry.io/otel/log v0.21.0/go.mod h1:iReetQrZL9Wyg84cCkOoCmqDHS5RCFfyxC7J+r8fn8g=
 go.opentelemetry.io/otel/metric v0.20.0/go.mod h1:598I5tYlH1vzBjn+BTuhzTCSb/9debfNp6R3s7Pr1eU=
@@ -1749,13 +1728,12 @@ go.opentelemetry.io/otel/metric v1.39.0/go.mod h1:jrZSWL33sD7bBxg1xjrqyDjnuzTUB0
 go.opentelemetry.io/otel/metric v1.40.0/go.mod h1:ib/crwQH7N3r5kfiBZQbwrTge743UDc7DTFVZrrXnqc=
 go.opentelemetry.io/otel/metric v1.42.0/go.mod h1:RlUN/7vTU7Ao/diDkEpQpnz3/92J9ko05BIwxYa2SSI=
 go.opentelemetry.io/otel/metric v1.43.0/go.mod h1:RDnPtIxvqlgO8GRW18W6Z/4P462ldprJtfxHxyKd2PY=
-go.opentelemetry.io/otel/metric v1.44.0 h1:1w0gILTcHdr3YI+ixLyjemwrVnsMURbTZFrSYCdDdmc=
 go.opentelemetry.io/otel/metric v1.44.0/go.mod h1:8O7hanEPBNgEMmybD3s2VBKcgWOCsA6tzHBPODAiquo=
 go.opentelemetry.io/otel/metric v1.45.0 h1:7Eg1uH7CJ5cXv9is6tnBe1FI6rj1nwUdbFypRm3br/M=
 go.opentelemetry.io/otel/metric v1.45.0/go.mod h1:HAPbm1nd3p1PmFH7v2dR+6BjXxw+Lq4a2+pndMAm08s=
-go.opentelemetry.io/otel/metric/x v0.66.0 h1:YkCrx1zLOChi9ZcZ6euupOcsgzbVlec7D/xoEU1+cTA=
 go.opentelemetry.io/otel/metric/x v0.66.0/go.mod h1:d1+BDj9t96do0/1LoU1ayfCv79ZgNE41qbhBvnMOBZk=
 go.opentelemetry.io/otel/metric/x v0.67.0 h1:PcicCNZFkZ4bXfSooXdo3WN7RBOVOtjVdo1wD358Uns=
+go.opentelemetry.io/otel/metric/x v0.67.0/go.mod h1:FBjCWZe6wgcqxcMtjdGiClDKXb2YxxXii0CXftE4QtI=
 go.opentelemetry.io/otel/oteltest v0.20.0/go.mod h1:L7bgKf9ZB7qCwT9Up7i9/pn0PWIa9FqQ2IQ8LoxiGnw=
 go.opentelemetry.io/otel/sdk v0.20.0/go.mod h1:g/IcepuwNsoiX5Byy2nNV0ySUF1em498m7hBWC279Yc=
 go.opentelemetry.io/otel/sdk v1.21.0/go.mod h1:Nna6Yv7PWTdgJHVRD9hIYywQBRx7pbox6nwBnZIxl/E=
@@ -1776,17 +1754,13 @@ go.opentelemetry.io/otel/sdk v1.39.0/go.mod h1:vDojkC4/jsTJsE+kh+LXYQlbL8CgrEcwm
 go.opentelemetry.io/otel/sdk v1.40.0/go.mod h1:Ph7EFdYvxq72Y8Li9q8KebuYUr2KoeyHx0DRMKrYBUE=
 go.opentelemetry.io/otel/sdk v1.42.0/go.mod h1:rGHCAxd9DAph0joO4W6OPwxjNTYWghRWmkHuGbayMts=
 go.opentelemetry.io/otel/sdk v1.43.0/go.mod h1:P+IkVU3iWukmiit/Yf9AWvpyRDlUeBaRg6Y+C58QHzg=
-go.opentelemetry.io/otel/sdk v1.44.0 h1:nHYwb9lK+fJPU/dnT6s7W7Z8itMWyqrnVfbheVYrZ58=
 go.opentelemetry.io/otel/sdk v1.44.0/go.mod h1:Osuydd3Se74nqjAKxid74N5eC+jfEqfTegHRnq58oK0=
 go.opentelemetry.io/otel/sdk v1.45.0 h1:4VVSMgQ83dUgW2aoX5f6JgLvHwIvzcuLnF9lUdCSpCw=
 go.opentelemetry.io/otel/sdk v1.45.0/go.mod h1:Sr40LgXV7DsKMMJMKOhUWOgMWTfAaqvm2kF0g7ilwuA=
-go.opentelemetry.io/otel/sdk/log v0.20.0 h1:vM3xI7TQgKPiSghe6urZtAkyFY7SodrSpC83CffDFuY=
-go.opentelemetry.io/otel/sdk/log v0.20.0/go.mod h1:Knej2nmsTUzN79T2eeXdRsjjPcoxoq2pUyUHz9TFyyU=
 go.opentelemetry.io/otel/sdk/log v0.21.0 h1:QsE7XSR0ktQdKmRKGnR+f1ObGF32WG+7MER/P9KgmYc=
 go.opentelemetry.io/otel/sdk/log v0.21.0/go.mod h1:m9mApjCoD2/1QuKCAptjv+BrG9WKOvQLVdNx+iBldTo=
-go.opentelemetry.io/otel/sdk/log/logtest v0.20.0 h1:OqdRZ1guyzamK3M6LlRsmGqRrjkHWw6WZOKKli5ELpg=
-go.opentelemetry.io/otel/sdk/log/logtest v0.20.0/go.mod h1:PuMIlm7zAt7c3z8zfOI5ox4iT1Z87We+PF6YoINux/M=
 go.opentelemetry.io/otel/sdk/log/logtest v0.21.0 h1:X+JBBgKlswCGYsmgL0CnoUUtlE//VB345c84jYAYkdQ=
+go.opentelemetry.io/otel/sdk/log/logtest v0.21.0/go.mod h1:HD1575K8e6sIFBBDd5tZB3t9DlMytWXq9FuR+Y4rfjE=
 go.opentelemetry.io/otel/sdk/metric v1.28.0/go.mod h1:cWPjykihLAPvXKi4iZc1dpER3Jdq2Z0YLse3moQUCpg=
 go.opentelemetry.io/otel/sdk/metric v1.29.0/go.mod h1:6zZLdCl2fkauYoZIOn/soQIDSWFmNSRcICarHfuhNJQ=
 go.opentelemetry.io/otel/sdk/metric v1.30.0/go.mod h1:waS6P3YqFNzeP01kuo/MBBYqaoBJl7efRQHOaydhy1Y=
@@ -1801,7 +1775,6 @@ go.opentelemetry.io/otel/sdk/metric v1.39.0/go.mod h1:xq9HEVH7qeX69/JnwEfp6fVq5w
 go.opentelemetry.io/otel/sdk/metric v1.40.0/go.mod h1:4Z2bGMf0KSK3uRjlczMOeMhKU2rhUqdWNoKcYrtcBPg=
 go.opentelemetry.io/otel/sdk/metric v1.42.0/go.mod h1:Ua6AAlDKdZ7tdvaQKfSmnFTdHx37+J4ba8MwVCYM5hc=
 go.opentelemetry.io/otel/sdk/metric v1.43.0/go.mod h1:C/RJtwSEJ5hzTiUz5pXF1kILHStzb9zFlIEe85bhj6A=
-go.opentelemetry.io/otel/sdk/metric v1.44.0 h1:3LlKgI+VjbVsjNRFZJZAJ30WjXC5VkNRks6si09iEfI=
 go.opentelemetry.io/otel/sdk/metric v1.44.0/go.mod h1:5B5pMARnXxKhltooO4xUuCBorl65a4EpnTalObqOigA=
 go.opentelemetry.io/otel/sdk/metric v1.45.0 h1:oVFszMfyj1Am6s24Vtc7wBb8BKLcwepJjNEYILuiE3o=
 go.opentelemetry.io/otel/sdk/metric v1.45.0/go.mod h1:vUWUxDZvu1WVRj8JA8S0AdhsPrZoDpA2DdZauIh4mDA=
@@ -1826,7 +1799,6 @@ go.opentelemetry.io/otel/trace v1.39.0/go.mod h1:88w4/PnZSazkGzz/w84VHpQafiU4Etq
 go.opentelemetry.io/otel/trace v1.40.0/go.mod h1:zeAhriXecNGP/s2SEG3+Y8X9ujcJOTqQ5RgdEJcawiA=
 go.opentelemetry.io/otel/trace v1.42.0/go.mod h1:f3K9S+IFqnumBkKhRJMeaZeNk9epyhnCmQh/EysQCdc=
 go.opentelemetry.io/otel/trace v1.43.0/go.mod h1:/QJhyVBUUswCphDVxq+8mld+AvhXZLhe+8WVFxiFff0=
-go.opentelemetry.io/otel/trace v1.44.0 h1:jxF5CsGYCe74MCRx2X4g7WsY/VBKRqqpNvXlX/6gtIk=
 go.opentelemetry.io/otel/trace v1.44.0/go.mod h1:oLl1jrMQAVo6v3GAggN+1VH9VIz9iUSvW53sW1Q8PIE=
 go.opentelemetry.io/otel/trace v1.45.0 h1:l/mP6Uv7oNO7/TblbhpbgMidxhq1uO/rPsikOyVhxag=
 go.opentelemetry.io/otel/trace v1.45.0/go.mod h1:qoJJA2xNMnxRrdISU/kLtfUH2wNeQbiv+jhs/CxI8bc=
@@ -1837,8 +1809,6 @@ go.opentelemetry.io/proto/otlp v1.0.0/go.mod h1:Sy6pihPLfYHkr3NkUbEhGHFhINUSI/v8
 go.opentelemetry.io/proto/otlp v1.7.0/go.mod h1:fSKjH6YJ7HDlwzltzyMj036AJ3ejJLCgCSHGj4efDDo=
 go.opentelemetry.io/proto/otlp v1.7.1/go.mod h1:b2rVh6rfI/s2pHWNlB7ILJcRALpcNDzKhACevjI+ZnE=
 go.opentelemetry.io/proto/otlp v1.9.0/go.mod h1:xE+Cx5E/eEHw+ISFkwPLwCZefwVjY+pqKg1qcK03+/4=
-go.opentelemetry.io/proto/otlp v1.10.0 h1:IQRWgT5srOCYfiWnpqUYz9CVmbO8bFmKcwYxpuCSL2g=
-go.opentelemetry.io/proto/otlp v1.10.0/go.mod h1:/CV4QoCR/S9yaPj8utp3lvQPoqMtxXdzn7ozvvozVqk=
 go.opentelemetry.io/proto/otlp v1.11.0 h1:5rrYs0Ykyj50sdU/JU0x8etU+LubXWb+gED6TbEdMIk=
 go.opentelemetry.io/proto/otlp v1.11.0/go.mod h1:SmVizdCOAm3XBtG1g1NnOdhW6jtddT72hLMhv8VwA8E=
 go.uber.org/atomic v1.3.2/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
@@ -2152,8 +2122,6 @@ golang.org/x/net v0.52.0/go.mod h1:R1MAz7uMZxVMualyPXb+VaqGSa3LIaUqk0eEt3w36Sw=
 golang.org/x/net v0.53.0/go.mod h1:JvMuJH7rrdiCfbeHoo3fCQU24Lf5JJwT9W3sJFulfgs=
 golang.org/x/net v0.54.0/go.mod h1:Sj4oj8jK6XmHpBZU/zWHw3BV3abl4Kvi+Ut7cQcY+cQ=
 golang.org/x/net v0.55.0/go.mod h1:L5U2KuzuOe1lY7Z+aWVIKK6qEeJXnXV9yzGA+WCHJww=
-golang.org/x/net v0.56.0 h1:Rw8j/hFzGvJUZwNBXnAtf5sVDVt+65SK2C7IxCxZt5o=
-golang.org/x/net v0.56.0/go.mod h1:D3Ku6r+V6JROoZK144D2XfMHFcMq/0zSfLelVTCFKec=
 golang.org/x/net v0.57.0 h1:K5+3DljvIuDG9/Jv9rvyMywYNFCQ9RSUY6OOTTkT+tE=
 golang.org/x/net v0.57.0/go.mod h1:KpXc8iv+r3XplLAG/f7Jsf9RPszJzdR0f58q9vGOuEU=
 golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
@@ -2868,7 +2836,6 @@ google.golang.org/genproto/googleapis/api v0.0.0-20260401024825-9d38bb4040a9/go.
 google.golang.org/genproto/googleapis/api v0.0.0-20260511170946-3700d4141b60/go.mod h1:7yoXV7RIh5gblj/xVYoogxAWvA9wUeVbpsK/M694l00=
 google.golang.org/genproto/googleapis/api v0.0.0-20260519071638-aa98bba5eb94/go.mod h1:1dCETSCY2YKZNXQE3h4fun3TYwF5p8jejRKZgfWAgAY=
 google.golang.org/genproto/googleapis/api v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:q4lMZS6kskjT5HvCPrnnypcDPVJqT/f4nfxmkE7gryY=
-google.golang.org/genproto/googleapis/api v0.0.0-20260618152121-87f3d3e198d3 h1:ctPmKL12ZsoKAlmPUsoW70zEDiYF+/H6aLieXxgAU0k=
 google.golang.org/genproto/googleapis/api v0.0.0-20260618152121-87f3d3e198d3/go.mod h1:Z4WJ5pJOYWFWcHEQUelD5QaZDknIQkpIL/+fyJOT9+A=
 google.golang.org/genproto/googleapis/api v0.0.0-20260803160001-6ac0973c030d h1:FarXi840EJWSHYTN3ERkADbPWjl307+FGrA22KAVjjc=
 google.golang.org/genproto/googleapis/api v0.0.0-20260803160001-6ac0973c030d/go.mod h1:K/+WGbmBY7aNW1HDw1fJnKYo10i0DkAX6pows00dLig=
@@ -3051,7 +3018,6 @@ google.golang.org/genproto/googleapis/rpc v0.0.0-20260519071638-aa98bba5eb94/go.
 google.golang.org/genproto/googleapis/rpc v0.0.0-20260523011958-0a33c5d7ca68/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
 google.golang.org/genproto/googleapis/rpc v0.0.0-20260526163538-3dc84a4a5aaa/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
 google.golang.org/genproto/googleapis/rpc v0.0.0-20260610212136-7ab31c22f7ad/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
-google.golang.org/genproto/googleapis/rpc v0.0.0-20260618152121-87f3d3e198d3 h1:phvBWCAQMGN1945mp5fjCXP6jEF0+a0+4TjokS4sxNY=
 google.golang.org/genproto/googleapis/rpc v0.0.0-20260618152121-87f3d3e198d3/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
 google.golang.org/genproto/googleapis/rpc v0.0.0-20260803160001-6ac0973c030d h1:IL4hdHzcUv2l/gcg98/Rj3FbtE6axwqslOW8SW0C+S0=
 google.golang.org/genproto/googleapis/rpc v0.0.0-20260803160001-6ac0973c030d/go.mod h1:4Hqkh8ycfw05ld/3BWL7rJOSfebL2Q+DVDeRgYgxUU8=
@@ -3148,8 +3114,6 @@ google.golang.org/grpc v1.79.2/go.mod h1:KmT0Kjez+0dde/v2j9vzwoAScgEPx/Bw1CYChhH
 google.golang.org/grpc v1.79.3/go.mod h1:KmT0Kjez+0dde/v2j9vzwoAScgEPx/Bw1CYChhHLrHQ=
 google.golang.org/grpc v1.80.0/go.mod h1:ho/dLnxwi3EDJA4Zghp7k2Ec1+c2jqup0bFkw07bwF4=
 google.golang.org/grpc v1.81.1/go.mod h1:xGH9GfzOyMTGIOXBJmXt+BX/V0kcdQbdcuwQ/zNw42I=
-google.golang.org/grpc v1.82.1 h1:NnAxzGRA0677vCa4BUkOAnO5+FfQqVl9iUXeD0IqcGE=
-google.golang.org/grpc v1.82.1/go.mod h1:yzTZ1TB1Z3SG+LIYaI+WiE8D5+PZ3ArnrSp8zF3+/ZA=
 google.golang.org/grpc v1.83.0 h1:JeNZEKJFbQxArAMl+hiytHauacDNqJUllNfmIMmpqnQ=
 google.golang.org/grpc v1.83.0/go.mod h1:kDyl6SKsiHKt0uylY5gtn5cEjkrIOhQOGDgIc4JGwzQ=
 google.golang.org/grpc/cmd/protoc-gen-go-grpc v1.1.0/go.mod h1:6Kw0yEErY5E/yWrBtf03jp27GLLJujG4z/JK95pnjjw=

scripts/node-reset.sh

@@ -181,6 +181,14 @@ for dir in "${WIPE_DIRS[@]}"; do
         ! \( "${KEEP_TOP[@]}" \) -delete 2>/dev/null || true
 done
 
+# The wipe above keeps directories, so the pre-cutover cluster/host-N/node-M
+# skeleton would survive beside the node-<id> dirs written now. predastore owns
+# this tree, not setup.sh, so removing it outright loses no installed ownership.
+if ! $KEEP_DATA && [ -d "$DATA_DIR/predastore/cluster" ]; then
+    log "removing the predastore cluster tree"
+    run sudo rm -rf "$DATA_DIR/predastore/cluster"
+fi
+
 # Directories may survive — a mountpoint cannot be removed, and neither can the
 # path leading down to one. Files may not: a surviving file is state that would
 # carry into the new cluster, which is the exact failure this script exists to

spinifex/admin/admin.go

@@ -951,39 +951,54 @@ func SetupAWSCredentials(accessKey, secretKey, region, certPath, bindIP string)
 	return nil
 }
 
-// PredastoreClusterNode is one [[node]] entry of the Predastore topology: a
-// role pinned to a host.
+// PredastoreClusterNode is one [[host.node]] entry of the Predastore topology:
+// a role pinned to a host, answering on a port unique within it.
 type PredastoreClusterNode struct {
 	ID     int
 	HostID int
 	Role   string
+	Port   int
 }
 
-// Predastore node roles: shard storage holds erasure-coded object shards,
-// state replicas form the Raft quorum over global state.
+// Predastore node roles: a gate serves the S3 API, blob nodes hold
+// erasure-coded object shards, and meta nodes form the Raft quorum over global
+// state.
 const (
-	predastoreRoleShardStorage = "shard-storage"
-	predastoreRoleStateReplica = "state-replica"
+	predastoreRoleGate = "gate"
+	predastoreRoleBlob = "blob"
+	predastoreRoleMeta = "meta"
+)
+
+// Predastore node ports. Only a gate binds a socket clients reach; the others
+// are still required and must not collide within a host, so each role gets its
+// own base and colocated siblings count up from it.
+const (
+	predastoreGatePort     = 8443
+	predastoreBlobBasePort = 6660
+	predastoreMetaBasePort = 7660
 )
 
 // PredastoreTopology derives the cluster nodes for a set of machines: each
-// machine hosts one shard-storage node and one state replica. Node IDs are
-// unique across roles, so storage takes 1..n and the replicas n+1..2n.
+// machine hosts a gate, one blob node and one meta node. Node IDs are unique
+// across the whole file, so gates take 1..n, blobs n+1..2n and metas 2n+1..3n.
 func PredastoreTopology(nodes []PredastoreNodeConfig) []PredastoreClusterNode {
-	out := make([]PredastoreClusterNode, 0, len(nodes)*2)
+	out := make([]PredastoreClusterNode, 0, len(nodes)*3)
 	for _, n := range nodes {
-		out = append(out, PredastoreClusterNode{ID: n.ID, HostID: n.ID, Role: predastoreRoleShardStorage})
+		out = append(out, PredastoreClusterNode{ID: n.ID, HostID: n.ID, Role: predastoreRoleGate, Port: predastoreGatePort})
 	}
 	for _, n := range nodes {
-		out = append(out, PredastoreClusterNode{ID: n.ID + len(nodes), HostID: n.ID, Role: predastoreRoleStateReplica})
+		out = append(out, PredastoreClusterNode{ID: n.ID + len(nodes), HostID: n.ID, Role: predastoreRoleBlob, Port: predastoreBlobBasePort})
+	}
+	for _, n := range nodes {
+		out = append(out, PredastoreClusterNode{ID: n.ID + 2*len(nodes), HostID: n.ID, Role: predastoreRoleMeta, Port: predastoreMetaBasePort})
 	}
 	return out
 }
 
 // GenerateMultiNodePredastoreConfig produces a complete predastore.toml for a
 // multi-node Predastore cluster. Each machine becomes one [[host]] — a
-// predastore process owning a socket on port 6660 and a data directory —
-// carrying the shard-storage and state-replica nodes pinned to it.
+// predastore process owning a data directory and a TLS identity — carrying the
+// gate, blob and meta nodes pinned to it under [[host.node]].
 //
 // dataDir is the Spinifex data root; the hosts' data directories are absolute
 // beneath it, since the service runs under systemd with no dependable working
@@ -1049,14 +1064,14 @@ func FindNodeIDByIP(nodes []PredastoreNodeConfig, ip string) int {
 }
 
 // ParsePredastoreHostIDFromConfig parses a predastore.toml string and returns
-// the ID of the [[host]] whose public address matches the given IP, or 0 if
-// not found. The host is what an operator places on a machine; the nodes
-// pinned to it follow from the topology.
+// the ID of the [[host]] whose addr matches the given IP, or 0 if not found.
+// A host addr names one machine and never carries a port; the ports belong to
+// the nodes pinned to it.
 func ParsePredastoreHostIDFromConfig(tomlContent string, ip string) int {
 	var cfg struct {
 		Hosts []struct {
-			ID         int    `toml:"id"`
-			PublicAddr string `toml:"public_addr"`
+			ID   int    `toml:"id"`
+			Addr string `toml:"addr"`
 		} `toml:"host"`
 	}
 	if err := toml.Unmarshal([]byte(tomlContent), &cfg); err != nil {
@@ -1064,12 +1079,7 @@ func ParsePredastoreHostIDFromConfig(tomlContent string, ip string) int {
 		return 0
 	}
 	for _, h := range cfg.Hosts {
-		host, _, err := net.SplitHostPort(h.PublicAddr)
-		if err != nil {
-			// A bare address without a port is still a usable match.
-			host = h.PublicAddr
-		}
-		if host == ip {
+		if h.Addr == ip {
 			return h.ID
 		}
 	}

spinifex/admin/admin_test.go

@@ -802,7 +802,7 @@ func TestCreateServiceDirectories_Idempotent(t *testing.T) {
 func TestGenerateMultiNodePredastoreConfig_Success(t *testing.T) {
 	tmpl := `{{range .Nodes}}[[host]]
 id = {{.ID}}
-public_addr = "{{.Host}}:6660"
+addr = "{{.Host}}"
 data_dir = "{{$.PredastoreDataDir}}"
 {{end}}`
 	nodes := []PredastoreNodeConfig{
@@ -813,18 +813,19 @@ data_dir = "{{$.PredastoreDataDir}}"
 
 	result, err := GenerateMultiNodePredastoreConfig(tmpl, nodes, "AK", "SK", "us-east-1", "nats-token", "/config", "/var/lib/spinifex", "10.0.0.1", 0, NorthstarCredentials{})
 	require.NoError(t, err)
-	assert.Contains(t, result, `public_addr = "10.0.0.1:6660"`)
-	assert.Contains(t, result, `public_addr = "10.0.0.3:6660"`)
+	assert.Contains(t, result, `addr = "10.0.0.1"`)
+	assert.Contains(t, result, `addr = "10.0.0.3"`)
 	assert.Contains(t, result, `data_dir = "/var/lib/spinifex/predastore/cluster"`)
 }
 
-// Each machine hosts one shard-storage node and one state replica, with node
-// IDs unique across both roles so the topology validates.
+// Each machine hosts a gate, a blob node and a meta node, with node IDs unique
+// across all three roles so the topology validates.
 func TestGenerateMultiNodePredastoreConfig_Topology(t *testing.T) {
-	tmpl := `{{range .ClusterNodes}}[[node]]
+	tmpl := `{{range .ClusterNodes}}[[host.node]]
 id = {{.ID}}
 host_id = {{.HostID}}
 role = "{{.Role}}"
+port = {{.Port}}
 {{end}}`
 	nodes := []PredastoreNodeConfig{
 		{ID: 1, Host: "10.0.0.1"},
@@ -835,10 +836,11 @@ role = "{{.Role}}"
 	result, err := GenerateMultiNodePredastoreConfig(tmpl, nodes, "AK", "SK", "us-east-1", "nats-token", "/config", "/var/lib/spinifex", "10.0.0.1", 0, NorthstarCredentials{})
 	require.NoError(t, err)
 
-	assert.Contains(t, result, "id = 1\nhost_id = 1\nrole = \"shard-storage\"")
-	assert.Contains(t, result, "id = 3\nhost_id = 3\nrole = \"shard-storage\"")
-	assert.Contains(t, result, "id = 4\nhost_id = 1\nrole = \"state-replica\"")
-	assert.Contains(t, result, "id = 6\nhost_id = 3\nrole = \"state-replica\"")
+	assert.Contains(t, result, "id = 1\nhost_id = 1\nrole = \"gate\"\nport = 8443")
+	assert.Contains(t, result, "id = 4\nhost_id = 1\nrole = \"blob\"\nport = 6660")
+	assert.Contains(t, result, "id = 6\nhost_id = 3\nrole = \"blob\"\nport = 6660")
+	assert.Contains(t, result, "id = 7\nhost_id = 1\nrole = \"meta\"\nport = 7660")
+	assert.Contains(t, result, "id = 9\nhost_id = 3\nrole = \"meta\"\nport = 7660")
 }
 
 func TestPredastoreTopology_UniqueIDsAcrossRoles(t *testing.T) {
@@ -847,15 +849,16 @@ func TestPredastoreTopology_UniqueIDsAcrossRoles(t *testing.T) {
 		{ID: 2, Host: "10.0.0.2"},
 	})
 
-	require.Len(t, topology, 4)
+	require.Len(t, topology, 6)
 	seen := map[int]bool{}
 	for _, n := range topology {
 		assert.False(t, seen[n.ID], "duplicate node id %d", n.ID)
 		seen[n.ID] = true
 		assert.Contains(t, []int{1, 2}, n.HostID)
 	}
-	assert.Equal(t, "shard-storage", topology[0].Role)
-	assert.Equal(t, "state-replica", topology[2].Role)
+	assert.Equal(t, "gate", topology[0].Role)
+	assert.Equal(t, "blob", topology[2].Role)
+	assert.Equal(t, "meta", topology[4].Role)
 }
 
 // The northstar template fields were declared but never assigned, so every
@@ -927,18 +930,23 @@ func TestParsePredastoreHostIDFromConfig(t *testing.T) {
 	tomlContent := `
 [[host]]
 id = 1
-bind_addr = "0.0.0.0:6660"
-public_addr = "10.0.0.1:6660"
+bind_addr = "0.0.0.0"
+addr = "10.0.0.1"
+
+[[host.node]]
+id = 1
+role = "gate"
+port = 8443
 
 [[host]]
 id = 2
-bind_addr = "0.0.0.0:6660"
-public_addr = "10.0.0.2:6660"
+bind_addr = "0.0.0.0"
+addr = "10.0.0.2"
 
 [[host]]
 id = 3
-bind_addr = "0.0.0.0:6660"
-public_addr = "10.0.0.3:6660"
+bind_addr = "0.0.0.0"
+addr = "10.0.0.3"
 `
 	assert.Equal(t, 2, ParsePredastoreHostIDFromConfig(tomlContent, "10.0.0.2"))
 	assert.Equal(t, 0, ParsePredastoreHostIDFromConfig(tomlContent, "10.0.0.99"))
@@ -946,15 +954,15 @@ public_addr = "10.0.0.3:6660"
 	assert.Equal(t, 0, ParsePredastoreHostIDFromConfig("", "10.0.0.1"))
 }
 
-// A public_addr without a port must still match, so a hand-edited config that
-// omits it resolves to a host rather than silently to zero.
-func TestParsePredastoreHostIDFromConfig_AddressWithoutPort(t *testing.T) {
+// A host addr never carries a port, so one written with a node's port attached
+// names no machine and must not resolve to a host.
+func TestParsePredastoreHostIDFromConfig_AddressWithPortDoesNotMatch(t *testing.T) {
 	tomlContent := `
 [[host]]
 id = 7
-public_addr = "10.0.0.7"
+addr = "10.0.0.7:6660"
 `
-	assert.Equal(t, 7, ParsePredastoreHostIDFromConfig(tomlContent, "10.0.0.7"))
+	assert.Equal(t, 0, ParsePredastoreHostIDFromConfig(tomlContent, "10.0.0.7"))
 }
 
 // --- Integration: Full config generation flow ---

spinifex/daemon/daemon_handlers.go

@@ -347,13 +347,10 @@ func (d *Daemon) handleNodeStatus(msg *nats.Msg) {
 	// OVN roles are probed only on DB quorum members; compute-only nodes skip
 	// the shell-out entirely.
 	var wg sync.WaitGroup
-	wg.Add(2)
-	go func() { defer wg.Done(); resp.NATSRole = d.queryNATSRole() }()
-	go func() { defer wg.Done(); resp.PredastoreRole = d.queryPredastoreRole() }()
+	wg.Go(func() { resp.NATSRole = d.queryNATSRole() })
 	if d.isOVNDBQuorumMember() {
-		wg.Add(2)
-		go func() { defer wg.Done(); resp.OVNNBRole = host.OVNDBRole(host.OVNNBTarget, host.OVNNBSchema) }()
-		go func() { defer wg.Done(); resp.OVNSBRole = host.OVNDBRole(host.OVNSBTarget, host.OVNSBSchema) }()
+		wg.Go(func() { resp.OVNNBRole = host.OVNDBRole(host.OVNNBTarget, host.OVNNBSchema) })
+		wg.Go(func() { resp.OVNSBRole = host.OVNDBRole(host.OVNSBTarget, host.OVNSBSchema) })
 	}
 	wg.Wait()
 
@@ -364,8 +361,7 @@ const (
 	roleLeader   = "leader"
 	roleFollower = "follower"
 
-	natsMonitorPort  = 8222
-	predastoreDBPort = 6660
+	natsMonitorPort = 8222
 )
 
 // queryNATSRole queries the local NATS monitoring endpoint to determine this
@@ -378,16 +374,6 @@ func (d *Daemon) queryNATSRole() string {
 	return fetchNATSRole(url, roleHTTPClient)
 }
 
-// queryPredastoreRole queries the local Predastore DB status endpoint to
-// determine this node's Raft role. Returns "leader", "follower", or "".
-func (d *Daemon) queryPredastoreRole() string {
-	if !d.config.HasService("predastore") {
-		return ""
-	}
-	url := "https://" + net.JoinHostPort(d.daemonIP(), strconv.Itoa(predastoreDBPort)) + "/status"
-	return fetchPredastoreRole(url, roleTLSHTTPClient)
-}
-
 // isOVNDBQuorumMember reports whether this node hosts the clustered OVN NB/SB
 // databases. Compute-only nodes short-circuit OVN role probes to "" without
 // shelling out to ovn-appctl.
@@ -401,10 +387,6 @@ func (d *Daemon) isOVNDBQuorumMember() bool {
 
 var roleHTTPClient = &http.Client{Timeout: 500 * time.Millisecond}
 
-var roleTLSHTTPClient = &http.Client{
-	Timeout: 500 * time.Millisecond,
-}
-
 // fetchNATSRole queries a NATS /varz endpoint and returns "leader", "follower", or "".
 func fetchNATSRole(url string, client *http.Client) string {
 	resp, err := client.Get(url) //nolint:noctx // internal monitoring call
@@ -525,29 +507,6 @@ func resolveVMGPU(att gpu.GPUAttachment, byMdev, byPCI map[string]gpu.PoolEntry)
 	return nil
 }
 
-// fetchPredastoreRole queries a Predastore /status endpoint and returns "leader", "follower", or "".
-func fetchPredastoreRole(url string, client *http.Client) string {
-	resp, err := client.Get(url) //nolint:noctx // internal monitoring call
-	if err != nil {
-		slog.Debug("Failed to query Predastore status", "err", err)
-		return ""
-	}
-	defer resp.Body.Close()
-
-	var status struct {
-		IsLeader bool `json:"is_leader"`
-	}
-	if err := json.NewDecoder(resp.Body).Decode(&status); err != nil {
-		slog.Debug("Failed to decode Predastore status", "err", err)
-		return ""
-	}
-
-	if status.IsLeader {
-		return roleLeader
-	}
-	return roleFollower
-}
-
 // handleNodeVMs responds with the list of VMs running on this node.
 // Used by the CLI: spx get vms.
 func (d *Daemon) handleNodeVMs(msg *nats.Msg) {

spinifex/daemon/daemon_handlers_storage.go

@@ -2,10 +2,8 @@ package daemon
 
 import (
 	"log/slog"
-	"net"
 	"os"
 	"path/filepath"
-	"strconv"
 
 	"github.com/mulgadc/spinifex/spinifex/types"
 	"github.com/nats-io/nats.go"
@@ -17,8 +15,7 @@ import (
 type predastoreTOML struct {
 	RS      predastoreRS       `toml:"rs"`
 	Hosts   []predastoreHost   `toml:"host"`
-	Nodes   []predastoreNode   `toml:"node"`
-	Buckets []predastoreBucket `toml:"buckets"`
+	Buckets []predastoreBucket `toml:"bucket"`
 }
 
 type predastoreRS struct {
@@ -26,30 +23,32 @@ type predastoreRS struct {
 	Parity int `toml:"parity"`
 }
 
-// predastoreHost is one predastore process; the nodes pinned to it are all
-// reachable at its public address.
+// predastoreHost is one predastore process, as written under [[host]]. Addr
+// names the machine and carries no port; the nodes pinned to it supply their
+// own.
 type predastoreHost struct {
-	ID         int    `toml:"id"`
-	PublicAddr string `toml:"public_addr"`
+	ID    int              `toml:"id"`
+	Addr  string           `toml:"addr"`
+	Nodes []predastoreNode `toml:"node"`
 }
 
-// predastoreNode is a role pinned to a host. Its address is the host's, so
-// reporting a node means resolving its host_id.
+// predastoreNode is a role pinned to the host it is written under. Its address
+// is that host's addr joined with this port.
 type predastoreNode struct {
-	ID     int    `toml:"id"`
-	HostID int    `toml:"host_id"`
-	Role   string `toml:"role"`
+	ID   int    `toml:"id"`
+	Role string `toml:"role"`
+	Port int    `toml:"port"`
 }
 
-// Predastore node roles, as written in the [[node]] blocks.
+// Predastore node roles, as written in the [[host.node]] blocks.
 const (
-	predastoreRoleShardStorage = "shard-storage"
-	predastoreRoleStateReplica = "state-replica"
+	predastoreRoleGate = "gate"
+	predastoreRoleBlob = "blob"
+	predastoreRoleMeta = "meta"
 )
 
 type predastoreBucket struct {
 	Name   string `toml:"name"`
-	Type   string `toml:"type"`
 	Region string `toml:"region"`
 }
 
@@ -80,35 +79,33 @@ func (d *Daemon) handleStorageConfig(msg *nats.Msg) {
 		},
 	}
 
-	// A node's address is its host's: every node pinned to a host is served
-	// by that one process, keyed apart within it.
-	hostAddrs := make(map[int]predastoreHost, len(cfg.Hosts))
+	// A node's address is its host's addr plus its own port: colocated nodes
+	// share a machine but never a port.
 	for _, h := range cfg.Hosts {
-		hostAddrs[h.ID] = h
-	}
-
-	for _, n := range cfg.Nodes {
-		host, port := splitPredastoreAddr(hostAddrs[n.HostID].PublicAddr)
-		switch n.Role {
-		case predastoreRoleStateReplica:
-			resp.DBNodes = append(resp.DBNodes, types.StorageDBNode{ID: n.ID, Host: host, Port: port})
-		case predastoreRoleShardStorage:
-			resp.ShardNodes = append(resp.ShardNodes, types.StorageShardNode{ID: n.ID, Host: host, Port: port})
-		default:
-			slog.Warn("handleStorageConfig: unknown predastore node role", "node", n.ID, "role", n.Role)
+		for _, n := range h.Nodes {
+			switch n.Role {
+			case predastoreRoleMeta:
+				resp.MetaNodes = append(resp.MetaNodes, types.StorageMetaNode{ID: n.ID, Host: h.Addr, Port: n.Port})
+			case predastoreRoleBlob:
+				resp.BlobNodes = append(resp.BlobNodes, types.StorageBlobNode{ID: n.ID, Host: h.Addr, Port: n.Port})
+			case predastoreRoleGate:
+				// A gate serves the S3 API and holds no cluster state, so it
+				// belongs to neither table this report carries.
+			default:
+				slog.Warn("handleStorageConfig: unknown predastore node role", "node", n.ID, "role", n.Role)
+			}
 		}
 	}
-	if resp.DBNodes == nil {
-		resp.DBNodes = []types.StorageDBNode{}
+	if resp.MetaNodes == nil {
+		resp.MetaNodes = []types.StorageMetaNode{}
 	}
-	if resp.ShardNodes == nil {
-		resp.ShardNodes = []types.StorageShardNode{}
+	if resp.BlobNodes == nil {
+		resp.BlobNodes = []types.StorageBlobNode{}
 	}
 
 	for _, b := range cfg.Buckets {
 		resp.Buckets = append(resp.Buckets, types.StorageBucket{
 			Name:   b.Name,
-			Type:   b.Type,
 			Region: b.Region,
 		})
 	}
@@ -118,18 +115,3 @@ func (d *Daemon) handleStorageConfig(msg *nats.Msg) {
 
 	respondWithJSON(msg, resp)
 }
-
-// splitPredastoreAddr splits a host's "ip:port" into its parts. An address
-// missing or malformed in the config yields the raw value and a zero port
-// rather than dropping the node from the report entirely.
-func splitPredastoreAddr(addr string) (string, int) {
-	host, portStr, err := net.SplitHostPort(addr)
-	if err != nil {
-		return addr, 0
-	}
-	port, err := strconv.Atoi(portStr)
-	if err != nil {
-		return host, 0
-	}
-	return host, port
-}

spinifex/daemon/daemon_handlers_storage_test.go

@@ -1,82 +1,165 @@
 package daemon
 
 import (
+	"encoding/json"
+	"os"
+	"path/filepath"
 	"testing"
+	"time"
 
+	"github.com/mulgadc/spinifex/spinifex/types"
+	"github.com/nats-io/nats.go"
 	toml "github.com/pelletier/go-toml/v2"
 	"github.com/stretchr/testify/assert"
 	"github.com/stretchr/testify/require"
 )
 
-// The storage status report is built by parsing predastore.toml directly, so
-// it degrades silently when the schema moves under it: a config that no longer
-// matches yields an empty topology rather than an error. This pins the parse
-// to the [[host]]/[[node]] dialect the templates actually emit.
-func TestPredastoreTOML_ParsesClusterTopology(t *testing.T) {
-	content := `
+// clusterTOML is the two-host topology both tests below read: host 1 runs a
+// gate, a blob node and a meta replica; host 2 runs a second blob node.
+const clusterTOML = `
+version = 1
+region = "ap-southeast-2"
+
 [rs]
 data = 2
 parity = 1
 
 [[host]]
 id = 1
-bind_addr = "0.0.0.0:6660"
-public_addr = "10.0.0.1:6660"
+bind_addr = "0.0.0.0"
+addr = "10.0.0.1"
 data_dir = "/var/lib/spinifex/predastore/cluster"
 
+[[host.node]]
+id = 1
+role = "gate"
+port = 8443
+
+[[host.node]]
+id = 2
+role = "blob"
+port = 6660
+
+[[host.node]]
+id = 3
+role = "meta"
+port = 7660
+
 [[host]]
 id = 2
-bind_addr = "0.0.0.0:6660"
-public_addr = "10.0.0.2:6660"
+bind_addr = "0.0.0.0"
+addr = "10.0.0.2"
 data_dir = "/var/lib/spinifex/predastore/cluster"
 
-[[node]]
-id = 1
-host_id = 1
-role = "shard-storage"
+[[host.node]]
+id = 4
+role = "blob"
+port = 6660
 
-[[node]]
-id = 2
-host_id = 2
-role = "shard-storage"
-
-[[node]]
-id = 3
-host_id = 1
-role = "state-replica"
-
-[[buckets]]
+[[bucket]]
 name = "predastore"
-type = "distributed"
 region = "ap-southeast-2"
 `
 
+// storageConfigResponse installs cfg as the node's predastore.toml, drives the
+// handler over NATS and returns what it answered.
+func storageConfigResponse(t *testing.T, cfg string, reply string) []byte {
+	t.Helper()
+
+	dir := t.TempDir()
+	if cfg != "" {
+		require.NoError(t, os.MkdirAll(filepath.Join(dir, "predastore"), 0o750))
+		require.NoError(t, os.WriteFile(filepath.Join(dir, "predastore", "predastore.toml"), []byte(cfg), 0o600))
+	}
+
+	nc, err := nats.Connect(sharedNATSURL)
+	require.NoError(t, err)
+	t.Cleanup(nc.Close)
+
+	replies := make(chan []byte, 1)
+	sub, err := nc.Subscribe(reply, func(msg *nats.Msg) { replies <- msg.Data })
+	require.NoError(t, err)
+	t.Cleanup(func() { _ = sub.Unsubscribe() })
+
+	d := &Daemon{configPath: filepath.Join(dir, "spinifex.toml")}
+	d.handleStorageConfig(&nats.Msg{Subject: "storage.config", Reply: reply, Sub: sub})
+
+	select {
+	case payload := <-replies:
+		return payload
+	case <-time.After(5 * time.Second):
+		t.Fatal("handleStorageConfig sent no response")
+		return nil
+	}
+}
+
+// TestHandleStorageConfig_SplitsNodesByRole pins what the gateway builds its
+// storage report from: every node is reported at its host address and its own
+// port, meta and blob in separate tables, and the gate in neither — it serves
+// the S3 API and holds no cluster state to report.
+func TestHandleStorageConfig_SplitsNodesByRole(t *testing.T) {
+	payload := storageConfigResponse(t, clusterTOML, "test.storage.config.ok")
+
+	var resp types.StorageConfigResponse
+	require.NoError(t, json.Unmarshal(payload, &resp))
+
+	assert.Equal(t, 2, resp.Encoding.DataShards)
+	assert.Equal(t, 1, resp.Encoding.ParityShards)
+
+	require.Len(t, resp.MetaNodes, 1)
+	assert.Equal(t, types.StorageMetaNode{ID: 3, Host: "10.0.0.1", Port: 7660}, resp.MetaNodes[0])
+
+	require.Len(t, resp.BlobNodes, 2)
+	assert.Equal(t, types.StorageBlobNode{ID: 2, Host: "10.0.0.1", Port: 6660}, resp.BlobNodes[0])
+	assert.Equal(t, types.StorageBlobNode{ID: 4, Host: "10.0.0.2", Port: 6660}, resp.BlobNodes[1])
+
+	require.Len(t, resp.Buckets, 1)
+	assert.Equal(t, "predastore", resp.Buckets[0].Name)
+	assert.Equal(t, "ap-southeast-2", resp.Buckets[0].Region)
+}
+
+// TestHandleStorageConfig_ReportsAMissingConfig covers the node that has no
+// predastore config: the report is an error rather than an empty topology,
+// which would read as a cluster with no storage at all.
+func TestHandleStorageConfig_ReportsAMissingConfig(t *testing.T) {
+	payload := storageConfigResponse(t, "", "test.storage.config.absent")
+
+	assert.Contains(t, string(payload), "InternalError")
+}
+
+// TestHandleStorageConfig_ReportsAMalformedConfig covers a config that parses
+// as neither the current schema nor anything else: the same error, rather than
+// a half-read topology.
+func TestHandleStorageConfig_ReportsAMalformedConfig(t *testing.T) {
+	payload := storageConfigResponse(t, "not = [toml", "test.storage.config.malformed")
+
+	assert.Contains(t, string(payload), "InternalError")
+}
+
+// The storage status report is built by parsing predastore.toml directly, so
+// it degrades silently when the schema moves under it: a config that no longer
+// matches yields an empty topology rather than an error. This pins the parse
+// to the [[host]]/[[host.node]] dialect the templates actually emit.
+func TestPredastoreTOML_ParsesClusterTopology(t *testing.T) {
 	var cfg predastoreTOML
-	require.NoError(t, toml.Unmarshal([]byte(content), &cfg))
+	require.NoError(t, toml.Unmarshal([]byte(clusterTOML), &cfg))
 
 	assert.Equal(t, 2, cfg.RS.Data)
 	assert.Equal(t, 1, cfg.RS.Parity)
 	require.Len(t, cfg.Hosts, 2)
-	require.Len(t, cfg.Nodes, 3)
+	require.Len(t, cfg.Buckets, 1)
 
-	assert.Equal(t, "10.0.0.2:6660", cfg.Hosts[1].PublicAddr)
-	assert.Equal(t, predastoreRoleShardStorage, cfg.Nodes[0].Role)
-	assert.Equal(t, 1, cfg.Nodes[2].HostID)
-	assert.Equal(t, predastoreRoleStateReplica, cfg.Nodes[2].Role)
-}
-
-func TestSplitPredastoreAddr(t *testing.T) {
-	host, port := splitPredastoreAddr("10.0.0.1:6660")
-	assert.Equal(t, "10.0.0.1", host)
-	assert.Equal(t, 6660, port)
-
-	// A hand-edited address without a port must still name its node rather
-	// than dropping it from the report.
-	host, port = splitPredastoreAddr("10.0.0.1")
-	assert.Equal(t, "10.0.0.1", host)
-	assert.Equal(t, 0, port)
-
-	host, port = splitPredastoreAddr("")
-	assert.Empty(t, host)
-	assert.Equal(t, 0, port)
+	assert.Equal(t, "10.0.0.2", cfg.Hosts[1].Addr)
+	require.Len(t, cfg.Hosts[0].Nodes, 3)
+	assert.Equal(t, predastoreRoleGate, cfg.Hosts[0].Nodes[0].Role)
+	assert.Equal(t, 8443, cfg.Hosts[0].Nodes[0].Port)
+	assert.Equal(t, predastoreRoleBlob, cfg.Hosts[0].Nodes[1].Role)
+	assert.Equal(t, predastoreRoleMeta, cfg.Hosts[0].Nodes[2].Role)
+	assert.Equal(t, 7660, cfg.Hosts[0].Nodes[2].Port)
+
+	require.Len(t, cfg.Hosts[1].Nodes, 1)
+	assert.Equal(t, 4, cfg.Hosts[1].Nodes[0].ID)
+	assert.Equal(t, predastoreRoleBlob, cfg.Hosts[1].Nodes[0].Role)
+	assert.Equal(t, 6660, cfg.Hosts[1].Nodes[0].Port)
+	assert.Equal(t, "predastore", cfg.Buckets[0].Name)
 }

spinifex/daemon/roles_test.go

@@ -76,38 +76,3 @@ func TestFetchNATSRole_BadJSON(t *testing.T) {
 	role := fetchNATSRole(srv.URL, srv.Client())
 	assert.Empty(t, role)
 }
-
-func TestFetchPredastoreRole_Leader(t *testing.T) {
-	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		w.Write([]byte(`{"node_id":"1","state":"Leader","is_leader":true}`))
-	}))
-	defer srv.Close()
-
-	role := fetchPredastoreRole(srv.URL, srv.Client())
-	assert.Equal(t, "leader", role)
-}
-
-func TestFetchPredastoreRole_Follower(t *testing.T) {
-	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		w.Write([]byte(`{"node_id":"2","state":"Follower","is_leader":false}`))
-	}))
-	defer srv.Close()
-
-	role := fetchPredastoreRole(srv.URL, srv.Client())
-	assert.Equal(t, "follower", role)
-}
-
-func TestFetchPredastoreRole_Unreachable(t *testing.T) {
-	role := fetchPredastoreRole("https://127.0.0.1:1/status", &http.Client{})
-	assert.Empty(t, role)
-}
-
-func TestFetchPredastoreRole_BadJSON(t *testing.T) {
-	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
-		w.Write([]byte(`not json`))
-	}))
-	defer srv.Close()
-
-	role := fetchPredastoreRole(srv.URL, srv.Client())
-	assert.Empty(t, role)
-}

spinifex/gateway/ecr_principal.go

@@ -8,7 +8,7 @@ import (
 
 	"github.com/aws/aws-sdk-go/aws"
 	"github.com/aws/aws-sdk-go/service/iam"
-	"github.com/mulgadc/predastore/auth"
+	"github.com/mulgadc/predastore/pkg/auth"
 	"github.com/mulgadc/spinifex/spinifex/awserrors"
 	gateway_ecrauth "github.com/mulgadc/spinifex/spinifex/gateway/ecrauth"
 	handlers_iam "github.com/mulgadc/spinifex/spinifex/handlers/iam"

spinifex/gateway/gateway.go

@@ -16,10 +16,10 @@ import (
 	"github.com/go-chi/chi/v5"
 	"github.com/go-chi/chi/v5/middleware"
 	"github.com/google/uuid"
-	"github.com/mulgadc/predastore/auth"
+	"github.com/mulgadc/predastore/pkg/auth"
 	"github.com/mulgadc/predastore/pkg/iampolicy"
+	"github.com/mulgadc/predastore/pkg/ratelimit"
 	"github.com/mulgadc/predastore/pkg/sigv4"
-	"github.com/mulgadc/predastore/ratelimit"
 	"github.com/mulgadc/spinifex/spinifex/awserrors"
 	gateway_bedrock "github.com/mulgadc/spinifex/spinifex/gateway/bedrock"
 	gateway_ecr "github.com/mulgadc/spinifex/spinifex/gateway/ecr"

spinifex/gateway/gateway_test.go

@@ -17,7 +17,7 @@ import (
 	awscreds "github.com/aws/aws-sdk-go/aws/credentials"
 	awssession "github.com/aws/aws-sdk-go/aws/session"
 	awsec2 "github.com/aws/aws-sdk-go/service/ec2"
-	"github.com/mulgadc/predastore/ratelimit"
+	"github.com/mulgadc/predastore/pkg/ratelimit"
 	"github.com/mulgadc/spinifex/spinifex/awserrors"
 	handlers_iam "github.com/mulgadc/spinifex/spinifex/handlers/iam"
 	"github.com/mulgadc/spinifex/spinifex/testutil"

spinifex/gateway/rds.go

@@ -5,7 +5,7 @@ import (
 	"log/slog"
 	"net/http"
 
-	"github.com/mulgadc/predastore/auth"
+	"github.com/mulgadc/predastore/pkg/auth"
 	"github.com/mulgadc/spinifex/spinifex/awserrors"
 	gateway_rds "github.com/mulgadc/spinifex/spinifex/gateway/rds"
 )

spinifex/gateway/spx/storage.go

@@ -15,8 +15,8 @@ import (
 	"github.com/nats-io/nats.go"
 )
 
-// DBNodeStatus is a single DB node's live status merged with its config.
-type DBNodeStatus struct {
+// MetaNodeStatus is a single meta node's live status merged with its config.
+type MetaNodeStatus struct {
 	ID         int    `json:"id"`
 	Host       string `json:"host"`
 	Port       int    `json:"port"`
@@ -32,10 +32,10 @@ type DBNodeStatus struct {
 
 // StorageStatusOutput is the response for GetStorageStatus.
 type StorageStatusOutput struct {
-	Encoding   StorageEncodingOutput    `json:"encoding"`
-	DBNodes    []DBNodeStatus           `json:"db_nodes"`
-	ShardNodes []types.StorageShardNode `json:"shard_nodes"`
-	Buckets    []types.StorageBucket    `json:"buckets"`
+	Encoding  StorageEncodingOutput   `json:"encoding"`
+	MetaNodes []MetaNodeStatus        `json:"meta_nodes"`
+	BlobNodes []types.StorageBlobNode `json:"blob_nodes"`
+	Buckets   []types.StorageBucket   `json:"buckets"`
 }
 
 // StorageEncodingOutput adds the type label to the encoding config.
@@ -49,8 +49,8 @@ var storageHTTPClient = &http.Client{
 	Timeout: 1 * time.Second,
 }
 
-// GetStorageStatus fetches predastore topology via NATS, then queries each DB
-// node's /status and /health endpoints in parallel.
+// GetStorageStatus fetches predastore topology via NATS, then queries each
+// meta node's /status and /health endpoints in parallel.
 func GetStorageStatus(nc *nats.Conn) (*StorageStatusOutput, error) {
 	msg, err := nc.Request("spinifex.storage.config", []byte("{}"), 3*time.Second)
 	if err != nil {
@@ -62,22 +62,22 @@ func GetStorageStatus(nc *nats.Conn) (*StorageStatusOutput, error) {
 		return nil, fmt.Errorf("parse storage config: %w", err)
 	}
 
-	dbStatuses := make([]DBNodeStatus, len(cfg.DBNodes))
+	metaStatuses := make([]MetaNodeStatus, len(cfg.MetaNodes))
 	var wg sync.WaitGroup
 	ctx, cancel := context.WithTimeout(context.Background(), 1*time.Second)
 	defer cancel()
 
-	for i, db := range cfg.DBNodes {
-		dbStatuses[i] = DBNodeStatus{
-			ID:   db.ID,
-			Host: db.Host,
-			Port: db.Port,
+	for i, meta := range cfg.MetaNodes {
+		metaStatuses[i] = MetaNodeStatus{
+			ID:   meta.ID,
+			Host: meta.Host,
+			Port: meta.Port,
 		}
 		wg.Add(1)
 		go func(idx int, host string, port int) {
 			defer wg.Done()
-			queryDBNodeStatus(ctx, &dbStatuses[idx], host, port)
-		}(i, db.Host, db.Port)
+			queryMetaNodeStatus(ctx, &metaStatuses[idx], host, port)
+		}(i, meta.Host, meta.Port)
 	}
 	wg.Wait()
 
@@ -87,9 +87,9 @@ func GetStorageStatus(nc *nats.Conn) (*StorageStatusOutput, error) {
 			DataShards:   cfg.Encoding.DataShards,
 			ParityShards: cfg.Encoding.ParityShards,
 		},
-		DBNodes:    dbStatuses,
-		ShardNodes: cfg.ShardNodes,
-		Buckets:    cfg.Buckets,
+		MetaNodes: metaStatuses,
+		BlobNodes: cfg.BlobNodes,
+		Buckets:   cfg.Buckets,
 	}, nil
 }
 
@@ -105,7 +105,7 @@ type predastoreStatusResponse struct {
 	IsLeader   bool   `json:"is_leader"`
 }
 
-func queryDBNodeStatus(ctx context.Context, out *DBNodeStatus, host string, port int) {
+func queryMetaNodeStatus(ctx context.Context, out *MetaNodeStatus, host string, port int) {
 	// Resolve 0.0.0.0 to a routable address for HTTPS.
 	queryHost := host
 	if queryHost == "0.0.0.0" {
@@ -119,7 +119,7 @@ func queryDBNodeStatus(ctx context.Context, out *DBNodeStatus, host string, port
 	}
 	resp, err := storageHTTPClient.Do(req)
 	if err != nil {
-		slog.Debug("queryDBNodeStatus: health check failed", "host", host, "port", port, "err", err)
+		slog.Debug("queryMetaNodeStatus: health check failed", "host", host, "port", port, "err", err)
 		return
 	}
 	resp.Body.Close()
@@ -132,14 +132,14 @@ func queryDBNodeStatus(ctx context.Context, out *DBNodeStatus, host string, port
 	}
 	resp, err = storageHTTPClient.Do(req)
 	if err != nil {
-		slog.Debug("queryDBNodeStatus: status check failed", "host", host, "port", port, "err", err)
+		slog.Debug("queryMetaNodeStatus: status check failed", "host", host, "port", port, "err", err)
 		return
 	}
 	defer resp.Body.Close()
 
 	var status predastoreStatusResponse
 	if err := json.NewDecoder(resp.Body).Decode(&status); err != nil {
-		slog.Debug("queryDBNodeStatus: failed to decode status", "host", host, "port", port, "err", err)
+		slog.Debug("queryMetaNodeStatus: failed to decode status", "host", host, "port", port, "err", err)
 		return
 	}
 

spinifex/gateway/spx/storage_test.go

@@ -22,15 +22,15 @@ func TestGetStorageStatus_Success(t *testing.T) {
 	sub, err := nc.Subscribe("spinifex.storage.config", func(msg *nats.Msg) {
 		resp := types.StorageConfigResponse{
 			Encoding: types.StorageEncoding{DataShards: 2, ParityShards: 1},
-			DBNodes: []types.StorageDBNode{
+			MetaNodes: []types.StorageMetaNode{
 				{ID: 1, Host: "127.0.0.1", Port: 0}, // port set below
 			},
-			ShardNodes: []types.StorageShardNode{
+			BlobNodes: []types.StorageBlobNode{
 				{ID: 1, Host: "0.0.0.0", Port: 9991},
 				{ID: 2, Host: "0.0.0.0", Port: 9992},
 			},
 			Buckets: []types.StorageBucket{
-				{Name: "predastore", Type: "distributed", Region: "ap-southeast-2"},
+				{Name: "predastore", Region: "ap-southeast-2"},
 			},
 		}
 		data, _ := json.Marshal(resp)
@@ -46,13 +46,13 @@ func TestGetStorageStatus_Success(t *testing.T) {
 	assert.Equal(t, "Reed-Solomon", out.Encoding.Type)
 	assert.Equal(t, 2, out.Encoding.DataShards)
 	assert.Equal(t, 1, out.Encoding.ParityShards)
-	assert.Len(t, out.ShardNodes, 2)
+	assert.Len(t, out.BlobNodes, 2)
 	assert.Len(t, out.Buckets, 1)
 	assert.Equal(t, "predastore", out.Buckets[0].Name)
-	// DB node health check will fail (no real predastore running) — that's expected
-	require.Len(t, out.DBNodes, 1)
-	assert.Equal(t, 1, out.DBNodes[0].ID)
-	assert.False(t, out.DBNodes[0].Healthy) // no predastore server to respond
+	// Meta node health check will fail (no real predastore running) — expected
+	require.Len(t, out.MetaNodes, 1)
+	assert.Equal(t, 1, out.MetaNodes[0].ID)
+	assert.False(t, out.MetaNodes[0].Healthy) // no predastore server to respond
 }
 
 func TestGetStorageStatus_NoNATSResponse(t *testing.T) {
@@ -63,8 +63,8 @@ func TestGetStorageStatus_NoNATSResponse(t *testing.T) {
 	assert.Contains(t, err.Error(), "storage config request")
 }
 
-func TestQueryDBNodeStatus_HealthyNode(t *testing.T) {
-	// Start a mock predastore DB server
+func TestQueryMetaNodeStatus_HealthyNode(t *testing.T) {
+	// Start a mock predastore meta node server
 	mux := http.NewServeMux()
 	mux.HandleFunc("/health", func(w http.ResponseWriter, r *http.Request) {
 		w.Header().Set("Content-Type", "application/json")
@@ -104,8 +104,8 @@ func TestQueryDBNodeStatus_HealthyNode(t *testing.T) {
 	port, err := strconv.Atoi(u.Port())
 	require.NoError(t, err)
 
-	out := &DBNodeStatus{ID: 1, Host: host, Port: port}
-	queryDBNodeStatus(t.Context(), out, host, port)
+	out := &MetaNodeStatus{ID: 1, Host: host, Port: port}
+	queryMetaNodeStatus(t.Context(), out, host, port)
 
 	assert.True(t, out.Healthy)
 	assert.Equal(t, "Leader", out.State)
@@ -115,9 +115,9 @@ func TestQueryDBNodeStatus_HealthyNode(t *testing.T) {
 	assert.Equal(t, "998", out.AppliedIdx)
 }
 
-func TestQueryDBNodeStatus_UnreachableNode(t *testing.T) {
-	out := &DBNodeStatus{ID: 1, Host: "127.0.0.1", Port: 1} // port 1 won't respond
-	queryDBNodeStatus(t.Context(), out, "127.0.0.1", 1)
+func TestQueryMetaNodeStatus_UnreachableNode(t *testing.T) {
+	out := &MetaNodeStatus{ID: 1, Host: "127.0.0.1", Port: 1} // port 1 won't respond
+	queryMetaNodeStatus(t.Context(), out, "127.0.0.1", 1)
 
 	assert.False(t, out.Healthy)
 	assert.Empty(t, out.State)

spinifex/handlers/sts/assume_role.go

@@ -21,7 +21,7 @@ import (
 	"github.com/aws/aws-sdk-go/service/sts"
 	"github.com/nats-io/nats.go/jetstream"
 
-	"github.com/mulgadc/predastore/auth"
+	"github.com/mulgadc/predastore/pkg/auth"
 	"github.com/mulgadc/spinifex/spinifex/awserrors"
 	handlers_iam "github.com/mulgadc/spinifex/spinifex/handlers/iam"
 	"github.com/mulgadc/spinifex/spinifex/utils"

spinifex/handlers/sts/assume_role_with_web_identity.go

@@ -17,7 +17,7 @@ import (
 	"github.com/golang-jwt/jwt/v5"
 	"github.com/nats-io/nats.go/jetstream"
 
-	"github.com/mulgadc/predastore/auth"
+	"github.com/mulgadc/predastore/pkg/auth"
 	"github.com/mulgadc/spinifex/spinifex/awserrors"
 	handlers_iam "github.com/mulgadc/spinifex/spinifex/handlers/iam"
 )

spinifex/migrate/VERSIONS.md

@@ -7,8 +7,8 @@ Single source of truth for the current schema version of every config file Spini
 | `nats.conf` | `3` | `cmd/spinifex/cmd/templates/nats.conf` |
 | `awsgw.toml` | `2` | `cmd/spinifex/cmd/templates/awsgw.toml` |
 | `spinifex.toml` | `4` | `cmd/spinifex/cmd/templates/spinifex.toml` |
-| `predastore.toml` | `5` | `cmd/spinifex/cmd/templates/predastore.toml` |
-| `predastore-multinode.toml` | `5` | `cmd/spinifex/cmd/templates/predastore-multinode.toml` |
+| `predastore.toml` | `1` | `cmd/spinifex/cmd/templates/predastore.toml` |
+| `predastore-multinode.toml` | `1` | `cmd/spinifex/cmd/templates/predastore-multinode.toml` |
 
 ## How versions are stamped
 
@@ -19,15 +19,12 @@ Single source of truth for the current schema version of every config file Spini
 
 | Target | Migration | File |
 |---|---|---|
-| `predastore.toml` | `4` → `5`: `[[db]]`/`[[nodes]]` → `[[host]]`/`[[node]]`, relocate node data, recover raft | `spinifex/migrate/predastoretopology/004_predastore_topology.go` |
 | `spinifex.toml` | `3` → `4`: rename `[nodes.*.predastore]` `node_id` → `host_id` | `spinifex/migrate/005_spinifex_predastore_host_id.go` |
 
 ## Where migrations live
 
 Register a `ConfigMigration` against `DefaultRegistry` in a new numbered file under `spinifex/migrate/`, and bump the version in both the template and the table above.
 
-A migration that needs more than the framework and the standard library goes in its own package beside `migrate` instead, blank-imported from `cmd/spinifex/cmd/admin_upgrade.go`. `migrate` is imported by every handler that stamps a KV bucket version, so a heavy dependency there reaches the whole tree — `predastoretopology` needs Predastore's cluster runtime, which is why it sits apart.
-
 ## Framework
 
 The framework (`migrate.go`, `version_readers.go`) covers both kinds of target:

spinifex/migrate/predastoretopology/004_predastore_topology.go

@@ -1,475 +0,0 @@
-// Package predastoretopology migrates a Predastore install onto the v5 cluster
-// topology: hosts owning a data directory, nodes pinned to them by role.
-//
-// It sits outside the migrate package because it needs Predastore's cluster
-// runtime to rewrite raft state, and migrate is imported by every handler that
-// stamps a KV bucket version. Only the spx binary that runs `admin upgrade`
-// links this.
-package predastoretopology
-
-import (
-	"fmt"
-	"os"
-	"path/filepath"
-	"regexp"
-	"sort"
-	"strings"
-
-	toml "github.com/pelletier/go-toml/v2"
-
-	"github.com/mulgadc/predastore/clusterrun"
-	"github.com/mulgadc/spinifex/spinifex/migrate"
-)
-
-// The predastore config target, relative to the Spinifex config directory.
-const (
-	predastoreTarget  = "predastore.toml"
-	predastoreRelPath = "predastore/predastore.toml"
-	// predastoreClusterVersion is the schema the cluster topology lands on;
-	// the install templates stamp it directly.
-	predastoreClusterVersion = 5
-)
-
-// predastoreHostPort is the port a v5 host binds. Every v4 [[db]] entry used
-// it too, so a migrated topology keeps the same firewall surface.
-const predastoreHostPort = 6660
-
-// Predastore v5 node roles, mirroring internal/cluster.Role.
-const (
-	predastoreRoleShard   = "shard-storage"
-	predastoreRoleReplica = "state-replica"
-)
-
-// predastoreV4Node is the v4 shape of a [[db]] or [[nodes]] entry, reduced to
-// the fields the migration needs. A frozen copy keeps the migration
-// independent of predastore's config surface, which no longer parses these.
-type predastoreV4Node struct {
-	ID   int    `toml:"id"`
-	Host string `toml:"host"`
-	Path string `toml:"path"`
-}
-
-// predastoreV4Config is the v4 topology. Only the two tables the migration
-// rewrites are declared; every other key rides through the rewrite as text.
-type predastoreV4Config struct {
-	DB    []predastoreV4Node `toml:"db"`
-	Nodes []predastoreV4Node `toml:"nodes"`
-}
-
-// predastoreHost is one v5 [[host]]: a predastore process owning a socket and
-// a data directory.
-type predastoreHost struct {
-	ID         int
-	BindAddr   string
-	PublicAddr string
-	DataDir    string
-}
-
-// predastoreNode is one v5 [[node]]: a role pinned to a host.
-type predastoreNode struct {
-	ID     int
-	HostID int
-	Role   string
-}
-
-// predastoreMove relocates one node's state from its v4 directory to the v5
-// one derived from the host data directory and the node id.
-type predastoreMove struct {
-	NodeID int
-	Role   string
-	From   string
-	To     string
-}
-
-// predastoreTopology is everything derived from a v4 config: the tables to
-// write, the directories to move, and the replica set any relocated state
-// replica must be recovered onto.
-type predastoreTopology struct {
-	Hosts      []predastoreHost
-	Nodes      []predastoreNode
-	Moves      []predastoreMove
-	ReplicaIDs []int
-}
-
-// predastoreHostTableRe detects a config that already carries the v5 topology,
-// so a part-applied migration can be re-run.
-var predastoreHostTableRe = regexp.MustCompile(`(?m)^\s*\[\[host\]\]\s*$`)
-
-// predastoreTopLevelAddrRe matches the v4 gateway address keys, which must go:
-// a scalar host cannot coexist with the [[host]] tables.
-var predastoreTopLevelAddrRe = regexp.MustCompile(`^(host|port)\s*=`)
-
-// predastoreTableHeaderRe captures the name of a table or table-array header.
-var predastoreTableHeaderRe = regexp.MustCompile(`^\[\[?([^\[\]]+)\]?\]$`)
-
-func init() {
-	migrate.DefaultRegistry.RegisterConfigTarget(predastoreTarget, predastoreRelPath, &migrate.TOMLVersionReader{})
-	migrate.DefaultRegistry.RegisterConfig(predastoreTarget, migrate.ConfigMigration{
-		FromVersion: 4,
-		ToVersion:   predastoreClusterVersion,
-		Description: "predastore [[db]]/[[nodes]] → [[host]]/[[node]]; relocate node data and recover raft",
-		Run:         migratePredastoreTopology,
-	})
-}
-
-// migratePredastoreTopology carries a v4 install onto the v5 cluster topology:
-// the data moves first, since a failure there leaves a config still describing
-// where the data actually is.
-func migratePredastoreTopology(ctx migrate.ConfigContext) error {
-	path := filepath.Join(ctx.ConfigDir, predastoreRelPath)
-
-	data, err := os.ReadFile(path)
-	if err != nil {
-		return fmt.Errorf("read %s: %w", path, err)
-	}
-
-	if predastoreHostTableRe.Match(data) {
-		ctx.Logger.Info("Predastore config already carries [[host]] tables, leaving it alone", "path", path)
-		return nil
-	}
-
-	var old predastoreV4Config
-	if err := toml.Unmarshal(data, &old); err != nil {
-		return fmt.Errorf("parse %s: %w", path, err)
-	}
-
-	topo, err := predastoreTopologyFromV4(old, ctx.DataDir)
-	if err != nil {
-		return err
-	}
-
-	if err := relocatePredastoreData(ctx, topo); err != nil {
-		return err
-	}
-
-	out, err := rewritePredastoreConfig(data, topo)
-	if err != nil {
-		return err
-	}
-
-	info, err := os.Stat(path)
-	if err != nil {
-		return fmt.Errorf("stat %s: %w", path, err)
-	}
-	if err := os.WriteFile(path, out, info.Mode()); err != nil {
-		return fmt.Errorf("write %s: %w", path, err)
-	}
-	return nil
-}
-
-// predastoreTopologyFromV4 derives the v5 topology from a v4 config.
-//
-// A machine is one host. Shard nodes keep their ids, because object metadata
-// records which node holds each shard; the state replicas are the ones that
-// move, taking ids above the shard range.
-func predastoreTopologyFromV4(old predastoreV4Config, spxRoot string) (*predastoreTopology, error) {
-	if len(old.DB) == 0 || len(old.Nodes) == 0 {
-		return nil, fmt.Errorf("predastore config has %d [[db]] and %d [[nodes]] entries; not a v4 distributed topology", len(old.DB), len(old.Nodes))
-	}
-
-	db := append([]predastoreV4Node(nil), old.DB...)
-	shards := append([]predastoreV4Node(nil), old.Nodes...)
-	sort.Slice(db, func(i, j int) bool { return db[i].ID < db[j].ID })
-	sort.Slice(shards, func(i, j int) bool { return shards[i].ID < shards[j].ID })
-
-	// A single-node install lists every db peer on one address; that is one
-	// machine, so one host carries the lot.
-	singleMachine := true
-	for _, n := range db[1:] {
-		if n.Host != db[0].Host {
-			singleMachine = false
-			break
-		}
-	}
-
-	dataDir := predastoreDataDir(spxRoot)
-	basePath := predastoreBasePath(spxRoot)
-
-	topo := &predastoreTopology{}
-	hostByAddr := make(map[string]int, len(db))
-
-	if singleMachine {
-		hostByAddr[db[0].Host] = 1
-		topo.Hosts = append(topo.Hosts, predastoreHost{
-			ID:         1,
-			BindAddr:   fmt.Sprintf("%s:%d", db[0].Host, predastoreHostPort),
-			PublicAddr: fmt.Sprintf("%s:%d", db[0].Host, predastoreHostPort),
-			DataDir:    dataDir,
-		})
-	} else {
-		for _, n := range db {
-			if _, dup := hostByAddr[n.Host]; dup {
-				return nil, fmt.Errorf("predastore config lists two [[db]] entries on %s; cannot derive one host per machine", n.Host)
-			}
-			hostByAddr[n.Host] = n.ID
-			topo.Hosts = append(topo.Hosts, predastoreHost{
-				ID: n.ID,
-				// The v4 multi-node template bound every interface; keep that
-				// rather than narrowing an operator's reachability on upgrade.
-				BindAddr:   fmt.Sprintf("0.0.0.0:%d", predastoreHostPort),
-				PublicAddr: fmt.Sprintf("%s:%d", n.Host, predastoreHostPort),
-				DataDir:    dataDir,
-			})
-		}
-	}
-
-	for _, n := range shards {
-		hostID, ok := hostByAddr[n.Host]
-		if !ok {
-			return nil, fmt.Errorf("predastore shard node %d is on %s, which has no [[db]] entry to place it", n.ID, n.Host)
-		}
-		topo.Nodes = append(topo.Nodes, predastoreNode{ID: n.ID, HostID: hostID, Role: predastoreRoleShard})
-		topo.Moves = append(topo.Moves, predastoreMove{
-			NodeID: n.ID,
-			Role:   predastoreRoleShard,
-			From:   predastoreV4Path(basePath, n.Path),
-			To:     filepath.Join(dataDir, fmt.Sprintf("node-%d", n.ID)),
-		})
-	}
-
-	// State replica ids start above the shard range so the two never collide.
-	offset := shards[len(shards)-1].ID
-	for _, n := range db {
-		id := n.ID + offset
-		hostID := hostByAddr[n.Host]
-		topo.Nodes = append(topo.Nodes, predastoreNode{ID: id, HostID: hostID, Role: predastoreRoleReplica})
-		topo.ReplicaIDs = append(topo.ReplicaIDs, id)
-		topo.Moves = append(topo.Moves, predastoreMove{
-			NodeID: id,
-			Role:   predastoreRoleReplica,
-			From:   predastoreV4Path(basePath, n.Path),
-			To:     filepath.Join(dataDir, fmt.Sprintf("node-%d", id)),
-		})
-	}
-
-	return topo, nil
-}
-
-// relocatePredastoreData moves each node's state into the v5 layout and
-// rewrites the raft configuration of every state replica it moves.
-//
-// Only this machine's directories exist, but the config describes the whole
-// cluster, so a missing source is the normal multi-node case rather than an
-// error. A destination that already exists is a re-run.
-func relocatePredastoreData(ctx migrate.ConfigContext, topo *predastoreTopology) error {
-	for _, m := range topo.Moves {
-		moved, err := movePredastoreNodeDir(ctx, m)
-		if err != nil {
-			return err
-		}
-		if !moved || m.Role != predastoreRoleReplica {
-			continue
-		}
-
-		// The rename preserved this directory's ownership, which is what
-		// recovery must leave behind on everything it writes into it.
-		uid, gid, err := dirOwner(m.To)
-		if err != nil {
-			return err
-		}
-
-		// The replica set is renumbered, so the configuration persisted here
-		// names servers that no longer exist. Raft will not bootstrap over
-		// existing state to correct that; it has to be rewritten.
-		recovered, err := clusterrun.RecoverStateReplica(m.To, m.NodeID, topo.ReplicaIDs)
-		if err != nil {
-			return fmt.Errorf("recover raft configuration for state replica %d in %s: %w", m.NodeID, m.To, err)
-		}
-
-		// Recovery reopened badger and wrote a fresh snapshot as root, both of
-		// which create files. Runs even when there was nothing to recover,
-		// which still opens the stores.
-		if err := chownTree(m.To, uid, gid); err != nil {
-			return err
-		}
-		ctx.Logger.Info("Migrated predastore state replica", "nodeID", m.NodeID, "dataDir", m.To, "raftRecovered", recovered)
-	}
-	return nil
-}
-
-// movePredastoreNodeDir moves one node directory, reporting whether the
-// destination now holds state that came from a v4 layout.
-func movePredastoreNodeDir(ctx migrate.ConfigContext, m predastoreMove) (bool, error) {
-	if _, err := os.Stat(m.From); err != nil {
-		if !os.IsNotExist(err) {
-			return false, fmt.Errorf("stat %s: %w", m.From, err)
-		}
-		// Already moved by an earlier run; the replica still needs recovering
-		// if that run failed before it got there.
-		if _, err := os.Stat(m.To); err == nil {
-			return true, nil
-		}
-		return false, nil
-	}
-
-	if _, err := os.Stat(m.To); err == nil {
-		return false, fmt.Errorf("cannot move %s to %s: destination already exists", m.From, m.To)
-	}
-
-	// Taken before the rename, which carries it to the destination; the data
-	// dir created below has to match or predastore cannot traverse into it.
-	uid, gid, err := dirOwner(m.From)
-	if err != nil {
-		return false, err
-	}
-
-	dataDir := filepath.Dir(m.To)
-	if err := os.MkdirAll(dataDir, 0750); err != nil {
-		return false, fmt.Errorf("create %s: %w", dataDir, err)
-	}
-	// Unconditional, so a re-run also repairs a data dir left root-owned by an
-	// earlier run that failed after creating it.
-	if err := chown(dataDir, uid, gid); err != nil {
-		return false, err
-	}
-
-	if err := os.Rename(m.From, m.To); err != nil {
-		return false, fmt.Errorf("move %s to %s: %w", m.From, m.To, err)
-	}
-	ctx.Logger.Info("Moved predastore node data", "nodeID", m.NodeID, "role", m.Role, "from", m.From, "to", m.To)
-	return true, nil
-}
-
-// rewritePredastoreConfig replaces the v4 topology tables and the gateway
-// address keys with the v5 tables, leaving every other table as the operator
-// left it. Re-rendering from the template would instead discard their buckets,
-// credentials and rate limits.
-func rewritePredastoreConfig(data []byte, topo *predastoreTopology) ([]byte, error) {
-	lines := strings.Split(string(data), "\n")
-
-	var out, pending []string
-	var table string
-	dropping := false
-	wroteTopology := false
-	wroteGatewayNote := false
-
-	// flush emits the buffered comment/blank block ahead of a line that keeps it.
-	flush := func() {
-		out = append(out, pending...)
-		pending = nil
-	}
-
-	for _, line := range lines {
-		trimmed := strings.TrimSpace(line)
-
-		if m := predastoreTableHeaderRe.FindStringSubmatch(trimmed); m != nil {
-			table = strings.TrimSpace(m[1])
-			// The comment block introducing a dropped table goes with it.
-			if table == "db" || table == "nodes" {
-				pending = nil
-				dropping = true
-				if !wroteTopology {
-					out = append(out, predastoreTopologyBlock(topo)...)
-					wroteTopology = true
-				}
-				continue
-			}
-			dropping = false
-			flush()
-			out = append(out, line)
-			continue
-		}
-
-		blankOrComment := trimmed == "" || strings.HasPrefix(trimmed, "#")
-
-		if dropping {
-			// Buffer separators and trailing comments; a key line means the
-			// comments above it described the dropped table, so let them go.
-			if blankOrComment {
-				pending = append(pending, line)
-			} else {
-				pending = nil
-			}
-			continue
-		}
-
-		if blankOrComment {
-			pending = append(pending, line)
-			continue
-		}
-
-		if table == "" && predastoreTopLevelAddrRe.MatchString(trimmed) {
-			pending = nil
-			if !wroteGatewayNote {
-				out = append(out, predastoreGatewayNote...)
-				wroteGatewayNote = true
-			}
-			continue
-		}
-
-		flush()
-		out = append(out, line)
-	}
-	flush()
-
-	if !wroteTopology {
-		return nil, fmt.Errorf("predastore config has no [[db]] or [[nodes]] tables to replace")
-	}
-	return []byte(strings.Join(out, "\n")), nil
-}
-
-// predastoreGatewayNote replaces the top-level host/port keys, which the v5
-// schema cannot accept.
-var predastoreGatewayNote = []string{
-	"",
-	"# The gateway listen address comes from the daemon CLI options: a top-level",
-	"# host key would collide with the [[host]] topology tables below.",
-}
-
-// predastoreTopologyBlock renders the v5 tables, matching the layout the
-// install templates generate so a migrated config reads like a fresh one.
-func predastoreTopologyBlock(topo *predastoreTopology) []string {
-	out := []string{
-		"",
-		"# One host per machine: a predastore process owning a socket and a data",
-		"# directory. bind_addr binds every interface; public_addr is what the other",
-		"# hosts dial.",
-	}
-	for _, h := range topo.Hosts {
-		out = append(out,
-			"",
-			"[[host]]",
-			fmt.Sprintf("id = %d", h.ID),
-			fmt.Sprintf("bind_addr = %q", h.BindAddr),
-			fmt.Sprintf("public_addr = %q", h.PublicAddr),
-			fmt.Sprintf("data_dir = %q", h.DataDir),
-		)
-	}
-
-	out = append(out,
-		"",
-		"# Nodes are roles pinned to a host: shard-storage nodes hold erasure-coded",
-		"# object shards, state replicas form the Raft quorum over global state.",
-	)
-	for _, n := range topo.Nodes {
-		out = append(out,
-			"",
-			"[[node]]",
-			fmt.Sprintf("id = %d", n.ID),
-			fmt.Sprintf("host_id = %d", n.HostID),
-			fmt.Sprintf("role = %q", n.Role),
-		)
-	}
-	return out
-}
-
-// predastoreV4Path resolves a v4 node path, which is relative to the
-// predastore base path unless the operator made it absolute.
-func predastoreV4Path(basePath, path string) string {
-	if filepath.IsAbs(path) {
-		return filepath.Clean(path)
-	}
-	return filepath.Join(basePath, path)
-}
-
-// predastoreBasePath mirrors SPINIFEX_PREDASTORE_BASE_PATH in
-// build/systemd/spinifex-predastore.service; v4 node paths resolve against it.
-func predastoreBasePath(spxRoot string) string {
-	return filepath.Join(spxRoot, "predastore")
-}
-
-// predastoreDataDir mirrors admin.PredastoreDataDir. Duplicated to keep the
-// migrate package free of the admin dependency; if either moves, both must.
-func predastoreDataDir(spxRoot string) string {
-	return filepath.Join(spxRoot, "predastore", "cluster")
-}

spinifex/migrate/predastoretopology/004_predastore_topology_test.go

@@ -1,265 +0,0 @@
-package predastoretopology
-
-import (
-	"flag"
-	"fmt"
-	"log/slog"
-	"os"
-	"path/filepath"
-	"strings"
-	"testing"
-
-	"github.com/mulgadc/predastore/s3"
-	"github.com/mulgadc/spinifex/spinifex/migrate"
-	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
-)
-
-var updateGolden = flag.Bool("update-golden", false, "rewrite the predastore migration golden files")
-
-// predastoreFixture lays out a v4 install under a temporary root: the config
-// in place, and a data directory for each node this machine owns.
-func predastoreFixture(t *testing.T, name string, nodeDirs []string) (configDir, dataDir string) {
-	t.Helper()
-
-	root := t.TempDir()
-	configDir = filepath.Join(root, "etc")
-	dataDir = filepath.Join(root, "var")
-
-	require.NoError(t, os.MkdirAll(filepath.Join(configDir, "predastore"), 0750))
-
-	src, err := os.ReadFile(filepath.Join("testdata", name))
-	require.NoError(t, err)
-	require.NoError(t, os.WriteFile(filepath.Join(configDir, predastoreRelPath), src, 0640))
-
-	for _, dir := range nodeDirs {
-		full := filepath.Join(dataDir, "predastore", dir)
-		require.NoError(t, os.MkdirAll(full, 0750))
-		// A marker per directory proves the contents travelled, not just the path.
-		require.NoError(t, os.WriteFile(filepath.Join(full, "marker"), []byte(dir), 0640))
-	}
-	return configDir, dataDir
-}
-
-// runPredastoreMigration drives the real registry path, so registration, the
-// backup and the version stamp are covered alongside the rewrite itself.
-func runPredastoreMigration(t *testing.T, configDir, dataDir string) {
-	t.Helper()
-	require.NoError(t, migrate.DefaultRegistry.RunConfig(predastoreTarget, configDir, dataDir))
-}
-
-// dataDirPlaceholder stands in for the test's temporary root, which is
-// rendered into data_dir and cannot be checked in.
-const dataDirPlaceholder = "{{DATA_DIR}}"
-
-// assertGolden compares the migrated config against its golden file, which is
-// checked in so a review sees the exact config an operator ends up with.
-func assertGolden(t *testing.T, configDir, dataDir, golden string) {
-	t.Helper()
-
-	raw, err := os.ReadFile(filepath.Join(configDir, predastoreRelPath))
-	require.NoError(t, err)
-	got := strings.ReplaceAll(string(raw), dataDir, dataDirPlaceholder)
-
-	path := filepath.Join("testdata", golden)
-	if *updateGolden {
-		require.NoError(t, os.WriteFile(path, []byte(got), 0640))
-		return
-	}
-	want, err := os.ReadFile(path)
-	require.NoError(t, err)
-	assert.Equal(t, string(want), got)
-}
-
-// readMigratedConfig parses the migrated config the way predastore does, so
-// the test fails on anything predastore would reject rather than on text.
-func readMigratedConfig(t *testing.T, configDir, dataDir string) *s3.Config {
-	t.Helper()
-
-	cfg := &s3.Config{
-		ConfigPath: filepath.Join(configDir, predastoreRelPath),
-		BasePath:   filepath.Join(dataDir, "predastore"),
-	}
-	require.NoError(t, cfg.ReadConfig())
-	return cfg
-}
-
-// TestPredastoreTopologySingleNode covers the common install: three db peers
-// and three shard nodes on one machine collapse to one host running all six.
-func TestPredastoreTopologySingleNode(t *testing.T) {
-	configDir, dataDir := predastoreFixture(t, "predastore_v4_singlenode.toml", []string{
-		"distributed/db/node-1", "distributed/db/node-2", "distributed/db/node-3",
-		"distributed/nodes/node-1", "distributed/nodes/node-2", "distributed/nodes/node-3",
-	})
-
-	runPredastoreMigration(t, configDir, dataDir)
-	assertGolden(t, configDir, dataDir, "predastore_v5_singlenode.golden")
-
-	cfg := readMigratedConfig(t, configDir, dataDir)
-
-	require.Len(t, cfg.Hosts, 1)
-	assert.Equal(t, 1, cfg.Hosts[0].ID)
-	assert.Equal(t, "127.0.0.1:6660", cfg.Hosts[0].BindAddr)
-	assert.Equal(t, "127.0.0.1:6660", cfg.Hosts[0].PublicAddr)
-	assert.Equal(t, filepath.Join(dataDir, "predastore", "cluster"), cfg.Hosts[0].DataDir)
-
-	require.Len(t, cfg.ClusterNodes, 6)
-	for i, n := range cfg.ClusterNodes {
-		assert.Equal(t, i+1, n.ID)
-		assert.Equal(t, 1, n.HostID)
-		if i < 3 {
-			assert.Equal(t, "shard-storage", string(n.Role))
-		} else {
-			assert.Equal(t, "state-replica", string(n.Role))
-		}
-	}
-
-	// Shard nodes keep their ids because object metadata records which node
-	// holds each shard; the state replicas are the ones that move.
-	cluster := filepath.Join(dataDir, "predastore", "cluster")
-	assertNodeDirContents(t, cluster, map[int]string{
-		1: "distributed/nodes/node-1",
-		2: "distributed/nodes/node-2",
-		3: "distributed/nodes/node-3",
-		4: "distributed/db/node-1",
-		5: "distributed/db/node-2",
-		6: "distributed/db/node-3",
-	})
-
-	assert.NoDirExists(t, filepath.Join(dataDir, "predastore", "distributed", "db", "node-1"))
-	assert.NoDirExists(t, filepath.Join(dataDir, "predastore", "distributed", "nodes", "node-1"))
-
-	// Everything outside the topology is the operator's, and must survive.
-	raw, err := os.ReadFile(filepath.Join(configDir, predastoreRelPath))
-	require.NoError(t, err)
-	for _, keep := range []string{
-		`[ratelimit.action."s3:PutObject"]`,
-		"access_keys_bucket = \"spinifex-iam-access-keys\"",
-		"interval_seconds = 300",
-		`{ bucket = "predastore", actions = ["s3:ListBucket",  "s3:GetObject",`,
-	} {
-		assert.Contains(t, string(raw), keep)
-	}
-	assert.NotContains(t, string(raw), "\nhost = \"0.0.0.0\"")
-	assert.NotContains(t, string(raw), "[[db]]")
-	assert.NotContains(t, string(raw), "[[nodes]]")
-}
-
-// TestPredastoreTopologyMultiNode covers a three-machine cluster from the
-// perspective of one member, which only holds its own node directories.
-func TestPredastoreTopologyMultiNode(t *testing.T) {
-	configDir, dataDir := predastoreFixture(t, "predastore_v4_multinode.toml", []string{
-		"distributed/db/node-2", "distributed/nodes/node-2",
-	})
-
-	runPredastoreMigration(t, configDir, dataDir)
-	assertGolden(t, configDir, dataDir, "predastore_v5_multinode.golden")
-
-	cfg := readMigratedConfig(t, configDir, dataDir)
-
-	require.Len(t, cfg.Hosts, 3)
-	for i, h := range cfg.Hosts {
-		assert.Equal(t, i+1, h.ID)
-		// Binding every interface preserves the v4 template's reachability;
-		// only the advertised address is per machine.
-		assert.Equal(t, "0.0.0.0:6660", h.BindAddr)
-		assert.Equal(t, fmt.Sprintf("10.0.0.%d:6660", h.ID), h.PublicAddr)
-		assert.Equal(t, filepath.Join(dataDir, "predastore", "cluster"), h.DataDir)
-	}
-
-	require.Len(t, cfg.ClusterNodes, 6)
-	byID := map[int]struct {
-		host int
-		role string
-	}{}
-	for _, n := range cfg.ClusterNodes {
-		byID[n.ID] = struct {
-			host int
-			role string
-		}{n.HostID, string(n.Role)}
-	}
-	for id, want := range map[int]struct {
-		host int
-		role string
-	}{
-		1: {1, "shard-storage"}, 2: {2, "shard-storage"}, 3: {3, "shard-storage"},
-		4: {1, "state-replica"}, 5: {2, "state-replica"}, 6: {3, "state-replica"},
-	} {
-		assert.Equal(t, want, byID[id], "node %d", id)
-	}
-
-	// Only this machine's directories exist; the rest belong to its peers.
-	cluster := filepath.Join(dataDir, "predastore", "cluster")
-	assertNodeDirContents(t, cluster, map[int]string{
-		2: "distributed/nodes/node-2",
-		5: "distributed/db/node-2",
-	})
-	assert.NoDirExists(t, filepath.Join(cluster, "node-1"))
-	assert.NoDirExists(t, filepath.Join(cluster, "node-4"))
-}
-
-// TestPredastoreTopologyRerunIsNoop covers a run that failed after writing the
-// config but before the version was stamped: the migration is invoked directly,
-// so the version gate cannot mask a missing guard on the already-migrated file.
-func TestPredastoreTopologyRerunIsNoop(t *testing.T) {
-	configDir, dataDir := predastoreFixture(t, "predastore_v4_singlenode.toml", []string{
-		"distributed/db/node-1", "distributed/nodes/node-1",
-	})
-	ctx := migrate.ConfigContext{ConfigDir: configDir, DataDir: dataDir, Logger: slog.Default()}
-
-	require.NoError(t, migratePredastoreTopology(ctx))
-	first, err := os.ReadFile(filepath.Join(configDir, predastoreRelPath))
-	require.NoError(t, err)
-
-	require.NoError(t, migratePredastoreTopology(ctx))
-	second, err := os.ReadFile(filepath.Join(configDir, predastoreRelPath))
-	require.NoError(t, err)
-
-	assert.Equal(t, string(first), string(second))
-	assertNodeDirContents(t, filepath.Join(dataDir, "predastore", "cluster"), map[int]string{
-		1: "distributed/nodes/node-1",
-		4: "distributed/db/node-1",
-	})
-}
-
-// TestPredastoreTopologyRejectsNonV4Config fails loudly rather than writing a
-// config with no topology at all.
-func TestPredastoreTopologyRejectsNonV4Config(t *testing.T) {
-	configDir := filepath.Join(t.TempDir(), "etc")
-	require.NoError(t, os.MkdirAll(filepath.Join(configDir, "predastore"), 0750))
-	require.NoError(t, os.WriteFile(filepath.Join(configDir, predastoreRelPath),
-		[]byte("version = \"4\"\nregion = \"ap-southeast-2\"\n"), 0640))
-
-	err := migratePredastoreTopology(migrate.ConfigContext{
-		ConfigDir: configDir,
-		DataDir:   t.TempDir(),
-		Logger:    slog.Default(),
-	})
-	require.Error(t, err)
-	assert.Contains(t, err.Error(), "not a v4 distributed topology")
-}
-
-// TestPredastoreVersionReaderReadsFixtures ties the migration's FromVersion to
-// what the reader actually finds in a shipped config.
-func TestPredastoreVersionReaderReadsFixtures(t *testing.T) {
-	r := &migrate.TOMLVersionReader{}
-	for _, name := range []string{"predastore_v4_singlenode.toml", "predastore_v4_multinode.toml"} {
-		v, err := r.ReadVersion(filepath.Join("testdata", name))
-		require.NoError(t, err)
-		assert.Equal(t, 4, v, name)
-	}
-}
-
-// assertNodeDirContents checks that each node directory holds the marker
-// written into the v4 directory it came from.
-func assertNodeDirContents(t *testing.T, clusterDir string, want map[int]string) {
-	t.Helper()
-	for id, origin := range want {
-		marker := filepath.Join(clusterDir, fmt.Sprintf("node-%d", id), "marker")
-		got, err := os.ReadFile(marker)
-		if !assert.NoError(t, err, "node-%d", id) {
-			continue
-		}
-		assert.Equal(t, origin, string(got), "node-%d", id)
-	}
-}

spinifex/migrate/predastoretopology/ownership.go

@@ -1,66 +0,0 @@
-package predastoretopology
-
-import (
-	"fmt"
-	"io/fs"
-	"os"
-	"path/filepath"
-	"syscall"
-)
-
-// The migration runs as root under `spx admin upgrade`, but predastore runs as
-// its own service user. Anything root creates along the way is root-owned and
-// unreachable to it, so the migration has to hand each path back.
-
-// dirOwner reports the uid and gid owning path. The migration takes the
-// ownership it applies from the data it is moving rather than from a service
-// user name, so a dev install running as the invoking user needs no special
-// case.
-func dirOwner(path string) (int, int, error) {
-	info, err := os.Stat(path)
-	if err != nil {
-		return 0, 0, fmt.Errorf("stat %s: %w", path, err)
-	}
-	st, ok := info.Sys().(*syscall.Stat_t)
-	if !ok {
-		return 0, 0, fmt.Errorf("cannot read ownership of %s", path)
-	}
-	return int(st.Uid), int(st.Gid), nil
-}
-
-// chown gives path to uid:gid.
-//
-// Only root can hand a file to another user, and only root ever creates one
-// owned by the wrong user, so an unprivileged run has nothing to correct.
-func chown(path string, uid, gid int) error {
-	if os.Geteuid() != 0 {
-		return nil
-	}
-	if err := os.Lchown(path, uid, gid); err != nil {
-		return fmt.Errorf("chown %s to %d:%d: %w", path, uid, gid, err)
-	}
-	return nil
-}
-
-// chownTree gives path and everything beneath it to uid:gid.
-//
-// os.Root scopes the walk to the tree, so a symlink planted inside it cannot
-// redirect a chown running as root.
-func chownTree(path string, uid, gid int) error {
-	if os.Geteuid() != 0 {
-		return nil
-	}
-
-	root, err := os.OpenRoot(path)
-	if err != nil {
-		return fmt.Errorf("open %s: %w", path, err)
-	}
-	defer root.Close()
-
-	return fs.WalkDir(root.FS(), ".", func(p string, _ fs.DirEntry, err error) error {
-		if err != nil {
-			return fmt.Errorf("walk %s: %w", filepath.Join(path, p), err)
-		}
-		return chown(filepath.Join(path, p), uid, gid)
-	})
-}

spinifex/migrate/predastoretopology/ownership_test.go

@@ -1,94 +0,0 @@
-package predastoretopology
-
-import (
-	"fmt"
-	"os"
-	"path/filepath"
-	"testing"
-
-	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
-)
-
-// nobodyUID is an unprivileged id the root-only assertions move data to, so
-// "inherited the owner" cannot be confused with "happened to already match".
-const nobodyUID, nobodyGID = 65534, 65534
-
-// TestPredastoreDataDirInheritsNodeOwnership pins the ownership of the data
-// dir the migration creates. Predastore runs as its own service user while the
-// migration runs as root, and cannot traverse a root-owned parent.
-func TestPredastoreDataDirInheritsNodeOwnership(t *testing.T) {
-	nodeDirs := []string{
-		"distributed/db/node-1", "distributed/db/node-2", "distributed/db/node-3",
-		"distributed/nodes/node-1", "distributed/nodes/node-2", "distributed/nodes/node-3",
-	}
-	configDir, dataDir := predastoreFixture(t, "predastore_v4_singlenode.toml", nodeDirs)
-
-	wantUID, wantGID := os.Geteuid(), os.Getegid()
-	if os.Geteuid() == 0 {
-		// Stand the fixture up as an unprivileged user owns it, which is the
-		// shape of a real install the root migration has to preserve.
-		for _, dir := range nodeDirs {
-			require.NoError(t, chownTree(filepath.Join(dataDir, "predastore", dir), nobodyUID, nobodyGID))
-		}
-		wantUID, wantGID = nobodyUID, nobodyGID
-	}
-
-	runPredastoreMigration(t, configDir, dataDir)
-
-	cluster := filepath.Join(dataDir, "predastore", "cluster")
-	uid, gid, err := dirOwner(cluster)
-	require.NoError(t, err)
-	assert.Equal(t, wantUID, uid, "data dir owner")
-	assert.Equal(t, wantGID, gid, "data dir group")
-
-	// The node dirs arrive by rename, so they keep their owner rather than
-	// gaining one; assert it anyway, since recovery writes into them.
-	for id := 1; id <= 6; id++ {
-		uid, gid, err := dirOwner(filepath.Join(cluster, fmt.Sprintf("node-%d", id)))
-		require.NoError(t, err)
-		assert.Equal(t, wantUID, uid, "node-%d owner", id)
-		assert.Equal(t, wantGID, gid, "node-%d group", id)
-	}
-}
-
-// TestChownIsNoopWhenUnprivileged covers the guard that lets the migration run
-// as an ordinary user, where there is no ownership to correct and every chown
-// would fail.
-func TestChownIsNoopWhenUnprivileged(t *testing.T) {
-	if os.Geteuid() == 0 {
-		t.Skip("asserts the unprivileged path")
-	}
-
-	dir := t.TempDir()
-	require.NoError(t, os.WriteFile(filepath.Join(dir, "file"), []byte("x"), 0640))
-
-	require.NoError(t, chown(dir, nobodyUID, nobodyGID))
-	require.NoError(t, chownTree(dir, nobodyUID, nobodyGID))
-
-	uid, _, err := dirOwner(dir)
-	require.NoError(t, err)
-	assert.Equal(t, os.Geteuid(), uid, "ownership must be left alone")
-}
-
-// TestChownTreeCoversNestedContents guards the recursive case: raft recovery
-// writes a snapshot directory and badger segments below the node dir.
-func TestChownTreeCoversNestedContents(t *testing.T) {
-	if os.Geteuid() != 0 {
-		t.Skip("chown to another user requires root")
-	}
-
-	dir := t.TempDir()
-	nested := filepath.Join(dir, "snapshots", "1-2-3")
-	require.NoError(t, os.MkdirAll(nested, 0750))
-	require.NoError(t, os.WriteFile(filepath.Join(nested, "state.bin"), []byte("x"), 0640))
-
-	require.NoError(t, chownTree(dir, nobodyUID, nobodyGID))
-
-	for _, path := range []string{dir, nested, filepath.Join(nested, "state.bin")} {
-		uid, gid, err := dirOwner(path)
-		require.NoError(t, err)
-		assert.Equal(t, nobodyUID, uid, path)
-		assert.Equal(t, nobodyGID, gid, path)
-	}
-}

spinifex/migrate/predastoretopology/testdata/predastore_v4_multinode.toml

@@ -1,97 +0,0 @@
-# Predastore S3-compatible storage configuration file in TOML format.
-# Generated for multi-node cluster
-version = "4"
-region = "ap-southeast-2"
-
-host = "0.0.0.0"
-port = 8443
-
-debug = false
-
-[rs]
-data = 2
-parity = 1
-
-[[db]]
-id = 1
-host = "10.0.0.1"
-port = 6660
-path = "distributed/db/node-1/"
-access_key_id = "AKIAIOSFODNN7EXAMPLE"
-secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
-leader = true
-
-[[db]]
-id = 2
-host = "10.0.0.2"
-port = 6660
-path = "distributed/db/node-2/"
-access_key_id = "AKIAIOSFODNN7EXAMPLE"
-secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
-
-[[db]]
-id = 3
-host = "10.0.0.3"
-port = 6660
-path = "distributed/db/node-3/"
-access_key_id = "AKIAIOSFODNN7EXAMPLE"
-secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
-
-[[nodes]]
-id = 1
-host = "10.0.0.1"
-port = 9991
-path = "distributed/nodes/node-1/"
-
-[[nodes]]
-id = 2
-host = "10.0.0.2"
-port = 9991
-path = "distributed/nodes/node-2/"
-
-[[nodes]]
-id = 3
-host = "10.0.0.3"
-port = 9991
-path = "distributed/nodes/node-3/"
-
-[[buckets]]
-name = "predastore"
-region = "ap-southeast-2"
-type = "distributed"
-public = false
-encryption = ""
-account_id = "000000000000"
-
-# Northstar DNS zone storage. Read-only for the northstar service; the control
-# plane writes zone files with the system credentials.
-[[buckets]]
-name = "northstar-zones"
-region = "ap-southeast-2"
-type = "distributed"
-public = false
-encryption = ""
-account_id = "000000000000"
-
-[[auth]]
-access_key_id = "AKIAIOSFODNN7EXAMPLE"
-secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
-account_id = "000000000000"
-policy = [
-  { bucket = "predastore", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
-  { bucket = "northstar-zones", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"] },
-]
-
-# API request throttling (per-account, per-action token bucket)
-[ratelimit]
-enabled = false
-rate = 100          # Default sustained requests per second per account per action
-burst = 500         # Default maximum burst capacity
-
-# IAM authentication via NATS KV (enables multi-account S3 access)
-[iam]
-nats_url = "nats://10.0.0.2:4222"
-nats_token = "s3cr3t"
-nats_ca_cert = "/etc/spinifex/ca.pem"
-master_key_path = "/etc/spinifex/master.key"
-access_keys_bucket = "spinifex-iam-access-keys"

spinifex/migrate/predastoretopology/testdata/predastore_v4_singlenode.toml

@@ -1,109 +0,0 @@
-# Predastore S3-compatible storage configuration file in TOML format.
-# MULTI NODE used for testing
-version = "4"
-region = "ap-southeast-2"
-
-# Optionally configure port and host, or specify via daemon CLI options
-host = "0.0.0.0"
-port = 8443
-
-# Enable verbose debugging
-debug = false
-
-# Optionally configure base-path if full directory path not defined for "fs" buckets
-# base_path = "/home/predastore/data/"
-
-# Specify reed solomon encoding -
-# RS(2,1), 2 data, 1 parity. Minimum
-# RS(3,2), 3 data, 2 parity. Recommended for production
-[rs]
-data = 2
-parity = 1
-
-[[db]]
-id = 1
-host = "127.0.0.1"
-port = 6660
-#raft_port = 7660       # Optional: Raft consensus port (default: port + 1000)
-#advertise_host = ""    # Optional: Address to advertise to other Raft nodes
-path = "distributed/db/node-1/"
-access_key_id = "AKIAIOSFODNN7EXAMPLE"
-secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
-leader = true      # Bootstrap leader hint
-
-[[db]]
-id = 2
-host = "127.0.0.1"
-port = 6661
-path = "distributed/db/node-2/"
-access_key_id = "AKIAIOSFODNN7EXAMPLE"
-secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
-
-[[db]]
-id = 3
-host = "127.0.0.1"
-port = 6662
-path = "distributed/db/node-3/"
-access_key_id = "AKIAIOSFODNN7EXAMPLE"
-secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
-
-# Shard nodes
-# Using distributed hash ring for object > shard placement.
-[[nodes]]
-id = 1
-host = "127.0.0.1"
-port = 9991
-path = "distributed/nodes/node-1/"
-
-[[nodes]]
-id = 2
-host = "127.0.0.1"
-port = 9992
-path = "distributed/nodes/node-2/"
-
-[[nodes]]
-id = 3
-host = "127.0.0.1"
-port = 9993
-path = "distributed/nodes/node-3/"
-
-[[buckets]]
-name = "predastore"
-region = "ap-southeast-2"
-type = "distributed"
-public = false
-encryption = ""
-account_id = "000000000000"
-
-[[auth]]
-access_key_id = "AKIAIOSFODNN7EXAMPLE"
-secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
-account_id = "000000000000"
-policy = [
-  { bucket = "predastore", actions = ["s3:ListBucket",  "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
-]
-
-# API request throttling (per-account, per-action token bucket)
-[ratelimit]
-enabled = false
-rate = 100          # Default sustained requests per second per account per action
-burst = 500         # Default maximum burst capacity
-
-[ratelimit.action."s3:PutObject"]
-rate = 50
-burst = 200
-
-[ratelimit.action."s3:DeleteObject"]
-rate = 50
-burst = 200
-
-# IAM authentication via NATS KV (enables multi-account S3 access)
-[iam]
-nats_url = "nats://localhost:4222"
-nats_token = "s3cr3t"
-nats_ca_cert = "/etc/spinifex/ca.pem"
-master_key_path = "/etc/spinifex/master.key"
-access_keys_bucket = "spinifex-iam-access-keys"
-
-[compaction]
-interval_seconds = 300

spinifex/migrate/predastoretopology/testdata/predastore_v5_multinode.golden

@@ -1,109 +0,0 @@
-# Predastore S3-compatible storage configuration file in TOML format.
-# Generated for multi-node cluster
-version = "5"
-region = "ap-southeast-2"
-
-# The gateway listen address comes from the daemon CLI options: a top-level
-# host key would collide with the [[host]] topology tables below.
-
-debug = false
-
-[rs]
-data = 2
-parity = 1
-
-# One host per machine: a predastore process owning a socket and a data
-# directory. bind_addr binds every interface; public_addr is what the other
-# hosts dial.
-
-[[host]]
-id = 1
-bind_addr = "0.0.0.0:6660"
-public_addr = "10.0.0.1:6660"
-data_dir = "{{DATA_DIR}}/predastore/cluster"
-
-[[host]]
-id = 2
-bind_addr = "0.0.0.0:6660"
-public_addr = "10.0.0.2:6660"
-data_dir = "{{DATA_DIR}}/predastore/cluster"
-
-[[host]]
-id = 3
-bind_addr = "0.0.0.0:6660"
-public_addr = "10.0.0.3:6660"
-data_dir = "{{DATA_DIR}}/predastore/cluster"
-
-# Nodes are roles pinned to a host: shard-storage nodes hold erasure-coded
-# object shards, state replicas form the Raft quorum over global state.
-
-[[node]]
-id = 1
-host_id = 1
-role = "shard-storage"
-
-[[node]]
-id = 2
-host_id = 2
-role = "shard-storage"
-
-[[node]]
-id = 3
-host_id = 3
-role = "shard-storage"
-
-[[node]]
-id = 4
-host_id = 1
-role = "state-replica"
-
-[[node]]
-id = 5
-host_id = 2
-role = "state-replica"
-
-[[node]]
-id = 6
-host_id = 3
-role = "state-replica"
-
-[[buckets]]
-name = "predastore"
-region = "ap-southeast-2"
-type = "distributed"
-public = false
-encryption = ""
-account_id = "000000000000"
-
-# Northstar DNS zone storage. Read-only for the northstar service; the control
-# plane writes zone files with the system credentials.
-[[buckets]]
-name = "northstar-zones"
-region = "ap-southeast-2"
-type = "distributed"
-public = false
-encryption = ""
-account_id = "000000000000"
-
-[[auth]]
-access_key_id = "AKIAIOSFODNN7EXAMPLE"
-secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
-account_id = "000000000000"
-policy = [
-  { bucket = "predastore", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
-  { bucket = "northstar-zones", actions = ["s3:ListBucket", "s3:GetObject", "s3:PutObject", "s3:DeleteObject"] },
-]
-
-# API request throttling (per-account, per-action token bucket)
-[ratelimit]
-enabled = false
-rate = 100          # Default sustained requests per second per account per action
-burst = 500         # Default maximum burst capacity
-
-# IAM authentication via NATS KV (enables multi-account S3 access)
-[iam]
-nats_url = "nats://10.0.0.2:4222"
-nats_token = "s3cr3t"
-nats_ca_cert = "/etc/spinifex/ca.pem"
-master_key_path = "/etc/spinifex/master.key"
-access_keys_bucket = "spinifex-iam-access-keys"

spinifex/migrate/predastoretopology/testdata/predastore_v5_singlenode.golden

@@ -1,104 +0,0 @@
-# Predastore S3-compatible storage configuration file in TOML format.
-# MULTI NODE used for testing
-version = "5"
-region = "ap-southeast-2"
-
-# The gateway listen address comes from the daemon CLI options: a top-level
-# host key would collide with the [[host]] topology tables below.
-
-# Enable verbose debugging
-debug = false
-
-# Optionally configure base-path if full directory path not defined for "fs" buckets
-# base_path = "/home/predastore/data/"
-
-# Specify reed solomon encoding -
-# RS(2,1), 2 data, 1 parity. Minimum
-# RS(3,2), 3 data, 2 parity. Recommended for production
-[rs]
-data = 2
-parity = 1
-
-# One host per machine: a predastore process owning a socket and a data
-# directory. bind_addr binds every interface; public_addr is what the other
-# hosts dial.
-
-[[host]]
-id = 1
-bind_addr = "127.0.0.1:6660"
-public_addr = "127.0.0.1:6660"
-data_dir = "{{DATA_DIR}}/predastore/cluster"
-
-# Nodes are roles pinned to a host: shard-storage nodes hold erasure-coded
-# object shards, state replicas form the Raft quorum over global state.
-
-[[node]]
-id = 1
-host_id = 1
-role = "shard-storage"
-
-[[node]]
-id = 2
-host_id = 1
-role = "shard-storage"
-
-[[node]]
-id = 3
-host_id = 1
-role = "shard-storage"
-
-[[node]]
-id = 4
-host_id = 1
-role = "state-replica"
-
-[[node]]
-id = 5
-host_id = 1
-role = "state-replica"
-
-[[node]]
-id = 6
-host_id = 1
-role = "state-replica"
-
-[[buckets]]
-name = "predastore"
-region = "ap-southeast-2"
-type = "distributed"
-public = false
-encryption = ""
-account_id = "000000000000"
-
-[[auth]]
-access_key_id = "AKIAIOSFODNN7EXAMPLE"
-secret_access_key = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY"
-account_id = "000000000000"
-policy = [
-  { bucket = "predastore", actions = ["s3:ListBucket",  "s3:GetObject", "s3:PutObject", "s3:DeleteObject", "s3:ListAllMyBuckets"] },
-]
-
-# API request throttling (per-account, per-action token bucket)
-[ratelimit]
-enabled = false
-rate = 100          # Default sustained requests per second per account per action
-burst = 500         # Default maximum burst capacity
-
-[ratelimit.action."s3:PutObject"]
-rate = 50
-burst = 200
-
-[ratelimit.action."s3:DeleteObject"]
-rate = 50
-burst = 200
-
-# IAM authentication via NATS KV (enables multi-account S3 access)
-[iam]
-nats_url = "nats://localhost:4222"
-nats_token = "s3cr3t"
-nats_ca_cert = "/etc/spinifex/ca.pem"
-master_key_path = "/etc/spinifex/master.key"
-access_keys_bucket = "spinifex-iam-access-keys"
-
-[compaction]
-interval_seconds = 300

spinifex/services/awsgw/awsgw.go

@@ -13,7 +13,7 @@ import (
 	"strings"
 	"time"
 
-	"github.com/mulgadc/predastore/ratelimit"
+	"github.com/mulgadc/predastore/pkg/ratelimit"
 	"github.com/mulgadc/spinifex/internal/tlsconfig"
 	"github.com/mulgadc/spinifex/spinifex/admin"
 	"github.com/mulgadc/spinifex/spinifex/config"

spinifex/services/predastore/predastore.go

@@ -1,6 +1,7 @@
 package predastore
 
 import (
+	"cmp"
 	"context"
 	"fmt"
 	"log/slog"
@@ -8,50 +9,48 @@ import (
 	"os/signal"
 	"sync"
 	"syscall"
-	"time"
 
-	"github.com/mulgadc/predastore/clusterrun"
+	pds "github.com/mulgadc/predastore"
 	"github.com/mulgadc/predastore/pkg/masterkey"
-	"github.com/mulgadc/predastore/s3"
 	"github.com/mulgadc/spinifex/spinifex/utils"
-	"golang.org/x/sync/errgroup"
 )
 
 var serviceName = "predastore"
 
-// Config holds the configuration for the predastore service.
+// Config is what spinifex.toml and the service flags contribute. Everything
+// else — the cluster topology, the buckets, the credentials — comes from the
+// predastore configuration file at ConfigPath.
 type Config struct {
 	ConfigPath string
 	Port       int
-	Host       string
-	Debug      bool
-	BasePath   string
-	TlsCert    string
-	TlsKey     string
+	// Host is where the S3 API listens, and only that: the cluster plane binds
+	// the host's own address from the predastore configuration file.
+	Host    string
+	TlsCert string
+	TlsKey  string
+
+	// BasePath is where this service keeps its pid file. Predastore itself has
+	// no base path — its directories come from the configuration file — but
+	// Stop and Status find the running process through this one.
+	BasePath string
 
 	// EncryptionKeyFile is the path to this node's 32-byte AES-256 master
 	// key for predastore at-rest encryption. Each node has its own key;
 	// fragments are only ever opened on the node that sealed them.
 	EncryptionKeyFile string
 
-	// HostID selects which host of the predastore topology this process is.
-	// A host owns one socket and one data directory, and runs the nodes
-	// pinned to it. Zero runs every node in the topology in this process,
-	// which is the single-node deployment and needs no intra-cluster network.
+	// HostID selects which [[host]] of the predastore topology this process
+	// runs: the S3 gate pinned to it and every cluster node beside it. It must
+	// name a host the configuration file declares.
 	HostID int
-
-	// Profiling
-	PprofEnabled    bool
-	PprofOutputPath string
 }
 
-// Service wraps the predastore S3 gateway and the cluster runtime beneath it.
+// Service runs one predastore host in this process.
 type Service struct {
 	Config *Config
-	server *s3.Server
 
 	// stop cancels the process context Start runs under, so Shutdown can
-	// drain the gateway and the nodes together. Guarded because Shutdown is
+	// drain the gate and the nodes together. Guarded because Shutdown is
 	// called from a different goroutine than Start.
 	mu   sync.Mutex
 	stop context.CancelFunc
@@ -69,7 +68,8 @@ func New(config any) (svc *Service, err error) {
 	return svc, nil
 }
 
-// Start starts the predastore service.
+// Start serves this host until the service context is cancelled, blocking for
+// as long as it serves.
 func (svc *Service) Start() (int, error) {
 	if svc.Config.EncryptionKeyFile == "" {
 		return 0, fmt.Errorf("predastore encryption key file is required (set EncryptionKeyFile)")
@@ -79,109 +79,86 @@ func (svc *Service) Start() (int, error) {
 		return 0, fmt.Errorf("write pid file: %w", err)
 	}
 
-	rt, err := svc.buildRuntime()
+	cfg, err := pds.LoadConfig(svc.Config.ConfigPath)
+	if err != nil {
+		return 0, fmt.Errorf("read predastore config %s: %w", svc.Config.ConfigPath, err)
+	}
+	hostID, err := svc.mergeHost(cfg)
 	if err != nil {
 		return 0, err
 	}
 
-	server, err := s3.NewServer(
-		s3.WithConfigPath(svc.Config.ConfigPath),
-		s3.WithAddress(svc.Config.Host, svc.Config.Port),
-		s3.WithTLS(svc.Config.TlsCert, svc.Config.TlsKey),
-		s3.WithBasePath(svc.Config.BasePath),
-		s3.WithDebug(svc.Config.Debug),
-		s3.WithPprof(svc.Config.PprofEnabled, svc.Config.PprofOutputPath),
-		s3.WithEncryptionKeyFile(svc.Config.EncryptionKeyFile),
-		s3.WithPreparedBackend(rt.Backend),
-	)
-	if err != nil {
-		rt.Close()
-		slog.Error("Failed to create predastore server", "error", err)
-		return 0, err
-	}
-
-	// One context for the whole service: a signal, a fatal gateway error or
-	// Shutdown all stop the cluster nodes and the gateway together.
-	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
-	defer stop()
-
-	svc.mu.Lock()
-	svc.server = server
-	svc.stop = stop
-	svc.mu.Unlock()
-
-	g, gctx := errgroup.WithContext(ctx)
-	g.Go(func() error { return rt.Run(gctx) })
-	g.Go(func() error {
-		// Serving before consensus settles would fail writes that would have
-		// succeeded a moment later; a timeout degrades rather than aborts,
-		// since the state client retries.
-		if err := rt.WaitReady(30 * time.Second); err != nil {
-			slog.Warn("No predastore leader elected within timeout, serving anyway", "error", err)
-		}
-		return svc.serve(gctx, server)
-	})
-
-	if err := g.Wait(); err != nil {
-		slog.Error("Predastore service exited", "error", err)
-		return 0, err
-	}
-
-	return os.Getpid(), nil
-}
-
-// buildRuntime reads the predastore topology and assembles the cluster nodes
-// this process is responsible for, returning the backend the S3 gateway runs
-// on top of.
-func (svc *Service) buildRuntime() (*clusterrun.Runtime, error) {
-	cfg := &s3.Config{ConfigPath: svc.Config.ConfigPath, BasePath: svc.Config.BasePath}
-	if err := cfg.ReadConfig(); err != nil {
-		return nil, fmt.Errorf("read predastore config %s: %w", svc.Config.ConfigPath, err)
-	}
-
-	// An unset host means this process is the whole cluster, so it runs every
-	// node over the in-process pipe and opens no intra-cluster socket.
-	nodeIDs := clusterrun.AllNodeIDs(cfg)
-	if svc.Config.HostID > 0 {
-		nodeIDs = clusterrun.NodeIDsForHost(cfg, svc.Config.HostID)
-		if len(nodeIDs) == 0 {
-			return nil, fmt.Errorf("predastore host %d has no nodes in %s", svc.Config.HostID, svc.Config.ConfigPath)
-		}
-	}
-
 	key, err := masterkey.Load(svc.Config.EncryptionKeyFile)
 	if err != nil {
-		return nil, fmt.Errorf("load predastore master key: %w", err)
+		return 0, fmt.Errorf("load predastore master key: %w", err)
 	}
 
-	rt, err := clusterrun.Build(cfg, nodeIDs, svc.Config.TlsCert, svc.Config.TlsKey, key)
-	if err != nil {
-		return nil, fmt.Errorf("build predastore cluster runtime: %w", err)
+	// One context for the whole service: a signal, a fatal node error or
+	// Shutdown stops the gate and every local node together.
+	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
+	defer stop()
+
+	svc.mu.Lock()
+	svc.stop = stop
+	svc.mu.Unlock()
+
+	if err := pds.Run(ctx, pds.Options{
+		Config:    cfg,
+		HostID:    hostID,
+		MasterKey: key,
+	}); err != nil {
+		slog.Error("Predastore service exited", "error", err)
+		return 0, err
 	}
-	return rt, nil
+	return os.Getpid(), nil
 }
 
-// serve runs the S3 gateway until the context is cancelled, then drains it
-// within a bounded grace period. A gateway that cannot bind is fatal: leaving
-// the cluster running headless would look healthy while serving nothing.
-func (svc *Service) serve(ctx context.Context, server *s3.Server) error {
-	if err := server.ListenAndServeAsync(); err != nil {
-		return fmt.Errorf("start predastore s3 gateway: %w", err)
+// mergeHost settles the host-local fields spinifex.toml owns into the parsed
+// configuration, then rechecks the merged tree: a flag can supply an address or
+// a port the file never had, and the collision checks only mean something once
+// those are in place. It returns the host this process runs.
+func (svc *Service) mergeHost(cfg *pds.Config) (pds.HostID, error) {
+	if svc.Config.HostID <= 0 {
+		return 0, fmt.Errorf("predastore host id is required and must name a [[host]] in %s", svc.Config.ConfigPath)
+	}
+	hostID := pds.HostID(svc.Config.HostID)
+
+	var host *pds.HostConfig
+	for i := range cfg.Hosts {
+		if cfg.Hosts[i].ID == hostID {
+			host = &cfg.Hosts[i]
+			break
+		}
+	}
+	if host == nil {
+		return 0, fmt.Errorf("predastore host %d is not in %s", svc.Config.HostID, svc.Config.ConfigPath)
 	}
 
-	select {
-	case <-ctx.Done():
-	case err := <-server.ServeError():
-		return fmt.Errorf("predastore s3 gateway: %w", err)
-	}
+	// host.BindAddr is left as the file has it: it is the cluster plane, there
+	// is no flag for it, and predastore falls back to the host's dial address
+	// when it is empty.
+	host.TLSCert = cmp.Or(svc.Config.TlsCert, host.TLSCert)
+	host.TLSKey = cmp.Or(svc.Config.TlsKey, host.TLSKey)
 
-	slog.Info("Shutting down predastore service")
-	shutdownCtx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
-	defer cancel()
-	if err := server.Shutdown(shutdownCtx); err != nil {
-		return fmt.Errorf("shut down predastore s3 gateway: %w", err)
+	// The endpoint spinifex.toml advertises is this host's gate, found by role:
+	// nothing fixes its position among the host's nodes.
+	gate := -1
+	for i := range host.Nodes {
+		if host.Nodes[i].Role == pds.RoleGate {
+			gate = i
+			break
+		}
 	}
-	return nil
+	if gate < 0 {
+		return 0, fmt.Errorf("predastore host %d runs no gate node in %s", svc.Config.HostID, svc.Config.ConfigPath)
+	}
+	// The address spinifex.toml carries is the S3 endpoint, which is the gate's
+	// alone. Settling it on the host would put raft and blob traffic on the
+	// public interface with it, which is the one place they must never be.
+	host.Nodes[gate].BindAddr = cmp.Or(svc.Config.Host, host.Nodes[gate].BindAddr)
+	host.Nodes[gate].Port = cmp.Or(svc.Config.Port, host.Nodes[gate].Port)
+
+	return hostID, cfg.Validate()
 }
 
 // Stop stops the predastore service.
@@ -195,8 +172,8 @@ func (svc *Service) Status() (string, error) {
 }
 
 // Shutdown gracefully shuts down the predastore service. When this process is
-// the one running the cluster it cancels the service context, which drains the
-// gateway and every local node; otherwise it signals the running process.
+// the one running the host it cancels the service context, which drains the
+// gate and every local node; otherwise it signals the running process.
 func (svc *Service) Shutdown() error {
 	svc.mu.Lock()
 	stop := svc.stop

spinifex/services/predastore/predastore_integration_test.go

@@ -9,6 +9,7 @@ import (
 	"fmt"
 	"io"
 	"net/http"
+	"os"
 	"slices"
 	"strings"
 	"testing"
@@ -22,6 +23,15 @@ import (
 	"github.com/stretchr/testify/assert"
 )
 
+// TestMain drains the shared predastore fixture once every test has run. The
+// fixture outlives any single test on purpose, so this is the only point at
+// which its cluster goroutines can be stopped rather than abandoned.
+func TestMain(m *testing.M) {
+	code := m.Run()
+	testpredastore.Stop()
+	os.Exit(code)
+}
+
 // Test credentials from config, mirrored from the shared fixture
 // (testpredastore.AccessKey etc.) so this file has a single source of truth
 // for the daemon's seeded auth and default buckets.

spinifex/services/predastore/predastore_runtime_test.go

@@ -1,504 +0,0 @@
-package predastore
-
-import (
-	"context"
-	"crypto/ecdsa"
-	"crypto/elliptic"
-	"crypto/rand"
-	"crypto/x509"
-	"crypto/x509/pkix"
-	"encoding/pem"
-	"fmt"
-	"math/big"
-	"net"
-	"os"
-	"path/filepath"
-	"strconv"
-	"strings"
-	"sync/atomic"
-	"testing"
-	"time"
-
-	"github.com/mulgadc/predastore/clusterrun"
-	"github.com/mulgadc/predastore/s3"
-	"github.com/stretchr/testify/assert"
-	"github.com/stretchr/testify/require"
-)
-
-const (
-	roleShardStorage = "shard-storage"
-	roleStateReplica = "state-replica"
-)
-
-// nodeIDSeq hands out node ids that are unique within this test binary. The
-// pipe transport's listener registry is process-wide, and this package's
-// integration tests hold a fixture daemon on the low ids for the life of the
-// binary, so topologies here must not reuse them.
-var nodeIDSeq atomic.Int64
-
-func nextNodeIDs(n int) []int {
-	ids := make([]int, n)
-	for i := range ids {
-		ids[i] = 100 + int(nodeIDSeq.Add(1))
-	}
-	return ids
-}
-
-// hostSpec and nodeSpec are one entry each of a topology's [[host]] and
-// [[node]] tables.
-type hostSpec struct {
-	id      int
-	dataDir string
-}
-
-type nodeSpec struct {
-	id     int
-	hostID int
-	role   string
-}
-
-// writeTopology writes a predastore config for hosts and nodes, returning its
-// path. Hosts bind an ephemeral port so several topologies can coexist in one
-// test binary.
-func writeTopology(t *testing.T, dir string, hosts []hostSpec, nodes []nodeSpec) string {
-	t.Helper()
-
-	var b strings.Builder
-	fmt.Fprintf(&b, "version = \"1.0\"\nregion = \"us-east-1\"\nbase_path = %q\n\n[rs]\ndata = 1\nparity = 1\n", dir)
-	for _, h := range hosts {
-		fmt.Fprintf(&b, "\n[[host]]\nid = %d\nbind_addr = \"127.0.0.1:0\"\npublic_addr = \"127.0.0.1:0\"\ndata_dir = %q\n", h.id, h.dataDir)
-	}
-	for _, n := range nodes {
-		fmt.Fprintf(&b, "\n[[node]]\nid = %d\nhost_id = %d\nrole = %q\n", n.id, n.hostID, n.role)
-	}
-
-	path := filepath.Join(dir, "predastore.toml")
-	require.NoError(t, os.WriteFile(path, []byte(b.String()), 0600))
-	return path
-}
-
-// writeMasterKey writes a 32-byte master key at mode. The mode matters:
-// masterkey.Load is fail-closed on anything group- or other-readable.
-func writeMasterKey(t *testing.T, dir string, mode os.FileMode) string {
-	t.Helper()
-
-	key := make([]byte, 32)
-	_, err := rand.Read(key)
-	require.NoError(t, err)
-
-	path := filepath.Join(dir, "master.key")
-	require.NoError(t, os.WriteFile(path, key, mode))
-	return path
-}
-
-// writeCertificate writes a self-signed P-256 certificate serving both the
-// intra-cluster QUIC socket and the S3 gateway. Nothing verifies it: the peers
-// are this process and the tests' own dialers.
-func writeCertificate(t *testing.T, dir string) (certPath, keyPath string) {
-	t.Helper()
-
-	priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
-	require.NoError(t, err)
-
-	tmpl := x509.Certificate{
-		SerialNumber:          big.NewInt(1),
-		Subject:               pkix.Name{CommonName: "localhost"},
-		NotBefore:             time.Now().Add(-time.Hour),
-		NotAfter:              time.Now().Add(24 * time.Hour),
-		KeyUsage:              x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment,
-		ExtKeyUsage:           []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
-		IPAddresses:           []net.IP{net.ParseIP("127.0.0.1")},
-		DNSNames:              []string{"localhost"},
-		BasicConstraintsValid: true,
-	}
-	der, err := x509.CreateCertificate(rand.Reader, &tmpl, &tmpl, &priv.PublicKey, priv)
-	require.NoError(t, err)
-
-	certPath = filepath.Join(dir, "tls.crt")
-	require.NoError(t, os.WriteFile(certPath, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}), 0600))
-
-	keyDER, err := x509.MarshalECPrivateKey(priv)
-	require.NoError(t, err)
-	keyPath = filepath.Join(dir, "tls.key")
-	require.NoError(t, os.WriteFile(keyPath, pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: keyDER}), 0600))
-
-	return certPath, keyPath
-}
-
-// freePort reserves and releases a loopback port, so a gateway under test
-// binds somewhere the fixture daemon and other packages are not.
-func freePort(t *testing.T) int {
-	t.Helper()
-
-	ln, err := net.Listen("tcp", "127.0.0.1:0")
-	require.NoError(t, err)
-	defer ln.Close()
-	return ln.Addr().(*net.TCPAddr).Port
-}
-
-// newTwoHostConfig returns the config for the first host of a two-host
-// topology: the shard stores run here, the state replica belongs to the other
-// host's process. Returns the local node ids and the remote one.
-func newTwoHostConfig(t *testing.T) (cfg *Config, local []int, remote int) {
-	t.Helper()
-
-	dir := t.TempDir()
-	certPath, keyPath := writeCertificate(t, dir)
-	ids := nextNodeIDs(3)
-
-	hosts := []hostSpec{
-		{id: 1, dataDir: filepath.Join(dir, "host-1")},
-		{id: 2, dataDir: filepath.Join(dir, "host-2")},
-	}
-	nodes := []nodeSpec{
-		{id: ids[0], hostID: 1, role: roleShardStorage},
-		{id: ids[1], hostID: 1, role: roleShardStorage},
-		{id: ids[2], hostID: 2, role: roleStateReplica},
-	}
-
-	cfg = &Config{
-		ConfigPath:        writeTopology(t, dir, hosts, nodes),
-		Host:              "127.0.0.1",
-		Port:              freePort(t),
-		BasePath:          dir,
-		TlsCert:           certPath,
-		TlsKey:            keyPath,
-		EncryptionKeyFile: writeMasterKey(t, dir, 0600),
-		HostID:            1,
-	}
-	return cfg, ids[:2], ids[2]
-}
-
-// newSingleHostConfig returns the single-node deployment's config: one host
-// running every node in this process, over the in-process pipe.
-func newSingleHostConfig(t *testing.T) *Config {
-	t.Helper()
-
-	dir := t.TempDir()
-	certPath, keyPath := writeCertificate(t, dir)
-	ids := nextNodeIDs(3)
-
-	hosts := []hostSpec{{id: 1, dataDir: filepath.Join(dir, "host-1")}}
-	nodes := []nodeSpec{
-		{id: ids[0], hostID: 1, role: roleShardStorage},
-		{id: ids[1], hostID: 1, role: roleShardStorage},
-		{id: ids[2], hostID: 1, role: roleStateReplica},
-	}
-
-	// HostID is left zero: this process is the whole cluster.
-	return &Config{
-		ConfigPath:        writeTopology(t, dir, hosts, nodes),
-		Host:              "127.0.0.1",
-		Port:              freePort(t),
-		BasePath:          dir,
-		TlsCert:           certPath,
-		TlsKey:            keyPath,
-		EncryptionKeyFile: writeMasterKey(t, dir, 0600),
-	}
-}
-
-// newGateway builds the S3 server serve runs, on top of an already-built
-// cluster runtime.
-func newGateway(t *testing.T, cfg *Config, rt *clusterrun.Runtime) *s3.Server {
-	t.Helper()
-
-	server, err := s3.NewServer(
-		s3.WithConfigPath(cfg.ConfigPath),
-		s3.WithAddress(cfg.Host, cfg.Port),
-		s3.WithTLS(cfg.TlsCert, cfg.TlsKey),
-		s3.WithBasePath(cfg.BasePath),
-		s3.WithEncryptionKeyFile(cfg.EncryptionKeyFile),
-		s3.WithPreparedBackend(rt.Backend),
-	)
-	require.NoError(t, err)
-	return server
-}
-
-// requireGatewayAccepts waits for the S3 gateway to accept connections, which
-// is also the point at which Start has published the cancel func Shutdown
-// needs.
-func requireGatewayAccepts(t *testing.T, cfg *Config) {
-	t.Helper()
-
-	addr := net.JoinHostPort(cfg.Host, strconv.Itoa(cfg.Port))
-	require.Eventually(t, func() bool {
-		conn, err := net.DialTimeout("tcp", addr, time.Second)
-		if err != nil {
-			return false
-		}
-		conn.Close()
-		return true
-	}, 30*time.Second, 100*time.Millisecond, "gateway never accepted connections on %s", addr)
-}
-
-// TestBuildRuntimeErrors covers every way assembling the cluster runtime can
-// fail before a single node is running.
-func TestBuildRuntimeErrors(t *testing.T) {
-	testCases := []struct {
-		name    string
-		config  func(t *testing.T, dir string) *Config
-		wantErr string
-	}{
-		{
-			name: "Unreadable Config",
-			config: func(t *testing.T, dir string) *Config {
-				return &Config{
-					ConfigPath:        filepath.Join(dir, "absent.toml"),
-					BasePath:          dir,
-					EncryptionKeyFile: writeMasterKey(t, dir, 0600),
-				}
-			},
-			wantErr: "read predastore config",
-		},
-		{
-			name: "Host Owns No Nodes",
-			config: func(t *testing.T, dir string) *Config {
-				ids := nextNodeIDs(1)
-				return &Config{
-					ConfigPath: writeTopology(t, dir,
-						[]hostSpec{{id: 1, dataDir: filepath.Join(dir, "host-1")}},
-						[]nodeSpec{{id: ids[0], hostID: 1, role: roleStateReplica}}),
-					BasePath:          dir,
-					EncryptionKeyFile: writeMasterKey(t, dir, 0600),
-					HostID:            9,
-				}
-			},
-			wantErr: "predastore host 9 has no nodes in",
-		},
-		{
-			name: "Missing Master Key",
-			config: func(t *testing.T, dir string) *Config {
-				ids := nextNodeIDs(1)
-				return &Config{
-					ConfigPath: writeTopology(t, dir,
-						[]hostSpec{{id: 1, dataDir: filepath.Join(dir, "host-1")}},
-						[]nodeSpec{{id: ids[0], hostID: 1, role: roleStateReplica}}),
-					BasePath:          dir,
-					EncryptionKeyFile: filepath.Join(dir, "absent.key"),
-				}
-			},
-			wantErr: "load predastore master key",
-		},
-		{
-			name: "World Readable Master Key",
-			config: func(t *testing.T, dir string) *Config {
-				ids := nextNodeIDs(1)
-				return &Config{
-					ConfigPath: writeTopology(t, dir,
-						[]hostSpec{{id: 1, dataDir: filepath.Join(dir, "host-1")}},
-						[]nodeSpec{{id: ids[0], hostID: 1, role: roleStateReplica}}),
-					BasePath:          dir,
-					EncryptionKeyFile: writeMasterKey(t, dir, 0644),
-				}
-			},
-			wantErr: "load predastore master key",
-		},
-		{
-			name: "Remote Nodes Without TLS Material",
-			config: func(t *testing.T, dir string) *Config {
-				ids := nextNodeIDs(2)
-				return &Config{
-					ConfigPath: writeTopology(t, dir,
-						[]hostSpec{
-							{id: 1, dataDir: filepath.Join(dir, "host-1")},
-							{id: 2, dataDir: filepath.Join(dir, "host-2")},
-						},
-						[]nodeSpec{
-							{id: ids[0], hostID: 1, role: roleShardStorage},
-							{id: ids[1], hostID: 2, role: roleStateReplica},
-						}),
-					BasePath:          dir,
-					EncryptionKeyFile: writeMasterKey(t, dir, 0600),
-					HostID:            1,
-					// TlsCert and TlsKey deliberately left empty: reaching the
-					// other host needs the intra-cluster socket.
-				}
-			},
-			wantErr: "build predastore cluster runtime",
-		},
-	}
-
-	for _, tc := range testCases {
-		t.Run(tc.name, func(t *testing.T) {
-			svc, err := New(tc.config(t, t.TempDir()))
-			require.NoError(t, err)
-
-			rt, err := svc.buildRuntime()
-
-			require.Error(t, err)
-			assert.Nil(t, rt)
-			assert.Contains(t, err.Error(), tc.wantErr)
-		})
-	}
-}
-
-// TestBuildRuntimeRunsOnlyThisHostsNodes verifies host selection: a host runs
-// the nodes pinned to it and leaves the rest to the processes that own them.
-func TestBuildRuntimeRunsOnlyThisHostsNodes(t *testing.T) {
-	cfg, local, remote := newTwoHostConfig(t)
-
-	svc, err := New(cfg)
-	require.NoError(t, err)
-
-	rt, err := svc.buildRuntime()
-	require.NoError(t, err)
-	t.Cleanup(rt.Close)
-
-	assert.NotNil(t, rt.Backend)
-	for _, id := range local {
-		assert.DirExists(t, filepath.Join(cfg.BasePath, "host-1", fmt.Sprintf("node-%d", id)))
-	}
-	assert.NoDirExists(t, filepath.Join(cfg.BasePath, "host-2", fmt.Sprintf("node-%d", remote)))
-}
-
-// TestServeReportsGatewayFailures covers the two ways the S3 gateway can fail
-// the service: refusing to start, and dying once started.
-func TestServeReportsGatewayFailures(t *testing.T) {
-	testCases := []struct {
-		name    string
-		tls     func(cfg *Config)
-		wantErr string
-	}{
-		{
-			name:    "Missing TLS Material",
-			tls:     func(cfg *Config) { cfg.TlsCert, cfg.TlsKey = "", "" },
-			wantErr: "start predastore s3 gateway",
-		},
-		{
-			name: "Unloadable Certificate",
-			tls: func(cfg *Config) {
-				cfg.TlsCert = filepath.Join(cfg.BasePath, "absent.crt")
-				cfg.TlsKey = filepath.Join(cfg.BasePath, "absent.key")
-			},
-			wantErr: "predastore s3 gateway",
-		},
-	}
-
-	for _, tc := range testCases {
-		t.Run(tc.name, func(t *testing.T) {
-			cfg, _, _ := newTwoHostConfig(t)
-			svc, err := New(cfg)
-			require.NoError(t, err)
-
-			rt, err := svc.buildRuntime()
-			require.NoError(t, err)
-			t.Cleanup(rt.Close)
-
-			// The runtime is built first: reaching the other host's node needs
-			// the certificate this case is about to take away.
-			tc.tls(cfg)
-			gateway := newGateway(t, cfg, rt)
-
-			ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
-			defer cancel()
-
-			err = svc.serve(ctx, gateway)
-
-			require.Error(t, err)
-			assert.Contains(t, err.Error(), tc.wantErr)
-		})
-	}
-}
-
-// TestServeDrainsOnContextCancel verifies the gateway stops serving when the
-// service context is cancelled, rather than outliving the cluster beneath it.
-func TestServeDrainsOnContextCancel(t *testing.T) {
-	cfg, _, _ := newTwoHostConfig(t)
-	svc, err := New(cfg)
-	require.NoError(t, err)
-
-	rt, err := svc.buildRuntime()
-	require.NoError(t, err)
-	t.Cleanup(rt.Close)
-
-	ctx, cancel := context.WithCancel(context.Background())
-	defer cancel()
-
-	gateway := newGateway(t, cfg, rt)
-	served := make(chan error, 1)
-	go func() { served <- svc.serve(ctx, gateway) }()
-
-	requireGatewayAccepts(t, cfg)
-	cancel()
-
-	select {
-	case err := <-served:
-		assert.NoError(t, err)
-	case <-time.After(30 * time.Second):
-		t.Fatal("serve did not return after the context was cancelled")
-	}
-}
-
-// TestStartFailsWhenGatewayCannotServe verifies a fatal gateway error takes
-// the whole service down: a cluster left running headless would look healthy
-// while serving nothing.
-func TestStartFailsWhenGatewayCannotServe(t *testing.T) {
-	cfg := newSingleHostConfig(t)
-	cfg.TlsCert = filepath.Join(cfg.BasePath, "absent.crt")
-	cfg.TlsKey = filepath.Join(cfg.BasePath, "absent.key")
-
-	svc, err := New(cfg)
-	require.NoError(t, err)
-
-	type result struct {
-		pid int
-		err error
-	}
-	done := make(chan result, 1)
-	go func() {
-		pid, err := svc.Start()
-		done <- result{pid: pid, err: err}
-	}()
-
-	select {
-	case res := <-done:
-		require.Error(t, res.err)
-		assert.Zero(t, res.pid)
-		assert.Contains(t, res.err.Error(), "predastore s3 gateway")
-	case <-time.After(60 * time.Second):
-		t.Fatal("Start did not return after the gateway failed")
-	}
-}
-
-// TestStartServesUntilShutdown covers the service lifecycle end to end: Start
-// brings the cluster nodes and the gateway up together, and Shutdown drains
-// both by cancelling the context they share.
-func TestStartServesUntilShutdown(t *testing.T) {
-	cfg := newSingleHostConfig(t)
-	svc, err := New(cfg)
-	require.NoError(t, err)
-
-	type result struct {
-		pid int
-		err error
-	}
-	done := make(chan result, 1)
-	go func() {
-		pid, err := svc.Start()
-		done <- result{pid: pid, err: err}
-	}()
-
-	// Shutdown must not run before Start publishes its cancel func, or it
-	// falls through to the pid file and signals this test binary.
-	requireGatewayAccepts(t, cfg)
-	require.NoError(t, svc.Shutdown())
-
-	select {
-	case res := <-done:
-		require.NoError(t, res.err)
-		assert.Equal(t, os.Getpid(), res.pid)
-	case <-time.After(60 * time.Second):
-		t.Fatal("Start did not return after Shutdown")
-	}
-}
-
-// TestShutdownWithoutRunningCluster verifies the fallback path: with no
-// cluster in this process there is no context to cancel, so Shutdown goes
-// through the pid file instead.
-func TestShutdownWithoutRunningCluster(t *testing.T) {
-	svc, err := New(&Config{BasePath: t.TempDir()})
-	require.NoError(t, err)
-
-	// No pid file was ever written, so this reports rather than signals.
-	assert.Error(t, svc.Shutdown())
-}

spinifex/services/predastore/predastore_start_test.go

@@ -0,0 +1,216 @@
+package predastore
+
+import (
+	"crypto/ecdsa"
+	"crypto/elliptic"
+	"crypto/rand"
+	"crypto/x509"
+	"crypto/x509/pkix"
+	"encoding/pem"
+	"math/big"
+	"net"
+	"os"
+	"path/filepath"
+	"strconv"
+	"testing"
+	"time"
+
+	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
+)
+
+// The cluster these tests start binds 127.0.0.2 so nothing it registers — the
+// in-process pipe transport keys its listeners by address and port, for the
+// life of the binary — can collide with the shared fixture on 127.0.0.1.
+const (
+	startAddr     = "127.0.0.2"
+	startBlobPort = 26660
+	startMetaPort = 27660
+)
+
+// writeMasterKey writes a 32-byte master key at mode. The mode matters:
+// masterkey.Load is fail-closed on anything group- or other-readable.
+func writeMasterKey(t *testing.T, dir string, mode os.FileMode) string {
+	t.Helper()
+
+	key := make([]byte, 32)
+	_, err := rand.Read(key)
+	require.NoError(t, err)
+
+	path := filepath.Join(dir, "master.key")
+	require.NoError(t, os.WriteFile(path, key, mode))
+	return path
+}
+
+// writeCertificate writes a self-signed P-256 certificate for the S3 gate.
+// Nothing verifies it: the only client is this test's own dialer.
+func writeCertificate(t *testing.T, dir string) (certPath, keyPath string) {
+	t.Helper()
+
+	priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
+	require.NoError(t, err)
+
+	tmpl := x509.Certificate{
+		SerialNumber:          big.NewInt(1),
+		Subject:               pkix.Name{CommonName: "localhost"},
+		NotBefore:             time.Now().Add(-time.Hour),
+		NotAfter:              time.Now().Add(24 * time.Hour),
+		KeyUsage:              x509.KeyUsageDigitalSignature | x509.KeyUsageKeyEncipherment,
+		ExtKeyUsage:           []x509.ExtKeyUsage{x509.ExtKeyUsageServerAuth},
+		IPAddresses:           []net.IP{net.ParseIP(startAddr)},
+		BasicConstraintsValid: true,
+	}
+	der, err := x509.CreateCertificate(rand.Reader, &tmpl, &tmpl, &priv.PublicKey, priv)
+	require.NoError(t, err)
+
+	certPath = filepath.Join(dir, "tls.crt")
+	require.NoError(t, os.WriteFile(certPath, pem.EncodeToMemory(&pem.Block{Type: "CERTIFICATE", Bytes: der}), 0600))
+
+	der, err = x509.MarshalECPrivateKey(priv)
+	require.NoError(t, err)
+	keyPath = filepath.Join(dir, "tls.key")
+	require.NoError(t, os.WriteFile(keyPath, pem.EncodeToMemory(&pem.Block{Type: "EC PRIVATE KEY", Bytes: der}), 0600))
+
+	return certPath, keyPath
+}
+
+// freePort reserves and releases a port on the test address, so the gate binds
+// somewhere no other test in this binary has taken.
+func freePort(t *testing.T) int {
+	t.Helper()
+
+	ln, err := net.Listen("tcp", net.JoinHostPort(startAddr, "0"))
+	require.NoError(t, err)
+	defer ln.Close()
+	return ln.Addr().(*net.TCPAddr).Port
+}
+
+// servableHost writes the smallest topology a host can actually serve: a gate
+// for the S3 API, the one blob node RS(1,0) needs, and a meta replica, since
+// the gate reaches the metadata plane through at least one.
+func servableHost(t *testing.T, dir string, gatePort int) string {
+	t.Helper()
+
+	certPath, keyPath := writeCertificate(t, dir)
+	return writeTopology(t, dir,
+		hostSpec{
+			id:      1,
+			addr:    startAddr,
+			dataDir: filepath.Join(dir, "data"),
+			tlsCert: certPath,
+			tlsKey:  keyPath,
+		},
+		[]nodeSpec{
+			{id: 1, role: "gate", port: gatePort},
+			{id: 2, role: "blob", port: startBlobPort},
+			{id: 3, role: "meta", port: startMetaPort},
+		})
+}
+
+// TestStartRejectsUnreadableConfig covers the first thing Start reads after the
+// pid file: a configuration path that names no file stops the service before
+// any node is built.
+func TestStartRejectsUnreadableConfig(t *testing.T) {
+	dir := t.TempDir()
+	svc, err := New(&Config{
+		ConfigPath:        filepath.Join(dir, "absent.toml"),
+		BasePath:          dir,
+		EncryptionKeyFile: writeMasterKey(t, dir, 0600),
+		HostID:            1,
+	})
+	require.NoError(t, err)
+
+	pid, err := svc.Start()
+
+	require.Error(t, err)
+	assert.Zero(t, pid)
+	assert.Contains(t, err.Error(), "read predastore config")
+}
+
+// TestStartRejectsUnknownHost covers the merge failing inside Start: the error
+// travels out unwrapped, because it already names the host and the file.
+func TestStartRejectsUnknownHost(t *testing.T) {
+	dir := t.TempDir()
+	svc, err := New(&Config{
+		ConfigPath:        singleHost(t, dir),
+		BasePath:          dir,
+		EncryptionKeyFile: writeMasterKey(t, dir, 0600),
+		HostID:            9,
+	})
+	require.NoError(t, err)
+
+	pid, err := svc.Start()
+
+	require.Error(t, err)
+	assert.Zero(t, pid)
+	assert.Contains(t, err.Error(), "predastore host 9 is not in")
+}
+
+// TestStartRejectsUnprotectedMasterKey covers the at-rest key: predastore
+// refuses one any other account can read, and the service must surface that
+// rather than start without encryption.
+func TestStartRejectsUnprotectedMasterKey(t *testing.T) {
+	dir := t.TempDir()
+	svc, err := New(&Config{
+		ConfigPath:        singleHost(t, dir),
+		BasePath:          dir,
+		EncryptionKeyFile: writeMasterKey(t, dir, 0644),
+		HostID:            1,
+	})
+	require.NoError(t, err)
+
+	pid, err := svc.Start()
+
+	require.Error(t, err)
+	assert.Zero(t, pid)
+	assert.Contains(t, err.Error(), "load predastore master key")
+}
+
+// TestStartServesUntilShutdown covers the service lifecycle end to end: Start
+// blocks for as long as the host serves, and Shutdown cancels the context the
+// gate and the local nodes share, which is what lets Start return at all.
+func TestStartServesUntilShutdown(t *testing.T) {
+	dir := t.TempDir()
+	basePath := t.TempDir()
+	gatePort := freePort(t)
+	svc, err := New(&Config{
+		ConfigPath:        servableHost(t, dir, gatePort),
+		BasePath:          basePath,
+		EncryptionKeyFile: writeMasterKey(t, dir, 0600),
+		HostID:            1,
+	})
+	require.NoError(t, err)
+
+	type result struct {
+		pid int
+		err error
+	}
+	done := make(chan result, 1)
+	go func() {
+		pid, err := svc.Start()
+		done <- result{pid: pid, err: err}
+	}()
+
+	// Shutdown must not run before Start has published its cancel func, and an
+	// accepted connection is the first observable point at which it has.
+	addr := net.JoinHostPort(startAddr, strconv.Itoa(gatePort))
+	require.Eventually(t, func() bool {
+		conn, err := net.DialTimeout("tcp", addr, time.Second)
+		if err != nil {
+			return false
+		}
+		conn.Close()
+		return true
+	}, 60*time.Second, 100*time.Millisecond, "gate never accepted a connection on %s", addr)
+
+	assert.FileExists(t, filepath.Join(basePath, serviceName+".pid"))
+	require.NoError(t, svc.Shutdown())
+
+	select {
+	case res := <-done:
+		require.NoError(t, res.err)
+		assert.Equal(t, os.Getpid(), res.pid)
+	case <-time.After(60 * time.Second):
+		t.Fatal("Start did not return after Shutdown")
+	}
+}

spinifex/services/predastore/predastore_test.go

@@ -1,118 +1,146 @@
 package predastore
 
 import (
+	"context"
+	"fmt"
+	"os"
+	"os/exec"
+	"path/filepath"
+	"strings"
 	"testing"
 
+	pds "github.com/mulgadc/predastore"
+	"github.com/mulgadc/spinifex/spinifex/utils"
 	"github.com/stretchr/testify/assert"
+	"github.com/stretchr/testify/require"
 )
 
+// hostSpec and nodeSpec are the [[host]] and [[host.node]] tables of a
+// configuration file under test. An empty field is left out of the file, which
+// is how a test leaves a host-local value for the service flags to supply.
+type hostSpec struct {
+	id       int
+	bindAddr string
+	addr     string
+	dataDir  string
+	tlsCert  string
+	tlsKey   string
+}
+
+type nodeSpec struct {
+	id   int
+	role string
+	port int
+}
+
+// writeTopology writes a predastore configuration file for host and its nodes
+// and returns its path. RS(1,0) is the narrowest code the file format allows,
+// so one blob node satisfies the stripe-width check.
+func writeTopology(t *testing.T, dir string, host hostSpec, nodes []nodeSpec) string {
+	t.Helper()
+
+	var b strings.Builder
+	fmt.Fprint(&b, "version = 1\nregion = \"us-east-1\"\n\n[rs]\ndata = 1\nparity = 0\n\n")
+	fmt.Fprintf(&b, "[[host]]\nid = %d\n", host.id)
+	for _, field := range [][2]string{
+		{"bind_addr", host.bindAddr},
+		{"addr", host.addr},
+		{"data_dir", host.dataDir},
+		{"tls_cert", host.tlsCert},
+		{"tls_key", host.tlsKey},
+	} {
+		if field[1] != "" {
+			fmt.Fprintf(&b, "%s = %q\n", field[0], field[1])
+		}
+	}
+	for _, n := range nodes {
+		fmt.Fprintf(&b, "\n[[host.node]]\nid = %d\nrole = %q\nport = %d\n", n.id, n.role, n.port)
+	}
+
+	path := filepath.Join(dir, "predastore.toml")
+	require.NoError(t, os.WriteFile(path, []byte(b.String()), 0600))
+	return path
+}
+
+// singleHost is the topology most of these tests merge against: a gate and the
+// one blob node RS(1,0) needs, on a host that names every host-local field so
+// a flag has something to beat.
+func singleHost(t *testing.T, dir string) string {
+	t.Helper()
+	return writeTopology(t, dir,
+		hostSpec{
+			id:       1,
+			bindAddr: "10.0.0.1",
+			addr:     "10.0.0.1",
+			dataDir:  filepath.Join(dir, "data"),
+			tlsCert:  filepath.Join(dir, "file.crt"),
+			tlsKey:   filepath.Join(dir, "file.key"),
+		},
+		[]nodeSpec{
+			{id: 1, role: "gate", port: 8443},
+			{id: 2, role: "blob", port: 6660},
+		})
+}
+
+// mergeHost loads the configuration at path and merges cfg's host-local fields
+// into it, which is the pairing Start performs.
+func mergeHost(t *testing.T, cfg *Config) (*pds.Config, error) {
+	t.Helper()
+
+	svc, err := New(cfg)
+	require.NoError(t, err)
+
+	parsed, err := pds.LoadConfig(cfg.ConfigPath)
+	require.NoError(t, err)
+	_, err = svc.mergeHost(parsed)
+	return parsed, err
+}
+
+// gateOf is the gate node of the first host, which is where the merged S3 port
+// lands.
+func gateOf(t *testing.T, cfg *pds.Config) pds.NodeConfig {
+	t.Helper()
+
+	for _, n := range cfg.Hosts[0].Nodes {
+		if n.Role == pds.RoleGate {
+			return n
+		}
+	}
+	t.Fatal("configuration declares no gate node")
+	return pds.NodeConfig{}
+}
+
 // TestNew tests the service constructor.
 func TestNew(t *testing.T) {
 	cfg := &Config{
 		ConfigPath:        "/tmp/test-config.toml",
 		Port:              8443,
 		Host:              "0.0.0.0",
-		Debug:             false,
 		BasePath:          "/tmp/predastore",
 		TlsCert:           "/tmp/cert.pem",
 		TlsKey:            "/tmp/key.pem",
 		EncryptionKeyFile: "/tmp/encryption.key",
+		HostID:            2,
 	}
 
 	svc, err := New(cfg)
 
-	assert.NoError(t, err)
-	assert.NotNil(t, svc)
-	assert.NotNil(t, svc.Config)
-	assert.Equal(t, "/tmp/test-config.toml", svc.Config.ConfigPath)
-	assert.Equal(t, 8443, svc.Config.Port)
-	assert.Equal(t, "0.0.0.0", svc.Config.Host)
-	assert.False(t, svc.Config.Debug)
-	assert.Equal(t, "/tmp/predastore", svc.Config.BasePath)
-	assert.Equal(t, "/tmp/cert.pem", svc.Config.TlsCert)
-	assert.Equal(t, "/tmp/key.pem", svc.Config.TlsKey)
-	assert.Equal(t, "/tmp/encryption.key", svc.Config.EncryptionKeyFile)
+	require.NoError(t, err)
+	require.NotNil(t, svc)
+	// The same pointer, not a copy: the start command mutates the config it
+	// hands over right up to the call.
+	assert.Same(t, cfg, svc.Config)
 }
 
-// TestNewWithNilConfig tests that New handles nil config correctly.
-func TestNewWithNilConfig(t *testing.T) {
-	cfg := &Config{}
-	svc, err := New(cfg)
+// TestNewRejectsForeignConfig pins the type assertion in New: the service
+// registry hands it an any, so a config meant for another service must be
+// rejected rather than silently ignored.
+func TestNewRejectsForeignConfig(t *testing.T) {
+	svc, err := New(struct{ Port int }{Port: 8443})
 
-	assert.NoError(t, err)
-	assert.NotNil(t, svc)
-	assert.NotNil(t, svc.Config)
-}
-
-// TestConfigDefaults tests Config struct default values.
-func TestConfigDefaults(t *testing.T) {
-	cfg := &Config{}
-
-	assert.Empty(t, cfg.ConfigPath)
-	assert.Empty(t, cfg.Host)
-	assert.Equal(t, 0, cfg.Port)
-	assert.False(t, cfg.Debug)
-	assert.Empty(t, cfg.BasePath)
-	assert.Empty(t, cfg.TlsCert)
-	assert.Empty(t, cfg.TlsKey)
-	assert.Empty(t, cfg.EncryptionKeyFile)
-}
-
-// TestConfigWithDebug tests Config with debug enabled.
-func TestConfigWithDebug(t *testing.T) {
-	cfg := &Config{
-		Debug:      true,
-		ConfigPath: "/etc/predastore/config.toml",
-		Port:       8443,
-	}
-
-	assert.True(t, cfg.Debug)
-	assert.Equal(t, "/etc/predastore/config.toml", cfg.ConfigPath)
-	assert.Equal(t, 8443, cfg.Port)
-}
-
-// TestConfigFullyPopulated tests a fully populated config.
-func TestConfigFullyPopulated(t *testing.T) {
-	cfg := &Config{
-		ConfigPath:        "/etc/predastore/config.toml",
-		Port:              9443,
-		Host:              "127.0.0.1",
-		Debug:             true,
-		BasePath:          "/var/lib/predastore",
-		TlsCert:           "/etc/ssl/certs/predastore.crt",
-		TlsKey:            "/etc/ssl/private/predastore.key",
-		EncryptionKeyFile: "/etc/spinifex/predastore/encryption.key",
-	}
-
-	assert.Equal(t, "/etc/predastore/config.toml", cfg.ConfigPath)
-	assert.Equal(t, 9443, cfg.Port)
-	assert.Equal(t, "127.0.0.1", cfg.Host)
-	assert.True(t, cfg.Debug)
-	assert.Equal(t, "/var/lib/predastore", cfg.BasePath)
-	assert.Equal(t, "/etc/ssl/certs/predastore.crt", cfg.TlsCert)
-	assert.Equal(t, "/etc/ssl/private/predastore.key", cfg.TlsKey)
-	assert.Equal(t, "/etc/spinifex/predastore/encryption.key", cfg.EncryptionKeyFile)
-}
-
-// TestServiceMethods tests the service interface methods.
-func TestServiceMethods(t *testing.T) {
-	cfg := &Config{
-		ConfigPath: "/tmp/test-config.toml",
-		Port:       8443,
-		Host:       "127.0.0.1",
-	}
-
-	svc, err := New(cfg)
-	assert.NoError(t, err)
-
-	// Test Status - returns "stopped" when no PID file exists
-	status, err := svc.Status()
-	assert.NoError(t, err)
-	assert.Equal(t, "stopped", status)
-
-	// Test Reload - returns no error
-	err = svc.Reload()
-	assert.NoError(t, err)
+	require.Error(t, err)
+	assert.Nil(t, svc)
+	assert.Contains(t, err.Error(), "invalid config type")
 }
 
 // TestServiceNameConstant tests the serviceName constant.
@@ -120,221 +148,253 @@ func TestServiceNameConstant(t *testing.T) {
 	assert.Equal(t, "predastore", serviceName)
 }
 
-// TestConfigPortRange tests port validation.
-func TestConfigPortRange(t *testing.T) {
-	testCases := []struct {
-		name     string
-		port     int
-		valid    bool
-		expected int
-	}{
-		{"Standard HTTPS", 443, true, 443},
-		{"Custom Port", 8443, true, 8443},
-		{"High Port", 65535, true, 65535},
-		{"Port 80", 80, true, 80},
-		{"Port 8080", 8080, true, 8080},
-	}
-
-	for _, tc := range testCases {
-		t.Run(tc.name, func(t *testing.T) {
-			cfg := &Config{
-				Port: tc.port,
-			}
-
-			assert.Equal(t, tc.expected, cfg.Port)
-			if tc.valid {
-				assert.Positive(t, cfg.Port)
-				assert.LessOrEqual(t, cfg.Port, 65535)
-			}
-		})
-	}
+// TestReloadIsANoOp pins the current contract: predastore has no reload, and
+// the service reports success rather than an error a caller would have to
+// special-case.
+func TestReloadIsANoOp(t *testing.T) {
+	svc, err := New(&Config{})
+	require.NoError(t, err)
+	assert.NoError(t, svc.Reload())
 }
 
-// TestConfigHostnames tests host configuration.
-func TestConfigHostnames(t *testing.T) {
-	testCases := []struct {
-		name     string
-		host     string
-		expected string
-	}{
-		{"Localhost", "127.0.0.1", "127.0.0.1"},
-		{"All Interfaces", "0.0.0.0", "0.0.0.0"},
-		{"Localhost Name", "localhost", "localhost"},
-		{"Custom Domain", "s3.example.com", "s3.example.com"},
-	}
-
-	for _, tc := range testCases {
-		t.Run(tc.name, func(t *testing.T) {
-			cfg := &Config{
-				Host: tc.host,
-			}
-
-			assert.Equal(t, tc.expected, cfg.Host)
-			assert.NotEmpty(t, cfg.Host)
-		})
-	}
-}
-
-// TestConfigBasePath tests base path configuration.
-func TestConfigBasePath(t *testing.T) {
-	testCases := []struct {
-		name     string
-		basePath string
-	}{
-		{"Absolute Path", "/var/lib/predastore"},
-		{"Relative Path", "data/predastore"},
-		{"Temp Directory", "/tmp/predastore"},
-		{"User Directory", "/home/user/predastore"},
-	}
-
-	for _, tc := range testCases {
-		t.Run(tc.name, func(t *testing.T) {
-			cfg := &Config{
-				BasePath: tc.basePath,
-			}
-
-			assert.Equal(t, tc.basePath, cfg.BasePath)
-		})
-	}
-}
-
-// TestConfigTLSPaths tests TLS certificate and key paths.
-func TestConfigTLSPaths(t *testing.T) {
+// TestMergeHostFlagsBeatFile covers the precedence spinifex.toml relies on:
+// the service flags own the host-local fields, so a value from them replaces
+// whatever the predastore file carried.
+func TestMergeHostFlagsBeatFile(t *testing.T) {
+	dir := t.TempDir()
 	cfg := &Config{
-		TlsCert: "/etc/ssl/certs/server.crt",
-		TlsKey:  "/etc/ssl/private/server.key",
+		ConfigPath: singleHost(t, dir),
+		Host:       "0.0.0.0",
+		Port:       19443,
+		TlsCert:    filepath.Join(dir, "flag.crt"),
+		TlsKey:     filepath.Join(dir, "flag.key"),
+		HostID:     1,
 	}
 
-	assert.NotEmpty(t, cfg.TlsCert)
-	assert.NotEmpty(t, cfg.TlsKey)
-	assert.Contains(t, cfg.TlsCert, ".crt")
-	assert.Contains(t, cfg.TlsKey, ".key")
+	merged, err := mergeHost(t, cfg)
+
+	require.NoError(t, err)
+	assert.Equal(t, cfg.TlsCert, merged.Hosts[0].TLSCert)
+	assert.Equal(t, cfg.TlsKey, merged.Hosts[0].TLSKey)
+	assert.Equal(t, 19443, gateOf(t, merged).Port)
+	// The dial address is the cluster's business, not this host's flags.
+	assert.Equal(t, "10.0.0.1", merged.Hosts[0].Addr)
 }
 
-// TestConfigMinimal tests minimal required config.
-func TestConfigMinimal(t *testing.T) {
-	cfg := &Config{
-		ConfigPath: "/tmp/config.toml",
-		Port:       8443,
-		Host:       "127.0.0.1",
-	}
+// TestMergeHostBindsTheGateNotTheCluster is the plane split: the address
+// spinifex.toml carries is the public S3 endpoint, so settling it on the host
+// would put raft and blob traffic on the public interface along with it.
+func TestMergeHostBindsTheGateNotTheCluster(t *testing.T) {
+	dir := t.TempDir()
 
-	svc, err := New(cfg)
-	assert.NoError(t, err)
-	assert.NotNil(t, svc)
+	merged, err := mergeHost(t, &Config{
+		ConfigPath: singleHost(t, dir),
+		Host:       "0.0.0.0",
+		HostID:     1,
+	})
 
-	// Minimal config should work
-	assert.NotEmpty(t, cfg.ConfigPath)
-	assert.Positive(t, cfg.Port)
-	assert.NotEmpty(t, cfg.Host)
+	require.NoError(t, err)
+	assert.Equal(t, "0.0.0.0", gateOf(t, merged).BindAddr, "S3 answers on every interface")
+	assert.Equal(t, "10.0.0.1", merged.Hosts[0].BindAddr, "raft and blob stay on the host address")
 }
 
-// TestServiceStartWithoutConfig tests Start method behavior
-// Note: This will fail without proper config file, which is expected.
-func TestServiceStartWithoutConfig(t *testing.T) {
-	// Skip this test - it requires actual config file and will block/exit
-	// This is covered by integration tests instead
-	t.Skip("Skipping test that requires actual predastore config file - covered by integration tests")
+// TestMergeHostWithoutAnS3AddressLeavesBothPlanesOnTheHost covers the
+// single-homed case: nothing to split, so the gate follows the host and the
+// file alone decides.
+func TestMergeHostWithoutAnS3AddressLeavesBothPlanesOnTheHost(t *testing.T) {
+	dir := t.TempDir()
+
+	merged, err := mergeHost(t, &Config{ConfigPath: singleHost(t, dir), HostID: 1})
+
+	require.NoError(t, err)
+	assert.Empty(t, gateOf(t, merged).BindAddr, "the gate names no address of its own")
+	assert.Equal(t, "10.0.0.1", merged.Hosts[0].BindAddr)
 }
 
-// TestServiceStartRejectsMissingEncryptionKey ensures Start fails fast when
-// EncryptionKeyFile is unset, before touching the pid file or binding.
-func TestServiceStartRejectsMissingEncryptionKey(t *testing.T) {
-	cfg := &Config{
-		ConfigPath: "/tmp/test-config.toml",
-		Port:       18443,
-		Host:       "127.0.0.1",
+// TestMergeHostReturnsTheResolvedHostID pins what Start hands predastore.Run:
+// the host whose nodes this process serves, resolved once rather than
+// reconverted at the call.
+func TestMergeHostReturnsTheResolvedHostID(t *testing.T) {
+	dir := t.TempDir()
+	svc, err := New(&Config{ConfigPath: singleHost(t, dir), HostID: 1})
+	require.NoError(t, err)
+	parsed, err := pds.LoadConfig(svc.Config.ConfigPath)
+	require.NoError(t, err)
+
+	hostID, err := svc.mergeHost(parsed)
+
+	require.NoError(t, err)
+	assert.Equal(t, pds.HostID(1), hostID)
+}
+
+// TestMergeHostKeepsFileValues is the other half of the precedence rule: an
+// unset flag must leave the file's value alone rather than blanking it.
+func TestMergeHostKeepsFileValues(t *testing.T) {
+	dir := t.TempDir()
+	cfg := &Config{ConfigPath: singleHost(t, dir), HostID: 1}
+
+	merged, err := mergeHost(t, cfg)
+
+	require.NoError(t, err)
+	assert.Equal(t, "10.0.0.1", merged.Hosts[0].BindAddr)
+	assert.Equal(t, filepath.Join(dir, "file.crt"), merged.Hosts[0].TLSCert)
+	assert.Equal(t, filepath.Join(dir, "file.key"), merged.Hosts[0].TLSKey)
+	assert.Equal(t, 8443, gateOf(t, merged).Port)
+}
+
+// TestMergeHostLeavesTheClusterPlaneToTheFile covers the one host-local field
+// no service flag owns. There is no flag for the cluster plane, so a file that
+// names only the dial address is left alone here and resolved by predastore,
+// which is the single place that decides what an empty bind address means.
+func TestMergeHostLeavesTheClusterPlaneToTheFile(t *testing.T) {
+	dir := t.TempDir()
+	path := writeTopology(t, dir,
+		hostSpec{id: 1, addr: "10.0.0.1", dataDir: filepath.Join(dir, "data")},
+		[]nodeSpec{{id: 1, role: "gate", port: 8443}, {id: 2, role: "blob", port: 6660}})
+
+	merged, err := mergeHost(t, &Config{ConfigPath: path, Host: "0.0.0.0", HostID: 1})
+
+	require.NoError(t, err)
+	assert.Empty(t, merged.Hosts[0].BindAddr, "the S3 address must not settle on the host")
+	assert.Equal(t, "10.0.0.1", pds.HostBindAddr(merged.Hosts[0]), "predastore resolves it to the dial address")
+}
+
+// TestMergeHostRejectsMissingHostID covers the one field with no default: the
+// host id selects which [[host]] this process runs, and zero names none.
+func TestMergeHostRejectsMissingHostID(t *testing.T) {
+	dir := t.TempDir()
+
+	_, err := mergeHost(t, &Config{ConfigPath: singleHost(t, dir)})
+
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "host id is required")
+}
+
+// TestMergeHostRejectsUnknownHostID covers a host id the configuration file
+// never declares: nothing pins nodes to it, so the process would serve an
+// empty cluster.
+func TestMergeHostRejectsUnknownHostID(t *testing.T) {
+	dir := t.TempDir()
+
+	_, err := mergeHost(t, &Config{ConfigPath: singleHost(t, dir), HostID: 9})
+
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "predastore host 9 is not in")
+}
+
+// TestMergeHostRejectsHostWithoutGate covers a host running no gate: the
+// endpoint spinifex.toml advertises for this node would answer no S3 request,
+// and the port the flags carry would have nowhere to land.
+func TestMergeHostRejectsHostWithoutGate(t *testing.T) {
+	dir := t.TempDir()
+	path := writeTopology(t, dir,
+		hostSpec{id: 1, addr: "10.0.0.1", dataDir: filepath.Join(dir, "data")},
+		[]nodeSpec{{id: 1, role: "blob", port: 6660}, {id: 2, role: "meta", port: 7660}})
+
+	_, err := mergeHost(t, &Config{ConfigPath: path, HostID: 1, Port: 18443})
+
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "runs no gate node")
+}
+
+// TestMergeHostRevalidatesMergedConfig is why the merge ends in Validate: a
+// port that only arrives by flag can still collide with a sibling node's, and
+// the file alone was coherent.
+func TestMergeHostRevalidatesMergedConfig(t *testing.T) {
+	dir := t.TempDir()
+	cfg := &Config{ConfigPath: singleHost(t, dir), HostID: 1, Port: 6660}
+
+	_, err := mergeHost(t, cfg)
+
+	require.Error(t, err)
+	assert.Contains(t, err.Error(), "both use port 6660")
+}
+
+// TestStartRejectsMissingEncryptionKey ensures Start fails fast when
+// EncryptionKeyFile is unset, before touching the pid file or the config.
+func TestStartRejectsMissingEncryptionKey(t *testing.T) {
+	svc, err := New(&Config{
+		ConfigPath: filepath.Join(t.TempDir(), "predastore.toml"),
 		BasePath:   t.TempDir(),
-		TlsCert:    "/tmp/cert.pem",
-		TlsKey:     "/tmp/key.pem",
-		// EncryptionKeyFile deliberately left empty.
-	}
-	svc, err := New(cfg)
-	assert.NoError(t, err)
+		HostID:     1,
+	})
+	require.NoError(t, err)
 
-	_, err = svc.Start()
-	assert.Error(t, err)
+	pid, err := svc.Start()
+
+	require.Error(t, err)
+	assert.Zero(t, pid)
 	assert.Contains(t, err.Error(), "encryption key file is required")
 }
 
-// TestConfigDebugFlag tests debug flag behavior.
-func TestConfigDebugFlag(t *testing.T) {
-	// Debug enabled
-	cfg1 := &Config{
-		Debug: true,
-	}
-	assert.True(t, cfg1.Debug)
+// TestStatusReportsStoppedWithoutPidFile covers the resting state: no pid file
+// under BasePath means nothing is running, not an error.
+func TestStatusReportsStoppedWithoutPidFile(t *testing.T) {
+	svc, err := New(&Config{BasePath: t.TempDir()})
+	require.NoError(t, err)
 
-	// Debug disabled
-	cfg2 := &Config{
-		Debug: false,
-	}
-	assert.False(t, cfg2.Debug)
+	status, err := svc.Status()
 
-	// Default should be false
-	cfg3 := &Config{}
-	assert.False(t, cfg3.Debug)
+	require.NoError(t, err)
+	assert.Equal(t, "stopped", status)
 }
 
-// TestConfigValidation tests validation of required fields.
-func TestConfigValidation(t *testing.T) {
-	// Empty config - fields should be empty/zero
-	cfg := &Config{}
-	assert.Empty(t, cfg.ConfigPath)
-	assert.Equal(t, 0, cfg.Port)
-	assert.Empty(t, cfg.Host)
+// TestStatusAndStopUseThePidFile covers the out-of-process path: Stop and
+// Status find a predastore started by an earlier invocation through the pid
+// file BasePath holds, and Stop clears it once the process is gone.
+func TestStatusAndStopUseThePidFile(t *testing.T) {
+	basePath := t.TempDir()
+	svc, err := New(&Config{BasePath: basePath})
+	require.NoError(t, err)
 
-	// Valid config
-	cfg2 := &Config{
-		ConfigPath: "/etc/predastore/config.toml",
-		Port:       8443,
-		Host:       "0.0.0.0",
-	}
-	assert.NotEmpty(t, cfg2.ConfigPath)
-	assert.Positive(t, cfg2.Port)
-	assert.NotEmpty(t, cfg2.Host)
+	child := exec.CommandContext(t.Context(), "sleep", "60")
+	require.NoError(t, child.Start())
+	// Reaped as it dies, not at cleanup: an unwaited child of this process
+	// lingers as a zombie, which still answers signal 0, so Stop would wait out
+	// its whole grace period before escalating to SIGKILL.
+	reaped := make(chan struct{})
+	go func() {
+		defer close(reaped)
+		_ = child.Wait()
+	}()
+	t.Cleanup(func() { <-reaped })
+	require.NoError(t, utils.WritePidFileTo(basePath, serviceName, child.Process.Pid))
+
+	status, err := svc.Status()
+	require.NoError(t, err)
+	assert.Equal(t, fmt.Sprintf("running (pid: %d)", child.Process.Pid), status)
+
+	require.NoError(t, svc.Stop())
+	assert.NoFileExists(t, filepath.Join(basePath, serviceName+".pid"))
+
+	status, err = svc.Status()
+	require.NoError(t, err)
+	assert.Equal(t, "stopped", status)
 }
 
-// TestMultipleServiceInstances tests creating multiple service instances.
-func TestMultipleServiceInstances(t *testing.T) {
-	cfg1 := &Config{
-		ConfigPath: "/tmp/config1.toml",
-		Port:       8443,
-		Host:       "127.0.0.1",
-	}
+// TestShutdownWithoutRunningClusterFallsBackToPidFile covers the fallback: with
+// no cluster in this process there is no context to cancel, so Shutdown goes
+// through the pid file, which reports rather than signalling anything.
+func TestShutdownWithoutRunningClusterFallsBackToPidFile(t *testing.T) {
+	svc, err := New(&Config{BasePath: t.TempDir()})
+	require.NoError(t, err)
 
-	cfg2 := &Config{
-		ConfigPath: "/tmp/config2.toml",
-		Port:       9443,
-		Host:       "127.0.0.1",
-	}
-
-	svc1, err := New(cfg1)
-	assert.NoError(t, err)
-	assert.NotNil(t, svc1)
-
-	svc2, err := New(cfg2)
-	assert.NoError(t, err)
-	assert.NotNil(t, svc2)
-
-	// Services should be independent
-	assert.NotEqual(t, svc1.Config.Port, svc2.Config.Port)
-	assert.NotEqual(t, svc1.Config.ConfigPath, svc2.Config.ConfigPath)
+	assert.Error(t, svc.Shutdown())
 }
 
-// TestConfigPointerPreservation tests that config pointer is preserved.
-func TestConfigPointerPreservation(t *testing.T) {
-	cfg := &Config{
-		Port: 8443,
-		Host: "127.0.0.1",
-	}
+// TestShutdownCancelsTheServiceContext covers the in-process path: once Start
+// has published its cancel func, Shutdown cancels the context the gate and the
+// local nodes share instead of signalling through the pid file.
+func TestShutdownCancelsTheServiceContext(t *testing.T) {
+	svc, err := New(&Config{BasePath: t.TempDir()})
+	require.NoError(t, err)
 
-	svc, err := New(cfg)
-	assert.NoError(t, err)
+	ctx, cancel := context.WithCancel(context.Background())
+	defer cancel()
+	svc.mu.Lock()
+	svc.stop = cancel
+	svc.mu.Unlock()
 
-	// Modify original config
-	cfg.Port = 9443
-
-	// Service should reflect the change (same pointer)
-	assert.Equal(t, 9443, svc.Config.Port)
+	require.NoError(t, svc.Shutdown())
+	assert.Error(t, ctx.Err())
 }

spinifex/services/spinifexui/frontend/src/queries/admin.ts

@@ -80,7 +80,7 @@ interface GetVMsOutput {
   vms: VMInfo[]
 }
 
-interface DBNodeStatus {
+interface MetaNodeStatus {
   id: number
   host: string
   port: number
@@ -94,7 +94,7 @@ interface DBNodeStatus {
   is_leader: boolean
 }
 
-interface ShardNode {
+interface BlobNode {
   id: number
   host: string
   port: number
@@ -102,7 +102,6 @@ interface ShardNode {
 
 interface StorageBucket {
   name: string
-  type: string
   region: string
 }
 
@@ -112,8 +111,8 @@ interface StorageStatusOutput {
     data_shards: number
     parity_shards: number
   }
-  db_nodes: DBNodeStatus[]
-  shard_nodes: ShardNode[]
+  meta_nodes: MetaNodeStatus[]
+  blob_nodes: BlobNode[]
   buckets: StorageBucket[]
 }
 
@@ -126,8 +125,8 @@ export type {
   GetNodesOutput,
   VMInfo,
   GetVMsOutput,
-  DBNodeStatus,
-  ShardNode,
+  MetaNodeStatus,
+  BlobNode,
   StorageBucket,
   StorageStatusOutput,
 }

spinifex/services/spinifexui/frontend/src/routes/_auth/s3/service-metrics.tsx

@@ -7,8 +7,8 @@ import { Card, CardContent, CardHeader, CardTitle } from "@/components/ui/card"
 import { useAdmin } from "@/contexts/admin-context"
 import {
   adminStorageStatusQueryOptions,
-  type DBNodeStatus,
-  type ShardNode,
+  type BlobNode,
+  type MetaNodeStatus,
 } from "@/queries/admin"
 
 export const Route = createFileRoute("/_auth/s3/service-metrics")({
@@ -31,11 +31,11 @@ function ServiceMetricsPage() {
   }
 
   const encoding = data?.encoding
-  const dbNodes = data?.db_nodes ?? []
-  const shardNodes = data?.shard_nodes ?? []
+  const metaNodes = data?.meta_nodes ?? []
+  const blobNodes = data?.blob_nodes ?? []
 
-  const allHealthy = dbNodes.length > 0 && dbNodes.every((n) => n.healthy)
-  const anyUnhealthy = dbNodes.some((n) => !n.healthy)
+  const allHealthy = metaNodes.length > 0 && metaNodes.every((n) => n.healthy)
+  const anyUnhealthy = metaNodes.some((n) => !n.healthy)
 
   return (
     <>
@@ -62,22 +62,22 @@ function ServiceMetricsPage() {
                 </div>
               </div>
               <div>
-                <span className="text-muted-foreground">DB Nodes</span>
+                <span className="text-muted-foreground">Meta Nodes</span>
                 <div className="mt-1 font-mono font-medium">
-                  {dbNodes.length}
+                  {metaNodes.length}
                 </div>
               </div>
               <div>
-                <span className="text-muted-foreground">Shard Nodes</span>
+                <span className="text-muted-foreground">Blob Nodes</span>
                 <div className="mt-1 font-mono font-medium">
-                  {shardNodes.length}
+                  {blobNodes.length}
                 </div>
               </div>
               <div>
                 <span className="text-muted-foreground">Cluster Health</span>
                 <div className="mt-1">
                   <ClusterHealthBadge
-                    nodeCount={dbNodes.length}
+                    nodeCount={metaNodes.length}
                     allHealthy={allHealthy}
                     anyUnhealthy={anyUnhealthy}
                   />
@@ -97,32 +97,32 @@ function ServiceMetricsPage() {
           </CardContent>
         </Card>
 
-        {/* Section 2: DB Consensus Nodes */}
+        {/* Section 2: Meta Consensus Nodes */}
         <Card>
           <CardHeader>
-            <CardTitle className="text-sm">DB Consensus Nodes (Raft)</CardTitle>
+            <CardTitle className="text-sm">
+              Meta Consensus Nodes (Raft)
+            </CardTitle>
           </CardHeader>
           <CardContent>
-            {dbNodes.length === 0 ? (
+            {metaNodes.length === 0 ? (
               <p className="text-xs text-muted-foreground">No data</p>
             ) : (
-              <DBNodesTable nodes={dbNodes} />
+              <MetaNodesTable nodes={metaNodes} />
             )}
           </CardContent>
         </Card>
 
-        {/* Section 3: Shard Nodes */}
+        {/* Section 3: Blob Nodes */}
         <Card>
           <CardHeader>
-            <CardTitle className="text-sm">
-              Object Storage Shard Nodes
-            </CardTitle>
+            <CardTitle className="text-sm">Object Storage Blob Nodes</CardTitle>
           </CardHeader>
           <CardContent>
-            {shardNodes.length === 0 ? (
+            {blobNodes.length === 0 ? (
               <p className="text-xs text-muted-foreground">No data</p>
             ) : (
-              <ShardNodesTable nodes={shardNodes} />
+              <BlobNodesTable nodes={blobNodes} />
             )}
           </CardContent>
         </Card>
@@ -152,7 +152,7 @@ function ClusterHealthBadge({
   return <Badge variant="secondary">Unknown</Badge>
 }
 
-function DBNodesTable({ nodes }: { nodes: DBNodeStatus[] }) {
+function MetaNodesTable({ nodes }: { nodes: MetaNodeStatus[] }) {
   return (
     <div className="overflow-x-auto">
       <table className="w-full text-xs">
@@ -209,7 +209,7 @@ function DBNodesTable({ nodes }: { nodes: DBNodeStatus[] }) {
   )
 }
 
-function ShardNodesTable({ nodes }: { nodes: ShardNode[] }) {
+function BlobNodesTable({ nodes }: { nodes: BlobNode[] }) {
   return (
     <div className="overflow-x-auto">
       <table className="w-full text-xs">

spinifex/types/node.go

@@ -62,8 +62,11 @@ type NodeStatusResponse struct {
 	InstanceTypes  []InstanceTypeCap `json:"instance_types"`
 
 	// Leader roles for clustered services (empty string = service not running or not clustered)
-	NATSRole       string `json:"nats_role,omitempty"`       // "leader" or "follower"
-	PredastoreRole string `json:"predastore_role,omitempty"` // "leader" or "follower"
+	NATSRole string `json:"nats_role,omitempty"` // "leader" or "follower"
+
+	// Never populated: predastore's meta nodes expose no status surface. Kept so
+	// the JSON contract and the UI are unchanged when the probe is restored.
+	PredastoreRole string `json:"predastore_role,omitempty"`
 
 	// OVN DB Raft roles ("leader"/"follower"/""); empty when the node is not an
 	// OVN DB cluster member or OVN is standalone.

spinifex/types/storage.go

@@ -3,10 +3,10 @@ package types
 // StorageConfigResponse is returned by the spinifex.storage.config NATS topic.
 // Contains predastore topology with credentials stripped.
 type StorageConfigResponse struct {
-	Encoding   StorageEncoding    `json:"encoding"`
-	DBNodes    []StorageDBNode    `json:"db_nodes"`
-	ShardNodes []StorageShardNode `json:"shard_nodes"`
-	Buckets    []StorageBucket    `json:"buckets"`
+	Encoding  StorageEncoding   `json:"encoding"`
+	MetaNodes []StorageMetaNode `json:"meta_nodes"`
+	BlobNodes []StorageBlobNode `json:"blob_nodes"`
+	Buckets   []StorageBucket   `json:"buckets"`
 }
 
 // StorageEncoding describes the Reed-Solomon erasure coding configuration.
@@ -15,15 +15,17 @@ type StorageEncoding struct {
 	ParityShards int `json:"parity_shards"`
 }
 
-// StorageDBNode describes a predastore distributed database node (no credentials).
-type StorageDBNode struct {
+// StorageMetaNode describes a predastore meta node, one member of the Raft
+// quorum over global state (no credentials).
+type StorageMetaNode struct {
 	ID   int    `json:"id"`
 	Host string `json:"host"`
 	Port int    `json:"port"`
 }
 
-// StorageShardNode describes a predastore object storage shard node.
-type StorageShardNode struct {
+// StorageBlobNode describes a predastore blob node, which holds erasure-coded
+// object shards.
+type StorageBlobNode struct {
 	ID   int    `json:"id"`
 	Host string `json:"host"`
 	Port int    `json:"port"`
@@ -32,6 +34,5 @@ type StorageShardNode struct {
 // StorageBucket describes a configured S3 bucket.
 type StorageBucket struct {
 	Name   string `json:"name"`
-	Type   string `json:"type"`
 	Region string `json:"region"`
 }

tests/fixtures/predastore/predastore.go

@@ -1,10 +1,10 @@
-// Package predastore starts a real predastore daemon for tests that need to
+// Package predastore starts a real predastore cluster for tests that need to
 // exercise an actual S3-compatible backend rather than a mock. It is
 // deliberately its own leaf package (not folded into spinifex/testutil,
 // which is imported by most of the module's test files for lightweight NATS
-// helpers): starting a daemon here builds a whole predastore cluster runtime
-// — shard stores, Raft replicas and their transports — whose goroutines run
-// for the life of the test binary. Folding this file into the shared testutil
+// helpers): starting one here runs a whole predastore host — blob nodes, Raft
+// meta replicas and the S3 gate in front of them — whose goroutines run for
+// the life of the test binary. Folding this file into the shared testutil
 // package would pull all of that into every test binary that imports
 // testutil, and trip up any unrelated goroutine-leak check (go.uber.org/goleak)
 // running in that same binary. Only callers that actually need a real
@@ -20,6 +20,7 @@ import (
 	"crypto/x509"
 	"crypto/x509/pkix"
 	"encoding/pem"
+	"fmt"
 	"log/slog"
 	"math/big"
 	"net"
@@ -27,6 +28,7 @@ import (
 	"os"
 	"path/filepath"
 	"strconv"
+	"strings"
 	"sync"
 	"testing"
 	"time"
@@ -35,23 +37,22 @@ import (
 	"github.com/aws/aws-sdk-go/aws/credentials"
 	"github.com/aws/aws-sdk-go/aws/session"
 	"github.com/aws/aws-sdk-go/service/s3"
-	"github.com/mulgadc/predastore/clusterrun"
+	pds "github.com/mulgadc/predastore"
 	"github.com/mulgadc/predastore/pkg/masterkey"
-	predastoreserver "github.com/mulgadc/predastore/s3"
 	"github.com/mulgadc/spinifex/tests/fixtures/scratch"
 )
 
-// Fixed connection details for the shared predastore fixture daemon started
-// by Start. Every cluster node runs in-process, so these values never need to
+// Fixed connection details for the shared predastore fixture started by
+// Start. Every cluster node runs in-process, so these values never need to
 // vary per caller or per test run.
 const (
 	Host   = "127.0.0.1:18443"
 	Region = "us-east-1"
 	// AccessKey/SecretKey are the well-known AWS SDK example credentials
 	// (docs.aws.amazon.com/IAM/latest/UserGuide), used only to authenticate
-	// against this ephemeral, localhost-only test daemon — not a real secret.
-	AccessKey = "AKIAIOSFODNN7EXAMPLE"                     //nolint:gosec // well-known AWS SDK example key, test-only daemon
-	SecretKey = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" //nolint:gosec // well-known AWS SDK example secret, test-only daemon
+	// against this ephemeral, localhost-only test cluster — not a real secret.
+	AccessKey = "AKIAIOSFODNN7EXAMPLE"                     //nolint:gosec // well-known AWS SDK example key, test-only cluster
+	SecretKey = "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY" //nolint:gosec // well-known AWS SDK example secret, test-only cluster
 
 	// DefaultBucket and DefaultBucket2 are pre-created by Start itself,
 	// matching the bucket set services/predastore's own integration tests
@@ -67,7 +68,31 @@ const (
 	testPort = 18443
 )
 
-// Fixture describes a running predastore daemon ready for real
+// The topology Start writes. One host runs every node, so they talk over the
+// in-process pipe and only the gate binds a socket. RS(3,2) spreads a stripe
+// over 5 distinct blob nodes, which is exactly how many there are — so every
+// PutObject lands one shard on each — and 3 meta replicas form the quorum.
+const (
+	fixtureHostID = 1
+	rsData        = 3
+	rsParity      = 2
+
+	gateNodeID    = 1
+	firstBlobNode = 2
+	blobNodes     = rsData + rsParity
+	firstMetaNode = firstBlobNode + blobNodes
+	metaNodes     = 3
+
+	// Cluster ports are never dialled — the pipe transport keys its registry
+	// by them, nothing binds them — but each must be unique within the host.
+	firstBlobPort = 16660
+	firstMetaPort = 17660
+
+	// accountID owns every bucket the fixture credentials create.
+	accountID = "123456789012"
+)
+
+// Fixture describes a running predastore cluster ready for real
 // S3/viperblock clients: a reachable endpoint and its default buckets
 // created. Its TLS cert is self-signed, so clients skip verification.
 type Fixture struct {
@@ -75,15 +100,15 @@ type Fixture struct {
 	Region    string
 	AccessKey string
 	SecretKey string
-	// DataDir is the daemon's base_path: it contains store/node-N/ for each
-	// configured node (db/, *.seg, *.idx, state.json), the exact layout a
-	// segment-inspection tool like segscan expects. The daemon keeps writing
-	// here for the life of the test binary, so callers that read it directly
-	// (rather than through the S3 API) must copy it first.
+	// DataDir is the host's data root: it contains node-N/ for each node that
+	// keeps state (db/, *.seg, *.idx), the exact layout a segment-inspection
+	// tool like segscan expects. The cluster keeps writing here for the life
+	// of the test binary, so callers that read it directly (rather than
+	// through the S3 API) must copy it first.
 	DataDir string
 }
 
-// Package-level singleton: one predastore daemon per test binary (per Go
+// Package-level singleton: one predastore cluster per test binary (per Go
 // package under test), amortising its startup cost across every test that
 // calls Start instead of paying it per-test. Guarded by a mutex rather than
 // sync.Once because a failed first attempt should not wedge every later
@@ -92,19 +117,21 @@ var (
 	mu      sync.Mutex
 	started bool
 	fixture *Fixture
+	stop    context.CancelFunc
+	drained chan struct{}
 )
 
 // fixtureDirPrefix names each run's fixture directory. It is a constant
 // because the sweep in Start matches on it.
 const fixtureDirPrefix = "predastore-fixture-"
 
-// Start starts a real predastore daemon the first time it's called within a
+// Start starts a real predastore cluster the first time it's called within a
 // test binary and returns connection details for it; every later call in
-// the same process returns the already-running fixture. The daemon
+// the same process returns the already-running fixture. The cluster
 // deliberately outlives any individual test — its temp dir uses
 // os.MkdirTemp rather than t.TempDir() so a finished test's cleanup can't
-// delete files the shared daemon still has open — and is left running until
-// the test process exits.
+// delete files the shared cluster still has open — and runs until Stop, or
+// until the test process exits.
 func Start(t *testing.T) *Fixture {
 	t.Helper()
 	mu.Lock()
@@ -114,12 +141,12 @@ func Start(t *testing.T) *Fixture {
 		return fixture
 	}
 
-	// The daemon runs until the process exits, so there is no point at which
-	// this run can remove its own directory. Reclaiming it is left to a later
-	// run's sweep, which is also what recovers a run killed mid-flight.
+	// The cluster can outlive this run, so there is no point at which it can
+	// remove its own directory. Reclaiming it is left to a later run's sweep,
+	// which is also what recovers a run killed mid-flight.
 	scratch.SweepAbandoned(os.TempDir(), fixtureDirPrefix, scratch.DefaultMaxAge)
 
-	testDir, err := os.MkdirTemp("", fixtureDirPrefix+"*") //nolint:usetesting // shared daemon outlives individual tests
+	testDir, err := os.MkdirTemp("", fixtureDirPrefix+"*") //nolint:usetesting // shared cluster outlives individual tests
 	if err != nil {
 		t.Fatalf("predastore fixture: create temp dir: %v", err)
 	}
@@ -139,7 +166,7 @@ func Start(t *testing.T) *Fixture {
 
 	// SSL_CERT_FILE injects the cert into the OS trust store for clients that
 	// verify it rather than skipping verification (objectstore's S3 client,
-	// for one). t.Setenv reverts when this test ends, while the daemon lives
+	// for one). t.Setenv reverts when this test ends, while the cluster lives
 	// on for the whole binary, so the pool is loaded here and now: crypto/x509
 	// caches it behind a sync.Once, and this call is what fixes our cert in it
 	// for every later test in the process.
@@ -148,8 +175,8 @@ func Start(t *testing.T) *Fixture {
 		t.Fatalf("predastore fixture: load system cert pool: %v", err)
 	}
 
-	// Predastore mandates a 32-byte master key at mode 0600 (rejected
-	// otherwise by internal/keyfile.Load).
+	// Predastore mandates a 32-byte master key at mode 0600 (masterkey.Load is
+	// fail-closed on anything group- or other-readable).
 	encryptionKeyPath := filepath.Join(testDir, "encryption.key")
 	testEncryptionKey := make([]byte, 32)
 	if _, err := rand.Read(testEncryptionKey); err != nil {
@@ -159,141 +186,56 @@ func Start(t *testing.T) *Fixture {
 		t.Fatalf("predastore fixture: write encryption key: %v", err)
 	}
 
-	// One host carrying every node, so the whole cluster runs in this process
-	// over the in-process pipe: no sockets to bind, no intra-cluster certs.
-	// RS(3,2) needs 5 shard-storage nodes; 3 state replicas form the quorum.
-	configPath := filepath.Join(testDir, "predastore_test.toml")
-	configContent := `version = "1.0"
-region = "us-east-1"
-debug = false
-disable_logging = false
-base_path = "` + testDir + `/"
-
-[rs]
-data = 3
-parity = 2
-
-[[host]]
-id = 1
-bind_addr = "127.0.0.1:16660"
-public_addr = "127.0.0.1:16660"
-data_dir = "` + filepath.Join(testDir, "cluster") + `"
-
-[[node]]
-id = 1
-host_id = 1
-role = "shard-storage"
-
-[[node]]
-id = 2
-host_id = 1
-role = "shard-storage"
-
-[[node]]
-id = 3
-host_id = 1
-role = "shard-storage"
-
-[[node]]
-id = 4
-host_id = 1
-role = "shard-storage"
-
-[[node]]
-id = 5
-host_id = 1
-role = "shard-storage"
-
-[[node]]
-id = 6
-host_id = 1
-role = "state-replica"
-
-[[node]]
-id = 7
-host_id = 1
-role = "state-replica"
-
-[[node]]
-id = 8
-host_id = 1
-role = "state-replica"
-
-[[auth]]
-access_key_id = "` + AccessKey + `"
-secret_access_key = "` + SecretKey + `"
-account_id = "123456789012"
-`
-
-	if err := os.WriteFile(configPath, []byte(configContent), 0644); err != nil { //nolint:gosec // ephemeral test-only config, contains no real secrets
+	// The config goes through the real file and the real loader rather than a
+	// hand-built struct, so the fixture trips over the same strict decode and
+	// topology validation an operator's install would.
+	dataDir := filepath.Join(testDir, "data")
+	configPath := filepath.Join(testDir, "predastore.toml")
+	config := topology(dataDir, certPath, keyPath, encryptionKeyPath)
+	if err := os.WriteFile(configPath, []byte(config), 0600); err != nil {
 		t.Fatalf("predastore fixture: write config: %v", err)
 	}
-
-	// The S3 gateway runs on top of a backend the caller wires: the cluster
-	// runtime owns the shard stores and the Raft replicas, and this process
-	// runs all of them.
-	cfg := &predastoreserver.Config{ConfigPath: configPath, BasePath: testDir}
-	if err := cfg.ReadConfig(); err != nil {
-		t.Fatalf("predastore fixture: read config: %v", err)
+	cfg, err := pds.LoadConfig(configPath)
+	if err != nil {
+		t.Fatalf("predastore fixture: load config: %v", err)
 	}
+
 	key, err := masterkey.Load(encryptionKeyPath)
 	if err != nil {
 		t.Fatalf("predastore fixture: load master key: %v", err)
 	}
-	rt, err := clusterrun.Build(cfg, clusterrun.AllNodeIDs(cfg), certPath, keyPath, key)
-	if err != nil {
-		t.Fatalf("predastore fixture: build cluster runtime: %v", err)
-	}
 
-	// Built directly against predastore/s3 rather than spinifex's own
-	// services/predastore wrapper: that wrapper pulls in spinifex/utils,
-	// and spinifex/utils' own tests import spinifex/testutil, which would be
-	// an import cycle if this package were reached from there. The
-	// wrapper's only other job — pidfile bookkeeping and signal-triggered
-	// shutdown — is irrelevant for a fixture daemon that lives for the
-	// whole test binary anyway.
-	server, err := predastoreserver.NewServer(
-		predastoreserver.WithConfigPath(configPath),
-		predastoreserver.WithAddress(testHost, testPort),
-		predastoreserver.WithTLS(certPath, keyPath),
-		predastoreserver.WithBasePath(testDir),
-		predastoreserver.WithDebug(false),
-		predastoreserver.WithPprof(false, ""),
-		predastoreserver.WithEncryptionKeyFile(encryptionKeyPath),
-		predastoreserver.WithPreparedBackend(rt.Backend),
-	)
-	if err != nil {
-		t.Fatalf("predastore fixture: create server: %v", err)
-	}
-
-	// The runtime is as permanent as the daemon it backs: an uncancellable
-	// context, left running until the test process exits. A t.Fatalf below
-	// takes the binary down with it, so nothing is left orphaned.
+	// Run blocks for as long as it serves and drains everything it started on
+	// the way out, so the cancel and the done channel are the whole handle
+	// Stop needs. A t.Fatalf below takes the binary down with it, so nothing
+	// is left orphaned either way.
+	ctx, cancel := context.WithCancel(context.Background())
+	done := make(chan struct{})
 	go func() {
-		if err := rt.Run(context.Background()); err != nil {
-			slog.Error("predastore fixture: cluster runtime exited", "error", err)
+		defer close(done)
+		if err := pds.Run(ctx, pds.Options{
+			Config:    cfg,
+			HostID:    pds.HostID(fixtureHostID),
+			MasterKey: key,
+		}); err != nil && ctx.Err() == nil {
+			slog.Error("predastore fixture: cluster exited", "error", err)
 		}
 	}()
+	stop, drained = cancel, done
 
-	// Writes need a committed leader; serving before one exists would fail
-	// the bucket creation below for no reason other than timing.
-	if err := rt.WaitReady(30 * time.Second); err != nil {
-		t.Fatalf("predastore fixture: no leader elected: %v", err)
+	// The gate holds off serving until the local Raft quorum has a leader, so
+	// an accepted connection also means writes will commit.
+	if !waitForReady(30*time.Second, done) {
+		cancel()
+		t.Fatal("predastore fixture: S3 gate did not become ready")
 	}
 
-	if err := server.ListenAndServeAsync(); err != nil {
-		t.Fatalf("predastore fixture: start server: %v", err)
-	}
-
-	if !waitForReady(10 * time.Second) {
-		t.Fatal("predastore fixture: server did not become ready")
-	}
-
-	// Create the default buckets via the S3 API so they're registered in
-	// distributed globalState (config buckets aren't visible to ListBuckets).
+	// Create the default buckets through the S3 API so they land in the meta
+	// plane; config-defined buckets are not visible to ListBuckets.
 	setupClient := s3Client(AccessKey, SecretKey)
 	for _, bucket := range []string{DefaultBucket, DefaultBucket2} {
 		if _, err := setupClient.CreateBucket(&s3.CreateBucketInput{Bucket: aws.String(bucket)}); err != nil {
+			cancel()
 			t.Fatalf("predastore fixture: create bucket %s: %v", bucket, err)
 		}
 	}
@@ -303,7 +245,7 @@ account_id = "123456789012"
 		Region:    Region,
 		AccessKey: AccessKey,
 		SecretKey: SecretKey,
-		DataDir:   testDir,
+		DataDir:   dataDir,
 	}
 	started = true
 	t.Logf("predastore fixture started, test dir: %s", testDir)
@@ -311,9 +253,54 @@ account_id = "123456789012"
 	return fixture
 }
 
+// Stop cancels the cluster and waits for it to drain. It is for a TestMain
+// that wants the goroutines gone before the binary reports; tests themselves
+// share the one cluster and must not stop it. Calling it without a running
+// fixture, or twice, is a no-op.
+func Stop() {
+	mu.Lock()
+	defer mu.Unlock()
+
+	if !started {
+		return
+	}
+	stop()
+	select {
+	case <-drained:
+	case <-time.After(30 * time.Second):
+		slog.Error("predastore fixture: cluster did not drain")
+	}
+	started, fixture = false, nil
+}
+
+// topology is the fixture's predastore configuration file: one host, the gate
+// on the fixed S3 port, and the blob and meta nodes the erasure code needs.
+func topology(dataDir, certPath, keyPath, encryptionKeyPath string) string {
+	var b strings.Builder
+
+	fmt.Fprintf(&b, "version = 1\nregion = %q\n\n[rs]\ndata = %d\nparity = %d\n\n", Region, rsData, rsParity)
+	fmt.Fprintf(&b, "[[host]]\nid = %d\nbind_addr = %q\naddr = %q\ndata_dir = %q\ntls_cert = %q\ntls_key = %q\nencryption_key = %q\n\n",
+		fixtureHostID, testHost, testHost, dataDir, certPath, keyPath, encryptionKeyPath)
+
+	fmt.Fprintf(&b, "[[host.node]]\nid = %d\nrole = \"gate\"\nport = %d\n\n", gateNodeID, testPort)
+	for i := range blobNodes {
+		fmt.Fprintf(&b, "[[host.node]]\nid = %d\nrole = \"blob\"\nport = %d\n\n", firstBlobNode+i, firstBlobPort+i)
+	}
+	for i := range metaNodes {
+		fmt.Fprintf(&b, "[[host.node]]\nid = %d\nrole = \"meta\"\nport = %d\n\n", firstMetaNode+i, firstMetaPort+i)
+	}
+
+	// No policy: a config-defined account is a trusted service account, so the
+	// gate skips the policy check for it entirely.
+	fmt.Fprintf(&b, "[[auth]]\naccess_key_id = %q\nsecret_access_key = %q\naccount_id = %q\n",
+		AccessKey, SecretKey, accountID)
+
+	return b.String()
+}
+
 // generateCertificate writes a self-signed TLS certificate and key for the
-// fixture daemon's S3 HTTPS frontend. Clients reach it with
-// InsecureSkipVerify, so nothing needs the cert in a trust store.
+// fixture's S3 HTTPS frontend. Clients reach it with InsecureSkipVerify, so
+// nothing needs the cert in a trust store.
 func generateCertificate(certPath, keyPath string) error {
 	priv, err := ecdsa.GenerateKey(elliptic.P256(), rand.Reader)
 	if err != nil {
@@ -375,9 +362,10 @@ func generateCertificate(certPath, keyPath string) error {
 	return nil
 }
 
-// waitForReady polls the fixture daemon's HTTPS endpoint until it accepts
-// connections or timeout elapses.
-func waitForReady(timeout time.Duration) bool {
+// waitForReady polls the S3 endpoint until it answers, the cluster exits or
+// timeout elapses. A cluster that failed to start closes done, which is the
+// difference between waiting out the timeout and reporting at once.
+func waitForReady(timeout time.Duration, done <-chan struct{}) bool {
 	client := &http.Client{
 		Transport: &http.Transport{
 			TLSClientConfig: &tls.Config{InsecureSkipVerify: true}, //nolint:gosec // self-signed cert generated by this same fixture, localhost-only
@@ -387,11 +375,14 @@ func waitForReady(timeout time.Duration) bool {
 
 	deadline := time.Now().Add(timeout)
 	for time.Now().Before(deadline) {
+		select {
+		case <-done:
+			return false
+		default:
+		}
 		resp, err := client.Get("https://" + net.JoinHostPort(testHost, strconv.Itoa(testPort)) + "/")
 		if err == nil {
 			resp.Body.Close()
-			// Give a bit more time for config to load.
-			time.Sleep(500 * time.Millisecond)
 			return true
 		}
 		time.Sleep(100 * time.Millisecond)
@@ -399,8 +390,8 @@ func waitForReady(timeout time.Duration) bool {
 	return false
 }
 
-// s3Client creates an AWS S3 client against the fixture daemon for fixture
-// setup only (bucket creation); test bodies build their own clients against
+// s3Client creates an AWS S3 client against the fixture for fixture setup
+// only (bucket creation); test bodies build their own clients against
 // whatever bucket/credentials their scenario needs.
 func s3Client(accessKey, secretKey string) *s3.S3 {
 	tr := &http.Transport{

tests/integration/main_test.go

@@ -7,6 +7,8 @@ import (
 	"os"
 	"path/filepath"
 	"testing"
+
+	testpredastore "github.com/mulgadc/spinifex/tests/fixtures/predastore"
 )
 
 // TestMain amortises the embedded NATS+JetStream server across every test in
@@ -49,6 +51,10 @@ func TestMain(m *testing.M) {
 
 	code := m.Run()
 
+	// The predastore fixture outlives any single test on purpose, so this is
+	// the only point at which its cluster goroutines can be drained.
+	testpredastore.Stop()
+
 	report := suiteConformance.report(policy, mode)
 	fmt.Fprintln(os.Stderr, report)
 	if blocking := suiteConformance.blocking(policy, mode); blocking != 0 {

tests/integration/segscan_oracle_test.go

@@ -17,10 +17,10 @@ import (
 )
 
 // segscanOracleNode is the fixture node this test scans. The fixture's RS
-// config is data=3/parity=2 across all 5 configured nodes (see
-// tests/fixtures/predastore.Start's config), so every PutObject places
-// exactly one shard — one live extent — on every node, node-1 included.
-const segscanOracleNode = "node-1"
+// config is data=3/parity=2 across exactly 5 blob nodes (see
+// tests/fixtures/predastore's topology), so every PutObject places exactly
+// one shard — one live extent — on every one of them, node-2 included.
+const segscanOracleNode = "node-2"
 
 // TestSegscanOracle_PutOverwriteDelete drives real S3 writes against the
 // predastore fixture and asserts scripts/segscan's live / dead-tombstoned /
@@ -50,7 +50,7 @@ func TestSegscanOracle_PutOverwriteDelete(t *testing.T) {
 		_, _ = cli.DeleteBucket(&s3.DeleteBucketInput{Bucket: aws.String(bucket)})
 	})
 
-	nodeDir := filepath.Join(fixture.DataDir, "store", segscanOracleNode)
+	nodeDir := filepath.Join(fixture.DataDir, segscanOracleNode)
 	const key = "segscan-oracle/object"
 
 	scan := func() *segscanoracle.Report {