buckit/testing/cluster

Buckit / MinIO Test Cluster Manager

cluster.sh is a single-script tool that builds and manages Docker-based test clusters for both Buckit (the target server under development) and MinIO (used as a migration source). Every node runs inside a privileged Docker container with:

• systemd as PID 1 — the same init model as a bare-metal deployment
• XFS loopback drives — real formatted volumes, not bind mounts
• SSH — full shell access to any node for debugging or data loading

Both cluster types are managed via docker compose and can co-exist on the same host simultaneously, enabling end-to-end migration testing in a single terminal.

---

Prerequisites

Requirement	Notes
Docker Desktop ≥ 4.x (or Docker Engine)	Compose v2 must be available (docker compose)
bash ≥ 4	Ships with macOS via Homebrew; native on Linux
~1 GB free disk per node-drive	Default: 4 nodes × 4 drives × 1 GB = 16 GB per cluster
ARM64 host (Apple Silicon / linux/arm64)	The MinIO image is built from the ARM64 RPM; the Buckit image adapts to any architecture

Docker Desktop on Mac requires privileged containers to be allowed (enabled by default; no extra setting needed).

---

Quick Start

# ── Buckit cluster (4 nodes, 4 drives each) ──────────────────────────────────
./cluster.sh create

# ── MinIO single-node (migration source) ─────────────────────────────────────
./cluster.sh minio start

# ── MinIO 4-node distributed cluster ─────────────────────────────────────────
./cluster.sh minio start --nodes 4

# ── Tear everything down ─────────────────────────────────────────────────────
./cluster.sh destroy
./cluster.sh minio stop

---

File Layout

testing/cluster/
├── cluster.sh            # Main CLI — all commands live here
├── dockerfile-gen.sh     # Generates Dockerfile for the Buckit cluster
├── entrypoint.sh         # Entrypoint for Buckit cluster nodes
├── Dockerfile.minio      # Static Dockerfile for MinIO nodes (Rocky Linux 9)
├── minio-entrypoint.sh   # Entrypoint for MinIO nodes
└── README.md             # This file

# Generated at runtime (not committed):
├── Dockerfile            # Generated by dockerfile-gen.sh for Buckit
├── docker-compose.yml    # Generated compose for Buckit cluster
├── docker-compose.minio.yml  # Generated compose for MinIO cluster
├── .state/               # Saved Buckit cluster config
└── .state-minio/         # Saved MinIO cluster config

---

How It Works

Containers

Each node runs as a privileged Docker container using the host's Linux kernel. Privilege is required for two reasons:

1. modprobe loop — loads the loop block-device module so loopback images can be attached.
2. mount — mounts the formatted loopback devices inside the container.

The container's init process is systemd (/sbin/init), started by the entrypoint script after drives are set up. Systemd then brings up the target service (Buckit or MinIO) exactly as it would on a real server.

Loopback XFS Drives

Each node's drives are sparse image files stored in a named Docker volume (so data survives container restarts until destroy / minio stop is called).

At container start the entrypoint:

1. Calls fallocate to create each image file the first time.
2. Formats it with mkfs.xfs -f.
3. Attaches it to a loop device via losetup.
4. Mounts the loop device at the expected path.

On subsequent starts (e.g. after docker compose restart) the image files already exist, so only the losetup + mount steps are repeated.

Docker Desktop on Mac runs containers inside a Linux VM. Loop device attachments made by one container run can leak into the VM kernel across restarts. The entrypoint detects and detaches stale loop devices (losetup -d) before creating new ones.

Dockerfile Generation (Buckit)

Because the Buckit cluster supports multiple base images (Ubuntu, Debian, Rocky Linux, Fedora, …), the Dockerfile is generated at create/expand time by dockerfile-gen.sh. The generator detects the package manager from the image name and emits the correct apt-get or dnf install command.

The MinIO image is always Rocky Linux 9, so it uses a static Dockerfile.minio instead.

Docker Compose Generation

cluster.sh generates docker-compose.yml / docker-compose.minio.yml at runtime. This allows the same script to handle different node counts, drive counts, port bases, and cluster names without a templating engine.

State Files

After create / minio start, the chosen configuration is written to .state/config or .state-minio/config. Subsequent commands (expand, destroy, minio stop) source that file so you don't have to repeat flags.

---

Buckit Cluster

create

Builds the cluster image and starts N nodes.

./cluster.sh create [options]

What happens:

1. Builds the current repo's Buckit binary for Linux into testing/cluster/buckit.
2. dockerfile-gen.sh generates a Dockerfile tailored to the chosen base image.
3. docker-compose.yml is generated with one service per node.
4. docker compose up -d --build builds the image and starts all containers.
5. Each container's entrypoint creates loopback XFS drives, sets the root SSH password, then hands off to systemd.

Example:

# Defaults: 4 nodes, 4 drives × 1 G, ubuntu:24.04
./cluster.sh create

# 8 nodes, Rocky Linux 9, 2 G drives, 512 M RAM each
./cluster.sh create --nodes 8 --image rockylinux:9 --drive-size 2G --memory 512M

# Named cluster (useful when running multiple clusters in parallel)
./cluster.sh create --name my-test --nodes 4

# Enable the single-trip GET prototype in the generated Buckit service env
./cluster.sh create --fast-get 1

expand

Adds more nodes to a running cluster without restarting the existing ones.

./cluster.sh expand --nodes NUM

--nodes here means how many to add, not the new total.

# Add 2 more nodes to the current cluster (total becomes 6 if 4 were created)
./cluster.sh expand --nodes 2

What happens:

1. Loads the saved state to recover the original node count and settings.
2. Regenerates docker-compose.yml for the new total.
3. Runs docker compose up -d --build — Docker Compose only starts the new services.

destroy

Stops all containers and removes volumes (drive data is lost).

./cluster.sh destroy

# Destroy a named cluster
./cluster.sh destroy --name my-test

Removes Dockerfile, docker-compose.yml, and .state/.

Options Reference

Flag	Default	Description
-n, --nodes NUM	4	Number of cluster nodes
-d, --drives NUM	4	Drives per node
-s, --drive-size SIZE	1G	Size per drive (e.g. 500M, 2G)
-m, --memory SIZE	256M	RAM limit per container
-i, --image IMAGE	ubuntu:24.04	Base Docker image
--ssh-base-port PORT	2201	First host port mapped to node SSH
--ssh-password PASS	buckitadmin	Root password for SSH
--fast-get VALUE	0	BUCKIT_FAST_GET value for Buckit nodes
-N, --name NAME	buckit-test	Compose project / cluster name

Port Allocation

Ports are allocated sequentially from the base. With defaults and 4 nodes:

Node	API (S3)	Console	SSH
node1	9000	9001	2201
node2	9002	9003	2202
node3	9004	9005	2203
node4	9006	9007	2204

Formula:

API port     = --base-port (9000) + (i−1) × 2
Console port = API port + 1
SSH port     = --ssh-base-port (2201) + (i−1)

SSH Access

# Connect to node 1 (password: buckitadmin)
ssh root@localhost -p 2201

# Connect to node 3
ssh root@localhost -p 2203

S3 Credentials

The default S3 access key / secret set on all Buckit nodes:

Key	Value
Access key	buckitadmin
Secret key	buckitadmin

Supported Base Images

The image must include systemd. Alpine is not supported.

Family	Tested images
Debian / Ubuntu	ubuntu:24.04, ubuntu:22.04, debian:12, debian:11
RHEL-family	rockylinux:9, almalinux:9, fedora:40, amazonlinux:2023, quay.io/centos/centos:stream9

---

MinIO Cluster

The MinIO cluster uses a static Dockerfile.minio based on Rocky Linux 9. It installs MinIO from the official ARM64 RPM following the MinIO RHEL deployment guide.

minio start — Single Node

./cluster.sh minio start [options]

Starts one standalone MinIO container. This is the simplest setup for migration testing when you don't need erasure coding on the source.

# Defaults: 1 node, 4 drives × 1 G
./cluster.sh minio start

# Custom credentials and drive size
./cluster.sh minio start \
  --root-user admin --root-password secret \
  --drives 2 --drive-size 2G

Output:

MinIO container 'minio-migration' is starting.

  API:     http://localhost:19000
  Console: http://localhost:19001
  SSH:     ssh root@localhost -p 2299  (password: minioadmin)
  User:    minioadmin / minioadmin

minio start --nodes N — Distributed Cluster

./cluster.sh minio start --nodes N [options]

Starts an N-node distributed MinIO cluster. Each node:

• Gets its own named Docker volume for drive backing images.
• Uses hostname minio1…minioN on the shared Docker bridge network.
• Has MINIO_VOLUMES set to http://minio{1...N}:9000/mnt/data/drive{0...D-1} — MinIO's brace- expansion URL that points every node at the full topology.

MinIO recommends at least 4 nodes for distributed erasure coding.

# 4-node cluster, 4 drives each (MinIO's recommended minimum)
./cluster.sh minio start --nodes 4

# 4-node cluster with larger drives and a custom name
./cluster.sh minio start --nodes 4 --drives 4 --drive-size 2G \
  --name staging-minio --root-user myadmin --root-password mypw

Output (4 nodes):

MinIO cluster 'minio-migration' is starting (4 nodes).

  User: minioadmin  Pass: minioadmin  SSH pass: minioadmin

  Node      API                           Console                         SSH
  ----      ---                           -------                         ---
  node1     http://localhost:19000        http://localhost:19001           ssh root@localhost -p 2299
  node2     http://localhost:19002        http://localhost:19003           ssh root@localhost -p 2300
  node3     http://localhost:19004        http://localhost:19005           ssh root@localhost -p 2301
  node4     http://localhost:19006        http://localhost:19007           ssh root@localhost -p 2302

minio stop

Stops and removes all MinIO containers and their drive volumes.

./cluster.sh minio stop

# Stop a named cluster
./cluster.sh minio stop --name staging-minio

Options Reference

Flag	Default	Description
-n, --nodes NUM	1	Number of MinIO nodes
--drives NUM	4	Drives per node
--drive-size SIZE	1G	Size per drive
--api-port PORT	19000	First host API port
--console-port PORT	19001	First host console port (single-node only)
--ssh-port PORT	2299	First host SSH port
--root-user USER	minioadmin	MINIO_ROOT_USER
--root-password PASS	minioadmin	MINIO_ROOT_PASSWORD
--ssh-password PASS	minioadmin	Root SSH password
--name NAME	minio-migration	Compose project / cluster name

Port Allocation

Ports are allocated sequentially from the base. With defaults and 4 nodes:

Node	API (S3)	Console	SSH
minio1	19000	19001	2299
minio2	19002	19003	2300
minio3	19004	19005	2301
minio4	19006	19007	2302

Formula:

API port     = --api-port (19000) + (i−1) × 2
Console port = API port + 1
SSH port     = --ssh-port (2299) + (i−1)

The defaults are chosen to avoid overlap with the Buckit cluster's range (9000–9xxx / 2201–22xx), so both clusters can run at the same time.

SSH Access

# Connect to minio1 (password: minioadmin)
ssh root@localhost -p 2299

# Connect to minio3 in a 4-node cluster
ssh root@localhost -p 2301

---

Migration Testing Workflow

The typical workflow is to start a MinIO cluster (source), populate it with test data, start a Buckit cluster (target), connect the two networks so they can reach each other, then run mc mirror to migrate objects and verify integrity.

1. Start the MinIO source

# Single-node MinIO (simple cases)
./cluster.sh minio start

# Or a 4-node distributed MinIO (closer to production)
./cluster.sh minio start --nodes 4

2. Load test data into MinIO

# Configure the MinIO Client (mc)
mc alias set source http://localhost:19000 minioadmin minioadmin

# Create a bucket and upload data
mc mb source/my-bucket
mc cp --recursive /path/to/data source/my-bucket/

3. Start the Buckit target cluster

./cluster.sh create

# Configure mc to talk to Buckit node 1
mc alias set target http://localhost:9000 buckitadmin buckitadmin

4. Connect the Docker networks

By default the two clusters sit on separate bridge networks. To let mc mirror run container-to-container (bypassing host port forwarding), connect the MinIO containers to the Buckit network:

# Single-node MinIO
docker network connect buckit-test-net minio-migration

# 4-node MinIO cluster
docker network connect buckit-test-net minio-migration-node1
docker network connect buckit-test-net minio-migration-node2
docker network connect buckit-test-net minio-migration-node3
docker network connect buckit-test-net minio-migration-node4

Now containers on either network can resolve minio / minio1…minioN and node1…nodeN by hostname.

5. Mirror data to Buckit

mc mirror source/my-bucket target/my-bucket

6. Verify

mc diff source/my-bucket target/my-bucket

7. Tear down

./cluster.sh minio stop
./cluster.sh destroy

---

Defaults Reference

Buckit cluster

Setting	Default
Cluster name	buckit-test
Nodes	4
Drives per node	4
Drive size	1G
Memory per node	256M
Base image	ubuntu:24.04
S3 API base port	9000
SSH base port	2201
Root / S3 credentials	buckitadmin / buckitadmin

MinIO cluster

Setting	Default
Cluster name	minio-migration
Nodes	1
Drives per node	4
Drive size	1G
Base image	rockylinux:9 (fixed)
S3 API base port	19000
SSH base port	2299
Root credentials	minioadmin / minioadmin
SSH password	minioadmin