cluster.sh is a single-script tool that builds and manages Docker-based test clusters for both Buckit (the target server under development) and MinIO (used as a migration source). Every node runs inside a privileged Docker container with:

systemd as PID 1 — the same init model as a bare-metal deployment
XFS loopback drives — real formatted volumes, not bind mounts
SSH — full shell access to any node for debugging or data loading

Both cluster types are managed via docker compose and can co-exist on the same host simultaneously, enabling end-to-end migration testing in a single terminal.

Prerequisites

Requirement	Notes
Docker Desktop ≥ 4.x (or Docker Engine)	Compose v2 must be available (`docker compose`)
`bash` ≥ 4	Ships with macOS via Homebrew; native on Linux
~1 GB free disk per node-drive	Default: 4 nodes × 4 drives × 1 GB = 16 GB per cluster
ARM64 host (Apple Silicon / linux/arm64)	The MinIO image is built from the ARM64 RPM; the Buckit image adapts to any architecture

Docker Desktop on Mac requires privileged containers to be allowed (enabled by default; no extra setting needed).

Quick Start

# ── Buckit cluster (4 nodes, 4 drives each) ──────────────────────────────────
./cluster.sh create

# ── MinIO single-node (migration source) ─────────────────────────────────────
./cluster.sh minio start

# ── MinIO 4-node distributed cluster ─────────────────────────────────────────
./cluster.sh minio start --nodes 4

# ── Tear everything down ─────────────────────────────────────────────────────
./cluster.sh destroy
./cluster.sh minio stop

File Layout

testing/cluster/
├── cluster.sh            # Main CLI — all commands live here
├── dockerfile-gen.sh     # Generates Dockerfile for the Buckit cluster
├── entrypoint.sh         # Entrypoint for Buckit cluster nodes
├── Dockerfile.minio      # Static Dockerfile for MinIO nodes (Rocky Linux 9)
├── minio-entrypoint.sh   # Entrypoint for MinIO nodes
└── README.md             # This file

# Generated at runtime (not committed):
├── Dockerfile            # Generated by dockerfile-gen.sh for Buckit
├── docker-compose.yml    # Generated compose for Buckit cluster
├── docker-compose.minio.yml  # Generated compose for MinIO cluster
├── .state/               # Saved Buckit cluster config
└── .state-minio/         # Saved MinIO cluster config

How It Works

Containers

Each node runs as a privileged Docker container using the host's Linux kernel. Privilege is required for two reasons:

modprobe loop — loads the loop block-device module so loopback images can be attached.
mount — mounts the formatted loopback devices inside the container.

The container's init process is systemd (/sbin/init), started by the entrypoint script after drives are set up. Systemd then brings up the target service (Buckit or MinIO) exactly as it would on a real server.

Loopback XFS Drives

Each node's drives are sparse image files stored in a named Docker volume (so data survives container restarts until destroy / minio stop is called).

At container start the entrypoint:

Calls fallocate to create each image file the first time.
Formats it with mkfs.xfs -f.
Attaches it to a loop device via losetup.
Mounts the loop device at the expected path.

On subsequent starts (e.g. after docker compose restart) the image files already exist, so only the losetup + mount steps are repeated.

Docker Desktop on Mac runs containers inside a Linux VM. Loop device attachments made by one container run can leak into the VM kernel across restarts. The entrypoint detects and detaches stale loop devices (losetup -d) before creating new ones.

Dockerfile Generation (Buckit)

Because the Buckit cluster supports multiple base images (Ubuntu, Debian, Rocky Linux, Fedora, …), the Dockerfile is generated at create/expand time by dockerfile-gen.sh. The generator detects the package manager from the image name and emits the correct apt-get or dnf install command.

The MinIO image is always Rocky Linux 9, so it uses a static Dockerfile.minio instead.

Docker Compose Generation

cluster.sh generates docker-compose.yml / docker-compose.minio.yml at runtime. This allows the same script to handle different node counts, drive counts, port bases, and cluster names without a templating engine.

State Files

After create / minio start, the chosen configuration is written to .state/config or .state-minio/config. Subsequent commands (expand, destroy, minio stop) source that file so you don't have to repeat flags.

Buckit Cluster

`create`

Builds the cluster image and starts N nodes.

./cluster.sh create [options]

What happens:

Builds the current repo's Buckit binary for Linux into testing/cluster/buckit.
dockerfile-gen.sh generates a Dockerfile tailored to the chosen base image.
docker-compose.yml is generated with one service per node.
docker compose up -d --build builds the image and starts all containers.
Each container's entrypoint creates loopback XFS drives, sets the root SSH password, then hands off to systemd.

With --empty, the same host and drive provisioning happens but the image does not include the buckit binary and no Buckit service is started. Use this when you want blank deployment hosts with a specific node and drive topology.

Example:

# Defaults: 4 nodes, 4 drives × 1 G, ubuntu:24.04
./cluster.sh create

# 8 nodes, Rocky Linux 9, 2 G drives, 512 M RAM each
./cluster.sh create --nodes 8 --image rockylinux:9 --drive-size 2G --memory 512M

# Empty deployment cluster: 6 hosts with 8 drives each, no Buckit service
./cluster.sh create --empty --nodes 6 --drives 8

# Named cluster (useful when running multiple clusters in parallel)
./cluster.sh create --name my-test --nodes 4

# Enable the single-trip GET prototype in the generated Buckit service env
./cluster.sh create --fast-get 1

`expand`

Adds more nodes to a running cluster without restarting the existing ones.

./cluster.sh expand --nodes NUM

--nodes here means how many to add, not the new total.

# Add 2 more nodes to the current cluster (total becomes 6 if 4 were created)
./cluster.sh expand --nodes 2

What happens:

Loads the saved state to recover the original node count and settings.
Regenerates docker-compose.yml for the new total.
Runs docker compose up -d --build — Docker Compose only starts the new services.

`destroy`

Stops all containers and removes volumes (drive data is lost).

./cluster.sh destroy

# Destroy a named cluster
./cluster.sh destroy --name my-test

Removes Dockerfile, docker-compose.yml, and .state/.

Options Reference

Flag	Default	Description
`-n, --nodes NUM`	`4`	Number of cluster nodes
`-d, --drives NUM`	`4`	Drives per node
`-s, --drive-size SIZE`	`1G`	Size per drive (e.g. `500M`, `2G`)
`-m, --memory SIZE`	`256M`	RAM limit per container
`-i, --image IMAGE`	`ubuntu:24.04`	Base Docker image
`--ssh-base-port PORT`	`2201`	First host port mapped to node SSH
`--ssh-password PASS`	`buckitadmin`	Root password for SSH
`--fast-get VALUE`	`0`	`BUCKIT_FAST_GET` value for Buckit nodes
`--empty`	disabled	Create hosts and drives only; skip Buckit install/start
`-N, --name NAME`	`buckit-test`	Compose project / cluster name

Port Allocation

Ports are allocated sequentially from the base. With defaults and 4 nodes:

Node	API (S3)	Console	SSH
node1	`9000`	`9001`	`2201`
node2	`9002`	`9003`	`2202`
node3	`9004`	`9005`	`2203`
node4	`9006`	`9007`	`2204`

Formula:

API port     = --base-port (9000) + (i−1) × 2
Console port = API port + 1
SSH port     = --ssh-base-port (2201) + (i−1)

SSH Access

# Connect to node 1 (password: buckitadmin)
ssh root@localhost -p 2201

# Connect to node 3
ssh root@localhost -p 2203

S3 Credentials

The default S3 access key / secret set on all Buckit nodes:

Key	Value
Access key	`buckitadmin`
Secret key	`buckitadmin`

Supported Base Images

The image must include systemd. Alpine is not supported.

Family	Tested images
Debian / Ubuntu	`ubuntu:24.04`, `ubuntu:22.04`, `debian:12`, `debian:11`
RHEL-family	`rockylinux:9`, `almalinux:9`, `fedora:40`, `amazonlinux:2023`, `quay.io/centos/centos:stream9`

MinIO Cluster

The MinIO cluster uses a static Dockerfile.minio based on Rocky Linux 9. It installs MinIO from the official ARM64 RPM following the MinIO RHEL deployment guide.

`minio start` — Single Node

./cluster.sh minio start [options]

Starts one standalone MinIO container. This is the simplest setup for migration testing when you don't need erasure coding on the source.

# Defaults: 1 node, 4 drives × 1 G
./cluster.sh minio start

# Custom credentials and drive size
./cluster.sh minio start \
  --root-user admin --root-password secret \
  --drives 2 --drive-size 2G

Output:

MinIO container 'minio-migration' is starting.

  API:     http://localhost:19000
  Console: http://localhost:19001
  SSH:     ssh root@localhost -p 2299  (password: minioadmin)
  User:    minioadmin / minioadmin

`minio start --nodes N` — Distributed Cluster

./cluster.sh minio start --nodes N [options]

Starts an N-node distributed MinIO cluster. Each node:

Gets its own named Docker volume for drive backing images.
Uses hostname minio1…minioN on the shared Docker bridge network.
Has MINIO_VOLUMES set to http://minio{1...N}:9000/mnt/data/drive{0...D-1} — MinIO's brace- expansion URL that points every node at the full topology.

MinIO recommends at least 4 nodes for distributed erasure coding.

# 4-node cluster, 4 drives each (MinIO's recommended minimum)
./cluster.sh minio start --nodes 4

# 4-node cluster with larger drives and a custom name
./cluster.sh minio start --nodes 4 --drives 4 --drive-size 2G \
  --name staging-minio --root-user myadmin --root-password mypw

Output (4 nodes):

MinIO cluster 'minio-migration' is starting (4 nodes).

  User: minioadmin  Pass: minioadmin  SSH pass: minioadmin

  Node      API                           Console                         SSH
  ----      ---                           -------                         ---
  node1     http://localhost:19000        http://localhost:19001           ssh root@localhost -p 2299
  node2     http://localhost:19002        http://localhost:19003           ssh root@localhost -p 2300
  node3     http://localhost:19004        http://localhost:19005           ssh root@localhost -p 2301
  node4     http://localhost:19006        http://localhost:19007           ssh root@localhost -p 2302

`minio stop`

Stops and removes all MinIO containers and their drive volumes.

./cluster.sh minio stop

# Stop a named cluster
./cluster.sh minio stop --name staging-minio

Options Reference

Flag	Default	Description
`-n, --nodes NUM`	`1`	Number of MinIO nodes
`--drives NUM`	`4`	Drives per node
`--drive-size SIZE`	`1G`	Size per drive
`--api-port PORT`	`19000`	First host API port
`--console-port PORT`	`19001`	First host console port (single-node only)
`--ssh-port PORT`	`2299`	First host SSH port
`--root-user USER`	`minioadmin`	`MINIO_ROOT_USER`
`--root-password PASS`	`minioadmin`	`MINIO_ROOT_PASSWORD`
`--ssh-password PASS`	`minioadmin`	Root SSH password
`--name NAME`	`minio-migration`	Compose project / cluster name

Port Allocation

Ports are allocated sequentially from the base. With defaults and 4 nodes:

Node	API (S3)	Console	SSH
minio1	`19000`	`19001`	`2299`
minio2	`19002`	`19003`	`2300`
minio3	`19004`	`19005`	`2301`
minio4	`19006`	`19007`	`2302`

Formula:

API port     = --api-port (19000) + (i−1) × 2
Console port = API port + 1
SSH port     = --ssh-port (2299) + (i−1)

The defaults are chosen to avoid overlap with the Buckit cluster's range (9000–9xxx / 2201–22xx), so both clusters can run at the same time.

SSH Access

# Connect to minio1 (password: minioadmin)
ssh root@localhost -p 2299

# Connect to minio3 in a 4-node cluster
ssh root@localhost -p 2301

Migration Testing Workflow

The typical workflow is to start a MinIO cluster (source), populate it with test data, start a Buckit cluster (target), connect the two networks so they can reach each other, then run mc mirror to migrate objects and verify integrity.

1. Start the MinIO source

# Single-node MinIO (simple cases)
./cluster.sh minio start

# Or a 4-node distributed MinIO (closer to production)
./cluster.sh minio start --nodes 4

2. Load test data into MinIO

# Configure the MinIO Client (mc)
mc alias set source http://localhost:19000 minioadmin minioadmin

# Create a bucket and upload data
mc mb source/my-bucket
mc cp --recursive /path/to/data source/my-bucket/

3. Start the Buckit target cluster

./cluster.sh create

# Configure mc to talk to Buckit node 1
mc alias set target http://localhost:9000 buckitadmin buckitadmin

4. Connect the Docker networks

By default the two clusters sit on separate bridge networks. To let mc mirror run container-to-container (bypassing host port forwarding), connect the MinIO containers to the Buckit network:

# Single-node MinIO
docker network connect buckit-test-net minio-migration

# 4-node MinIO cluster
docker network connect buckit-test-net minio-migration-node1
docker network connect buckit-test-net minio-migration-node2
docker network connect buckit-test-net minio-migration-node3
docker network connect buckit-test-net minio-migration-node4

Now containers on either network can resolve minio / minio1…minioN and node1…nodeN by hostname.

5. Mirror data to Buckit

mc mirror source/my-bucket target/my-bucket

6. Verify

mc diff source/my-bucket target/my-bucket

7. Tear down

./cluster.sh minio stop
./cluster.sh destroy

Defaults Reference

Buckit cluster

Setting	Default
Cluster name	`buckit-test`
Nodes	`4`
Drives per node	`4`
Drive size	`1G`
Memory per node	`256M`
Base image	`ubuntu:24.04`
S3 API base port	`9000`
SSH base port	`2201`
Root / S3 credentials	`buckitadmin` / `buckitadmin`

MinIO cluster

Setting	Default
Cluster name	`minio-migration`
Nodes	`1`
Drives per node	`4`
Drive size	`1G`
Base image	`rockylinux:9` (fixed)
S3 API base port	`19000`
SSH base port	`2299`
Root credentials	`minioadmin` / `minioadmin`
SSH password	`minioadmin`

README.md Unescape Escape

Buckit / MinIO Test Cluster Manager

Prerequisites

Quick Start

File Layout

How It Works

Containers

Loopback XFS Drives

Dockerfile Generation (Buckit)

Docker Compose Generation

State Files

Buckit Cluster

create

expand

destroy

Options Reference

Port Allocation

SSH Access

S3 Credentials

Supported Base Images

MinIO Cluster

minio start — Single Node

minio start --nodes N — Distributed Cluster

minio stop

Options Reference

Port Allocation

SSH Access

Migration Testing Workflow

1. Start the MinIO source

2. Load test data into MinIO

3. Start the Buckit target cluster

4. Connect the Docker networks

5. Mirror data to Buckit

6. Verify

7. Tear down

Defaults Reference

Buckit cluster

MinIO cluster

README.md

`create`

`expand`

`destroy`

`minio start` — Single Node

`minio start --nodes N` — Distributed Cluster

`minio stop`