rustfs/.docker/test/site-replication

Site Replication Docker Compose Test

Purpose

This directory contains a local three-site RustFS site replication check. It is intended to verify the admin site-replication flow against real containers:

• three independent RustFS sites start successfully
• the MinIO-compatible mc admin replicate add command configures all three sites
• a bucket and object written to site 1 are replicated to site 2 and site 3
• mc admin replicate status can read the resulting site-replication state

The compose file uses named volumes so the test does not require preparing host bind-mount directories. Because Docker named volumes usually share the same physical device in local desktop environments, this test compose defaults RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true. Keep that setting limited to local test and CI environments.

Files

• docker-compose.yml: three RustFS sites, a volume permission helper, and a one-shot setup/check container
• run-object-flow-check.sh: host-side upload/download verification for replicated 10 MiB to 100 MiB objects

Ports

Default host endpoints:

• Site 1 API: http://127.0.0.1:9000
• Site 1 Console: http://127.0.0.1:9001
• Site 2 API: http://127.0.0.1:9010
• Site 2 Console: http://127.0.0.1:9011
• Site 3 API: http://127.0.0.1:9020
• Site 3 Console: http://127.0.0.1:9021

Default credentials are rustfsadmin / rustfsadmin. These are for local testing only.

Run

From the repository root:

docker compose -f .docker/test/site-replication/docker-compose.yml up

To test a locally built image instead of Docker Hub rustfs/rustfs:latest, set RUSTFS_SITE_REPL_IMAGE:

docker build -f Dockerfile.source -t rustfs-site-repl-local:latest .
RUSTFS_SITE_REPL_IMAGE=rustfs-site-repl-local:latest \
docker compose -f .docker/test/site-replication/docker-compose.yml up

For a detached run:

docker compose -f .docker/test/site-replication/docker-compose.yml up -d
docker compose -f .docker/test/site-replication/docker-compose.yml logs -f site-replication-setup

Test Flow

The compose stack performs these steps:

1. site-replication-volume-permission-helper fixes ownership on all named volumes for the RustFS runtime user.
2. rustfs-site1, rustfs-site2, and rustfs-site3 start as separate RustFS sites.
3. Each site exposes its S3 API and Console on a unique host port.
4. Health checks wait for /health/ready on each RustFS container.
5. site-replication-setup configures mc aliases for all three sites.
6. The setup container waits until mc admin info succeeds for all sites.
7. It runs:

mc admin replicate add site1 site2 site3

8. It creates the test bucket on site 1 and uploads from-site1.txt.
9. It polls site 2 and site 3 until the replicated object is visible.
10. It prints mc admin replicate status site1.

The setup container exits with status 0 only after the object replication check passes.

Object Flow Check

After the compose setup succeeds, run the larger object flow check from the repository root:

.docker/test/site-replication/run-object-flow-check.sh

The script creates five local files and uploads them from different sites:

• 10 MiB from site 1
• 25 MiB from site 2
• 50 MiB from site 3
• 75 MiB from site 1
• 100 MiB from site 2

For each uploaded object, the script waits for replication to the other two sites, downloads the object from those sites, and verifies both byte size and SHA-256 checksum. It uses a temporary mc config directory, so it does not overwrite existing host aliases.

The default bucket is site-repl-flow-check. Override it when needed:

RUSTFS_SITE_REPL_FLOW_BUCKET='site-repl-large-flow' \
.docker/test/site-replication/run-object-flow-check.sh

The script keeps uploaded objects under a timestamped prefix. Override the prefix for repeatable runs:

RUSTFS_SITE_REPL_FLOW_PREFIX='manual-check-001' \
.docker/test/site-replication/run-object-flow-check.sh

If replication is slow on the local machine, increase polling:

RUSTFS_SITE_REPL_WAIT_ATTEMPTS=180 \
RUSTFS_SITE_REPL_WAIT_SLEEP_SECONDS=2 \
.docker/test/site-replication/run-object-flow-check.sh

Optional Settings

Override local test credentials:

RUSTFS_SITE_REPL_ACCESS_KEY='localadmin' \
RUSTFS_SITE_REPL_SECRET_KEY='localadmin-secret' \
docker compose -f .docker/test/site-replication/docker-compose.yml up

Use a different test bucket:

RUSTFS_SITE_REPL_BUCKET='site-repl-check' \
docker compose -f .docker/test/site-replication/docker-compose.yml up

Enable ILM expiry rule replication during site setup:

RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY=true \
docker compose -f .docker/test/site-replication/docker-compose.yml up

Use this only when the test needs lifecycle expiry metadata included in site replication.

Manual Checks

After the setup container succeeds, you can inspect the sites with mc from the host:

mc alias set site1 http://127.0.0.1:9000 rustfsadmin rustfsadmin
mc alias set site2 http://127.0.0.1:9010 rustfsadmin rustfsadmin
mc alias set site3 http://127.0.0.1:9020 rustfsadmin rustfsadmin

mc admin replicate info site1
mc admin replicate status site1
mc stat site2/site-repl-demo/from-site1.txt
mc stat site3/site-repl-demo/from-site1.txt

After larger object flow checks, replication should converge without a growing queue:

mc admin replicate status site1
mc admin replicate status site2
mc admin replicate status site3

Useful Docker commands:

docker compose -f .docker/test/site-replication/docker-compose.yml ps
docker compose -f .docker/test/site-replication/docker-compose.yml logs --no-color --tail=200
docker compose -f .docker/test/site-replication/docker-compose.yml logs --no-color site-replication-setup

Cleanup

Remove containers and named volumes:

docker compose -f .docker/test/site-replication/docker-compose.yml down -v

Use down -v before rerunning the full setup from scratch. Site replication state is persisted in the named volumes, so rerunning without deleting volumes may attempt to add an already-configured replication topology.