mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-05 21:07:43 +00:00
唐小鸭
4efefd67b5
* fix(site-replication): Add Docker Compose setup for site replication testing - Fixed the site replication flag issue and resolved the replication storm. - Introduced a new directory for site replication tests with Docker Compose. - Created `docker-compose.yml` to define three RustFS sites and a setup container. - Added `README.md` to document the purpose, usage, and test flow of the replication setup. - Implemented `run-object-flow-check.sh` script to verify object replication across sites. - Configured health checks and volume permissions for the RustFS containers. - Enabled customization of access keys, bucket names, and other parameters via environment variables. * fix * fix
184 lines
6.0 KiB
Markdown
184 lines
6.0 KiB
Markdown
# Site Replication Docker Compose Test
|
|
|
|
## Purpose
|
|
|
|
This directory contains a local three-site RustFS site replication check. It is intended to verify the admin site-replication flow against real containers:
|
|
|
|
- three independent RustFS sites start successfully
|
|
- the MinIO-compatible `mc admin replicate add` command configures all three sites
|
|
- a bucket and object written to site 1 are replicated to site 2 and site 3
|
|
- `mc admin replicate status` can read the resulting site-replication state
|
|
|
|
The compose file uses named volumes so the test does not require preparing host bind-mount directories.
|
|
Because Docker named volumes usually share the same physical device in local desktop environments, this test compose defaults `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true`. Keep that setting limited to local test and CI environments.
|
|
|
|
## Files
|
|
|
|
- `docker-compose.yml`: three RustFS sites, a volume permission helper, and a one-shot setup/check container
|
|
- `run-object-flow-check.sh`: host-side upload/download verification for replicated 10 MiB to 100 MiB objects
|
|
|
|
## Ports
|
|
|
|
Default host endpoints:
|
|
|
|
- Site 1 API: `http://127.0.0.1:9000`
|
|
- Site 1 Console: `http://127.0.0.1:9001`
|
|
- Site 2 API: `http://127.0.0.1:9010`
|
|
- Site 2 Console: `http://127.0.0.1:9011`
|
|
- Site 3 API: `http://127.0.0.1:9020`
|
|
- Site 3 Console: `http://127.0.0.1:9021`
|
|
|
|
Default credentials are `rustfsadmin` / `rustfsadmin`. These are for local testing only.
|
|
|
|
## Run
|
|
|
|
From the repository root:
|
|
|
|
```bash
|
|
docker compose -f .docker/test/site-replication/docker-compose.yml up
|
|
```
|
|
|
|
To test a locally built image instead of Docker Hub `rustfs/rustfs:latest`, set `RUSTFS_SITE_REPL_IMAGE`:
|
|
|
|
```bash
|
|
docker build -f Dockerfile.source -t rustfs-site-repl-local:latest .
|
|
RUSTFS_SITE_REPL_IMAGE=rustfs-site-repl-local:latest \
|
|
docker compose -f .docker/test/site-replication/docker-compose.yml up
|
|
```
|
|
|
|
For a detached run:
|
|
|
|
```bash
|
|
docker compose -f .docker/test/site-replication/docker-compose.yml up -d
|
|
docker compose -f .docker/test/site-replication/docker-compose.yml logs -f site-replication-setup
|
|
```
|
|
|
|
## Test Flow
|
|
|
|
The compose stack performs these steps:
|
|
|
|
1. `site-replication-volume-permission-helper` fixes ownership on all named volumes for the RustFS runtime user.
|
|
2. `rustfs-site1`, `rustfs-site2`, and `rustfs-site3` start as separate RustFS sites.
|
|
3. Each site exposes its S3 API and Console on a unique host port.
|
|
4. Health checks wait for `/health/ready` on each RustFS container.
|
|
5. `site-replication-setup` configures `mc` aliases for all three sites.
|
|
6. The setup container waits until `mc admin info` succeeds for all sites.
|
|
7. It runs:
|
|
|
|
```bash
|
|
mc admin replicate add site1 site2 site3
|
|
```
|
|
|
|
8. It creates the test bucket on site 1 and uploads `from-site1.txt`.
|
|
9. It polls site 2 and site 3 until the replicated object is visible.
|
|
10. It prints `mc admin replicate status site1`.
|
|
|
|
The setup container exits with status `0` only after the object replication check passes.
|
|
|
|
## Object Flow Check
|
|
|
|
After the compose setup succeeds, run the larger object flow check from the repository root:
|
|
|
|
```bash
|
|
.docker/test/site-replication/run-object-flow-check.sh
|
|
```
|
|
|
|
The script creates five local files and uploads them from different sites:
|
|
|
|
- 10 MiB from site 1
|
|
- 25 MiB from site 2
|
|
- 50 MiB from site 3
|
|
- 75 MiB from site 1
|
|
- 100 MiB from site 2
|
|
|
|
For each uploaded object, the script waits for replication to the other two sites, downloads the object from those sites, and verifies both byte size and SHA-256 checksum. It uses a temporary `mc` config directory, so it does not overwrite existing host aliases.
|
|
|
|
The default bucket is `site-repl-flow-check`. Override it when needed:
|
|
|
|
```bash
|
|
RUSTFS_SITE_REPL_FLOW_BUCKET='site-repl-large-flow' \
|
|
.docker/test/site-replication/run-object-flow-check.sh
|
|
```
|
|
|
|
The script keeps uploaded objects under a timestamped prefix. Override the prefix for repeatable runs:
|
|
|
|
```bash
|
|
RUSTFS_SITE_REPL_FLOW_PREFIX='manual-check-001' \
|
|
.docker/test/site-replication/run-object-flow-check.sh
|
|
```
|
|
|
|
If replication is slow on the local machine, increase polling:
|
|
|
|
```bash
|
|
RUSTFS_SITE_REPL_WAIT_ATTEMPTS=180 \
|
|
RUSTFS_SITE_REPL_WAIT_SLEEP_SECONDS=2 \
|
|
.docker/test/site-replication/run-object-flow-check.sh
|
|
```
|
|
|
|
## Optional Settings
|
|
|
|
Override local test credentials:
|
|
|
|
```bash
|
|
RUSTFS_SITE_REPL_ACCESS_KEY='localadmin' \
|
|
RUSTFS_SITE_REPL_SECRET_KEY='localadmin-secret' \
|
|
docker compose -f .docker/test/site-replication/docker-compose.yml up
|
|
```
|
|
|
|
Use a different test bucket:
|
|
|
|
```bash
|
|
RUSTFS_SITE_REPL_BUCKET='site-repl-check' \
|
|
docker compose -f .docker/test/site-replication/docker-compose.yml up
|
|
```
|
|
|
|
Enable ILM expiry rule replication during site setup:
|
|
|
|
```bash
|
|
RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY=true \
|
|
docker compose -f .docker/test/site-replication/docker-compose.yml up
|
|
```
|
|
|
|
Use this only when the test needs lifecycle expiry metadata included in site replication.
|
|
|
|
## Manual Checks
|
|
|
|
After the setup container succeeds, you can inspect the sites with `mc` from the host:
|
|
|
|
```bash
|
|
mc alias set site1 http://127.0.0.1:9000 rustfsadmin rustfsadmin
|
|
mc alias set site2 http://127.0.0.1:9010 rustfsadmin rustfsadmin
|
|
mc alias set site3 http://127.0.0.1:9020 rustfsadmin rustfsadmin
|
|
|
|
mc admin replicate info site1
|
|
mc admin replicate status site1
|
|
mc stat site2/site-repl-demo/from-site1.txt
|
|
mc stat site3/site-repl-demo/from-site1.txt
|
|
```
|
|
|
|
After larger object flow checks, replication should converge without a growing queue:
|
|
|
|
```bash
|
|
mc admin replicate status site1
|
|
mc admin replicate status site2
|
|
mc admin replicate status site3
|
|
```
|
|
|
|
Useful Docker commands:
|
|
|
|
```bash
|
|
docker compose -f .docker/test/site-replication/docker-compose.yml ps
|
|
docker compose -f .docker/test/site-replication/docker-compose.yml logs --no-color --tail=200
|
|
docker compose -f .docker/test/site-replication/docker-compose.yml logs --no-color site-replication-setup
|
|
```
|
|
|
|
## Cleanup
|
|
|
|
Remove containers and named volumes:
|
|
|
|
```bash
|
|
docker compose -f .docker/test/site-replication/docker-compose.yml down -v
|
|
```
|
|
|
|
Use `down -v` before rerunning the full setup from scratch. Site replication state is persisted in the named volumes, so rerunning without deleting volumes may attempt to add an already-configured replication topology.
|