mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 08:18:18 +00:00
fix(site-replication): site replication flag issue and resolved the replication storm (#4120)
* fix(site-replication): Add Docker Compose setup for site replication testing - Fixed the site replication flag issue and resolved the replication storm. - Introduced a new directory for site replication tests with Docker Compose. - Created `docker-compose.yml` to define three RustFS sites and a setup container. - Added `README.md` to document the purpose, usage, and test flow of the replication setup. - Implemented `run-object-flow-check.sh` script to verify object replication across sites. - Configured health checks and volume permissions for the RustFS containers. - Enabled customization of access keys, bucket names, and other parameters via environment variables. * fix * fix
This commit is contained in:
唐小鸭
@@ -0,0 +1,91 @@
|
||||
# Rio compatibility compose files
|
||||
|
||||
These compose files prepare 4-node, 4-disk clusters for rio/rio-v2 storage format compatibility checks. All disks are bind-mounted under `.docker/compat/data` so the on-disk files remain available on the host.
|
||||
|
||||
## Clusters
|
||||
|
||||
```bash
|
||||
docker compose -f .docker/compat/docker-compose.rustfs-beta5.yml up -d --build
|
||||
docker compose -f .docker/compat/docker-compose.minio.yml up -d
|
||||
docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
|
||||
```
|
||||
|
||||
Default API endpoints:
|
||||
|
||||
- RustFS `1.0.0-beta.5`: `http://127.0.0.1:9100`
|
||||
- MinIO: `http://127.0.0.1:9200`
|
||||
- current main with `rio-v2`: `http://127.0.0.1:9300`
|
||||
|
||||
## Reading old datasets with rio-v2
|
||||
|
||||
Stop the writer cluster before mounting its disks into the rio-v2 cluster.
|
||||
|
||||
```bash
|
||||
docker compose -f .docker/compat/docker-compose.rustfs-beta5.yml down
|
||||
RUSTFS_RIO_V2_DATASET=./data/rustfs-beta5 \
|
||||
docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
|
||||
|
||||
docker compose -f .docker/compat/docker-compose.minio.yml down
|
||||
RUSTFS_RIO_V2_DATASET=./data/minio \
|
||||
docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
|
||||
```
|
||||
|
||||
## 200G object mix
|
||||
|
||||
Use the same bucket/object matrix against the beta5 and MinIO endpoints, then read it back through the rio-v2 endpoint. A practical 200G mix is:
|
||||
|
||||
- 1 KiB x 1024
|
||||
- 1 MiB x 1024
|
||||
- 64 MiB x 512
|
||||
- 1 GiB x 64
|
||||
- 8 GiB x 12
|
||||
- 6 GiB x 1
|
||||
|
||||
Compression is enabled by default for RustFS and MinIO. Server-side KMS/SSE settings are intentionally left to environment variables or mounted key directories so real key material is not committed. For SSE-C cases, run the clusters with TLS because MinIO requires HTTPS for SSE-C.
|
||||
|
||||
## High-concurrency write/read stress
|
||||
|
||||
Use `run_rw_compat_stress.sh` to generate a manifest on an old endpoint, then verify the same objects through the rio-v2 endpoint after mounting the old disks.
|
||||
|
||||
```bash
|
||||
.docker/compat/run_rw_compat_stress.sh \
|
||||
--mode write \
|
||||
--endpoint http://127.0.0.1:9100 \
|
||||
--access-key rustfsadmin \
|
||||
--secret-key rustfsadmin \
|
||||
--bucket compat-beta5 \
|
||||
--concurrency 96
|
||||
|
||||
RUSTFS_RIO_V2_DATASET=./data/rustfs-beta5 \
|
||||
docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
|
||||
|
||||
.docker/compat/run_rw_compat_stress.sh \
|
||||
--mode verify \
|
||||
--endpoint http://127.0.0.1:9300 \
|
||||
--access-key rustfsadmin \
|
||||
--secret-key rustfsadmin \
|
||||
--bucket compat-beta5 \
|
||||
--concurrency 96 \
|
||||
--manifest target/compat/rw-stress-YYYYmmdd-HHMMSS/manifest.csv
|
||||
```
|
||||
|
||||
For encrypted datasets, add `--encryption sse-s3`, `--encryption sse-kms --sse-kms-key-id <key-id>`, or `--encryption sse-c --sse-c-key-file <raw-32-byte-key-file>` to both the write and verify commands.
|
||||
|
||||
## 5 GiB encrypted compatibility run
|
||||
|
||||
The `5g` profile covers 1 KiB, 1 MiB, 16 MiB, 64 MiB, and 1 GiB objects and totals exactly 5 GiB. Generate `compat-key.key` under `.docker/compat/kms/rustfs-compat`, enable local KMS on both RustFS clusters, then use the same encryption arguments while writing with beta5 and verifying with rio-v2. Set non-default local test credentials first because distributed listeners reject the built-in default credentials.
|
||||
|
||||
```bash
|
||||
export COMPAT_ACCESS_KEY='<non-default-access-key>'
|
||||
export COMPAT_SECRET_KEY='<non-default-secret-key>'
|
||||
|
||||
RUSTFS_ACCESS_KEY="$COMPAT_ACCESS_KEY" RUSTFS_SECRET_KEY="$COMPAT_SECRET_KEY" \
|
||||
RUSTFS_KMS_ENABLE=true RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS=true \
|
||||
docker compose -f .docker/compat/docker-compose.rustfs-beta5.yml up -d
|
||||
|
||||
.docker/compat/run_rw_compat_stress.sh \
|
||||
--mode write --endpoint http://127.0.0.1:9100 \
|
||||
--access-key "$COMPAT_ACCESS_KEY" --secret-key "$COMPAT_SECRET_KEY" \
|
||||
--bucket compat-beta5-sse-s3 --profile 5g --concurrency 16 \
|
||||
--encryption sse-s3
|
||||
```
|
||||
@@ -0,0 +1,88 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0.
|
||||
|
||||
x-minio-env: &minio-env
|
||||
MINIO_ROOT_USER: "${MINIO_ROOT_USER:-minioadmin}"
|
||||
MINIO_ROOT_PASSWORD: "${MINIO_ROOT_PASSWORD:-minioadmin}"
|
||||
MINIO_COMPRESSION_ENABLE: "${MINIO_COMPRESSION_ENABLE:-on}"
|
||||
MINIO_COMPRESSION_ALLOW_ENCRYPTION: "${MINIO_COMPRESSION_ALLOW_ENCRYPTION:-on}"
|
||||
MINIO_COMPRESSION_EXTENSIONS: "${MINIO_COMPRESSION_EXTENSIONS:-.txt,.log,.csv,.json,.tar,.xml,.bin}"
|
||||
MINIO_COMPRESSION_MIME_TYPES: "${MINIO_COMPRESSION_MIME_TYPES:-text/*,application/json,application/xml,binary/octet-stream}"
|
||||
|
||||
x-minio-node: &minio-node
|
||||
image: "${MINIO_IMAGE:-quay.io/minio/minio:latest}"
|
||||
command: server --console-address ":9001" "http://minio{1...4}:9000/data/disk{1...4}"
|
||||
environment: *minio-env
|
||||
networks:
|
||||
- minio-compat-net
|
||||
restart: unless-stopped
|
||||
|
||||
services:
|
||||
minio-permission-helper:
|
||||
image: alpine:3.23
|
||||
command: sh -c "mkdir -p /compat-data && chown -R 1000:1000 /compat-data"
|
||||
volumes:
|
||||
- ./data/minio:/compat-data
|
||||
restart: "no"
|
||||
|
||||
minio1:
|
||||
<<: *minio-node
|
||||
hostname: minio1
|
||||
depends_on:
|
||||
minio-permission-helper:
|
||||
condition: service_completed_successfully
|
||||
volumes:
|
||||
- ./data/minio/node1/disk1:/data/disk1
|
||||
- ./data/minio/node1/disk2:/data/disk2
|
||||
- ./data/minio/node1/disk3:/data/disk3
|
||||
- ./data/minio/node1/disk4:/data/disk4
|
||||
ports:
|
||||
- "${MINIO_API_PORT:-9200}:9000"
|
||||
- "${MINIO_CONSOLE_PORT:-9201}:9001"
|
||||
|
||||
minio2:
|
||||
<<: *minio-node
|
||||
hostname: minio2
|
||||
depends_on:
|
||||
minio-permission-helper:
|
||||
condition: service_completed_successfully
|
||||
volumes:
|
||||
- ./data/minio/node2/disk1:/data/disk1
|
||||
- ./data/minio/node2/disk2:/data/disk2
|
||||
- ./data/minio/node2/disk3:/data/disk3
|
||||
- ./data/minio/node2/disk4:/data/disk4
|
||||
ports:
|
||||
- "${MINIO_NODE2_PORT:-9202}:9000"
|
||||
|
||||
minio3:
|
||||
<<: *minio-node
|
||||
hostname: minio3
|
||||
depends_on:
|
||||
minio-permission-helper:
|
||||
condition: service_completed_successfully
|
||||
volumes:
|
||||
- ./data/minio/node3/disk1:/data/disk1
|
||||
- ./data/minio/node3/disk2:/data/disk2
|
||||
- ./data/minio/node3/disk3:/data/disk3
|
||||
- ./data/minio/node3/disk4:/data/disk4
|
||||
ports:
|
||||
- "${MINIO_NODE3_PORT:-9203}:9000"
|
||||
|
||||
minio4:
|
||||
<<: *minio-node
|
||||
hostname: minio4
|
||||
depends_on:
|
||||
minio-permission-helper:
|
||||
condition: service_completed_successfully
|
||||
volumes:
|
||||
- ./data/minio/node4/disk1:/data/disk1
|
||||
- ./data/minio/node4/disk2:/data/disk2
|
||||
- ./data/minio/node4/disk3:/data/disk3
|
||||
- ./data/minio/node4/disk4:/data/disk4
|
||||
ports:
|
||||
- "${MINIO_NODE4_PORT:-9204}:9000"
|
||||
|
||||
networks:
|
||||
minio-compat-net:
|
||||
driver: bridge
|
||||
@@ -0,0 +1,104 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0.
|
||||
|
||||
x-rustfs-env: &rustfs-env
|
||||
RUSTFS_VOLUMES: "http://rustfs-beta5-node{1...4}:9000/data/disk{1...4}"
|
||||
RUSTFS_ADDRESS: ":9000"
|
||||
RUSTFS_CONSOLE_ADDRESS: ":9001"
|
||||
RUSTFS_CONSOLE_ENABLE: "true"
|
||||
RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS: "*"
|
||||
RUSTFS_ACCESS_KEY: "${RUSTFS_ACCESS_KEY:-rustfsadmin}"
|
||||
RUSTFS_SECRET_KEY: "${RUSTFS_SECRET_KEY:-rustfsadmin}"
|
||||
RUSTFS_OBS_LOGGER_LEVEL: "${RUSTFS_OBS_LOGGER_LEVEL:-info}"
|
||||
RUSTFS_OBS_LOG_DIRECTORY: "/logs"
|
||||
RUSTFS_COMPRESSION_ENABLED: "${RUSTFS_COMPRESSION_ENABLED:-true}"
|
||||
RUSTFS_COMPRESSION_EXTENSIONS: "${RUSTFS_COMPRESSION_EXTENSIONS:-.txt,.log,.csv,.json,.tar,.xml,.bin}"
|
||||
RUSTFS_COMPRESSION_MIME_TYPES: "${RUSTFS_COMPRESSION_MIME_TYPES:-text/*,application/json,application/xml,binary/octet-stream}"
|
||||
RUSTFS_KMS_ENABLE: "${RUSTFS_KMS_ENABLE:-false}"
|
||||
RUSTFS_KMS_BACKEND: "${RUSTFS_KMS_BACKEND:-local}"
|
||||
RUSTFS_KMS_KEY_DIR: "${RUSTFS_KMS_KEY_DIR:-/kms}"
|
||||
RUSTFS_KMS_DEFAULT_KEY_ID: "${RUSTFS_KMS_DEFAULT_KEY_ID:-compat-key}"
|
||||
RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS: "${RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS:-false}"
|
||||
RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}"
|
||||
|
||||
x-rustfs-node: &rustfs-node
|
||||
image: "${RUSTFS_BETA5_IMAGE:-rustfs/rustfs:1.0.0-beta.5}"
|
||||
build:
|
||||
context: ../..
|
||||
dockerfile: Dockerfile
|
||||
args:
|
||||
RELEASE: "1.0.0-beta.5"
|
||||
environment: *rustfs-env
|
||||
depends_on:
|
||||
rustfs-beta5-permission-helper:
|
||||
condition: service_completed_successfully
|
||||
networks:
|
||||
- rustfs-beta5-net
|
||||
restart: unless-stopped
|
||||
|
||||
services:
|
||||
rustfs-beta5-permission-helper:
|
||||
image: alpine:3.23
|
||||
command: sh -c "mkdir -p /compat-data /kms && chown -R 10001:10001 /compat-data /kms"
|
||||
volumes:
|
||||
- ./data/rustfs-beta5:/compat-data
|
||||
- ./kms/rustfs-compat:/kms
|
||||
restart: "no"
|
||||
|
||||
rustfs-beta5-node1:
|
||||
<<: *rustfs-node
|
||||
hostname: rustfs-beta5-node1
|
||||
volumes:
|
||||
- ./data/rustfs-beta5/node1/disk1:/data/disk1
|
||||
- ./data/rustfs-beta5/node1/disk2:/data/disk2
|
||||
- ./data/rustfs-beta5/node1/disk3:/data/disk3
|
||||
- ./data/rustfs-beta5/node1/disk4:/data/disk4
|
||||
- ./data/rustfs-beta5/logs/node1:/logs
|
||||
- ./kms/rustfs-compat:/kms
|
||||
ports:
|
||||
- "${RUSTFS_BETA5_API_PORT:-9100}:9000"
|
||||
- "${RUSTFS_BETA5_CONSOLE_PORT:-9101}:9001"
|
||||
|
||||
rustfs-beta5-node2:
|
||||
<<: *rustfs-node
|
||||
hostname: rustfs-beta5-node2
|
||||
volumes:
|
||||
- ./data/rustfs-beta5/node2/disk1:/data/disk1
|
||||
- ./data/rustfs-beta5/node2/disk2:/data/disk2
|
||||
- ./data/rustfs-beta5/node2/disk3:/data/disk3
|
||||
- ./data/rustfs-beta5/node2/disk4:/data/disk4
|
||||
- ./data/rustfs-beta5/logs/node2:/logs
|
||||
- ./kms/rustfs-compat:/kms
|
||||
ports:
|
||||
- "${RUSTFS_BETA5_NODE2_PORT:-9102}:9000"
|
||||
|
||||
rustfs-beta5-node3:
|
||||
<<: *rustfs-node
|
||||
hostname: rustfs-beta5-node3
|
||||
volumes:
|
||||
- ./data/rustfs-beta5/node3/disk1:/data/disk1
|
||||
- ./data/rustfs-beta5/node3/disk2:/data/disk2
|
||||
- ./data/rustfs-beta5/node3/disk3:/data/disk3
|
||||
- ./data/rustfs-beta5/node3/disk4:/data/disk4
|
||||
- ./data/rustfs-beta5/logs/node3:/logs
|
||||
- ./kms/rustfs-compat:/kms
|
||||
ports:
|
||||
- "${RUSTFS_BETA5_NODE3_PORT:-9103}:9000"
|
||||
|
||||
rustfs-beta5-node4:
|
||||
<<: *rustfs-node
|
||||
hostname: rustfs-beta5-node4
|
||||
volumes:
|
||||
- ./data/rustfs-beta5/node4/disk1:/data/disk1
|
||||
- ./data/rustfs-beta5/node4/disk2:/data/disk2
|
||||
- ./data/rustfs-beta5/node4/disk3:/data/disk3
|
||||
- ./data/rustfs-beta5/node4/disk4:/data/disk4
|
||||
- ./data/rustfs-beta5/logs/node4:/logs
|
||||
- ./kms/rustfs-compat:/kms
|
||||
ports:
|
||||
- "${RUSTFS_BETA5_NODE4_PORT:-9104}:9000"
|
||||
|
||||
networks:
|
||||
rustfs-beta5-net:
|
||||
driver: bridge
|
||||
@@ -0,0 +1,115 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0.
|
||||
|
||||
x-rustfs-rio-v2-env: &rustfs-rio-v2-env
|
||||
RUSTFS_VOLUMES: "http://rustfs-rio-v2-node{1...4}:9000/data/disk{1...4}"
|
||||
RUSTFS_ADDRESS: ":9000"
|
||||
RUSTFS_CONSOLE_ADDRESS: ":9001"
|
||||
RUSTFS_CONSOLE_ENABLE: "true"
|
||||
RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS: "*"
|
||||
RUSTFS_ACCESS_KEY: "${RUSTFS_ACCESS_KEY:-rustfsadmin}"
|
||||
RUSTFS_SECRET_KEY: "${RUSTFS_SECRET_KEY:-rustfsadmin}"
|
||||
RUSTFS_OBS_LOGGER_LEVEL: "${RUSTFS_OBS_LOGGER_LEVEL:-info}"
|
||||
RUSTFS_OBS_LOG_DIRECTORY: "/logs"
|
||||
RUSTFS_COMPRESSION_ENABLED: "${RUSTFS_COMPRESSION_ENABLED:-true}"
|
||||
RUSTFS_COMPRESSION_EXTENSIONS: "${RUSTFS_COMPRESSION_EXTENSIONS:-.txt,.log,.csv,.json,.tar,.xml,.bin}"
|
||||
RUSTFS_COMPRESSION_MIME_TYPES: "${RUSTFS_COMPRESSION_MIME_TYPES:-text/*,application/json,application/xml,binary/octet-stream}"
|
||||
RUSTFS_KMS_ENABLE: "${RUSTFS_KMS_ENABLE:-false}"
|
||||
RUSTFS_KMS_BACKEND: "${RUSTFS_KMS_BACKEND:-local}"
|
||||
RUSTFS_KMS_KEY_DIR: "${RUSTFS_KMS_KEY_DIR:-/kms}"
|
||||
RUSTFS_KMS_DEFAULT_KEY_ID: "${RUSTFS_KMS_DEFAULT_KEY_ID:-compat-key}"
|
||||
RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS: "${RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS:-false}"
|
||||
RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}"
|
||||
|
||||
x-rustfs-rio-v2-node: &rustfs-rio-v2-node
|
||||
image: "${RUSTFS_RIO_V2_IMAGE:-rustfs/rustfs:compat-rio-v2}"
|
||||
entrypoint:
|
||||
- /bin/sh
|
||||
- -c
|
||||
- |
|
||||
until getent hosts rustfs-rio-v2-node1 >/dev/null &&
|
||||
getent hosts rustfs-rio-v2-node2 >/dev/null &&
|
||||
getent hosts rustfs-rio-v2-node3 >/dev/null &&
|
||||
getent hosts rustfs-rio-v2-node4 >/dev/null; do
|
||||
sleep 1
|
||||
done
|
||||
exec /entrypoint.sh /usr/bin/rustfs
|
||||
build:
|
||||
context: ../..
|
||||
dockerfile: Dockerfile.source
|
||||
args:
|
||||
RUSTFS_BUILD_FEATURES: "rio-v2"
|
||||
environment: *rustfs-rio-v2-env
|
||||
depends_on:
|
||||
rustfs-rio-v2-permission-helper:
|
||||
condition: service_completed_successfully
|
||||
networks:
|
||||
- rustfs-rio-v2-net
|
||||
restart: unless-stopped
|
||||
|
||||
services:
|
||||
rustfs-rio-v2-permission-helper:
|
||||
image: alpine:3.23
|
||||
command: sh -c "mkdir -p /compat-data /kms && chown -R 10001:10001 /compat-data /kms"
|
||||
volumes:
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}:/compat-data
|
||||
- ./kms/rustfs-compat:/kms
|
||||
restart: "no"
|
||||
|
||||
rustfs-rio-v2-node1:
|
||||
<<: *rustfs-rio-v2-node
|
||||
hostname: rustfs-rio-v2-node1
|
||||
volumes:
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node1/disk1:/data/disk1
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node1/disk2:/data/disk2
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node1/disk3:/data/disk3
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node1/disk4:/data/disk4
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/logs/node1:/logs
|
||||
- ./kms/rustfs-compat:/kms
|
||||
ports:
|
||||
- "${RUSTFS_RIO_V2_API_PORT:-9300}:9000"
|
||||
- "${RUSTFS_RIO_V2_CONSOLE_PORT:-9301}:9001"
|
||||
|
||||
rustfs-rio-v2-node2:
|
||||
<<: *rustfs-rio-v2-node
|
||||
hostname: rustfs-rio-v2-node2
|
||||
volumes:
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node2/disk1:/data/disk1
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node2/disk2:/data/disk2
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node2/disk3:/data/disk3
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node2/disk4:/data/disk4
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/logs/node2:/logs
|
||||
- ./kms/rustfs-compat:/kms
|
||||
ports:
|
||||
- "${RUSTFS_RIO_V2_NODE2_PORT:-9302}:9000"
|
||||
|
||||
rustfs-rio-v2-node3:
|
||||
<<: *rustfs-rio-v2-node
|
||||
hostname: rustfs-rio-v2-node3
|
||||
volumes:
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node3/disk1:/data/disk1
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node3/disk2:/data/disk2
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node3/disk3:/data/disk3
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node3/disk4:/data/disk4
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/logs/node3:/logs
|
||||
- ./kms/rustfs-compat:/kms
|
||||
ports:
|
||||
- "${RUSTFS_RIO_V2_NODE3_PORT:-9303}:9000"
|
||||
|
||||
rustfs-rio-v2-node4:
|
||||
<<: *rustfs-rio-v2-node
|
||||
hostname: rustfs-rio-v2-node4
|
||||
volumes:
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node4/disk1:/data/disk1
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node4/disk2:/data/disk2
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node4/disk3:/data/disk3
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node4/disk4:/data/disk4
|
||||
- ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/logs/node4:/logs
|
||||
- ./kms/rustfs-compat:/kms
|
||||
ports:
|
||||
- "${RUSTFS_RIO_V2_NODE4_PORT:-9304}:9000"
|
||||
|
||||
networks:
|
||||
rustfs-rio-v2-net:
|
||||
driver: bridge
|
||||
Executable
+639
@@ -0,0 +1,639 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
# High-concurrency S3 read/write stress runner for rio/rio-v2 format compatibility.
|
||||
# Write a manifest on an old endpoint, then verify the same manifest through rio-v2.
|
||||
|
||||
MODE="write"
|
||||
ENDPOINT=""
|
||||
ACCESS_KEY="${AWS_ACCESS_KEY_ID:-}"
|
||||
SECRET_KEY="${AWS_SECRET_ACCESS_KEY:-}"
|
||||
BUCKET="compat-rw-stress"
|
||||
REGION="us-east-1"
|
||||
CONCURRENCY=64
|
||||
OUT_DIR=""
|
||||
WORK_DIR=""
|
||||
MANIFEST=""
|
||||
PROFILE="200g"
|
||||
OBJECT_SPEC=""
|
||||
DATA_PATTERN="compressible"
|
||||
ENCRYPTION="none"
|
||||
SSE_KMS_KEY_ID=""
|
||||
SSE_C_KEY_FILE=""
|
||||
CLIENT="mc"
|
||||
AWS_BIN="${AWS_BIN:-aws}"
|
||||
MC_BIN="${MC_BIN:-}"
|
||||
KEEP_PAYLOADS=false
|
||||
DRY_RUN=false
|
||||
RESUME=false
|
||||
|
||||
usage() {
|
||||
cat <<'USAGE'
|
||||
Usage:
|
||||
.docker/compat/run_rw_compat_stress.sh --mode <write|verify|mixed> \
|
||||
--endpoint <url> --access-key <ak> --secret-key <sk> [options]
|
||||
|
||||
Modes:
|
||||
write Create bucket, upload objects concurrently, and write manifest.csv.
|
||||
verify Read objects concurrently from --endpoint and verify against manifest.csv.
|
||||
mixed Write and verify against the same endpoint.
|
||||
|
||||
Required:
|
||||
--endpoint S3 endpoint URL, for example http://127.0.0.1:9100
|
||||
--access-key S3 access key
|
||||
--secret-key S3 secret key
|
||||
|
||||
Core options:
|
||||
--bucket Bucket name (default: compat-rw-stress)
|
||||
--region Region (default: us-east-1)
|
||||
--concurrency Parallel object operations (default: 64)
|
||||
--out-dir Output directory (default: target/compat/rw-stress-<timestamp>)
|
||||
--work-dir Payload scratch directory (default: <out-dir>/payloads)
|
||||
--manifest Manifest path (default: <out-dir>/manifest.csv for write/mixed)
|
||||
--profile compact | 5g | 200g (default: 200g)
|
||||
--object-spec Override profile. Format: size:count,size:count
|
||||
Example: 1KiB:1024,1MiB:1024,64MiB:512,1GiB:64
|
||||
--data-pattern compressible | random | mixed (default: compressible)
|
||||
--keep-payloads Do not delete local payload files after upload
|
||||
--resume Skip write tasks that already have rows in <out-dir>/tasks/write-rows
|
||||
--dry-run Print planned tasks and commands without executing S3 operations
|
||||
--client mc | aws (default: mc)
|
||||
--mc-bin Path to mc binary (default: first tmp/mc.* or mc in PATH)
|
||||
--aws-bin Path to aws binary (used with --client aws)
|
||||
|
||||
Encryption options:
|
||||
--encryption none | sse-s3 | sse-kms | sse-c (default: none)
|
||||
--sse-kms-key-id KMS key id for --encryption sse-kms
|
||||
--sse-c-key-file Raw 32-byte SSE-C key file for --encryption sse-c
|
||||
|
||||
Examples:
|
||||
# Generate the old RustFS beta5 dataset.
|
||||
.docker/compat/run_rw_compat_stress.sh \
|
||||
--mode write --endpoint http://127.0.0.1:9100 \
|
||||
--access-key rustfsadmin --secret-key rustfsadmin \
|
||||
--bucket compat-beta5 --concurrency 96
|
||||
|
||||
# Verify that dataset after mounting beta5 disks into the rio-v2 compose.
|
||||
.docker/compat/run_rw_compat_stress.sh \
|
||||
--mode verify --endpoint http://127.0.0.1:9300 \
|
||||
--access-key rustfsadmin --secret-key rustfsadmin \
|
||||
--bucket compat-beta5 --concurrency 96 \
|
||||
--manifest target/compat/rw-stress-YYYYmmdd-HHMMSS/manifest.csv
|
||||
|
||||
# Faster smoke run.
|
||||
.docker/compat/run_rw_compat_stress.sh \
|
||||
--mode mixed --endpoint http://127.0.0.1:9300 \
|
||||
--access-key rustfsadmin --secret-key rustfsadmin \
|
||||
--profile compact --concurrency 16
|
||||
USAGE
|
||||
}
|
||||
|
||||
die() {
|
||||
echo "ERROR: $*" >&2
|
||||
exit 1
|
||||
}
|
||||
|
||||
require_cmd() {
|
||||
command -v "$1" >/dev/null 2>&1 || die "command not found: $1"
|
||||
}
|
||||
|
||||
parse_args() {
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--mode) MODE="$2"; shift 2 ;;
|
||||
--endpoint) ENDPOINT="$2"; shift 2 ;;
|
||||
--access-key) ACCESS_KEY="$2"; shift 2 ;;
|
||||
--secret-key) SECRET_KEY="$2"; shift 2 ;;
|
||||
--bucket) BUCKET="$2"; shift 2 ;;
|
||||
--region) REGION="$2"; shift 2 ;;
|
||||
--concurrency) CONCURRENCY="$2"; shift 2 ;;
|
||||
--out-dir) OUT_DIR="$2"; shift 2 ;;
|
||||
--work-dir) WORK_DIR="$2"; shift 2 ;;
|
||||
--manifest) MANIFEST="$2"; shift 2 ;;
|
||||
--profile) PROFILE="$2"; shift 2 ;;
|
||||
--object-spec) OBJECT_SPEC="$2"; shift 2 ;;
|
||||
--data-pattern) DATA_PATTERN="$2"; shift 2 ;;
|
||||
--encryption) ENCRYPTION="$2"; shift 2 ;;
|
||||
--sse-kms-key-id) SSE_KMS_KEY_ID="$2"; shift 2 ;;
|
||||
--sse-c-key-file) SSE_C_KEY_FILE="$2"; shift 2 ;;
|
||||
--client) CLIENT="$2"; shift 2 ;;
|
||||
--mc-bin) MC_BIN="$2"; shift 2 ;;
|
||||
--aws-bin) AWS_BIN="$2"; shift 2 ;;
|
||||
--keep-payloads) KEEP_PAYLOADS=true; shift ;;
|
||||
--resume) RESUME=true; shift ;;
|
||||
--dry-run) DRY_RUN=true; shift ;;
|
||||
-h|--help) usage; exit 0 ;;
|
||||
*) die "unknown arg: $1" ;;
|
||||
esac
|
||||
done
|
||||
}
|
||||
|
||||
validate_args() {
|
||||
[[ "$MODE" =~ ^(write|verify|mixed)$ ]] || die "--mode must be write, verify, or mixed"
|
||||
[[ "$CLIENT" =~ ^(mc|aws)$ ]] || die "--client must be mc or aws"
|
||||
[[ "$PROFILE" =~ ^(compact|5g|200g)$ ]] || die "--profile must be compact, 5g, or 200g"
|
||||
[[ "$DATA_PATTERN" =~ ^(compressible|random|mixed)$ ]] || die "--data-pattern must be compressible, random, or mixed"
|
||||
[[ "$ENCRYPTION" =~ ^(none|sse-s3|sse-kms|sse-c)$ ]] || die "--encryption must be none, sse-s3, sse-kms, or sse-c"
|
||||
[[ -n "$ENDPOINT" && -n "$ACCESS_KEY" && -n "$SECRET_KEY" ]] || die "--endpoint/--access-key/--secret-key are required"
|
||||
[[ "$CONCURRENCY" =~ ^[0-9]+$ && "$CONCURRENCY" -gt 0 ]] || die "--concurrency must be a positive integer"
|
||||
if [[ "$ENCRYPTION" == "sse-kms" && -z "$SSE_KMS_KEY_ID" ]]; then
|
||||
die "--sse-kms-key-id is required for --encryption sse-kms"
|
||||
fi
|
||||
if [[ "$ENCRYPTION" == "sse-c" ]]; then
|
||||
[[ -n "$SSE_C_KEY_FILE" && -f "$SSE_C_KEY_FILE" ]] || die "--sse-c-key-file must point to an existing key file"
|
||||
fi
|
||||
if [[ "$MODE" == "verify" && -z "$MANIFEST" ]]; then
|
||||
die "--manifest is required for --mode verify"
|
||||
fi
|
||||
}
|
||||
|
||||
setup_paths() {
|
||||
if [[ -z "$OUT_DIR" ]]; then
|
||||
OUT_DIR="target/compat/rw-stress-$(date +%Y%m%d-%H%M%S)"
|
||||
fi
|
||||
if [[ -z "$WORK_DIR" ]]; then
|
||||
WORK_DIR="$OUT_DIR/payloads"
|
||||
fi
|
||||
if [[ -z "$MANIFEST" ]]; then
|
||||
MANIFEST="$OUT_DIR/manifest.csv"
|
||||
fi
|
||||
|
||||
mkdir -p "$OUT_DIR" "$WORK_DIR" "$OUT_DIR/tasks" "$OUT_DIR/logs"
|
||||
TASKS_FILE="$OUT_DIR/tasks/tasks.tsv"
|
||||
WRITE_ROWS_DIR="$OUT_DIR/tasks/write-rows"
|
||||
VERIFY_ROWS_DIR="$OUT_DIR/tasks/verify-rows"
|
||||
MC_CONFIG_DIR_LOCAL="$OUT_DIR/mc-config"
|
||||
MC_ALIAS="compat"
|
||||
mkdir -p "$WRITE_ROWS_DIR" "$VERIFY_ROWS_DIR"
|
||||
}
|
||||
|
||||
resolve_mc_bin() {
|
||||
if [[ -n "$MC_BIN" ]]; then
|
||||
echo "$MC_BIN"
|
||||
return
|
||||
fi
|
||||
|
||||
local candidate
|
||||
candidate="$(find tmp -maxdepth 1 -type f -name 'mc.*' -perm -111 2>/dev/null | sort | tail -n 1 || true)"
|
||||
if [[ -n "$candidate" ]]; then
|
||||
echo "$candidate"
|
||||
return
|
||||
fi
|
||||
|
||||
command -v mc 2>/dev/null || true
|
||||
}
|
||||
|
||||
size_to_bytes() {
|
||||
local raw="$1"
|
||||
local num unit
|
||||
if [[ "$raw" =~ ^([0-9]+)(B|KiB|MiB|GiB|KB|MB|GB)?$ ]]; then
|
||||
num="${BASH_REMATCH[1]}"
|
||||
unit="${BASH_REMATCH[2]:-B}"
|
||||
else
|
||||
die "invalid size: $raw"
|
||||
fi
|
||||
|
||||
case "$unit" in
|
||||
B) echo "$num" ;;
|
||||
KiB) echo $((num * 1024)) ;;
|
||||
MiB) echo $((num * 1024 * 1024)) ;;
|
||||
GiB) echo $((num * 1024 * 1024 * 1024)) ;;
|
||||
KB) echo $((num * 1000)) ;;
|
||||
MB) echo $((num * 1000 * 1000)) ;;
|
||||
GB) echo $((num * 1000 * 1000 * 1000)) ;;
|
||||
*) die "invalid size unit: $unit" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
profile_spec() {
|
||||
if [[ -n "$OBJECT_SPEC" ]]; then
|
||||
echo "$OBJECT_SPEC"
|
||||
return
|
||||
fi
|
||||
|
||||
case "$PROFILE" in
|
||||
compact)
|
||||
echo "1KiB:64,1MiB:64,16MiB:16,128MiB:4"
|
||||
;;
|
||||
5g)
|
||||
echo "1KiB:1024,1MiB:255,16MiB:64,64MiB:28,1GiB:2"
|
||||
;;
|
||||
200g)
|
||||
echo "1KiB:1024,1MiB:1024,64MiB:512,1GiB:64,8GiB:12,6GiB:1"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
content_for_index() {
|
||||
local index="$1"
|
||||
case $((index % 3)) in
|
||||
0) echo "txt|text/plain" ;;
|
||||
1) echo "json|application/json" ;;
|
||||
2) echo "bin|binary/octet-stream" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
generate_tasks() {
|
||||
local spec item size count bytes i content ext mime key seed index=0
|
||||
spec="$(profile_spec)"
|
||||
: > "$TASKS_FILE"
|
||||
|
||||
IFS=',' read -r -a items <<< "$spec"
|
||||
for item in "${items[@]}"; do
|
||||
item="${item//[[:space:]]/}"
|
||||
[[ -n "$item" ]] || continue
|
||||
[[ "$item" =~ ^([^:]+):([0-9]+)$ ]] || die "invalid object spec item: $item"
|
||||
size="${BASH_REMATCH[1]}"
|
||||
count="${BASH_REMATCH[2]}"
|
||||
bytes="$(size_to_bytes "$size")"
|
||||
|
||||
for ((i = 1; i <= count; i++)); do
|
||||
content="$(content_for_index "$index")"
|
||||
ext="${content%%|*}"
|
||||
mime="${content#*|}"
|
||||
key="rw-stress/${size}/obj-$(printf '%06d' "$i").${ext}"
|
||||
seed="${BUCKET}:${key}:${bytes}"
|
||||
printf '%s\t%s\t%s\t%s\t%s\n' "$key" "$size" "$bytes" "$mime" "$seed" >> "$TASKS_FILE"
|
||||
index=$((index + 1))
|
||||
done
|
||||
done
|
||||
}
|
||||
|
||||
write_row_file_for_key() {
|
||||
local key="$1"
|
||||
echo "$WRITE_ROWS_DIR/${key//\//_}.csv"
|
||||
}
|
||||
|
||||
prepare_write_input() {
|
||||
WRITE_INPUT="$TASKS_FILE"
|
||||
if [[ "$RESUME" != "true" ]]; then
|
||||
return
|
||||
fi
|
||||
|
||||
WRITE_INPUT="$OUT_DIR/tasks/write-input.tsv"
|
||||
: > "$WRITE_INPUT"
|
||||
|
||||
local line key _size _bytes _mime _seed row_file
|
||||
while IFS= read -r line || [[ -n "$line" ]]; do
|
||||
IFS=$'\t' read -r key _size _bytes _mime _seed <<< "$line"
|
||||
row_file="$(write_row_file_for_key "$key")"
|
||||
[[ -s "$row_file" ]] && continue
|
||||
printf '%s\n' "$line" >> "$WRITE_INPUT"
|
||||
done < "$TASKS_FILE"
|
||||
}
|
||||
|
||||
print_plan() {
|
||||
local total_objects total_bytes profile_label
|
||||
if [[ -f "$TASKS_FILE" ]]; then
|
||||
total_objects="$(wc -l < "$TASKS_FILE" | tr -d ' ')"
|
||||
total_bytes="$(awk -F '\t' '{sum += $3} END {print sum + 0}' "$TASKS_FILE")"
|
||||
profile_label="$(profile_spec)"
|
||||
else
|
||||
total_objects="$(awk 'END {count = NR - 1; if (count < 0) count = 0; print count}' "$MANIFEST")"
|
||||
total_bytes="$(awk -F ',' 'NR > 1 {sum += $3} END {print sum + 0}' "$MANIFEST")"
|
||||
profile_label="from manifest"
|
||||
fi
|
||||
|
||||
cat <<PLAN
|
||||
Mode: $MODE
|
||||
Endpoint: $ENDPOINT
|
||||
Bucket: $BUCKET
|
||||
Profile: $profile_label
|
||||
Objects: $total_objects
|
||||
Bytes: $total_bytes
|
||||
Concurrency: $CONCURRENCY
|
||||
Encryption: $ENCRYPTION
|
||||
Client: $CLIENT
|
||||
Manifest: $MANIFEST
|
||||
Out dir: $OUT_DIR
|
||||
Work dir: $WORK_DIR
|
||||
PLAN
|
||||
}
|
||||
|
||||
aws_base() {
|
||||
AWS_ACCESS_KEY_ID="$ACCESS_KEY" AWS_SECRET_ACCESS_KEY="$SECRET_KEY" AWS_DEFAULT_REGION="$REGION" \
|
||||
"$AWS_BIN" --endpoint-url "$ENDPOINT" "$@"
|
||||
}
|
||||
|
||||
mc_base() {
|
||||
"$MC_BIN" --config-dir "$MC_CONFIG_DIR_LOCAL" "$@"
|
||||
}
|
||||
|
||||
aws_cp_args() {
|
||||
case "$ENCRYPTION" in
|
||||
none) ;;
|
||||
sse-s3) printf '%s\n' "--sse" "AES256" ;;
|
||||
sse-kms) printf '%s\n' "--sse" "aws:kms" "--sse-kms-key-id" "$SSE_KMS_KEY_ID" ;;
|
||||
sse-c) printf '%s\n' "--sse-c" "AES256" "--sse-c-key" "fileb://$SSE_C_KEY_FILE" ;;
|
||||
esac
|
||||
}
|
||||
|
||||
mc_enc_target() {
|
||||
printf '%s/%s/rw-stress/' "$MC_ALIAS" "$BUCKET"
|
||||
}
|
||||
|
||||
mc_cp_args() {
|
||||
local target
|
||||
target="$(mc_enc_target)"
|
||||
case "$ENCRYPTION" in
|
||||
none) ;;
|
||||
sse-s3) printf '%s\n' "--enc-s3" "$target" ;;
|
||||
sse-kms) printf '%s\n' "--enc-kms" "${target}=${SSE_KMS_KEY_ID}" ;;
|
||||
sse-c)
|
||||
local key_b64
|
||||
key_b64="$(base64 < "$SSE_C_KEY_FILE" | tr -d '\n')"
|
||||
printf '%s\n' "--enc-c" "${target}=${key_b64}"
|
||||
;;
|
||||
esac
|
||||
}
|
||||
|
||||
aws_get_args() {
|
||||
if [[ "$ENCRYPTION" == "sse-c" ]]; then
|
||||
printf '%s\n' "--sse-c" "AES256" "--sse-c-key" "fileb://$SSE_C_KEY_FILE"
|
||||
fi
|
||||
}
|
||||
|
||||
mc_get_args() {
|
||||
if [[ "$ENCRYPTION" == "sse-c" ]]; then
|
||||
local target key_b64
|
||||
target="$(mc_enc_target)"
|
||||
key_b64="$(base64 < "$SSE_C_KEY_FILE" | tr -d '\n')"
|
||||
printf '%s\n' "--enc-c" "${target}=${key_b64}"
|
||||
fi
|
||||
}
|
||||
|
||||
setup_mc_alias() {
|
||||
if [[ "$CLIENT" != "mc" || "$DRY_RUN" == "true" ]]; then
|
||||
return
|
||||
fi
|
||||
mkdir -p "$MC_CONFIG_DIR_LOCAL"
|
||||
mc_base alias set "$MC_ALIAS" "$ENDPOINT" "$ACCESS_KEY" "$SECRET_KEY" --api S3v4 --path auto >/dev/null
|
||||
}
|
||||
|
||||
create_bucket_if_needed() {
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
echo "[DRY-RUN] create bucket if missing: $BUCKET"
|
||||
return
|
||||
fi
|
||||
|
||||
if [[ "$CLIENT" == "mc" ]]; then
|
||||
mc_base mb --ignore-existing --region "$REGION" "$MC_ALIAS/$BUCKET" >/dev/null
|
||||
return
|
||||
fi
|
||||
|
||||
if aws_base s3api head-bucket --bucket "$BUCKET" >/dev/null 2>&1; then
|
||||
return
|
||||
fi
|
||||
|
||||
aws_base s3api create-bucket --bucket "$BUCKET" >/dev/null
|
||||
}
|
||||
|
||||
payload_path_for_key() {
|
||||
local key="$1"
|
||||
echo "$WORK_DIR/${key//\//_}"
|
||||
}
|
||||
|
||||
generate_payload() {
|
||||
local file="$1"
|
||||
local bytes="$2"
|
||||
local seed="$3"
|
||||
local pattern="$DATA_PATTERN"
|
||||
mkdir -p "$(dirname "$file")"
|
||||
|
||||
if [[ "$pattern" == "mixed" ]]; then
|
||||
if [[ $((bytes % 2)) -eq 0 ]]; then
|
||||
pattern="compressible"
|
||||
else
|
||||
pattern="random"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "$pattern" == "random" ]]; then
|
||||
head -c "$bytes" /dev/zero | openssl enc -aes-256-ctr -nosalt -pass "pass:$seed" -out "$file"
|
||||
else
|
||||
yes "$seed payload-for-rio-compatibility" | tr '\n' ' ' | head -c "$bytes" > "$file"
|
||||
fi
|
||||
}
|
||||
|
||||
sha256_file() {
|
||||
shasum -a 256 "$1" | awk '{print $1}'
|
||||
}
|
||||
|
||||
sha256_object() {
|
||||
local key="$1"
|
||||
shift
|
||||
if [[ "$CLIENT" == "mc" ]]; then
|
||||
mc_base cat "$@" "$MC_ALIAS/$BUCKET/$key" | shasum -a 256 | awk '{print $1}'
|
||||
else
|
||||
aws_base s3 cp "s3://$BUCKET/$key" - --no-progress "$@" | shasum -a 256 | awk '{print $1}'
|
||||
fi
|
||||
}
|
||||
|
||||
collect_args() {
|
||||
local generator="$1"
|
||||
extra_args=()
|
||||
while IFS= read -r arg; do
|
||||
extra_args+=("$arg")
|
||||
done < <("$generator")
|
||||
}
|
||||
|
||||
write_one() {
|
||||
local line="$1"
|
||||
local key size bytes mime seed file sha row_file
|
||||
local -a extra_args
|
||||
IFS=$'\t' read -r key size bytes mime seed <<< "$line"
|
||||
file="$(payload_path_for_key "$key")"
|
||||
row_file="$(write_row_file_for_key "$key")"
|
||||
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
echo "[DRY-RUN] upload $bytes bytes to s3://$BUCKET/$key content-type=$mime"
|
||||
printf '%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "DRY_RUN" "$ENCRYPTION" > "$row_file"
|
||||
return
|
||||
fi
|
||||
|
||||
generate_payload "$file" "$bytes" "$seed"
|
||||
sha="$(sha256_file "$file")"
|
||||
|
||||
if [[ "$CLIENT" == "mc" ]]; then
|
||||
collect_args mc_cp_args
|
||||
mc_base cp --quiet --attr "Content-Type=$mime" ${extra_args[@]+"${extra_args[@]}"} "$file" "$MC_ALIAS/$BUCKET/$key" \
|
||||
> "$OUT_DIR/logs/${key//\//_}.put.log" 2>&1
|
||||
else
|
||||
collect_args aws_cp_args
|
||||
aws_base s3 cp "$file" "s3://$BUCKET/$key" --no-progress --content-type "$mime" ${extra_args[@]+"${extra_args[@]}"} \
|
||||
> "$OUT_DIR/logs/${key//\//_}.put.log" 2>&1
|
||||
fi
|
||||
|
||||
printf '%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "$sha" "$ENCRYPTION" > "$row_file"
|
||||
if [[ "$KEEP_PAYLOADS" != "true" ]]; then
|
||||
rm -f "$file"
|
||||
fi
|
||||
}
|
||||
|
||||
verify_one() {
|
||||
local line="$1"
|
||||
local key size bytes mime expected encryption actual row_file
|
||||
local -a extra_args
|
||||
IFS=',' read -r key size bytes mime expected encryption <<< "$line"
|
||||
row_file="$VERIFY_ROWS_DIR/${key//\//_}.csv"
|
||||
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
echo "[DRY-RUN] verify s3://$BUCKET/$key expected=$expected"
|
||||
printf '%s,%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "$expected" "DRY_RUN" "dry-run" > "$row_file"
|
||||
return
|
||||
fi
|
||||
|
||||
if [[ "$CLIENT" == "mc" ]]; then
|
||||
collect_args mc_get_args
|
||||
else
|
||||
collect_args aws_get_args
|
||||
fi
|
||||
if ! actual="$(sha256_object "$key" ${extra_args[@]+"${extra_args[@]}"})"; then
|
||||
printf '%s,%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "$expected" "ERROR" "download failed" > "$row_file"
|
||||
return 1
|
||||
fi
|
||||
|
||||
if [[ "$actual" == "$expected" ]]; then
|
||||
printf '%s,%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "$expected" "$actual" "ok" > "$row_file"
|
||||
else
|
||||
printf '%s,%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "$expected" "$actual" "sha256-mismatch" > "$row_file"
|
||||
return 1
|
||||
fi
|
||||
}
|
||||
|
||||
run_parallel_tasks() {
|
||||
local action="$1"
|
||||
local input_file="$2"
|
||||
local failure_file="$OUT_DIR/tasks/${action}.failed"
|
||||
local line
|
||||
rm -f "$failure_file"
|
||||
|
||||
while IFS= read -r line || [[ -n "$line" ]]; do
|
||||
while [[ "$(jobs -rp | wc -l | tr -d ' ')" -ge "$CONCURRENCY" ]]; do
|
||||
sleep 0.2
|
||||
done
|
||||
|
||||
({
|
||||
if [[ "$action" == "write" ]]; then
|
||||
write_one "$line"
|
||||
else
|
||||
verify_one "$line"
|
||||
fi
|
||||
} || touch "$failure_file") &
|
||||
done < "$input_file"
|
||||
|
||||
wait
|
||||
[[ ! -f "$failure_file" ]]
|
||||
}
|
||||
|
||||
combine_write_manifest() {
|
||||
echo "key,size,bytes,content_type,sha256,encryption" > "$MANIFEST"
|
||||
find "$WRITE_ROWS_DIR" -type f -name '*.csv' -print0 | sort -z | xargs -0 cat >> "$MANIFEST"
|
||||
|
||||
local expected actual
|
||||
expected="$(wc -l < "$TASKS_FILE" | tr -d ' ')"
|
||||
actual="$(( $(wc -l < "$MANIFEST" | tr -d ' ') - 1 ))"
|
||||
if [[ "$actual" -ne "$expected" ]]; then
|
||||
die "manifest row count mismatch: expected $expected, got $actual"
|
||||
fi
|
||||
}
|
||||
|
||||
prepare_verify_input() {
|
||||
VERIFY_INPUT="$OUT_DIR/tasks/verify-input.csv"
|
||||
tail -n +2 "$MANIFEST" > "$VERIFY_INPUT"
|
||||
}
|
||||
|
||||
combine_verify_summary() {
|
||||
VERIFY_SUMMARY="$OUT_DIR/verify-summary.csv"
|
||||
echo "key,size,bytes,content_type,expected_sha256,actual_sha256,status" > "$VERIFY_SUMMARY"
|
||||
find "$VERIFY_ROWS_DIR" -type f -name '*.csv' -print0 | sort -z | xargs -0 cat >> "$VERIFY_SUMMARY"
|
||||
|
||||
local failed
|
||||
if [[ "$DRY_RUN" == "true" ]]; then
|
||||
echo "Verification dry run complete. Summary: $VERIFY_SUMMARY"
|
||||
return
|
||||
fi
|
||||
|
||||
failed="$(awk -F ',' 'NR > 1 && $7 != "ok" {count++} END {print count + 0}' "$VERIFY_SUMMARY")"
|
||||
local expected actual
|
||||
expected="$(wc -l < "$VERIFY_INPUT" | tr -d ' ')"
|
||||
actual="$(( $(wc -l < "$VERIFY_SUMMARY" | tr -d ' ') - 1 ))"
|
||||
if [[ "$actual" -ne "$expected" ]]; then
|
||||
echo "Verification row count mismatch: expected $expected, got $actual" >&2
|
||||
return 1
|
||||
fi
|
||||
if [[ "$failed" -ne 0 ]]; then
|
||||
echo "Verification failed: $failed object(s). See $VERIFY_SUMMARY" >&2
|
||||
return 1
|
||||
fi
|
||||
echo "Verification passed. Summary: $VERIFY_SUMMARY"
|
||||
}
|
||||
|
||||
export_functions() {
|
||||
export ENDPOINT ACCESS_KEY SECRET_KEY BUCKET REGION ENCRYPTION SSE_KMS_KEY_ID SSE_C_KEY_FILE AWS_BIN
|
||||
export CLIENT MC_BIN MC_CONFIG_DIR_LOCAL MC_ALIAS
|
||||
export WORK_DIR OUT_DIR WRITE_ROWS_DIR VERIFY_ROWS_DIR DATA_PATTERN KEEP_PAYLOADS DRY_RUN
|
||||
export -f aws_base mc_base aws_cp_args aws_get_args mc_enc_target mc_cp_args mc_get_args payload_path_for_key generate_payload sha256_file sha256_object write_one verify_one
|
||||
}
|
||||
|
||||
main() {
|
||||
parse_args "$@"
|
||||
validate_args
|
||||
setup_paths
|
||||
|
||||
if [[ "$DRY_RUN" != "true" ]]; then
|
||||
if [[ "$CLIENT" == "mc" ]]; then
|
||||
MC_BIN="$(resolve_mc_bin)"
|
||||
[[ -n "$MC_BIN" ]] || die "mc binary not found; pass --mc-bin or put mc in PATH"
|
||||
require_cmd "$MC_BIN"
|
||||
else
|
||||
require_cmd "$AWS_BIN"
|
||||
fi
|
||||
require_cmd shasum
|
||||
require_cmd head
|
||||
require_cmd yes
|
||||
require_cmd openssl
|
||||
elif [[ "$CLIENT" == "mc" ]]; then
|
||||
MC_BIN="$(resolve_mc_bin)"
|
||||
fi
|
||||
|
||||
setup_mc_alias
|
||||
|
||||
if [[ "$MODE" == "write" || "$MODE" == "mixed" ]]; then
|
||||
local write_failed=false
|
||||
generate_tasks
|
||||
prepare_write_input
|
||||
print_plan
|
||||
create_bucket_if_needed
|
||||
export_functions
|
||||
if ! run_parallel_tasks write "$WRITE_INPUT"; then
|
||||
write_failed=true
|
||||
fi
|
||||
combine_write_manifest
|
||||
echo "Write manifest: $MANIFEST"
|
||||
if [[ "$write_failed" == "true" ]]; then
|
||||
die "one or more write tasks failed; see $OUT_DIR/logs"
|
||||
fi
|
||||
fi
|
||||
|
||||
if [[ "$MODE" == "verify" || "$MODE" == "mixed" ]]; then
|
||||
local verify_failed=false
|
||||
[[ -f "$MANIFEST" ]] || die "manifest not found: $MANIFEST"
|
||||
if [[ "$MODE" == "verify" ]]; then
|
||||
cp "$MANIFEST" "$OUT_DIR/manifest.csv"
|
||||
MANIFEST="$OUT_DIR/manifest.csv"
|
||||
fi
|
||||
prepare_verify_input
|
||||
print_plan
|
||||
export_functions
|
||||
if ! run_parallel_tasks verify "$VERIFY_INPUT"; then
|
||||
verify_failed=true
|
||||
fi
|
||||
combine_verify_summary
|
||||
if [[ "$verify_failed" == "true" ]]; then
|
||||
die "one or more verify tasks failed; see $VERIFY_SUMMARY"
|
||||
fi
|
||||
fi
|
||||
}
|
||||
|
||||
main "$@"
|
||||
@@ -0,0 +1,183 @@
|
||||
# Site Replication Docker Compose Test
|
||||
|
||||
## Purpose
|
||||
|
||||
This directory contains a local three-site RustFS site replication check. It is intended to verify the admin site-replication flow against real containers:
|
||||
|
||||
- three independent RustFS sites start successfully
|
||||
- the MinIO-compatible `mc admin replicate add` command configures all three sites
|
||||
- a bucket and object written to site 1 are replicated to site 2 and site 3
|
||||
- `mc admin replicate status` can read the resulting site-replication state
|
||||
|
||||
The compose file uses named volumes so the test does not require preparing host bind-mount directories.
|
||||
Because Docker named volumes usually share the same physical device in local desktop environments, this test compose defaults `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true`. Keep that setting limited to local test and CI environments.
|
||||
|
||||
## Files
|
||||
|
||||
- `docker-compose.yml`: three RustFS sites, a volume permission helper, and a one-shot setup/check container
|
||||
- `run-object-flow-check.sh`: host-side upload/download verification for replicated 10 MiB to 100 MiB objects
|
||||
|
||||
## Ports
|
||||
|
||||
Default host endpoints:
|
||||
|
||||
- Site 1 API: `http://127.0.0.1:9000`
|
||||
- Site 1 Console: `http://127.0.0.1:9001`
|
||||
- Site 2 API: `http://127.0.0.1:9010`
|
||||
- Site 2 Console: `http://127.0.0.1:9011`
|
||||
- Site 3 API: `http://127.0.0.1:9020`
|
||||
- Site 3 Console: `http://127.0.0.1:9021`
|
||||
|
||||
Default credentials are `rustfsadmin` / `rustfsadmin`. These are for local testing only.
|
||||
|
||||
## Run
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml up
|
||||
```
|
||||
|
||||
To test a locally built image instead of Docker Hub `rustfs/rustfs:latest`, set `RUSTFS_SITE_REPL_IMAGE`:
|
||||
|
||||
```bash
|
||||
docker build -f Dockerfile.source -t rustfs-site-repl-local:latest .
|
||||
RUSTFS_SITE_REPL_IMAGE=rustfs-site-repl-local:latest \
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml up
|
||||
```
|
||||
|
||||
For a detached run:
|
||||
|
||||
```bash
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml up -d
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml logs -f site-replication-setup
|
||||
```
|
||||
|
||||
## Test Flow
|
||||
|
||||
The compose stack performs these steps:
|
||||
|
||||
1. `site-replication-volume-permission-helper` fixes ownership on all named volumes for the RustFS runtime user.
|
||||
2. `rustfs-site1`, `rustfs-site2`, and `rustfs-site3` start as separate RustFS sites.
|
||||
3. Each site exposes its S3 API and Console on a unique host port.
|
||||
4. Health checks wait for `/health/ready` on each RustFS container.
|
||||
5. `site-replication-setup` configures `mc` aliases for all three sites.
|
||||
6. The setup container waits until `mc admin info` succeeds for all sites.
|
||||
7. It runs:
|
||||
|
||||
```bash
|
||||
mc admin replicate add site1 site2 site3
|
||||
```
|
||||
|
||||
8. It creates the test bucket on site 1 and uploads `from-site1.txt`.
|
||||
9. It polls site 2 and site 3 until the replicated object is visible.
|
||||
10. It prints `mc admin replicate status site1`.
|
||||
|
||||
The setup container exits with status `0` only after the object replication check passes.
|
||||
|
||||
## Object Flow Check
|
||||
|
||||
After the compose setup succeeds, run the larger object flow check from the repository root:
|
||||
|
||||
```bash
|
||||
.docker/test/site-replication/run-object-flow-check.sh
|
||||
```
|
||||
|
||||
The script creates five local files and uploads them from different sites:
|
||||
|
||||
- 10 MiB from site 1
|
||||
- 25 MiB from site 2
|
||||
- 50 MiB from site 3
|
||||
- 75 MiB from site 1
|
||||
- 100 MiB from site 2
|
||||
|
||||
For each uploaded object, the script waits for replication to the other two sites, downloads the object from those sites, and verifies both byte size and SHA-256 checksum. It uses a temporary `mc` config directory, so it does not overwrite existing host aliases.
|
||||
|
||||
The default bucket is `site-repl-flow-check`. Override it when needed:
|
||||
|
||||
```bash
|
||||
RUSTFS_SITE_REPL_FLOW_BUCKET='site-repl-large-flow' \
|
||||
.docker/test/site-replication/run-object-flow-check.sh
|
||||
```
|
||||
|
||||
The script keeps uploaded objects under a timestamped prefix. Override the prefix for repeatable runs:
|
||||
|
||||
```bash
|
||||
RUSTFS_SITE_REPL_FLOW_PREFIX='manual-check-001' \
|
||||
.docker/test/site-replication/run-object-flow-check.sh
|
||||
```
|
||||
|
||||
If replication is slow on the local machine, increase polling:
|
||||
|
||||
```bash
|
||||
RUSTFS_SITE_REPL_WAIT_ATTEMPTS=180 \
|
||||
RUSTFS_SITE_REPL_WAIT_SLEEP_SECONDS=2 \
|
||||
.docker/test/site-replication/run-object-flow-check.sh
|
||||
```
|
||||
|
||||
## Optional Settings
|
||||
|
||||
Override local test credentials:
|
||||
|
||||
```bash
|
||||
RUSTFS_SITE_REPL_ACCESS_KEY='localadmin' \
|
||||
RUSTFS_SITE_REPL_SECRET_KEY='localadmin-secret' \
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml up
|
||||
```
|
||||
|
||||
Use a different test bucket:
|
||||
|
||||
```bash
|
||||
RUSTFS_SITE_REPL_BUCKET='site-repl-check' \
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml up
|
||||
```
|
||||
|
||||
Enable ILM expiry rule replication during site setup:
|
||||
|
||||
```bash
|
||||
RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY=true \
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml up
|
||||
```
|
||||
|
||||
Use this only when the test needs lifecycle expiry metadata included in site replication.
|
||||
|
||||
## Manual Checks
|
||||
|
||||
After the setup container succeeds, you can inspect the sites with `mc` from the host:
|
||||
|
||||
```bash
|
||||
mc alias set site1 http://127.0.0.1:9000 rustfsadmin rustfsadmin
|
||||
mc alias set site2 http://127.0.0.1:9010 rustfsadmin rustfsadmin
|
||||
mc alias set site3 http://127.0.0.1:9020 rustfsadmin rustfsadmin
|
||||
|
||||
mc admin replicate info site1
|
||||
mc admin replicate status site1
|
||||
mc stat site2/site-repl-demo/from-site1.txt
|
||||
mc stat site3/site-repl-demo/from-site1.txt
|
||||
```
|
||||
|
||||
After larger object flow checks, replication should converge without a growing queue:
|
||||
|
||||
```bash
|
||||
mc admin replicate status site1
|
||||
mc admin replicate status site2
|
||||
mc admin replicate status site3
|
||||
```
|
||||
|
||||
Useful Docker commands:
|
||||
|
||||
```bash
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml ps
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml logs --no-color --tail=200
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml logs --no-color site-replication-setup
|
||||
```
|
||||
|
||||
## Cleanup
|
||||
|
||||
Remove containers and named volumes:
|
||||
|
||||
```bash
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml down -v
|
||||
```
|
||||
|
||||
Use `down -v` before rerunning the full setup from scratch. Site replication state is persisted in the named volumes, so rerunning without deleting volumes may attempt to add an already-configured replication topology.
|
||||
@@ -0,0 +1,243 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
services:
|
||||
rustfs-site1:
|
||||
image: ${RUSTFS_SITE_REPL_IMAGE:-rustfs/rustfs:latest}
|
||||
container_name: rustfs-site-repl-1
|
||||
security_opt:
|
||||
- "no-new-privileges:true"
|
||||
ports:
|
||||
- "9000:9000"
|
||||
- "9001:9001"
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=/data/rustfs{0...3}
|
||||
- RUSTFS_ADDRESS=0.0.0.0:9000
|
||||
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
|
||||
- RUSTFS_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}
|
||||
- RUSTFS_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_SITE_REPL_LOG_LEVEL:-info}
|
||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
||||
volumes:
|
||||
- site1_data_0:/data/rustfs0
|
||||
- site1_data_1:/data/rustfs1
|
||||
- site1_data_2:/data/rustfs2
|
||||
- site1_data_3:/data/rustfs3
|
||||
- site1_logs:/app/logs
|
||||
networks:
|
||||
- rustfs-site-replication
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health/ready"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 30
|
||||
start_period: 20s
|
||||
depends_on:
|
||||
site-replication-volume-permission-helper:
|
||||
condition: service_completed_successfully
|
||||
|
||||
rustfs-site2:
|
||||
image: ${RUSTFS_SITE_REPL_IMAGE:-rustfs/rustfs:latest}
|
||||
container_name: rustfs-site-repl-2
|
||||
security_opt:
|
||||
- "no-new-privileges:true"
|
||||
ports:
|
||||
- "9010:9000"
|
||||
- "9011:9001"
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=/data/rustfs{0...3}
|
||||
- RUSTFS_ADDRESS=0.0.0.0:9000
|
||||
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
|
||||
- RUSTFS_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}
|
||||
- RUSTFS_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_SITE_REPL_LOG_LEVEL:-info}
|
||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
||||
volumes:
|
||||
- site2_data_0:/data/rustfs0
|
||||
- site2_data_1:/data/rustfs1
|
||||
- site2_data_2:/data/rustfs2
|
||||
- site2_data_3:/data/rustfs3
|
||||
- site2_logs:/app/logs
|
||||
networks:
|
||||
- rustfs-site-replication
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health/ready"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 30
|
||||
start_period: 20s
|
||||
depends_on:
|
||||
site-replication-volume-permission-helper:
|
||||
condition: service_completed_successfully
|
||||
|
||||
rustfs-site3:
|
||||
image: ${RUSTFS_SITE_REPL_IMAGE:-rustfs/rustfs:latest}
|
||||
container_name: rustfs-site-repl-3
|
||||
security_opt:
|
||||
- "no-new-privileges:true"
|
||||
ports:
|
||||
- "9020:9000"
|
||||
- "9021:9001"
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=/data/rustfs{0...3}
|
||||
- RUSTFS_ADDRESS=0.0.0.0:9000
|
||||
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
|
||||
- RUSTFS_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}
|
||||
- RUSTFS_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_SITE_REPL_LOG_LEVEL:-info}
|
||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
||||
volumes:
|
||||
- site3_data_0:/data/rustfs0
|
||||
- site3_data_1:/data/rustfs1
|
||||
- site3_data_2:/data/rustfs2
|
||||
- site3_data_3:/data/rustfs3
|
||||
- site3_logs:/app/logs
|
||||
networks:
|
||||
- rustfs-site-replication
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health/ready"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 30
|
||||
start_period: 20s
|
||||
depends_on:
|
||||
site-replication-volume-permission-helper:
|
||||
condition: service_completed_successfully
|
||||
|
||||
site-replication-setup:
|
||||
image: minio/mc:latest
|
||||
container_name: rustfs-site-repl-setup
|
||||
depends_on:
|
||||
rustfs-site1:
|
||||
condition: service_healthy
|
||||
rustfs-site2:
|
||||
condition: service_healthy
|
||||
rustfs-site3:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
- RUSTFS_SITE_REPL_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}
|
||||
- RUSTFS_SITE_REPL_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}
|
||||
- RUSTFS_SITE_REPL_BUCKET=${RUSTFS_SITE_REPL_BUCKET:-site-repl-demo}
|
||||
- RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY=${RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY:-false}
|
||||
networks:
|
||||
- rustfs-site-replication
|
||||
entrypoint: ["/bin/sh", "-c"]
|
||||
command:
|
||||
- |
|
||||
set -eu
|
||||
|
||||
access_key="$${RUSTFS_SITE_REPL_ACCESS_KEY}"
|
||||
secret_key="$${RUSTFS_SITE_REPL_SECRET_KEY}"
|
||||
bucket="$${RUSTFS_SITE_REPL_BUCKET}"
|
||||
|
||||
mc alias set site1 http://rustfs-site1:9000 "$${access_key}" "$${secret_key}"
|
||||
mc alias set site2 http://rustfs-site2:9000 "$${access_key}" "$${secret_key}"
|
||||
mc alias set site3 http://rustfs-site3:9000 "$${access_key}" "$${secret_key}"
|
||||
|
||||
for site in site1 site2 site3; do
|
||||
until mc ls "$${site}" >/dev/null 2>&1; do
|
||||
echo "waiting for $${site} S3 API"
|
||||
sleep 2
|
||||
done
|
||||
done
|
||||
|
||||
ilm_flag=""
|
||||
if [ "$${RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY}" = "true" ]; then
|
||||
ilm_flag="--replicate-ilm-expiry"
|
||||
fi
|
||||
|
||||
echo "configuring 3-site replication"
|
||||
if ! mc admin replicate add site1 site2 site3 $${ilm_flag}; then
|
||||
echo "replicate add failed; showing current replication info before exiting"
|
||||
mc admin replicate info site1 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mc mb --ignore-existing "site1/$${bucket}"
|
||||
printf 'rustfs 3-site replication check\n' > /tmp/site-replication-check.txt
|
||||
mc cp /tmp/site-replication-check.txt "site1/$${bucket}/from-site1.txt"
|
||||
|
||||
for site in site2 site3; do
|
||||
for attempt in $$(seq 1 60); do
|
||||
if mc stat "$${site}/$${bucket}/from-site1.txt" >/dev/null 2>&1; then
|
||||
echo "$${site} received replicated object"
|
||||
break
|
||||
fi
|
||||
if [ "$${attempt}" = "60" ]; then
|
||||
echo "$${site} did not receive replicated object in time"
|
||||
exit 1
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
done
|
||||
|
||||
mc admin replicate status site1
|
||||
echo "3-site replication example is ready"
|
||||
restart: "no"
|
||||
|
||||
site-replication-volume-permission-helper:
|
||||
image: alpine:3.23.4
|
||||
volumes:
|
||||
- site1_data_0:/site1/data0
|
||||
- site1_data_1:/site1/data1
|
||||
- site1_data_2:/site1/data2
|
||||
- site1_data_3:/site1/data3
|
||||
- site1_logs:/site1/logs
|
||||
- site2_data_0:/site2/data0
|
||||
- site2_data_1:/site2/data1
|
||||
- site2_data_2:/site2/data2
|
||||
- site2_data_3:/site2/data3
|
||||
- site2_logs:/site2/logs
|
||||
- site3_data_0:/site3/data0
|
||||
- site3_data_1:/site3/data1
|
||||
- site3_data_2:/site3/data2
|
||||
- site3_data_3:/site3/data3
|
||||
- site3_logs:/site3/logs
|
||||
command: >
|
||||
sh -c "
|
||||
chown -R 10001:10001 /site1 /site2 /site3 &&
|
||||
echo 'site replication volume permissions fixed'
|
||||
"
|
||||
networks:
|
||||
- rustfs-site-replication
|
||||
restart: "no"
|
||||
|
||||
networks:
|
||||
rustfs-site-replication:
|
||||
|
||||
volumes:
|
||||
site1_data_0:
|
||||
site1_data_1:
|
||||
site1_data_2:
|
||||
site1_data_3:
|
||||
site1_logs:
|
||||
site2_data_0:
|
||||
site2_data_1:
|
||||
site2_data_2:
|
||||
site2_data_3:
|
||||
site2_logs:
|
||||
site3_data_0:
|
||||
site3_data_1:
|
||||
site3_data_2:
|
||||
site3_data_3:
|
||||
site3_logs:
|
||||
+191
@@ -0,0 +1,191 @@
|
||||
#!/bin/sh
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
set -eu
|
||||
|
||||
ACCESS_KEY="${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}"
|
||||
SECRET_KEY="${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}"
|
||||
BUCKET="${RUSTFS_SITE_REPL_FLOW_BUCKET:-site-repl-flow-check}"
|
||||
PREFIX="${RUSTFS_SITE_REPL_FLOW_PREFIX:-flow-$(date +%Y%m%d-%H%M%S)}"
|
||||
WAIT_ATTEMPTS="${RUSTFS_SITE_REPL_WAIT_ATTEMPTS:-90}"
|
||||
WAIT_SLEEP_SECONDS="${RUSTFS_SITE_REPL_WAIT_SLEEP_SECONDS:-2}"
|
||||
|
||||
SITE1_ENDPOINT="${RUSTFS_SITE1_ENDPOINT:-http://127.0.0.1:9000}"
|
||||
SITE2_ENDPOINT="${RUSTFS_SITE2_ENDPOINT:-http://127.0.0.1:9010}"
|
||||
SITE3_ENDPOINT="${RUSTFS_SITE3_ENDPOINT:-http://127.0.0.1:9020}"
|
||||
|
||||
command_exists() {
|
||||
command -v "$1" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
require_command() {
|
||||
if ! command_exists "$1"; then
|
||||
echo "missing required command: $1" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
checksum_file() {
|
||||
if command_exists sha256sum; then
|
||||
sha256sum "$1" | awk '{print $1}'
|
||||
elif command_exists shasum; then
|
||||
shasum -a 256 "$1" | awk '{print $1}'
|
||||
else
|
||||
echo "missing required command: sha256sum or shasum" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
site_endpoint() {
|
||||
case "$1" in
|
||||
site1) printf '%s\n' "$SITE1_ENDPOINT" ;;
|
||||
site2) printf '%s\n' "$SITE2_ENDPOINT" ;;
|
||||
site3) printf '%s\n' "$SITE3_ENDPOINT" ;;
|
||||
*) echo "unknown site alias: $1" >&2; exit 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
other_sites() {
|
||||
case "$1" in
|
||||
site1) printf '%s\n' "site2 site3" ;;
|
||||
site2) printf '%s\n' "site1 site3" ;;
|
||||
site3) printf '%s\n' "site1 site2" ;;
|
||||
*) echo "unknown source site: $1" >&2; exit 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
wait_for_object() {
|
||||
site="$1"
|
||||
object="$2"
|
||||
attempt=1
|
||||
|
||||
while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
|
||||
if mc stat "$site/$BUCKET/$object" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
sleep "$WAIT_SLEEP_SECONDS"
|
||||
attempt=$((attempt + 1))
|
||||
done
|
||||
|
||||
echo "object was not replicated in time: $site/$BUCKET/$object" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
wait_for_bucket() {
|
||||
site="$1"
|
||||
attempt=1
|
||||
|
||||
while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
|
||||
if mc stat "$site/$BUCKET" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
sleep "$WAIT_SLEEP_SECONDS"
|
||||
attempt=$((attempt + 1))
|
||||
done
|
||||
|
||||
echo "bucket was not replicated in time: $site/$BUCKET" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
require_command mc
|
||||
require_command dd
|
||||
require_command awk
|
||||
require_command date
|
||||
require_command mktemp
|
||||
require_command tr
|
||||
require_command wc
|
||||
|
||||
WORK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/rustfs-site-repl-flow.XXXXXX")"
|
||||
MC_CONFIG_DIR="$WORK_DIR/mc"
|
||||
export MC_CONFIG_DIR
|
||||
|
||||
cleanup() {
|
||||
rm -rf "$WORK_DIR"
|
||||
}
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
mkdir -p "$MC_CONFIG_DIR" "$WORK_DIR/src" "$WORK_DIR/downloads"
|
||||
|
||||
for site in site1 site2 site3; do
|
||||
mc alias set "$site" "$(site_endpoint "$site")" "$ACCESS_KEY" "$SECRET_KEY" >/dev/null
|
||||
done
|
||||
|
||||
for site in site1 site2 site3; do
|
||||
echo "checking S3 API for $site"
|
||||
mc ls "$site" >/dev/null
|
||||
done
|
||||
|
||||
echo "ensuring bucket exists on site1: $BUCKET"
|
||||
mc mb --ignore-existing "site1/$BUCKET" >/dev/null
|
||||
|
||||
for site in site1 site2 site3; do
|
||||
wait_for_bucket "$site"
|
||||
done
|
||||
|
||||
cat <<'EOF' | while read -r size_mb source_site object_name; do
|
||||
10 site1 object-010m.bin
|
||||
25 site2 object-025m.bin
|
||||
50 site3 object-050m.bin
|
||||
75 site1 object-075m.bin
|
||||
100 site2 object-100m.bin
|
||||
EOF
|
||||
src_file="$WORK_DIR/src/$object_name"
|
||||
object="$PREFIX/$object_name"
|
||||
|
||||
echo "creating ${size_mb}MiB file: $object_name"
|
||||
dd if=/dev/urandom of="$src_file" bs=1048576 count="$size_mb" >/dev/null 2>&1
|
||||
src_checksum="$(checksum_file "$src_file")"
|
||||
src_bytes="$(wc -c < "$src_file" | tr -d ' ')"
|
||||
|
||||
echo "uploading $object_name to $source_site ($src_bytes bytes)"
|
||||
mc cp "$src_file" "$source_site/$BUCKET/$object" >/dev/null
|
||||
|
||||
for site in $(other_sites "$source_site"); do
|
||||
wait_for_object "$site" "$object"
|
||||
|
||||
dst_file="$WORK_DIR/downloads/$site-$object_name"
|
||||
verified=false
|
||||
attempt=1
|
||||
while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
|
||||
rm -f "$dst_file"
|
||||
echo "downloading $object_name from $site"
|
||||
if mc cp "$site/$BUCKET/$object" "$dst_file" >/dev/null 2>&1; then
|
||||
dst_checksum="$(checksum_file "$dst_file")"
|
||||
dst_bytes="$(wc -c < "$dst_file" | tr -d ' ')"
|
||||
|
||||
if [ "$dst_checksum" = "$src_checksum" ] && [ "$dst_bytes" = "$src_bytes" ]; then
|
||||
verified=true
|
||||
break
|
||||
fi
|
||||
fi
|
||||
|
||||
sleep "$WAIT_SLEEP_SECONDS"
|
||||
attempt=$((attempt + 1))
|
||||
done
|
||||
|
||||
if [ "$verified" != "true" ]; then
|
||||
echo "download verification failed for $site/$BUCKET/$object" >&2
|
||||
echo "expected checksum: $src_checksum" >&2
|
||||
echo "expected bytes: $src_bytes" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
echo "verified replicated downloads for $object_name"
|
||||
done
|
||||
|
||||
echo "site replication object flow check passed"
|
||||
echo "bucket: $BUCKET"
|
||||
echo "prefix: $PREFIX"
|
||||
Reference in New Issue
Block a user