mirror of
https://github.com/rustfs/rustfs.git
synced 2026-07-26 08:18:18 +00:00
fix(site-replication): site replication flag issue and resolved the replication storm (#4120)
* fix(site-replication): Add Docker Compose setup for site replication testing - Fixed the site replication flag issue and resolved the replication storm. - Introduced a new directory for site replication tests with Docker Compose. - Created `docker-compose.yml` to define three RustFS sites and a setup container. - Added `README.md` to document the purpose, usage, and test flow of the replication setup. - Implemented `run-object-flow-check.sh` script to verify object replication across sites. - Configured health checks and volume permissions for the RustFS containers. - Enabled customization of access keys, bucket names, and other parameters via environment variables. * fix * fix
This commit is contained in:
唐小鸭
@@ -0,0 +1,183 @@
|
||||
# Site Replication Docker Compose Test
|
||||
|
||||
## Purpose
|
||||
|
||||
This directory contains a local three-site RustFS site replication check. It is intended to verify the admin site-replication flow against real containers:
|
||||
|
||||
- three independent RustFS sites start successfully
|
||||
- the MinIO-compatible `mc admin replicate add` command configures all three sites
|
||||
- a bucket and object written to site 1 are replicated to site 2 and site 3
|
||||
- `mc admin replicate status` can read the resulting site-replication state
|
||||
|
||||
The compose file uses named volumes so the test does not require preparing host bind-mount directories.
|
||||
Because Docker named volumes usually share the same physical device in local desktop environments, this test compose defaults `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true`. Keep that setting limited to local test and CI environments.
|
||||
|
||||
## Files
|
||||
|
||||
- `docker-compose.yml`: three RustFS sites, a volume permission helper, and a one-shot setup/check container
|
||||
- `run-object-flow-check.sh`: host-side upload/download verification for replicated 10 MiB to 100 MiB objects
|
||||
|
||||
## Ports
|
||||
|
||||
Default host endpoints:
|
||||
|
||||
- Site 1 API: `http://127.0.0.1:9000`
|
||||
- Site 1 Console: `http://127.0.0.1:9001`
|
||||
- Site 2 API: `http://127.0.0.1:9010`
|
||||
- Site 2 Console: `http://127.0.0.1:9011`
|
||||
- Site 3 API: `http://127.0.0.1:9020`
|
||||
- Site 3 Console: `http://127.0.0.1:9021`
|
||||
|
||||
Default credentials are `rustfsadmin` / `rustfsadmin`. These are for local testing only.
|
||||
|
||||
## Run
|
||||
|
||||
From the repository root:
|
||||
|
||||
```bash
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml up
|
||||
```
|
||||
|
||||
To test a locally built image instead of Docker Hub `rustfs/rustfs:latest`, set `RUSTFS_SITE_REPL_IMAGE`:
|
||||
|
||||
```bash
|
||||
docker build -f Dockerfile.source -t rustfs-site-repl-local:latest .
|
||||
RUSTFS_SITE_REPL_IMAGE=rustfs-site-repl-local:latest \
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml up
|
||||
```
|
||||
|
||||
For a detached run:
|
||||
|
||||
```bash
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml up -d
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml logs -f site-replication-setup
|
||||
```
|
||||
|
||||
## Test Flow
|
||||
|
||||
The compose stack performs these steps:
|
||||
|
||||
1. `site-replication-volume-permission-helper` fixes ownership on all named volumes for the RustFS runtime user.
|
||||
2. `rustfs-site1`, `rustfs-site2`, and `rustfs-site3` start as separate RustFS sites.
|
||||
3. Each site exposes its S3 API and Console on a unique host port.
|
||||
4. Health checks wait for `/health/ready` on each RustFS container.
|
||||
5. `site-replication-setup` configures `mc` aliases for all three sites.
|
||||
6. The setup container waits until `mc admin info` succeeds for all sites.
|
||||
7. It runs:
|
||||
|
||||
```bash
|
||||
mc admin replicate add site1 site2 site3
|
||||
```
|
||||
|
||||
8. It creates the test bucket on site 1 and uploads `from-site1.txt`.
|
||||
9. It polls site 2 and site 3 until the replicated object is visible.
|
||||
10. It prints `mc admin replicate status site1`.
|
||||
|
||||
The setup container exits with status `0` only after the object replication check passes.
|
||||
|
||||
## Object Flow Check
|
||||
|
||||
After the compose setup succeeds, run the larger object flow check from the repository root:
|
||||
|
||||
```bash
|
||||
.docker/test/site-replication/run-object-flow-check.sh
|
||||
```
|
||||
|
||||
The script creates five local files and uploads them from different sites:
|
||||
|
||||
- 10 MiB from site 1
|
||||
- 25 MiB from site 2
|
||||
- 50 MiB from site 3
|
||||
- 75 MiB from site 1
|
||||
- 100 MiB from site 2
|
||||
|
||||
For each uploaded object, the script waits for replication to the other two sites, downloads the object from those sites, and verifies both byte size and SHA-256 checksum. It uses a temporary `mc` config directory, so it does not overwrite existing host aliases.
|
||||
|
||||
The default bucket is `site-repl-flow-check`. Override it when needed:
|
||||
|
||||
```bash
|
||||
RUSTFS_SITE_REPL_FLOW_BUCKET='site-repl-large-flow' \
|
||||
.docker/test/site-replication/run-object-flow-check.sh
|
||||
```
|
||||
|
||||
The script keeps uploaded objects under a timestamped prefix. Override the prefix for repeatable runs:
|
||||
|
||||
```bash
|
||||
RUSTFS_SITE_REPL_FLOW_PREFIX='manual-check-001' \
|
||||
.docker/test/site-replication/run-object-flow-check.sh
|
||||
```
|
||||
|
||||
If replication is slow on the local machine, increase polling:
|
||||
|
||||
```bash
|
||||
RUSTFS_SITE_REPL_WAIT_ATTEMPTS=180 \
|
||||
RUSTFS_SITE_REPL_WAIT_SLEEP_SECONDS=2 \
|
||||
.docker/test/site-replication/run-object-flow-check.sh
|
||||
```
|
||||
|
||||
## Optional Settings
|
||||
|
||||
Override local test credentials:
|
||||
|
||||
```bash
|
||||
RUSTFS_SITE_REPL_ACCESS_KEY='localadmin' \
|
||||
RUSTFS_SITE_REPL_SECRET_KEY='localadmin-secret' \
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml up
|
||||
```
|
||||
|
||||
Use a different test bucket:
|
||||
|
||||
```bash
|
||||
RUSTFS_SITE_REPL_BUCKET='site-repl-check' \
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml up
|
||||
```
|
||||
|
||||
Enable ILM expiry rule replication during site setup:
|
||||
|
||||
```bash
|
||||
RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY=true \
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml up
|
||||
```
|
||||
|
||||
Use this only when the test needs lifecycle expiry metadata included in site replication.
|
||||
|
||||
## Manual Checks
|
||||
|
||||
After the setup container succeeds, you can inspect the sites with `mc` from the host:
|
||||
|
||||
```bash
|
||||
mc alias set site1 http://127.0.0.1:9000 rustfsadmin rustfsadmin
|
||||
mc alias set site2 http://127.0.0.1:9010 rustfsadmin rustfsadmin
|
||||
mc alias set site3 http://127.0.0.1:9020 rustfsadmin rustfsadmin
|
||||
|
||||
mc admin replicate info site1
|
||||
mc admin replicate status site1
|
||||
mc stat site2/site-repl-demo/from-site1.txt
|
||||
mc stat site3/site-repl-demo/from-site1.txt
|
||||
```
|
||||
|
||||
After larger object flow checks, replication should converge without a growing queue:
|
||||
|
||||
```bash
|
||||
mc admin replicate status site1
|
||||
mc admin replicate status site2
|
||||
mc admin replicate status site3
|
||||
```
|
||||
|
||||
Useful Docker commands:
|
||||
|
||||
```bash
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml ps
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml logs --no-color --tail=200
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml logs --no-color site-replication-setup
|
||||
```
|
||||
|
||||
## Cleanup
|
||||
|
||||
Remove containers and named volumes:
|
||||
|
||||
```bash
|
||||
docker compose -f .docker/test/site-replication/docker-compose.yml down -v
|
||||
```
|
||||
|
||||
Use `down -v` before rerunning the full setup from scratch. Site replication state is persisted in the named volumes, so rerunning without deleting volumes may attempt to add an already-configured replication topology.
|
||||
@@ -0,0 +1,243 @@
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
services:
|
||||
rustfs-site1:
|
||||
image: ${RUSTFS_SITE_REPL_IMAGE:-rustfs/rustfs:latest}
|
||||
container_name: rustfs-site-repl-1
|
||||
security_opt:
|
||||
- "no-new-privileges:true"
|
||||
ports:
|
||||
- "9000:9000"
|
||||
- "9001:9001"
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=/data/rustfs{0...3}
|
||||
- RUSTFS_ADDRESS=0.0.0.0:9000
|
||||
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
|
||||
- RUSTFS_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}
|
||||
- RUSTFS_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_SITE_REPL_LOG_LEVEL:-info}
|
||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
||||
volumes:
|
||||
- site1_data_0:/data/rustfs0
|
||||
- site1_data_1:/data/rustfs1
|
||||
- site1_data_2:/data/rustfs2
|
||||
- site1_data_3:/data/rustfs3
|
||||
- site1_logs:/app/logs
|
||||
networks:
|
||||
- rustfs-site-replication
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health/ready"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 30
|
||||
start_period: 20s
|
||||
depends_on:
|
||||
site-replication-volume-permission-helper:
|
||||
condition: service_completed_successfully
|
||||
|
||||
rustfs-site2:
|
||||
image: ${RUSTFS_SITE_REPL_IMAGE:-rustfs/rustfs:latest}
|
||||
container_name: rustfs-site-repl-2
|
||||
security_opt:
|
||||
- "no-new-privileges:true"
|
||||
ports:
|
||||
- "9010:9000"
|
||||
- "9011:9001"
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=/data/rustfs{0...3}
|
||||
- RUSTFS_ADDRESS=0.0.0.0:9000
|
||||
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
|
||||
- RUSTFS_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}
|
||||
- RUSTFS_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_SITE_REPL_LOG_LEVEL:-info}
|
||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
||||
volumes:
|
||||
- site2_data_0:/data/rustfs0
|
||||
- site2_data_1:/data/rustfs1
|
||||
- site2_data_2:/data/rustfs2
|
||||
- site2_data_3:/data/rustfs3
|
||||
- site2_logs:/app/logs
|
||||
networks:
|
||||
- rustfs-site-replication
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health/ready"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 30
|
||||
start_period: 20s
|
||||
depends_on:
|
||||
site-replication-volume-permission-helper:
|
||||
condition: service_completed_successfully
|
||||
|
||||
rustfs-site3:
|
||||
image: ${RUSTFS_SITE_REPL_IMAGE:-rustfs/rustfs:latest}
|
||||
container_name: rustfs-site-repl-3
|
||||
security_opt:
|
||||
- "no-new-privileges:true"
|
||||
ports:
|
||||
- "9020:9000"
|
||||
- "9021:9001"
|
||||
environment:
|
||||
- RUSTFS_VOLUMES=/data/rustfs{0...3}
|
||||
- RUSTFS_ADDRESS=0.0.0.0:9000
|
||||
- RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
|
||||
- RUSTFS_CONSOLE_ENABLE=true
|
||||
- RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
|
||||
- RUSTFS_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}
|
||||
- RUSTFS_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}
|
||||
- RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_SITE_REPL_LOG_LEVEL:-info}
|
||||
- RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
|
||||
volumes:
|
||||
- site3_data_0:/data/rustfs0
|
||||
- site3_data_1:/data/rustfs1
|
||||
- site3_data_2:/data/rustfs2
|
||||
- site3_data_3:/data/rustfs3
|
||||
- site3_logs:/app/logs
|
||||
networks:
|
||||
- rustfs-site-replication
|
||||
restart: unless-stopped
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health/ready"]
|
||||
interval: 10s
|
||||
timeout: 5s
|
||||
retries: 30
|
||||
start_period: 20s
|
||||
depends_on:
|
||||
site-replication-volume-permission-helper:
|
||||
condition: service_completed_successfully
|
||||
|
||||
site-replication-setup:
|
||||
image: minio/mc:latest
|
||||
container_name: rustfs-site-repl-setup
|
||||
depends_on:
|
||||
rustfs-site1:
|
||||
condition: service_healthy
|
||||
rustfs-site2:
|
||||
condition: service_healthy
|
||||
rustfs-site3:
|
||||
condition: service_healthy
|
||||
environment:
|
||||
- RUSTFS_SITE_REPL_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}
|
||||
- RUSTFS_SITE_REPL_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}
|
||||
- RUSTFS_SITE_REPL_BUCKET=${RUSTFS_SITE_REPL_BUCKET:-site-repl-demo}
|
||||
- RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY=${RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY:-false}
|
||||
networks:
|
||||
- rustfs-site-replication
|
||||
entrypoint: ["/bin/sh", "-c"]
|
||||
command:
|
||||
- |
|
||||
set -eu
|
||||
|
||||
access_key="$${RUSTFS_SITE_REPL_ACCESS_KEY}"
|
||||
secret_key="$${RUSTFS_SITE_REPL_SECRET_KEY}"
|
||||
bucket="$${RUSTFS_SITE_REPL_BUCKET}"
|
||||
|
||||
mc alias set site1 http://rustfs-site1:9000 "$${access_key}" "$${secret_key}"
|
||||
mc alias set site2 http://rustfs-site2:9000 "$${access_key}" "$${secret_key}"
|
||||
mc alias set site3 http://rustfs-site3:9000 "$${access_key}" "$${secret_key}"
|
||||
|
||||
for site in site1 site2 site3; do
|
||||
until mc ls "$${site}" >/dev/null 2>&1; do
|
||||
echo "waiting for $${site} S3 API"
|
||||
sleep 2
|
||||
done
|
||||
done
|
||||
|
||||
ilm_flag=""
|
||||
if [ "$${RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY}" = "true" ]; then
|
||||
ilm_flag="--replicate-ilm-expiry"
|
||||
fi
|
||||
|
||||
echo "configuring 3-site replication"
|
||||
if ! mc admin replicate add site1 site2 site3 $${ilm_flag}; then
|
||||
echo "replicate add failed; showing current replication info before exiting"
|
||||
mc admin replicate info site1 || true
|
||||
exit 1
|
||||
fi
|
||||
|
||||
mc mb --ignore-existing "site1/$${bucket}"
|
||||
printf 'rustfs 3-site replication check\n' > /tmp/site-replication-check.txt
|
||||
mc cp /tmp/site-replication-check.txt "site1/$${bucket}/from-site1.txt"
|
||||
|
||||
for site in site2 site3; do
|
||||
for attempt in $$(seq 1 60); do
|
||||
if mc stat "$${site}/$${bucket}/from-site1.txt" >/dev/null 2>&1; then
|
||||
echo "$${site} received replicated object"
|
||||
break
|
||||
fi
|
||||
if [ "$${attempt}" = "60" ]; then
|
||||
echo "$${site} did not receive replicated object in time"
|
||||
exit 1
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
done
|
||||
|
||||
mc admin replicate status site1
|
||||
echo "3-site replication example is ready"
|
||||
restart: "no"
|
||||
|
||||
site-replication-volume-permission-helper:
|
||||
image: alpine:3.23.4
|
||||
volumes:
|
||||
- site1_data_0:/site1/data0
|
||||
- site1_data_1:/site1/data1
|
||||
- site1_data_2:/site1/data2
|
||||
- site1_data_3:/site1/data3
|
||||
- site1_logs:/site1/logs
|
||||
- site2_data_0:/site2/data0
|
||||
- site2_data_1:/site2/data1
|
||||
- site2_data_2:/site2/data2
|
||||
- site2_data_3:/site2/data3
|
||||
- site2_logs:/site2/logs
|
||||
- site3_data_0:/site3/data0
|
||||
- site3_data_1:/site3/data1
|
||||
- site3_data_2:/site3/data2
|
||||
- site3_data_3:/site3/data3
|
||||
- site3_logs:/site3/logs
|
||||
command: >
|
||||
sh -c "
|
||||
chown -R 10001:10001 /site1 /site2 /site3 &&
|
||||
echo 'site replication volume permissions fixed'
|
||||
"
|
||||
networks:
|
||||
- rustfs-site-replication
|
||||
restart: "no"
|
||||
|
||||
networks:
|
||||
rustfs-site-replication:
|
||||
|
||||
volumes:
|
||||
site1_data_0:
|
||||
site1_data_1:
|
||||
site1_data_2:
|
||||
site1_data_3:
|
||||
site1_logs:
|
||||
site2_data_0:
|
||||
site2_data_1:
|
||||
site2_data_2:
|
||||
site2_data_3:
|
||||
site2_logs:
|
||||
site3_data_0:
|
||||
site3_data_1:
|
||||
site3_data_2:
|
||||
site3_data_3:
|
||||
site3_logs:
|
||||
+191
@@ -0,0 +1,191 @@
|
||||
#!/bin/sh
|
||||
# Copyright 2024 RustFS Team
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
set -eu
|
||||
|
||||
ACCESS_KEY="${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}"
|
||||
SECRET_KEY="${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}"
|
||||
BUCKET="${RUSTFS_SITE_REPL_FLOW_BUCKET:-site-repl-flow-check}"
|
||||
PREFIX="${RUSTFS_SITE_REPL_FLOW_PREFIX:-flow-$(date +%Y%m%d-%H%M%S)}"
|
||||
WAIT_ATTEMPTS="${RUSTFS_SITE_REPL_WAIT_ATTEMPTS:-90}"
|
||||
WAIT_SLEEP_SECONDS="${RUSTFS_SITE_REPL_WAIT_SLEEP_SECONDS:-2}"
|
||||
|
||||
SITE1_ENDPOINT="${RUSTFS_SITE1_ENDPOINT:-http://127.0.0.1:9000}"
|
||||
SITE2_ENDPOINT="${RUSTFS_SITE2_ENDPOINT:-http://127.0.0.1:9010}"
|
||||
SITE3_ENDPOINT="${RUSTFS_SITE3_ENDPOINT:-http://127.0.0.1:9020}"
|
||||
|
||||
command_exists() {
|
||||
command -v "$1" >/dev/null 2>&1
|
||||
}
|
||||
|
||||
require_command() {
|
||||
if ! command_exists "$1"; then
|
||||
echo "missing required command: $1" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
checksum_file() {
|
||||
if command_exists sha256sum; then
|
||||
sha256sum "$1" | awk '{print $1}'
|
||||
elif command_exists shasum; then
|
||||
shasum -a 256 "$1" | awk '{print $1}'
|
||||
else
|
||||
echo "missing required command: sha256sum or shasum" >&2
|
||||
exit 1
|
||||
fi
|
||||
}
|
||||
|
||||
site_endpoint() {
|
||||
case "$1" in
|
||||
site1) printf '%s\n' "$SITE1_ENDPOINT" ;;
|
||||
site2) printf '%s\n' "$SITE2_ENDPOINT" ;;
|
||||
site3) printf '%s\n' "$SITE3_ENDPOINT" ;;
|
||||
*) echo "unknown site alias: $1" >&2; exit 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
other_sites() {
|
||||
case "$1" in
|
||||
site1) printf '%s\n' "site2 site3" ;;
|
||||
site2) printf '%s\n' "site1 site3" ;;
|
||||
site3) printf '%s\n' "site1 site2" ;;
|
||||
*) echo "unknown source site: $1" >&2; exit 1 ;;
|
||||
esac
|
||||
}
|
||||
|
||||
wait_for_object() {
|
||||
site="$1"
|
||||
object="$2"
|
||||
attempt=1
|
||||
|
||||
while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
|
||||
if mc stat "$site/$BUCKET/$object" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
sleep "$WAIT_SLEEP_SECONDS"
|
||||
attempt=$((attempt + 1))
|
||||
done
|
||||
|
||||
echo "object was not replicated in time: $site/$BUCKET/$object" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
wait_for_bucket() {
|
||||
site="$1"
|
||||
attempt=1
|
||||
|
||||
while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
|
||||
if mc stat "$site/$BUCKET" >/dev/null 2>&1; then
|
||||
return 0
|
||||
fi
|
||||
sleep "$WAIT_SLEEP_SECONDS"
|
||||
attempt=$((attempt + 1))
|
||||
done
|
||||
|
||||
echo "bucket was not replicated in time: $site/$BUCKET" >&2
|
||||
return 1
|
||||
}
|
||||
|
||||
require_command mc
|
||||
require_command dd
|
||||
require_command awk
|
||||
require_command date
|
||||
require_command mktemp
|
||||
require_command tr
|
||||
require_command wc
|
||||
|
||||
WORK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/rustfs-site-repl-flow.XXXXXX")"
|
||||
MC_CONFIG_DIR="$WORK_DIR/mc"
|
||||
export MC_CONFIG_DIR
|
||||
|
||||
cleanup() {
|
||||
rm -rf "$WORK_DIR"
|
||||
}
|
||||
trap cleanup EXIT INT TERM
|
||||
|
||||
mkdir -p "$MC_CONFIG_DIR" "$WORK_DIR/src" "$WORK_DIR/downloads"
|
||||
|
||||
for site in site1 site2 site3; do
|
||||
mc alias set "$site" "$(site_endpoint "$site")" "$ACCESS_KEY" "$SECRET_KEY" >/dev/null
|
||||
done
|
||||
|
||||
for site in site1 site2 site3; do
|
||||
echo "checking S3 API for $site"
|
||||
mc ls "$site" >/dev/null
|
||||
done
|
||||
|
||||
echo "ensuring bucket exists on site1: $BUCKET"
|
||||
mc mb --ignore-existing "site1/$BUCKET" >/dev/null
|
||||
|
||||
for site in site1 site2 site3; do
|
||||
wait_for_bucket "$site"
|
||||
done
|
||||
|
||||
cat <<'EOF' | while read -r size_mb source_site object_name; do
|
||||
10 site1 object-010m.bin
|
||||
25 site2 object-025m.bin
|
||||
50 site3 object-050m.bin
|
||||
75 site1 object-075m.bin
|
||||
100 site2 object-100m.bin
|
||||
EOF
|
||||
src_file="$WORK_DIR/src/$object_name"
|
||||
object="$PREFIX/$object_name"
|
||||
|
||||
echo "creating ${size_mb}MiB file: $object_name"
|
||||
dd if=/dev/urandom of="$src_file" bs=1048576 count="$size_mb" >/dev/null 2>&1
|
||||
src_checksum="$(checksum_file "$src_file")"
|
||||
src_bytes="$(wc -c < "$src_file" | tr -d ' ')"
|
||||
|
||||
echo "uploading $object_name to $source_site ($src_bytes bytes)"
|
||||
mc cp "$src_file" "$source_site/$BUCKET/$object" >/dev/null
|
||||
|
||||
for site in $(other_sites "$source_site"); do
|
||||
wait_for_object "$site" "$object"
|
||||
|
||||
dst_file="$WORK_DIR/downloads/$site-$object_name"
|
||||
verified=false
|
||||
attempt=1
|
||||
while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
|
||||
rm -f "$dst_file"
|
||||
echo "downloading $object_name from $site"
|
||||
if mc cp "$site/$BUCKET/$object" "$dst_file" >/dev/null 2>&1; then
|
||||
dst_checksum="$(checksum_file "$dst_file")"
|
||||
dst_bytes="$(wc -c < "$dst_file" | tr -d ' ')"
|
||||
|
||||
if [ "$dst_checksum" = "$src_checksum" ] && [ "$dst_bytes" = "$src_bytes" ]; then
|
||||
verified=true
|
||||
break
|
||||
fi
|
||||
fi
|
||||
|
||||
sleep "$WAIT_SLEEP_SECONDS"
|
||||
attempt=$((attempt + 1))
|
||||
done
|
||||
|
||||
if [ "$verified" != "true" ]; then
|
||||
echo "download verification failed for $site/$BUCKET/$object" >&2
|
||||
echo "expected checksum: $src_checksum" >&2
|
||||
echo "expected bytes: $src_bytes" >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
echo "verified replicated downloads for $object_name"
|
||||
done
|
||||
|
||||
echo "site replication object flow check passed"
|
||||
echo "bucket: $BUCKET"
|
||||
echo "prefix: $PREFIX"
|
||||
+4
-2
@@ -55,8 +55,10 @@ docs/heal-scanner-logging-governance.md
|
||||
docs/benchmark/rustfs-target-bench/
|
||||
docs/benchmark/*.md
|
||||
.codegraph/*
|
||||
.docker/compat/data/*
|
||||
.docker/compat/kms/*
|
||||
.docker/test/compat/data/*
|
||||
.docker/test/compat/kms/*
|
||||
reasonix.toml
|
||||
.reasonix/*
|
||||
|
||||
# nix stuff
|
||||
result*
|
||||
|
||||
@@ -4008,6 +4008,11 @@ async fn replicate_object_with_multipart<S: ReplicationObjectIO>(ctx: MultipartR
|
||||
uploaded_parts,
|
||||
&PutObjectOptions {
|
||||
user_metadata,
|
||||
internal: AdvancedPutOptions {
|
||||
replication_status: ReplicationStatusType::Replica,
|
||||
replication_request: true,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
|
||||
@@ -109,6 +109,14 @@ impl ObjectOptions {
|
||||
}
|
||||
|
||||
pub fn put_replication_state(&self) -> ReplicationState {
|
||||
if self
|
||||
.delete_replication
|
||||
.as_ref()
|
||||
.is_some_and(|state| !state.replica_status.is_empty())
|
||||
{
|
||||
return self.delete_replication.clone().unwrap_or_default();
|
||||
}
|
||||
|
||||
let rs = match rustfs_utils::http::get_str(&self.user_defined, rustfs_utils::http::SUFFIX_REPLICATION_STATUS) {
|
||||
Some(v) => v,
|
||||
None => return ReplicationState::default(),
|
||||
@@ -797,6 +805,21 @@ mod tests {
|
||||
assert_eq!(versions[0].version_id, Some(last_version));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn put_replication_state_preserves_replica_status() {
|
||||
let opts = ObjectOptions {
|
||||
delete_replication: Some(ReplicationState {
|
||||
replica_status: ReplicationStatusType::Replica,
|
||||
..Default::default()
|
||||
}),
|
||||
..Default::default()
|
||||
};
|
||||
|
||||
let state = opts.put_replication_state();
|
||||
|
||||
assert_eq!(state.composite_replication_status(), ReplicationStatusType::Replica);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn versions_after_marker_handles_uuid_version_marker() {
|
||||
let first_version = Uuid::parse_str("11111111-2222-3333-4444-555555555555").unwrap();
|
||||
|
||||
@@ -63,7 +63,7 @@ use crate::table_catalog;
|
||||
use bytes::Bytes;
|
||||
use futures::StreamExt;
|
||||
use http::{HeaderMap, Uri};
|
||||
use rustfs_filemeta::{ReplicationStatusType, ReplicationType};
|
||||
use rustfs_filemeta::ReplicationType;
|
||||
use rustfs_s3_ops::S3Operation;
|
||||
use rustfs_targets::EventName;
|
||||
use rustfs_utils::CompressionAlgorithm;
|
||||
@@ -560,8 +560,13 @@ impl DefaultMultipartUsecase {
|
||||
..Default::default()
|
||||
};
|
||||
let mt2 = obj_info.user_defined.clone();
|
||||
let replicate_options =
|
||||
get_must_replicate_options(&mt2, "".to_string(), ReplicationStatusType::Empty, ReplicationType::Object, opts.clone());
|
||||
let replicate_options = get_must_replicate_options(
|
||||
&mt2,
|
||||
"".to_string(),
|
||||
opts.delete_marker_replication_status(),
|
||||
ReplicationType::Object,
|
||||
opts.clone(),
|
||||
);
|
||||
let dsc = must_replicate(&bucket, &key, replicate_options).await;
|
||||
|
||||
if dsc.replicate_any() {
|
||||
@@ -694,8 +699,13 @@ impl DefaultMultipartUsecase {
|
||||
.await
|
||||
.map_err(ApiError::from)?;
|
||||
|
||||
let replicate_options =
|
||||
get_must_replicate_options(&mt2, "".to_string(), ReplicationStatusType::Empty, ReplicationType::Object, opts.clone());
|
||||
let replicate_options = get_must_replicate_options(
|
||||
&mt2,
|
||||
"".to_string(),
|
||||
opts.delete_marker_replication_status(),
|
||||
ReplicationType::Object,
|
||||
opts.clone(),
|
||||
);
|
||||
let dsc = must_replicate(&bucket, &key, replicate_options).await;
|
||||
if dsc.replicate_any() {
|
||||
insert_str(&mut opts.user_defined, SUFFIX_REPLICATION_TIMESTAMP, jiff::Zoned::now().to_string());
|
||||
|
||||
@@ -2994,8 +2994,13 @@ impl DefaultObjectUsecase {
|
||||
.map(|ctx| ctx.request_id.clone())
|
||||
.unwrap_or_else(|| request_context::RequestContext::fallback().request_id);
|
||||
|
||||
let repoptions =
|
||||
get_must_replicate_options(&mt2, "".to_string(), ReplicationStatusType::Empty, ReplicationType::Object, opts.clone());
|
||||
let repoptions = get_must_replicate_options(
|
||||
&mt2,
|
||||
"".to_string(),
|
||||
opts.delete_marker_replication_status(),
|
||||
ReplicationType::Object,
|
||||
opts.clone(),
|
||||
);
|
||||
let dsc = must_replicate(&bucket, &key, repoptions).await;
|
||||
|
||||
if dsc.replicate_any() {
|
||||
@@ -3105,8 +3110,13 @@ impl DefaultObjectUsecase {
|
||||
|
||||
let e_tag = obj_info.etag.clone().map(|etag| to_s3s_etag(&etag));
|
||||
|
||||
let repoptions =
|
||||
get_must_replicate_options(&mt2, "".to_string(), ReplicationStatusType::Empty, ReplicationType::Object, opts);
|
||||
let repoptions = get_must_replicate_options(
|
||||
&mt2,
|
||||
"".to_string(),
|
||||
opts.delete_marker_replication_status(),
|
||||
ReplicationType::Object,
|
||||
opts,
|
||||
);
|
||||
|
||||
let dsc = must_replicate(&bucket, &key, repoptions).await;
|
||||
let expiration = resolve_put_object_expiration(&bucket, &obj_info).await;
|
||||
|
||||
@@ -16,15 +16,16 @@ use super::{BucketVersioningSys, Result, StorageError};
|
||||
use crate::storage::storage_api::options_consumer::contract::{object::HTTPPreconditions, range::HTTPRangeSpec};
|
||||
use http::header::{IF_MATCH, IF_NONE_MATCH};
|
||||
use http::{HeaderMap, HeaderValue};
|
||||
use rustfs_filemeta::ReplicationStatusType;
|
||||
use rustfs_utils::http::{
|
||||
AMZ_BUCKET_REPLICATION_STATUS, SUFFIX_FORCE_DELETE, SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE, SUFFIX_REPLICATION_SSEC_CRC,
|
||||
SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_REQUEST, SUFFIX_SOURCE_VERSION_ID, get_header,
|
||||
insert_header_map, is_encryption_metadata_key, is_internal_key,
|
||||
};
|
||||
use rustfs_utils::http::{
|
||||
AMZ_META_UNENCRYPTED_CONTENT_LENGTH, AMZ_META_UNENCRYPTED_CONTENT_MD5, AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER,
|
||||
AMZ_OBJECT_LOCK_MODE_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER,
|
||||
};
|
||||
use rustfs_utils::http::{
|
||||
SUFFIX_FORCE_DELETE, SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE, SUFFIX_REPLICATION_SSEC_CRC, SUFFIX_SOURCE_DELETEMARKER,
|
||||
SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_REQUEST, SUFFIX_SOURCE_VERSION_ID, get_header, insert_header_map,
|
||||
is_encryption_metadata_key, is_internal_key,
|
||||
};
|
||||
use s3s::header::X_AMZ_OBJECT_LOCK_MODE;
|
||||
use s3s::header::X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE;
|
||||
|
||||
@@ -275,6 +276,7 @@ pub fn get_complete_multipart_upload_opts(headers: &HeaderMap<HeaderValue>) -> s
|
||||
replication_request,
|
||||
..Default::default()
|
||||
};
|
||||
apply_replica_status_from_headers(headers, &mut opts);
|
||||
|
||||
fill_conditional_writes_opts_from_header(headers, &mut opts)?;
|
||||
Ok(opts)
|
||||
@@ -297,6 +299,7 @@ pub fn copy_src_opts(_bucket: &str, _object: &str, headers: &HeaderMap<HeaderVal
|
||||
|
||||
pub fn put_opts_from_headers(headers: &HeaderMap<HeaderValue>, metadata: HashMap<String, String>) -> Result<ObjectOptions> {
|
||||
let mut opts = get_default_opts(headers, metadata, false)?;
|
||||
apply_replica_status_from_headers(headers, &mut opts);
|
||||
if get_header(headers, SUFFIX_SOURCE_REPLICATION_REQUEST).as_deref() == Some("true") {
|
||||
opts.replication_request = true;
|
||||
if let Some(v) = get_header(headers, SUFFIX_SOURCE_MTIME) {
|
||||
@@ -313,6 +316,16 @@ pub fn put_opts_from_headers(headers: &HeaderMap<HeaderValue>, metadata: HashMap
|
||||
Ok(opts)
|
||||
}
|
||||
|
||||
fn apply_replica_status_from_headers(headers: &HeaderMap<HeaderValue>, opts: &mut ObjectOptions) {
|
||||
if headers
|
||||
.get(AMZ_BUCKET_REPLICATION_STATUS)
|
||||
.and_then(|value| value.to_str().ok())
|
||||
.is_some_and(|status| status.eq_ignore_ascii_case(ReplicationStatusType::Replica.as_str()))
|
||||
{
|
||||
opts.set_replica_status(ReplicationStatusType::Replica);
|
||||
}
|
||||
}
|
||||
|
||||
/// Creates default options for getting an object from a bucket.
|
||||
pub fn get_default_opts(
|
||||
_headers: &HeaderMap<HeaderValue>,
|
||||
@@ -788,13 +801,15 @@ mod tests {
|
||||
use super::{
|
||||
ENV_REJECT_ARCHIVE_CONTENT_ENCODING, SUPPORTED_HEADERS, copy_dst_opts, copy_src_opts, del_opts,
|
||||
detect_content_type_from_object_name, extract_metadata, extract_metadata_from_mime,
|
||||
extract_metadata_from_mime_with_object_name, filter_object_metadata, get_default_opts, get_opts, parse_copy_source_range,
|
||||
put_opts, put_opts_from_headers, validate_archive_content_encoding,
|
||||
extract_metadata_from_mime_with_object_name, filter_object_metadata, get_complete_multipart_upload_opts,
|
||||
get_default_opts, get_opts, parse_copy_source_range, put_opts, put_opts_from_headers, validate_archive_content_encoding,
|
||||
};
|
||||
use http::{HeaderMap, HeaderValue};
|
||||
use rustfs_filemeta::ReplicationStatusType;
|
||||
use rustfs_utils::http::{
|
||||
AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, AMZ_OBJECT_LOCK_MODE_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER,
|
||||
SUFFIX_FORCE_DELETE, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_REQUEST, insert_header,
|
||||
AMZ_BUCKET_REPLICATION_STATUS, AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, AMZ_OBJECT_LOCK_MODE_LOWER,
|
||||
AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER, SUFFIX_FORCE_DELETE, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_REQUEST,
|
||||
insert_header,
|
||||
};
|
||||
use s3s::S3ErrorCode;
|
||||
use std::collections::HashMap;
|
||||
@@ -1126,6 +1141,32 @@ mod tests {
|
||||
assert!(opts_invalid.mod_time.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_put_opts_from_headers_with_replica_status() {
|
||||
let mut headers = HeaderMap::new();
|
||||
headers.insert(
|
||||
AMZ_BUCKET_REPLICATION_STATUS,
|
||||
HeaderValue::from_static(ReplicationStatusType::Replica.as_str()),
|
||||
);
|
||||
|
||||
let opts = put_opts_from_headers(&headers, HashMap::new()).expect("replica status header should parse");
|
||||
|
||||
assert_eq!(opts.delete_marker_replication_status(), ReplicationStatusType::Replica);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_complete_multipart_opts_with_replica_status() {
|
||||
let mut headers = HeaderMap::new();
|
||||
headers.insert(
|
||||
AMZ_BUCKET_REPLICATION_STATUS,
|
||||
HeaderValue::from_static(ReplicationStatusType::Replica.as_str()),
|
||||
);
|
||||
|
||||
let opts = get_complete_multipart_upload_opts(&headers).expect("replica status header should parse");
|
||||
|
||||
assert_eq!(opts.delete_marker_replication_status(), ReplicationStatusType::Replica);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_get_default_opts_with_metadata() {
|
||||
let headers = create_test_headers();
|
||||
|
||||
Reference in New Issue
Block a user