fix(site-replication): site replication flag issue and resolved the replication storm (#4120)

* fix(site-replication): Add Docker Compose setup for site replication testing

- Fixed the site replication flag issue and resolved the replication storm.
- Introduced a new directory for site replication tests with Docker Compose.
- Created `docker-compose.yml` to define three RustFS sites and a setup container.
- Added `README.md` to document the purpose, usage, and test flow of the replication setup.
- Implemented `run-object-flow-check.sh` script to verify object replication across sites.
- Configured health checks and volume permissions for the RustFS containers.
- Enabled customization of access keys, bucket names, and other parameters via environment variables.

* fix

* fix

.docker/test/site-replication/README.md

@@ -0,0 +1,183 @@
+# Site Replication Docker Compose Test
+
+## Purpose
+
+This directory contains a local three-site RustFS site replication check. It is intended to verify the admin site-replication flow against real containers:
+
+- three independent RustFS sites start successfully
+- the MinIO-compatible `mc admin replicate add` command configures all three sites
+- a bucket and object written to site 1 are replicated to site 2 and site 3
+- `mc admin replicate status` can read the resulting site-replication state
+
+The compose file uses named volumes so the test does not require preparing host bind-mount directories.
+Because Docker named volumes usually share the same physical device in local desktop environments, this test compose defaults `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true`. Keep that setting limited to local test and CI environments.
+
+## Files
+
+- `docker-compose.yml`: three RustFS sites, a volume permission helper, and a one-shot setup/check container
+- `run-object-flow-check.sh`: host-side upload/download verification for replicated 10 MiB to 100 MiB objects
+
+## Ports
+
+Default host endpoints:
+
+- Site 1 API: `http://127.0.0.1:9000`
+- Site 1 Console: `http://127.0.0.1:9001`
+- Site 2 API: `http://127.0.0.1:9010`
+- Site 2 Console: `http://127.0.0.1:9011`
+- Site 3 API: `http://127.0.0.1:9020`
+- Site 3 Console: `http://127.0.0.1:9021`
+
+Default credentials are `rustfsadmin` / `rustfsadmin`. These are for local testing only.
+
+## Run
+
+From the repository root:
+
+```bash
+docker compose -f .docker/test/site-replication/docker-compose.yml up
+```
+
+To test a locally built image instead of Docker Hub `rustfs/rustfs:latest`, set `RUSTFS_SITE_REPL_IMAGE`:
+
+```bash
+docker build -f Dockerfile.source -t rustfs-site-repl-local:latest .
+RUSTFS_SITE_REPL_IMAGE=rustfs-site-repl-local:latest \
+docker compose -f .docker/test/site-replication/docker-compose.yml up
+```
+
+For a detached run:
+
+```bash
+docker compose -f .docker/test/site-replication/docker-compose.yml up -d
+docker compose -f .docker/test/site-replication/docker-compose.yml logs -f site-replication-setup
+```
+
+## Test Flow
+
+The compose stack performs these steps:
+
+1. `site-replication-volume-permission-helper` fixes ownership on all named volumes for the RustFS runtime user.
+2. `rustfs-site1`, `rustfs-site2`, and `rustfs-site3` start as separate RustFS sites.
+3. Each site exposes its S3 API and Console on a unique host port.
+4. Health checks wait for `/health/ready` on each RustFS container.
+5. `site-replication-setup` configures `mc` aliases for all three sites.
+6. The setup container waits until `mc admin info` succeeds for all sites.
+7. It runs:
+
+```bash
+mc admin replicate add site1 site2 site3
+```
+
+8. It creates the test bucket on site 1 and uploads `from-site1.txt`.
+9. It polls site 2 and site 3 until the replicated object is visible.
+10. It prints `mc admin replicate status site1`.
+
+The setup container exits with status `0` only after the object replication check passes.
+
+## Object Flow Check
+
+After the compose setup succeeds, run the larger object flow check from the repository root:
+
+```bash
+.docker/test/site-replication/run-object-flow-check.sh
+```
+
+The script creates five local files and uploads them from different sites:
+
+- 10 MiB from site 1
+- 25 MiB from site 2
+- 50 MiB from site 3
+- 75 MiB from site 1
+- 100 MiB from site 2
+
+For each uploaded object, the script waits for replication to the other two sites, downloads the object from those sites, and verifies both byte size and SHA-256 checksum. It uses a temporary `mc` config directory, so it does not overwrite existing host aliases.
+
+The default bucket is `site-repl-flow-check`. Override it when needed:
+
+```bash
+RUSTFS_SITE_REPL_FLOW_BUCKET='site-repl-large-flow' \
+.docker/test/site-replication/run-object-flow-check.sh
+```
+
+The script keeps uploaded objects under a timestamped prefix. Override the prefix for repeatable runs:
+
+```bash
+RUSTFS_SITE_REPL_FLOW_PREFIX='manual-check-001' \
+.docker/test/site-replication/run-object-flow-check.sh
+```
+
+If replication is slow on the local machine, increase polling:
+
+```bash
+RUSTFS_SITE_REPL_WAIT_ATTEMPTS=180 \
+RUSTFS_SITE_REPL_WAIT_SLEEP_SECONDS=2 \
+.docker/test/site-replication/run-object-flow-check.sh
+```
+
+## Optional Settings
+
+Override local test credentials:
+
+```bash
+RUSTFS_SITE_REPL_ACCESS_KEY='localadmin' \
+RUSTFS_SITE_REPL_SECRET_KEY='localadmin-secret' \
+docker compose -f .docker/test/site-replication/docker-compose.yml up
+```
+
+Use a different test bucket:
+
+```bash
+RUSTFS_SITE_REPL_BUCKET='site-repl-check' \
+docker compose -f .docker/test/site-replication/docker-compose.yml up
+```
+
+Enable ILM expiry rule replication during site setup:
+
+```bash
+RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY=true \
+docker compose -f .docker/test/site-replication/docker-compose.yml up
+```
+
+Use this only when the test needs lifecycle expiry metadata included in site replication.
+
+## Manual Checks
+
+After the setup container succeeds, you can inspect the sites with `mc` from the host:
+
+```bash
+mc alias set site1 http://127.0.0.1:9000 rustfsadmin rustfsadmin
+mc alias set site2 http://127.0.0.1:9010 rustfsadmin rustfsadmin
+mc alias set site3 http://127.0.0.1:9020 rustfsadmin rustfsadmin
+
+mc admin replicate info site1
+mc admin replicate status site1
+mc stat site2/site-repl-demo/from-site1.txt
+mc stat site3/site-repl-demo/from-site1.txt
+```
+
+After larger object flow checks, replication should converge without a growing queue:
+
+```bash
+mc admin replicate status site1
+mc admin replicate status site2
+mc admin replicate status site3
+```
+
+Useful Docker commands:
+
+```bash
+docker compose -f .docker/test/site-replication/docker-compose.yml ps
+docker compose -f .docker/test/site-replication/docker-compose.yml logs --no-color --tail=200
+docker compose -f .docker/test/site-replication/docker-compose.yml logs --no-color site-replication-setup
+```
+
+## Cleanup
+
+Remove containers and named volumes:
+
+```bash
+docker compose -f .docker/test/site-replication/docker-compose.yml down -v
+```
+
+Use `down -v` before rerunning the full setup from scratch. Site replication state is persisted in the named volumes, so rerunning without deleting volumes may attempt to add an already-configured replication topology.

.docker/test/site-replication/docker-compose.yml

@@ -0,0 +1,243 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+services:
+  rustfs-site1:
+    image: ${RUSTFS_SITE_REPL_IMAGE:-rustfs/rustfs:latest}
+    container_name: rustfs-site-repl-1
+    security_opt:
+      - "no-new-privileges:true"
+    ports:
+      - "9000:9000"
+      - "9001:9001"
+    environment:
+      - RUSTFS_VOLUMES=/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=0.0.0.0:9000
+      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_SITE_REPL_LOG_LEVEL:-info}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+    volumes:
+      - site1_data_0:/data/rustfs0
+      - site1_data_1:/data/rustfs1
+      - site1_data_2:/data/rustfs2
+      - site1_data_3:/data/rustfs3
+      - site1_logs:/app/logs
+    networks:
+      - rustfs-site-replication
+    restart: unless-stopped
+    healthcheck:
+      test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health/ready"]
+      interval: 10s
+      timeout: 5s
+      retries: 30
+      start_period: 20s
+    depends_on:
+      site-replication-volume-permission-helper:
+        condition: service_completed_successfully
+
+  rustfs-site2:
+    image: ${RUSTFS_SITE_REPL_IMAGE:-rustfs/rustfs:latest}
+    container_name: rustfs-site-repl-2
+    security_opt:
+      - "no-new-privileges:true"
+    ports:
+      - "9010:9000"
+      - "9011:9001"
+    environment:
+      - RUSTFS_VOLUMES=/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=0.0.0.0:9000
+      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_SITE_REPL_LOG_LEVEL:-info}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+    volumes:
+      - site2_data_0:/data/rustfs0
+      - site2_data_1:/data/rustfs1
+      - site2_data_2:/data/rustfs2
+      - site2_data_3:/data/rustfs3
+      - site2_logs:/app/logs
+    networks:
+      - rustfs-site-replication
+    restart: unless-stopped
+    healthcheck:
+      test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health/ready"]
+      interval: 10s
+      timeout: 5s
+      retries: 30
+      start_period: 20s
+    depends_on:
+      site-replication-volume-permission-helper:
+        condition: service_completed_successfully
+
+  rustfs-site3:
+    image: ${RUSTFS_SITE_REPL_IMAGE:-rustfs/rustfs:latest}
+    container_name: rustfs-site-repl-3
+    security_opt:
+      - "no-new-privileges:true"
+    ports:
+      - "9020:9000"
+      - "9021:9001"
+    environment:
+      - RUSTFS_VOLUMES=/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=0.0.0.0:9000
+      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_SITE_REPL_LOG_LEVEL:-info}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+    volumes:
+      - site3_data_0:/data/rustfs0
+      - site3_data_1:/data/rustfs1
+      - site3_data_2:/data/rustfs2
+      - site3_data_3:/data/rustfs3
+      - site3_logs:/app/logs
+    networks:
+      - rustfs-site-replication
+    restart: unless-stopped
+    healthcheck:
+      test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health/ready"]
+      interval: 10s
+      timeout: 5s
+      retries: 30
+      start_period: 20s
+    depends_on:
+      site-replication-volume-permission-helper:
+        condition: service_completed_successfully
+
+  site-replication-setup:
+    image: minio/mc:latest
+    container_name: rustfs-site-repl-setup
+    depends_on:
+      rustfs-site1:
+        condition: service_healthy
+      rustfs-site2:
+        condition: service_healthy
+      rustfs-site3:
+        condition: service_healthy
+    environment:
+      - RUSTFS_SITE_REPL_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SITE_REPL_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_SITE_REPL_BUCKET=${RUSTFS_SITE_REPL_BUCKET:-site-repl-demo}
+      - RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY=${RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY:-false}
+    networks:
+      - rustfs-site-replication
+    entrypoint: ["/bin/sh", "-c"]
+    command:
+      - |
+        set -eu
+
+        access_key="$${RUSTFS_SITE_REPL_ACCESS_KEY}"
+        secret_key="$${RUSTFS_SITE_REPL_SECRET_KEY}"
+        bucket="$${RUSTFS_SITE_REPL_BUCKET}"
+
+        mc alias set site1 http://rustfs-site1:9000 "$${access_key}" "$${secret_key}"
+        mc alias set site2 http://rustfs-site2:9000 "$${access_key}" "$${secret_key}"
+        mc alias set site3 http://rustfs-site3:9000 "$${access_key}" "$${secret_key}"
+
+        for site in site1 site2 site3; do
+          until mc ls "$${site}" >/dev/null 2>&1; do
+            echo "waiting for $${site} S3 API"
+            sleep 2
+          done
+        done
+
+        ilm_flag=""
+        if [ "$${RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY}" = "true" ]; then
+          ilm_flag="--replicate-ilm-expiry"
+        fi
+
+        echo "configuring 3-site replication"
+        if ! mc admin replicate add site1 site2 site3 $${ilm_flag}; then
+          echo "replicate add failed; showing current replication info before exiting"
+          mc admin replicate info site1 || true
+          exit 1
+        fi
+
+        mc mb --ignore-existing "site1/$${bucket}"
+        printf 'rustfs 3-site replication check\n' > /tmp/site-replication-check.txt
+        mc cp /tmp/site-replication-check.txt "site1/$${bucket}/from-site1.txt"
+
+        for site in site2 site3; do
+          for attempt in $$(seq 1 60); do
+            if mc stat "$${site}/$${bucket}/from-site1.txt" >/dev/null 2>&1; then
+              echo "$${site} received replicated object"
+              break
+            fi
+            if [ "$${attempt}" = "60" ]; then
+              echo "$${site} did not receive replicated object in time"
+              exit 1
+            fi
+            sleep 2
+          done
+        done
+
+        mc admin replicate status site1
+        echo "3-site replication example is ready"
+    restart: "no"
+
+  site-replication-volume-permission-helper:
+    image: alpine:3.23.4
+    volumes:
+      - site1_data_0:/site1/data0
+      - site1_data_1:/site1/data1
+      - site1_data_2:/site1/data2
+      - site1_data_3:/site1/data3
+      - site1_logs:/site1/logs
+      - site2_data_0:/site2/data0
+      - site2_data_1:/site2/data1
+      - site2_data_2:/site2/data2
+      - site2_data_3:/site2/data3
+      - site2_logs:/site2/logs
+      - site3_data_0:/site3/data0
+      - site3_data_1:/site3/data1
+      - site3_data_2:/site3/data2
+      - site3_data_3:/site3/data3
+      - site3_logs:/site3/logs
+    command: >
+      sh -c "
+        chown -R 10001:10001 /site1 /site2 /site3 &&
+        echo 'site replication volume permissions fixed'
+      "
+    networks:
+      - rustfs-site-replication
+    restart: "no"
+
+networks:
+  rustfs-site-replication:
+
+volumes:
+  site1_data_0:
+  site1_data_1:
+  site1_data_2:
+  site1_data_3:
+  site1_logs:
+  site2_data_0:
+  site2_data_1:
+  site2_data_2:
+  site2_data_3:
+  site2_logs:
+  site3_data_0:
+  site3_data_1:
+  site3_data_2:
+  site3_data_3:
+  site3_logs:

.docker/test/site-replication/run-object-flow-check.sh

@@ -0,0 +1,191 @@
+#!/bin/sh
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -eu
+
+ACCESS_KEY="${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}"
+SECRET_KEY="${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}"
+BUCKET="${RUSTFS_SITE_REPL_FLOW_BUCKET:-site-repl-flow-check}"
+PREFIX="${RUSTFS_SITE_REPL_FLOW_PREFIX:-flow-$(date +%Y%m%d-%H%M%S)}"
+WAIT_ATTEMPTS="${RUSTFS_SITE_REPL_WAIT_ATTEMPTS:-90}"
+WAIT_SLEEP_SECONDS="${RUSTFS_SITE_REPL_WAIT_SLEEP_SECONDS:-2}"
+
+SITE1_ENDPOINT="${RUSTFS_SITE1_ENDPOINT:-http://127.0.0.1:9000}"
+SITE2_ENDPOINT="${RUSTFS_SITE2_ENDPOINT:-http://127.0.0.1:9010}"
+SITE3_ENDPOINT="${RUSTFS_SITE3_ENDPOINT:-http://127.0.0.1:9020}"
+
+command_exists() {
+  command -v "$1" >/dev/null 2>&1
+}
+
+require_command() {
+  if ! command_exists "$1"; then
+    echo "missing required command: $1" >&2
+    exit 1
+  fi
+}
+
+checksum_file() {
+  if command_exists sha256sum; then
+    sha256sum "$1" | awk '{print $1}'
+  elif command_exists shasum; then
+    shasum -a 256 "$1" | awk '{print $1}'
+  else
+    echo "missing required command: sha256sum or shasum" >&2
+    exit 1
+  fi
+}
+
+site_endpoint() {
+  case "$1" in
+    site1) printf '%s\n' "$SITE1_ENDPOINT" ;;
+    site2) printf '%s\n' "$SITE2_ENDPOINT" ;;
+    site3) printf '%s\n' "$SITE3_ENDPOINT" ;;
+    *) echo "unknown site alias: $1" >&2; exit 1 ;;
+  esac
+}
+
+other_sites() {
+  case "$1" in
+    site1) printf '%s\n' "site2 site3" ;;
+    site2) printf '%s\n' "site1 site3" ;;
+    site3) printf '%s\n' "site1 site2" ;;
+    *) echo "unknown source site: $1" >&2; exit 1 ;;
+  esac
+}
+
+wait_for_object() {
+  site="$1"
+  object="$2"
+  attempt=1
+
+  while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
+    if mc stat "$site/$BUCKET/$object" >/dev/null 2>&1; then
+      return 0
+    fi
+    sleep "$WAIT_SLEEP_SECONDS"
+    attempt=$((attempt + 1))
+  done
+
+  echo "object was not replicated in time: $site/$BUCKET/$object" >&2
+  return 1
+}
+
+wait_for_bucket() {
+  site="$1"
+  attempt=1
+
+  while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
+    if mc stat "$site/$BUCKET" >/dev/null 2>&1; then
+      return 0
+    fi
+    sleep "$WAIT_SLEEP_SECONDS"
+    attempt=$((attempt + 1))
+  done
+
+  echo "bucket was not replicated in time: $site/$BUCKET" >&2
+  return 1
+}
+
+require_command mc
+require_command dd
+require_command awk
+require_command date
+require_command mktemp
+require_command tr
+require_command wc
+
+WORK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/rustfs-site-repl-flow.XXXXXX")"
+MC_CONFIG_DIR="$WORK_DIR/mc"
+export MC_CONFIG_DIR
+
+cleanup() {
+  rm -rf "$WORK_DIR"
+}
+trap cleanup EXIT INT TERM
+
+mkdir -p "$MC_CONFIG_DIR" "$WORK_DIR/src" "$WORK_DIR/downloads"
+
+for site in site1 site2 site3; do
+  mc alias set "$site" "$(site_endpoint "$site")" "$ACCESS_KEY" "$SECRET_KEY" >/dev/null
+done
+
+for site in site1 site2 site3; do
+  echo "checking S3 API for $site"
+  mc ls "$site" >/dev/null
+done
+
+echo "ensuring bucket exists on site1: $BUCKET"
+mc mb --ignore-existing "site1/$BUCKET" >/dev/null
+
+for site in site1 site2 site3; do
+  wait_for_bucket "$site"
+done
+
+cat <<'EOF' | while read -r size_mb source_site object_name; do
+10 site1 object-010m.bin
+25 site2 object-025m.bin
+50 site3 object-050m.bin
+75 site1 object-075m.bin
+100 site2 object-100m.bin
+EOF
+  src_file="$WORK_DIR/src/$object_name"
+  object="$PREFIX/$object_name"
+
+  echo "creating ${size_mb}MiB file: $object_name"
+  dd if=/dev/urandom of="$src_file" bs=1048576 count="$size_mb" >/dev/null 2>&1
+  src_checksum="$(checksum_file "$src_file")"
+  src_bytes="$(wc -c < "$src_file" | tr -d ' ')"
+
+  echo "uploading $object_name to $source_site ($src_bytes bytes)"
+  mc cp "$src_file" "$source_site/$BUCKET/$object" >/dev/null
+
+  for site in $(other_sites "$source_site"); do
+    wait_for_object "$site" "$object"
+
+    dst_file="$WORK_DIR/downloads/$site-$object_name"
+    verified=false
+    attempt=1
+    while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
+      rm -f "$dst_file"
+      echo "downloading $object_name from $site"
+      if mc cp "$site/$BUCKET/$object" "$dst_file" >/dev/null 2>&1; then
+        dst_checksum="$(checksum_file "$dst_file")"
+        dst_bytes="$(wc -c < "$dst_file" | tr -d ' ')"
+
+        if [ "$dst_checksum" = "$src_checksum" ] && [ "$dst_bytes" = "$src_bytes" ]; then
+          verified=true
+          break
+        fi
+      fi
+
+      sleep "$WAIT_SLEEP_SECONDS"
+      attempt=$((attempt + 1))
+    done
+
+    if [ "$verified" != "true" ]; then
+      echo "download verification failed for $site/$BUCKET/$object" >&2
+      echo "expected checksum: $src_checksum" >&2
+      echo "expected bytes:    $src_bytes" >&2
+      exit 1
+    fi
+  done
+
+  echo "verified replicated downloads for $object_name"
+done
+
+echo "site replication object flow check passed"
+echo "bucket: $BUCKET"
+echo "prefix: $PREFIX"