diff --git a/.docker/compat/README.md b/.docker/test/compat/README.md similarity index 100% rename from .docker/compat/README.md rename to .docker/test/compat/README.md diff --git a/.docker/compat/docker-compose.minio.yml b/.docker/test/compat/docker-compose.minio.yml similarity index 100% rename from .docker/compat/docker-compose.minio.yml rename to .docker/test/compat/docker-compose.minio.yml diff --git a/.docker/compat/docker-compose.rustfs-beta5.yml b/.docker/test/compat/docker-compose.rustfs-beta5.yml similarity index 100% rename from .docker/compat/docker-compose.rustfs-beta5.yml rename to .docker/test/compat/docker-compose.rustfs-beta5.yml diff --git a/.docker/compat/docker-compose.rustfs-rio-v2.yml b/.docker/test/compat/docker-compose.rustfs-rio-v2.yml similarity index 100% rename from .docker/compat/docker-compose.rustfs-rio-v2.yml rename to .docker/test/compat/docker-compose.rustfs-rio-v2.yml diff --git a/.docker/compat/run_rw_compat_stress.sh b/.docker/test/compat/run_rw_compat_stress.sh similarity index 100% rename from .docker/compat/run_rw_compat_stress.sh rename to .docker/test/compat/run_rw_compat_stress.sh diff --git a/.docker/test/site-replication/README.md b/.docker/test/site-replication/README.md new file mode 100644 index 000000000..c753697ea --- /dev/null +++ b/.docker/test/site-replication/README.md @@ -0,0 +1,183 @@ +# Site Replication Docker Compose Test + +## Purpose + +This directory contains a local three-site RustFS site replication check. It is intended to verify the admin site-replication flow against real containers: + +- three independent RustFS sites start successfully +- the MinIO-compatible `mc admin replicate add` command configures all three sites +- a bucket and object written to site 1 are replicated to site 2 and site 3 +- `mc admin replicate status` can read the resulting site-replication state + +The compose file uses named volumes so the test does not require preparing host bind-mount directories. +Because Docker named volumes usually share the same physical device in local desktop environments, this test compose defaults `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true`. Keep that setting limited to local test and CI environments. + +## Files + +- `docker-compose.yml`: three RustFS sites, a volume permission helper, and a one-shot setup/check container +- `run-object-flow-check.sh`: host-side upload/download verification for replicated 10 MiB to 100 MiB objects + +## Ports + +Default host endpoints: + +- Site 1 API: `http://127.0.0.1:9000` +- Site 1 Console: `http://127.0.0.1:9001` +- Site 2 API: `http://127.0.0.1:9010` +- Site 2 Console: `http://127.0.0.1:9011` +- Site 3 API: `http://127.0.0.1:9020` +- Site 3 Console: `http://127.0.0.1:9021` + +Default credentials are `rustfsadmin` / `rustfsadmin`. These are for local testing only. + +## Run + +From the repository root: + +```bash +docker compose -f .docker/test/site-replication/docker-compose.yml up +``` + +To test a locally built image instead of Docker Hub `rustfs/rustfs:latest`, set `RUSTFS_SITE_REPL_IMAGE`: + +```bash +docker build -f Dockerfile.source -t rustfs-site-repl-local:latest . +RUSTFS_SITE_REPL_IMAGE=rustfs-site-repl-local:latest \ +docker compose -f .docker/test/site-replication/docker-compose.yml up +``` + +For a detached run: + +```bash +docker compose -f .docker/test/site-replication/docker-compose.yml up -d +docker compose -f .docker/test/site-replication/docker-compose.yml logs -f site-replication-setup +``` + +## Test Flow + +The compose stack performs these steps: + +1. `site-replication-volume-permission-helper` fixes ownership on all named volumes for the RustFS runtime user. +2. `rustfs-site1`, `rustfs-site2`, and `rustfs-site3` start as separate RustFS sites. +3. Each site exposes its S3 API and Console on a unique host port. +4. Health checks wait for `/health/ready` on each RustFS container. +5. `site-replication-setup` configures `mc` aliases for all three sites. +6. The setup container waits until `mc admin info` succeeds for all sites. +7. It runs: + +```bash +mc admin replicate add site1 site2 site3 +``` + +8. It creates the test bucket on site 1 and uploads `from-site1.txt`. +9. It polls site 2 and site 3 until the replicated object is visible. +10. It prints `mc admin replicate status site1`. + +The setup container exits with status `0` only after the object replication check passes. + +## Object Flow Check + +After the compose setup succeeds, run the larger object flow check from the repository root: + +```bash +.docker/test/site-replication/run-object-flow-check.sh +``` + +The script creates five local files and uploads them from different sites: + +- 10 MiB from site 1 +- 25 MiB from site 2 +- 50 MiB from site 3 +- 75 MiB from site 1 +- 100 MiB from site 2 + +For each uploaded object, the script waits for replication to the other two sites, downloads the object from those sites, and verifies both byte size and SHA-256 checksum. It uses a temporary `mc` config directory, so it does not overwrite existing host aliases. + +The default bucket is `site-repl-flow-check`. Override it when needed: + +```bash +RUSTFS_SITE_REPL_FLOW_BUCKET='site-repl-large-flow' \ +.docker/test/site-replication/run-object-flow-check.sh +``` + +The script keeps uploaded objects under a timestamped prefix. Override the prefix for repeatable runs: + +```bash +RUSTFS_SITE_REPL_FLOW_PREFIX='manual-check-001' \ +.docker/test/site-replication/run-object-flow-check.sh +``` + +If replication is slow on the local machine, increase polling: + +```bash +RUSTFS_SITE_REPL_WAIT_ATTEMPTS=180 \ +RUSTFS_SITE_REPL_WAIT_SLEEP_SECONDS=2 \ +.docker/test/site-replication/run-object-flow-check.sh +``` + +## Optional Settings + +Override local test credentials: + +```bash +RUSTFS_SITE_REPL_ACCESS_KEY='localadmin' \ +RUSTFS_SITE_REPL_SECRET_KEY='localadmin-secret' \ +docker compose -f .docker/test/site-replication/docker-compose.yml up +``` + +Use a different test bucket: + +```bash +RUSTFS_SITE_REPL_BUCKET='site-repl-check' \ +docker compose -f .docker/test/site-replication/docker-compose.yml up +``` + +Enable ILM expiry rule replication during site setup: + +```bash +RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY=true \ +docker compose -f .docker/test/site-replication/docker-compose.yml up +``` + +Use this only when the test needs lifecycle expiry metadata included in site replication. + +## Manual Checks + +After the setup container succeeds, you can inspect the sites with `mc` from the host: + +```bash +mc alias set site1 http://127.0.0.1:9000 rustfsadmin rustfsadmin +mc alias set site2 http://127.0.0.1:9010 rustfsadmin rustfsadmin +mc alias set site3 http://127.0.0.1:9020 rustfsadmin rustfsadmin + +mc admin replicate info site1 +mc admin replicate status site1 +mc stat site2/site-repl-demo/from-site1.txt +mc stat site3/site-repl-demo/from-site1.txt +``` + +After larger object flow checks, replication should converge without a growing queue: + +```bash +mc admin replicate status site1 +mc admin replicate status site2 +mc admin replicate status site3 +``` + +Useful Docker commands: + +```bash +docker compose -f .docker/test/site-replication/docker-compose.yml ps +docker compose -f .docker/test/site-replication/docker-compose.yml logs --no-color --tail=200 +docker compose -f .docker/test/site-replication/docker-compose.yml logs --no-color site-replication-setup +``` + +## Cleanup + +Remove containers and named volumes: + +```bash +docker compose -f .docker/test/site-replication/docker-compose.yml down -v +``` + +Use `down -v` before rerunning the full setup from scratch. Site replication state is persisted in the named volumes, so rerunning without deleting volumes may attempt to add an already-configured replication topology. diff --git a/.docker/test/site-replication/docker-compose.yml b/.docker/test/site-replication/docker-compose.yml new file mode 100644 index 000000000..fa5a4cd11 --- /dev/null +++ b/.docker/test/site-replication/docker-compose.yml @@ -0,0 +1,243 @@ +# Copyright 2024 RustFS Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +services: + rustfs-site1: + image: ${RUSTFS_SITE_REPL_IMAGE:-rustfs/rustfs:latest} + container_name: rustfs-site-repl-1 + security_opt: + - "no-new-privileges:true" + ports: + - "9000:9000" + - "9001:9001" + environment: + - RUSTFS_VOLUMES=/data/rustfs{0...3} + - RUSTFS_ADDRESS=0.0.0.0:9000 + - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001 + - RUSTFS_CONSOLE_ENABLE=true + - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=* + - RUSTFS_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin} + - RUSTFS_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin} + - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_SITE_REPL_LOG_LEVEL:-info} + - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true} + volumes: + - site1_data_0:/data/rustfs0 + - site1_data_1:/data/rustfs1 + - site1_data_2:/data/rustfs2 + - site1_data_3:/data/rustfs3 + - site1_logs:/app/logs + networks: + - rustfs-site-replication + restart: unless-stopped + healthcheck: + test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health/ready"] + interval: 10s + timeout: 5s + retries: 30 + start_period: 20s + depends_on: + site-replication-volume-permission-helper: + condition: service_completed_successfully + + rustfs-site2: + image: ${RUSTFS_SITE_REPL_IMAGE:-rustfs/rustfs:latest} + container_name: rustfs-site-repl-2 + security_opt: + - "no-new-privileges:true" + ports: + - "9010:9000" + - "9011:9001" + environment: + - RUSTFS_VOLUMES=/data/rustfs{0...3} + - RUSTFS_ADDRESS=0.0.0.0:9000 + - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001 + - RUSTFS_CONSOLE_ENABLE=true + - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=* + - RUSTFS_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin} + - RUSTFS_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin} + - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_SITE_REPL_LOG_LEVEL:-info} + - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true} + volumes: + - site2_data_0:/data/rustfs0 + - site2_data_1:/data/rustfs1 + - site2_data_2:/data/rustfs2 + - site2_data_3:/data/rustfs3 + - site2_logs:/app/logs + networks: + - rustfs-site-replication + restart: unless-stopped + healthcheck: + test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health/ready"] + interval: 10s + timeout: 5s + retries: 30 + start_period: 20s + depends_on: + site-replication-volume-permission-helper: + condition: service_completed_successfully + + rustfs-site3: + image: ${RUSTFS_SITE_REPL_IMAGE:-rustfs/rustfs:latest} + container_name: rustfs-site-repl-3 + security_opt: + - "no-new-privileges:true" + ports: + - "9020:9000" + - "9021:9001" + environment: + - RUSTFS_VOLUMES=/data/rustfs{0...3} + - RUSTFS_ADDRESS=0.0.0.0:9000 + - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001 + - RUSTFS_CONSOLE_ENABLE=true + - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=* + - RUSTFS_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin} + - RUSTFS_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin} + - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_SITE_REPL_LOG_LEVEL:-info} + - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true} + volumes: + - site3_data_0:/data/rustfs0 + - site3_data_1:/data/rustfs1 + - site3_data_2:/data/rustfs2 + - site3_data_3:/data/rustfs3 + - site3_logs:/app/logs + networks: + - rustfs-site-replication + restart: unless-stopped + healthcheck: + test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health/ready"] + interval: 10s + timeout: 5s + retries: 30 + start_period: 20s + depends_on: + site-replication-volume-permission-helper: + condition: service_completed_successfully + + site-replication-setup: + image: minio/mc:latest + container_name: rustfs-site-repl-setup + depends_on: + rustfs-site1: + condition: service_healthy + rustfs-site2: + condition: service_healthy + rustfs-site3: + condition: service_healthy + environment: + - RUSTFS_SITE_REPL_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin} + - RUSTFS_SITE_REPL_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin} + - RUSTFS_SITE_REPL_BUCKET=${RUSTFS_SITE_REPL_BUCKET:-site-repl-demo} + - RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY=${RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY:-false} + networks: + - rustfs-site-replication + entrypoint: ["/bin/sh", "-c"] + command: + - | + set -eu + + access_key="$${RUSTFS_SITE_REPL_ACCESS_KEY}" + secret_key="$${RUSTFS_SITE_REPL_SECRET_KEY}" + bucket="$${RUSTFS_SITE_REPL_BUCKET}" + + mc alias set site1 http://rustfs-site1:9000 "$${access_key}" "$${secret_key}" + mc alias set site2 http://rustfs-site2:9000 "$${access_key}" "$${secret_key}" + mc alias set site3 http://rustfs-site3:9000 "$${access_key}" "$${secret_key}" + + for site in site1 site2 site3; do + until mc ls "$${site}" >/dev/null 2>&1; do + echo "waiting for $${site} S3 API" + sleep 2 + done + done + + ilm_flag="" + if [ "$${RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY}" = "true" ]; then + ilm_flag="--replicate-ilm-expiry" + fi + + echo "configuring 3-site replication" + if ! mc admin replicate add site1 site2 site3 $${ilm_flag}; then + echo "replicate add failed; showing current replication info before exiting" + mc admin replicate info site1 || true + exit 1 + fi + + mc mb --ignore-existing "site1/$${bucket}" + printf 'rustfs 3-site replication check\n' > /tmp/site-replication-check.txt + mc cp /tmp/site-replication-check.txt "site1/$${bucket}/from-site1.txt" + + for site in site2 site3; do + for attempt in $$(seq 1 60); do + if mc stat "$${site}/$${bucket}/from-site1.txt" >/dev/null 2>&1; then + echo "$${site} received replicated object" + break + fi + if [ "$${attempt}" = "60" ]; then + echo "$${site} did not receive replicated object in time" + exit 1 + fi + sleep 2 + done + done + + mc admin replicate status site1 + echo "3-site replication example is ready" + restart: "no" + + site-replication-volume-permission-helper: + image: alpine:3.23.4 + volumes: + - site1_data_0:/site1/data0 + - site1_data_1:/site1/data1 + - site1_data_2:/site1/data2 + - site1_data_3:/site1/data3 + - site1_logs:/site1/logs + - site2_data_0:/site2/data0 + - site2_data_1:/site2/data1 + - site2_data_2:/site2/data2 + - site2_data_3:/site2/data3 + - site2_logs:/site2/logs + - site3_data_0:/site3/data0 + - site3_data_1:/site3/data1 + - site3_data_2:/site3/data2 + - site3_data_3:/site3/data3 + - site3_logs:/site3/logs + command: > + sh -c " + chown -R 10001:10001 /site1 /site2 /site3 && + echo 'site replication volume permissions fixed' + " + networks: + - rustfs-site-replication + restart: "no" + +networks: + rustfs-site-replication: + +volumes: + site1_data_0: + site1_data_1: + site1_data_2: + site1_data_3: + site1_logs: + site2_data_0: + site2_data_1: + site2_data_2: + site2_data_3: + site2_logs: + site3_data_0: + site3_data_1: + site3_data_2: + site3_data_3: + site3_logs: diff --git a/.docker/test/site-replication/run-object-flow-check.sh b/.docker/test/site-replication/run-object-flow-check.sh new file mode 100755 index 000000000..f351e2587 --- /dev/null +++ b/.docker/test/site-replication/run-object-flow-check.sh @@ -0,0 +1,191 @@ +#!/bin/sh +# Copyright 2024 RustFS Team +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. + +set -eu + +ACCESS_KEY="${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}" +SECRET_KEY="${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}" +BUCKET="${RUSTFS_SITE_REPL_FLOW_BUCKET:-site-repl-flow-check}" +PREFIX="${RUSTFS_SITE_REPL_FLOW_PREFIX:-flow-$(date +%Y%m%d-%H%M%S)}" +WAIT_ATTEMPTS="${RUSTFS_SITE_REPL_WAIT_ATTEMPTS:-90}" +WAIT_SLEEP_SECONDS="${RUSTFS_SITE_REPL_WAIT_SLEEP_SECONDS:-2}" + +SITE1_ENDPOINT="${RUSTFS_SITE1_ENDPOINT:-http://127.0.0.1:9000}" +SITE2_ENDPOINT="${RUSTFS_SITE2_ENDPOINT:-http://127.0.0.1:9010}" +SITE3_ENDPOINT="${RUSTFS_SITE3_ENDPOINT:-http://127.0.0.1:9020}" + +command_exists() { + command -v "$1" >/dev/null 2>&1 +} + +require_command() { + if ! command_exists "$1"; then + echo "missing required command: $1" >&2 + exit 1 + fi +} + +checksum_file() { + if command_exists sha256sum; then + sha256sum "$1" | awk '{print $1}' + elif command_exists shasum; then + shasum -a 256 "$1" | awk '{print $1}' + else + echo "missing required command: sha256sum or shasum" >&2 + exit 1 + fi +} + +site_endpoint() { + case "$1" in + site1) printf '%s\n' "$SITE1_ENDPOINT" ;; + site2) printf '%s\n' "$SITE2_ENDPOINT" ;; + site3) printf '%s\n' "$SITE3_ENDPOINT" ;; + *) echo "unknown site alias: $1" >&2; exit 1 ;; + esac +} + +other_sites() { + case "$1" in + site1) printf '%s\n' "site2 site3" ;; + site2) printf '%s\n' "site1 site3" ;; + site3) printf '%s\n' "site1 site2" ;; + *) echo "unknown source site: $1" >&2; exit 1 ;; + esac +} + +wait_for_object() { + site="$1" + object="$2" + attempt=1 + + while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do + if mc stat "$site/$BUCKET/$object" >/dev/null 2>&1; then + return 0 + fi + sleep "$WAIT_SLEEP_SECONDS" + attempt=$((attempt + 1)) + done + + echo "object was not replicated in time: $site/$BUCKET/$object" >&2 + return 1 +} + +wait_for_bucket() { + site="$1" + attempt=1 + + while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do + if mc stat "$site/$BUCKET" >/dev/null 2>&1; then + return 0 + fi + sleep "$WAIT_SLEEP_SECONDS" + attempt=$((attempt + 1)) + done + + echo "bucket was not replicated in time: $site/$BUCKET" >&2 + return 1 +} + +require_command mc +require_command dd +require_command awk +require_command date +require_command mktemp +require_command tr +require_command wc + +WORK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/rustfs-site-repl-flow.XXXXXX")" +MC_CONFIG_DIR="$WORK_DIR/mc" +export MC_CONFIG_DIR + +cleanup() { + rm -rf "$WORK_DIR" +} +trap cleanup EXIT INT TERM + +mkdir -p "$MC_CONFIG_DIR" "$WORK_DIR/src" "$WORK_DIR/downloads" + +for site in site1 site2 site3; do + mc alias set "$site" "$(site_endpoint "$site")" "$ACCESS_KEY" "$SECRET_KEY" >/dev/null +done + +for site in site1 site2 site3; do + echo "checking S3 API for $site" + mc ls "$site" >/dev/null +done + +echo "ensuring bucket exists on site1: $BUCKET" +mc mb --ignore-existing "site1/$BUCKET" >/dev/null + +for site in site1 site2 site3; do + wait_for_bucket "$site" +done + +cat <<'EOF' | while read -r size_mb source_site object_name; do +10 site1 object-010m.bin +25 site2 object-025m.bin +50 site3 object-050m.bin +75 site1 object-075m.bin +100 site2 object-100m.bin +EOF + src_file="$WORK_DIR/src/$object_name" + object="$PREFIX/$object_name" + + echo "creating ${size_mb}MiB file: $object_name" + dd if=/dev/urandom of="$src_file" bs=1048576 count="$size_mb" >/dev/null 2>&1 + src_checksum="$(checksum_file "$src_file")" + src_bytes="$(wc -c < "$src_file" | tr -d ' ')" + + echo "uploading $object_name to $source_site ($src_bytes bytes)" + mc cp "$src_file" "$source_site/$BUCKET/$object" >/dev/null + + for site in $(other_sites "$source_site"); do + wait_for_object "$site" "$object" + + dst_file="$WORK_DIR/downloads/$site-$object_name" + verified=false + attempt=1 + while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do + rm -f "$dst_file" + echo "downloading $object_name from $site" + if mc cp "$site/$BUCKET/$object" "$dst_file" >/dev/null 2>&1; then + dst_checksum="$(checksum_file "$dst_file")" + dst_bytes="$(wc -c < "$dst_file" | tr -d ' ')" + + if [ "$dst_checksum" = "$src_checksum" ] && [ "$dst_bytes" = "$src_bytes" ]; then + verified=true + break + fi + fi + + sleep "$WAIT_SLEEP_SECONDS" + attempt=$((attempt + 1)) + done + + if [ "$verified" != "true" ]; then + echo "download verification failed for $site/$BUCKET/$object" >&2 + echo "expected checksum: $src_checksum" >&2 + echo "expected bytes: $src_bytes" >&2 + exit 1 + fi + done + + echo "verified replicated downloads for $object_name" +done + +echo "site replication object flow check passed" +echo "bucket: $BUCKET" +echo "prefix: $PREFIX" diff --git a/.gitignore b/.gitignore index 967ed3483..32ee4889f 100644 --- a/.gitignore +++ b/.gitignore @@ -55,8 +55,10 @@ docs/heal-scanner-logging-governance.md docs/benchmark/rustfs-target-bench/ docs/benchmark/*.md .codegraph/* -.docker/compat/data/* -.docker/compat/kms/* +.docker/test/compat/data/* +.docker/test/compat/kms/* +reasonix.toml +.reasonix/* # nix stuff result* diff --git a/crates/ecstore/src/bucket/replication/replication_resyncer.rs b/crates/ecstore/src/bucket/replication/replication_resyncer.rs index 0f8f6fb7a..112fa6104 100644 --- a/crates/ecstore/src/bucket/replication/replication_resyncer.rs +++ b/crates/ecstore/src/bucket/replication/replication_resyncer.rs @@ -4008,6 +4008,11 @@ async fn replicate_object_with_multipart(ctx: MultipartR uploaded_parts, &PutObjectOptions { user_metadata, + internal: AdvancedPutOptions { + replication_status: ReplicationStatusType::Replica, + replication_request: true, + ..Default::default() + }, ..Default::default() }, ) diff --git a/crates/ecstore/src/object_api/types.rs b/crates/ecstore/src/object_api/types.rs index c6ce799a5..abd0edaf0 100644 --- a/crates/ecstore/src/object_api/types.rs +++ b/crates/ecstore/src/object_api/types.rs @@ -109,6 +109,14 @@ impl ObjectOptions { } pub fn put_replication_state(&self) -> ReplicationState { + if self + .delete_replication + .as_ref() + .is_some_and(|state| !state.replica_status.is_empty()) + { + return self.delete_replication.clone().unwrap_or_default(); + } + let rs = match rustfs_utils::http::get_str(&self.user_defined, rustfs_utils::http::SUFFIX_REPLICATION_STATUS) { Some(v) => v, None => return ReplicationState::default(), @@ -797,6 +805,21 @@ mod tests { assert_eq!(versions[0].version_id, Some(last_version)); } + #[test] + fn put_replication_state_preserves_replica_status() { + let opts = ObjectOptions { + delete_replication: Some(ReplicationState { + replica_status: ReplicationStatusType::Replica, + ..Default::default() + }), + ..Default::default() + }; + + let state = opts.put_replication_state(); + + assert_eq!(state.composite_replication_status(), ReplicationStatusType::Replica); + } + #[test] fn versions_after_marker_handles_uuid_version_marker() { let first_version = Uuid::parse_str("11111111-2222-3333-4444-555555555555").unwrap(); diff --git a/rustfs/src/app/multipart_usecase.rs b/rustfs/src/app/multipart_usecase.rs index 3904d12fa..b179f718e 100644 --- a/rustfs/src/app/multipart_usecase.rs +++ b/rustfs/src/app/multipart_usecase.rs @@ -63,7 +63,7 @@ use crate::table_catalog; use bytes::Bytes; use futures::StreamExt; use http::{HeaderMap, Uri}; -use rustfs_filemeta::{ReplicationStatusType, ReplicationType}; +use rustfs_filemeta::ReplicationType; use rustfs_s3_ops::S3Operation; use rustfs_targets::EventName; use rustfs_utils::CompressionAlgorithm; @@ -560,8 +560,13 @@ impl DefaultMultipartUsecase { ..Default::default() }; let mt2 = obj_info.user_defined.clone(); - let replicate_options = - get_must_replicate_options(&mt2, "".to_string(), ReplicationStatusType::Empty, ReplicationType::Object, opts.clone()); + let replicate_options = get_must_replicate_options( + &mt2, + "".to_string(), + opts.delete_marker_replication_status(), + ReplicationType::Object, + opts.clone(), + ); let dsc = must_replicate(&bucket, &key, replicate_options).await; if dsc.replicate_any() { @@ -694,8 +699,13 @@ impl DefaultMultipartUsecase { .await .map_err(ApiError::from)?; - let replicate_options = - get_must_replicate_options(&mt2, "".to_string(), ReplicationStatusType::Empty, ReplicationType::Object, opts.clone()); + let replicate_options = get_must_replicate_options( + &mt2, + "".to_string(), + opts.delete_marker_replication_status(), + ReplicationType::Object, + opts.clone(), + ); let dsc = must_replicate(&bucket, &key, replicate_options).await; if dsc.replicate_any() { insert_str(&mut opts.user_defined, SUFFIX_REPLICATION_TIMESTAMP, jiff::Zoned::now().to_string()); diff --git a/rustfs/src/app/object_usecase.rs b/rustfs/src/app/object_usecase.rs index 5c8ff5c78..8b8659a09 100644 --- a/rustfs/src/app/object_usecase.rs +++ b/rustfs/src/app/object_usecase.rs @@ -2994,8 +2994,13 @@ impl DefaultObjectUsecase { .map(|ctx| ctx.request_id.clone()) .unwrap_or_else(|| request_context::RequestContext::fallback().request_id); - let repoptions = - get_must_replicate_options(&mt2, "".to_string(), ReplicationStatusType::Empty, ReplicationType::Object, opts.clone()); + let repoptions = get_must_replicate_options( + &mt2, + "".to_string(), + opts.delete_marker_replication_status(), + ReplicationType::Object, + opts.clone(), + ); let dsc = must_replicate(&bucket, &key, repoptions).await; if dsc.replicate_any() { @@ -3105,8 +3110,13 @@ impl DefaultObjectUsecase { let e_tag = obj_info.etag.clone().map(|etag| to_s3s_etag(&etag)); - let repoptions = - get_must_replicate_options(&mt2, "".to_string(), ReplicationStatusType::Empty, ReplicationType::Object, opts); + let repoptions = get_must_replicate_options( + &mt2, + "".to_string(), + opts.delete_marker_replication_status(), + ReplicationType::Object, + opts, + ); let dsc = must_replicate(&bucket, &key, repoptions).await; let expiration = resolve_put_object_expiration(&bucket, &obj_info).await; diff --git a/rustfs/src/storage/options.rs b/rustfs/src/storage/options.rs index 337853b48..01c1152e1 100644 --- a/rustfs/src/storage/options.rs +++ b/rustfs/src/storage/options.rs @@ -16,15 +16,16 @@ use super::{BucketVersioningSys, Result, StorageError}; use crate::storage::storage_api::options_consumer::contract::{object::HTTPPreconditions, range::HTTPRangeSpec}; use http::header::{IF_MATCH, IF_NONE_MATCH}; use http::{HeaderMap, HeaderValue}; +use rustfs_filemeta::ReplicationStatusType; +use rustfs_utils::http::{ + AMZ_BUCKET_REPLICATION_STATUS, SUFFIX_FORCE_DELETE, SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE, SUFFIX_REPLICATION_SSEC_CRC, + SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_REQUEST, SUFFIX_SOURCE_VERSION_ID, get_header, + insert_header_map, is_encryption_metadata_key, is_internal_key, +}; use rustfs_utils::http::{ AMZ_META_UNENCRYPTED_CONTENT_LENGTH, AMZ_META_UNENCRYPTED_CONTENT_MD5, AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, AMZ_OBJECT_LOCK_MODE_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER, }; -use rustfs_utils::http::{ - SUFFIX_FORCE_DELETE, SUFFIX_REPLICATION_ACTUAL_OBJECT_SIZE, SUFFIX_REPLICATION_SSEC_CRC, SUFFIX_SOURCE_DELETEMARKER, - SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_REQUEST, SUFFIX_SOURCE_VERSION_ID, get_header, insert_header_map, - is_encryption_metadata_key, is_internal_key, -}; use s3s::header::X_AMZ_OBJECT_LOCK_MODE; use s3s::header::X_AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE; @@ -275,6 +276,7 @@ pub fn get_complete_multipart_upload_opts(headers: &HeaderMap) -> s replication_request, ..Default::default() }; + apply_replica_status_from_headers(headers, &mut opts); fill_conditional_writes_opts_from_header(headers, &mut opts)?; Ok(opts) @@ -297,6 +299,7 @@ pub fn copy_src_opts(_bucket: &str, _object: &str, headers: &HeaderMap, metadata: HashMap) -> Result { let mut opts = get_default_opts(headers, metadata, false)?; + apply_replica_status_from_headers(headers, &mut opts); if get_header(headers, SUFFIX_SOURCE_REPLICATION_REQUEST).as_deref() == Some("true") { opts.replication_request = true; if let Some(v) = get_header(headers, SUFFIX_SOURCE_MTIME) { @@ -313,6 +316,16 @@ pub fn put_opts_from_headers(headers: &HeaderMap, metadata: HashMap Ok(opts) } +fn apply_replica_status_from_headers(headers: &HeaderMap, opts: &mut ObjectOptions) { + if headers + .get(AMZ_BUCKET_REPLICATION_STATUS) + .and_then(|value| value.to_str().ok()) + .is_some_and(|status| status.eq_ignore_ascii_case(ReplicationStatusType::Replica.as_str())) + { + opts.set_replica_status(ReplicationStatusType::Replica); + } +} + /// Creates default options for getting an object from a bucket. pub fn get_default_opts( _headers: &HeaderMap, @@ -788,13 +801,15 @@ mod tests { use super::{ ENV_REJECT_ARCHIVE_CONTENT_ENCODING, SUPPORTED_HEADERS, copy_dst_opts, copy_src_opts, del_opts, detect_content_type_from_object_name, extract_metadata, extract_metadata_from_mime, - extract_metadata_from_mime_with_object_name, filter_object_metadata, get_default_opts, get_opts, parse_copy_source_range, - put_opts, put_opts_from_headers, validate_archive_content_encoding, + extract_metadata_from_mime_with_object_name, filter_object_metadata, get_complete_multipart_upload_opts, + get_default_opts, get_opts, parse_copy_source_range, put_opts, put_opts_from_headers, validate_archive_content_encoding, }; use http::{HeaderMap, HeaderValue}; + use rustfs_filemeta::ReplicationStatusType; use rustfs_utils::http::{ - AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, AMZ_OBJECT_LOCK_MODE_LOWER, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER, - SUFFIX_FORCE_DELETE, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_REQUEST, insert_header, + AMZ_BUCKET_REPLICATION_STATUS, AMZ_OBJECT_LOCK_LEGAL_HOLD_LOWER, AMZ_OBJECT_LOCK_MODE_LOWER, + AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE_LOWER, SUFFIX_FORCE_DELETE, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_REQUEST, + insert_header, }; use s3s::S3ErrorCode; use std::collections::HashMap; @@ -1126,6 +1141,32 @@ mod tests { assert!(opts_invalid.mod_time.is_none()); } + #[test] + fn test_put_opts_from_headers_with_replica_status() { + let mut headers = HeaderMap::new(); + headers.insert( + AMZ_BUCKET_REPLICATION_STATUS, + HeaderValue::from_static(ReplicationStatusType::Replica.as_str()), + ); + + let opts = put_opts_from_headers(&headers, HashMap::new()).expect("replica status header should parse"); + + assert_eq!(opts.delete_marker_replication_status(), ReplicationStatusType::Replica); + } + + #[test] + fn test_complete_multipart_opts_with_replica_status() { + let mut headers = HeaderMap::new(); + headers.insert( + AMZ_BUCKET_REPLICATION_STATUS, + HeaderValue::from_static(ReplicationStatusType::Replica.as_str()), + ); + + let opts = get_complete_multipart_upload_opts(&headers).expect("replica status header should parse"); + + assert_eq!(opts.delete_marker_replication_status(), ReplicationStatusType::Replica); + } + #[test] fn test_get_default_opts_with_metadata() { let headers = create_test_headers();