mirror of
https://github.com/rustfs/rustfs.git
synced 2026-10-29 07:17:52 +00:00
Compare commits
51 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 1afaf4491e | |||
| bdea44f332 | |||
| 8ba3309624 | |||
| cd2fa50e50 | |||
| 077b961232 | |||
| e1bc731331 | |||
| bb43ef861a | |||
| 4047ed6c8d | |||
| ae6bbaef27 | |||
| a7c875b48f | |||
| 42dd76d3b2 | |||
| 9aeecad4a9 | |||
| 3738b32df1 | |||
| d997b07505 | |||
| 0632a64f91 | |||
| cbf6b02905 | |||
| ed2c3eaf77 | |||
| 22243e791d | |||
| c35de00e8a | |||
| 0e03bcda49 | |||
| c3269ad4c2 | |||
| 0fcd34c331 | |||
| ddc5f97a04 | |||
| ff62c810b4 | |||
| 950db7412c | |||
| 4e7a8fabb8 | |||
| b0f4e66c92 | |||
| e9da1fa074 | |||
| 023adb1696 | |||
| 0f69c073de | |||
| 7a5e1efa95 | |||
| aa7f55946c | |||
| 785b44a696 | |||
| 415bd2310c | |||
| 3f469ab257 | |||
| 31f446bed1 | |||
| d2cdf057cc | |||
| 717e8b9c87 | |||
| 5886368178 | |||
| db4b8290e7 | |||
| 5829c66f21 | |||
| 9c199aefd1 | |||
| 4c02ad4025 | |||
| e31a22febb | |||
| 91a6c5bad8 | |||
| a58ac62618 | |||
| 36c6c2e34b | |||
| d6729a1138 | |||
| 07a9d15ab2 | |||
| d9c3eeea0c | |||
| 892d3ba3f7 |
@@ -101,6 +101,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
|
||||
### SSE and on-disk storage invariants
|
||||
|
||||
- `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
|
||||
- `GHSA-wqmc-vjgv-jrpw`: SSE-C persisted and listed the plaintext MD5 as the ETag, allowing list-only users to confirm or brute-force low-entropy content and correlate equal plaintext across keys. Lesson: encrypted-object ETags must not expose deterministic plaintext fingerprints; protect them with per-object key material and keep listing, conditional, copy, and multipart semantics consistent.
|
||||
|
||||
### Object Lock and retention invariants
|
||||
|
||||
@@ -126,7 +127,7 @@ rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonical
|
||||
rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
|
||||
rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
|
||||
rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
|
||||
rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
|
||||
rg -n "HashReader|EncryptReader|try_resolve_etag|ETag|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
|
||||
rg -n "ObjectLock|object_lock|retention|COMPLIANCE|GOVERNANCE|delete_prefix|lifecycle|scanner" rustfs crates
|
||||
rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
|
||||
```
|
||||
@@ -145,5 +146,5 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
|
||||
- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
|
||||
- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
|
||||
- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
|
||||
- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.
|
||||
- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together; for ETags, upload equal plaintext under different keys and assert list-only callers cannot derive or correlate the plaintext digest, including multipart objects.
|
||||
- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths.
|
||||
|
||||
@@ -29,6 +29,7 @@ script-tests: ## Run shell script tests
|
||||
./scripts/test_entrypoint_credentials.sh
|
||||
./scripts/test_internode_grpc_ab_bench.sh
|
||||
./scripts/test_object_batch_bench_enhanced.sh
|
||||
./scripts/test_package_service_scripts.sh
|
||||
./scripts/test_hotpath_warp_ab_gate.sh
|
||||
./scripts/test_hotpath_warp_abba.sh
|
||||
./scripts/test_scanner_validation_harness.sh
|
||||
|
||||
@@ -24,7 +24,9 @@ on:
|
||||
- '.github/actions/**'
|
||||
- '.github/workflows/**'
|
||||
- 'scripts/release/create_or_update_release.sh'
|
||||
- 'scripts/release/package_service_scripts.sh'
|
||||
- 'scripts/release/package_versions.sh'
|
||||
- 'scripts/test_package_service_scripts.sh'
|
||||
- 'scripts/test_package_versions.sh'
|
||||
- 'scripts/security/check_performance_ab_workflow.sh'
|
||||
- 'scripts/security/check_preview_release_workflow.sh'
|
||||
@@ -40,7 +42,9 @@ on:
|
||||
- '.github/actions/**'
|
||||
- '.github/workflows/**'
|
||||
- 'scripts/release/create_or_update_release.sh'
|
||||
- 'scripts/release/package_service_scripts.sh'
|
||||
- 'scripts/release/package_versions.sh'
|
||||
- 'scripts/test_package_service_scripts.sh'
|
||||
- 'scripts/test_package_versions.sh'
|
||||
- 'scripts/security/check_performance_ab_workflow.sh'
|
||||
- 'scripts/security/check_preview_release_workflow.sh'
|
||||
@@ -158,6 +162,9 @@ jobs:
|
||||
- name: Check package version contract
|
||||
run: ./scripts/test_package_versions.sh
|
||||
|
||||
- name: Check package service lifecycle contract
|
||||
run: ./scripts/test_package_service_scripts.sh
|
||||
|
||||
dependency-review:
|
||||
name: Dependency Review
|
||||
runs-on: ubuntu-latest
|
||||
|
||||
@@ -542,6 +542,54 @@ jobs:
|
||||
if-no-files-found: error
|
||||
retention-days: 30
|
||||
|
||||
# Docker Hub's overview is a separate `full_description` field that
|
||||
# `docker push` never touches, so it drifted from README.md and kept
|
||||
# linking to pages that no longer exist (#7995). Republish it from the
|
||||
# commit the images were built from, only when images were pushed.
|
||||
sync-dockerhub-description:
|
||||
name: Sync Docker Hub Description
|
||||
needs: [ build-check, build-docker ]
|
||||
if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
|
||||
runs-on: ubuntu-latest
|
||||
timeout-minutes: 10
|
||||
steps:
|
||||
- name: Checkout repository
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
with:
|
||||
persist-credentials: false
|
||||
ref: ${{ needs.build-check.outputs.source_ref }}
|
||||
sparse-checkout: README.md
|
||||
sparse-checkout-cone-mode: false
|
||||
|
||||
- name: Sync Docker Hub description
|
||||
uses: peter-evans/dockerhub-description@1b9a80c056b620d92cedb9d9b5a223409c68ddfa # v5.0.0
|
||||
with:
|
||||
username: ${{ env.DOCKERHUB_USERNAME }}
|
||||
password: ${{ secrets.DOCKERHUB_TOKEN }}
|
||||
repository: ${{ env.REGISTRY_DOCKERHUB }}
|
||||
readme-filepath: ./README.md
|
||||
# README.md links to docs/ and CONTRIBUTING.md by relative path;
|
||||
# rewrite those to github.com URLs so they resolve on Docker Hub.
|
||||
enable-url-completion: true
|
||||
|
||||
# Docker Hub caps `full_description` at 25,000 bytes and the action
|
||||
# truncates to fit with only a warning, which would publish a README
|
||||
# cut off mid-sentence. README.md is already near the cap, so read the
|
||||
# published overview back and fail if it hit the limit.
|
||||
- name: Check published description was not truncated
|
||||
env:
|
||||
REPOSITORY: ${{ env.REGISTRY_DOCKERHUB }}
|
||||
run: |
|
||||
limit=25000
|
||||
published=$(curl -sSf "https://hub.docker.com/v2/repositories/${REPOSITORY}/" \
|
||||
| jq -j '.full_description' | wc -c | tr -d ' ')
|
||||
echo "Published overview: ${published} bytes (Docker Hub limit: ${limit})"
|
||||
# Truncation lands within one UTF-8 character of the cap.
|
||||
if [ "$published" -ge $((limit - 4)) ]; then
|
||||
echo "::error file=README.md::Docker Hub overview was truncated at ${limit} bytes; trim README.md before the next release."
|
||||
exit 1
|
||||
fi
|
||||
|
||||
# Docker build summary
|
||||
docker-summary:
|
||||
name: Docker Build Summary
|
||||
|
||||
@@ -117,10 +117,15 @@ jobs:
|
||||
|
||||
cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
|
||||
# RustFS Environment Configuration
|
||||
# See https://rustfs.com/docs/ for more information
|
||||
# See https://docs.rustfs.com for more information
|
||||
# RUSTFS_ACCESS_KEY=<changeme>
|
||||
# RUSTFS_SECRET_KEY=<changeme>
|
||||
# RUSTFS_VOLUMES=""
|
||||
# RUSTFS_ROOT_USER=""
|
||||
# RUSTFS_ROOT_PASSWORD=""
|
||||
# RUSTFS_ADDRESS=":9000"
|
||||
# RUSTFS_CONSOLE_ADDRESS=":9001"
|
||||
# RUSTFS_CONSOLE_ENABLE=true
|
||||
# RUSTFS_OBS_LOGGER_LEVEL=error
|
||||
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
|
||||
ENVEOF
|
||||
|
||||
# dpkg versions must start with a digit and cannot contain hyphens;
|
||||
@@ -146,37 +151,13 @@ jobs:
|
||||
/etc/default/rustfs
|
||||
CONFFILES
|
||||
|
||||
cat > "${PKG_DIR}/DEBIAN/postinst" << 'POSTINST'
|
||||
#!/bin/bash
|
||||
set -e
|
||||
if ! getent passwd rustfs > /dev/null 2>&1; then
|
||||
useradd -r -s /bin/false -d /opt/rustfs rustfs
|
||||
fi
|
||||
mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
|
||||
chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
|
||||
if [ -d /run/systemd/system ]; then
|
||||
systemctl daemon-reload
|
||||
fi
|
||||
echo "RustFS installed. Configure /etc/default/rustfs then: systemctl start rustfs"
|
||||
POSTINST
|
||||
./scripts/release/package_service_scripts.sh after-install > "${PKG_DIR}/DEBIAN/postinst"
|
||||
chmod 755 "${PKG_DIR}/DEBIAN/postinst"
|
||||
|
||||
cat > "${PKG_DIR}/DEBIAN/prerm" << 'PRERM'
|
||||
#!/bin/bash
|
||||
set -e
|
||||
if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
|
||||
systemctl stop rustfs
|
||||
fi
|
||||
PRERM
|
||||
./scripts/release/package_service_scripts.sh before-remove > "${PKG_DIR}/DEBIAN/prerm"
|
||||
chmod 755 "${PKG_DIR}/DEBIAN/prerm"
|
||||
|
||||
cat > "${PKG_DIR}/DEBIAN/postrm" << 'POSTRM'
|
||||
#!/bin/bash
|
||||
set -e
|
||||
if [ -d /run/systemd/system ]; then
|
||||
systemctl daemon-reload
|
||||
fi
|
||||
POSTRM
|
||||
./scripts/release/package_service_scripts.sh after-remove > "${PKG_DIR}/DEBIAN/postrm"
|
||||
chmod 755 "${PKG_DIR}/DEBIAN/postrm"
|
||||
|
||||
cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
|
||||
@@ -214,10 +195,15 @@ jobs:
|
||||
mkdir -p ./tmp-pkg/etc/default
|
||||
cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
|
||||
# RustFS Environment Configuration
|
||||
# See https://rustfs.com/docs/ for more information
|
||||
# See https://docs.rustfs.com for more information
|
||||
# RUSTFS_ACCESS_KEY=<changeme>
|
||||
# RUSTFS_SECRET_KEY=<changeme>
|
||||
# RUSTFS_VOLUMES=""
|
||||
# RUSTFS_ROOT_USER=""
|
||||
# RUSTFS_ROOT_PASSWORD=""
|
||||
# RUSTFS_ADDRESS=":9000"
|
||||
# RUSTFS_CONSOLE_ADDRESS=":9001"
|
||||
# RUSTFS_CONSOLE_ENABLE=true
|
||||
# RUSTFS_OBS_LOGGER_LEVEL=error
|
||||
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
|
||||
ENVEOF
|
||||
|
||||
fpm -s dir -t rpm \
|
||||
@@ -231,35 +217,13 @@ jobs:
|
||||
--description "High-performance distributed object storage" \
|
||||
--url "https://rustfs.com" \
|
||||
--license "Apache-2.0" \
|
||||
--after-install <(cat << 'POSTINST'
|
||||
#!/bin/bash
|
||||
set -e
|
||||
if ! getent passwd rustfs > /dev/null 2>&1; then
|
||||
useradd -r -s /bin/false -d /opt/rustfs rustfs
|
||||
fi
|
||||
mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
|
||||
chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
|
||||
if [ -d /run/systemd/system ]; then
|
||||
systemctl daemon-reload
|
||||
fi
|
||||
POSTINST
|
||||
) \
|
||||
--before-remove <(cat << 'PRERM'
|
||||
#!/bin/bash
|
||||
set -e
|
||||
if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
|
||||
systemctl stop rustfs
|
||||
fi
|
||||
PRERM
|
||||
) \
|
||||
--after-remove <(cat << 'POSTRM'
|
||||
#!/bin/bash
|
||||
set -e
|
||||
if [ -d /run/systemd/system ]; then
|
||||
systemctl daemon-reload
|
||||
fi
|
||||
POSTRM
|
||||
) \
|
||||
--before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
|
||||
--before-upgrade <(./scripts/release/package_service_scripts.sh rpm-before-upgrade) \
|
||||
--after-install <(./scripts/release/package_service_scripts.sh after-install) \
|
||||
--after-upgrade <(./scripts/release/package_service_scripts.sh after-install) \
|
||||
--before-remove <(./scripts/release/package_service_scripts.sh before-remove) \
|
||||
--after-remove <(./scripts/release/package_service_scripts.sh after-remove) \
|
||||
--rpm-posttrans <(./scripts/release/package_service_scripts.sh rpm-posttrans) \
|
||||
--config-files /etc/default/rustfs \
|
||||
"rustfs-nightly-${DEB_DATE}/usr/bin/rustfs=/usr/bin/rustfs" \
|
||||
./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
|
||||
|
||||
@@ -328,10 +328,15 @@ jobs:
|
||||
|
||||
cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
|
||||
# RustFS Environment Configuration
|
||||
# See https://rustfs.com/docs/ for more information
|
||||
# See https://docs.rustfs.com for more information
|
||||
# RUSTFS_ACCESS_KEY=<changeme>
|
||||
# RUSTFS_SECRET_KEY=<changeme>
|
||||
# RUSTFS_VOLUMES=""
|
||||
# RUSTFS_ROOT_USER=""
|
||||
# RUSTFS_ROOT_PASSWORD=""
|
||||
# RUSTFS_ADDRESS=":9000"
|
||||
# RUSTFS_CONSOLE_ADDRESS=":9001"
|
||||
# RUSTFS_CONSOLE_ENABLE=true
|
||||
# RUSTFS_OBS_LOGGER_LEVEL=error
|
||||
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
|
||||
ENVEOF
|
||||
|
||||
cat > "${PKG_DIR}/DEBIAN/control" << EOF
|
||||
@@ -354,37 +359,13 @@ jobs:
|
||||
/etc/default/rustfs
|
||||
CONFFILES
|
||||
|
||||
cat > "${PKG_DIR}/DEBIAN/postinst" << 'POSTINST'
|
||||
#!/bin/bash
|
||||
set -e
|
||||
if ! getent passwd rustfs > /dev/null 2>&1; then
|
||||
useradd -r -s /bin/false -d /opt/rustfs rustfs
|
||||
fi
|
||||
mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
|
||||
chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
|
||||
if [ -d /run/systemd/system ]; then
|
||||
systemctl daemon-reload
|
||||
fi
|
||||
echo "RustFS installed. Configure /etc/default/rustfs then: systemctl start rustfs"
|
||||
POSTINST
|
||||
./scripts/release/package_service_scripts.sh after-install > "${PKG_DIR}/DEBIAN/postinst"
|
||||
chmod 755 "${PKG_DIR}/DEBIAN/postinst"
|
||||
|
||||
cat > "${PKG_DIR}/DEBIAN/prerm" << 'PRERM'
|
||||
#!/bin/bash
|
||||
set -e
|
||||
if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
|
||||
systemctl stop rustfs
|
||||
fi
|
||||
PRERM
|
||||
./scripts/release/package_service_scripts.sh before-remove > "${PKG_DIR}/DEBIAN/prerm"
|
||||
chmod 755 "${PKG_DIR}/DEBIAN/prerm"
|
||||
|
||||
cat > "${PKG_DIR}/DEBIAN/postrm" << 'POSTRM'
|
||||
#!/bin/bash
|
||||
set -e
|
||||
if [ -d /run/systemd/system ]; then
|
||||
systemctl daemon-reload
|
||||
fi
|
||||
POSTRM
|
||||
./scripts/release/package_service_scripts.sh after-remove > "${PKG_DIR}/DEBIAN/postrm"
|
||||
chmod 755 "${PKG_DIR}/DEBIAN/postrm"
|
||||
|
||||
cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
|
||||
@@ -422,10 +403,15 @@ jobs:
|
||||
mkdir -p ./tmp-pkg/etc/default
|
||||
cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
|
||||
# RustFS Environment Configuration
|
||||
# See https://rustfs.com/docs/ for more information
|
||||
# See https://docs.rustfs.com for more information
|
||||
# RUSTFS_ACCESS_KEY=<changeme>
|
||||
# RUSTFS_SECRET_KEY=<changeme>
|
||||
# RUSTFS_VOLUMES=""
|
||||
# RUSTFS_ROOT_USER=""
|
||||
# RUSTFS_ROOT_PASSWORD=""
|
||||
# RUSTFS_ADDRESS=":9000"
|
||||
# RUSTFS_CONSOLE_ADDRESS=":9001"
|
||||
# RUSTFS_CONSOLE_ENABLE=true
|
||||
# RUSTFS_OBS_LOGGER_LEVEL=error
|
||||
# RUSTFS_OBS_LOG_DIRECTORY="/var/log/rustfs/"
|
||||
ENVEOF
|
||||
|
||||
fpm -s dir -t rpm \
|
||||
@@ -439,35 +425,13 @@ jobs:
|
||||
--description "High-performance distributed object storage" \
|
||||
--url "https://rustfs.com" \
|
||||
--license "Apache-2.0" \
|
||||
--after-install <(cat <<'POSTINST'
|
||||
#!/bin/bash
|
||||
set -e
|
||||
if ! getent passwd rustfs > /dev/null 2>&1; then
|
||||
useradd -r -s /bin/false -d /opt/rustfs rustfs
|
||||
fi
|
||||
mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
|
||||
chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
|
||||
if [ -d /run/systemd/system ]; then
|
||||
systemctl daemon-reload
|
||||
fi
|
||||
POSTINST
|
||||
) \
|
||||
--before-remove <(cat <<'PRERM'
|
||||
#!/bin/bash
|
||||
set -e
|
||||
if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
|
||||
systemctl stop rustfs
|
||||
fi
|
||||
PRERM
|
||||
) \
|
||||
--after-remove <(cat <<'POSTRM'
|
||||
#!/bin/bash
|
||||
set -e
|
||||
if [ -d /run/systemd/system ]; then
|
||||
systemctl daemon-reload
|
||||
fi
|
||||
POSTRM
|
||||
) \
|
||||
--before-install <(./scripts/release/package_service_scripts.sh rpm-before-install) \
|
||||
--before-upgrade <(./scripts/release/package_service_scripts.sh rpm-before-upgrade) \
|
||||
--after-install <(./scripts/release/package_service_scripts.sh after-install) \
|
||||
--after-upgrade <(./scripts/release/package_service_scripts.sh after-install) \
|
||||
--before-remove <(./scripts/release/package_service_scripts.sh before-remove) \
|
||||
--after-remove <(./scripts/release/package_service_scripts.sh after-remove) \
|
||||
--rpm-posttrans <(./scripts/release/package_service_scripts.sh rpm-posttrans) \
|
||||
--config-files /etc/default/rustfs \
|
||||
./bin/rustfs=/usr/bin/rustfs \
|
||||
./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
|
||||
|
||||
@@ -4,13 +4,14 @@
|
||||
# behavior under drive and node loss against the erasure-coding contract and
|
||||
# snapshots health-endpoint responses at every tier.
|
||||
#
|
||||
# A single-node 4 drives (SNMD): hide 1/2/3 drives, restore
|
||||
# A single-node 4 drives (SNMD): revoke access to 1/2/3 drives, restore
|
||||
# B multi-node 4x1 (one drive per node): stop 1/2/3 nodes, restore
|
||||
# C multi-node 4x4 (16 drives, EC:4): stop 1 node (read-quorum boundary),
|
||||
# stop 2 nodes, restore
|
||||
# C2 multi-node 4x4 with EC:8: 2 nodes down puts 8 drives online -- reads
|
||||
# satisfy the EC read quorum while the lock majority is broken (the
|
||||
# reported divergence window: reads 503 with lock_quorum_unavailable)
|
||||
# D multi-node 4x1: stop 1/2 nodes, then restore
|
||||
#
|
||||
# Expectations come from product source (default_parity_count, erasure set
|
||||
# sizing). By default a "reads refused although the read quorum is met"
|
||||
@@ -32,6 +33,11 @@ on:
|
||||
description: 'Direct .deb URL. Required unless the nightly default is wanted.'
|
||||
required: false
|
||||
type: string
|
||||
auto_testing_ref:
|
||||
description: 'auto-testing branch, tag, or SHA (defaults to main)'
|
||||
required: false
|
||||
default: main
|
||||
type: string
|
||||
strict:
|
||||
description: 'Fail the suite when reads are refused despite a met read quorum'
|
||||
type: boolean
|
||||
@@ -52,7 +58,7 @@ on:
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
# The suite stops services and hides drive dirs on the shared fleet; only one
|
||||
# The suite stops services and revokes drive access on the shared fleet; only one
|
||||
# functional suite may touch the environment at a time.
|
||||
concurrency:
|
||||
group: rustfs-shared-functional-tests-v2
|
||||
@@ -92,6 +98,7 @@ jobs:
|
||||
printf 'FUNCTIONAL_ARTIFACTS_DIR=%s\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
|
||||
printf 'LOG_FILE=%s/suite.log\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
|
||||
printf 'REPORT_FILE=%s/report.md\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
|
||||
printf 'RESULTS_FILE=%s/results.json\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
|
||||
printf 'EVIDENCE_DIR=%s/evidence\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
|
||||
printf 'TMPDIR=%s-scratch\n' "${FUNCTIONAL_ARTIFACTS_DIR}"
|
||||
} >> "${GITHUB_ENV}"
|
||||
@@ -109,16 +116,24 @@ jobs:
|
||||
uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
|
||||
with:
|
||||
repository: rustfs/auto-testing
|
||||
ref: ${{ steps.chain.outputs.testing_sha || 'main' }}
|
||||
ref: ${{ steps.chain.outputs.testing_sha || inputs.auto_testing_ref || 'main' }}
|
||||
token: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
path: auto-testing
|
||||
persist-credentials: false
|
||||
|
||||
- name: Record auto-testing revision
|
||||
run: |
|
||||
set -euo pipefail
|
||||
AUTO_TESTING_SHA="$(git -C auto-testing rev-parse HEAD)"
|
||||
printf 'AUTO_TESTING_SHA=%s\n' "${AUTO_TESTING_SHA}" >> "${GITHUB_ENV}"
|
||||
echo "auto-testing revision: ${AUTO_TESTING_SHA}"
|
||||
|
||||
- name: Show environment
|
||||
run: |
|
||||
uname -a
|
||||
jq --version
|
||||
aws --version
|
||||
echo "auto-testing: ${AUTO_TESTING_SHA}"
|
||||
df -h /data | tail -1
|
||||
|
||||
- name: Cleanup environment (before)
|
||||
@@ -134,14 +149,14 @@ jobs:
|
||||
CHAIN_MANIFEST: ${{ inputs.chain_manifest }}
|
||||
run: python3 scripts/prepare_functional_package.py prepare
|
||||
|
||||
- name: Run fault-tolerance scenarios (A, B, C, C2)
|
||||
- name: Run fault-tolerance scenarios (A, B, C, C2, D, E)
|
||||
timeout-minutes: 45
|
||||
id: test
|
||||
# Standalone case failures are reported by the backlog manager.
|
||||
# Chain evidence separately requires complete passing cases.
|
||||
continue-on-error: true
|
||||
run: |
|
||||
ARGS=(--all -y --package-url "${{ steps.chain_package.outputs.package_url || inputs.package_url || env.RUSTFS_NIGHTLY_PACKAGE_URL }}" --log-file "${LOG_FILE}")
|
||||
ARGS=(--all -y --package-url "${{ steps.chain_package.outputs.package_url || inputs.package_url || env.RUSTFS_NIGHTLY_PACKAGE_URL }}" --log-file "${LOG_FILE}" --results-file "${RESULTS_FILE}")
|
||||
if [ "${{ inputs.strict }}" = "true" ]; then
|
||||
ARGS+=(--strict)
|
||||
fi
|
||||
@@ -158,12 +173,13 @@ jobs:
|
||||
echo ""
|
||||
echo "- Run: ${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
|
||||
echo "- Package: ${{ inputs.package_url || 'nightly (R2 latest)' }}"
|
||||
echo "- auto-testing: \`${AUTO_TESTING_SHA}\`"
|
||||
echo "- Strict mode: ${{ inputs.strict || 'false' }}"
|
||||
echo ""
|
||||
echo "## Per-probe results"
|
||||
echo ""
|
||||
echo '```'
|
||||
grep -E '^FT-(CASE|SUMMARY|REPORT)' "${LOG_FILE}" || echo "(no FT-CASE lines found)"
|
||||
grep -E '^FT-(CASE|COMPLETE|SUMMARY|REPORT)' "${LOG_FILE}" || echo "(no FT-CASE lines found)"
|
||||
echo '```'
|
||||
echo ""
|
||||
echo "## Health snapshots"
|
||||
@@ -174,19 +190,30 @@ jobs:
|
||||
done
|
||||
} > "${REPORT_FILE}"
|
||||
cat "${REPORT_FILE}" >> "${GITHUB_STEP_SUMMARY}"
|
||||
# Product gate: failing FT cases keep the run green — the report above and
|
||||
# the backlog issue manager carry the signal. Only harness/environment
|
||||
# breakdowns (no case verdicts at all) turn the workflow red.
|
||||
FT_CASES="$(grep -cE '^FT-CASE:' "${LOG_FILE}" 2>/dev/null || true)"
|
||||
FT_FAILED="$(grep -cE '^FT-CASE: .* verdict=UNEXPECTED' "${LOG_FILE}" 2>/dev/null || true)"
|
||||
FT_PASSED=$(( ${FT_CASES:-0} - ${FT_FAILED:-0} ))
|
||||
HARNESS_OK=0
|
||||
if [ "${{ steps.test.outcome }}" = "success" ]; then
|
||||
HARNESS_OK=1
|
||||
elif [ "${{ steps.test.outcome }}" = "failure" ] && [ "${FT_CASES}" -gt 0 ] && [ "${FT_PASSED}" -ge 1 ]; then
|
||||
HARNESS_OK=1
|
||||
fi
|
||||
[ "${HARNESS_OK}" = "1" ]
|
||||
printf 'complete=false\n' >> "${GITHUB_OUTPUT}"
|
||||
jq -e --arg testing_sha "${AUTO_TESTING_SHA}" '
|
||||
.schema_version == 1 and
|
||||
.auto_testing_sha == $testing_sha and
|
||||
.selected_scenarios == ["A", "B", "C", "C2", "D", "E"] and
|
||||
.complete == true and
|
||||
.counts.expected == 44 and
|
||||
.counts.seen == 44 and
|
||||
.counts.harness_errors == 0 and
|
||||
(.expected_cases | length) == 44 and
|
||||
(.expected_cases | unique | length) == 44 and
|
||||
(.seen_cases | length) == 44 and
|
||||
(.seen_cases | unique | length) == 44 and
|
||||
(.seen_cases | sort) == (.expected_cases | sort) and
|
||||
(.missing_cases | length) == 0 and
|
||||
(.harness_errors | length) == 0 and
|
||||
(.cases | length) == 44 and
|
||||
([.cases[].case_id] | sort) == (.expected_cases | sort) and
|
||||
([.cases[].case_id] | unique | length) == 44 and
|
||||
all(.cases[]; .verdict == "pass" or .verdict == "known-divergence" or .verdict == "UNEXPECTED") and
|
||||
.counts.unexpected == ([.cases[] | select(.verdict == "UNEXPECTED")] | length) and
|
||||
.counts.known_divergence == ([.cases[] | select(.verdict == "known-divergence")] | length)
|
||||
' "${RESULTS_FILE}" >/dev/null
|
||||
printf 'complete=true\n' >> "${GITHUB_OUTPUT}"
|
||||
|
||||
- name: Manage backlog issues (dedup / label / auto-close)
|
||||
timeout-minutes: 2
|
||||
@@ -196,7 +223,7 @@ jobs:
|
||||
# green run. Never acts on cancelled runs. Logic lives in
|
||||
# auto-testing/scripts/issue_manager.py, which parses the
|
||||
# FT-CASE verdict lines from the suite log.
|
||||
if: ${{ always() && steps.evidence.outcome == 'success' }}
|
||||
if: ${{ always() && steps.evidence.outcome == 'success' && steps.chain_report.outputs.complete == 'true' }}
|
||||
continue-on-error: true
|
||||
env:
|
||||
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
@@ -272,6 +299,7 @@ jobs:
|
||||
name: rustfs-fault-tolerance-${{ github.run_id }}-${{ github.run_attempt }}
|
||||
path: |
|
||||
${{ env.FUNCTIONAL_ARTIFACTS_DIR }}/report.md
|
||||
${{ env.FUNCTIONAL_ARTIFACTS_DIR }}/results.json
|
||||
${{ env.FUNCTIONAL_ARTIFACTS_DIR }}/suite.log
|
||||
${{ env.FUNCTIONAL_ARTIFACTS_DIR }}/evidence/
|
||||
if-no-files-found: warn
|
||||
|
||||
@@ -54,7 +54,10 @@ jobs:
|
||||
- name: Resolve published candidate
|
||||
id: candidate
|
||||
env:
|
||||
GH_TOKEN: ${{ github.token }}
|
||||
# The staleness probe in resolve_functional_candidate.py reads the
|
||||
# private rustfs/auto-testing repo; github.token cannot, and a probe
|
||||
# failure silently keeps a stale testing_sha pinned for the chain.
|
||||
GH_TOKEN: ${{ secrets.PF_TESTING_GH_TOKEN }}
|
||||
BUILD_RUN_ID: ${{ inputs.build_run_id }}
|
||||
BUILD_RUN_ATTEMPT: ${{ inputs.build_run_attempt }}
|
||||
CHAIN_OUTPUT: ${{ runner.temp }}/chain-candidate.json
|
||||
|
||||
@@ -61,10 +61,13 @@ on:
|
||||
permissions:
|
||||
contents: read
|
||||
|
||||
# The default performance nodes overlap the other suites' remote VMs, even
|
||||
# though the runner differs. Hold the shared lock through cleanup as well.
|
||||
# Workflow-level so the lock is held through cleanup as well.
|
||||
concurrency:
|
||||
group: rustfs-shared-functional-tests-v2
|
||||
# Own group: the performance suite runs on the pf-testing fleet - a fully
|
||||
# separate environment from the smoke-testing fleet the functional suites
|
||||
# use. No shared node, no shared runner: it must never queue behind them
|
||||
# (or block them).
|
||||
group: rustfs-performance-suite
|
||||
cancel-in-progress: false
|
||||
|
||||
defaults:
|
||||
|
||||
Generated
+86
-77
@@ -685,9 +685,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "async-compression"
|
||||
version = "0.4.47"
|
||||
version = "0.4.48"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ef217a77a86a6e3dab9a5b3c81dc445b603fe743a90c1cb10a2f2144628d8cfa"
|
||||
checksum = "fb61aea1a7def73ee7c350a184f0e70b32c182344e2e75bf70c9b621b83417fd"
|
||||
dependencies = [
|
||||
"compression-codecs",
|
||||
"compression-core",
|
||||
@@ -812,7 +812,7 @@ checksum = "82f6aeea286b8eb4dd3431a1be1b59d290ace00f5bfd8e2a159bc2a05e2c1667"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -961,9 +961,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "aws-runtime"
|
||||
version = "1.9.2"
|
||||
version = "1.9.4"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ef47857a1d4488b528f4a5d5715fa7c3300820897824152234d3fa22b1426657"
|
||||
checksum = "25b43ad47adc2517efe3d706559d94b97e50e80e0321b3cadbc9f77cee88adcd"
|
||||
dependencies = [
|
||||
"aws-credential-types",
|
||||
"aws-sigv4",
|
||||
@@ -989,9 +989,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "aws-sdk-kms"
|
||||
version = "1.118.0"
|
||||
version = "1.120.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c243864bc3754be9f0001414e0162fdf1370fa62c70b0cfbe1da6a6221d553c7"
|
||||
checksum = "d6b0fe38fee2ba5b6cd24d32d08365b314ae1adea123649e061a7eb6300b6f5b"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"aws-credential-types",
|
||||
@@ -1015,9 +1015,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "aws-sdk-s3"
|
||||
version = "1.146.1"
|
||||
version = "1.148.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "2cd651b4400d4011b8927b83a9552bf90ff11e6e5da0b9f0a7583247aceec971"
|
||||
checksum = "8a61c1db3987ab6c8740fb87f248864687d50549401115244a84f02c3047b2d5"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"aws-credential-types",
|
||||
@@ -1052,9 +1052,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "aws-sdk-sso"
|
||||
version = "1.109.0"
|
||||
version = "1.111.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "c3cfe74df5d9ad2fedd691973ad3521ebf4f27a3c68c792556686aedb5519bab"
|
||||
checksum = "65700d75f6da78c89a61a49f1539979400a4609d3329376b1dc2133d7a1b2b90"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"aws-credential-types",
|
||||
@@ -1078,9 +1078,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "aws-sdk-ssooidc"
|
||||
version = "1.111.0"
|
||||
version = "1.113.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "81b0ec31ed6191bd11350aae4b2004198f2db21350cb0a20c57e0a92e55dd161"
|
||||
checksum = "eb1147c254af7b6cee3ace90ae7da6f21366ce9a9b899035fa8a0fdf586eb75a"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"aws-credential-types",
|
||||
@@ -1104,9 +1104,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "aws-sdk-sts"
|
||||
version = "1.114.0"
|
||||
version = "1.116.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "ef45745026107ec30c4ef86bd8ae4b002e7e5f6a86e4225240bdf6b06a0b944a"
|
||||
checksum = "035c7392e751d20bf1060d625d023587eb14099646110a9cff7c697bfe4b253a"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"aws-credential-types",
|
||||
@@ -1131,9 +1131,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "aws-sigv4"
|
||||
version = "1.5.1"
|
||||
version = "1.5.3"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "723c2234ad7511ceef63eab016b7ba6ff7c55590fefb96fa8467af014a07309f"
|
||||
checksum = "31d955e76ff96acd555bf06fa0fa6d5bf9335fa84ae7c64481b20ae61d231f70"
|
||||
dependencies = [
|
||||
"aws-credential-types",
|
||||
"aws-smithy-eventstream",
|
||||
@@ -1737,7 +1737,7 @@ dependencies = [
|
||||
"prettyplease 0.3.0",
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -1984,9 +1984,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "cc"
|
||||
version = "1.4.6"
|
||||
version = "1.4.7"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a3eb0f42d6c360dc3f8a821f6bf2fdea7f72bfd36b3076eb0e6d1e9e0752fff4"
|
||||
checksum = "54413ede23c2daf518f35156dfde027feb2374004d63bd497f983c8db9c0e313"
|
||||
dependencies = [
|
||||
"find-msvc-tools",
|
||||
"jobserver",
|
||||
@@ -1996,9 +1996,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "cfg-if"
|
||||
version = "1.0.4"
|
||||
version = "1.0.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9330f8b2ff13f34540b44e946ef35111825727b38d33286ef986142615121801"
|
||||
checksum = "4e7648175b45a9a48536d676f68d918270699102aa8dab5496df06904c914600"
|
||||
|
||||
[[package]]
|
||||
name = "cfg_aliases"
|
||||
@@ -2141,7 +2141,7 @@ dependencies = [
|
||||
"heck 0.5.0",
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -2222,9 +2222,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "compression-codecs"
|
||||
version = "0.4.42"
|
||||
version = "0.4.43"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "257c7085cbb71be72d8fb97edff08b03d86d5d9f2222b9cc34a6bec87093bc10"
|
||||
checksum = "bef16c47ba2797aa6a909cc37d39911f3a6743811fe7408ac0b0cc0276b656e9"
|
||||
dependencies = [
|
||||
"brotli 8.0.4",
|
||||
"bzip2",
|
||||
@@ -2798,7 +2798,7 @@ dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"strsim",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -2842,7 +2842,7 @@ checksum = "2ac7135c3ef02b2f7833bbeb1be5ba7f966dcde8a87c6b87f65a778d71a02785"
|
||||
dependencies = [
|
||||
"darling_core 0.24.1",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -3343,7 +3343,7 @@ checksum = "3a711bd5a06ec85d9683e142d28a98ad0679a7220669f5c1b8589e57e9b29e46"
|
||||
dependencies = [
|
||||
"datafusion-doc",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -3715,7 +3715,7 @@ checksum = "2e2b94854e8576378ccda7c8de8a66ed8b4e8acbd2c50ec3418ea6c8aaf4b567"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -3792,9 +3792,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "dial9"
|
||||
version = "0.5.0"
|
||||
version = "0.5.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "49b11bce55173f77d9003cce8f129b4af889a4639005c53c3498cef933bc895c"
|
||||
checksum = "bab33c9f10e909fb3f1f6af2a9d976398ea793b90d1d286b4e23647a7d131d53"
|
||||
dependencies = [
|
||||
"bon",
|
||||
"dial9-core",
|
||||
@@ -3809,9 +3809,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "dial9-core"
|
||||
version = "0.5.0"
|
||||
version = "0.5.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9e8cbbc8955394be626249a3b52ddd6bf664373661eaeae70d87eb12bf6f20b6"
|
||||
checksum = "ffd7231006b074b02765ae929bded4757a414224b06c7db324d3e6e902f182ee"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"bon",
|
||||
@@ -3842,9 +3842,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "dial9-metrique"
|
||||
version = "0.5.0"
|
||||
version = "0.5.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "d42fe0e3e14e8780c4ecf9ee14c183dbcb9b1ae3612cc9a689e7dc83b13adef1"
|
||||
checksum = "f987e56dbafcab5998acc8e77eb91aaf66fa5441a691b38e99c8058c70725f58"
|
||||
dependencies = [
|
||||
"dial9-core",
|
||||
"dial9-trace-format",
|
||||
@@ -3856,9 +3856,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "dial9-perf-self-profile"
|
||||
version = "0.5.0"
|
||||
version = "0.5.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a8f65948455c504bf08576c7b5cfe93bfcaa486d661dc4ee85c2a6309ab89629"
|
||||
checksum = "d81f74ada03919614ca5b1ba767271d00980376a5a0e0fc43f8c265bf90413d4"
|
||||
dependencies = [
|
||||
"blazesym",
|
||||
"bon",
|
||||
@@ -3875,9 +3875,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "dial9-tokio-telemetry"
|
||||
version = "0.5.0"
|
||||
version = "0.5.1"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "af1244091367c805b5d98a6a590f8ab992efda06e6d2560a6967ef898ffd30a2"
|
||||
checksum = "934c027f628c59ae4c9664ca4dce758843c3da027dee5052d358f86a5d292ae6"
|
||||
dependencies = [
|
||||
"bon",
|
||||
"bytes",
|
||||
@@ -3991,7 +3991,7 @@ checksum = "c6232dd377dcc64799954cbd3a9bb882e9cdc1308ccd87b1c098f1fb2eaf82a8"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -4410,9 +4410,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "find-msvc-tools"
|
||||
version = "0.1.12"
|
||||
version = "0.1.13"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "3e0f1c7c3a72c66fd80abe965175f7523475c0489a87d3ff9d6e8c87d87a9d2d"
|
||||
checksum = "ef25905e51abafe4dcea6c15fec58c57b601cdbd0ee53d22ea1d3016c587d39b"
|
||||
|
||||
[[package]]
|
||||
name = "findshlibs"
|
||||
@@ -4597,7 +4597,7 @@ checksum = "9fb9654ba8355388abeb8dcb4fc62f511300867002afc858860463bdd9fe0c44"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -5306,9 +5306,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "hotpath"
|
||||
version = "0.25.1"
|
||||
version = "0.26.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "366c5db5a8e4643bdde121a1d6f35cfa035e7850b8b8ea94c6960a139d44e93a"
|
||||
checksum = "f37234f85a8d1bda6b84d486245dd0847c5b33cca10d5f3691375e66704de0f8"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"async-channel",
|
||||
@@ -5318,7 +5318,7 @@ dependencies = [
|
||||
"flate2",
|
||||
"futures-channel",
|
||||
"futures-core",
|
||||
"futures-util",
|
||||
"futures-sink",
|
||||
"hdrhistogram",
|
||||
"hotpath-macros",
|
||||
"http 1.5.0",
|
||||
@@ -5339,9 +5339,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "hotpath-macros"
|
||||
version = "0.25.1"
|
||||
version = "0.26.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "846bde0d9600d98434e1aac376977d7718bfe3d2f5312a041b7c59a6a466c51a"
|
||||
checksum = "2ac39b5517f540cab511bdbd5a4dc8cfc62858d6cd26a095463c1d2acfb3f290"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
@@ -6751,7 +6751,7 @@ dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"rustversion",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -8220,7 +8220,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "2bfe0f4c752e450fc2faf62654f1c134747922825d5b04ca717b8874f41a40c0"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -9054,7 +9054,7 @@ checksum = "92ecd8964f8453721699a1ed72037b0db49ce2f5a5138486ee89bed6f67cdf3a"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -11111,9 +11111,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "rustix"
|
||||
version = "1.1.4"
|
||||
version = "1.1.5"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "b6fe4565b9518b83ef4f91bb47ce29620ca828bd32cb7e408f0062e9930ba190"
|
||||
checksum = "891efababe418670775f199f0d233d84843c227a0949a883ce15b37c78d6629d"
|
||||
dependencies = [
|
||||
"bitflags 2.13.2",
|
||||
"errno",
|
||||
@@ -11259,8 +11259,7 @@ checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
|
||||
[[package]]
|
||||
name = "s3s"
|
||||
version = "0.16.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9d29a4db6a51f1cd3269a36354ce4aa7db2be8816cee42a642b9c85cd3769577"
|
||||
source = "git+https://github.com/s3s-project/s3s.git?rev=50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6#50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6"
|
||||
dependencies = [
|
||||
"arc-swap",
|
||||
"arrayvec",
|
||||
@@ -11278,7 +11277,6 @@ dependencies = [
|
||||
"http 1.5.0",
|
||||
"http-body 1.1.0",
|
||||
"http-body-util",
|
||||
"httparse",
|
||||
"hyper",
|
||||
"itoa",
|
||||
"jiff",
|
||||
@@ -11291,6 +11289,7 @@ dependencies = [
|
||||
"pin-project-lite",
|
||||
"quick-xml",
|
||||
"regex",
|
||||
"s3s-multipart",
|
||||
"s3s-rfc2047",
|
||||
"s3s-sigv2",
|
||||
"s3s-sigv4",
|
||||
@@ -11315,11 +11314,23 @@ dependencies = [
|
||||
"zeroize",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "s3s-multipart"
|
||||
version = "0.17.0-alpha.1"
|
||||
source = "git+https://github.com/s3s-project/s3s.git?rev=50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6#50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6"
|
||||
dependencies = [
|
||||
"arrayvec",
|
||||
"bytes",
|
||||
"futures-core",
|
||||
"httparse",
|
||||
"memchr",
|
||||
"thiserror 2.0.20",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
name = "s3s-rfc2047"
|
||||
version = "0.16.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "9370a2f353a7929b79ae7b5f692b4758c59aa19526b18feff6a977e8ff17da5a"
|
||||
source = "git+https://github.com/s3s-project/s3s.git?rev=50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6#50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6"
|
||||
dependencies = [
|
||||
"base64-simd",
|
||||
"thiserror 2.0.20",
|
||||
@@ -11328,8 +11339,7 @@ dependencies = [
|
||||
[[package]]
|
||||
name = "s3s-sigv2"
|
||||
version = "0.16.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "f277b63ffc9a2135c49d52726199b7436c2f86746d625496984f5bb5cabef050"
|
||||
source = "git+https://github.com/s3s-project/s3s.git?rev=50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6#50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6"
|
||||
dependencies = [
|
||||
"base64-simd",
|
||||
"hmac 0.13.0",
|
||||
@@ -11342,8 +11352,7 @@ dependencies = [
|
||||
[[package]]
|
||||
name = "s3s-sigv4"
|
||||
version = "0.16.0"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "a979a401f6ae4035bc22e4ac0f7a9a5923ba93841c931e8e28652fe211d991bc"
|
||||
source = "git+https://github.com/s3s-project/s3s.git?rev=50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6#50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6"
|
||||
dependencies = [
|
||||
"arrayvec",
|
||||
"base64-simd",
|
||||
@@ -11575,7 +11584,7 @@ checksum = "e7a5d71263a5a7d47b41f6b3f06ba276f10cc18b0931f1799f710578e2309348"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -11639,7 +11648,7 @@ checksum = "8d3b1629de253c70a0508c3899572da79ca359fdab27c7920ff00406df418906"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -11684,7 +11693,7 @@ dependencies = [
|
||||
"darling 0.24.1",
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -11719,7 +11728,7 @@ checksum = "a22144e767da4ddd8416dbf383700542ffd8a5dc493dfecedfe1fe3ad03c98ae"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -12316,9 +12325,9 @@ dependencies = [
|
||||
|
||||
[[package]]
|
||||
name = "syn"
|
||||
version = "3.0.5"
|
||||
version = "3.0.6"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "12df2e0110f65b775f769bb17ef989067a1d931b2eb822bd4346631eeada89f9"
|
||||
checksum = "8593e8e72159ed2257d083c7a454a85cbf854f37a0966d8d483aff8c8a3ebcee"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
@@ -12365,7 +12374,7 @@ checksum = "901704edd0dfe137f1987838ee4f259e4e063c31371bdb423f7ae38ec6f77f02"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -12559,7 +12568,7 @@ checksum = "bc04cd3e1236dd4a98afca4569f2deb3f120e5422a4023be2cb683f8486292af"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -12696,7 +12705,7 @@ checksum = "78773a2a397f451582ce068015985c33193cf6dea8b74d2a639fe457b2f07b0e"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
@@ -13212,9 +13221,9 @@ checksum = "5c1cb5db39152898a79168971543b1cb5020dff7fe43c8dc468b0885f5e29df5"
|
||||
|
||||
[[package]]
|
||||
name = "unicode-ident"
|
||||
version = "1.0.24"
|
||||
version = "1.0.26"
|
||||
source = "registry+https://github.com/rust-lang/crates.io-index"
|
||||
checksum = "e6e4313cd5fcd3dad5cafa179702e2b244f760991f45397d14d4ebf38247da75"
|
||||
checksum = "d245f478577f809a851594d02313b640fb437e0bb33866753cff937863096954"
|
||||
|
||||
[[package]]
|
||||
name = "unicode-normalization"
|
||||
@@ -13456,7 +13465,7 @@ dependencies = [
|
||||
"bumpalo",
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
"wasm-bindgen-shared",
|
||||
]
|
||||
|
||||
@@ -14031,7 +14040,7 @@ checksum = "33811428bee40dbceb6d545e95754741d17a6aef9a4849f0fd62e2ba4f412a78"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
"synstructure 0.14.0",
|
||||
]
|
||||
|
||||
@@ -14072,7 +14081,7 @@ checksum = "f75b4683f6c7f45248d4d64056a24298c6281e0993356d7d1b4a1a962ef10d4a"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
"synstructure 0.14.0",
|
||||
]
|
||||
|
||||
@@ -14126,7 +14135,7 @@ checksum = "34df6fc39dbd26ddc9c10e6a2984476e13acce22e64e4487636ef494369225da"
|
||||
dependencies = [
|
||||
"proc-macro2",
|
||||
"quote",
|
||||
"syn 3.0.5",
|
||||
"syn 3.0.6",
|
||||
]
|
||||
|
||||
[[package]]
|
||||
|
||||
+9
-9
@@ -145,7 +145,7 @@ rustfs-zip = { path = "./crates/zip", version = "1.0.1" }
|
||||
async-channel = "2.5.0"
|
||||
async_zip = { default-features = false, version = "0.0.19" }
|
||||
mysql_async = { default-features = false, version = "0.37.1" }
|
||||
async-compression = { version = "0.4.47" }
|
||||
async-compression = { version = "0.4.48" }
|
||||
async-recursion = "1.1.1"
|
||||
async-trait = "0.1.92"
|
||||
async-nats = { version = "0.50.0", default-features = false }
|
||||
@@ -244,9 +244,9 @@ atoi = "3.1.0"
|
||||
atomic_enum = "0.3.0"
|
||||
aws-config = { version = "1.12.0" }
|
||||
aws-credential-types = { version = "1.3.0" }
|
||||
aws-sdk-kms = { default-features = false, version = "1.118.0" }
|
||||
aws-sdk-s3 = { default-features = false, version = "1.146.1" }
|
||||
aws-sdk-sts = { default-features = false, version = "1.114.0" }
|
||||
aws-sdk-kms = { default-features = false, version = "1.120.0" }
|
||||
aws-sdk-s3 = { default-features = false, version = "1.148.0" }
|
||||
aws-sdk-sts = { default-features = false, version = "1.116.0" }
|
||||
aws-smithy-async = { version = "1.3.0" }
|
||||
aws-smithy-http-client = { default-features = false, version = "1.4.2" }
|
||||
aws-smithy-runtime-api = { version = "1.16.2" }
|
||||
@@ -310,11 +310,11 @@ regex = { version = "1.13.1" }
|
||||
rumqttc = { package = "rumqttc-next", version = "0.34.0" }
|
||||
redis = { version = "1.7.0" }
|
||||
rustify = { version = "0.7", default-features = false }
|
||||
rustix = { version = "1.1.4" }
|
||||
rustix = { version = "1.1.5" }
|
||||
rust-embed = { version = "8.12.0" }
|
||||
rustc-hash = { version = "2.1.3" }
|
||||
s3s = { version = "0.16.0", features = ["minio"] }
|
||||
s3s-sigv4 = { version = "0.16.0" }
|
||||
s3s = { git = "https://github.com/s3s-project/s3s.git", rev = "50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6", features = ["minio"] }
|
||||
s3s-sigv4 = { git = "https://github.com/s3s-project/s3s.git", rev = "50c94aeb1e5ea9ef8e7393ef6bbc87b72f6891c6" }
|
||||
serial_test = "4.0.1"
|
||||
shadow-rs = { default-features = false, version = "2.0.0" }
|
||||
siphasher = "1.0.3"
|
||||
@@ -351,7 +351,7 @@ zstd = "0.14.0"
|
||||
# Observability and Metrics
|
||||
metrics = "0.24.6"
|
||||
metrics-util = "0.20"
|
||||
dial9 = { version = "0.5.0", default-features = false }
|
||||
dial9 = { version = "0.5.1", default-features = false }
|
||||
opentelemetry = { version = "0.32.0" }
|
||||
opentelemetry-appender-tracing = { version = "0.32.0" }
|
||||
opentelemetry-otlp = { version = "0.32.0" }
|
||||
@@ -375,7 +375,7 @@ dav-server = "0.11.0"
|
||||
# Performance Analysis and Memory Profiling
|
||||
rustfs-mimalloc = { version = "0.5.4" }
|
||||
# Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
|
||||
hotpath = { version = "0.25.1", default-features = false }
|
||||
hotpath = { version = "0.26.0", default-features = false }
|
||||
# Snapshot testing for output format regression detection
|
||||
insta = { version = "1.48" }
|
||||
|
||||
|
||||
@@ -663,6 +663,15 @@ pub const ENV_OBJECT_LOCK_RPC_DETACHED_LIMIT: &str = "RUSTFS_OBJECT_LOCK_RPC_DET
|
||||
/// Default per-peer budget of detached (timed-out but still running) lock RPCs: 256.
|
||||
pub const DEFAULT_OBJECT_LOCK_RPC_DETACHED_LIMIT: usize = 256;
|
||||
|
||||
/// Environment variable for the maximum number of in-flight lock acquisition
|
||||
/// RPCs admitted to one peer. Requests beyond this bound fail fast as
|
||||
/// retryable contention so a slow endpoint cannot accumulate an unbounded
|
||||
/// queue while its channel remains healthy.
|
||||
pub const ENV_OBJECT_LOCK_RPC_REQUEST_LIMIT: &str = "RUSTFS_OBJECT_LOCK_RPC_REQUEST_LIMIT";
|
||||
|
||||
/// Default per-peer in-flight lock acquisition RPC admission limit.
|
||||
pub const DEFAULT_OBJECT_LOCK_RPC_REQUEST_LIMIT: usize = 128;
|
||||
|
||||
/// Environment variable to enable object namespace lock diagnostics.
|
||||
///
|
||||
/// When enabled, RustFS emits slow lock acquisition and long lock hold
|
||||
|
||||
@@ -55,6 +55,13 @@ pub const ENV_OBS_LOG_FILENAME: &str = "RUSTFS_OBS_LOG_FILENAME";
|
||||
pub const ENV_OBS_LOG_ROTATION_TIME: &str = "RUSTFS_OBS_LOG_ROTATION_TIME";
|
||||
pub const ENV_OBS_LOG_KEEP_FILES: &str = "RUSTFS_OBS_LOG_KEEP_FILES";
|
||||
|
||||
/// Span lifecycle events emitted by the JSON log layers (`none`|`close`|`full`).
|
||||
///
|
||||
/// Defaults to `none`: a record per span lifecycle multiplies log volume on
|
||||
/// object hot paths, and emitting the ancestor span list on every record
|
||||
/// produced multi-megabyte lines that journald truncated (backlog#2642).
|
||||
pub const ENV_OBS_LOG_SPAN_EVENTS: &str = "RUSTFS_OBS_LOG_SPAN_EVENTS";
|
||||
|
||||
/// Log cleanup related configurations
|
||||
pub const ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES: &str = "RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES";
|
||||
pub const ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES: &str = "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES";
|
||||
@@ -99,6 +106,8 @@ pub const DEFAULT_OBS_LOG_DRY_RUN: bool = false;
|
||||
pub const DEFAULT_OBS_LOG_MATCH_MODE_PREFIX: &str = "prefix";
|
||||
pub const DEFAULT_OBS_LOG_MATCH_MODE: &str = "suffix";
|
||||
|
||||
pub const DEFAULT_OBS_LOG_SPAN_EVENTS: &str = "none";
|
||||
|
||||
/// Default values for observability configuration
|
||||
// ### Supported Environment Values
|
||||
// - `production` - Secure file-only logging
|
||||
@@ -141,6 +150,7 @@ mod tests {
|
||||
assert_eq!(ENV_OBS_LOG_FILENAME, "RUSTFS_OBS_LOG_FILENAME");
|
||||
assert_eq!(ENV_OBS_LOG_ROTATION_TIME, "RUSTFS_OBS_LOG_ROTATION_TIME");
|
||||
assert_eq!(ENV_OBS_LOG_KEEP_FILES, "RUSTFS_OBS_LOG_KEEP_FILES");
|
||||
assert_eq!(ENV_OBS_LOG_SPAN_EVENTS, "RUSTFS_OBS_LOG_SPAN_EVENTS");
|
||||
assert_eq!(ENV_OBS_TRACES_EXPORT_ENABLED, "RUSTFS_OBS_TRACES_EXPORT_ENABLED");
|
||||
assert_eq!(ENV_OBS_METRICS_EXPORT_ENABLED, "RUSTFS_OBS_METRICS_EXPORT_ENABLED");
|
||||
assert_eq!(ENV_OBS_LOGS_EXPORT_ENABLED, "RUSTFS_OBS_LOGS_EXPORT_ENABLED");
|
||||
@@ -178,6 +188,7 @@ mod tests {
|
||||
assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_GZIP, "gzip");
|
||||
assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM_ZSTD, "zstd");
|
||||
assert_eq!(DEFAULT_OBS_LOG_MATCH_MODE_PREFIX, "prefix");
|
||||
assert_eq!(DEFAULT_OBS_LOG_SPAN_EVENTS, "none");
|
||||
assert_eq!(DEFAULT_OBS_LOG_MATCH_MODE, "suffix");
|
||||
assert_eq!(DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM, "zstd");
|
||||
}
|
||||
|
||||
@@ -184,7 +184,7 @@ async fn four_node_health_inventory_metrics_and_audit_delivery_are_consistent()
|
||||
let (status, metrics_body) = admin_request(
|
||||
&node.url,
|
||||
Method::GET,
|
||||
"/rustfs/admin/v3/metrics?n=1&by-host=true&by-disk=true",
|
||||
"/rustfs/admin/v3/realtime?n=1&by-host=true&by-disk=true",
|
||||
None,
|
||||
&dist.cluster.access_key,
|
||||
&dist.cluster.secret_key,
|
||||
@@ -257,7 +257,7 @@ async fn node_admin_body(dist: &DistCluster, node: usize, path: &str) -> TestRes
|
||||
}
|
||||
|
||||
async fn http_put_counts(dist: &DistCluster, node: usize) -> TestResult<[u64; 2]> {
|
||||
let body = node_admin_body(dist, node, "/rustfs/admin/v3/metrics?types=512&by-host=true&n=1").await?;
|
||||
let body = node_admin_body(dist, node, "/rustfs/admin/v3/realtime?types=512&by-host=true&n=1").await?;
|
||||
let sample: RealtimeMetrics = serde_json::from_str(body.lines().next().ok_or("empty HTTP metrics stream")?)?;
|
||||
assert!(sample.errors.is_empty(), "HTTP metrics returned errors: {:?}", sample.errors);
|
||||
let http = sample.aggregated.http.ok_or("HTTP metrics missing at WARN log level")?;
|
||||
@@ -400,17 +400,36 @@ async fn verify_write_observations_during_peer_failure(dist: &DistCluster, bucke
|
||||
let observations = storage["info"]["observations"]
|
||||
.as_array()
|
||||
.ok_or("recovery omitted observations")?;
|
||||
let disks = storage["info"]["disks"].as_array().ok_or("recovery omitted disks")?;
|
||||
Ok(observations.len() == 4
|
||||
&& observations
|
||||
.iter()
|
||||
.all(|item| item["status"] == "succeeded" && item["cached"] == false))
|
||||
.all(|item| item["status"] == "succeeded" && item["cached"] == false)
|
||||
&& disks.len() == 16
|
||||
&& disks.iter().all(|disk| {
|
||||
disk["state"].as_str().is_some_and(|state| state.eq_ignore_ascii_case("ok"))
|
||||
&& disk["runtimeState"]
|
||||
.as_str()
|
||||
.is_some_and(|state| state.eq_ignore_ascii_case("online"))
|
||||
}))
|
||||
},
|
||||
"peer probes recover to fresh successful observations",
|
||||
"peer probes and remote disks recover to fresh successful observations",
|
||||
)
|
||||
.await?;
|
||||
wait_for_ready(&dist.cluster).await?;
|
||||
assert!(observed_put(dist, 0, bucket, "recovered").await?.is_success());
|
||||
assert_eq!(http_put_counts(dist, 0).await?, [baseline[0][0] + 1, baseline[0][1] + 2]);
|
||||
let recovered_before = http_put_counts(dist, 0).await?;
|
||||
wait_until(
|
||||
Duration::from_secs(90),
|
||||
|| async { Ok(observed_put(dist, 0, bucket, "recovered").await?.is_success()) },
|
||||
"recovered PUT after remote disk and lock convergence",
|
||||
)
|
||||
.await?;
|
||||
let recovered_after = http_put_counts(dist, 0).await?;
|
||||
assert_eq!(recovered_after[0], recovered_before[0] + 1);
|
||||
assert!(
|
||||
recovered_after[1] >= baseline[0][1] + 2,
|
||||
"recovery retries must retain the two intentional sub-quorum failures: before={recovered_before:?}, after={recovered_after:?}"
|
||||
);
|
||||
for node in 0..dist.cluster.nodes.len() {
|
||||
assert_object_bytes(&dist.client(node)?, bucket, "healthy-0", b"write-observation").await?;
|
||||
assert_object_bytes(&dist.client(node)?, bucket, "recovered", b"write-observation").await?;
|
||||
|
||||
@@ -391,8 +391,9 @@ pub mod data_usage {
|
||||
#[cfg(feature = "test-util")]
|
||||
pub use crate::data_movement::scanner_backlog::test_util::NativeScannerPauseBacklogWriteFault;
|
||||
pub use crate::data_movement::scanner_backlog::{
|
||||
MAX_SCANNER_PAUSE_BACKLOG_BYTES, ScannerPauseBacklogRetirementPlan, ScannerPauseBacklogRetirementPlanner,
|
||||
ScannerPauseBacklogRetirementReplica, register_scanner_pause_backlog_retirement_planner,
|
||||
MAX_SCANNER_PAUSE_BACKLOG_BYTES, ScannerPauseBacklogRetirementError, ScannerPauseBacklogRetirementPlan,
|
||||
ScannerPauseBacklogRetirementPlanner, ScannerPauseBacklogRetirementReplica,
|
||||
register_scanner_pause_backlog_retirement_planner,
|
||||
};
|
||||
pub use crate::data_usage::{
|
||||
DATA_USAGE_CACHE_NAME, apply_bucket_usage_memory_overlay, compute_bucket_usage,
|
||||
|
||||
@@ -35,6 +35,7 @@ use crate::bucket::lifecycle::tier_sweeper::{
|
||||
use crate::disk::RUSTFS_META_BUCKET;
|
||||
use crate::error::{Error, Result as EcstoreResult};
|
||||
use crate::object_api::{ObjectInfo, ObjectOptions};
|
||||
use crate::services::notification_sys::acquire_remote_version_state_writer_fleet_proof;
|
||||
use crate::services::tier::{tier::TierConfigMgr, warm_backend::TransitionCandidateProbe};
|
||||
use crate::storage_api_contracts::{
|
||||
list::ListOperations as _,
|
||||
@@ -1237,6 +1238,12 @@ pub async fn delete_transition_candidate_for_operator(
|
||||
lease
|
||||
.validate_remote_version_id(remote_version_id)
|
||||
.map_err(TransitionOperatorError::Remote)?;
|
||||
if remote_version_requires_fleet_proof(remote_version_id) && acquire_remote_version_state_writer_fleet_proof().is_none() {
|
||||
return Err(TransitionOperatorError::Remote(std::io::Error::new(
|
||||
std::io::ErrorKind::Unsupported,
|
||||
"opaque remote tier version cleanup requires the operator-attested live fleet capability gate",
|
||||
)));
|
||||
}
|
||||
let before_delete_probe = lease
|
||||
.probe_transition_candidate_for(&transaction.remote_object, transaction.transaction_id)
|
||||
.await
|
||||
@@ -1845,6 +1852,11 @@ async fn recover_unknown_upload_outcome(
|
||||
))
|
||||
}
|
||||
TransitionCandidateProbe::VersionedPresent(version_id) => {
|
||||
if remote_version_requires_fleet_proof(&version_id) && acquire_remote_version_state_writer_fleet_proof().is_none() {
|
||||
return Ok(TransitionTransactionRecoveryOutcome::RetainedAmbiguous(
|
||||
IlmRecoveryErrorCode::RemoteVersionUnknown,
|
||||
));
|
||||
}
|
||||
cleanup_recovered_unknown_upload_candidate(api, transaction, TransitionRemoteVersion::versioned(version_id)).await
|
||||
}
|
||||
TransitionCandidateProbe::Ambiguous => Ok(TransitionTransactionRecoveryOutcome::RetainedAmbiguous(
|
||||
@@ -1940,6 +1952,14 @@ fn transition_source_lookup_options(transaction: &TransitionTransaction) -> Obje
|
||||
async fn delete_transition_remote_candidate(api: Arc<ECStore>, transaction: &TransitionTransaction) -> EcstoreResult<()> {
|
||||
let version_id = transaction.remote_version.tier_delete_version_id().unwrap_or_default();
|
||||
let version_id_exact = transaction.remote_version.kind == TransitionRemoteVersionKind::Versioned;
|
||||
if version_id_exact
|
||||
&& remote_version_requires_fleet_proof(version_id)
|
||||
&& acquire_remote_version_state_writer_fleet_proof().is_none()
|
||||
{
|
||||
return Err(Error::other(
|
||||
"opaque remote tier version cleanup requires the operator-attested live fleet capability gate",
|
||||
));
|
||||
}
|
||||
delete_object_from_remote_tier_idempotent_with_manager_and_identity(
|
||||
&transaction.remote_object,
|
||||
version_id,
|
||||
@@ -1953,6 +1973,10 @@ async fn delete_transition_remote_candidate(api: Arc<ECStore>, transaction: &Tra
|
||||
.map_err(Error::other)
|
||||
}
|
||||
|
||||
fn remote_version_requires_fleet_proof(version_id: &str) -> bool {
|
||||
!version_id.is_empty() && Uuid::parse_str(version_id).is_err()
|
||||
}
|
||||
|
||||
pub async fn recover_transition_transaction_records(
|
||||
api: Arc<ECStore>,
|
||||
limit: usize,
|
||||
@@ -2439,6 +2463,14 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn opaque_remote_versions_require_the_fleet_proof_boundary() {
|
||||
assert!(!remote_version_requires_fleet_proof(""));
|
||||
assert!(!remote_version_requires_fleet_proof(&Uuid::new_v4().to_string()));
|
||||
assert!(remote_version_requires_fleet_proof("null"));
|
||||
assert!(remote_version_requires_fleet_proof("b2-opaque-version"));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn remote_version_distinguishes_unknown_unversioned_and_versioned() {
|
||||
assert_eq!(TransitionRemoteVersion::known_from_put_response("").tier_delete_version_id(), None);
|
||||
|
||||
@@ -1464,9 +1464,13 @@ pub(crate) async fn load_bucket_metadata_parse_with_presence(
|
||||
"bucket incarnation sidecar is missing for new-format metadata: {bucket}"
|
||||
)));
|
||||
}
|
||||
} else if incarnation.is_some() {
|
||||
return Err(Error::other("bucket incarnation sidecar exists without bucket metadata"));
|
||||
}
|
||||
// A sidecar without `.metadata.bin` is the window between the two legacy
|
||||
// migration writes (rustfs/rustfs#8003): the sidecar lands first, and a
|
||||
// crash or lost namespace lease before the metadata write leaves the
|
||||
// bucket in this state on every node. Report it as not persisted so the
|
||||
// migration runs again; the writers adopt the stored incarnation instead
|
||||
// of minting a new one, so the sidecar keeps its authority.
|
||||
|
||||
bm.default_timestamps();
|
||||
|
||||
|
||||
@@ -2504,6 +2504,16 @@ impl BucketMetadataSys {
|
||||
if !persisted {
|
||||
metadata = BucketMetadata::new(bucket);
|
||||
metadata.created = bucket_info.created.unwrap_or(OffsetDateTime::UNIX_EPOCH);
|
||||
// An interrupted migration may already have published this
|
||||
// bucket's incarnation; re-read it under the transaction lock so
|
||||
// the retry never replaces an identity other nodes fenced on. A
|
||||
// retired incarnation is residue of a deleted bucket and must not
|
||||
// come back, or heal would reclaim the bucket's new objects.
|
||||
if let Some(stored) = load_bucket_incarnation(self.object_store(), bucket).await?
|
||||
&& !crate::bucket::retirement::is_retired(self.object_store(), bucket, stored).await?
|
||||
{
|
||||
metadata.bucket_incarnation_id = stored;
|
||||
}
|
||||
} else if metadata.bucket_incarnation_id.is_nil() {
|
||||
metadata.bucket_incarnation_id = Uuid::new_v4();
|
||||
}
|
||||
@@ -3556,6 +3566,98 @@ mod tests {
|
||||
assert!(err.to_string().contains("sidecar is missing"));
|
||||
}
|
||||
|
||||
/// rustfs/rustfs#8003: the legacy migration writes the incarnation sidecar
|
||||
/// before `.metadata.bin`. A crash or lost namespace lease between the two
|
||||
/// left every pre-existing bucket answering 500 on every request, with no
|
||||
/// path back. The sidecar-only state must load as a legacy bucket, and the
|
||||
/// retried migration must keep the stored incarnation.
|
||||
#[tokio::test]
|
||||
async fn issue_8003_sidecar_without_metadata_loads_as_legacy_and_migrates_with_its_incarnation() {
|
||||
let (dirs, ecstore) = isolated_store_over_temp_disks().await;
|
||||
let sys = BucketMetadataSys::new(ecstore.clone());
|
||||
let bucket = "issue-8003-sidecar-only";
|
||||
for dir in &dirs {
|
||||
std::fs::create_dir_all(dir.path().join(bucket)).unwrap();
|
||||
}
|
||||
let stored = Uuid::new_v4();
|
||||
save_bucket_incarnation(ecstore.clone(), bucket, stored)
|
||||
.await
|
||||
.expect("simulate the interrupted migration's first write");
|
||||
|
||||
let (fabricated, persisted) = load_bucket_metadata_parse_with_presence(ecstore.clone(), bucket, true)
|
||||
.await
|
||||
.expect("a sidecar without metadata must read as a legacy bucket, not fail closed");
|
||||
assert!(!persisted);
|
||||
assert!(!fabricated.bucket_incarnation_sidecar);
|
||||
|
||||
let (_, fabricated_read) = sys
|
||||
.get_config(bucket)
|
||||
.await
|
||||
.expect("request-path metadata reads must not fail closed on the sidecar-only state");
|
||||
assert!(fabricated_read);
|
||||
|
||||
let migrated = sys
|
||||
.get_authoritative_metadata(bucket)
|
||||
.await
|
||||
.expect("the retried legacy migration must complete");
|
||||
assert!(migrated.bucket_incarnation_sidecar);
|
||||
assert_eq!(
|
||||
migrated.bucket_incarnation_id, stored,
|
||||
"the retry must adopt the published incarnation instead of minting a new one"
|
||||
);
|
||||
assert_eq!(sys.get_bucket_incarnation_id(bucket).await.unwrap(), stored);
|
||||
|
||||
let on_disk = sys.get_config_from_disk(bucket).await.expect("metadata is now persisted");
|
||||
assert!(on_disk.bucket_incarnation_sidecar);
|
||||
assert_eq!(on_disk.bucket_incarnation_id, stored);
|
||||
assert_eq!(load_bucket_incarnation(ecstore, bucket).await.unwrap(), Some(stored));
|
||||
}
|
||||
|
||||
/// A sidecar left behind by a deleted bucket names a retired incarnation.
|
||||
/// The migration must mint a new identity for a same-name volume instead
|
||||
/// of re-publishing the retired one, or heal would treat the bucket's new
|
||||
/// objects as reclaimable residue.
|
||||
#[tokio::test]
|
||||
async fn issue_8003_sidecar_only_migration_does_not_adopt_a_retired_incarnation() {
|
||||
let (dirs, ecstore) = isolated_store_over_temp_disks().await;
|
||||
let sys = BucketMetadataSys::new(ecstore.clone());
|
||||
let bucket = "issue-8003-retired-sidecar";
|
||||
for dir in &dirs {
|
||||
std::fs::create_dir_all(dir.path().join(bucket)).unwrap();
|
||||
}
|
||||
let retired = Uuid::new_v4();
|
||||
save_bucket_incarnation(ecstore.clone(), bucket, retired)
|
||||
.await
|
||||
.expect("residual sidecar");
|
||||
crate::bucket::retirement::commit_retirement(
|
||||
ecstore.clone(),
|
||||
bucket,
|
||||
retired,
|
||||
&ObjectOptions {
|
||||
max_parity: true,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("retirement record");
|
||||
|
||||
let migrated = sys
|
||||
.get_authoritative_metadata(bucket)
|
||||
.await
|
||||
.expect("the migration must still complete for the same-name volume");
|
||||
assert!(migrated.bucket_incarnation_sidecar);
|
||||
assert_ne!(
|
||||
migrated.bucket_incarnation_id, retired,
|
||||
"a retired incarnation must never be re-published"
|
||||
);
|
||||
assert!(!migrated.bucket_incarnation_id.is_nil());
|
||||
assert_eq!(
|
||||
load_bucket_incarnation(ecstore, bucket).await.unwrap(),
|
||||
Some(migrated.bucket_incarnation_id),
|
||||
"the sidecar must be rewritten to the new incarnation"
|
||||
);
|
||||
}
|
||||
|
||||
/// Concurrent cache misses for one bucket must collapse into a single disk
|
||||
/// load.
|
||||
///
|
||||
|
||||
@@ -199,6 +199,38 @@ fn has_raw_status(err: &SdkError<HeadObjectError>, status: u16) -> bool {
|
||||
err.raw_response().is_some_and(|r| r.status().as_u16() == status)
|
||||
}
|
||||
|
||||
fn raw_header_value(err: &SdkError<HeadObjectError>, name: &str) -> Option<String> {
|
||||
err.raw_response()
|
||||
.and_then(|response| response.headers().get(name))
|
||||
.filter(|value| !value.is_empty())
|
||||
.map(ToOwned::to_owned)
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
enum DeleteMarkerHeadOutcome {
|
||||
Missing,
|
||||
Exists,
|
||||
Ambiguous,
|
||||
}
|
||||
|
||||
fn classify_delete_marker_head_error(err: &SdkError<HeadObjectError>) -> DeleteMarkerHeadOutcome {
|
||||
let (is_not_found, code) = err
|
||||
.as_service_error()
|
||||
.map(|service_err| (service_err.is_not_found(), service_err.code()))
|
||||
.unwrap_or((false, None));
|
||||
|
||||
// Prefer the raw HTTP status when present. HEAD responses commonly have no
|
||||
// body, and a contradictory XML code must not turn a 403/5xx into success.
|
||||
match err.raw_response().map(|response| response.status().as_u16()) {
|
||||
Some(404) => DeleteMarkerHeadOutcome::Missing,
|
||||
Some(405) => DeleteMarkerHeadOutcome::Exists,
|
||||
Some(_) => DeleteMarkerHeadOutcome::Ambiguous,
|
||||
None if is_not_found => DeleteMarkerHeadOutcome::Missing,
|
||||
None if !is_retryable_delete_replication_head_error(false, code) => DeleteMarkerHeadOutcome::Exists,
|
||||
None => DeleteMarkerHeadOutcome::Ambiguous,
|
||||
}
|
||||
}
|
||||
|
||||
fn metadata_requires_existing_target(op_type: ReplicationType, object_info: &ObjectInfo) -> bool {
|
||||
op_type == ReplicationType::Metadata
|
||||
&& object_info
|
||||
@@ -1809,32 +1841,14 @@ async fn verify_resync_head_result(
|
||||
(roi.size, None)
|
||||
}
|
||||
Err(err) if roi.delete_marker => {
|
||||
// Verifying a replicated delete marker: only a
|
||||
// definitive 404/NoSuchKey or 405/MethodNotAllowed
|
||||
// confirms the marker propagated. Any other
|
||||
// (retryable/ambiguous) HEAD error leaves the outcome
|
||||
// unverified, so it must count as failed — not as a
|
||||
// blanket success (backlog#862 / #799 B13).
|
||||
// A HEAD carries no body, so the SDK only synthesizes an error
|
||||
// code for 404; a target answering the delete marker's version
|
||||
// with 405 (RustFS, MinIO, AWS) leaves `code()` empty and the
|
||||
// raw status is the only evidence (rustfs/backlog#2479).
|
||||
// A marker *version purge* (`version_purge_status` set) asks
|
||||
// for the opposite outcome: the marker must be gone from the
|
||||
// target, so only absence verifies it and a 405 proves the
|
||||
// marker is still there.
|
||||
let retryable = {
|
||||
let (is_not_found, code) = err
|
||||
.as_service_error()
|
||||
.map(|se| (se.is_not_found(), se.code()))
|
||||
.unwrap_or((false, None));
|
||||
let not_found = is_not_found || has_raw_status(&err, 404);
|
||||
if roi.version_purge_status.is_empty() {
|
||||
let code = code.or_else(|| has_raw_status(&err, 405).then_some("405"));
|
||||
is_retryable_delete_replication_head_error(not_found, code)
|
||||
} else {
|
||||
!not_found
|
||||
}
|
||||
// A marker creation is verified by either absence (404) or
|
||||
// presence (405). A marker-version purge has the opposite goal:
|
||||
// only absence proves completion. All other outcomes remain
|
||||
// failed (backlog#862 / #799 B13).
|
||||
let retryable = if roi.version_purge_status.is_empty() {
|
||||
matches!(classify_delete_marker_head_error(&err), DeleteMarkerHeadOutcome::Ambiguous)
|
||||
} else {
|
||||
!matches!(classify_delete_marker_head_error(&err), DeleteMarkerHeadOutcome::Missing)
|
||||
};
|
||||
if retryable {
|
||||
st.failed_count += 1;
|
||||
@@ -3553,7 +3567,7 @@ async fn replicate_delete_to_target(
|
||||
}
|
||||
}
|
||||
|
||||
if dobj.delete_object.delete_marker && dobj.delete_object.delete_marker_version_id.is_some() {
|
||||
if !is_version_purge && dobj.delete_object.delete_marker && dobj.delete_object.delete_marker_version_id.is_some() {
|
||||
match head_object_for_worker(
|
||||
tgt_client.as_ref(),
|
||||
&tgt_client.bucket,
|
||||
@@ -3562,20 +3576,48 @@ async fn replicate_delete_to_target(
|
||||
)
|
||||
.await
|
||||
{
|
||||
Ok(_) => {}
|
||||
Ok(head) if head.delete_marker == Some(true) => {
|
||||
// Some targets expose an existing marker as a successful HEAD
|
||||
// with the marker bit instead of the usual 405. Treat that as
|
||||
// positive convergence evidence and retain the target version
|
||||
// id for any later marker purge.
|
||||
rinfo.target_delete_marker_version_id = head.version_id.filter(|version_id| !version_id.is_empty());
|
||||
rinfo.replication_status = ReplicationStatusType::Completed;
|
||||
return rinfo;
|
||||
}
|
||||
Ok(_) => {
|
||||
// A successful HEAD without the delete-marker bit proves that
|
||||
// the addressed version is a regular object (or that the
|
||||
// target omitted the marker contract). Creating a marker in
|
||||
// that state would hide a live version, so fail closed.
|
||||
rinfo.replication_status = ReplicationStatusType::Failed;
|
||||
rinfo.error = Some("target version is not a delete marker".to_string());
|
||||
return rinfo;
|
||||
}
|
||||
Err(e) => {
|
||||
let non_retryable = matches!(
|
||||
e.as_ref(),
|
||||
SdkError::ServiceError(service_err)
|
||||
if is_retryable_delete_replication_head_error(
|
||||
service_err.err().is_not_found(),
|
||||
service_err.err().code(),
|
||||
)
|
||||
);
|
||||
if non_retryable {
|
||||
rinfo.replication_status = ReplicationStatusType::Failed;
|
||||
rinfo.error = Some(e.to_string());
|
||||
return rinfo;
|
||||
match classify_delete_marker_head_error(e.as_ref()) {
|
||||
DeleteMarkerHeadOutcome::Exists => {
|
||||
// An explicit HEAD of a delete-marker version is expected
|
||||
// to answer 405 (often without an SDK error code). That
|
||||
// is positive evidence the marker already exists, so the
|
||||
// creation is complete and must not issue another
|
||||
// versionless DELETE that would mint a second marker.
|
||||
if let Some(version_id) = raw_header_value(e.as_ref(), "x-amz-version-id") {
|
||||
rinfo.target_delete_marker_version_id = Some(version_id);
|
||||
}
|
||||
rinfo.replication_status = ReplicationStatusType::Completed;
|
||||
return rinfo;
|
||||
}
|
||||
DeleteMarkerHeadOutcome::Missing => {}
|
||||
DeleteMarkerHeadOutcome::Ambiguous => {
|
||||
// Only a definitive not-found means the marker is absent
|
||||
// and may be created. Keep transport and ambiguous
|
||||
// failures fail-closed instead of creating blindly.
|
||||
rinfo.replication_status = ReplicationStatusType::Failed;
|
||||
rinfo.error = Some(e.to_string());
|
||||
mark_replication_target_offline_if_needed(&tgt_client, &e).await;
|
||||
return rinfo;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -6529,6 +6571,110 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
fn marker_creation_dobj(target_arn: &str) -> DeletedObjectReplicationInfo {
|
||||
let mut state = ReplicationState::default();
|
||||
state.targets.insert(target_arn.to_string(), ReplicationStatusType::Pending);
|
||||
DeletedObjectReplicationInfo {
|
||||
bucket: "source".to_string(),
|
||||
target_arn: target_arn.to_string(),
|
||||
op_type: ReplicationType::Heal,
|
||||
delete_object: ReplicationDeletedObject {
|
||||
object_name: "gone.txt".to_string(),
|
||||
delete_marker: true,
|
||||
delete_marker_version_id: Some(Uuid::new_v4()),
|
||||
replication_state: Some(state),
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
/// A target that already has the marker answers a versioned HEAD with a
|
||||
/// bodiless 405. The live worker must settle the marker as completed and
|
||||
/// avoid minting another marker with a versionless DELETE.
|
||||
#[tokio::test]
|
||||
async fn live_delete_marker_worker_accepts_bodiless_405_without_delete() {
|
||||
let (endpoint, server) = spawn_scripted_target_server(1, |_| {
|
||||
"HTTP/1.1 405 Method Not Allowed\r\nX-Amz-Version-Id: target-marker\r\nContent-Length: 0\r\nConnection: close\r\n\r\n"
|
||||
.to_string()
|
||||
});
|
||||
let target = test_target_client(endpoint);
|
||||
register_test_target(&target).await;
|
||||
let dobj = marker_creation_dobj(&target.arn);
|
||||
|
||||
let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
|
||||
|
||||
assert_eq!(rinfo.replication_status, ReplicationStatusType::Completed, "{rinfo:?}");
|
||||
assert!(rinfo.error.is_none(), "{rinfo:?}");
|
||||
assert_eq!(rinfo.target_delete_marker_version_id.as_deref(), Some("target-marker"));
|
||||
let seen = server.join().expect("test HTTP server should finish");
|
||||
assert_eq!(seen.len(), 1, "an existing marker must not be re-created: {seen:?}");
|
||||
assert!(seen[0].starts_with("HEAD "), "{seen:?}");
|
||||
}
|
||||
|
||||
/// A successful HEAD that does not identify a delete marker is not
|
||||
/// convergence evidence: creating a marker would hide a live version.
|
||||
#[tokio::test]
|
||||
async fn live_delete_marker_worker_rejects_successful_regular_head_without_delete() {
|
||||
let (endpoint, server) = spawn_scripted_target_server(1, |_| empty_response("200 OK"));
|
||||
let target = test_target_client(endpoint);
|
||||
register_test_target(&target).await;
|
||||
let dobj = marker_creation_dobj(&target.arn);
|
||||
|
||||
let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
|
||||
|
||||
assert_eq!(rinfo.replication_status, ReplicationStatusType::Failed, "{rinfo:?}");
|
||||
assert_eq!(rinfo.error.as_deref(), Some("target version is not a delete marker"));
|
||||
let seen = server.join().expect("test HTTP server should finish");
|
||||
assert_eq!(seen.len(), 1, "a regular object must not be hidden by a new marker: {seen:?}");
|
||||
assert!(seen[0].starts_with("HEAD "), "{seen:?}");
|
||||
}
|
||||
|
||||
/// A definitive 404 means the marker is absent, so exactly one marker
|
||||
/// creation DELETE is still required.
|
||||
#[tokio::test]
|
||||
async fn live_delete_marker_worker_creates_marker_after_404() {
|
||||
let (endpoint, server) = spawn_scripted_target_server(2, |line| {
|
||||
if line.starts_with("HEAD ") {
|
||||
empty_response("404 Not Found")
|
||||
} else if line.starts_with("DELETE ") {
|
||||
empty_response("204 No Content")
|
||||
} else {
|
||||
empty_response("500 Unexpected")
|
||||
}
|
||||
});
|
||||
let target = test_target_client(endpoint);
|
||||
register_test_target(&target).await;
|
||||
let dobj = marker_creation_dobj(&target.arn);
|
||||
|
||||
let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
|
||||
|
||||
assert_eq!(rinfo.replication_status, ReplicationStatusType::Completed, "{rinfo:?}");
|
||||
let seen = server.join().expect("test HTTP server should finish");
|
||||
assert_eq!(seen.len(), 2, "a missing marker should be created once: {seen:?}");
|
||||
assert!(seen[0].starts_with("HEAD "), "{seen:?}");
|
||||
assert!(seen[1].starts_with("DELETE "), "{seen:?}");
|
||||
assert!(!seen[1].contains("versionId="), "marker creation must remain versionless: {seen:?}");
|
||||
}
|
||||
|
||||
/// Ambiguous service errors must remain failed and must not fall through
|
||||
/// to a blind marker-creation DELETE.
|
||||
#[tokio::test]
|
||||
async fn live_delete_marker_worker_keeps_ambiguous_head_failure_closed() {
|
||||
let (endpoint, server) = spawn_scripted_target_server(1, |_| empty_response("503 Service Unavailable"));
|
||||
let target = test_target_client(endpoint);
|
||||
register_test_target(&target).await;
|
||||
let dobj = marker_creation_dobj(&target.arn);
|
||||
|
||||
let rinfo = replicate_delete_to_target(&dobj, target.clone(), None).await;
|
||||
|
||||
assert_eq!(rinfo.replication_status, ReplicationStatusType::Failed, "{rinfo:?}");
|
||||
assert!(rinfo.error.is_some(), "{rinfo:?}");
|
||||
let seen = server.join().expect("test HTTP server should finish");
|
||||
assert_eq!(seen.len(), 1, "an ambiguous HEAD must not trigger DELETE: {seen:?}");
|
||||
assert!(seen[0].starts_with("HEAD "), "{seen:?}");
|
||||
}
|
||||
|
||||
/// A target answers `HEAD ?versionId=<delete marker>` with a bodiless
|
||||
/// 405, so the SDK reports no error code. That is the marker having
|
||||
/// propagated, not a failure (rustfs/backlog#2479).
|
||||
|
||||
@@ -23,7 +23,7 @@ use std::{
|
||||
future::Future,
|
||||
io::ErrorKind,
|
||||
pin::Pin,
|
||||
sync::{Arc, OnceLock},
|
||||
sync::{Arc, OnceLock, atomic::AtomicBool},
|
||||
task::{Context, Poll},
|
||||
time::Duration,
|
||||
};
|
||||
@@ -234,6 +234,8 @@ pub struct ListPathRawOptions {
|
||||
pub skip_walkdir_total_timeout: bool,
|
||||
pub walkdir_timeout: Option<Duration>,
|
||||
pub walkdir_stall_timeout: Option<Duration>,
|
||||
/// Shared terminal state for bounded producer walks.
|
||||
pub producer_limit_reached: Option<Arc<AtomicBool>>,
|
||||
pub agreed: Option<AgreedFn>,
|
||||
pub partial: Option<PartialFn>,
|
||||
pub finished: Option<FinishedFn>,
|
||||
@@ -267,6 +269,7 @@ impl Clone for ListPathRawOptions {
|
||||
skip_walkdir_total_timeout: self.skip_walkdir_total_timeout,
|
||||
walkdir_timeout: self.walkdir_timeout,
|
||||
walkdir_stall_timeout: self.walkdir_stall_timeout,
|
||||
producer_limit_reached: self.producer_limit_reached.clone(),
|
||||
#[cfg(test)]
|
||||
test_reader_behaviors: self.test_reader_behaviors.clone(),
|
||||
#[cfg(test)]
|
||||
@@ -292,6 +295,7 @@ fn walk_dir_options(opts: &ListPathRawOptions) -> WalkDirOptions {
|
||||
skip_total_timeout: opts.skip_walkdir_total_timeout,
|
||||
timeout_ms: opts.walkdir_timeout.map(duration_millis),
|
||||
stall_timeout_ms: opts.walkdir_stall_timeout.map(duration_millis),
|
||||
producer_limit_reached: opts.producer_limit_reached.clone(),
|
||||
..Default::default()
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1170,6 +1170,11 @@ impl RemoteDisk {
|
||||
self.health.force_runtime_state_for_test(state);
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub fn force_offline_for_test(&self) {
|
||||
self.health.force_offline_for_test();
|
||||
}
|
||||
|
||||
/// Same as [`DiskHealthTracker::reset_for_store_init_retry`]: undo a transient faulty mark before another format load attempt.
|
||||
pub fn reset_health_for_store_init_retry(&self) {
|
||||
self.health.reset_for_store_init_retry(&self.endpoint);
|
||||
@@ -2136,7 +2141,7 @@ impl RemoteDisk {
|
||||
let file_info_bin = encode_file_info_msgpack(fi)?;
|
||||
let mut client = self.get_client().await?;
|
||||
let mut request = Request::new(RenameDataRequest {
|
||||
bucket_incarnation_id: crate::store::bucket_heal_scope(dst_volume)
|
||||
bucket_incarnation_id: crate::store::bucket_heal_scope_for_object(dst_volume, dst_path)
|
||||
.map(|scope| scope.incarnation.as_bytes().to_vec().into())
|
||||
.unwrap_or_default(),
|
||||
disk: self.endpoint.to_string(),
|
||||
|
||||
@@ -47,10 +47,37 @@ fn attach_lock_mutation_body_digest<T: rustfs_protos::CanonicalMutationBody>(req
|
||||
|
||||
/// Work to run if an RPC that already timed out for its caller completes later.
|
||||
type LateCompletion<T> = Option<Box<dyn FnOnce(T) -> Pin<Box<dyn Future<Output = ()> + Send>> + Send>>;
|
||||
type LateFailure = LateCompletion<()>;
|
||||
|
||||
struct RpcCleanup<T> {
|
||||
late: LateCompletion<T>,
|
||||
late_failure: LateFailure,
|
||||
admission: Option<LockRequestAdmission>,
|
||||
}
|
||||
|
||||
impl<T> RpcCleanup<T> {
|
||||
fn new(late: LateCompletion<T>, late_failure: LateFailure, admission: Option<LockRequestAdmission>) -> Self {
|
||||
Self {
|
||||
late,
|
||||
late_failure,
|
||||
admission,
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// The liveness window is this many RPC deadlines: a peer that completed a
|
||||
/// lock RPC within it is slow, not gone, and keeps its channel on a timeout.
|
||||
const LOCK_RPC_LIVENESS_WINDOW_DEADLINES: u32 = 2;
|
||||
const LOCK_RPC_REQUEST_BACKOFF_THRESHOLD: u32 = 3;
|
||||
const LOCK_RPC_REQUEST_BACKOFF_MAX: Duration = Duration::from_secs(30);
|
||||
const LATE_RELEASE_RETRY_DELAYS: [Duration; 3] = [Duration::from_millis(100), Duration::from_millis(500), Duration::from_secs(1)];
|
||||
const LATE_RELEASE_IN_FLIGHT_LIMIT: usize = 32;
|
||||
const TIMEOUT_LOG_STATE_MAX: usize = 1024;
|
||||
const TIMEOUT_LOG_STATE_RETENTION: Duration = Duration::from_secs(600);
|
||||
|
||||
fn is_request_breaker_operation(op: &'static str) -> bool {
|
||||
matches!(op, "lock" | "lock_batch")
|
||||
}
|
||||
|
||||
/// Recent history of the shared lock channel to one peer (issue #7363).
|
||||
///
|
||||
@@ -64,15 +91,44 @@ struct LockPeerChannelHealth {
|
||||
last_success: Option<Instant>,
|
||||
last_eviction: Option<Instant>,
|
||||
consecutive_timeouts: u32,
|
||||
request_consecutive_timeouts: u32,
|
||||
request_backoff_until: Option<Instant>,
|
||||
request_backoff: Duration,
|
||||
request_probe_generation: Option<u64>,
|
||||
next_probe_generation: u64,
|
||||
request_in_flight: usize,
|
||||
late_release_in_flight: usize,
|
||||
/// Timed-out RPCs still running in the background for this peer.
|
||||
detached_rpcs: usize,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, Default)]
|
||||
struct TimeoutLogHealth {
|
||||
last_logged: Option<Instant>,
|
||||
suppressed: u64,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy)]
|
||||
struct LockRequestAdmission {
|
||||
probe_generation: Option<u64>,
|
||||
}
|
||||
|
||||
impl LockRequestAdmission {
|
||||
fn is_probe(self) -> bool {
|
||||
self.probe_generation.is_some()
|
||||
}
|
||||
}
|
||||
|
||||
fn lock_peer_channel_health() -> &'static Mutex<HashMap<String, LockPeerChannelHealth>> {
|
||||
static HEALTH: OnceLock<Mutex<HashMap<String, LockPeerChannelHealth>>> = OnceLock::new();
|
||||
HEALTH.get_or_init(Mutex::default)
|
||||
}
|
||||
|
||||
fn timeout_log_health() -> &'static Mutex<HashMap<(String, &'static str), TimeoutLogHealth>> {
|
||||
static HEALTH: OnceLock<Mutex<HashMap<(String, &'static str), TimeoutLogHealth>>> = OnceLock::new();
|
||||
HEALTH.get_or_init(Mutex::default)
|
||||
}
|
||||
|
||||
fn with_lock_peer_health<R>(addr: &str, update: impl FnOnce(&mut LockPeerChannelHealth) -> R) -> R {
|
||||
let mut peers = lock_peer_channel_health()
|
||||
.lock()
|
||||
@@ -343,17 +399,163 @@ impl RemoteClient {
|
||||
)
|
||||
}
|
||||
|
||||
fn request_limit() -> usize {
|
||||
rustfs_utils::get_env_usize(
|
||||
rustfs_config::ENV_OBJECT_LOCK_RPC_REQUEST_LIMIT,
|
||||
rustfs_config::DEFAULT_OBJECT_LOCK_RPC_REQUEST_LIMIT,
|
||||
)
|
||||
.max(1)
|
||||
}
|
||||
|
||||
fn liveness_window(deadline: Duration) -> Duration {
|
||||
deadline.saturating_mul(LOCK_RPC_LIVENESS_WINDOW_DEADLINES)
|
||||
}
|
||||
|
||||
fn record_rpc_success(&self) {
|
||||
fn record_rpc_success(&self, op: &'static str, admission: Option<LockRequestAdmission>) {
|
||||
with_lock_peer_health(&self.addr, |health| {
|
||||
health.last_success = Some(Instant::now());
|
||||
health.consecutive_timeouts = 0;
|
||||
if is_request_breaker_operation(op) {
|
||||
let is_probe = admission.is_some_and(LockRequestAdmission::is_probe);
|
||||
if health.request_backoff_until.is_none() || is_probe {
|
||||
health.request_consecutive_timeouts = 0;
|
||||
health.request_backoff_until = None;
|
||||
health.request_backoff = Duration::ZERO;
|
||||
}
|
||||
if is_probe && admission.is_some_and(|admission| health.request_probe_generation == admission.probe_generation) {
|
||||
health.request_probe_generation = None;
|
||||
}
|
||||
}
|
||||
});
|
||||
}
|
||||
|
||||
fn admit_late_release(&self) -> bool {
|
||||
with_lock_peer_health(&self.addr, |health| {
|
||||
if health.late_release_in_flight >= LATE_RELEASE_IN_FLIGHT_LIMIT {
|
||||
false
|
||||
} else {
|
||||
health.late_release_in_flight += 1;
|
||||
true
|
||||
}
|
||||
})
|
||||
}
|
||||
|
||||
fn release_late_release(&self) {
|
||||
with_lock_peer_health(&self.addr, |health| {
|
||||
health.late_release_in_flight = health.late_release_in_flight.saturating_sub(1);
|
||||
});
|
||||
}
|
||||
|
||||
/// Admit a new lock acquisition unless this peer is in the request-level
|
||||
/// backoff window. A single half-open probe is allowed after the window;
|
||||
/// channel liveness and request admission are deliberately separate.
|
||||
fn admit_lock_request(&self, op: &'static str) -> Option<LockRequestAdmission> {
|
||||
if !is_request_breaker_operation(op) {
|
||||
return Some(LockRequestAdmission { probe_generation: None });
|
||||
}
|
||||
|
||||
let now = Instant::now();
|
||||
let (admission, suppression_reason) = with_lock_peer_health(&self.addr, |health| {
|
||||
let degraded = health.request_backoff_until.is_some() || health.request_consecutive_timeouts > 0;
|
||||
if degraded && health.request_in_flight >= Self::request_limit() {
|
||||
return (None, "in_flight_limit");
|
||||
}
|
||||
let Some(until) = health.request_backoff_until else {
|
||||
health.request_in_flight += 1;
|
||||
return (Some(LockRequestAdmission { probe_generation: None }), "none");
|
||||
};
|
||||
if now < until || health.request_probe_generation.is_some() {
|
||||
return (None, "breaker_open");
|
||||
}
|
||||
health.next_probe_generation = health.next_probe_generation.saturating_add(1);
|
||||
let generation = health.next_probe_generation;
|
||||
health.request_probe_generation = Some(generation);
|
||||
health.request_in_flight += 1;
|
||||
(
|
||||
Some(LockRequestAdmission {
|
||||
probe_generation: Some(generation),
|
||||
}),
|
||||
"none",
|
||||
)
|
||||
});
|
||||
if admission.is_none() {
|
||||
rustfs_io_metrics::lock_metrics::record_remote_lock_request_suppressed(&self.addr, op, suppression_reason);
|
||||
}
|
||||
admission
|
||||
}
|
||||
|
||||
fn release_lock_request(&self, op: &'static str, admission: Option<LockRequestAdmission>) {
|
||||
if is_request_breaker_operation(op) && admission.is_some() {
|
||||
with_lock_peer_health(&self.addr, |health| {
|
||||
health.request_in_flight = health.request_in_flight.saturating_sub(1);
|
||||
if admission.is_some_and(|admission| health.request_probe_generation == admission.probe_generation) {
|
||||
health.request_probe_generation = None;
|
||||
}
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
fn record_request_timeout(&self, op: &'static str, deadline: Duration) {
|
||||
if !is_request_breaker_operation(op) {
|
||||
return;
|
||||
}
|
||||
|
||||
let now = Instant::now();
|
||||
with_lock_peer_health(&self.addr, |health| {
|
||||
health.request_consecutive_timeouts = health.request_consecutive_timeouts.saturating_add(1);
|
||||
if health.request_consecutive_timeouts < LOCK_RPC_REQUEST_BACKOFF_THRESHOLD {
|
||||
return;
|
||||
}
|
||||
|
||||
let initial = deadline / 2;
|
||||
let next = if health.request_backoff.is_zero() {
|
||||
initial.max(Duration::from_millis(1))
|
||||
} else {
|
||||
health.request_backoff.saturating_mul(2)
|
||||
};
|
||||
health.request_backoff = next.min(LOCK_RPC_REQUEST_BACKOFF_MAX);
|
||||
health.request_backoff_until = Some(now + health.request_backoff);
|
||||
});
|
||||
}
|
||||
|
||||
fn timeout_log_decision(&self, op: &'static str, interval: Duration) -> (bool, u64) {
|
||||
let now = Instant::now();
|
||||
let mut peers = timeout_log_health().lock().unwrap_or_else(std::sync::PoisonError::into_inner);
|
||||
if peers.len() >= TIMEOUT_LOG_STATE_MAX {
|
||||
peers.retain(|_, state| {
|
||||
state
|
||||
.last_logged
|
||||
.is_some_and(|at| now.saturating_duration_since(at) < TIMEOUT_LOG_STATE_RETENTION)
|
||||
});
|
||||
}
|
||||
let key = (self.addr.clone(), op);
|
||||
if peers.len() >= TIMEOUT_LOG_STATE_MAX
|
||||
&& !peers.contains_key(&key)
|
||||
&& let Some(eviction_key) = peers.keys().next().cloned()
|
||||
{
|
||||
peers.remove(&eviction_key);
|
||||
}
|
||||
let state = peers.entry(key).or_default();
|
||||
if state
|
||||
.last_logged
|
||||
.is_some_and(|at| now.saturating_duration_since(at) < interval)
|
||||
{
|
||||
state.suppressed = state.suppressed.saturating_add(1);
|
||||
return (false, state.suppressed);
|
||||
}
|
||||
|
||||
let suppressed = state.suppressed;
|
||||
state.last_logged = Some(now);
|
||||
state.suppressed = 0;
|
||||
(true, suppressed)
|
||||
}
|
||||
|
||||
fn bounded_acquire_request(request: &LockRequest, deadline: Duration) -> LockRequest {
|
||||
let mut bounded = request.clone();
|
||||
bounded.acquire_timeout = bounded.acquire_timeout.min(deadline);
|
||||
bounded
|
||||
}
|
||||
|
||||
/// Apply the per-peer eviction policy after a failed RPC.
|
||||
async fn maybe_evict_connection(
|
||||
&self,
|
||||
@@ -406,6 +608,8 @@ impl RemoteClient {
|
||||
resource_summary: &str,
|
||||
handle: JoinHandle<std::result::Result<T, tonic::Status>>,
|
||||
late: LateCompletion<T>,
|
||||
late_failure: LateFailure,
|
||||
admission: Option<LockRequestAdmission>,
|
||||
) {
|
||||
let limit = Self::detached_rpc_limit();
|
||||
let admitted = with_lock_peer_health(&self.addr, |health| {
|
||||
@@ -418,6 +622,10 @@ impl RemoteClient {
|
||||
});
|
||||
if !admitted {
|
||||
handle.abort();
|
||||
self.release_lock_request(op, admission);
|
||||
if let Some(late_failure) = late_failure {
|
||||
drop(tokio::spawn(late_failure(())));
|
||||
}
|
||||
rustfs_io_metrics::lock_metrics::record_remote_lock_rpc_detached(op, "aborted");
|
||||
debug!(
|
||||
addr = %self.addr,
|
||||
@@ -432,12 +640,19 @@ impl RemoteClient {
|
||||
let addr = self.addr.clone();
|
||||
tokio::spawn(async move {
|
||||
let outcome = handle.await;
|
||||
with_lock_peer_health(&addr, |health| health.detached_rpcs = health.detached_rpcs.saturating_sub(1));
|
||||
with_lock_peer_health(&addr, |health| {
|
||||
health.detached_rpcs = health.detached_rpcs.saturating_sub(1);
|
||||
if is_request_breaker_operation(op) && admission.is_some() {
|
||||
health.request_in_flight = health.request_in_flight.saturating_sub(1);
|
||||
if admission.is_some_and(|admission| health.request_probe_generation == admission.probe_generation) {
|
||||
health.request_probe_generation = None;
|
||||
}
|
||||
}
|
||||
});
|
||||
match outcome {
|
||||
Ok(Ok(response)) => {
|
||||
with_lock_peer_health(&addr, |health| {
|
||||
health.last_success = Some(Instant::now());
|
||||
health.consecutive_timeouts = 0;
|
||||
});
|
||||
rustfs_io_metrics::lock_metrics::record_remote_lock_rpc_late_completion(op, "success");
|
||||
if let Some(late) = late {
|
||||
@@ -445,6 +660,9 @@ impl RemoteClient {
|
||||
}
|
||||
}
|
||||
Ok(Err(status)) => {
|
||||
if let Some(late_failure) = late_failure {
|
||||
late_failure(()).await;
|
||||
}
|
||||
rustfs_io_metrics::lock_metrics::record_remote_lock_rpc_late_completion(op, "error");
|
||||
debug!(
|
||||
addr = %addr,
|
||||
@@ -455,6 +673,9 @@ impl RemoteClient {
|
||||
);
|
||||
}
|
||||
Err(join_error) => {
|
||||
if let Some(late_failure) = late_failure {
|
||||
late_failure(()).await;
|
||||
}
|
||||
rustfs_io_metrics::lock_metrics::record_remote_lock_rpc_late_completion(op, "join_error");
|
||||
debug!(addr = %addr, op, error = %join_error, "Detached remote lock RPC task ended abnormally");
|
||||
}
|
||||
@@ -485,24 +706,79 @@ impl RemoteClient {
|
||||
}))
|
||||
}
|
||||
|
||||
fn late_release_failure_hook(&self, lock_ids: Vec<LockId>) -> LateFailure {
|
||||
let client = self.clone();
|
||||
Some(Box::new(move |_| {
|
||||
Box::pin(async move {
|
||||
client.release_late_acquisitions(lock_ids).await;
|
||||
})
|
||||
}))
|
||||
}
|
||||
|
||||
/// A lock granted after its caller stopped waiting is an orphan until its
|
||||
/// lease expires; hand it back right away, best effort.
|
||||
async fn release_late_acquisitions(&self, lock_ids: Vec<LockId>) {
|
||||
let outcome = match self.release_locks_batch(&lock_ids).await {
|
||||
Ok(released) if released.iter().all(|released| *released) => "released",
|
||||
Ok(_) => "partial",
|
||||
Err(_) => "failed",
|
||||
};
|
||||
async fn release_late_acquisitions(&self, mut lock_ids: Vec<LockId>) {
|
||||
if !self.admit_late_release() {
|
||||
rustfs_io_metrics::lock_metrics::record_remote_lock_late_release("failed");
|
||||
let (log_warning, suppressed_timeouts) = self.timeout_log_decision("late_release", Self::rpc_timeout());
|
||||
if log_warning {
|
||||
warn!(
|
||||
addr = %self.addr,
|
||||
count = lock_ids.len(),
|
||||
outcome = "budget_exhausted",
|
||||
suppressed_timeouts,
|
||||
"Skipped late remote lock release because the cleanup budget is exhausted"
|
||||
);
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
let original_count = lock_ids.len();
|
||||
let mut outcome = "failed";
|
||||
for (attempt, delay) in std::iter::once(Duration::ZERO).chain(LATE_RELEASE_RETRY_DELAYS).enumerate() {
|
||||
if !delay.is_zero() {
|
||||
tokio::time::sleep(delay).await;
|
||||
}
|
||||
|
||||
match self.release_locks_batch(&lock_ids).await {
|
||||
Ok(released) => {
|
||||
lock_ids = lock_ids
|
||||
.into_iter()
|
||||
.zip(released)
|
||||
.filter_map(|(lock_id, released)| (!released).then_some(lock_id))
|
||||
.collect();
|
||||
if lock_ids.is_empty() {
|
||||
outcome = "released";
|
||||
break;
|
||||
}
|
||||
outcome = if attempt == LATE_RELEASE_RETRY_DELAYS.len() {
|
||||
"partial"
|
||||
} else {
|
||||
"failed"
|
||||
};
|
||||
}
|
||||
Err(_) => {
|
||||
outcome = "failed";
|
||||
}
|
||||
}
|
||||
}
|
||||
self.release_late_release();
|
||||
rustfs_io_metrics::lock_metrics::record_remote_lock_late_release(outcome);
|
||||
if outcome == "released" {
|
||||
debug!(addr = %self.addr, count = lock_ids.len(), "Released remote locks granted after their caller timed out");
|
||||
debug!(addr = %self.addr, count = original_count, "Released remote locks granted after their caller timed out");
|
||||
} else {
|
||||
warn!(
|
||||
addr = %self.addr,
|
||||
count = lock_ids.len(),
|
||||
outcome,
|
||||
"Could not release every remote lock granted after its caller timed out; the server lease will expire it"
|
||||
);
|
||||
let (log_warning, suppressed_timeouts) = self.timeout_log_decision("late_release", Self::rpc_timeout());
|
||||
if log_warning {
|
||||
warn!(
|
||||
addr = %self.addr,
|
||||
count = original_count,
|
||||
outcome,
|
||||
suppressed_timeouts,
|
||||
"Could not release every remote lock granted after its caller timed out; the server lease will expire it"
|
||||
);
|
||||
} else {
|
||||
debug!(addr = %self.addr, count = original_count, outcome, suppressed_timeouts, "Suppressed repeated late remote lock release warning");
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -512,26 +788,74 @@ impl RemoteClient {
|
||||
resource_summary: &str,
|
||||
deadline: Duration,
|
||||
future: Fut,
|
||||
late: LateCompletion<T>,
|
||||
cleanup: RpcCleanup<T>,
|
||||
) -> std::result::Result<T, LockError>
|
||||
where
|
||||
Fut: Future<Output = std::result::Result<T, tonic::Status>> + Send + 'static,
|
||||
T: Send + 'static,
|
||||
{
|
||||
let RpcCleanup {
|
||||
late,
|
||||
late_failure,
|
||||
admission,
|
||||
} = cleanup;
|
||||
let mut handle = tokio::spawn(future);
|
||||
match timeout(deadline, &mut handle).await {
|
||||
Ok(Ok(Ok(response))) => {
|
||||
self.record_rpc_success();
|
||||
self.record_rpc_success(op, admission);
|
||||
self.release_lock_request(op, admission);
|
||||
Ok(response)
|
||||
}
|
||||
Ok(Ok(Err(err))) => {
|
||||
let reason = err.to_string();
|
||||
if err.code() == tonic::Code::DeadlineExceeded {
|
||||
rustfs_io_metrics::lock_metrics::record_remote_lock_rpc_timeout(&self.addr, op);
|
||||
self.record_request_timeout(op, deadline);
|
||||
let (log_timeout, suppressed_timeouts) = self.timeout_log_decision(op, deadline);
|
||||
if Self::is_scanner_leader_lock(resource_summary) {
|
||||
debug!(
|
||||
addr = %self.addr,
|
||||
op,
|
||||
timeout_ms = deadline.as_millis(),
|
||||
resource_summary,
|
||||
"Remote lock RPC returned deadline exceeded for scanner leader lock"
|
||||
);
|
||||
} else if log_timeout {
|
||||
warn!(
|
||||
addr = %self.addr,
|
||||
op,
|
||||
timeout_ms = deadline.as_millis(),
|
||||
resource_summary,
|
||||
suppressed_timeouts,
|
||||
"Remote lock RPC returned deadline exceeded"
|
||||
);
|
||||
} else {
|
||||
debug!(
|
||||
addr = %self.addr,
|
||||
op,
|
||||
timeout_ms = deadline.as_millis(),
|
||||
resource_summary,
|
||||
suppressed_timeouts,
|
||||
"Suppressed repeated remote lock RPC deadline"
|
||||
);
|
||||
}
|
||||
self.maybe_evict_connection(op, &reason, resource_summary, EvictionTrigger::Timeout, deadline)
|
||||
.await;
|
||||
self.release_lock_request(op, admission);
|
||||
if let Some(late_failure) = late_failure {
|
||||
drop(tokio::spawn(late_failure(())));
|
||||
}
|
||||
return Err(LockError::timeout(format!("remote lock RPC {op} on {}", self.addr), deadline));
|
||||
}
|
||||
// Only evict (and re-dial) the cached channel when the failure is a genuine
|
||||
// transport problem. A server-produced application status (auth denied, peer
|
||||
// lock service not ready, invalid args, ...) arrives on a perfectly healthy
|
||||
// channel; evicting it just churns the connection and pushes the peer toward
|
||||
// the offline threshold for no benefit. See issue #4567.
|
||||
let transport_failure = Self::is_transport_failure(&err);
|
||||
if transport_failure {
|
||||
self.record_request_timeout(op, deadline);
|
||||
}
|
||||
if Self::is_scanner_leader_lock(resource_summary) {
|
||||
debug!(
|
||||
addr = %self.addr,
|
||||
@@ -543,6 +867,31 @@ impl RemoteClient {
|
||||
transport_failure,
|
||||
"Remote lock RPC returned tonic error for scanner leader lock"
|
||||
);
|
||||
} else if transport_failure {
|
||||
let (log_failure, suppressed_failures) = self.timeout_log_decision(op, deadline);
|
||||
if log_failure {
|
||||
warn!(
|
||||
addr = %self.addr,
|
||||
op,
|
||||
timeout_ms = deadline.as_millis(),
|
||||
resource_summary,
|
||||
tonic_code = ?err.code(),
|
||||
tonic_message = err.message(),
|
||||
transport_failure,
|
||||
suppressed_failures,
|
||||
"Remote lock RPC returned transport error"
|
||||
);
|
||||
} else {
|
||||
debug!(
|
||||
addr = %self.addr,
|
||||
op,
|
||||
timeout_ms = deadline.as_millis(),
|
||||
resource_summary,
|
||||
tonic_code = ?err.code(),
|
||||
suppressed_failures,
|
||||
"Suppressed repeated remote lock transport error"
|
||||
);
|
||||
}
|
||||
} else {
|
||||
warn!(
|
||||
addr = %self.addr,
|
||||
@@ -559,6 +908,10 @@ impl RemoteClient {
|
||||
self.maybe_evict_connection(op, &reason, resource_summary, EvictionTrigger::Transport, deadline)
|
||||
.await;
|
||||
}
|
||||
self.release_lock_request(op, admission);
|
||||
if transport_failure && let Some(late_failure) = late_failure {
|
||||
drop(tokio::spawn(late_failure(())));
|
||||
}
|
||||
Err(LockError::internal(format!("{op} RPC failed: {reason}")))
|
||||
}
|
||||
Ok(Err(join_error)) => {
|
||||
@@ -569,11 +922,17 @@ impl RemoteClient {
|
||||
error = %join_error,
|
||||
"Remote lock RPC task ended abnormally"
|
||||
);
|
||||
self.release_lock_request(op, admission);
|
||||
if let Some(late_failure) = late_failure {
|
||||
drop(tokio::spawn(late_failure(())));
|
||||
}
|
||||
Err(LockError::internal(format!("{op} RPC task failed: {join_error}")))
|
||||
}
|
||||
Err(_) => {
|
||||
let reason = format!("RPC timed out after {deadline:?}");
|
||||
rustfs_io_metrics::lock_metrics::record_remote_lock_rpc_timeout(&self.addr, op);
|
||||
self.record_request_timeout(op, deadline);
|
||||
let (log_timeout, suppressed_timeouts) = self.timeout_log_decision(op, deadline);
|
||||
if Self::is_scanner_leader_lock(resource_summary) {
|
||||
debug!(
|
||||
addr = %self.addr,
|
||||
@@ -582,18 +941,28 @@ impl RemoteClient {
|
||||
resource_summary,
|
||||
"Remote lock RPC timed out for scanner leader lock"
|
||||
);
|
||||
} else {
|
||||
} else if log_timeout {
|
||||
warn!(
|
||||
addr = %self.addr,
|
||||
op,
|
||||
timeout_ms = deadline.as_millis(),
|
||||
resource_summary,
|
||||
suppressed_timeouts,
|
||||
"Remote lock RPC timed out"
|
||||
);
|
||||
} else {
|
||||
debug!(
|
||||
addr = %self.addr,
|
||||
op,
|
||||
timeout_ms = deadline.as_millis(),
|
||||
resource_summary,
|
||||
suppressed_timeouts,
|
||||
"Suppressed repeated remote lock RPC timeout"
|
||||
);
|
||||
}
|
||||
self.maybe_evict_connection(op, &reason, resource_summary, EvictionTrigger::Timeout, deadline)
|
||||
.await;
|
||||
self.detach_timed_out_rpc(op, resource_summary, handle, late);
|
||||
self.detach_timed_out_rpc(op, resource_summary, handle, late, late_failure, admission);
|
||||
Err(LockError::timeout(format!("remote lock RPC {op} on {}", self.addr), deadline))
|
||||
}
|
||||
}
|
||||
@@ -621,6 +990,14 @@ impl RemoteClient {
|
||||
.collect()
|
||||
}
|
||||
|
||||
fn rpc_backoff_failure_response(request: &LockRequest) -> LockResponse {
|
||||
LockResponse::failure("Remote lock RPC timed out: peer request breaker is open", request.acquire_timeout)
|
||||
}
|
||||
|
||||
fn rpc_backoff_failure_batch(requests: &[LockRequest]) -> Vec<LockResponse> {
|
||||
requests.iter().map(Self::rpc_backoff_failure_response).collect()
|
||||
}
|
||||
|
||||
fn build_lock_info(request: &LockRequest, lock_info_json: Option<String>) -> LockInfo {
|
||||
if let Some(lock_info_json) = lock_info_json {
|
||||
match serde_json::from_str::<LockInfo>(&lock_info_json) {
|
||||
@@ -680,22 +1057,45 @@ impl RemoteClient {
|
||||
impl LockClient for RemoteClient {
|
||||
async fn acquire_lock(&self, request: &LockRequest) -> Result<LockResponse> {
|
||||
info!("remote acquire_exclusive for {}", request.resource);
|
||||
let mut client = self.get_client().await?;
|
||||
let admission = match self.admit_lock_request("lock") {
|
||||
Some(admission) => admission,
|
||||
None => return Ok(Self::rpc_backoff_failure_response(request)),
|
||||
};
|
||||
|
||||
let rpc_timeout = Self::rpc_timeout();
|
||||
let bounded_request = Self::bounded_acquire_request(request, rpc_timeout);
|
||||
let mut client = match self.get_client().await {
|
||||
Ok(client) => client,
|
||||
Err(err) => {
|
||||
self.record_request_timeout("lock", rpc_timeout);
|
||||
self.release_lock_request("lock", Some(admission));
|
||||
return Err(err);
|
||||
}
|
||||
};
|
||||
let resource_summary = request.resource.to_string();
|
||||
let mut req = Request::new(GenerallyLockRequest {
|
||||
args: serde_json::to_string(&request)
|
||||
.map_err(|e| LockError::internal(format!("Failed to serialize request: {e}")))?,
|
||||
});
|
||||
attach_lock_mutation_body_digest(&mut req)?;
|
||||
let args = match serde_json::to_string(&bounded_request) {
|
||||
Ok(args) => args,
|
||||
Err(err) => {
|
||||
self.release_lock_request("lock", Some(admission));
|
||||
return Err(LockError::internal(format!("Failed to serialize request: {err}")));
|
||||
}
|
||||
};
|
||||
let mut req = Request::new(GenerallyLockRequest { args });
|
||||
req.set_timeout(rpc_timeout);
|
||||
if let Err(err) = attach_lock_mutation_body_digest(&mut req) {
|
||||
self.release_lock_request("lock", Some(admission));
|
||||
return Err(err.into());
|
||||
}
|
||||
let late = self.late_release_hook(request.lock_id.clone());
|
||||
let late_failure = self.late_release_failure_hook(vec![request.lock_id.clone()]);
|
||||
|
||||
let resp = match self
|
||||
.execute_rpc(
|
||||
"lock",
|
||||
&resource_summary,
|
||||
Self::rpc_timeout(),
|
||||
rpc_timeout,
|
||||
async move { client.lock(req).await },
|
||||
late,
|
||||
RpcCleanup::new(late, late_failure, Some(admission)),
|
||||
)
|
||||
.await
|
||||
{
|
||||
@@ -725,26 +1125,54 @@ impl LockClient for RemoteClient {
|
||||
return Ok(Vec::new());
|
||||
}
|
||||
|
||||
let mut client = self.get_client().await?;
|
||||
let admission = match self.admit_lock_request("lock_batch") {
|
||||
Some(admission) => admission,
|
||||
None => return Ok(Self::rpc_backoff_failure_batch(requests)),
|
||||
};
|
||||
|
||||
let rpc_timeout = Self::rpc_timeout();
|
||||
let bounded_requests = requests
|
||||
.iter()
|
||||
.map(|request| Self::bounded_acquire_request(request, rpc_timeout))
|
||||
.collect::<Vec<_>>();
|
||||
let mut client = match self.get_client().await {
|
||||
Ok(client) => client,
|
||||
Err(err) => {
|
||||
self.record_request_timeout("lock_batch", rpc_timeout);
|
||||
self.release_lock_request("lock_batch", Some(admission));
|
||||
return Err(err);
|
||||
}
|
||||
};
|
||||
let resource_summary = Self::summarize_resources(requests);
|
||||
let mut req = Request::new(BatchGenerallyLockRequest {
|
||||
args: requests
|
||||
.iter()
|
||||
.map(|request| {
|
||||
serde_json::to_string(request).map_err(|e| LockError::internal(format!("Failed to serialize request: {e}")))
|
||||
})
|
||||
.collect::<Result<Vec<_>>>()?,
|
||||
});
|
||||
attach_lock_mutation_body_digest(&mut req)?;
|
||||
let args = match bounded_requests
|
||||
.iter()
|
||||
.map(|request| {
|
||||
serde_json::to_string(request).map_err(|e| LockError::internal(format!("Failed to serialize request: {e}")))
|
||||
})
|
||||
.collect::<Result<Vec<_>>>()
|
||||
{
|
||||
Ok(args) => args,
|
||||
Err(err) => {
|
||||
self.release_lock_request("lock_batch", Some(admission));
|
||||
return Err(err);
|
||||
}
|
||||
};
|
||||
let mut req = Request::new(BatchGenerallyLockRequest { args });
|
||||
req.set_timeout(rpc_timeout);
|
||||
if let Err(err) = attach_lock_mutation_body_digest(&mut req) {
|
||||
self.release_lock_request("lock_batch", Some(admission));
|
||||
return Err(err.into());
|
||||
}
|
||||
let late = self.late_release_batch_hook(requests.iter().map(|request| request.lock_id.clone()).collect());
|
||||
let late_failure = self.late_release_failure_hook(requests.iter().map(|request| request.lock_id.clone()).collect());
|
||||
|
||||
let resp = match self
|
||||
.execute_rpc(
|
||||
"lock_batch",
|
||||
&resource_summary,
|
||||
Self::rpc_timeout(),
|
||||
rpc_timeout,
|
||||
async move { client.lock_batch(req).await },
|
||||
late,
|
||||
RpcCleanup::new(late, late_failure, Some(admission)),
|
||||
)
|
||||
.await
|
||||
{
|
||||
@@ -791,7 +1219,7 @@ impl LockClient for RemoteClient {
|
||||
&resource_summary,
|
||||
Self::rpc_timeout(),
|
||||
async move { client.un_lock(req).await },
|
||||
None,
|
||||
RpcCleanup::new(None, None, None),
|
||||
)
|
||||
.await?
|
||||
.into_inner();
|
||||
@@ -825,7 +1253,7 @@ impl LockClient for RemoteClient {
|
||||
&resource_summary,
|
||||
Self::rpc_timeout(),
|
||||
async move { client.un_lock_batch(req).await },
|
||||
None,
|
||||
RpcCleanup::new(None, None, None),
|
||||
)
|
||||
.await?
|
||||
.into_inner();
|
||||
@@ -853,7 +1281,7 @@ impl LockClient for RemoteClient {
|
||||
&resource_summary,
|
||||
Self::rpc_timeout(),
|
||||
async move { client.refresh(req).await },
|
||||
None,
|
||||
RpcCleanup::new(None, None, None),
|
||||
)
|
||||
.await?
|
||||
.into_inner();
|
||||
@@ -879,7 +1307,7 @@ impl LockClient for RemoteClient {
|
||||
&resource_summary,
|
||||
Self::rpc_timeout(),
|
||||
async move { client.force_un_lock(req).await },
|
||||
None,
|
||||
RpcCleanup::new(None, None, None),
|
||||
)
|
||||
.await?
|
||||
.into_inner();
|
||||
@@ -894,26 +1322,31 @@ impl LockClient for RemoteClient {
|
||||
|
||||
// Since there's no direct status query in the gRPC service,
|
||||
// we attempt a non-blocking lock acquisition to check if the resource is available
|
||||
let status_request = Self::create_unlock_request(lock_id);
|
||||
let probe_lock_id = LockId::new_unique(&lock_id.resource);
|
||||
let rpc_timeout = Self::rpc_timeout();
|
||||
let status_request = Self::create_unlock_request(&probe_lock_id).with_ttl(rpc_timeout);
|
||||
let bounded_status_request = Self::bounded_acquire_request(&status_request, rpc_timeout);
|
||||
let resource_summary = status_request.resource.to_string();
|
||||
let mut client = self.get_client().await?;
|
||||
let args = serde_json::to_string(&status_request)
|
||||
let args = serde_json::to_string(&bounded_status_request)
|
||||
.map_err(|e| LockError::internal(format!("Failed to serialize request: {e}")))?;
|
||||
|
||||
// Try to acquire a very short-lived lock to test availability
|
||||
let mut req = Request::new(GenerallyLockRequest { args: args.clone() });
|
||||
req.set_timeout(rpc_timeout);
|
||||
attach_lock_mutation_body_digest(&mut req)?;
|
||||
// A probe lock granted after the deadline must not linger on the peer.
|
||||
let late = self.late_release_hook(lock_id.clone());
|
||||
let late = self.late_release_hook(probe_lock_id.clone());
|
||||
let late_failure = self.late_release_failure_hook(vec![probe_lock_id]);
|
||||
|
||||
// Try exclusive lock first with very short timeout
|
||||
let resp = match self
|
||||
.execute_rpc(
|
||||
"check_status",
|
||||
&resource_summary,
|
||||
Self::rpc_timeout(),
|
||||
rpc_timeout,
|
||||
async move { client.lock(req).await },
|
||||
late,
|
||||
RpcCleanup::new(late, late_failure, None),
|
||||
)
|
||||
.await
|
||||
{
|
||||
@@ -925,6 +1358,7 @@ impl LockClient for RemoteClient {
|
||||
// If we successfully acquired the lock, the resource was free.
|
||||
// Immediately release it on a best-effort basis.
|
||||
let mut release_req = Request::new(GenerallyLockRequest { args });
|
||||
release_req.set_timeout(rpc_timeout);
|
||||
attach_lock_mutation_body_digest(&mut release_req)?;
|
||||
if let Ok(mut client) = self.get_client().await {
|
||||
let _ = self
|
||||
@@ -933,7 +1367,7 @@ impl LockClient for RemoteClient {
|
||||
&resource_summary,
|
||||
Self::rpc_timeout(),
|
||||
async move { client.un_lock(release_req).await },
|
||||
None,
|
||||
RpcCleanup::new(None, None, None),
|
||||
)
|
||||
.await;
|
||||
}
|
||||
@@ -998,7 +1432,7 @@ impl LockClient for RemoteClient {
|
||||
Self::ONLINE_CHECK_RESOURCE,
|
||||
online_timeout,
|
||||
async move { client.ping(ping_req).await },
|
||||
None,
|
||||
RpcCleanup::new(None, None, None),
|
||||
)
|
||||
.await
|
||||
{
|
||||
@@ -1133,6 +1567,94 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn request_breaker_opens_after_three_timeouts_and_allows_one_probe() {
|
||||
let addr = "http://breaker-test";
|
||||
let client = RemoteClient::new(addr.to_string());
|
||||
reset_lock_peer_health_for_test(addr);
|
||||
|
||||
for _ in 0..LOCK_RPC_REQUEST_BACKOFF_THRESHOLD {
|
||||
client.record_request_timeout("lock", Duration::from_millis(100));
|
||||
}
|
||||
|
||||
assert!(
|
||||
client.admit_lock_request("lock").is_none(),
|
||||
"open breaker must reject requests during backoff"
|
||||
);
|
||||
with_lock_peer_health(addr, |health| {
|
||||
health.request_backoff_until = Some(Instant::now() - Duration::from_millis(1));
|
||||
health.request_probe_generation = None;
|
||||
health.request_in_flight = 0;
|
||||
});
|
||||
|
||||
let probe = client
|
||||
.admit_lock_request("lock")
|
||||
.expect("half-open breaker must admit one probe");
|
||||
assert!(probe.is_probe());
|
||||
assert!(client.admit_lock_request("lock").is_none(), "only one half-open probe may run");
|
||||
client.release_lock_request("lock", Some(probe));
|
||||
assert!(lock_peer_health_for_test(addr).request_probe_generation.is_none());
|
||||
reset_lock_peer_health_for_test(addr);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn old_request_completion_cannot_clear_a_new_probe() {
|
||||
let addr = "http://breaker-token-test";
|
||||
let client = RemoteClient::new(addr.to_string());
|
||||
reset_lock_peer_health_for_test(addr);
|
||||
with_lock_peer_health(addr, |health| {
|
||||
health.request_backoff_until = Some(Instant::now() - Duration::from_millis(1));
|
||||
health.request_probe_generation = None;
|
||||
health.request_in_flight = 1;
|
||||
});
|
||||
|
||||
let probe = client.admit_lock_request("lock").expect("probe should be admitted");
|
||||
client.release_lock_request("lock", Some(LockRequestAdmission { probe_generation: None }));
|
||||
assert!(lock_peer_health_for_test(addr).request_probe_generation.is_some());
|
||||
client.record_rpc_success("lock", Some(LockRequestAdmission { probe_generation: None }));
|
||||
assert!(lock_peer_health_for_test(addr).request_probe_generation.is_some());
|
||||
client.release_lock_request("lock", Some(probe));
|
||||
assert!(lock_peer_health_for_test(addr).request_probe_generation.is_none());
|
||||
reset_lock_peer_health_for_test(addr);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn stale_probe_completion_cannot_clear_a_later_generation() {
|
||||
let addr = "http://breaker-generation-test";
|
||||
let client = RemoteClient::new(addr.to_string());
|
||||
reset_lock_peer_health_for_test(addr);
|
||||
with_lock_peer_health(addr, |health| {
|
||||
health.request_backoff_until = Some(Instant::now() - Duration::from_millis(1));
|
||||
});
|
||||
|
||||
let first = client.admit_lock_request("lock").expect("first probe should be admitted");
|
||||
client.record_request_timeout("lock", Duration::from_millis(10));
|
||||
with_lock_peer_health(addr, |health| {
|
||||
health.request_backoff_until = Some(Instant::now() - Duration::from_millis(1));
|
||||
health.request_in_flight = 0;
|
||||
health.request_probe_generation = None;
|
||||
health.request_consecutive_timeouts = 0;
|
||||
});
|
||||
let second = client.admit_lock_request("lock").expect("second probe should be admitted");
|
||||
assert_ne!(first.probe_generation, second.probe_generation);
|
||||
|
||||
client.release_lock_request("lock", Some(first));
|
||||
assert_eq!(lock_peer_health_for_test(addr).request_probe_generation, second.probe_generation);
|
||||
client.release_lock_request("lock", Some(second));
|
||||
reset_lock_peer_health_for_test(addr);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bounded_acquire_request_matches_transport_deadline() {
|
||||
let request = test_lock_request(Duration::from_secs(30));
|
||||
let bounded = RemoteClient::bounded_acquire_request(&request, Duration::from_secs(3));
|
||||
assert_eq!(bounded.acquire_timeout, Duration::from_secs(3));
|
||||
|
||||
let request = test_lock_request(Duration::from_secs(1));
|
||||
let bounded = RemoteClient::bounded_acquire_request(&request, Duration::from_secs(3));
|
||||
assert_eq!(bounded.acquire_timeout, Duration::from_secs(1));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn acquired_lock_ids_picks_only_granted_batch_entries() {
|
||||
let lock_ids = vec![
|
||||
|
||||
@@ -41,7 +41,7 @@ use crate::config::com::{
|
||||
};
|
||||
use crate::data_movement;
|
||||
use crate::data_movement::backpressure::{self, DataMovementOperation};
|
||||
use crate::data_usage::DATA_USAGE_CACHE_NAME;
|
||||
use crate::data_usage::{DATA_USAGE_CACHE_NAME, DATA_USAGE_ROOT, load_data_usage_cache};
|
||||
use crate::disk::error::DiskError;
|
||||
use crate::disk::{BUCKET_META_PREFIX, DiskAPI, RUSTFS_META_BUCKET};
|
||||
use crate::error::{Error, Result};
|
||||
@@ -135,6 +135,7 @@ const DECOMMISSION_CAPACITY_RELEASE_FAILED: &str = "failed";
|
||||
const DECOMMISSION_CAPACITY_RELEASE_COMPLETED: &str = "completed";
|
||||
pub(crate) const DECOMMISSION_CAPACITY_TARGET_LOCK_PREFIX: &str = "decommission/capacity-target";
|
||||
const DECOMMISSION_CAPACITY_TARGET_LOCK_TIMEOUT: std::time::Duration = std::time::Duration::from_millis(250);
|
||||
const DECOMMISSION_CAPACITY_TARGET_GATE_MAX_ATTEMPTS: usize = 12;
|
||||
const DECOMMISSION_CAPACITY_TARGET_GATE_BUSY_PREFIX: &str = "target pool ";
|
||||
const DECOMMISSION_CAPACITY_TARGET_GATE_BUSY_SUFFIX: &str = " target capacity mutation gate is busy";
|
||||
const METRIC_DECOMMISSION_CAPACITY_CONFLICTS_TOTAL: &str = "rustfs_decommission_capacity_conflicts_total";
|
||||
@@ -147,12 +148,21 @@ const DECOMMISSION_LISTING_MAX_ATTEMPTS: usize = 3;
|
||||
const DECOMMISSION_LISTING_RETRY_DELAY: std::time::Duration = std::time::Duration::from_secs(5);
|
||||
pub(crate) const DECOMMISSION_ENTRY_MAX_ATTEMPTS: usize = 3;
|
||||
const DECOMMISSION_CAPACITY_INTENT_CONFLICT_MAX_ATTEMPTS: usize = 12;
|
||||
// A scanner pause backlog handoff conflict is a transient membership
|
||||
// transition: the native authority is expected to converge once surviving
|
||||
// replicas advance their durable commit. Bound the wait so a genuinely
|
||||
// divergent authority still reaches a terminal, attributable failure.
|
||||
const DECOMMISSION_SCANNER_BACKLOG_HANDOFF_MAX_ATTEMPTS: usize = 12;
|
||||
const DECOMMISSION_SOURCE_CLEANUP_RETRY_DELAY: std::time::Duration = std::time::Duration::from_millis(100);
|
||||
pub(crate) const DECOMMISSION_VERSION_COPY_ATTEMPTS: usize = 3;
|
||||
const DECOMMISSION_COPY_RETRY_DELAY: std::time::Duration = std::time::Duration::from_millis(50);
|
||||
const DECOMMISSION_SOURCE_CHANGED_EXHAUSTION_LIMIT: usize = 100;
|
||||
const DECOMMISSION_TERMINAL_RETRY_DELAY: std::time::Duration = std::time::Duration::from_secs(1);
|
||||
const DECOMMISSION_CANCEL_TARGET_LOCK_MAX_ATTEMPTS: usize = 3;
|
||||
|
||||
fn decommission_capacity_target_gate_retry_exhausted(attempt: usize) -> bool {
|
||||
attempt.saturating_add(1) >= DECOMMISSION_CAPACITY_TARGET_GATE_MAX_ATTEMPTS
|
||||
}
|
||||
const DECOMMISSION_DURABLE_ILM_RECEIPT_ROOT: &str = "decommission/ilm-receipts";
|
||||
const DECOMMISSION_DURABLE_ILM_MANIFEST_ROOT: &str = "decommission/ilm-manifests";
|
||||
const DECOMMISSION_DURABLE_ILM_RECEIPT_SCHEMA: &str = "v2";
|
||||
@@ -920,6 +930,29 @@ fn capacity_target_physical_bytes(data_bytes: usize, layout: DecommissionErasure
|
||||
Ok(capacity_mul_div_ceil(data_bytes, layout.width(), layout.data))
|
||||
}
|
||||
|
||||
async fn decommission_owned_physical_usage(sets: &Sets, layout: DecommissionErasureLayout) -> HashMap<i32, usize> {
|
||||
let mut usage = HashMap::with_capacity(sets.disk_set.len());
|
||||
for (set_index, set) in sets.disk_set.iter().enumerate() {
|
||||
let Ok(cache) = load_data_usage_cache(set.as_ref(), DATA_USAGE_CACHE_NAME).await else {
|
||||
continue;
|
||||
};
|
||||
let data_usage = cache.dui(DATA_USAGE_ROOT, &[]);
|
||||
if !data_usage.is_complete_bucket_usage_snapshot() {
|
||||
continue;
|
||||
}
|
||||
let Ok(logical_bytes) = usize::try_from(data_usage.objects_total_size) else {
|
||||
continue;
|
||||
};
|
||||
let Ok(set_index) = i32::try_from(set_index) else {
|
||||
continue;
|
||||
};
|
||||
// The data-usage snapshot excludes unrelated files on a shared mount.
|
||||
// Convert logical object bytes to the source set's physical EC footprint.
|
||||
usage.insert(set_index, capacity_mul_div_ceil(logical_bytes, layout.width(), layout.data));
|
||||
}
|
||||
usage
|
||||
}
|
||||
|
||||
fn worst_decommission_target_layout(targets: &[DecommissionPoolCapacityInfo]) -> Result<DecommissionErasureLayout> {
|
||||
targets
|
||||
.iter()
|
||||
@@ -996,6 +1029,30 @@ fn decommission_capacity_retry_kind(err: &Error, intent_conflict_attempt: usize)
|
||||
.then_some(DecommissionCapacityRetryKind::IntentConflict)
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
enum DecommissionScannerBacklogRetryKind {
|
||||
HandoffConflict,
|
||||
}
|
||||
|
||||
fn is_decommission_scanner_backlog_handoff_conflict(err: &Error) -> bool {
|
||||
data_movement::data_movement_stage_source_as::<data_movement::scanner_backlog::ScannerPauseBacklogRetirementError>(err)
|
||||
.is_some_and(data_movement::scanner_backlog::ScannerPauseBacklogRetirementError::is_retryable_handoff_conflict)
|
||||
}
|
||||
|
||||
/// Classify a retryable scanner pause backlog handoff conflict.
|
||||
///
|
||||
/// The typed planner error is the only signal consulted: string matching would
|
||||
/// turn unrelated retirement failures into retries. `handoff_conflict_attempt`
|
||||
/// is the number of retries already consumed by this entry.
|
||||
fn decommission_scanner_backlog_retry_kind(
|
||||
err: &Error,
|
||||
handoff_conflict_attempt: usize,
|
||||
) -> Option<DecommissionScannerBacklogRetryKind> {
|
||||
(handoff_conflict_attempt < DECOMMISSION_SCANNER_BACKLOG_HANDOFF_MAX_ATTEMPTS
|
||||
&& is_decommission_scanner_backlog_handoff_conflict(err))
|
||||
.then_some(DecommissionScannerBacklogRetryKind::HandoffConflict)
|
||||
}
|
||||
|
||||
fn ensure_decommission_capacity_target_fence(
|
||||
guard: &rustfs_lock::NamespaceLockGuard,
|
||||
target_pool_index: usize,
|
||||
@@ -1832,6 +1889,132 @@ fn ensure_external_decommission_target_admission(
|
||||
Err(Error::SlowDown)
|
||||
}
|
||||
|
||||
fn grow_decommission_target_reservation(
|
||||
meta: &mut PoolMeta,
|
||||
capacity_infos: &[DecommissionPoolCapacityInfo],
|
||||
source_pool_index: usize,
|
||||
target_pool_index: usize,
|
||||
expected_data_bytes: usize,
|
||||
mutation_id: uuid::Uuid,
|
||||
now: OffsetDateTime,
|
||||
) -> Result<bool> {
|
||||
let original = meta.clone();
|
||||
let result = (|| {
|
||||
let target_capacity = capacity_infos
|
||||
.iter()
|
||||
.find(|capacity| capacity.pool_index == target_pool_index)
|
||||
.ok_or_else(|| {
|
||||
decommission_capacity_blocked_error("target capacity snapshot is missing during reservation growth")
|
||||
})?;
|
||||
let pool_count = meta.pools.len();
|
||||
let pool = meta
|
||||
.pools
|
||||
.get_mut(source_pool_index)
|
||||
.ok_or_else(|| invalid_decommission_pool_index_error(pool_count, source_pool_index))?;
|
||||
let reservation = pool
|
||||
.decommission
|
||||
.as_mut()
|
||||
.and_then(|info| info.capacity_reservation.as_mut())
|
||||
.filter(|reservation| reservation.active())
|
||||
.ok_or_else(|| decommission_capacity_blocked_error("active reservation disappeared during reservation growth"))?;
|
||||
let target_layout = reservation
|
||||
.targets
|
||||
.iter()
|
||||
.find(|target| target.pool_index == target_pool_index)
|
||||
.map(|target| target.layout)
|
||||
.unwrap_or(target_capacity.layout);
|
||||
let data_bytes = expected_data_bytes.max(1);
|
||||
let required_target_bytes = capacity_target_physical_bytes(data_bytes, target_layout)?;
|
||||
let required_peak = required_target_bytes.saturating_mul(1usize.saturating_add(reservation.temporary_copies));
|
||||
let existing_target = reservation
|
||||
.targets
|
||||
.iter()
|
||||
.find(|target| target.pool_index == target_pool_index);
|
||||
if let Some(target) = existing_target
|
||||
&& target.pending_physical_bytes > 0
|
||||
&& target.pending_mutation_id != Some(mutation_id)
|
||||
{
|
||||
return Err(decommission_capacity_blocked_error(format!(
|
||||
"source pool {source_pool_index} target pool {target_pool_index} has an unresolved target capacity intent"
|
||||
)));
|
||||
}
|
||||
let existing_remaining = existing_target
|
||||
.map(|target| target.remaining_reserved_physical_bytes(reservation.temporary_copies))
|
||||
.unwrap_or_default();
|
||||
if required_peak <= existing_remaining {
|
||||
return Ok(false);
|
||||
}
|
||||
if reservation.source_data_equivalent_bytes > 0 {
|
||||
return Err(decommission_capacity_blocked_error(format!(
|
||||
"source pool {source_pool_index} reservation estimate is exhausted"
|
||||
)));
|
||||
}
|
||||
if let Some(target) = existing_target
|
||||
&& target_capacity.physical_free <= target.physical_free_at_reservation
|
||||
{
|
||||
return Err(decommission_capacity_blocked_error(format!(
|
||||
"source pool {source_pool_index} target pool {target_pool_index} reservation is exhausted"
|
||||
)));
|
||||
}
|
||||
let current_peak = reservation.remaining_peak_physical_bytes();
|
||||
let additional_peak = required_peak.saturating_sub(existing_remaining);
|
||||
let additional_predicted = additional_peak.div_ceil(1usize.saturating_add(reservation.temporary_copies));
|
||||
let additional_source_data = capacity_source_data_equivalent(additional_predicted, target_layout)?;
|
||||
reservation.source_data_equivalent_bytes = reservation
|
||||
.source_data_equivalent_bytes
|
||||
.saturating_add(additional_source_data);
|
||||
reservation.source_physical_bytes = reservation
|
||||
.source_physical_bytes
|
||||
.saturating_add(capacity_target_physical_bytes(additional_source_data, reservation.source_layout)?);
|
||||
reservation.predicted_physical_bytes = reservation
|
||||
.predicted_physical_bytes
|
||||
.saturating_add(additional_predicted)
|
||||
.max(capacity_target_physical_bytes(reservation.source_data_equivalent_bytes, target_layout)?);
|
||||
reservation.temporary_physical_bytes = reservation
|
||||
.predicted_physical_bytes
|
||||
.saturating_mul(reservation.temporary_copies);
|
||||
reservation.peak_physical_bytes = reservation
|
||||
.predicted_physical_bytes
|
||||
.saturating_add(reservation.temporary_physical_bytes);
|
||||
let allocation_growth = reservation.remaining_peak_physical_bytes().saturating_sub(current_peak);
|
||||
if let Some(target) = reservation
|
||||
.targets
|
||||
.iter_mut()
|
||||
.find(|target| target.pool_index == target_pool_index)
|
||||
{
|
||||
target.reserved_physical_bytes = target.reserved_physical_bytes.saturating_add(allocation_growth);
|
||||
target.physical_total_at_reservation = target.physical_total_at_reservation.max(target_capacity.physical_total);
|
||||
target.physical_free_at_reservation = target.physical_free_at_reservation.max(target_capacity.physical_free);
|
||||
} else {
|
||||
reservation.targets.push(DecommissionCapacityTarget {
|
||||
pool_index: target_pool_index,
|
||||
layout: target_layout,
|
||||
physical_total_at_reservation: target_capacity.physical_total,
|
||||
physical_free_at_reservation: target_capacity.physical_free,
|
||||
reserved_physical_bytes: allocation_growth,
|
||||
consumed_physical_bytes: 0,
|
||||
observed_physical_bytes: 0,
|
||||
inflight_physical_bytes: 0,
|
||||
pending_physical_bytes: 0,
|
||||
pending_mutation_id: None,
|
||||
temporary_mutations: Vec::new(),
|
||||
});
|
||||
}
|
||||
renew_decommission_capacity_reservation(reservation, now, true);
|
||||
pool.last_update = pool.last_update.max(now);
|
||||
Ok(true)
|
||||
})();
|
||||
if result.is_ok() {
|
||||
if let Err(err) = ensure_decommission_capacity_reservations_available(meta, capacity_infos, "reservation growth") {
|
||||
*meta = original;
|
||||
return Err(err);
|
||||
}
|
||||
} else {
|
||||
*meta = original;
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
fn ensure_decommission_target_owner_admission(
|
||||
meta: &PoolMeta,
|
||||
owner: DecommissionCapacityOwner,
|
||||
@@ -11373,10 +11556,11 @@ impl ECStore {
|
||||
}
|
||||
})
|
||||
.filter(|reservation| {
|
||||
reservation
|
||||
.targets
|
||||
.iter()
|
||||
.any(|target| target.pool_index == target_pool_index)
|
||||
!temporary_release
|
||||
|| reservation
|
||||
.targets
|
||||
.iter()
|
||||
.any(|target| target.pool_index == target_pool_index)
|
||||
})
|
||||
.map(|reservation| (owner, reservation.model_version))
|
||||
});
|
||||
@@ -11435,10 +11619,11 @@ impl ECStore {
|
||||
} else {
|
||||
reservation.admits_owner(owner, OffsetDateTime::now_utc())
|
||||
}) && reservation.model_version == model_version
|
||||
&& reservation
|
||||
.targets
|
||||
.iter()
|
||||
.any(|target| target.pool_index == target_pool_index)
|
||||
&& (!temporary_release
|
||||
|| reservation
|
||||
.targets
|
||||
.iter()
|
||||
.any(|target| target.pool_index == target_pool_index))
|
||||
})
|
||||
.is_some();
|
||||
if !owner_current {
|
||||
@@ -11447,9 +11632,22 @@ impl ECStore {
|
||||
));
|
||||
}
|
||||
let capacity_infos = self.get_decommission_all_pool_capacity_infos().await?;
|
||||
if !temporary_release {
|
||||
let reservation_grew = if temporary_release {
|
||||
false
|
||||
} else {
|
||||
let data_bytes = expected_data_bytes.unwrap_or(1).max(1);
|
||||
let grew = grow_decommission_target_reservation(
|
||||
&mut snapshot,
|
||||
&capacity_infos,
|
||||
source_pool_index,
|
||||
target_pool_index,
|
||||
data_bytes,
|
||||
mutation_id,
|
||||
OffsetDateTime::now_utc(),
|
||||
)?;
|
||||
ensure_decommission_capacity_reservations_available(&snapshot, &capacity_infos, "mutation")?;
|
||||
}
|
||||
grew
|
||||
};
|
||||
let target_layout = snapshot
|
||||
.pools
|
||||
.get(source_pool_index)
|
||||
@@ -11515,7 +11713,7 @@ impl ECStore {
|
||||
OffsetDateTime::now_utc(),
|
||||
)?
|
||||
};
|
||||
if pending_added > 0 {
|
||||
if reservation_grew || pending_added > 0 {
|
||||
let outcome = snapshot
|
||||
.save_no_lock_armed(self.pools.clone(), &mut save_guard, write_guard.lock_lost_signal(), &[source_pool_index])
|
||||
.await?;
|
||||
@@ -12357,7 +12555,11 @@ impl ECStore {
|
||||
Ok(())
|
||||
}
|
||||
|
||||
async fn get_decommission_pool_capacity_info(&self, idx: usize) -> Result<DecommissionPoolCapacityInfo> {
|
||||
async fn get_decommission_pool_capacity_info(
|
||||
&self,
|
||||
idx: usize,
|
||||
use_owned_usage: bool,
|
||||
) -> Result<DecommissionPoolCapacityInfo> {
|
||||
if let Some(sets) = self.pools.get(idx) {
|
||||
let mut info = sets.storage_info_snapshot().await;
|
||||
info.backend = StorageAdminApi::backend_info(self).await;
|
||||
@@ -12389,8 +12591,13 @@ impl ECStore {
|
||||
layout.data, layout.parity
|
||||
)));
|
||||
}
|
||||
let owned_physical_used = if use_owned_usage {
|
||||
Some(decommission_owned_physical_usage(sets, layout).await)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
let (physical_total, physical_free, physical_used) =
|
||||
decommission_physical_pool_capacity(&info.disks, idx, layout, space);
|
||||
decommission_physical_pool_capacity(&info.disks, idx, layout, space, owned_physical_used.as_ref());
|
||||
|
||||
Ok(DecommissionPoolCapacityInfo {
|
||||
pool_index: idx,
|
||||
@@ -12413,7 +12620,20 @@ impl ECStore {
|
||||
|
||||
let mut capacity_infos = Vec::with_capacity(self.pools.len());
|
||||
for idx in 0..self.pools.len() {
|
||||
capacity_infos.push(self.get_decommission_pool_capacity_info(idx).await?);
|
||||
capacity_infos.push(self.get_decommission_pool_capacity_info(idx, false).await?);
|
||||
}
|
||||
Ok(capacity_infos)
|
||||
}
|
||||
|
||||
async fn get_decommission_all_pool_capacity_infos_with_owned_usage(&self) -> Result<Vec<DecommissionPoolCapacityInfo>> {
|
||||
#[cfg(any(test, feature = "test-util"))]
|
||||
if let Some(capacity_infos) = take_decommission_capacity_info_override_for_test(self.id) {
|
||||
return Ok(capacity_infos);
|
||||
}
|
||||
|
||||
let mut capacity_infos = Vec::with_capacity(self.pools.len());
|
||||
for idx in 0..self.pools.len() {
|
||||
capacity_infos.push(self.get_decommission_pool_capacity_info(idx, true).await?);
|
||||
}
|
||||
Ok(capacity_infos)
|
||||
}
|
||||
@@ -12515,15 +12735,37 @@ impl ECStore {
|
||||
// different target so one mutation never holds two target gates.
|
||||
discard_decommission_capacity_target_permit_except(self.id, owner, None);
|
||||
}
|
||||
reservation
|
||||
if let Some((pool_index, _)) = reservation
|
||||
.targets
|
||||
.iter()
|
||||
.filter_map(candidate)
|
||||
.max_by_key(|(_, remaining)| *remaining)
|
||||
.map(|(pool_index, _)| pool_index)
|
||||
{
|
||||
return Ok(pool_index);
|
||||
}
|
||||
if reservation.source_data_equivalent_bytes > 0 {
|
||||
return Err(decommission_capacity_blocked_error(format!(
|
||||
"source pool {} has no target allocation for {expected_data_bytes} data bytes",
|
||||
owner.source_pool_index
|
||||
)));
|
||||
}
|
||||
let capacities = self.get_decommission_all_pool_capacity_infos().await?;
|
||||
let active_sources = active_decommission_source_indices(&pool_meta);
|
||||
capacities
|
||||
.into_iter()
|
||||
.filter(|capacity| {
|
||||
capacity.pool_index != owner.source_pool_index
|
||||
&& !active_sources.contains(&capacity.pool_index)
|
||||
&& pool_meta
|
||||
.pools
|
||||
.get(capacity.pool_index)
|
||||
.is_some_and(is_decommission_start_active_pool)
|
||||
})
|
||||
.max_by_key(|capacity| capacity.physical_free)
|
||||
.map(|capacity| capacity.pool_index)
|
||||
.ok_or_else(|| {
|
||||
decommission_capacity_blocked_error(format!(
|
||||
"source pool {} has no target allocation for {expected_data_bytes} data bytes",
|
||||
"source pool {} has no target pool for {expected_data_bytes} data bytes",
|
||||
owner.source_pool_index
|
||||
))
|
||||
})
|
||||
@@ -13019,7 +13261,7 @@ impl ECStore {
|
||||
let capacity_infos = if self.pools.is_empty() {
|
||||
Vec::new()
|
||||
} else {
|
||||
self.get_decommission_all_pool_capacity_infos().await?
|
||||
self.get_decommission_all_pool_capacity_infos_with_owned_usage().await?
|
||||
};
|
||||
let target_fence_proof = crate::services::notification_sys::acquire_decommission_target_fence_fleet_proof();
|
||||
let mut pool_meta = self.pool_meta.write().await;
|
||||
@@ -13720,6 +13962,20 @@ impl ECStore {
|
||||
match self.acquire_decommission_capacity_target_guard(target_pool_index).await {
|
||||
Ok(guard) => break guard,
|
||||
Err(err) if is_decommission_capacity_target_gate_busy(&err) => {
|
||||
if decommission_capacity_target_gate_retry_exhausted(wait_attempt) {
|
||||
warn!(
|
||||
event = EVENT_DECOMMISSION_STATE,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_POOLS,
|
||||
pool_index = idx,
|
||||
target_pool_index,
|
||||
attempts = wait_attempt.saturating_add(1),
|
||||
state = "capacity_gate_retry_exhausted",
|
||||
error = %err,
|
||||
"Decommission target capacity gate remained busy; pausing for supervised retry"
|
||||
);
|
||||
return Err(err);
|
||||
}
|
||||
wait_attempt = wait_attempt.saturating_add(1);
|
||||
}
|
||||
Err(err) => return Err(err),
|
||||
@@ -14058,6 +14314,7 @@ impl ECStore {
|
||||
for entry_attempt in 1..=DECOMMISSION_ENTRY_MAX_ATTEMPTS {
|
||||
let attempt_result = {
|
||||
let mut conflict_attempt = 0;
|
||||
let mut scanner_backlog_handoff_attempt = 0;
|
||||
loop {
|
||||
let result = self
|
||||
.decommission_entry_attempt(
|
||||
@@ -14072,6 +14329,7 @@ impl ECStore {
|
||||
replication_config.clone(),
|
||||
expected_bucket_incarnation_id,
|
||||
entry_attempt,
|
||||
scanner_backlog_handoff_attempt,
|
||||
source_changed_exhaustions.as_ref(),
|
||||
&mut counted_versions,
|
||||
)
|
||||
@@ -14080,17 +14338,48 @@ impl ECStore {
|
||||
.as_ref()
|
||||
.err()
|
||||
.and_then(|err| decommission_capacity_retry_kind(err, conflict_attempt));
|
||||
let retry_attempt = match retry {
|
||||
Some(DecommissionCapacityRetryKind::IntentConflict) => {
|
||||
conflict_attempt += 1;
|
||||
conflict_attempt
|
||||
if let Some(DecommissionCapacityRetryKind::IntentConflict) = retry {
|
||||
conflict_attempt += 1;
|
||||
let retry_delay =
|
||||
decommission_retry_backoff_delay(DECOMMISSION_SOURCE_CLEANUP_RETRY_DELAY, conflict_attempt);
|
||||
if wait_decommission_retry_backoff(&rx, retry_delay).await {
|
||||
decommission_cancel_signal_result(rx.is_cancelled())?;
|
||||
}
|
||||
None => break result,
|
||||
};
|
||||
let retry_delay = decommission_retry_backoff_delay(DECOMMISSION_SOURCE_CLEANUP_RETRY_DELAY, retry_attempt);
|
||||
if wait_decommission_retry_backoff(&rx, retry_delay).await {
|
||||
decommission_cancel_signal_result(rx.is_cancelled())?;
|
||||
continue;
|
||||
}
|
||||
// Scanner pause backlog handoff conflicts resolve when the
|
||||
// surviving membership converges. Retry on a bounded
|
||||
// backoff that never writes failure state, clears
|
||||
// `start_time`, or releases the capacity reservation.
|
||||
let scanner_retry = result
|
||||
.as_ref()
|
||||
.err()
|
||||
.and_then(|err| decommission_scanner_backlog_retry_kind(err, scanner_backlog_handoff_attempt));
|
||||
if let Some(DecommissionScannerBacklogRetryKind::HandoffConflict) = scanner_retry {
|
||||
scanner_backlog_handoff_attempt += 1;
|
||||
let retry_delay = decommission_retry_backoff_delay(
|
||||
DECOMMISSION_SOURCE_CLEANUP_RETRY_DELAY,
|
||||
scanner_backlog_handoff_attempt,
|
||||
);
|
||||
info!(
|
||||
event = EVENT_DECOMMISSION_ENTRY,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_POOLS,
|
||||
state = "scanner_backlog_handoff_retry",
|
||||
pool_index = idx,
|
||||
bucket = %bucket,
|
||||
object = %entry.name,
|
||||
attempt = scanner_backlog_handoff_attempt,
|
||||
max_attempts = DECOMMISSION_SCANNER_BACKLOG_HANDOFF_MAX_ATTEMPTS,
|
||||
retry_delay_ms = retry_delay.as_millis(),
|
||||
"Decommission scanner pause backlog handoff conflict; retrying entry"
|
||||
);
|
||||
if wait_decommission_retry_backoff(&rx, retry_delay).await {
|
||||
decommission_cancel_signal_result(rx.is_cancelled())?;
|
||||
}
|
||||
continue;
|
||||
}
|
||||
break result;
|
||||
}
|
||||
};
|
||||
match attempt_result {
|
||||
@@ -14138,6 +14427,7 @@ impl ECStore {
|
||||
replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
|
||||
expected_bucket_incarnation_id: Option<uuid::Uuid>,
|
||||
entry_attempt: usize,
|
||||
scanner_backlog_handoff_attempt: usize,
|
||||
source_changed_exhaustions: &AtomicUsize,
|
||||
counted_versions: &mut HashSet<(Option<uuid::Uuid>, bool)>,
|
||||
) -> Result<DecommissionEntryAttemptOutcome> {
|
||||
@@ -14197,27 +14487,56 @@ impl ECStore {
|
||||
if data_movement::scanner_backlog::is_scanner_pause_backlog(&bucket, &entry.name) {
|
||||
let outcome = self
|
||||
.retire_scanner_pause_backlog_entry(rx, idx, generation, Arc::clone(&set), fivs.clone(), capacity_owner)
|
||||
.await?;
|
||||
if matches!(outcome, DecommissionEntryAttemptOutcome::Complete) {
|
||||
let mut pool_meta = self.pool_meta.write().await;
|
||||
ensure_decommission_generation(&pool_meta, idx, generation)?;
|
||||
if let Some(version) = fivs.versions.first()
|
||||
&& counted_versions.insert((version.version_id, false))
|
||||
{
|
||||
count_decommission_item(&mut pool_meta, idx, decommission_item_size(version.size), false)?;
|
||||
.await;
|
||||
return match outcome {
|
||||
Ok(DecommissionEntryAttemptOutcome::Complete) => {
|
||||
let mut pool_meta = self.pool_meta.write().await;
|
||||
ensure_decommission_generation(&pool_meta, idx, generation)?;
|
||||
if let Some(version) = fivs.versions.first()
|
||||
&& counted_versions.insert((version.version_id, false))
|
||||
{
|
||||
count_decommission_item(&mut pool_meta, idx, decommission_item_size(version.size), false)?;
|
||||
}
|
||||
track_decommission_current_object(&mut pool_meta, idx, &bucket, &entry.name)?;
|
||||
drop(pool_meta);
|
||||
self.track_decommission_entry_progress_stage(
|
||||
idx,
|
||||
generation,
|
||||
&bucket,
|
||||
&entry.name,
|
||||
DECOMMISSION_STAGE_ENTRY_FINISHED,
|
||||
)
|
||||
.await?;
|
||||
Ok(DecommissionEntryAttemptOutcome::Complete)
|
||||
}
|
||||
track_decommission_current_object(&mut pool_meta, idx, &bucket, &entry.name)?;
|
||||
drop(pool_meta);
|
||||
self.track_decommission_entry_progress_stage(
|
||||
idx,
|
||||
generation,
|
||||
&bucket,
|
||||
&entry.name,
|
||||
DECOMMISSION_STAGE_ENTRY_FINISHED,
|
||||
)
|
||||
.await?;
|
||||
}
|
||||
return Ok(outcome);
|
||||
Ok(outcome) => Ok(outcome),
|
||||
Err(err) => {
|
||||
// The retry loop owns the handoff-conflict budget, so a
|
||||
// retryable conflict must not be attributed here. Once that
|
||||
// budget is exhausted the failure is terminal and belongs to
|
||||
// this entry rather than to whichever object was processed
|
||||
// last.
|
||||
let retryable_handoff = data_movement::data_movement_stage_source_as::<
|
||||
data_movement::scanner_backlog::ScannerPauseBacklogRetirementError,
|
||||
>(&err)
|
||||
.is_some_and(
|
||||
data_movement::scanner_backlog::ScannerPauseBacklogRetirementError::is_retryable_handoff_conflict,
|
||||
);
|
||||
let terminal = !retryable_handoff
|
||||
|| scanner_backlog_handoff_attempt >= DECOMMISSION_SCANNER_BACKLOG_HANDOFF_MAX_ATTEMPTS;
|
||||
if terminal {
|
||||
let mut pool_meta = self.pool_meta.write().await;
|
||||
ensure_decommission_generation(&pool_meta, idx, generation)?;
|
||||
if let Some(version) = fivs.versions.first()
|
||||
&& counted_versions.insert((version.version_id, true))
|
||||
{
|
||||
count_decommission_item(&mut pool_meta, idx, decommission_item_size(version.size), true)?;
|
||||
}
|
||||
track_decommission_current_object(&mut pool_meta, idx, &bucket, &entry.name)?;
|
||||
}
|
||||
Err(err)
|
||||
}
|
||||
};
|
||||
}
|
||||
|
||||
let pending_mutations = if let Some(owner) = capacity_owner {
|
||||
@@ -15263,7 +15582,7 @@ impl ECStore {
|
||||
.is_some_and(|reservation| !reservation.active())
|
||||
};
|
||||
if needs_capacity_reservation {
|
||||
let capacity_infos = self.get_decommission_all_pool_capacity_infos().await?;
|
||||
let capacity_infos = self.get_decommission_all_pool_capacity_infos_with_owned_usage().await?;
|
||||
let mut pool_meta = self.pool_meta.write().await;
|
||||
let version = pool_meta.version;
|
||||
pool_meta.version = POOL_META_VERSION;
|
||||
@@ -16127,7 +16446,7 @@ impl ECStore {
|
||||
self.ensure_decommission_rebalance_idle_after_refresh_under_start_gate()
|
||||
.await?;
|
||||
|
||||
let all_capacity_infos = self.get_decommission_all_pool_capacity_infos().await?;
|
||||
let all_capacity_infos = self.get_decommission_all_pool_capacity_infos_with_owned_usage().await?;
|
||||
let target_fence_proof_available =
|
||||
crate::services::notification_sys::acquire_decommission_target_fence_fleet_proof().is_some();
|
||||
// Signal cancellation before waiting for the movement writer so active
|
||||
@@ -21687,6 +22006,45 @@ mod tests {
|
||||
assert!(!is_decommission_target_capacity_error(&Error::SlowDown));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decommission_scanner_backlog_handoff_retry_is_typed_and_bounded() {
|
||||
use data_movement::scanner_backlog::ScannerPauseBacklogRetirementError;
|
||||
|
||||
let handoff = data_movement::data_movement_context_error(
|
||||
"scanner pause backlog retirement planner failed: competing proofs".to_string(),
|
||||
ScannerPauseBacklogRetirementError::HandoffConflict {
|
||||
reason: "competing maximum-membership commit proofs".to_string(),
|
||||
},
|
||||
);
|
||||
assert!(is_decommission_scanner_backlog_handoff_conflict(&handoff));
|
||||
assert_eq!(
|
||||
decommission_scanner_backlog_retry_kind(&handoff, DECOMMISSION_SCANNER_BACKLOG_HANDOFF_MAX_ATTEMPTS - 1),
|
||||
Some(DecommissionScannerBacklogRetryKind::HandoffConflict)
|
||||
);
|
||||
assert_eq!(
|
||||
decommission_scanner_backlog_retry_kind(&handoff, DECOMMISSION_SCANNER_BACKLOG_HANDOFF_MAX_ATTEMPTS),
|
||||
None,
|
||||
"a divergent handoff must not retry forever"
|
||||
);
|
||||
|
||||
for non_retryable in [
|
||||
ScannerPauseBacklogRetirementError::AuthorityConflict {
|
||||
reason: "larger source authority".to_string(),
|
||||
},
|
||||
ScannerPauseBacklogRetirementError::InvalidRecord {
|
||||
reason: "unread member".to_string(),
|
||||
},
|
||||
] {
|
||||
let err = data_movement::data_movement_context_error("planner failed".to_string(), non_retryable);
|
||||
assert!(!is_decommission_scanner_backlog_handoff_conflict(&err));
|
||||
assert_eq!(
|
||||
decommission_scanner_backlog_retry_kind(&err, 0),
|
||||
None,
|
||||
"only a handoff conflict may be retried"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn should_skip_decommission_delete_marker_characterizes_empty_marker_without_replication() {
|
||||
let version = rustfs_filemeta::FileInfo {
|
||||
@@ -22522,6 +22880,7 @@ fn decommission_physical_pool_capacity(
|
||||
pool_index: usize,
|
||||
layout: DecommissionErasureLayout,
|
||||
logical: PoolSpaceInfo,
|
||||
owned_physical_used: Option<&HashMap<i32, usize>>,
|
||||
) -> (usize, usize, usize) {
|
||||
let width = layout.width();
|
||||
let mut sets: HashMap<i32, Vec<&rustfs_madmin::Disk>> = HashMap::new();
|
||||
@@ -22561,13 +22920,20 @@ fn decommission_physical_pool_capacity(
|
||||
.map(|disk| disk.available_space as usize)
|
||||
.min()
|
||||
.unwrap_or_default();
|
||||
let max_used = set_disks
|
||||
.iter()
|
||||
.map(|disk| (disk.used_space as usize).max((disk.total_space as usize).saturating_sub(disk.available_space as usize)))
|
||||
.max()
|
||||
.unwrap_or_default();
|
||||
let physical_used_for_set = owned_physical_used
|
||||
.and_then(|owned| owned.get(&set_disks[0].set_index).copied())
|
||||
.unwrap_or_else(|| {
|
||||
set_disks
|
||||
.iter()
|
||||
.map(|disk| {
|
||||
(disk.used_space as usize).max((disk.total_space as usize).saturating_sub(disk.available_space as usize))
|
||||
})
|
||||
.max()
|
||||
.unwrap_or_default()
|
||||
.saturating_mul(width)
|
||||
});
|
||||
physical_total = physical_total.saturating_add(min_total.saturating_mul(width));
|
||||
physical_used = physical_used.saturating_add(max_used.saturating_mul(width));
|
||||
physical_used = physical_used.saturating_add(physical_used_for_set);
|
||||
if set_disks.len() >= width {
|
||||
physical_free = physical_free.saturating_add(min_free.saturating_mul(width));
|
||||
}
|
||||
@@ -22750,6 +23116,8 @@ pub(crate) fn fallback_free_capacity_dedup(disks: &[rustfs_madmin::Disk]) -> usi
|
||||
|
||||
#[cfg(test)]
|
||||
mod pools_tests {
|
||||
use std::collections::HashMap;
|
||||
|
||||
use super::DECOMMISSION_PROGRESS_SAVE_RETRY_BACKOFF;
|
||||
use super::persist_v3_pool_meta_for_test;
|
||||
use super::record_decommission_entry_error;
|
||||
@@ -22783,10 +23151,11 @@ mod pools_tests {
|
||||
ensure_decommission_start_rebalance_meta_allowed, ensure_decommission_start_target_capacity,
|
||||
ensure_decommission_terminal_operation_supported, ensure_decommission_unresolved_verification_disk_count,
|
||||
ensure_local_decommission_pool_leaders, ensure_pool_meta_write_fence, ensure_valid_decommission_pool_index, get_by_index,
|
||||
guard_decommission_cancelers, has_active_decommission_canceler, is_decommission_active, is_decommission_cancel_requested,
|
||||
load_decommission_entry_versions, local_decommission_queue_prefix, mark_decommission_bucket_done,
|
||||
merge_decommission_durable_ilm_receipts, merge_pool_meta_updates_for_save, merge_pool_status_refresh,
|
||||
missing_decommission_worker_prefix, next_decommission_capacity_generation, observe_decommission_terminal_reload_result,
|
||||
grow_decommission_target_reservation, guard_decommission_cancelers, has_active_decommission_canceler,
|
||||
is_decommission_active, is_decommission_cancel_requested, load_decommission_entry_versions,
|
||||
local_decommission_queue_prefix, mark_decommission_bucket_done, merge_decommission_durable_ilm_receipts,
|
||||
merge_pool_meta_updates_for_save, merge_pool_status_refresh, missing_decommission_worker_prefix,
|
||||
next_decommission_capacity_generation, observe_decommission_terminal_reload_result,
|
||||
parse_decommission_durable_ilm_receipt_path, pool_meta_has_active_decommission, publish_pool_meta_updates,
|
||||
read_pool_meta_replica, reconcile_decommission_meta_buckets, reconcile_decommission_unresolved_entries_for_completion,
|
||||
record_decommission_unresolved_entry, recover_decommission_capacity_reservations,
|
||||
@@ -22812,8 +23181,9 @@ mod pools_tests {
|
||||
with_decommission_entry_context,
|
||||
};
|
||||
use super::{
|
||||
DecommissionCapacityAdmission, DecommissionCapacityOwner, DecommissionCapacityReleaseProof,
|
||||
DecommissionCapacityReservation, DecommissionCapacityTemporaryMutation, decommission_capacity_mutation_id,
|
||||
DECOMMISSION_CAPACITY_TARGET_GATE_MAX_ATTEMPTS, DecommissionCapacityAdmission, DecommissionCapacityOwner,
|
||||
DecommissionCapacityReleaseProof, DecommissionCapacityReservation, DecommissionCapacityTemporaryMutation,
|
||||
decommission_capacity_mutation_id, decommission_capacity_target_gate_retry_exhausted,
|
||||
ensure_decommission_target_owner_admission, ensure_exact_delete_capacity_namespace_fences,
|
||||
ensure_external_decommission_target_admission, is_decommission_capacity_blocked_error,
|
||||
plan_exact_delete_capacity_reconciliations, record_decommission_target_consumption, release_decommission_target_inflight,
|
||||
@@ -22920,6 +23290,17 @@ mod pools_tests {
|
||||
identity: StdMutex<Option<(Vec<u8>, String)>>,
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decommission_target_gate_retry_is_bounded() {
|
||||
assert!(!decommission_capacity_target_gate_retry_exhausted(0));
|
||||
assert!(!decommission_capacity_target_gate_retry_exhausted(
|
||||
DECOMMISSION_CAPACITY_TARGET_GATE_MAX_ATTEMPTS - 2
|
||||
));
|
||||
assert!(decommission_capacity_target_gate_retry_exhausted(
|
||||
DECOMMISSION_CAPACITY_TARGET_GATE_MAX_ATTEMPTS - 1
|
||||
));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn pool_meta_recovery_reconciles_prepare_and_partial_commit_without_format_downgrade() {
|
||||
for previous_version in [POOL_META_V1_VERSION, POOL_META_VERSION, POOL_META_GENERATION_VERSION] {
|
||||
@@ -28341,6 +28722,44 @@ mod pools_tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decommission_capacity_reservation_grows_after_stale_empty_snapshot() {
|
||||
let now = OffsetDateTime::UNIX_EPOCH + Duration::minutes(1);
|
||||
let layout = DecommissionErasureLayout { data: 3, parity: 1 };
|
||||
let capacity_infos = vec![
|
||||
DecommissionPoolCapacityInfo::for_test(0, layout, 0, 0, 0),
|
||||
DecommissionPoolCapacityInfo::for_test(1, layout, 64, 64, 0),
|
||||
];
|
||||
let mut meta = PoolMeta {
|
||||
version: POOL_META_VERSION,
|
||||
pools: vec![decommission_test_pool_status(0, None), decommission_test_pool_status(1, None)],
|
||||
..Default::default()
|
||||
};
|
||||
meta.decommission(0, capacity_infos[0].space).unwrap();
|
||||
reserve_decommission_start_target_capacity(
|
||||
&mut meta,
|
||||
&[0],
|
||||
&capacity_infos,
|
||||
uuid::Uuid::new_v4(),
|
||||
1,
|
||||
now,
|
||||
DECOMMISSION_CAPACITY_MODEL_VERSION,
|
||||
)
|
||||
.expect("an empty scanner snapshot should still start decommission");
|
||||
let grew = grow_decommission_target_reservation(&mut meta, &capacity_infos, 0, 1, 1, uuid::Uuid::new_v4(), now)
|
||||
.expect("the reservation should grow against current target capacity");
|
||||
assert!(grew);
|
||||
let reservation = meta.pools[0]
|
||||
.decommission
|
||||
.as_ref()
|
||||
.and_then(|info| info.capacity_reservation.as_ref())
|
||||
.unwrap();
|
||||
assert_eq!(reservation.source_data_equivalent_bytes, 2);
|
||||
assert_eq!(reservation.predicted_physical_bytes, 3);
|
||||
assert_eq!(reservation.targets.len(), 1);
|
||||
assert_eq!(reservation.targets[0].reserved_physical_bytes, 6);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decommission_capacity_model_reserves_versions_delete_markers_and_temporary_copies_from_physical_usage() {
|
||||
let reservation = build_decommission_capacity_reservation(
|
||||
@@ -28897,11 +29316,46 @@ mod pools_tests {
|
||||
total: 0,
|
||||
used: 0,
|
||||
},
|
||||
None,
|
||||
);
|
||||
|
||||
assert_eq!(capacity, (400, 40, 360));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decommission_physical_capacity_prefers_rustfs_owned_usage_over_filesystem_usage() {
|
||||
let disks = [10_u64, 20, 30, 40]
|
||||
.into_iter()
|
||||
.enumerate()
|
||||
.map(|(disk_index, available_space)| rustfs_madmin::Disk {
|
||||
endpoint: format!("http://node-{disk_index}"),
|
||||
drive_path: format!("/disk-{disk_index}"),
|
||||
state: "ok".to_string(),
|
||||
total_space: 100,
|
||||
used_space: 100 - available_space,
|
||||
available_space,
|
||||
pool_index: 0,
|
||||
set_index: 0,
|
||||
disk_index: disk_index as i32,
|
||||
..Default::default()
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
let owned = HashMap::from([(0, 12)]);
|
||||
let capacity = decommission_physical_pool_capacity(
|
||||
&disks,
|
||||
0,
|
||||
DecommissionErasureLayout { data: 2, parity: 2 },
|
||||
PoolSpaceInfo {
|
||||
free: 0,
|
||||
total: 0,
|
||||
used: 0,
|
||||
},
|
||||
Some(&owned),
|
||||
);
|
||||
|
||||
assert_eq!(capacity, (400, 40, 12));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn decommission_terminal_transitions_release_capacity_reservations() {
|
||||
let now = OffsetDateTime::UNIX_EPOCH + Duration::hours(1);
|
||||
|
||||
@@ -1493,12 +1493,12 @@ impl Sets {
|
||||
object: &str,
|
||||
version_id: &str,
|
||||
opts: &HealOpts,
|
||||
retirement: Option<&crate::bucket::retirement::MarkerRetirementContext<'_>>,
|
||||
proof: crate::set_disk::AbsenceProofRequest<'_>,
|
||||
) -> Result<(HealResultItem, Option<Error>, Option<crate::set_disk::HealedObjectAbsence>)> {
|
||||
let mut absence = None;
|
||||
let (item, error) = self
|
||||
.get_disks_for_heal_object(object, opts)?
|
||||
.heal_object_with_retirement(bucket, object, version_id, opts, &mut absence, retirement)
|
||||
.heal_object_with_retirement(bucket, object, version_id, opts, &mut absence, proof)
|
||||
.await?;
|
||||
// A caller-owned lock does not expose its lease to this boundary.
|
||||
// Keep cleanup unverified when that lease cannot be checked here.
|
||||
|
||||
@@ -642,6 +642,21 @@ pub(crate) fn data_movement_stage_source(err: &Error) -> Option<&Error> {
|
||||
.downcast_ref::<Error>()
|
||||
}
|
||||
|
||||
/// Recover a concrete typed error wrapped by [`data_movement_stage_error`].
|
||||
pub(crate) fn data_movement_stage_source_as<T>(err: &Error) -> Option<&T>
|
||||
where
|
||||
T: std::error::Error + 'static,
|
||||
{
|
||||
let Error::Io(io_err) = err else {
|
||||
return None;
|
||||
};
|
||||
io_err
|
||||
.get_ref()?
|
||||
.downcast_ref::<DataMovementStageError>()?
|
||||
.source
|
||||
.downcast_ref::<T>()
|
||||
}
|
||||
|
||||
fn schedule_data_movement_multipart_abort_cleanup(
|
||||
store: Arc<ECStore>,
|
||||
target_pool_idx: usize,
|
||||
|
||||
@@ -12,6 +12,7 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
use crate::data_movement::data_movement_context_error;
|
||||
use crate::disk::RUSTFS_META_BUCKET;
|
||||
use crate::error::{Error, Result, is_err_object_not_found, is_err_version_not_found};
|
||||
use crate::object_api::ObjectOptions;
|
||||
@@ -27,6 +28,28 @@ use tokio::io::AsyncReadExt;
|
||||
pub const MAX_SCANNER_PAUSE_BACKLOG_BYTES: u64 = 64 * 1024;
|
||||
pub(crate) const SCANNER_PAUSE_BACKLOG_PATH: &str = "buckets/.scanner-pause-backlog.json";
|
||||
|
||||
/// Typed failure from the scanner-owned native retirement planner.
|
||||
///
|
||||
/// Decommission must preserve the source when the native authority cannot be
|
||||
/// proven. The handoff variant is deliberately narrower than other authority
|
||||
/// failures so the worker can retry without converting a transient membership
|
||||
/// transition into a terminal pool failure.
|
||||
#[derive(Debug, thiserror::Error)]
|
||||
pub enum ScannerPauseBacklogRetirementError {
|
||||
#[error("scanner pause backlog retirement has a retryable handoff conflict: {reason}")]
|
||||
HandoffConflict { reason: String },
|
||||
#[error("scanner pause backlog retirement has an unsafe authority conflict: {reason}")]
|
||||
AuthorityConflict { reason: String },
|
||||
#[error("scanner pause backlog retirement found an invalid native record: {reason}")]
|
||||
InvalidRecord { reason: String },
|
||||
}
|
||||
|
||||
impl ScannerPauseBacklogRetirementError {
|
||||
pub fn is_retryable_handoff_conflict(&self) -> bool {
|
||||
matches!(self, Self::HandoffConflict { .. })
|
||||
}
|
||||
}
|
||||
|
||||
/// A bounded, storage-fenced native replica. Only a confirmed missing object
|
||||
/// has no payload; read failures never enter the Scanner verifier.
|
||||
pub struct ScannerPauseBacklogRetirementReplica {
|
||||
@@ -44,7 +67,10 @@ pub struct ScannerPauseBacklogRetirementPlan {
|
||||
}
|
||||
|
||||
pub type ScannerPauseBacklogRetirementPlanner =
|
||||
fn(usize, &[ScannerPauseBacklogRetirementReplica]) -> std::result::Result<Option<ScannerPauseBacklogRetirementPlan>, String>;
|
||||
fn(
|
||||
usize,
|
||||
&[ScannerPauseBacklogRetirementReplica],
|
||||
) -> std::result::Result<Option<ScannerPauseBacklogRetirementPlan>, ScannerPauseBacklogRetirementError>;
|
||||
|
||||
static RETIREMENT_PLANNER: OnceLock<ScannerPauseBacklogRetirementPlanner> = OnceLock::new();
|
||||
|
||||
@@ -161,7 +187,8 @@ pub(crate) fn plan_scanner_pause_backlog_retirement(
|
||||
data: read.replica.data.clone(),
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
planner(source_pool_index, &snapshots).map_err(Error::other)
|
||||
planner(source_pool_index, &snapshots)
|
||||
.map_err(|err| data_movement_context_error(format!("scanner pause backlog retirement planner failed: {err}"), err))
|
||||
}
|
||||
|
||||
/// The native writer and retirement handoff use the same conditional, full-tail
|
||||
|
||||
@@ -30,6 +30,8 @@ use rustfs_filemeta::{FileInfo, ObjectPartInfo, RawFileInfo};
|
||||
use rustfs_madmin::{info_commands::DiskMetrics, metrics::TimedAction};
|
||||
#[cfg(not(test))]
|
||||
use std::sync::OnceLock;
|
||||
#[cfg(test)]
|
||||
use std::sync::atomic::AtomicBool;
|
||||
use std::{
|
||||
collections::HashMap,
|
||||
path::PathBuf,
|
||||
@@ -628,6 +630,8 @@ pub struct DiskHealthTracker {
|
||||
/// Authoritative atomically published runtime/status pair.
|
||||
state_snapshot: AtomicU64,
|
||||
transition_lock: std::sync::Mutex<()>,
|
||||
#[cfg(test)]
|
||||
test_forced_offline: AtomicBool,
|
||||
}
|
||||
|
||||
fn pack_health_state(runtime_state: RuntimeDriveHealthState, status: u32) -> u64 {
|
||||
@@ -975,6 +979,8 @@ impl DiskHealthTracker {
|
||||
last_capacity_probe_unix_secs: AtomicI64::new(0),
|
||||
state_snapshot: AtomicU64::new(pack_health_state(RuntimeDriveHealthState::Online, DISK_HEALTH_OK)),
|
||||
transition_lock: std::sync::Mutex::new(()),
|
||||
#[cfg(test)]
|
||||
test_forced_offline: AtomicBool::new(false),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -1045,6 +1051,13 @@ impl DiskHealthTracker {
|
||||
self.publish_state(state, status);
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub fn force_offline_for_test(&self) {
|
||||
self.test_forced_offline.store(true, Ordering::Release);
|
||||
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
|
||||
self.publish_state(RuntimeDriveHealthState::Offline, DISK_HEALTH_FAULTY);
|
||||
}
|
||||
|
||||
pub fn swap_ok_to_faulty(&self) -> bool {
|
||||
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
|
||||
let (_, status) = unpack_health_state(self.state_snapshot.load(Ordering::Acquire));
|
||||
@@ -1123,6 +1136,8 @@ impl DiskHealthTracker {
|
||||
/// Remote disks are marked faulty on timeout/network errors; the init loop retries with the
|
||||
/// same [`DiskStore`] handles, which would otherwise fail immediately at `is_faulty()`.
|
||||
pub fn reset_for_store_init_retry(&self, endpoint: &Endpoint) {
|
||||
#[cfg(test)]
|
||||
self.test_forced_offline.store(false, Ordering::Release);
|
||||
self.reset_for_store_init_retry_at(endpoint, current_unix_time());
|
||||
}
|
||||
|
||||
@@ -1142,6 +1157,10 @@ impl DiskHealthTracker {
|
||||
}
|
||||
|
||||
pub fn mark_recovery_success(&self, endpoint: &Endpoint, reason: &'static str) -> bool {
|
||||
#[cfg(test)]
|
||||
if self.test_forced_offline.load(Ordering::Acquire) {
|
||||
return false;
|
||||
}
|
||||
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
|
||||
let current = self.runtime_state();
|
||||
let next = match current {
|
||||
@@ -1440,6 +1459,11 @@ impl LocalDiskWrapper {
|
||||
self.health.force_runtime_state_for_test(state);
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub fn force_offline_for_test(&self) {
|
||||
self.health.force_offline_for_test();
|
||||
}
|
||||
|
||||
/// Same as [`DiskHealthTracker::reset_for_store_init_retry`]: undo a transient faulty mark before another format load attempt.
|
||||
pub fn reset_health_for_store_init_retry(&self) {
|
||||
self.health.reset_for_store_init_retry(&self.disk.endpoint());
|
||||
|
||||
@@ -244,13 +244,16 @@ impl StdError for ConditionalFileNotCommittedError {
|
||||
}
|
||||
}
|
||||
|
||||
fn classify_internode_missing_error(error: &InternodeHttpError) -> Option<DiskError> {
|
||||
fn classify_internode_disk_error(error: &InternodeHttpError) -> Option<DiskError> {
|
||||
if error.is_remote_file_not_found() {
|
||||
return Some(DiskError::FileNotFound);
|
||||
}
|
||||
if error.is_remote_volume_not_found() {
|
||||
return Some(DiskError::VolumeNotFound);
|
||||
}
|
||||
if error.is_remote_file_corrupt() {
|
||||
return Some(DiskError::FileCorrupt);
|
||||
}
|
||||
None
|
||||
}
|
||||
|
||||
@@ -526,13 +529,10 @@ fn io_error_chain_contains_kind(io_error: &std::io::Error, kind: std::io::ErrorK
|
||||
|
||||
impl From<std::io::Error> for DiskError {
|
||||
fn from(e: std::io::Error) -> Self {
|
||||
if let Some(error) = e.get_ref().and_then(|source| source.downcast_ref::<InternodeHttpError>()) {
|
||||
if error.is_remote_file_not_found() {
|
||||
return DiskError::FileNotFound;
|
||||
}
|
||||
if error.is_remote_volume_not_found() {
|
||||
return DiskError::VolumeNotFound;
|
||||
}
|
||||
if let Some(error) = e.get_ref().and_then(|source| source.downcast_ref::<InternodeHttpError>())
|
||||
&& let Some(classified) = classify_internode_disk_error(error)
|
||||
{
|
||||
return classified;
|
||||
}
|
||||
let e = match e.downcast::<TerminalReadError>() {
|
||||
Ok(terminal_error) => {
|
||||
@@ -541,7 +541,7 @@ impl From<std::io::Error> for DiskError {
|
||||
&& let Some(internode_error) = io_error
|
||||
.get_ref()
|
||||
.and_then(|source| source.downcast_ref::<InternodeHttpError>())
|
||||
&& let Some(classified) = classify_internode_missing_error(internode_error)
|
||||
&& let Some(classified) = classify_internode_disk_error(internode_error)
|
||||
{
|
||||
return classified;
|
||||
}
|
||||
@@ -963,6 +963,7 @@ mod tests {
|
||||
for (remote_error, expected) in [
|
||||
(rustfs_rio::new_test_remote_file_not_found_http_io_error(), DiskError::FileNotFound),
|
||||
(rustfs_rio::new_test_remote_volume_not_found_http_io_error(), DiskError::VolumeNotFound),
|
||||
(rustfs_rio::new_test_remote_file_corrupt_http_io_error(), DiskError::FileCorrupt),
|
||||
] {
|
||||
let wrapped = terminal_read_error_to_io(DiskError::Io(remote_error));
|
||||
assert_eq!(DiskError::from(wrapped), expected);
|
||||
@@ -1533,25 +1534,23 @@ mod tests {
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_internode_missing_errors_preserve_disk_error_types() {
|
||||
fn test_internode_disk_errors_preserve_disk_error_types() {
|
||||
let file_missing = DiskError::from(rustfs_rio::new_test_remote_file_not_found_http_io_error());
|
||||
let volume_missing = DiskError::from(rustfs_rio::new_test_remote_volume_not_found_http_io_error());
|
||||
let file_corrupt = DiskError::from(rustfs_rio::new_test_remote_file_corrupt_http_io_error());
|
||||
let unmarked_server_error = DiskError::from(rustfs_rio::new_test_internode_http_io_error(
|
||||
rustfs_rio::InternodeHttpErrorKind::HttpStatus(http::StatusCode::INTERNAL_SERVER_ERROR),
|
||||
));
|
||||
|
||||
assert_eq!(file_missing, DiskError::FileNotFound);
|
||||
assert_eq!(volume_missing, DiskError::VolumeNotFound);
|
||||
assert_eq!(file_corrupt, DiskError::FileCorrupt);
|
||||
assert!(matches!(unmarked_server_error, DiskError::Io(_)));
|
||||
for missing in [file_missing, volume_missing] {
|
||||
assert_eq!(missing.clone(), missing);
|
||||
for error in [file_missing, volume_missing, file_corrupt] {
|
||||
assert_eq!(error.clone(), error);
|
||||
assert_eq!(
|
||||
crate::disk::error_reduce::reduce_write_quorum_errs(
|
||||
&[Some(missing.clone()), Some(missing.clone()), None],
|
||||
&[],
|
||||
2
|
||||
),
|
||||
Some(missing)
|
||||
crate::disk::error_reduce::reduce_write_quorum_errs(&[Some(error.clone()), Some(error.clone()), None], &[], 2),
|
||||
Some(error)
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
+495
-121
@@ -94,6 +94,8 @@ const STALE_TMP_OBJECT_EXPIRY: Duration = Duration::from_secs(24 * 60 * 60);
|
||||
#[cfg(test)]
|
||||
tokio::task_local! {
|
||||
static DIRECTORY_LISTING_ENTRY_PROBE_COUNT: Arc<AtomicUsize>;
|
||||
/// `(sampled, complete)` directory reads issued by the delete-residue probe.
|
||||
static DELETE_RESIDUE_PROBE_READS: Arc<(AtomicUsize, AtomicUsize)>;
|
||||
}
|
||||
const RUSTFS_META_TMP_OLD_BUCKET: &str = ".rustfs.sys/tmp-old";
|
||||
const INLINE_METADATA_ROLLBACK_DIR_XOR: u128 = 0x7275737466735f696e6c696e655f7262;
|
||||
@@ -105,6 +107,21 @@ pub(crate) const RESERVED_DELETE_DATA_DIR_MARKER_PREFIX: &str = "reserve-delete-
|
||||
/// under-filled batch settles the common case without materializing large
|
||||
/// child sets; a full batch cannot prove no listable child hides behind it.
|
||||
const DELETE_RESIDUE_PROBE_LIMIT: i32 = 8;
|
||||
/// Most directory reads one delete-residue probe spends walking down through
|
||||
/// the ancestors of a deleted key before it gives up and lets the prefix
|
||||
/// surface. A genuine prefix settles within its depth (the first object's
|
||||
/// `xl.meta` ends the walk), so the budget only caps the cost of hiding a
|
||||
/// large residue tree, which then hides one level at a time instead.
|
||||
const DELETE_RESIDUE_PROBE_READ_BUDGET: usize = 32;
|
||||
|
||||
/// One directory read owed by the delete-residue probe.
|
||||
enum ResidueProbeStep {
|
||||
/// Read a bounded batch of `dir` and probe what it holds.
|
||||
Sample(String),
|
||||
/// Read all of `dir` once every child in `sampled` proved unlistable, and
|
||||
/// probe the rest.
|
||||
Remainder { dir: String, sampled: Vec<String> },
|
||||
}
|
||||
|
||||
/// A `part.N` file with a positive part number, the shape erasure data takes
|
||||
/// inside a version data dir.
|
||||
@@ -1882,6 +1899,61 @@ impl AsyncWrite for DirectWriter {
|
||||
}
|
||||
}
|
||||
|
||||
fn poll_write_vectored(
|
||||
self: std::pin::Pin<&mut Self>,
|
||||
cx: &mut std::task::Context<'_>,
|
||||
bufs: &[std::io::IoSlice<'_>],
|
||||
) -> std::task::Poll<std::io::Result<usize>> {
|
||||
// Bitrot writes arrive as [hash, data]. Coalesce both slices into the
|
||||
// aligned bounce buffer so the O_DIRECT path has the same byte-stream
|
||||
// contract as the buffered writer instead of relying on the trait's
|
||||
// first-slice fallback.
|
||||
let this = self.get_mut();
|
||||
loop {
|
||||
match &mut this.state {
|
||||
DirectWriteState::Busy(_) => {
|
||||
std::task::ready!(this.poll_drive_busy(cx))?;
|
||||
}
|
||||
DirectWriteState::Idle(inner_opt) => {
|
||||
let inner = inner_opt.as_mut().expect("idle direct writer must hold inner state");
|
||||
let capacity = inner.buf.len;
|
||||
let space = capacity - inner.filled;
|
||||
let mut remaining = space;
|
||||
let mut written = 0;
|
||||
for src in bufs {
|
||||
if remaining == 0 {
|
||||
break;
|
||||
}
|
||||
let take = src.len().min(remaining);
|
||||
let start = inner.filled + written;
|
||||
inner.buf.as_mut_slice()[start..start + take].copy_from_slice(&src[..take]);
|
||||
written += take;
|
||||
remaining -= take;
|
||||
}
|
||||
|
||||
if written == 0 {
|
||||
return std::task::Poll::Ready(Ok(0));
|
||||
}
|
||||
inner.filled += written;
|
||||
|
||||
if inner.filled == capacity {
|
||||
let mut inner = inner_opt.take().expect("idle direct writer must hold inner state");
|
||||
let handle = tokio::task::spawn_blocking(move || {
|
||||
let res = inner.flush_batch();
|
||||
(inner, res)
|
||||
});
|
||||
this.state = DirectWriteState::Busy(handle);
|
||||
}
|
||||
return std::task::Poll::Ready(Ok(written));
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn is_write_vectored(&self) -> bool {
|
||||
true
|
||||
}
|
||||
|
||||
fn poll_flush(self: std::pin::Pin<&mut Self>, cx: &mut std::task::Context<'_>) -> std::task::Poll<std::io::Result<()>> {
|
||||
// Only drive an in-flight batch to completion. Sub-alignment staged
|
||||
// bytes cannot be flushed mid-stream (they would misalign the next
|
||||
@@ -7539,7 +7611,7 @@ impl LocalDisk {
|
||||
objs_returned: &mut i32,
|
||||
skip_current_dir_object: bool,
|
||||
multipart_dir_to_skip: Option<HashSet<String>>,
|
||||
) -> Result<()>
|
||||
) -> Result<bool>
|
||||
where
|
||||
W: AsyncWrite + Unpin + Send,
|
||||
{
|
||||
@@ -7557,7 +7629,7 @@ impl LocalDisk {
|
||||
};
|
||||
|
||||
if opts.limit > 0 && *objs_returned >= opts.limit {
|
||||
return Ok(());
|
||||
return Ok(true);
|
||||
}
|
||||
|
||||
// TODO(backlog): add directory listing lock to prevent concurrent enumeration
|
||||
@@ -7618,12 +7690,12 @@ impl LocalDisk {
|
||||
return Err(DiskError::FileNotFound);
|
||||
}
|
||||
|
||||
return Ok(());
|
||||
return Ok(false);
|
||||
}
|
||||
};
|
||||
|
||||
if entries.is_empty() {
|
||||
return Ok(());
|
||||
return Ok(false);
|
||||
}
|
||||
|
||||
current = current.trim_matches('/').to_owned();
|
||||
@@ -7637,7 +7709,7 @@ impl LocalDisk {
|
||||
let entry = item.clone();
|
||||
// check limit
|
||||
if opts.limit > 0 && *objs_returned >= opts.limit {
|
||||
return Ok(());
|
||||
return Ok(true);
|
||||
}
|
||||
// check multipart dir
|
||||
if skip_current_dir_object
|
||||
@@ -7742,14 +7814,14 @@ impl LocalDisk {
|
||||
prefix = "".to_owned();
|
||||
|
||||
for entry in entries.iter() {
|
||||
if opts.limit > 0 && *objs_returned >= opts.limit {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
if entry.is_empty() {
|
||||
continue;
|
||||
}
|
||||
|
||||
if opts.limit > 0 && *objs_returned >= opts.limit {
|
||||
return Ok(true);
|
||||
}
|
||||
|
||||
let name = path_join_buf(&[current.as_str(), entry.as_str()]);
|
||||
|
||||
while let Some((last_name, _, _, _)) = dir_stack.last()
|
||||
@@ -7763,7 +7835,7 @@ impl LocalDisk {
|
||||
// moment the limit is reached, same as the check below this
|
||||
// loop guards against for the current entry itself.
|
||||
if opts.limit > 0 && *objs_returned >= opts.limit {
|
||||
return Ok(());
|
||||
return Ok(true);
|
||||
}
|
||||
|
||||
let (pop, skip_object, dir_to_skip, scan_required) = dir_stack.pop().expect("operation should succeed");
|
||||
@@ -7777,23 +7849,25 @@ impl LocalDisk {
|
||||
.await?;
|
||||
|
||||
let scan_path = pop.clone();
|
||||
if opts.recursive
|
||||
&& scan_required
|
||||
&& let Err(er) =
|
||||
Box::pin(self.scan_dir(pop, prefix.clone(), opts, out, objs_returned, skip_object, dir_to_skip)).await
|
||||
{
|
||||
if !er.is_metacache_output_stream_closed() {
|
||||
error!(
|
||||
event = EVENT_DISK_LOCAL_SCAN_FAILED,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
|
||||
path = %scan_path,
|
||||
operation = "scan_dir",
|
||||
error = ?er,
|
||||
"Disk local scan failed"
|
||||
);
|
||||
if opts.recursive && scan_required {
|
||||
match Box::pin(self.scan_dir(pop, prefix.clone(), opts, out, objs_returned, skip_object, dir_to_skip)).await {
|
||||
Ok(true) => return Ok(true),
|
||||
Ok(false) => {}
|
||||
Err(er) => {
|
||||
if !er.is_metacache_output_stream_closed() {
|
||||
error!(
|
||||
event = EVENT_DISK_LOCAL_SCAN_FAILED,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
|
||||
path = %scan_path,
|
||||
operation = "scan_dir",
|
||||
error = ?er,
|
||||
"Disk local scan failed"
|
||||
);
|
||||
}
|
||||
return Err(er);
|
||||
}
|
||||
}
|
||||
return Err(er);
|
||||
}
|
||||
}
|
||||
|
||||
@@ -7804,7 +7878,7 @@ impl LocalDisk {
|
||||
// tail, permanently skipping those keys on the next page instead
|
||||
// of just deferring them to it.
|
||||
if opts.limit > 0 && *objs_returned >= opts.limit {
|
||||
return Ok(());
|
||||
return Ok(true);
|
||||
}
|
||||
|
||||
let mut meta = MetaCacheEntry {
|
||||
@@ -7947,7 +8021,7 @@ impl LocalDisk {
|
||||
|
||||
while let Some((dir, skip_object, dir_to_skip, scan_required)) = dir_stack.pop() {
|
||||
if opts.limit > 0 && *objs_returned >= opts.limit {
|
||||
return Ok(());
|
||||
return Ok(true);
|
||||
}
|
||||
|
||||
write_metacache_obj(
|
||||
@@ -7960,27 +8034,29 @@ impl LocalDisk {
|
||||
.await?;
|
||||
|
||||
let scan_path = dir.clone();
|
||||
if opts.recursive
|
||||
&& scan_required
|
||||
&& let Err(er) =
|
||||
Box::pin(self.scan_dir(dir, prefix.clone(), opts, out, objs_returned, skip_object, dir_to_skip)).await
|
||||
{
|
||||
if !er.is_metacache_output_stream_closed() {
|
||||
error!(
|
||||
event = EVENT_DISK_LOCAL_SCAN_FAILED,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
|
||||
path = %scan_path,
|
||||
operation = "scan_dir",
|
||||
error = ?er,
|
||||
"Disk local recursive scan failed"
|
||||
);
|
||||
if opts.recursive && scan_required {
|
||||
match Box::pin(self.scan_dir(dir, prefix.clone(), opts, out, objs_returned, skip_object, dir_to_skip)).await {
|
||||
Ok(true) => return Ok(true),
|
||||
Ok(false) => {}
|
||||
Err(er) => {
|
||||
if !er.is_metacache_output_stream_closed() {
|
||||
error!(
|
||||
event = EVENT_DISK_LOCAL_SCAN_FAILED,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
|
||||
path = %scan_path,
|
||||
operation = "scan_dir",
|
||||
error = ?er,
|
||||
"Disk local recursive scan failed"
|
||||
);
|
||||
}
|
||||
return Err(er);
|
||||
}
|
||||
}
|
||||
return Err(er);
|
||||
}
|
||||
}
|
||||
|
||||
Ok(())
|
||||
Ok(false)
|
||||
}
|
||||
|
||||
/// Whether the backing directory of plain object `object_name` also holds
|
||||
@@ -8047,72 +8123,116 @@ impl LocalDisk {
|
||||
Ok(false)
|
||||
}
|
||||
|
||||
/// Whether the metadata-less directory `dir_name` holds nothing but the
|
||||
/// data dirs of deleted versions: it is itself a non-nil UUID directory of
|
||||
/// `part.N` files and delete-transaction markers, or every child is one.
|
||||
/// That is what an interrupted or deferred version delete leaves behind
|
||||
/// once the `xl.meta` is gone, and it must not surface as a prefix. Real
|
||||
/// object children are directories carrying their own `xl.meta`, so the
|
||||
/// first non-UUID child, stray file, or subdirectory inside a UUID child
|
||||
/// proves the directory is a genuine prefix. Reads are bounded: a
|
||||
/// Whether the metadata-less directory `dir_name` holds nothing that a
|
||||
/// listing could show: every leaf under it is the data dir of a deleted
|
||||
/// version (a non-nil UUID directory of `part.N` files and
|
||||
/// delete-transaction markers) or an empty directory. That is what an
|
||||
/// interrupted or deferred version delete leaves behind once the `xl.meta`
|
||||
/// is gone, and neither the object directory nor the date-style ancestors
|
||||
/// above it may surface as prefixes (#6898). Real objects are directories
|
||||
/// carrying their own `xl.meta`, so the first file met outside a UUID data
|
||||
/// dir, or any stray entry inside one, proves a genuine prefix and ends the
|
||||
/// walk at the depth of the first object.
|
||||
///
|
||||
/// Reads stay small on genuine prefixes: each directory is sampled with one
|
||||
/// bounded batch, the sampled children are probed depth-first, and the
|
||||
/// directory is only read in full once every sampled child proved to hold
|
||||
/// nothing listable. A total read budget makes a residue tree larger than
|
||||
/// the budget surface and be hidden one level down instead, so the cost on
|
||||
/// a genuine prefix never exceeds its depth in small directory reads. A
|
||||
/// directory that vanishes mid-probe holds nothing listable.
|
||||
async fn directory_is_delete_residue(&self, bucket: &str, dir_name: &str, stall: Option<Duration>) -> Result<bool> {
|
||||
let dir_name = dir_name.trim_end_matches(SLASH_SEPARATOR);
|
||||
let Some(entries) = self.read_dir_for_residue_probe(bucket, dir_name, stall).await? else {
|
||||
return Ok(false);
|
||||
};
|
||||
if entries.is_empty() {
|
||||
return Ok(false);
|
||||
}
|
||||
|
||||
let is_data_dir = dir_name
|
||||
.rsplit(SLASH_SEPARATOR)
|
||||
.next()
|
||||
.is_some_and(|name| Uuid::parse_str(name).is_ok_and(|uuid| !uuid.is_nil()));
|
||||
if is_data_dir && entries.iter().all(|entry| is_metadata_less_data_dir_entry(entry)) {
|
||||
return Ok(true);
|
||||
}
|
||||
|
||||
for entry in entries {
|
||||
let Some(child) = entry.strip_suffix(SLASH_SEPARATOR) else {
|
||||
return Ok(false);
|
||||
};
|
||||
if !Uuid::parse_str(child).is_ok_and(|uuid| !uuid.is_nil()) {
|
||||
let mut reads = 0usize;
|
||||
let mut pending = vec![ResidueProbeStep::Sample(dir_name.to_owned())];
|
||||
while let Some(step) = pending.pop() {
|
||||
if reads >= DELETE_RESIDUE_PROBE_READ_BUDGET {
|
||||
return Ok(false);
|
||||
}
|
||||
reads += 1;
|
||||
|
||||
let child_path = path_join_buf(&[dir_name, child]);
|
||||
let Some(child_entries) = self.read_dir_for_residue_probe(bucket, &child_path, stall).await? else {
|
||||
let (dir, entries, complete) = match step {
|
||||
ResidueProbeStep::Sample(dir) => {
|
||||
let Some((entries, complete)) = self.read_dir_for_residue_probe(bucket, &dir, stall, false).await? else {
|
||||
continue;
|
||||
};
|
||||
(dir, entries, complete)
|
||||
}
|
||||
ResidueProbeStep::Remainder { dir, sampled } => {
|
||||
let Some((entries, _)) = self.read_dir_for_residue_probe(bucket, &dir, stall, true).await? else {
|
||||
continue;
|
||||
};
|
||||
let entries = entries
|
||||
.into_iter()
|
||||
.filter(|entry| !sampled.contains(entry))
|
||||
.collect::<Vec<_>>();
|
||||
(dir, entries, true)
|
||||
}
|
||||
};
|
||||
|
||||
let is_data_dir = dir
|
||||
.rsplit(SLASH_SEPARATOR)
|
||||
.next()
|
||||
.is_some_and(|name| Uuid::parse_str(name).is_ok_and(|uuid| !uuid.is_nil()));
|
||||
if is_data_dir {
|
||||
if !entries.iter().all(|entry| is_metadata_less_data_dir_entry(entry)) {
|
||||
// A subdirectory, an `xl.meta`, or an unknown file inside a
|
||||
// UUID directory: not plain delete residue.
|
||||
return Ok(false);
|
||||
}
|
||||
if !complete {
|
||||
// Only the sampled part files were seen; the rest must be
|
||||
// read before the data dir counts as plain residue.
|
||||
pending.push(ResidueProbeStep::Remainder { dir, sampled: entries });
|
||||
}
|
||||
continue;
|
||||
};
|
||||
if !child_entries.iter().all(|entry| is_metadata_less_data_dir_entry(entry)) {
|
||||
return Ok(false);
|
||||
}
|
||||
|
||||
let mut children = Vec::with_capacity(entries.len());
|
||||
for entry in &entries {
|
||||
let Some(child) = entry.strip_suffix(SLASH_SEPARATOR) else {
|
||||
// A file outside a plain data dir: `xl.meta` or something
|
||||
// this probe does not understand. Either way, not residue.
|
||||
return Ok(false);
|
||||
};
|
||||
children.push(path_join_buf(&[&dir, child]));
|
||||
}
|
||||
if !complete {
|
||||
// Revisit the unsampled siblings only after every sampled
|
||||
// child, probed first, turned out to hold nothing listable.
|
||||
pending.push(ResidueProbeStep::Remainder { dir, sampled: entries });
|
||||
}
|
||||
pending.extend(children.into_iter().map(ResidueProbeStep::Sample));
|
||||
}
|
||||
|
||||
Ok(true)
|
||||
}
|
||||
|
||||
/// Read `dir` with a bounded batch first and a complete read only when the
|
||||
/// batch was full. `None` when the directory does not exist any more.
|
||||
async fn read_dir_for_residue_probe(&self, bucket: &str, dir: &str, stall: Option<Duration>) -> Result<Option<Vec<String>>> {
|
||||
for count in [DELETE_RESIDUE_PROBE_LIMIT, -1] {
|
||||
let entries = match with_walk_stall_timeout(stall, self.list_dir("", bucket, dir, count)).await {
|
||||
Ok(entries) => entries,
|
||||
Err(err) => {
|
||||
if err == DiskError::VolumeNotFound || err == Error::FileNotFound {
|
||||
return Ok(None);
|
||||
}
|
||||
|
||||
return Err(err);
|
||||
}
|
||||
};
|
||||
if count < 0 || entries.len() < count as usize {
|
||||
return Ok(Some(entries));
|
||||
/// Read `dir` for the residue probe: one bounded batch, or the complete
|
||||
/// directory when `complete` is set. The flag in the result says whether
|
||||
/// the batch held the whole directory. `None` when the directory does not
|
||||
/// exist any more.
|
||||
async fn read_dir_for_residue_probe(
|
||||
&self,
|
||||
bucket: &str,
|
||||
dir: &str,
|
||||
stall: Option<Duration>,
|
||||
complete: bool,
|
||||
) -> Result<Option<(Vec<String>, bool)>> {
|
||||
let count = if complete { -1 } else { DELETE_RESIDUE_PROBE_LIMIT };
|
||||
#[cfg(test)]
|
||||
let _ = DELETE_RESIDUE_PROBE_READS.try_with(|reads| {
|
||||
let counter = if complete { &reads.1 } else { &reads.0 };
|
||||
counter.fetch_add(1, Ordering::Relaxed)
|
||||
});
|
||||
match with_walk_stall_timeout(stall, self.list_dir("", bucket, dir, count)).await {
|
||||
Ok(entries) => {
|
||||
let complete = complete || entries.len() < DELETE_RESIDUE_PROBE_LIMIT as usize;
|
||||
Ok(Some((entries, complete)))
|
||||
}
|
||||
Err(err) if err == DiskError::VolumeNotFound || err == Error::FileNotFound => Ok(None),
|
||||
Err(err) => Err(err),
|
||||
}
|
||||
|
||||
Ok(None)
|
||||
}
|
||||
|
||||
/// Whether anything under `dir_name` would appear in a listing. With
|
||||
@@ -10060,21 +10180,25 @@ impl DiskAPI for LocalDisk {
|
||||
}
|
||||
}
|
||||
|
||||
self.scan_dir(
|
||||
opts.base_dir.clone(),
|
||||
opts.filter_prefix.clone().unwrap_or_default(),
|
||||
&opts,
|
||||
&mut out,
|
||||
&mut objs_returned,
|
||||
skip_current_dir_object,
|
||||
if multipart_dir_to_skip.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(multipart_dir_to_skip)
|
||||
},
|
||||
)
|
||||
.await?;
|
||||
let limit_reached = self
|
||||
.scan_dir(
|
||||
opts.base_dir.clone(),
|
||||
opts.filter_prefix.clone().unwrap_or_default(),
|
||||
&opts,
|
||||
&mut out,
|
||||
&mut objs_returned,
|
||||
skip_current_dir_object,
|
||||
if multipart_dir_to_skip.is_empty() {
|
||||
None
|
||||
} else {
|
||||
Some(multipart_dir_to_skip)
|
||||
},
|
||||
)
|
||||
.await?;
|
||||
|
||||
if let Some(flag) = opts.producer_limit_reached.as_ref() {
|
||||
flag.store(limit_reached, std::sync::atomic::Ordering::Release);
|
||||
}
|
||||
out.close().await?;
|
||||
Ok(())
|
||||
}
|
||||
@@ -18719,9 +18843,9 @@ mod test {
|
||||
let (fast_path_names, fast_path_probes) = scan_prefixes(&disk, bucket, true).await;
|
||||
|
||||
assert_eq!(conservative_names, expected_names);
|
||||
let mut expected_fast_path_names = expected_names.clone();
|
||||
expected_fast_path_names.push("stale/".to_owned());
|
||||
assert_eq!(fast_path_names, expected_fast_path_names);
|
||||
// The fast path hides the empty `stale/` chain too, at the cost of a
|
||||
// few bounded directory reads rather than the metadata probes.
|
||||
assert_eq!(fast_path_names, expected_names);
|
||||
let expected_probes = PREFIX_COUNT * 3 + 3;
|
||||
assert_eq!(conservative_probes, expected_probes);
|
||||
assert_eq!(fast_path_probes, 0);
|
||||
@@ -18832,9 +18956,9 @@ mod test {
|
||||
}
|
||||
|
||||
// Directories whose only content is a deleted version's data dir are
|
||||
// not prefixes; their ancestors stay ordinary directories until an
|
||||
// empty listing reclaims them.
|
||||
// not prefixes, and neither are their ancestors.
|
||||
assert_eq!(scan_names(&disk, bucket, "residue/2026/").await, Vec::<String>::new());
|
||||
assert_eq!(scan_names(&disk, bucket, "residue/").await, Vec::<String>::new());
|
||||
assert_eq!(scan_names(&disk, bucket, "committed/").await, Vec::<String>::new());
|
||||
|
||||
// UUID-named directories holding real objects, an object keyed by a
|
||||
@@ -18842,18 +18966,258 @@ mod test {
|
||||
assert_eq!(scan_names(&disk, bucket, "uploads/").await, vec![format!("uploads/{upload}/")]);
|
||||
assert_eq!(scan_names(&disk, bucket, "named/").await, vec![format!("named/{named}")]);
|
||||
assert_eq!(scan_names(&disk, bucket, "mixed/").await, vec!["mixed/child".to_owned()]);
|
||||
assert_eq!(
|
||||
scan_names(&disk, bucket, "").await,
|
||||
vec!["mixed/".to_owned(), "named/".to_owned(), "uploads/".to_owned()]
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_scan_dir_nonrecursive_fast_path_hides_delete_residue_ancestors() {
|
||||
use rustfs_filemeta::MetacacheReader;
|
||||
use tempfile::tempdir;
|
||||
|
||||
let dir = tempdir().expect("tempdir should be created");
|
||||
let bucket = "test-bucket";
|
||||
let bucket_dir = dir.path().join(bucket);
|
||||
|
||||
async fn write_object(object_dir: &Path, object_name: &str) {
|
||||
fs::create_dir_all(object_dir)
|
||||
.await
|
||||
.expect("object directory should be created");
|
||||
let mut metadata = FileMeta::default();
|
||||
let mut file_info = FileInfo::new(object_name, 1, 1);
|
||||
file_info.mod_time = Some(OffsetDateTime::now_utc());
|
||||
metadata.add_version(file_info).expect("metadata should be valid");
|
||||
fs::write(
|
||||
object_dir.join(STORAGE_FORMAT_FILE),
|
||||
metadata.marshal_msg().expect("metadata should encode"),
|
||||
)
|
||||
.await
|
||||
.expect("object metadata should be written");
|
||||
}
|
||||
|
||||
async fn write_residue(object_dir: &Path, committed: bool) {
|
||||
let residue = object_dir.join(Uuid::new_v4().to_string());
|
||||
fs::create_dir_all(&residue).await.expect("residue should be created");
|
||||
fs::write(residue.join("part.1"), b"stale")
|
||||
.await
|
||||
.expect("stale part should be written");
|
||||
if committed {
|
||||
fs::write(residue.join(format!("{DELETE_DATA_DIR_MARKER_PREFIX}{}", Uuid::new_v4())), [])
|
||||
.await
|
||||
.expect("delete marker should be written");
|
||||
}
|
||||
}
|
||||
|
||||
// The reported shape: a date-partitioned key deleted on an older build,
|
||||
// whose data dir survived without a delete-transaction marker. Every
|
||||
// ancestor up to `metrics/` holds nothing else.
|
||||
write_residue(&bucket_dir.join("metrics/kubelet/2026/08/28/23/74992556388248657933757.parquet"), false).await;
|
||||
// Several committed residues under one hour plus an empty sibling hour.
|
||||
write_residue(&bucket_dir.join("metrics/cpu/2026/08/28/22/a.parquet"), true).await;
|
||||
write_residue(&bucket_dir.join("metrics/cpu/2026/08/28/22/b.parquet"), true).await;
|
||||
fs::create_dir_all(bucket_dir.join("metrics/cpu/2026/08/28/21"))
|
||||
.await
|
||||
.expect("empty hour directory should be created");
|
||||
|
||||
// A live object deep under an otherwise identical tree keeps every
|
||||
// ancestor visible, even beside residue.
|
||||
write_residue(&bucket_dir.join("logs/default/2026/08/28/23/old.parquet"), false).await;
|
||||
write_object(
|
||||
&bucket_dir.join("logs/default/2026/08/28/23/live.parquet"),
|
||||
"logs/default/2026/08/28/23/live.parquet",
|
||||
)
|
||||
.await;
|
||||
|
||||
// A prefix with more residue directories than the probe budget stays
|
||||
// visible rather than costing an unbounded walk.
|
||||
for hour in 0..(DELETE_RESIDUE_PROBE_READ_BUDGET + 1) {
|
||||
write_residue(&bucket_dir.join(format!("bulk/2026/08/28/{hour:02}/a.parquet")), false).await;
|
||||
}
|
||||
// Exactly at the budget: `edge/` with N object dirs costs one sampled
|
||||
// read of `edge`, two reads per object dir (its own and its data dir),
|
||||
// and one remainder read of `edge` once its 8-entry sample was full.
|
||||
for object in 0..15 {
|
||||
write_residue(&bucket_dir.join(format!("edge-hide/{object:02}.parquet")), false).await;
|
||||
}
|
||||
for object in 0..16 {
|
||||
write_residue(&bucket_dir.join(format!("edge-show/{object:02}.parquet")), false).await;
|
||||
}
|
||||
|
||||
// A file that is not `xl.meta` outside a UUID data dir is not residue
|
||||
// either: the probe does not guess, the prefix surfaces.
|
||||
fs::create_dir_all(bucket_dir.join("stray/2026/08"))
|
||||
.await
|
||||
.expect("stray directory should be created");
|
||||
fs::write(bucket_dir.join("stray/2026/08/notes.txt"), b"?")
|
||||
.await
|
||||
.expect("stray file should be written");
|
||||
|
||||
// A UUID directory holding a subdirectory is not a data dir shape the
|
||||
// probe understands, so it surfaces even when the subdirectory is empty.
|
||||
let odd = Uuid::new_v4().to_string();
|
||||
fs::create_dir_all(bucket_dir.join("odd").join(&odd).join("sub"))
|
||||
.await
|
||||
.expect("odd directory should be created");
|
||||
|
||||
let endpoint =
|
||||
Endpoint::try_from(dir.path().to_str().expect("tempdir path should be UTF-8")).expect("endpoint should parse");
|
||||
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should initialize");
|
||||
|
||||
async fn scan_names(disk: &LocalDisk, bucket: &str, current: &str) -> Vec<String> {
|
||||
let (reader, mut writer) = tokio::io::duplex(64 * 1024);
|
||||
let mut output = MetacacheWriter::new(&mut writer);
|
||||
let opts = WalkDirOptions {
|
||||
bucket: bucket.to_string(),
|
||||
base_dir: current.to_string(),
|
||||
skip_hidden_prefix_check: true,
|
||||
..Default::default()
|
||||
};
|
||||
let mut objects_returned = 0;
|
||||
disk.scan_dir(
|
||||
current.to_string(),
|
||||
"".to_string(),
|
||||
&opts,
|
||||
&mut output,
|
||||
&mut objects_returned,
|
||||
false,
|
||||
None,
|
||||
)
|
||||
.await
|
||||
.expect("scan_dir should succeed");
|
||||
output.close().await.expect("metacache writer should close");
|
||||
drop(output);
|
||||
drop(writer);
|
||||
|
||||
let mut names = MetacacheReader::new(reader)
|
||||
.read_all()
|
||||
.await
|
||||
.expect("scan output should decode")
|
||||
.into_iter()
|
||||
.map(|entry| entry.name)
|
||||
.collect::<Vec<_>>();
|
||||
names.sort();
|
||||
names
|
||||
}
|
||||
|
||||
// Every level of a tree whose only leaves are deleted data dirs is
|
||||
// hidden, not just the object directory itself.
|
||||
assert_eq!(scan_names(&disk, bucket, "metrics/kubelet/2026/08/28/").await, Vec::<String>::new());
|
||||
assert_eq!(scan_names(&disk, bucket, "metrics/kubelet/").await, Vec::<String>::new());
|
||||
assert_eq!(scan_names(&disk, bucket, "metrics/cpu/2026/08/28/").await, Vec::<String>::new());
|
||||
assert_eq!(scan_names(&disk, bucket, "metrics/").await, Vec::<String>::new());
|
||||
assert_eq!(
|
||||
scan_names(&disk, bucket, "logs/default/2026/08/28/").await,
|
||||
vec!["logs/default/2026/08/28/23/".to_owned()]
|
||||
);
|
||||
assert_eq!(scan_names(&disk, bucket, "logs/").await, vec!["logs/default/".to_owned()]);
|
||||
assert_eq!(scan_names(&disk, bucket, "bulk/2026/08/").await, vec!["bulk/2026/08/28/".to_owned()]);
|
||||
// Listed directly, every object dir is probed on its own and hidden;
|
||||
// the budget only bites where the whole tree hangs off one entry, so
|
||||
// the root listing below shows `edge-show/` but not `edge-hide/`.
|
||||
assert_eq!(scan_names(&disk, bucket, "edge-hide/").await, Vec::<String>::new());
|
||||
assert_eq!(scan_names(&disk, bucket, "edge-show/").await, Vec::<String>::new());
|
||||
assert_eq!(scan_names(&disk, bucket, "stray/").await, vec!["stray/2026/".to_owned()]);
|
||||
assert_eq!(scan_names(&disk, bucket, "odd/").await, vec![format!("odd/{odd}/")]);
|
||||
assert_eq!(
|
||||
scan_names(&disk, bucket, "").await,
|
||||
vec![
|
||||
"committed/".to_owned(),
|
||||
"mixed/".to_owned(),
|
||||
"named/".to_owned(),
|
||||
"residue/".to_owned(),
|
||||
"uploads/".to_owned(),
|
||||
"bulk/".to_owned(),
|
||||
"edge-show/".to_owned(),
|
||||
"logs/".to_owned(),
|
||||
"odd/".to_owned(),
|
||||
"stray/".to_owned(),
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_scan_dir_nonrecursive_fast_path_residue_probe_reads_stay_within_prefix_depth() {
|
||||
use rustfs_filemeta::MetacacheReader;
|
||||
use tempfile::tempdir;
|
||||
|
||||
let dir = tempdir().expect("tempdir should be created");
|
||||
let bucket = "test-bucket";
|
||||
let bucket_dir = dir.path().join(bucket);
|
||||
|
||||
async fn write_object(object_dir: &Path, object_name: &str) {
|
||||
fs::create_dir_all(object_dir)
|
||||
.await
|
||||
.expect("object directory should be created");
|
||||
let mut metadata = FileMeta::default();
|
||||
let mut file_info = FileInfo::new(object_name, 1, 1);
|
||||
file_info.mod_time = Some(OffsetDateTime::now_utc());
|
||||
metadata.add_version(file_info).expect("metadata should be valid");
|
||||
fs::write(
|
||||
object_dir.join(STORAGE_FORMAT_FILE),
|
||||
metadata.marshal_msg().expect("metadata should encode"),
|
||||
)
|
||||
.await
|
||||
.expect("object metadata should be written");
|
||||
}
|
||||
|
||||
// A genuine date-partitioned prefix wider than the probe batch at two
|
||||
// levels: 20 days, each hour holding 100 objects.
|
||||
const DAYS: usize = 20;
|
||||
const OBJECTS: usize = 100;
|
||||
for day in 0..DAYS {
|
||||
for object in 0..OBJECTS {
|
||||
let key = format!("data/stream/2026/08/{day:02}/23/{object:03}.parquet");
|
||||
write_object(&bucket_dir.join(&key), &key).await;
|
||||
}
|
||||
}
|
||||
|
||||
let endpoint =
|
||||
Endpoint::try_from(dir.path().to_str().expect("tempdir path should be UTF-8")).expect("endpoint should parse");
|
||||
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should initialize");
|
||||
|
||||
let reads = Arc::new((AtomicUsize::new(0), AtomicUsize::new(0)));
|
||||
let (reader, mut writer) = tokio::io::duplex(64 * 1024);
|
||||
let mut output = MetacacheWriter::new(&mut writer);
|
||||
let opts = WalkDirOptions {
|
||||
bucket: bucket.to_string(),
|
||||
base_dir: "data/".to_string(),
|
||||
skip_hidden_prefix_check: true,
|
||||
..Default::default()
|
||||
};
|
||||
let mut objects_returned = 0;
|
||||
DELETE_RESIDUE_PROBE_READS
|
||||
.scope(
|
||||
Arc::clone(&reads),
|
||||
disk.scan_dir(
|
||||
"data/".to_string(),
|
||||
"".to_string(),
|
||||
&opts,
|
||||
&mut output,
|
||||
&mut objects_returned,
|
||||
false,
|
||||
None,
|
||||
),
|
||||
)
|
||||
.await
|
||||
.expect("scan_dir should succeed");
|
||||
output.close().await.expect("metacache writer should close");
|
||||
drop(output);
|
||||
drop(writer);
|
||||
|
||||
let names = MetacacheReader::new(reader)
|
||||
.read_all()
|
||||
.await
|
||||
.expect("scan output should decode")
|
||||
.into_iter()
|
||||
.map(|entry| entry.name)
|
||||
.collect::<Vec<_>>();
|
||||
assert_eq!(names, vec!["data/stream/".to_owned()]);
|
||||
|
||||
// stream, 2026, 08, one sampled day, its hour, one sampled object:
|
||||
// the walk ends at the first `xl.meta`, one bounded read per level,
|
||||
// without ever reading a wide directory in full.
|
||||
let (sampled, complete) = (reads.0.load(Ordering::Relaxed), reads.1.load(Ordering::Relaxed));
|
||||
assert_eq!(sampled, 6, "one sampled read per level down to the first object");
|
||||
assert_eq!(complete, 0, "a genuine prefix must never cost a complete directory read");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_scan_dir_nonrecursive_skips_dirs_with_only_hidden_delete_markers() {
|
||||
use rustfs_filemeta::MetacacheReader;
|
||||
@@ -21833,7 +22197,7 @@ mod test {
|
||||
#[cfg(target_os = "linux")]
|
||||
#[tokio::test]
|
||||
async fn direct_writer_state_machine_round_trips_over_plain_file() {
|
||||
use std::io::Read;
|
||||
use std::io::{IoSlice, Read};
|
||||
use tempfile::tempdir;
|
||||
|
||||
let dir = tempdir().expect("tempdir");
|
||||
@@ -21852,6 +22216,16 @@ mod test {
|
||||
|
||||
let mut writer = DirectWriter::from_std_file_for_test(file, align, capacity);
|
||||
let mut off = 0;
|
||||
if content.len() >= 2 {
|
||||
let split = content.len().min(300);
|
||||
let first = split / 2;
|
||||
let written = writer
|
||||
.write_vectored(&[IoSlice::new(&content[..first]), IoSlice::new(&content[first..split])])
|
||||
.await
|
||||
.expect("vectored write");
|
||||
assert_eq!(written, split, "vectored write must consume both hash/data-like slices");
|
||||
off = split;
|
||||
}
|
||||
while off < content.len() {
|
||||
let end = (off + 300).min(content.len());
|
||||
writer.write_all(&content[off..end]).await.expect("write_all");
|
||||
|
||||
@@ -70,7 +70,12 @@ use rustfs_filemeta::{FileInfo, ObjectPartInfo, RawFileInfo};
|
||||
use rustfs_madmin::info_commands::DiskMetrics;
|
||||
use rustfs_rio::ChunkReaderBox;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::{fmt::Debug, path::PathBuf, sync::Arc, time::Duration};
|
||||
use std::{
|
||||
fmt::Debug,
|
||||
path::PathBuf,
|
||||
sync::{Arc, atomic::AtomicBool},
|
||||
time::Duration,
|
||||
};
|
||||
use time::OffsetDateTime;
|
||||
use tokio::io::{AsyncRead, AsyncWrite};
|
||||
use uuid::Uuid;
|
||||
@@ -820,7 +825,7 @@ impl Disk {
|
||||
dst_volume: &str,
|
||||
dst_path: &str,
|
||||
) -> Result<RenameDataResp> {
|
||||
let Some(scope) = crate::store::bucket_heal_scope(dst_volume) else {
|
||||
let Some(scope) = crate::store::bucket_heal_scope_for_object(dst_volume, dst_path) else {
|
||||
return self
|
||||
.rename_data_borrowed_with_fence(src_volume, src_path, fi, dst_volume, dst_path, None)
|
||||
.await;
|
||||
@@ -979,6 +984,14 @@ impl Disk {
|
||||
Disk::Remote(remote_disk) => remote_disk.force_runtime_state_for_test(state),
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub fn force_offline_for_test(&self) {
|
||||
match self {
|
||||
Disk::Local(local_disk) => local_disk.force_offline_for_test(),
|
||||
Disk::Remote(remote_disk) => remote_disk.force_offline_for_test(),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
@@ -1471,6 +1484,12 @@ pub struct WalkDirOptions {
|
||||
// Override the remote stream stall timeout for long background walks.
|
||||
#[serde(default)]
|
||||
pub stall_timeout_ms: Option<u64>,
|
||||
|
||||
/// In-process completion state for bounded local walks. This is skipped
|
||||
/// from RPC serialization; remote peers retain the legacy natural-EOF
|
||||
/// behavior until they support an explicit capability.
|
||||
#[serde(skip)]
|
||||
pub producer_limit_reached: Option<Arc<AtomicBool>>,
|
||||
}
|
||||
|
||||
impl WalkDirOptions {
|
||||
@@ -1794,6 +1813,7 @@ mod tests {
|
||||
skip_total_timeout: false,
|
||||
timeout_ms: Some(10_000),
|
||||
stall_timeout_ms: Some(20_000),
|
||||
producer_limit_reached: None,
|
||||
};
|
||||
|
||||
assert_eq!(opts.bucket, "test-bucket");
|
||||
|
||||
@@ -2573,6 +2573,11 @@ impl Erasure {
|
||||
.with_deferred_parity_reopeners(deferred_reopeners);
|
||||
|
||||
let integrity = reader.integrity.clone();
|
||||
// Copy sources use demand-bound scheduling for backpressure, but retain
|
||||
// the ordinary GET read quorum. Only the opt-in GET rollout requires a
|
||||
// surplus source; available surplus shards and integrity proofs are
|
||||
// still verified before any reconstructed copy bytes are emitted.
|
||||
let require_surplus_source = reader.demand_bound_lockstep && matches!(decode_read_policy(), DecodeReadPolicy::Default);
|
||||
|
||||
let start = offset / self.block_size;
|
||||
let end = end_offset.saturating_sub(1) / self.block_size;
|
||||
@@ -2667,7 +2672,6 @@ impl Erasure {
|
||||
// `shards` are borrowed again below. In the `Stop` case that
|
||||
// drop is what cancels the still-in-flight prefetch read.
|
||||
let (flow, next): (Option<StripeFlow>, Option<StripeReadOutput>) = {
|
||||
let require_surplus_source = reader.demand_bound_lockstep;
|
||||
let read_fut = read_stripe_timed(&mut reader, stage_metrics_enabled);
|
||||
let emit_fut = self.emit_decoded_stripe(
|
||||
writer,
|
||||
@@ -2728,7 +2732,7 @@ impl Erasure {
|
||||
&mut written,
|
||||
&mut ret_err,
|
||||
stage_metrics_enabled,
|
||||
reader.demand_bound_lockstep,
|
||||
require_surplus_source,
|
||||
integrity.as_ref(),
|
||||
offset,
|
||||
)
|
||||
@@ -2771,7 +2775,7 @@ impl Erasure {
|
||||
&mut written,
|
||||
&mut ret_err,
|
||||
stage_metrics_enabled,
|
||||
reader.demand_bound_lockstep,
|
||||
require_surplus_source,
|
||||
integrity.as_ref(),
|
||||
offset,
|
||||
)
|
||||
@@ -3449,6 +3453,150 @@ mod tests {
|
||||
}
|
||||
}
|
||||
|
||||
async fn copy_source_bitrot_shards(erasure: &Erasure, payload: &[u8], corrupt_parity: bool) -> Vec<Vec<u8>> {
|
||||
let mut writers: Vec<_> = (0..erasure.data_shards + erasure.parity_shards)
|
||||
.map(|_| BitrotWriter::new(Cursor::new(Vec::new()), erasure.shard_size(), HashAlgorithm::HighwayHash256))
|
||||
.collect();
|
||||
for block in payload.chunks(erasure.block_size) {
|
||||
let shards = erasure.encode_data(block).expect("copy source should encode");
|
||||
for (index, shard) in shards.iter().enumerate() {
|
||||
let mut bytes = shard.to_vec();
|
||||
if corrupt_parity && index == erasure.data_shards {
|
||||
// A stale source can have a valid per-shard bitrot checksum.
|
||||
// Reframe the changed parity to exercise reconstruction verification.
|
||||
bytes[0] ^= 0x80;
|
||||
}
|
||||
writers[index].write(&bytes).await.expect("copy source should frame");
|
||||
}
|
||||
}
|
||||
writers.into_iter().map(|writer| writer.into_inner().into_inner()).collect()
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn copy_source_reads_exact_quorum_across_stripes_and_ranges() {
|
||||
const BLOCK_SIZE: usize = 768;
|
||||
let payload: Vec<_> = (0..BLOCK_SIZE * 3 + 37)
|
||||
.map(|index| (index.wrapping_mul(31) % 251) as u8)
|
||||
.collect();
|
||||
|
||||
for uses_legacy in [false, true] {
|
||||
let erasure = Erasure::new_with_options(12, 4, BLOCK_SIZE, uses_legacy);
|
||||
let buffers = copy_source_bitrot_shards(&erasure, &payload, false).await;
|
||||
for offline in [[0, 1, 2, 3], [0, 5, 12, 15]] {
|
||||
for (offset, length) in [(0, payload.len()), (BLOCK_SIZE + 17, payload.len() - BLOCK_SIZE - 17)] {
|
||||
let frame_offset = offset / BLOCK_SIZE * (erasure.shard_size() + HashAlgorithm::HighwayHash256.size());
|
||||
let positioned: Vec<_> = buffers.iter().map(|bytes| bytes[frame_offset..].to_vec()).collect();
|
||||
let (mut readers, handles) =
|
||||
readers_with_deferred_parity(&positioned, 12, erasure.shard_size(), &HashAlgorithm::HighwayHash256, &[]);
|
||||
for index in offline {
|
||||
readers[index] = None;
|
||||
}
|
||||
let mut output = Vec::new();
|
||||
let (written, error) = crate::set_disk::with_get_object_read_policy(
|
||||
crate::set_disk::GetObjectReadPolicy::CopySource,
|
||||
erasure.decode_with_stripe_handles(&mut output, readers, offset, length, payload.len(), None, handles),
|
||||
)
|
||||
.await;
|
||||
|
||||
assert!(error.is_none(), "legacy={uses_legacy}, offline={offline:?}, offset={offset}: {error:?}");
|
||||
assert_eq!(written, length);
|
||||
assert_eq!(output, payload[offset..offset + length]);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn copy_source_recovers_exact_quorum_after_first_stripe() {
|
||||
const BLOCK_SIZE: usize = 768;
|
||||
let payload: Vec<_> = (0..BLOCK_SIZE * 3 + 37)
|
||||
.map(|index| (index.wrapping_mul(31) % 251) as u8)
|
||||
.collect();
|
||||
for uses_legacy in [false, true] {
|
||||
let erasure = Erasure::new_with_options(12, 4, BLOCK_SIZE, uses_legacy);
|
||||
let buffers = copy_source_bitrot_shards(&erasure, &payload, false).await;
|
||||
let first_frame = erasure.shard_size() + HashAlgorithm::HighwayHash256.size();
|
||||
let (readers, handles) = readers_with_deferred_parity(
|
||||
&buffers,
|
||||
12,
|
||||
erasure.shard_size(),
|
||||
&HashAlgorithm::HighwayHash256,
|
||||
&[(0, first_frame), (1, first_frame), (2, first_frame), (3, first_frame)],
|
||||
);
|
||||
let mut output = Vec::new();
|
||||
let (written, error) = crate::set_disk::with_get_object_read_policy(
|
||||
crate::set_disk::GetObjectReadPolicy::CopySource,
|
||||
erasure.decode_with_stripe_handles(&mut output, readers, 0, payload.len(), payload.len(), None, handles),
|
||||
)
|
||||
.await;
|
||||
|
||||
assert!(error.is_none(), "legacy={uses_legacy}: late shard loss should recover: {error:?}");
|
||||
assert_eq!(written, payload.len());
|
||||
assert_eq!(output, payload);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn copy_source_below_read_quorum_fails_before_output() {
|
||||
const BLOCK_SIZE: usize = 768;
|
||||
let payload = vec![0x5a; BLOCK_SIZE * 2 + 37];
|
||||
for uses_legacy in [false, true] {
|
||||
let erasure = Erasure::new_with_options(12, 4, BLOCK_SIZE, uses_legacy);
|
||||
let buffers = copy_source_bitrot_shards(&erasure, &payload, false).await;
|
||||
let (mut readers, handles) =
|
||||
readers_with_deferred_parity(&buffers, 12, erasure.shard_size(), &HashAlgorithm::HighwayHash256, &[]);
|
||||
for reader in readers.iter_mut().take(5) {
|
||||
*reader = None;
|
||||
}
|
||||
let mut output = Vec::new();
|
||||
let (written, error) = crate::set_disk::with_get_object_read_policy(
|
||||
crate::set_disk::GetObjectReadPolicy::CopySource,
|
||||
erasure.decode_with_stripe_handles(&mut output, readers, 0, payload.len(), payload.len(), None, handles),
|
||||
)
|
||||
.await;
|
||||
|
||||
assert_eq!(written, 0);
|
||||
assert!(output.is_empty());
|
||||
assert_eq!(
|
||||
error
|
||||
.expect("eleven shards cannot reconstruct twelve data shards")
|
||||
.to_string(),
|
||||
Error::ErasureReadQuorum.to_string()
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn copy_source_rejects_corrupt_surplus_parity_before_output() {
|
||||
const BLOCK_SIZE: usize = 768;
|
||||
let payload = vec![0x5a; BLOCK_SIZE * 2 + 37];
|
||||
for uses_legacy in [false, true] {
|
||||
let erasure = Erasure::new_with_options(12, 4, BLOCK_SIZE, uses_legacy);
|
||||
let buffers = copy_source_bitrot_shards(&erasure, &payload, true).await;
|
||||
let (mut readers, handles) =
|
||||
readers_with_deferred_parity(&buffers, 12, erasure.shard_size(), &HashAlgorithm::HighwayHash256, &[]);
|
||||
for reader in readers.iter_mut().take(3) {
|
||||
*reader = None;
|
||||
}
|
||||
let mut output = Vec::new();
|
||||
let (written, error) = crate::set_disk::with_get_object_read_policy(
|
||||
crate::set_disk::GetObjectReadPolicy::CopySource,
|
||||
erasure.decode_with_stripe_handles(&mut output, readers, 0, payload.len(), payload.len(), None, handles),
|
||||
)
|
||||
.await;
|
||||
|
||||
assert_eq!(written, 0);
|
||||
assert!(output.is_empty());
|
||||
let error = error.expect("copy must reject inconsistent surplus parity despite valid bitrot framing");
|
||||
assert_eq!(error.kind(), ErrorKind::InvalidData);
|
||||
assert!(error.to_string().contains("inconsistent read source shards"));
|
||||
}
|
||||
}
|
||||
|
||||
/// Guards the shard-buffer reuse in `ParallelReader`: a multi-stripe
|
||||
/// `Erasure::decode` that reconstructs missing data shards on every stripe
|
||||
/// must still return byte-exact output. Reconstructed and parity buffers are
|
||||
|
||||
@@ -587,7 +587,9 @@ pub(crate) async fn verify_deep_parts(
|
||||
return Err(DiskError::FileCorrupt);
|
||||
}
|
||||
let part_status = statuses.get_mut(&part_index).ok_or(DiskError::FileCorrupt)?;
|
||||
let length = erasure.shard_file_offset(0, part.size, part.size);
|
||||
// Deep verification must consume every encoded stripe, including a
|
||||
// partial final stripe, before certifying the shard's integrity.
|
||||
let length = usize::try_from(erasure.shard_file_size(part.size as i64)).map_err(|_| DiskError::FileCorrupt)?;
|
||||
let mut readers = Vec::with_capacity(disks.len());
|
||||
for (index, disk) in disks.iter().enumerate() {
|
||||
if part_status[index] != CHECK_PART_UNKNOWN {
|
||||
@@ -631,6 +633,9 @@ pub(crate) async fn verify_deep_parts(
|
||||
part_status[index] = CHECK_PART_FILE_CORRUPT;
|
||||
}
|
||||
Err(error) => {
|
||||
if !matches!(error, DiskError::FileNotFound | DiskError::FileVersionNotFound | DiskError::FileCorrupt) {
|
||||
return Err(error);
|
||||
}
|
||||
readers.push(None);
|
||||
part_status[index] = conv_part_err_to_int(&Some(error));
|
||||
}
|
||||
|
||||
@@ -42,6 +42,7 @@ use std::sync::{
|
||||
atomic::{AtomicBool, AtomicUsize, Ordering},
|
||||
};
|
||||
use std::time::{Duration, Instant, SystemTime};
|
||||
use tokio::sync::Notify;
|
||||
use tokio::time::{sleep, timeout};
|
||||
use tokio_util::sync::CancellationToken;
|
||||
use tracing::{debug, error, info, warn};
|
||||
@@ -61,6 +62,14 @@ const TIER_CONFIG_RELOAD_RETRY_CAP: Duration = Duration::from_secs(5);
|
||||
const REMOTE_VERSION_STATE_PROBE_INTERVAL: Duration = Duration::from_secs(10);
|
||||
const REMOTE_VERSION_STATE_PROBE_TIMEOUT: Duration = Duration::from_secs(5);
|
||||
const REMOTE_VERSION_STATE_PROOF_TTL: Duration = Duration::from_secs(30);
|
||||
/// Fallback poll cadence while startup has not yet published the notification
|
||||
/// system. IAM finalization starts the probe before `init_notification_runtime`
|
||||
/// runs, so the first pass always fails closed; waiting a full probe interval
|
||||
/// there left a single node without its durable quota capability for ten
|
||||
/// seconds after `/health` reported ok (rustfs/rustfs#8014). Publication wakes
|
||||
/// the probe immediately through `NOTIFICATION_SYS_PUBLISHED`; this bound only
|
||||
/// covers a wakeup that races the availability check.
|
||||
const REMOTE_VERSION_STATE_PROBE_BOOTSTRAP_INTERVAL: Duration = Duration::from_millis(100);
|
||||
const CROSS_POOL_FENCE_SUPPORTED_VERSION: u32 = 2;
|
||||
const TIER_DELETE_JOURNAL_POLICY_SUPPORTED_VERSION: u32 = 3;
|
||||
const DECOMMISSION_TARGET_FENCE_POLICY_SUPPORTED_VERSION: u32 = 4;
|
||||
@@ -338,6 +347,9 @@ static LEGACY_TRANSITION_STATE_RECONCILE_FLEET_PROOF: OnceLock<std::sync::RwLock
|
||||
static ILM_RECOVERY_EXPORT_FLEET_PROOF: OnceLock<std::sync::RwLock<FleetCapabilityProofState>> = OnceLock::new();
|
||||
static TRANSITION_TRANSACTION_COMPACTION_FLEET_PROOF: OnceLock<std::sync::RwLock<FleetCapabilityProofState>> = OnceLock::new();
|
||||
static REMOTE_VERSION_STATE_PROBE_TOPOLOGY: OnceLock<String> = OnceLock::new();
|
||||
/// Signalled once `GLOBAL_NOTIFICATION_SYS` is published so the fleet probe can
|
||||
/// run its first real pass without waiting for the bootstrap poll.
|
||||
static NOTIFICATION_SYS_PUBLISHED: Notify = Notify::const_new();
|
||||
static ILM_RECOVERY_EXPORT_LOCAL_PROCESS_EPOCH: LazyLock<Uuid> = LazyLock::new(Uuid::new_v4);
|
||||
|
||||
fn cross_pool_fence_fleet_proof_slot() -> &'static std::sync::RwLock<FleetCapabilityProofState> {
|
||||
@@ -368,6 +380,24 @@ fn transition_transaction_compaction_fleet_proof_slot() -> &'static std::sync::R
|
||||
TRANSITION_TRANSACTION_COMPACTION_FLEET_PROOF.get_or_init(|| std::sync::RwLock::new(FleetCapabilityProofState::default()))
|
||||
}
|
||||
|
||||
fn all_fleet_capability_proof_slots() -> [&'static std::sync::RwLock<FleetCapabilityProofState>; 7] {
|
||||
[
|
||||
remote_version_state_fleet_proof_slot(),
|
||||
cross_pool_fence_fleet_proof_slot(),
|
||||
tier_delete_journal_fleet_proof_slot(),
|
||||
decommission_target_fence_fleet_proof_slot(),
|
||||
legacy_transition_state_reconcile_fleet_proof_slot(),
|
||||
ilm_recovery_export_fleet_proof_slot(),
|
||||
transition_transaction_compaction_fleet_proof_slot(),
|
||||
]
|
||||
}
|
||||
|
||||
fn revoke_all_fleet_capability_proofs() {
|
||||
for slot in all_fleet_capability_proof_slots() {
|
||||
revoke_fleet_capability_proof(slot);
|
||||
}
|
||||
}
|
||||
|
||||
fn revoke_fleet_capability_proof_state(state: &mut FleetCapabilityProofState) {
|
||||
if let Some(proof) = state.proof.take() {
|
||||
proof.generation.revoke();
|
||||
@@ -453,6 +483,20 @@ pub(crate) fn acquire_remote_version_state_fleet_proof() -> Option<RemoteVersion
|
||||
acquire_fleet_capability_proof_from(&state, expected_topology, Instant::now()).map(RemoteVersionStateFleetProofToken)
|
||||
}
|
||||
|
||||
pub(crate) fn acquire_remote_version_state_writer_fleet_proof() -> Option<RemoteVersionStateFleetProofToken> {
|
||||
let requested = rustfs_utils::get_env_bool(
|
||||
rustfs_config::ENV_TIER_REMOTE_VERSION_STATE_WRITE,
|
||||
rustfs_config::DEFAULT_TIER_REMOTE_VERSION_STATE_WRITE,
|
||||
);
|
||||
let fleet_confirmed = rustfs_utils::get_env_bool(
|
||||
rustfs_config::ENV_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED,
|
||||
rustfs_config::DEFAULT_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED,
|
||||
);
|
||||
(requested && fleet_confirmed)
|
||||
.then(acquire_remote_version_state_fleet_proof)
|
||||
.flatten()
|
||||
}
|
||||
|
||||
fn acquire_fleet_capability_proof_from(
|
||||
state: &FleetCapabilityProofState,
|
||||
expected_topology: &str,
|
||||
@@ -1212,15 +1256,7 @@ fn insert_remote_version_state_peer(peer_epochs: &mut BTreeMap<String, Uuid>, pe
|
||||
pub fn start_remote_version_state_fleet_probe(topology_fingerprint: String) {
|
||||
if REMOTE_VERSION_STATE_PROBE_TOPOLOGY.set(topology_fingerprint.clone()).is_err() {
|
||||
if REMOTE_VERSION_STATE_PROBE_TOPOLOGY.get() != Some(&topology_fingerprint) {
|
||||
for slot in [
|
||||
remote_version_state_fleet_proof_slot(),
|
||||
cross_pool_fence_fleet_proof_slot(),
|
||||
tier_delete_journal_fleet_proof_slot(),
|
||||
decommission_target_fence_fleet_proof_slot(),
|
||||
legacy_transition_state_reconcile_fleet_proof_slot(),
|
||||
ilm_recovery_export_fleet_proof_slot(),
|
||||
transition_transaction_compaction_fleet_proof_slot(),
|
||||
] {
|
||||
for slot in all_fleet_capability_proof_slots() {
|
||||
mark_fleet_capability_topology_conflict(slot);
|
||||
}
|
||||
}
|
||||
@@ -1228,53 +1264,59 @@ pub fn start_remote_version_state_fleet_probe(topology_fingerprint: String) {
|
||||
}
|
||||
|
||||
tokio::spawn(async move {
|
||||
let mut notification_sys_unavailable_logged = false;
|
||||
loop {
|
||||
let notification_sys = get_global_notification_sys();
|
||||
let remote_version_state_probe = async {
|
||||
match notification_sys.as_ref() {
|
||||
Some(notification_sys) => timeout(
|
||||
REMOTE_VERSION_STATE_PROBE_TIMEOUT,
|
||||
notification_sys.probe_remote_version_state_fleet(&topology_fingerprint),
|
||||
)
|
||||
.await
|
||||
.unwrap_or_else(|_| Err(Error::other("remote version state fleet capability probe timed out"))),
|
||||
None => Err(Error::other("remote version state fleet capability notification system is unavailable")),
|
||||
let Some(notification_sys) = get_global_notification_sys() else {
|
||||
// Startup publishes the notification system after this probe
|
||||
// is started. Stay failed closed, but retry on the bootstrap
|
||||
// cadence so a single node gains its capabilities as soon as
|
||||
// the system appears instead of one full probe interval later.
|
||||
revoke_all_fleet_capability_proofs();
|
||||
if !notification_sys_unavailable_logged {
|
||||
notification_sys_unavailable_logged = true;
|
||||
debug!(
|
||||
event = EVENT_NOTIFICATION_CAPABILITY_PROBE,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
subsystem = LOG_SUBSYSTEM_NOTIFICATION,
|
||||
state = "waiting_for_notification_system",
|
||||
"notification capability probe"
|
||||
);
|
||||
}
|
||||
let _ = timeout(REMOTE_VERSION_STATE_PROBE_BOOTSTRAP_INTERVAL, NOTIFICATION_SYS_PUBLISHED.notified()).await;
|
||||
continue;
|
||||
};
|
||||
notification_sys_unavailable_logged = false;
|
||||
let remote_version_state_probe = async {
|
||||
timeout(
|
||||
REMOTE_VERSION_STATE_PROBE_TIMEOUT,
|
||||
notification_sys.probe_remote_version_state_fleet(&topology_fingerprint),
|
||||
)
|
||||
.await
|
||||
.unwrap_or_else(|_| Err(Error::other("remote version state fleet capability probe timed out")))
|
||||
};
|
||||
let cross_pool_fence_probe = async {
|
||||
match notification_sys.as_ref() {
|
||||
Some(notification_sys) => timeout(
|
||||
REMOTE_VERSION_STATE_PROBE_TIMEOUT,
|
||||
notification_sys.probe_cross_pool_fence_fleet(&topology_fingerprint),
|
||||
)
|
||||
.await
|
||||
.unwrap_or_else(|_| Err(Error::other("cross-pool fence fleet capability probe timed out"))),
|
||||
None => Err(Error::other("cross-pool fence fleet capability notification system is unavailable")),
|
||||
}
|
||||
timeout(
|
||||
REMOTE_VERSION_STATE_PROBE_TIMEOUT,
|
||||
notification_sys.probe_cross_pool_fence_fleet(&topology_fingerprint),
|
||||
)
|
||||
.await
|
||||
.unwrap_or_else(|_| Err(Error::other("cross-pool fence fleet capability probe timed out")))
|
||||
};
|
||||
let recovery_export_probe = async {
|
||||
match notification_sys.as_ref() {
|
||||
Some(notification_sys) => timeout(
|
||||
REMOTE_VERSION_STATE_PROBE_TIMEOUT,
|
||||
notification_sys.probe_ilm_recovery_export_fleet(&topology_fingerprint),
|
||||
)
|
||||
.await
|
||||
.unwrap_or_else(|_| Err(Error::other("ILM recovery export fleet capability probe timed out"))),
|
||||
None => Err(Error::other("ILM recovery export fleet capability notification system is unavailable")),
|
||||
}
|
||||
timeout(
|
||||
REMOTE_VERSION_STATE_PROBE_TIMEOUT,
|
||||
notification_sys.probe_ilm_recovery_export_fleet(&topology_fingerprint),
|
||||
)
|
||||
.await
|
||||
.unwrap_or_else(|_| Err(Error::other("ILM recovery export fleet capability probe timed out")))
|
||||
};
|
||||
let transition_transaction_compaction_probe = async {
|
||||
match notification_sys.as_ref() {
|
||||
Some(notification_sys) => timeout(
|
||||
REMOTE_VERSION_STATE_PROBE_TIMEOUT,
|
||||
notification_sys.probe_transition_transaction_compaction_fleet(&topology_fingerprint),
|
||||
)
|
||||
.await
|
||||
.unwrap_or_else(|_| Err(Error::other("transition transaction compaction fleet capability probe timed out"))),
|
||||
None => Err(Error::other(
|
||||
"transition transaction compaction fleet capability notification system is unavailable",
|
||||
)),
|
||||
}
|
||||
timeout(
|
||||
REMOTE_VERSION_STATE_PROBE_TIMEOUT,
|
||||
notification_sys.probe_transition_transaction_compaction_fleet(&topology_fingerprint),
|
||||
)
|
||||
.await
|
||||
.unwrap_or_else(|_| Err(Error::other("transition transaction compaction fleet capability probe timed out")))
|
||||
};
|
||||
let (result, fence_probe, recovery_export_result, transition_transaction_compaction_result) = tokio::join!(
|
||||
remote_version_state_probe,
|
||||
@@ -1299,13 +1341,7 @@ pub fn start_remote_version_state_fleet_probe(topology_fingerprint: String) {
|
||||
.unwrap_or_else(std::sync::PoisonError::into_inner)
|
||||
.topology_conflict;
|
||||
if topology_conflict {
|
||||
revoke_fleet_capability_proof(remote_version_state_fleet_proof_slot());
|
||||
revoke_fleet_capability_proof(cross_pool_fence_fleet_proof_slot());
|
||||
revoke_fleet_capability_proof(tier_delete_journal_fleet_proof_slot());
|
||||
revoke_fleet_capability_proof(decommission_target_fence_fleet_proof_slot());
|
||||
revoke_fleet_capability_proof(legacy_transition_state_reconcile_fleet_proof_slot());
|
||||
revoke_fleet_capability_proof(ilm_recovery_export_fleet_proof_slot());
|
||||
revoke_fleet_capability_proof(transition_transaction_compaction_fleet_proof_slot());
|
||||
revoke_all_fleet_capability_proofs();
|
||||
} else if let Some(err) = publish_fleet_capability_probe_result(
|
||||
remote_version_state_fleet_proof_slot(),
|
||||
&topology_fingerprint,
|
||||
@@ -1428,9 +1464,11 @@ pub fn start_remote_version_state_fleet_probe(topology_fingerprint: String) {
|
||||
}
|
||||
|
||||
pub async fn new_global_notification_sys(eps: EndpointServerPools) -> Result<()> {
|
||||
let _ = GLOBAL_NOTIFICATION_SYS
|
||||
.set(Arc::new(NotificationSys::new(eps).await))
|
||||
.map_err(|_| Error::other("init notification_sys fail"));
|
||||
if GLOBAL_NOTIFICATION_SYS.set(Arc::new(NotificationSys::new(eps).await)).is_ok() {
|
||||
// `notify_one` stores a permit, so a probe that checks availability
|
||||
// just before this publication still wakes without losing the signal.
|
||||
NOTIFICATION_SYS_PUBLISHED.notify_one();
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -3905,6 +3943,49 @@ mod tests {
|
||||
stats
|
||||
}
|
||||
|
||||
/// rustfs/rustfs#8014: IAM finalization starts the fleet probe before the
|
||||
/// notification system is published. The first probe therefore fails
|
||||
/// closed, and a single node used to wait a full probe interval before the
|
||||
/// durable quota capability appeared, even though `/health` was already
|
||||
/// reporting ok. The probe must retry promptly while it waits for startup
|
||||
/// to publish the notification system.
|
||||
#[tokio::test]
|
||||
async fn fleet_probe_publishes_promptly_once_the_notification_system_appears() {
|
||||
if REMOTE_VERSION_STATE_PROBE_TOPOLOGY.get().is_some() || GLOBAL_NOTIFICATION_SYS.get().is_some() {
|
||||
// Process-wide state was bound by another test in this binary. The
|
||||
// startup ordering under test needs a fresh process; nextest
|
||||
// (the authoritative runner) always provides one.
|
||||
eprintln!("skipping: fleet probe globals already bound in this process");
|
||||
return;
|
||||
}
|
||||
|
||||
// Share the fingerprint every other proof-installing test in this crate
|
||||
// binds, so the plain `cargo test` fallback cannot see two topologies.
|
||||
start_remote_version_state_fleet_probe("object-transaction-fencing-test".to_string());
|
||||
sleep(Duration::from_millis(300)).await;
|
||||
assert!(
|
||||
acquire_cross_pool_fence_fleet_proof().is_none(),
|
||||
"no capability proof may exist before the notification system is published"
|
||||
);
|
||||
|
||||
let published_at = Instant::now();
|
||||
new_global_notification_sys(EndpointServerPools::default())
|
||||
.await
|
||||
.expect("single-node notification system initializes");
|
||||
|
||||
let observed = timeout(Duration::from_secs(2), async {
|
||||
while acquire_cross_pool_fence_fleet_proof().is_none() {
|
||||
sleep(Duration::from_millis(20)).await;
|
||||
}
|
||||
})
|
||||
.await;
|
||||
assert!(
|
||||
observed.is_ok(),
|
||||
"single-node cross-pool fence proof should publish within 2s of the notification system, waited {:?}",
|
||||
published_at.elapsed()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn merging_peer_rings_sums_each_tier_without_double_counting() {
|
||||
let mut cluster = DailyAllTierStats::from([("WARM".to_string(), ring(10))]);
|
||||
|
||||
@@ -9,11 +9,12 @@ use super::meta::{
|
||||
};
|
||||
use super::worker::{
|
||||
rebalance_max_attempts, rebalance_meta_lock_error, resolve_load_rebalance_stats_update_result,
|
||||
resolve_rebalance_meta_load_result, resolve_rebalance_meta_save_result, retry_rebalance_metadata_access,
|
||||
resolve_rebalance_deferred_last_error, resolve_rebalance_meta_load_result, resolve_rebalance_meta_save_result,
|
||||
retry_rebalance_metadata_access,
|
||||
};
|
||||
use super::{
|
||||
DiskStat, EVENT_REBALANCE_BUCKET, EVENT_REBALANCE_STATE, LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_REBALANCE, REBAL_META_NAME,
|
||||
RebalStatus, RebalanceInfo, RebalanceMeta, RebalanceStats, RebalanceStopPropagationRecord,
|
||||
RebalStatus, RebalanceDeferKind, RebalanceInfo, RebalanceMeta, RebalanceStats, RebalanceStopPropagationRecord,
|
||||
encode_rebalance_stop_propagation_record,
|
||||
};
|
||||
use crate::core::pools::{
|
||||
@@ -1103,6 +1104,7 @@ impl ECStore {
|
||||
bucket: String,
|
||||
last_error: String,
|
||||
expected_id: &str,
|
||||
kind: RebalanceDeferKind,
|
||||
) -> Result<()> {
|
||||
let mut rebalance_meta = self.rebalance_meta.write().await;
|
||||
ensure_rebalance_worker_active(rebalance_meta.as_ref(), expected_id, "defer rebalance bucket")?;
|
||||
@@ -1116,7 +1118,8 @@ impl ECStore {
|
||||
};
|
||||
|
||||
defer_bucket_in_rebalance_queue(pool_stat, &bucket)?;
|
||||
pool_stat.info.last_error = Some(last_error);
|
||||
let pending_entry_defer = pool_stat.info.last_error.clone();
|
||||
pool_stat.info.last_error = resolve_rebalance_deferred_last_error(kind, pending_entry_defer.as_deref(), &last_error);
|
||||
meta.last_refreshed_at = Some(OffsetDateTime::now_utc());
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -1209,7 +1209,7 @@ mod tests {
|
||||
cancel: &warning_cancel,
|
||||
};
|
||||
let warning_result = store
|
||||
.finish_rebalance_entry_after_cleanup(&warning_cleanup_context, async { Err(Error::SlowDown.into()) })
|
||||
.finish_rebalance_entry_after_cleanup(&warning_cleanup_context, async { Err(Error::FileAccessDenied.into()) })
|
||||
.await
|
||||
.expect("cleanup warnings should not fail the completed migration");
|
||||
assert!(matches!(warning_result, RebalanceEntryCleanupResult::Completed { warning: Some(_) }));
|
||||
@@ -1250,6 +1250,35 @@ mod tests {
|
||||
let pool_stats = &meta.as_ref().expect("rebalance metadata should exist").pool_stats[0];
|
||||
assert_eq!(pool_stats.bytes, 0, "deferred cleanup must not commit completion stats");
|
||||
assert_eq!(pool_stats.cleanup_warnings.count, 1, "deferred cleanup must not add a permanent warning");
|
||||
drop(meta);
|
||||
|
||||
let transient_guard = store
|
||||
.rebalance_run_guard(rebalance_id, "rebalance transient cleanup deferral test")
|
||||
.await
|
||||
.expect("rebalance transient cleanup deferral test guard should be acquired");
|
||||
let transient_cancel = CancellationToken::new();
|
||||
let transient_stats_updates = [&warning_version];
|
||||
let transient_cleanup_context = RebalanceEntryCleanupContext {
|
||||
run_guard: &transient_guard,
|
||||
pool_index: 0,
|
||||
bucket: "bucket",
|
||||
object: "object.bin",
|
||||
stats_updates: &transient_stats_updates,
|
||||
expected_id: rebalance_id,
|
||||
cancel: &transient_cancel,
|
||||
};
|
||||
let transient = store
|
||||
.finish_rebalance_entry_after_cleanup(&transient_cleanup_context, async { Err(Error::SlowDown.into()) })
|
||||
.await
|
||||
.expect("retryable cleanup failures should defer without failing the worker");
|
||||
assert!(matches!(transient, RebalanceEntryCleanupResult::Deferred { .. }));
|
||||
let meta = store.rebalance_meta.read().await;
|
||||
let pool_stats = &meta.as_ref().expect("rebalance metadata should exist").pool_stats[0];
|
||||
assert_eq!(pool_stats.bytes, 0, "retryable cleanup failures must not commit completion stats");
|
||||
assert_eq!(
|
||||
pool_stats.cleanup_warnings.count, 1,
|
||||
"retryable cleanup failures must not add a permanent warning"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
@@ -29,7 +29,7 @@ const REBAL_META_FMT: u16 = 1; // Replace with actual format value
|
||||
const REBAL_META_VER: u16 = 1; // Replace with actual version value
|
||||
pub(crate) const REBAL_META_NAME: &str = "rebalance.bin";
|
||||
const DEFAULT_REBALANCE_MAX_ATTEMPTS: usize = 3;
|
||||
pub(crate) const REBALANCE_SOURCE_CLEANUP_MAX_DEFERS: usize = 3;
|
||||
pub(crate) const REBALANCE_SOURCE_CLEANUP_MAX_DEFERS: usize = 8;
|
||||
const REBALANCE_MAX_ATTEMPTS_ENV: &str = "RUSTFS_REBALANCE_MAX_ATTEMPTS";
|
||||
const REBALANCE_STOP_PROPAGATION_ERROR_PREFIX: &str = "rebalance stop propagation incomplete: ";
|
||||
const REBALANCE_LISTING_RETRY_BASE_DELAY: Duration = Duration::from_millis(250);
|
||||
@@ -55,7 +55,7 @@ pub use types::{
|
||||
DiskStat, RebalSaveOpt, RebalStatus, RebalanceCleanupWarningEntry, RebalanceCleanupWarnings, RebalanceInfo, RebalanceMeta,
|
||||
RebalanceStats, RebalanceStopPropagationRecord,
|
||||
};
|
||||
use types::{RebalanceBucketConfigs, RebalanceBucketOutcome, RebalanceEntryOutcome};
|
||||
use types::{RebalanceBucketConfigs, RebalanceBucketOutcome, RebalanceDeferKind, RebalanceEntryOutcome};
|
||||
|
||||
#[cfg(any(test, feature = "test-util"))]
|
||||
pub async fn test_store_with_persisted_rebalance_meta(
|
||||
|
||||
@@ -34,27 +34,29 @@ use super::migration::{
|
||||
};
|
||||
use super::runtime::{
|
||||
RebalanceLocalActivationOutcome, commit_local_rebalance_worker_activation,
|
||||
commit_local_rebalance_worker_activation_candidate, should_fail_repeated_rebalance_bucket_defer,
|
||||
source_cleanup_defer_attempt, stage_local_rebalance_worker_activation,
|
||||
commit_local_rebalance_worker_activation_candidate, reached_rebalance_source_cleanup_defer_limit,
|
||||
should_fail_repeated_rebalance_bucket_defer, source_cleanup_defer_attempt, stage_local_rebalance_worker_activation,
|
||||
};
|
||||
use super::worker::{
|
||||
RebalanceEntryCleanupResult, ensure_rebalance_listing_disks_available, is_transient_rebalance_error,
|
||||
parse_rebalance_max_attempts, rebalance_listing_retry_delay, rebalance_migration_retry_delay,
|
||||
resolve_load_rebalance_stats_update_result, resolve_rebalance_bucket_error, resolve_rebalance_bucket_result,
|
||||
resolve_rebalance_entry_cleanup_delete_result, resolve_rebalance_file_info_versions_result,
|
||||
resolve_rebalance_meta_load_result, resolve_rebalance_meta_save_result, resolve_rebalance_migrate_result_error,
|
||||
resolve_rebalance_optional_bucket_config_result, resolve_rebalance_save_task_result, resolve_rebalance_stats_update_result,
|
||||
resolve_rebalance_terminal_error, resolve_rebalance_worker_result, run_rebalance_listing_with_retry,
|
||||
send_rebalance_done_signal, should_cleanup_rebalance_source_entry, should_count_rebalance_version_complete,
|
||||
should_defer_rebalance_entry_failure, should_retry_rebalance_listing, should_skip_rebalance_delete_marker,
|
||||
wait_rebalance_entry_tasks, wait_rebalance_listing_retry, with_rebalance_entry_context,
|
||||
resolve_rebalance_deferred_last_error, resolve_rebalance_entry_cleanup_delete_result,
|
||||
resolve_rebalance_file_info_versions_result, resolve_rebalance_meta_load_result, resolve_rebalance_meta_save_result,
|
||||
resolve_rebalance_migrate_result_error, resolve_rebalance_optional_bucket_config_result, resolve_rebalance_save_task_result,
|
||||
resolve_rebalance_stats_update_result, resolve_rebalance_terminal_error, resolve_rebalance_worker_result,
|
||||
run_rebalance_listing_with_retry, send_rebalance_done_signal, should_cleanup_rebalance_source_entry,
|
||||
should_count_rebalance_version_complete, should_defer_rebalance_entry_failure, should_retry_rebalance_listing,
|
||||
should_skip_rebalance_delete_marker, wait_rebalance_entry_tasks, wait_rebalance_listing_retry, with_rebalance_entry_context,
|
||||
};
|
||||
use super::{
|
||||
DiskStat, GetObjectReader, ObjectInfo, ObjectOptions, RebalSaveOpt, RebalStatus, RebalanceBucketConfigs,
|
||||
RebalanceBucketOutcome, RebalanceCleanupWarnings, RebalanceEntryOutcome, RebalanceInfo, RebalanceMeta, RebalanceStats,
|
||||
RebalanceStopPropagationRecord,
|
||||
RebalanceBucketOutcome, RebalanceCleanupWarnings, RebalanceDeferKind, RebalanceEntryOutcome, RebalanceInfo, RebalanceMeta,
|
||||
RebalanceStats, RebalanceStopPropagationRecord,
|
||||
};
|
||||
use super::{
|
||||
REBALANCE_DEFERRED_ENTRY_ERROR_PREFIX, REBALANCE_SOURCE_CLEANUP_DEFERRED_ERROR_PREFIX, REBALANCE_SOURCE_CLEANUP_MAX_DEFERS,
|
||||
};
|
||||
use super::{REBALANCE_DEFERRED_ENTRY_ERROR_PREFIX, REBALANCE_SOURCE_CLEANUP_DEFERRED_ERROR_PREFIX};
|
||||
use crate::bucket::replication::{ReplicationState, ReplicationStatusType, replication_state_to_filemeta};
|
||||
use crate::data_movement;
|
||||
use crate::data_movement::SourceCleanupError;
|
||||
@@ -1775,7 +1777,7 @@ fn test_resolve_rebalance_entry_cleanup_delete_result_ignores_not_found() {
|
||||
|
||||
#[test]
|
||||
fn test_resolve_rebalance_entry_cleanup_delete_result_returns_warning_for_failures() {
|
||||
let result = resolve_rebalance_entry_cleanup_delete_result(Err(Error::SlowDown.into()), "bucket-a", "obj.txt");
|
||||
let result = resolve_rebalance_entry_cleanup_delete_result(Err(Error::FileAccessDenied.into()), "bucket-a", "obj.txt");
|
||||
assert!(matches!(
|
||||
result,
|
||||
RebalanceEntryCleanupResult::Completed { warning: Some(ref message) }
|
||||
@@ -1783,6 +1785,122 @@ fn test_resolve_rebalance_entry_cleanup_delete_result_returns_warning_for_failur
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_resolve_rebalance_entry_cleanup_delete_result_defers_transient_failures() {
|
||||
let cases = [
|
||||
(Error::SlowDown, "slow down"),
|
||||
(
|
||||
Error::Lock(rustfs_lock::LockError::timeout("bucket-a/obj.txt@latest", Duration::from_secs(5))),
|
||||
"object lock timeout",
|
||||
),
|
||||
(
|
||||
Error::Lock(rustfs_lock::LockError::network(
|
||||
"peer unavailable",
|
||||
std::io::Error::from(std::io::ErrorKind::ConnectionReset),
|
||||
)),
|
||||
"object lock network failure",
|
||||
),
|
||||
(Error::ErasureWriteQuorum, "write quorum"),
|
||||
(Error::Io(std::io::Error::from(std::io::ErrorKind::TimedOut)), "io timeout"),
|
||||
(
|
||||
Error::other("Lock error: Lock acquisition timeout for resource 'bucket-a/obj.txt@latest' after 5s"),
|
||||
"rendered lock timeout text",
|
||||
),
|
||||
];
|
||||
|
||||
for (err, label) in cases {
|
||||
match resolve_rebalance_entry_cleanup_delete_result(Err(err.into()), "bucket-a", "obj.txt") {
|
||||
RebalanceEntryCleanupResult::Deferred { last_error } => {
|
||||
assert!(
|
||||
last_error.starts_with(REBALANCE_SOURCE_CLEANUP_DEFERRED_ERROR_PREFIX),
|
||||
"{label}: {last_error}"
|
||||
);
|
||||
assert!(last_error.contains("bucket-a/obj.txt"), "{label}: {last_error}");
|
||||
}
|
||||
RebalanceEntryCleanupResult::Completed { warning } => {
|
||||
panic!("{label} must defer source cleanup instead of completing the entry with warning {warning:?}")
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_resolve_rebalance_entry_cleanup_delete_result_defers_stage_wrapped_lock_timeout() {
|
||||
let err = data_movement::data_movement_stage_error_for_test(
|
||||
"rebalance",
|
||||
"delete_object",
|
||||
"bucket-a",
|
||||
"obj.txt",
|
||||
Error::Lock(rustfs_lock::LockError::timeout("bucket-a/obj.txt@latest", Duration::from_secs(5))),
|
||||
);
|
||||
|
||||
assert!(matches!(
|
||||
resolve_rebalance_entry_cleanup_delete_result(Err(err.into()), "bucket-a", "obj.txt"),
|
||||
RebalanceEntryCleanupResult::Deferred { .. }
|
||||
));
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_resolve_rebalance_entry_cleanup_delete_result_ignores_stage_wrapped_not_found() {
|
||||
let err = data_movement::data_movement_stage_error_for_test(
|
||||
"rebalance",
|
||||
"delete_object",
|
||||
"bucket-a",
|
||||
"obj.txt",
|
||||
Error::ObjectNotFound("bucket-a".to_string(), "obj.txt".to_string()),
|
||||
);
|
||||
|
||||
assert_eq!(
|
||||
resolve_rebalance_entry_cleanup_delete_result(Err(err.into()), "bucket-a", "obj.txt"),
|
||||
RebalanceEntryCleanupResult::Completed { warning: None }
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_resolve_rebalance_deferred_last_error_hides_retryable_cleanup_conflicts() {
|
||||
let entry_error = format!("{REBALANCE_DEFERRED_ENTRY_ERROR_PREFIX} timeout");
|
||||
assert_eq!(
|
||||
resolve_rebalance_deferred_last_error(RebalanceDeferKind::Entry, None, entry_error.as_str()),
|
||||
Some(entry_error.clone()),
|
||||
"transient migration deferrals must stay visible to the completion guards"
|
||||
);
|
||||
|
||||
let cleanup_error = format!("{REBALANCE_SOURCE_CLEANUP_DEFERRED_ERROR_PREFIX} lock acquisition timeout");
|
||||
assert_eq!(
|
||||
resolve_rebalance_deferred_last_error(RebalanceDeferKind::SourceCleanup, None, cleanup_error.as_str()),
|
||||
None,
|
||||
"a retryable source cleanup conflict is progress, not a pool failure"
|
||||
);
|
||||
|
||||
assert_eq!(
|
||||
resolve_rebalance_deferred_last_error(
|
||||
RebalanceDeferKind::SourceCleanup,
|
||||
Some(entry_error.as_str()),
|
||||
cleanup_error.as_str()
|
||||
),
|
||||
Some(entry_error.clone()),
|
||||
"a cleanup deferral for one bucket must not erase an unresolved migration deferral of the same pool"
|
||||
);
|
||||
assert_eq!(
|
||||
resolve_rebalance_deferred_last_error(
|
||||
RebalanceDeferKind::SourceCleanup,
|
||||
Some(cleanup_error.as_str()),
|
||||
cleanup_error.as_str()
|
||||
),
|
||||
None,
|
||||
"a stale retryable cleanup message must not survive as a pool failure"
|
||||
);
|
||||
assert_eq!(
|
||||
resolve_rebalance_deferred_last_error(
|
||||
RebalanceDeferKind::SourceCleanup,
|
||||
Some(entry_error.as_str()),
|
||||
entry_error.as_str()
|
||||
),
|
||||
Some(entry_error),
|
||||
"repeated cleanup deferrals must keep the pending migration deferral visible"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_resolve_rebalance_entry_cleanup_delete_result_defers_source_change() {
|
||||
let result = resolve_rebalance_entry_cleanup_delete_result(Err(SourceCleanupError::SourceChanged), "bucket-a", "obj.txt");
|
||||
@@ -1817,6 +1935,89 @@ fn test_source_cleanup_defer_does_not_fail_repeated_bucket_retry() {
|
||||
assert_eq!(source_cleanup_defer_attempt(&mut source_attempts, "bucket-c"), 1);
|
||||
assert_eq!(source_cleanup_defer_attempt(&mut source_attempts, "bucket-c"), 2);
|
||||
assert_eq!(source_cleanup_defer_attempt(&mut source_attempts, "bucket-c"), 3);
|
||||
|
||||
let mut bounded_attempts = std::collections::HashMap::new();
|
||||
for expected in 1..REBALANCE_SOURCE_CLEANUP_MAX_DEFERS {
|
||||
assert_eq!(source_cleanup_defer_attempt(&mut bounded_attempts, "bucket-d"), expected);
|
||||
assert!(
|
||||
!reached_rebalance_source_cleanup_defer_limit(expected),
|
||||
"a retryable cleanup conflict must stay retryable at deferral {expected}"
|
||||
);
|
||||
}
|
||||
assert_eq!(
|
||||
source_cleanup_defer_attempt(&mut bounded_attempts, "bucket-d"),
|
||||
REBALANCE_SOURCE_CLEANUP_MAX_DEFERS
|
||||
);
|
||||
assert!(
|
||||
reached_rebalance_source_cleanup_defer_limit(REBALANCE_SOURCE_CLEANUP_MAX_DEFERS),
|
||||
"an unreclaimable source replica must fail the bucket after the bounded deferral budget"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_defer_rebalance_bucket_keeps_pending_entry_defer_without_surfacing_cleanup_conflicts() {
|
||||
let id = "rebalance-defer-last-error";
|
||||
let meta = RebalanceMeta {
|
||||
id: id.to_string(),
|
||||
pool_stats: vec![RebalanceStats {
|
||||
participating: true,
|
||||
buckets: vec!["bucket-a".to_string(), "bucket-b".to_string()],
|
||||
info: RebalanceInfo {
|
||||
status: RebalStatus::Started,
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
}],
|
||||
..Default::default()
|
||||
};
|
||||
let store = test_store_with_rebalance_meta(meta);
|
||||
let cleanup_error = format!("{REBALANCE_SOURCE_CLEANUP_DEFERRED_ERROR_PREFIX} lock acquisition timeout");
|
||||
|
||||
store
|
||||
.defer_rebalance_bucket(0, "bucket-a".to_string(), cleanup_error.clone(), id, RebalanceDeferKind::SourceCleanup)
|
||||
.await
|
||||
.expect("a retryable cleanup conflict must be deferrable");
|
||||
{
|
||||
let meta = store.rebalance_meta.read().await;
|
||||
let pool_stat = &meta.as_ref().expect("rebalance metadata should exist").pool_stats[0];
|
||||
assert_eq!(
|
||||
pool_stat.info.last_error, None,
|
||||
"a retryable cleanup conflict is progress and must not surface as a pool failure"
|
||||
);
|
||||
assert_eq!(pool_stat.buckets, vec!["bucket-b".to_string(), "bucket-a".to_string()]);
|
||||
}
|
||||
|
||||
let entry_error = format!("{REBALANCE_DEFERRED_ENTRY_ERROR_PREFIX} slow down");
|
||||
{
|
||||
let mut meta = store.rebalance_meta.write().await;
|
||||
meta.as_mut().expect("rebalance metadata should exist").pool_stats[0]
|
||||
.info
|
||||
.last_error = Some(entry_error.clone());
|
||||
}
|
||||
store
|
||||
.defer_rebalance_bucket(0, "bucket-b".to_string(), cleanup_error, id, RebalanceDeferKind::SourceCleanup)
|
||||
.await
|
||||
.expect("a cleanup deferral must be accepted while another bucket still defers an entry");
|
||||
{
|
||||
let meta = store.rebalance_meta.read().await;
|
||||
let pool_stat = &meta.as_ref().expect("rebalance metadata should exist").pool_stats[0];
|
||||
assert_eq!(
|
||||
pool_stat.info.last_error,
|
||||
Some(entry_error),
|
||||
"a cleanup deferral must not erase the pending migration deferral that blocks goal completion"
|
||||
);
|
||||
assert!(has_deferred_rebalance_error(pool_stat));
|
||||
assert_eq!(pool_stat.buckets, vec!["bucket-a".to_string(), "bucket-b".to_string()]);
|
||||
}
|
||||
|
||||
let migration_error = format!("{REBALANCE_DEFERRED_ENTRY_ERROR_PREFIX} i/o timeout");
|
||||
store
|
||||
.defer_rebalance_bucket(0, "bucket-a".to_string(), migration_error.clone(), id, RebalanceDeferKind::Entry)
|
||||
.await
|
||||
.expect("migration deferrals must keep their last error");
|
||||
let meta = store.rebalance_meta.read().await;
|
||||
let pool_stat = &meta.as_ref().expect("rebalance metadata should exist").pool_stats[0];
|
||||
assert_eq!(pool_stat.info.last_error, Some(migration_error));
|
||||
}
|
||||
|
||||
#[test]
|
||||
|
||||
@@ -7,13 +7,14 @@ use super::meta::{
|
||||
validate_start_rebalance_state,
|
||||
};
|
||||
use super::worker::{
|
||||
resolve_rebalance_bucket_result, resolve_rebalance_meta_save_result, resolve_rebalance_save_task_result,
|
||||
resolve_rebalance_terminal_error, send_rebalance_done_signal,
|
||||
rebalance_max_attempts, resolve_rebalance_bucket_result, resolve_rebalance_meta_save_result,
|
||||
resolve_rebalance_save_task_result, resolve_rebalance_terminal_error, retry_rebalance_metadata_access,
|
||||
send_rebalance_done_signal,
|
||||
};
|
||||
use super::{
|
||||
EVENT_REBALANCE_BUCKET, EVENT_REBALANCE_STATE, LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_REBALANCE,
|
||||
REBALANCE_LISTING_RETRY_BASE_DELAY, REBALANCE_SOURCE_CLEANUP_DEFERRED_ERROR_PREFIX, RebalSaveOpt, RebalStatus,
|
||||
RebalanceBucketOutcome,
|
||||
REBALANCE_LISTING_RETRY_BASE_DELAY, REBALANCE_SOURCE_CLEANUP_DEFERRED_ERROR_PREFIX, REBALANCE_SOURCE_CLEANUP_MAX_DEFERS,
|
||||
RebalSaveOpt, RebalStatus, RebalanceBucketOutcome, RebalanceDeferKind,
|
||||
};
|
||||
use crate::error::{Error, Result};
|
||||
use crate::runtime::sources as runtime_sources;
|
||||
@@ -40,6 +41,12 @@ pub(super) fn source_cleanup_defer_attempt(deferred_attempts: &mut HashMap<Strin
|
||||
*attempts
|
||||
}
|
||||
|
||||
/// Retryable source cleanup conflicts are bounded per run, so a source replica that stays
|
||||
/// unreclaimable fails the bucket explicitly instead of deferring forever.
|
||||
pub(super) fn reached_rebalance_source_cleanup_defer_limit(attempt: usize) -> bool {
|
||||
attempt >= REBALANCE_SOURCE_CLEANUP_MAX_DEFERS
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
|
||||
pub(super) enum RebalanceLocalActivationOutcome {
|
||||
Started,
|
||||
@@ -443,13 +450,14 @@ impl ECStore {
|
||||
};
|
||||
|
||||
if terminal_state_present {
|
||||
if let Err(err) = store
|
||||
.save_rebalance_stats_inner(
|
||||
if let Err(err) = retry_rebalance_metadata_access(None, rebalance_max_attempts(), || {
|
||||
store.save_rebalance_stats_inner(
|
||||
pool_index,
|
||||
RebalSaveOpt::Stats,
|
||||
Some(save_rebalance_id.as_ref()),
|
||||
)
|
||||
.await
|
||||
})
|
||||
.await
|
||||
{
|
||||
let mut rebalance_meta = store.rebalance_meta.write().await;
|
||||
*rebalance_meta = previous_meta;
|
||||
@@ -469,9 +477,10 @@ impl ECStore {
|
||||
}
|
||||
|
||||
if !terminal_state_saved
|
||||
&& let Err(err) = store
|
||||
.save_rebalance_stats_for_id(pool_index, RebalSaveOpt::Stats, save_rebalance_id.as_ref())
|
||||
.await
|
||||
&& let Err(err) = retry_rebalance_metadata_access(None, rebalance_max_attempts(), || {
|
||||
store.save_rebalance_stats_for_id(pool_index, RebalSaveOpt::Stats, save_rebalance_id.as_ref())
|
||||
})
|
||||
.await
|
||||
{
|
||||
let wrapped = Error::other(format!("rebalance save_task stats save failed for pool {pool_index}: {err}"));
|
||||
error!("{} err: {:?}", msg, wrapped);
|
||||
@@ -623,6 +632,11 @@ impl ECStore {
|
||||
} else {
|
||||
0
|
||||
};
|
||||
let defer_kind = if source_cleanup_deferred {
|
||||
RebalanceDeferKind::SourceCleanup
|
||||
} else {
|
||||
RebalanceDeferKind::Entry
|
||||
};
|
||||
warn!(
|
||||
event = EVENT_REBALANCE_BUCKET,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
@@ -634,7 +648,7 @@ impl ECStore {
|
||||
"Deferred rebalance bucket after transient object failures"
|
||||
);
|
||||
if let Err(err) = self
|
||||
.defer_rebalance_bucket(pool_index, bucket.clone(), last_error.clone(), rebalance_id.as_ref())
|
||||
.defer_rebalance_bucket(pool_index, bucket.clone(), last_error.clone(), rebalance_id.as_ref(), defer_kind)
|
||||
.await
|
||||
{
|
||||
error!(
|
||||
@@ -654,10 +668,10 @@ impl ECStore {
|
||||
break;
|
||||
}
|
||||
if source_cleanup_deferred {
|
||||
if source_cleanup_attempt >= super::REBALANCE_SOURCE_CLEANUP_MAX_DEFERS {
|
||||
if reached_rebalance_source_cleanup_defer_limit(source_cleanup_attempt) {
|
||||
let err = Error::other(format!(
|
||||
"rebalance bucket {bucket} source cleanup remained unstable after {} deferrals: {last_error}",
|
||||
super::REBALANCE_SOURCE_CLEANUP_MAX_DEFERS
|
||||
REBALANCE_SOURCE_CLEANUP_MAX_DEFERS
|
||||
));
|
||||
warn!(
|
||||
event = EVENT_REBALANCE_BUCKET,
|
||||
|
||||
@@ -51,6 +51,18 @@ pub(super) enum RebalanceEntryOutcome {
|
||||
Deferred { last_error: String },
|
||||
}
|
||||
|
||||
/// Why a rebalance bucket was put back at the end of the queue.
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
|
||||
pub(super) enum RebalanceDeferKind {
|
||||
/// A transient object-migration failure. Persisting it as `lastError` keeps the pool
|
||||
/// from being completed at the free-space goal while the entry is still retried.
|
||||
Entry,
|
||||
/// A retryable source-cleanup conflict. The bucket stays queued and is retried, so a
|
||||
/// transient lock conflict must not be recorded as a permanent warning that would block
|
||||
/// pool completion for the rest of the run.
|
||||
SourceCleanup,
|
||||
}
|
||||
|
||||
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default, Serialize, Deserialize)]
|
||||
pub enum RebalStatus {
|
||||
#[default]
|
||||
|
||||
@@ -1,9 +1,9 @@
|
||||
use super::migration::MigrationVersionResult;
|
||||
use super::{
|
||||
DEFAULT_REBALANCE_MAX_ATTEMPTS, EVENT_REBALANCE_LISTING, LOG_COMPONENT_ECSTORE, LOG_SUBSYSTEM_REBALANCE, REBAL_META_NAME,
|
||||
REBALANCE_LISTING_RETRY_BASE_DELAY, REBALANCE_MAX_ATTEMPTS_ENV, REBALANCE_MIGRATION_LOCK_RETRY_CAP,
|
||||
REBALANCE_MIGRATION_RETRY_BASE_DELAY, REBALANCE_SOURCE_CLEANUP_DEFERRED_ERROR_PREFIX, RebalanceBucketConfigs,
|
||||
RebalanceBucketOutcome, RebalanceEntryOutcome, Result,
|
||||
REBALANCE_DEFERRED_ENTRY_ERROR_PREFIX, REBALANCE_LISTING_RETRY_BASE_DELAY, REBALANCE_MAX_ATTEMPTS_ENV,
|
||||
REBALANCE_MIGRATION_LOCK_RETRY_CAP, REBALANCE_MIGRATION_RETRY_BASE_DELAY, REBALANCE_SOURCE_CLEANUP_DEFERRED_ERROR_PREFIX,
|
||||
RebalanceBucketConfigs, RebalanceBucketOutcome, RebalanceDeferKind, RebalanceEntryOutcome, Result,
|
||||
};
|
||||
use crate::cache_value::metacache_set::{ListPathRawOptions, list_path_raw};
|
||||
use crate::core::pools::ListCallback;
|
||||
@@ -109,7 +109,13 @@ pub(super) fn resolve_rebalance_save_task_result(
|
||||
}
|
||||
|
||||
pub(super) fn resolve_rebalance_meta_save_result(result: Result<()>, stage: &str) -> Result<()> {
|
||||
result.map_err(|err| Error::other(format!("rebalance meta save failed during {stage}: {err}")))
|
||||
// Keep the source error reachable: the metadata retry policy classifies
|
||||
// transient lock timeouts by inspecting the source chain, so collapsing the
|
||||
// failure into a plain string here would make that retry a no-op.
|
||||
result.map_err(|err| {
|
||||
let rendered = format!("rebalance meta save failed during {stage}: {err}");
|
||||
crate::data_movement::data_movement_context_error(rendered, err)
|
||||
})
|
||||
}
|
||||
|
||||
pub(super) fn rebalance_meta_lock_error(err: rustfs_lock::LockError, mode: &'static str) -> Error {
|
||||
@@ -177,7 +183,7 @@ pub(super) fn resolve_rebalance_entry_cleanup_delete_result(
|
||||
) -> RebalanceEntryCleanupResult {
|
||||
match result {
|
||||
Ok(_) => RebalanceEntryCleanupResult::Completed { warning: None },
|
||||
Err(SourceCleanupError::Storage(err)) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => {
|
||||
Err(SourceCleanupError::Storage(err)) if is_source_cleanup_not_found(&err) => {
|
||||
RebalanceEntryCleanupResult::Completed { warning: None }
|
||||
}
|
||||
Err(SourceCleanupError::SourceChanged) => RebalanceEntryCleanupResult::Deferred {
|
||||
@@ -185,12 +191,25 @@ pub(super) fn resolve_rebalance_entry_cleanup_delete_result(
|
||||
"{REBALANCE_SOURCE_CLEANUP_DEFERRED_ERROR_PREFIX} source changed during cleanup preflight for {bucket}/{object_name}"
|
||||
),
|
||||
},
|
||||
// A transient cleanup failure is not evidence that the source replica is gone, so the
|
||||
// entry stays incomplete and the bucket is retried instead of recording a permanent
|
||||
// cleanup warning that would block pool completion.
|
||||
Err(SourceCleanupError::Storage(err)) if is_transient_rebalance_error(&err) => RebalanceEntryCleanupResult::Deferred {
|
||||
last_error: format!(
|
||||
"{REBALANCE_SOURCE_CLEANUP_DEFERRED_ERROR_PREFIX} transient source cleanup failure for {bucket}/{object_name} will be retried: {err}"
|
||||
),
|
||||
},
|
||||
Err(SourceCleanupError::Storage(err)) => RebalanceEntryCleanupResult::Completed {
|
||||
warning: Some(format!("rebalance cleanup delete failed for {bucket}/{object_name}: {err}")),
|
||||
},
|
||||
}
|
||||
}
|
||||
|
||||
fn is_source_cleanup_not_found(err: &Error) -> bool {
|
||||
let err = rebalance_error_source(err);
|
||||
is_err_object_not_found(err) || is_err_version_not_found(err)
|
||||
}
|
||||
|
||||
pub(super) fn resolve_rebalance_migrate_result_error(
|
||||
err: Option<Error>,
|
||||
pool_idx: usize,
|
||||
@@ -210,6 +229,23 @@ pub(super) fn should_defer_rebalance_entry_failure(err: &Error) -> bool {
|
||||
is_transient_rebalance_error(err)
|
||||
}
|
||||
|
||||
pub(super) fn resolve_rebalance_deferred_last_error(
|
||||
kind: RebalanceDeferKind,
|
||||
pending_entry_defer: Option<&str>,
|
||||
last_error: &str,
|
||||
) -> Option<String> {
|
||||
match kind {
|
||||
RebalanceDeferKind::Entry => Some(last_error.to_string()),
|
||||
// A retryable cleanup conflict is progress, not a pool failure, so it must not surface as
|
||||
// `lastError`. It also must not erase an unresolved migration deferral: that marker is the
|
||||
// only signal keeping the pool from completing at the free-space goal while an entry is
|
||||
// still retried, and the two deferrals can be reported by different buckets of one pool.
|
||||
RebalanceDeferKind::SourceCleanup => pending_entry_defer
|
||||
.filter(|pending| pending.starts_with(REBALANCE_DEFERRED_ENTRY_ERROR_PREFIX))
|
||||
.map(str::to_string),
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn resolve_load_rebalance_stats_update_result(result: Result<()>) -> Result<()> {
|
||||
result.map_err(|err| Error::other(format!("rebalance metadata stats refresh failed after load: {err}")))
|
||||
}
|
||||
@@ -299,7 +335,10 @@ fn is_rebalance_transient_io_error(err: &std::io::Error) -> bool {
|
||||
|
||||
fn is_rebalance_transient_message(message: &str) -> bool {
|
||||
let message = message.to_ascii_lowercase();
|
||||
message.contains("lock acquisition timed out")
|
||||
// `LockError::Timeout` renders "Lock acquisition timeout for resource ...", while the
|
||||
// namespace-lock layer renders "lock acquisition timed out on ..."; both are retryable.
|
||||
message.contains("lock acquisition timeout")
|
||||
|| message.contains("lock acquisition timed out")
|
||||
|| message.contains("remote lock rpc timed out")
|
||||
|| message.contains("keepalivetimedout")
|
||||
|| message.contains("i/o timeout")
|
||||
@@ -380,7 +419,8 @@ fn is_rebalance_lock_or_rpc_timeout(err: &Error) -> bool {
|
||||
|
||||
fn is_rebalance_lock_or_rpc_timeout_message(message: &str) -> bool {
|
||||
let message = message.to_ascii_lowercase();
|
||||
message.contains("lock acquisition timed out")
|
||||
message.contains("lock acquisition timeout")
|
||||
|| message.contains("lock acquisition timed out")
|
||||
|| message.contains("remote lock rpc timed out")
|
||||
|| message.contains("keepalivetimedout")
|
||||
}
|
||||
@@ -705,6 +745,29 @@ mod error_source_tests {
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn rebalance_metadata_retry_engages_for_wrapped_meta_save_lock_timeout() {
|
||||
let mut attempts = 0;
|
||||
let result = retry_rebalance_metadata_access(None, 3, || {
|
||||
attempts += 1;
|
||||
std::future::ready(resolve_rebalance_meta_save_result(
|
||||
Err(rebalance_meta_lock_error(
|
||||
rustfs_lock::LockError::timeout(".rustfs.sys/rebalance.bin@latest", Duration::from_secs(5)),
|
||||
"write",
|
||||
)),
|
||||
"save_rebalance_stats for pool 0 opt Stats",
|
||||
))
|
||||
})
|
||||
.await;
|
||||
|
||||
assert_eq!(attempts, 3, "a wrapped meta save lock timeout must stay retryable");
|
||||
let err = result.expect_err("persistent lock contention must not become success");
|
||||
assert!(matches!(
|
||||
rebalance_error_source(&err),
|
||||
Error::Lock(rustfs_lock::LockError::Timeout { .. })
|
||||
));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn rebalance_metadata_retry_cancels_a_pending_attempt() {
|
||||
struct DropProbe(Arc<std::sync::atomic::AtomicUsize>);
|
||||
@@ -807,4 +870,44 @@ mod error_source_tests {
|
||||
}
|
||||
}
|
||||
}
|
||||
#[test]
|
||||
fn rendered_lock_timeout_text_selects_the_lock_backoff() {
|
||||
// The lock backend renders a timeout as "Lock acquisition timeout for resource ...",
|
||||
// so the message matcher must recognize that text when the error arrives re-rendered
|
||||
// instead of as a typed `Error::Lock`.
|
||||
let rendered = rustfs_lock::LockError::timeout("bucket/object@latest", Duration::from_secs(5)).to_string();
|
||||
assert!(
|
||||
rendered.contains("Lock acquisition timeout for resource"),
|
||||
"unexpected lock timeout text: {rendered}"
|
||||
);
|
||||
|
||||
// The lock policy jitters the delay inside its own cap, so a far-out attempt identifies
|
||||
// the selected policy: the linear fallback would return `base * (attempt + 1)`.
|
||||
let far_attempt = 100;
|
||||
assert!(REBALANCE_MIGRATION_RETRY_BASE_DELAY * 101 > REBALANCE_MIGRATION_LOCK_RETRY_CAP);
|
||||
|
||||
let mut error = Error::other(format!("Lock error: {rendered}"));
|
||||
for depth in 0..=3 {
|
||||
assert!(
|
||||
is_transient_rebalance_error(&error),
|
||||
"rendered lock timeout lost retryability at depth {depth}: {error:?}"
|
||||
);
|
||||
assert!(
|
||||
is_rebalance_lock_or_rpc_timeout(&error),
|
||||
"rendered lock timeout lost the lock backoff at depth {depth}: {error:?}"
|
||||
);
|
||||
let delay = rebalance_migration_retry_delay(far_attempt, &error);
|
||||
assert!(
|
||||
delay <= REBALANCE_MIGRATION_LOCK_RETRY_CAP && delay >= Duration::from_millis(1),
|
||||
"rendered lock timeout must stay inside the lock backoff cap at depth {depth}: {delay:?}"
|
||||
);
|
||||
error = crate::data_movement::data_movement_stage_error_for_test(
|
||||
"rebalance_object",
|
||||
"put_object",
|
||||
"bucket",
|
||||
"baseline/00042.bin",
|
||||
error,
|
||||
);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -37,7 +37,7 @@ use rustfs_s3_client::{
|
||||
api_s3_datatypes::ListVersionsResult,
|
||||
credentials::{Credentials, SignatureType, Static, Value},
|
||||
provider_versions::validate_remote_version_id,
|
||||
transition_api::{BucketLookupType, Options, TransitionClient, TransitionCore},
|
||||
transition_api::{BucketLookupType, ObjectInfo, Options, TransitionClient, TransitionCore},
|
||||
transition_api::{ReadCloser, ReaderImpl},
|
||||
};
|
||||
use rustfs_utils::egress::validate_outbound_url;
|
||||
@@ -278,19 +278,7 @@ impl WarmBackendS3 {
|
||||
let mut stat_opts = GetObjectOptions::default();
|
||||
stat_opts.version_id.clone_from(&version.version_id);
|
||||
let info = self.client.stat_object(&self.bucket, &remote_object, &stat_opts).await?;
|
||||
let mut metadata = info.user_metadata;
|
||||
for (name, value) in &info.metadata {
|
||||
if (name
|
||||
.as_str()
|
||||
.starts_with(rustfs_utils::http::metadata_compat::RUSTFS_INTERNAL_PREFIX)
|
||||
|| name
|
||||
.as_str()
|
||||
.starts_with(rustfs_utils::http::metadata_compat::MINIO_INTERNAL_PREFIX))
|
||||
&& let Ok(value) = value.to_str()
|
||||
{
|
||||
metadata.insert(name.as_str().to_string(), value.to_string());
|
||||
}
|
||||
}
|
||||
let metadata = transition_candidate_metadata(info);
|
||||
if transition_candidate_metadata_matches(&metadata, identity)? {
|
||||
if matched_version.is_some() {
|
||||
return Ok(TransitionCandidateProbe::Ambiguous);
|
||||
@@ -313,6 +301,53 @@ impl WarmBackendS3 {
|
||||
advance_version_markers(&mut key_marker, &mut version_id_marker, &versions)?;
|
||||
}
|
||||
}
|
||||
|
||||
async fn probe_unversioned_transition_candidate_identity(
|
||||
&self,
|
||||
object: &str,
|
||||
identity: TransitionCandidateIdentity,
|
||||
) -> Result<TransitionCandidateProbe, std::io::Error> {
|
||||
let remote_object = self.get_dest(object);
|
||||
match self
|
||||
.client
|
||||
.stat_object(&self.bucket, &remote_object, &GetObjectOptions::default())
|
||||
.await
|
||||
{
|
||||
Ok(info) => {
|
||||
let metadata = transition_candidate_metadata(info);
|
||||
if transition_candidate_metadata_matches(&metadata, identity)? {
|
||||
Ok(TransitionCandidateProbe::UnversionedPresent)
|
||||
} else {
|
||||
Ok(TransitionCandidateProbe::Unsupported)
|
||||
}
|
||||
}
|
||||
Err(err) => {
|
||||
let response = to_error_response(&err);
|
||||
if response.code == S3ErrorCode::NoSuchKey {
|
||||
Ok(TransitionCandidateProbe::Missing)
|
||||
} else {
|
||||
Err(err)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
fn transition_candidate_metadata(info: ObjectInfo) -> HashMap<String, String> {
|
||||
let mut metadata = info.user_metadata;
|
||||
for (name, value) in &info.metadata {
|
||||
if (name
|
||||
.as_str()
|
||||
.starts_with(rustfs_utils::http::metadata_compat::RUSTFS_INTERNAL_PREFIX)
|
||||
|| name
|
||||
.as_str()
|
||||
.starts_with(rustfs_utils::http::metadata_compat::MINIO_INTERNAL_PREFIX))
|
||||
&& let Ok(value) = value.to_str()
|
||||
{
|
||||
metadata.insert(name.as_str().to_string(), value.to_string());
|
||||
}
|
||||
}
|
||||
metadata
|
||||
}
|
||||
|
||||
fn transition_candidate_metadata_matches(
|
||||
@@ -525,6 +560,13 @@ mod tests {
|
||||
}
|
||||
|
||||
async fn scripted_probe_fixture(responses: Vec<String>) -> Option<(WarmBackendS3, tokio::task::JoinHandle<Vec<String>>)> {
|
||||
scripted_probe_fixture_for_tier(responses, "s3").await
|
||||
}
|
||||
|
||||
async fn scripted_probe_fixture_for_tier(
|
||||
responses: Vec<String>,
|
||||
tier_type: &str,
|
||||
) -> Option<(WarmBackendS3, tokio::task::JoinHandle<Vec<String>>)> {
|
||||
let listener = match tokio::net::TcpListener::bind("127.0.0.1:0").await {
|
||||
Ok(listener) => listener,
|
||||
Err(err) if err.kind() == std::io::ErrorKind::PermissionDenied => return None,
|
||||
@@ -571,7 +613,7 @@ mod tests {
|
||||
max_retries: 1,
|
||||
..Default::default()
|
||||
},
|
||||
"s3",
|
||||
tier_type,
|
||||
)
|
||||
.await
|
||||
.expect("fixture client should build"),
|
||||
@@ -675,6 +717,37 @@ mod tests {
|
||||
assert!(requests[8].to_ascii_lowercase().contains("?versionid=historical-version"));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn unversioned_candidate_reconciler_uses_head_without_version_apis() {
|
||||
let identity = candidate_identity();
|
||||
let transaction_id = identity.transaction_id;
|
||||
let destination_id = rustfs_utils::crypto::hex(identity.destination_id);
|
||||
let response = format!(
|
||||
"HTTP/1.1 200 OK\r\nContent-Length: 0\r\nx-amz-meta-x-rustfs-internal-transition-transaction-id: {transaction_id}\r\nx-amz-meta-x-rustfs-internal-transition-tier-destination-id: {destination_id}\r\nConnection: close\r\n\r\n"
|
||||
);
|
||||
let Some((backend, fixture)) = scripted_probe_fixture_for_tier(vec![response], "r2").await else {
|
||||
return;
|
||||
};
|
||||
|
||||
assert_eq!(
|
||||
crate::services::tier::warm_backend::TransitionCandidateReconciler::probe_transition_candidate_for(
|
||||
&backend,
|
||||
"archive/object",
|
||||
identity,
|
||||
)
|
||||
.await
|
||||
.expect("R2 candidate probe should use the current unversioned object"),
|
||||
TransitionCandidateProbe::UnversionedPresent
|
||||
);
|
||||
let requests = fixture.await.expect("R2 candidate fixture should finish");
|
||||
assert_eq!(requests.len(), 1);
|
||||
let request = requests[0].to_ascii_lowercase();
|
||||
assert!(request.starts_with("head /bucket/archive/object"));
|
||||
assert!(!request.contains("versioning"));
|
||||
assert!(!request.contains("versions"));
|
||||
assert!(!request.contains("versionid="));
|
||||
}
|
||||
|
||||
fn legacy_probe_xml_response(body: &str) -> String {
|
||||
format!(
|
||||
"HTTP/1.1 200 OK\r\nContent-Type: application/xml\r\nContent-Length: {}\r\nConnection: close\r\n\r\n{body}",
|
||||
@@ -1089,6 +1162,10 @@ impl TransitionCandidateReconciler for WarmBackendS3 {
|
||||
object: &str,
|
||||
identity: TransitionCandidateIdentity,
|
||||
) -> Result<TransitionCandidateProbe, std::io::Error> {
|
||||
let capabilities = self.client.provider_version_capabilities();
|
||||
if !capabilities.bucket_versioning_state && !capabilities.list_object_versions {
|
||||
return self.probe_unversioned_transition_candidate_identity(object, identity).await;
|
||||
}
|
||||
let bucket_versioning = self.remote_bucket_versioning().await?;
|
||||
self.probe_transition_candidate_identity(object, identity, bucket_versioning)
|
||||
.await
|
||||
|
||||
@@ -3494,19 +3494,44 @@ fn dangling_delete_grace() -> time::Duration {
|
||||
/// Result of scanning one disk's copy of a directory prefix while deciding
|
||||
/// whether an orphan (metadata-less) directory tree can be safely purged.
|
||||
enum OrphanDirScan {
|
||||
/// The subtree holds object metadata or uncommitted data, so it must not be
|
||||
/// purged.
|
||||
HasData,
|
||||
/// The prefix contains only empty directories and/or UUID data directories
|
||||
/// carrying a committed delete marker.
|
||||
Purgeable {
|
||||
empty_dirs: Vec<String>,
|
||||
/// The prefix exists on this disk. `blocked` holds every directory that is
|
||||
/// itself, or an ancestor of, object metadata or uncommitted data (closed
|
||||
/// under taking parents, root included when anything under it is blocked);
|
||||
/// `dirs` is the pre-order list of every directory reached that is not
|
||||
/// blocked, and `committed_files` the erasure data and committed delete
|
||||
/// markers found in the unblocked UUID data dirs among them.
|
||||
Scanned {
|
||||
blocked: HashSet<String>,
|
||||
dirs: Vec<String>,
|
||||
committed_files: Vec<String>,
|
||||
},
|
||||
/// A directory read failed for a reason other than absence, so nothing
|
||||
/// under the prefix can be classified on this disk.
|
||||
Unreadable,
|
||||
/// The prefix does not exist on this disk.
|
||||
Missing,
|
||||
}
|
||||
|
||||
/// How long an orphan prefix whose purge scan met unpurgeable data is left
|
||||
/// alone before an empty listing scans it again.
|
||||
const ORPHAN_PURGE_BACKOFF: Duration = Duration::from_secs(60);
|
||||
/// Upper bound on remembered backoff entries; the oldest is dropped first.
|
||||
const ORPHAN_PURGE_BACKOFF_MAX_ENTRIES: usize = 4096;
|
||||
|
||||
/// Mark `dir` and every ancestor up to and including `root` as blocked.
|
||||
fn block_orphan_dir_chain(blocked: &mut HashSet<String>, root: &str, dir: &str) {
|
||||
let mut current = dir;
|
||||
loop {
|
||||
if !blocked.insert(current.to_owned()) || current == root {
|
||||
return;
|
||||
}
|
||||
let Some((parent, _)) = current.rsplit_once(SLASH_SEPARATOR) else {
|
||||
return;
|
||||
};
|
||||
current = parent;
|
||||
}
|
||||
}
|
||||
|
||||
fn is_safe_orphan_dir_entry(entry: &str) -> bool {
|
||||
let component = entry.strip_suffix(SLASH_SEPARATOR).unwrap_or(entry);
|
||||
!component.is_empty()
|
||||
@@ -6311,9 +6336,12 @@ impl SetDisks {
|
||||
.await
|
||||
}
|
||||
|
||||
/// Scan a single disk's copy of `prefix` and decide whether it is an orphan
|
||||
/// directory subtree. Only empty directories and UUID data directories with
|
||||
/// valid committed delete markers are purgeable; every child is still scanned.
|
||||
/// Scan a single disk's copy of `prefix` and classify every directory
|
||||
/// under it. Only empty directories and UUID data directories with valid
|
||||
/// committed delete markers are purgeable; anything else blocks its whole
|
||||
/// ancestor chain while sibling subtrees stay purgeable, so committed
|
||||
/// residue is still reclaimed when it shares an ancestor with residue from
|
||||
/// an older build that never wrote markers (#6898).
|
||||
async fn scan_orphan_dir(disk: &DiskStore, bucket: &str, prefix: &str) -> OrphanDirScan {
|
||||
let root = prefix.trim_end_matches(SLASH_SEPARATOR).to_string();
|
||||
let mut stack = vec![root.clone()];
|
||||
@@ -6321,6 +6349,7 @@ impl SetDisks {
|
||||
// so reversing it yields a safe children-first removal order.
|
||||
let mut dirs: Vec<String> = Vec::new();
|
||||
let mut committed_files: Vec<String> = Vec::new();
|
||||
let mut blocked: HashSet<String> = HashSet::new();
|
||||
let mut existed = false;
|
||||
|
||||
while let Some(dir) = stack.pop() {
|
||||
@@ -6335,16 +6364,18 @@ impl SetDisks {
|
||||
}
|
||||
// Classification must fail closed: committed residue is safe to
|
||||
// remove only after every reachable child was inspected.
|
||||
Err(_) => return OrphanDirScan::HasData,
|
||||
Err(_) => return OrphanDirScan::Unreadable,
|
||||
};
|
||||
|
||||
existed = true;
|
||||
let mut child_dirs = Vec::new();
|
||||
let mut files = Vec::new();
|
||||
|
||||
let mut has_data = false;
|
||||
for entry in entries {
|
||||
if !is_safe_orphan_dir_entry(&entry) {
|
||||
return OrphanDirScan::HasData;
|
||||
has_data = true;
|
||||
break;
|
||||
}
|
||||
match entry.strip_suffix(SLASH_SEPARATOR) {
|
||||
Some(child) => child_dirs.push(format!("{dir}{SLASH_SEPARATOR}{child}")),
|
||||
@@ -6352,28 +6383,29 @@ impl SetDisks {
|
||||
}
|
||||
}
|
||||
|
||||
if !files.is_empty() {
|
||||
if !has_data && !files.is_empty() {
|
||||
let data_dir_name = dir.rsplit(SLASH_SEPARATOR).next().unwrap_or_default();
|
||||
let is_uuid_data_dir = Uuid::parse_str(data_dir_name).is_ok_and(|uuid| !uuid.is_nil());
|
||||
let has_committed_delete = files.iter().any(|entry| is_committed_delete_marker(entry));
|
||||
has_data = !is_uuid_data_dir || !has_committed_delete || files.iter().any(|entry| entry == STORAGE_FORMAT_FILE);
|
||||
}
|
||||
|
||||
if !is_uuid_data_dir || !has_committed_delete || files.iter().any(|entry| entry == STORAGE_FORMAT_FILE) {
|
||||
return OrphanDirScan::HasData;
|
||||
}
|
||||
|
||||
committed_files.extend(files.into_iter().map(|entry| path_join_buf(&[&dir, &entry])));
|
||||
dirs.push(dir);
|
||||
stack.extend(child_dirs);
|
||||
if has_data {
|
||||
// Nothing below a blocked directory is ever removed, so its
|
||||
// children need no classification.
|
||||
block_orphan_dir_chain(&mut blocked, &root, &dir);
|
||||
continue;
|
||||
}
|
||||
|
||||
committed_files.extend(files.into_iter().map(|entry| format!("{dir}{SLASH_SEPARATOR}{entry}")));
|
||||
dirs.push(dir);
|
||||
stack.extend(child_dirs);
|
||||
}
|
||||
|
||||
if existed {
|
||||
OrphanDirScan::Purgeable {
|
||||
empty_dirs: dirs,
|
||||
OrphanDirScan::Scanned {
|
||||
blocked,
|
||||
dirs,
|
||||
committed_files,
|
||||
}
|
||||
} else {
|
||||
@@ -6468,43 +6500,101 @@ impl SetDisks {
|
||||
/// of this set (the caller should surface the original NotFound), and `Err` on
|
||||
/// a hard disk failure.
|
||||
pub(crate) async fn purge_orphan_dir_object(&self, bucket: &str, object: &str) -> disk::error::Result<bool> {
|
||||
let backoff_key = format!("{bucket}{SLASH_SEPARATOR}{object}");
|
||||
if self.orphan_purge_in_backoff(&backoff_key) {
|
||||
return Ok(false);
|
||||
}
|
||||
|
||||
let disks = self.get_disks_internal().await;
|
||||
|
||||
// Phase 1: classify every online disk. Refuse to purge if ANY disk holds
|
||||
// object data under the prefix, so a degraded/healable object is never
|
||||
// destroyed.
|
||||
let mut per_disk_dirs: Vec<(usize, Vec<String>, Vec<String>)> = Vec::new();
|
||||
let mut existed = false;
|
||||
// Phase 1: classify every online disk. A directory that holds object
|
||||
// data or uncommitted residue on ANY disk blocks itself and its
|
||||
// ancestors on every disk, so a degraded/healable object is never
|
||||
// destroyed; purgeable subtrees beside it are still reclaimed.
|
||||
let mut per_disk: Vec<(usize, Vec<String>, Vec<String>)> = Vec::new();
|
||||
let mut blocked: HashSet<String> = HashSet::new();
|
||||
for (i, disk) in disks.iter().enumerate() {
|
||||
let Some(disk) = disk else { continue };
|
||||
match Self::scan_orphan_dir(disk, bucket, object).await {
|
||||
OrphanDirScan::HasData => return Ok(false),
|
||||
OrphanDirScan::Purgeable {
|
||||
empty_dirs,
|
||||
OrphanDirScan::Unreadable => return Ok(false),
|
||||
OrphanDirScan::Scanned {
|
||||
blocked: disk_blocked,
|
||||
dirs,
|
||||
committed_files,
|
||||
} => {
|
||||
existed = true;
|
||||
per_disk_dirs.push((i, empty_dirs, committed_files));
|
||||
blocked.extend(disk_blocked);
|
||||
per_disk.push((i, dirs, committed_files));
|
||||
}
|
||||
OrphanDirScan::Missing => {}
|
||||
}
|
||||
}
|
||||
|
||||
if !existed {
|
||||
return Ok(false);
|
||||
if !blocked.is_empty() {
|
||||
// Whatever is purgeable goes now; what blocks the rest will still
|
||||
// block it on the next empty listing, so do not rescan for a while.
|
||||
self.record_orphan_purge_backoff(backoff_key);
|
||||
}
|
||||
|
||||
// Phase 2: remove only the files classified as committed residue, then
|
||||
// remove directories children-first. Every directory delete is
|
||||
// non-recursive, so a directory that concurrently gained an object fails
|
||||
// with DirectoryNotEmpty and is skipped — a racing PutObject is never
|
||||
// Phase 2: remove only the files classified as committed residue in
|
||||
// unblocked data dirs, then remove unblocked directories
|
||||
// children-first. Every directory delete is non-recursive, so a
|
||||
// directory that concurrently gained an object fails with
|
||||
// DirectoryNotEmpty and is skipped — a racing PutObject is never
|
||||
// clobbered.
|
||||
for (i, empty_dirs, committed_files) in per_disk_dirs {
|
||||
let mut purged = false;
|
||||
for (i, dirs, committed_files) in per_disk {
|
||||
let Some(disk) = disks[i].as_ref() else { continue };
|
||||
Self::delete_purgeable_orphan_entries(disk, bucket, object, empty_dirs, committed_files).await;
|
||||
let dirs = dirs.into_iter().filter(|dir| !blocked.contains(dir)).collect::<Vec<_>>();
|
||||
let committed_files = committed_files
|
||||
.into_iter()
|
||||
.filter(|file| {
|
||||
file.rsplit_once(SLASH_SEPARATOR)
|
||||
.is_some_and(|(data_dir, _)| !blocked.contains(data_dir))
|
||||
})
|
||||
.collect::<Vec<_>>();
|
||||
if dirs.is_empty() && committed_files.is_empty() {
|
||||
continue;
|
||||
}
|
||||
purged = true;
|
||||
Self::delete_purgeable_orphan_entries(disk, bucket, object, dirs, committed_files).await;
|
||||
}
|
||||
|
||||
Ok(true)
|
||||
Ok(purged)
|
||||
}
|
||||
|
||||
fn orphan_purge_in_backoff(&self, key: &str) -> bool {
|
||||
let backoff = self
|
||||
.orphan_purge_backoff
|
||||
.lock()
|
||||
.unwrap_or_else(|poisoned| poisoned.into_inner());
|
||||
backoff
|
||||
.get(key)
|
||||
.is_some_and(|scanned_at| scanned_at.elapsed() < ORPHAN_PURGE_BACKOFF)
|
||||
}
|
||||
|
||||
fn record_orphan_purge_backoff(&self, key: String) {
|
||||
let mut backoff = self
|
||||
.orphan_purge_backoff
|
||||
.lock()
|
||||
.unwrap_or_else(|poisoned| poisoned.into_inner());
|
||||
let now = Instant::now();
|
||||
backoff.retain(|_, scanned_at| now.duration_since(*scanned_at) < ORPHAN_PURGE_BACKOFF);
|
||||
if backoff.len() >= ORPHAN_PURGE_BACKOFF_MAX_ENTRIES
|
||||
&& let Some(oldest) = backoff
|
||||
.iter()
|
||||
.min_by_key(|(_, scanned_at)| **scanned_at)
|
||||
.map(|(key, _)| key.clone())
|
||||
{
|
||||
backoff.remove(&oldest);
|
||||
}
|
||||
backoff.insert(key, now);
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(crate) fn clear_orphan_purge_backoff(&self) {
|
||||
self.orphan_purge_backoff
|
||||
.lock()
|
||||
.unwrap_or_else(|poisoned| poisoned.into_inner())
|
||||
.clear();
|
||||
}
|
||||
|
||||
/// Reclaim orphaned physical data directories under `bucket/object` that no
|
||||
@@ -7621,13 +7711,15 @@ mod tests {
|
||||
.await
|
||||
.expect("committed delete marker should be written");
|
||||
|
||||
let OrphanDirScan::Purgeable {
|
||||
empty_dirs,
|
||||
let OrphanDirScan::Scanned {
|
||||
blocked,
|
||||
dirs: empty_dirs,
|
||||
committed_files,
|
||||
} = SetDisks::scan_orphan_dir(&disk, "bucket", "pfx/").await
|
||||
else {
|
||||
panic!("committed residue should be classified as purgeable");
|
||||
};
|
||||
assert!(blocked.is_empty(), "committed residue alone blocks nothing");
|
||||
|
||||
let nested_object = residue.join("nested");
|
||||
tokio::fs::create_dir_all(&nested_object)
|
||||
@@ -7669,13 +7761,15 @@ mod tests {
|
||||
.await
|
||||
.expect("committed delete marker should be written");
|
||||
|
||||
let OrphanDirScan::Purgeable {
|
||||
empty_dirs,
|
||||
let OrphanDirScan::Scanned {
|
||||
blocked,
|
||||
dirs: empty_dirs,
|
||||
committed_files,
|
||||
} = SetDisks::scan_orphan_dir(&disk, "bucket", "pfx/").await
|
||||
else {
|
||||
panic!("committed residue should be classified as purgeable");
|
||||
};
|
||||
assert!(blocked.is_empty(), "committed residue alone blocks nothing");
|
||||
tokio::fs::set_permissions(&residue, std::fs::Permissions::from_mode(0o555))
|
||||
.await
|
||||
.expect("residue directory should become read-only");
|
||||
|
||||
@@ -874,6 +874,7 @@ mod ctx;
|
||||
mod metadata;
|
||||
mod ops;
|
||||
pub(crate) use ops::bucket::BucketInfoQuorum;
|
||||
pub(crate) use ops::heal::AbsenceProofRequest;
|
||||
#[cfg(test)]
|
||||
pub(crate) use ops::heal::DanglingDeleteFailure;
|
||||
pub(crate) use ops::heal::HealedObjectAbsence;
|
||||
@@ -3900,6 +3901,12 @@ pub struct SetDisks {
|
||||
/// writes skip the global registry mutex (backlog#1315). `Arc` so clones of
|
||||
/// a set share one generation marker.
|
||||
capacity_dirty_generation: Arc<AtomicU64>,
|
||||
/// Orphan prefixes whose last purge scan met data that can never be
|
||||
/// purged by listing (residue without a committed marker, an in-flight
|
||||
/// write), keyed by `bucket/prefix` with the time of that scan. Empty
|
||||
/// listings of such a prefix are frequent and the scan reads the whole
|
||||
/// subtree, so it is not repeated within `ORPHAN_PURGE_BACKOFF` (#6898).
|
||||
orphan_purge_backoff: Arc<std::sync::Mutex<HashMap<String, std::time::Instant>>>,
|
||||
#[cfg(test)]
|
||||
storage_class_config_override: Arc<std::sync::RwLock<Option<Arc<storageclass::Config>>>>,
|
||||
#[cfg(test)]
|
||||
@@ -3908,6 +3915,72 @@ pub struct SetDisks {
|
||||
>,
|
||||
}
|
||||
|
||||
fn marker_purge_receipt_path(
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
version: Uuid,
|
||||
purge: &rustfs_common::mrf_channel::MrfDeleteMarkerPurge,
|
||||
) -> String {
|
||||
let object_hash = rustfs_utils::crypto::hex(Sha256::digest(object.as_bytes()));
|
||||
let marker_hash = rustfs_utils::crypto::hex(purge.marker_identity);
|
||||
format!(
|
||||
"marker-purge-receipts/{}/{}/{}-{}-{}.json",
|
||||
bucket, purge.bucket_incarnation_id, version, object_hash, marker_hash
|
||||
)
|
||||
}
|
||||
|
||||
impl SetDisks {
|
||||
/// Persist a quorum DELETE receipt outside the object marker itself. The
|
||||
/// receipt is written to the metadata bucket so it survives a node restart
|
||||
/// while an inaccessible member still holds the stale marker.
|
||||
pub(crate) async fn persist_marker_purge_receipt(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
version: Uuid,
|
||||
purge: &rustfs_common::mrf_channel::MrfDeleteMarkerPurge,
|
||||
) -> bool {
|
||||
let path = marker_purge_receipt_path(bucket, object, version, purge);
|
||||
let api = Arc::new(self.clone());
|
||||
crate::config::com::save_config_with_opts(
|
||||
api,
|
||||
&path,
|
||||
b"rustfs-marker-purge-receipt-v1".to_vec(),
|
||||
&ObjectOptions {
|
||||
max_parity: true,
|
||||
no_lock: true,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.is_ok()
|
||||
}
|
||||
|
||||
pub(crate) async fn has_marker_purge_receipt(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
version: Uuid,
|
||||
purge: &rustfs_common::mrf_channel::MrfDeleteMarkerPurge,
|
||||
) -> bool {
|
||||
let path = marker_purge_receipt_path(bucket, object, version, purge);
|
||||
crate::config::com::read_config_limited_preserve_empty(Arc::new(self.clone()), &path, 128)
|
||||
.await
|
||||
.is_ok()
|
||||
}
|
||||
|
||||
pub(crate) async fn consume_marker_purge_receipt(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
version: Uuid,
|
||||
purge: &rustfs_common::mrf_channel::MrfDeleteMarkerPurge,
|
||||
) {
|
||||
let path = marker_purge_receipt_path(bucket, object, version, purge);
|
||||
let _ = crate::config::com::delete_config_no_lock(Arc::new(self.clone()), &path).await;
|
||||
}
|
||||
}
|
||||
|
||||
/// Read every physical copy before selecting a version quorum. A minority
|
||||
/// legacy record is still evidence and must not disappear behind a majority
|
||||
/// not-found result. Only an explicit file/volume absence produces `None`;
|
||||
@@ -4690,6 +4763,7 @@ impl SetDisks {
|
||||
ctx,
|
||||
capacity_scope_cache: Arc::new(std::sync::RwLock::new(CapacityScopeCache::default())),
|
||||
capacity_dirty_generation: Arc::new(AtomicU64::new(u64::MAX)),
|
||||
orphan_purge_backoff: Arc::new(std::sync::Mutex::new(HashMap::new())),
|
||||
#[cfg(test)]
|
||||
storage_class_config_override: Arc::new(std::sync::RwLock::new(None)),
|
||||
#[cfg(test)]
|
||||
@@ -6399,6 +6473,28 @@ async fn verify_inline_part_bitrot(meta: &FileInfo) -> disk::error::Result<()> {
|
||||
.map_err(|_| DiskError::FileCorrupt)
|
||||
}
|
||||
|
||||
fn validate_deep_scan_results(results: &[usize], expected_parts: usize) -> disk::error::Result<()> {
|
||||
if results.len() != expected_parts {
|
||||
return Err(DiskError::other(format!(
|
||||
"incomplete deep scan result: expected {expected_parts} parts, received {}",
|
||||
results.len()
|
||||
)));
|
||||
}
|
||||
for (part, status) in results.iter().enumerate() {
|
||||
match *status {
|
||||
CHECK_PART_SUCCESS | CHECK_PART_FILE_NOT_FOUND | CHECK_PART_FILE_CORRUPT => {}
|
||||
CHECK_PART_DISK_NOT_FOUND => return Err(DiskError::DiskNotFound),
|
||||
crate::disk::CHECK_PART_VOLUME_NOT_FOUND => return Err(DiskError::VolumeNotFound),
|
||||
_ => {
|
||||
return Err(DiskError::other(format!(
|
||||
"incomplete deep scan result: part {part} has unverified status {status}"
|
||||
)));
|
||||
}
|
||||
}
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// disks_with_all_partsv2 is a corrected version based on Go implementation.
|
||||
/// It sets partsMetadata and onlineDisks when xl.meta is inexistant/corrupted or outdated.
|
||||
/// It also checks if the status of each part (corrupted, missing, ok) in each drive.
|
||||
@@ -6619,9 +6715,13 @@ async fn disks_with_all_parts(
|
||||
// it needs healing too.
|
||||
match disk.verify_file(bucket, object, meta).await {
|
||||
Ok(v) => {
|
||||
validate_deep_scan_results(&v.results, latest_meta.parts.len())?;
|
||||
verify_resp = v;
|
||||
}
|
||||
Err(err) => {
|
||||
if !matches!(err, DiskError::FileNotFound | DiskError::FileVersionNotFound | DiskError::FileCorrupt) {
|
||||
return Err(err);
|
||||
}
|
||||
debug!(
|
||||
event = EVENT_SET_DISK_HEAL,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
@@ -9337,6 +9437,175 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
async fn write_committed_residue(object_dir: &std::path::Path) -> std::path::PathBuf {
|
||||
let residue = object_dir.join(Uuid::new_v4().to_string());
|
||||
fs::create_dir_all(&residue)
|
||||
.await
|
||||
.expect("committed data directory should be created");
|
||||
fs::write(residue.join("part.1"), b"stale")
|
||||
.await
|
||||
.expect("stale part should be written");
|
||||
fs::write(
|
||||
residue.join(format!("{}{}", crate::disk::local::DELETE_DATA_DIR_MARKER_PREFIX, Uuid::new_v4())),
|
||||
[],
|
||||
)
|
||||
.await
|
||||
.expect("committed delete marker should be written");
|
||||
residue
|
||||
}
|
||||
|
||||
// #6898: residue from a build that never wrote delete markers blocks only
|
||||
// its own ancestor chain; a committed sibling subtree is still reclaimed
|
||||
// and the call reports the partial purge.
|
||||
#[tokio::test]
|
||||
async fn purge_orphan_dir_object_reclaims_committed_subtree_beside_blocked_subtree() {
|
||||
let (dir0, disk0) = make_single_local_disk().await;
|
||||
let (dir1, disk1) = make_single_local_disk().await;
|
||||
|
||||
let committed = write_committed_residue(&dir0.path().join("bucket/pfx/a/obj")).await;
|
||||
let unmarked = dir1.path().join("bucket/pfx/b/obj").join(Uuid::new_v4().to_string());
|
||||
fs::create_dir_all(&unmarked)
|
||||
.await
|
||||
.expect("unmarked residue should be created");
|
||||
fs::write(unmarked.join("part.1"), b"stale")
|
||||
.await
|
||||
.expect("stale part should be written");
|
||||
fs::create_dir_all(dir0.path().join("bucket/pfx/b/obj"))
|
||||
.await
|
||||
.expect("disk0 copy of the blocked chain should be created");
|
||||
|
||||
let set = make_set_disks_with(vec![Some(disk0), Some(disk1)]).await;
|
||||
let purged = set
|
||||
.purge_orphan_dir_object("bucket", "pfx/")
|
||||
.await
|
||||
.expect("scan should succeed");
|
||||
|
||||
assert!(purged, "the committed subtree must be reported as purged");
|
||||
assert!(!committed.exists(), "committed residue beside a blocked subtree must be reclaimed");
|
||||
assert!(!dir0.path().join("bucket/pfx/a").exists(), "the reclaimed subtree's directories must go");
|
||||
assert!(unmarked.join("part.1").exists(), "unmarked residue must survive");
|
||||
assert!(
|
||||
dir0.path().join("bucket/pfx/b/obj").exists(),
|
||||
"a directory blocked on another disk must be left alone on this one"
|
||||
);
|
||||
assert!(dir0.path().join("bucket/pfx").exists() && dir1.path().join("bucket/pfx").exists());
|
||||
}
|
||||
|
||||
// A data dir that is committed residue on one disk but still holds an
|
||||
// object below it on another disk is blocked on every disk.
|
||||
#[tokio::test]
|
||||
async fn purge_orphan_dir_object_blocks_committed_files_by_other_disks_data() {
|
||||
let (dir0, disk0) = make_single_local_disk().await;
|
||||
let (dir1, disk1) = make_single_local_disk().await;
|
||||
|
||||
let data_dir = Uuid::new_v4();
|
||||
let committed = dir0.path().join("bucket/pfx/x").join(data_dir.to_string());
|
||||
fs::create_dir_all(&committed)
|
||||
.await
|
||||
.expect("committed data directory should be created");
|
||||
fs::write(committed.join("part.1"), b"stale")
|
||||
.await
|
||||
.expect("stale part should be written");
|
||||
fs::write(
|
||||
committed.join(format!("{}{}", crate::disk::local::DELETE_DATA_DIR_MARKER_PREFIX, Uuid::new_v4())),
|
||||
[],
|
||||
)
|
||||
.await
|
||||
.expect("committed delete marker should be written");
|
||||
let nested = dir1.path().join("bucket/pfx/x").join(data_dir.to_string()).join("nested");
|
||||
fs::create_dir_all(&nested)
|
||||
.await
|
||||
.expect("nested object dir should be created");
|
||||
fs::write(nested.join(STORAGE_FORMAT_FILE), b"meta")
|
||||
.await
|
||||
.expect("nested object metadata should be written");
|
||||
|
||||
let set = make_set_disks_with(vec![Some(disk0), Some(disk1)]).await;
|
||||
let purged = set
|
||||
.purge_orphan_dir_object("bucket", "pfx/")
|
||||
.await
|
||||
.expect("scan should succeed");
|
||||
|
||||
assert!(!purged);
|
||||
assert!(
|
||||
committed.join("part.1").exists(),
|
||||
"committed files under a dir blocked elsewhere must remain"
|
||||
);
|
||||
assert!(nested.join(STORAGE_FORMAT_FILE).exists());
|
||||
}
|
||||
|
||||
// An unreadable directory anywhere under the prefix aborts the purge on
|
||||
// every disk before anything is deleted.
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
async fn purge_orphan_dir_object_aborts_when_a_directory_is_unreadable() {
|
||||
use std::os::unix::fs::PermissionsExt;
|
||||
|
||||
let (dir, disk) = make_single_local_disk().await;
|
||||
let committed = write_committed_residue(&dir.path().join("bucket/pfx/b/obj")).await;
|
||||
let sealed = dir.path().join("bucket/pfx/a");
|
||||
fs::create_dir_all(&sealed).await.expect("sealed directory should be created");
|
||||
fs::set_permissions(&sealed, std::fs::Permissions::from_mode(0o000))
|
||||
.await
|
||||
.expect("sealed directory should become unreadable");
|
||||
|
||||
let set = make_set_disks_with(vec![Some(disk)]).await;
|
||||
let purged = set.purge_orphan_dir_object("bucket", "pfx/").await;
|
||||
fs::set_permissions(&sealed, std::fs::Permissions::from_mode(0o755))
|
||||
.await
|
||||
.expect("sealed directory should be restored");
|
||||
|
||||
assert!(matches!(purged, Ok(false)), "an unreadable directory must abort the purge");
|
||||
assert!(committed.join("part.1").exists(), "nothing may be deleted once classification failed");
|
||||
}
|
||||
|
||||
// After a scan met unpurgeable data the prefix is not rescanned for a
|
||||
// while, even when new committed residue appears under it.
|
||||
#[tokio::test]
|
||||
async fn purge_orphan_dir_object_backs_off_after_blocked_scan() {
|
||||
let (dir, disk) = make_single_local_disk().await;
|
||||
let unmarked = dir.path().join("bucket/pfx/b/obj").join(Uuid::new_v4().to_string());
|
||||
fs::create_dir_all(&unmarked)
|
||||
.await
|
||||
.expect("unmarked residue should be created");
|
||||
fs::write(unmarked.join("part.1"), b"stale")
|
||||
.await
|
||||
.expect("stale part should be written");
|
||||
|
||||
let set = make_set_disks_with(vec![Some(disk)]).await;
|
||||
assert!(
|
||||
!set.purge_orphan_dir_object("bucket", "pfx/")
|
||||
.await
|
||||
.expect("scan should succeed")
|
||||
);
|
||||
|
||||
let committed = write_committed_residue(&dir.path().join("bucket/pfx/a/obj")).await;
|
||||
assert!(
|
||||
!set.purge_orphan_dir_object("bucket", "pfx/")
|
||||
.await
|
||||
.expect("scan should succeed"),
|
||||
"a prefix in backoff is not rescanned"
|
||||
);
|
||||
assert!(committed.exists());
|
||||
assert!(
|
||||
set.purge_orphan_dir_object("bucket", "pfx/a/")
|
||||
.await
|
||||
.expect("scan should succeed"),
|
||||
"backoff is per prefix, a narrower prefix still purges"
|
||||
);
|
||||
assert!(!committed.exists());
|
||||
|
||||
let committed = write_committed_residue(&dir.path().join("bucket/pfx/a/obj")).await;
|
||||
set.clear_orphan_purge_backoff();
|
||||
assert!(
|
||||
set.purge_orphan_dir_object("bucket", "pfx/")
|
||||
.await
|
||||
.expect("scan should succeed")
|
||||
);
|
||||
assert!(!committed.exists());
|
||||
assert!(unmarked.join("part.1").exists());
|
||||
}
|
||||
|
||||
// Build an `xl.meta` under `object_dir` whose versions reference `data_dirs`
|
||||
// (one Object version per data dir, each with its own version id).
|
||||
async fn write_object_meta_with_data_dirs(object_dir: &std::path::Path, bucket: &str, object: &str, data_dirs: &[Uuid]) {
|
||||
@@ -10866,6 +11135,50 @@ mod tests {
|
||||
assert_eq!(conv_part_err_to_int(&Some(other_err)), CHECK_PART_UNKNOWN); // Other errors should return UNKNOWN, not SUCCESS
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn deep_scan_results_accept_only_complete_verified_or_repairable_parts() {
|
||||
for (results, expected_parts) in [
|
||||
(vec![], 0),
|
||||
(vec![CHECK_PART_SUCCESS], 1),
|
||||
(vec![CHECK_PART_FILE_NOT_FOUND], 1),
|
||||
(vec![CHECK_PART_FILE_CORRUPT], 1),
|
||||
(vec![CHECK_PART_SUCCESS, CHECK_PART_FILE_NOT_FOUND, CHECK_PART_FILE_CORRUPT], 3),
|
||||
] {
|
||||
validate_deep_scan_results(&results, expected_parts)
|
||||
.expect("complete results must allow healthy or repairable parts");
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn deep_scan_results_reject_unknown_and_incomplete_observations() {
|
||||
for (results, expected_parts) in [
|
||||
(vec![CHECK_PART_UNKNOWN], 1),
|
||||
(vec![usize::MAX], 1),
|
||||
(vec![CHECK_PART_SUCCESS, CHECK_PART_UNKNOWN], 2),
|
||||
(vec![], 1),
|
||||
(vec![CHECK_PART_SUCCESS], 2),
|
||||
(vec![CHECK_PART_SUCCESS, CHECK_PART_SUCCESS], 1),
|
||||
(vec![CHECK_PART_SUCCESS], 0),
|
||||
] {
|
||||
let error =
|
||||
validate_deep_scan_results(&results, expected_parts).expect_err("unverified parts must not become healthy");
|
||||
assert!(matches!(error, DiskError::Io(_)), "an incomplete observation is not proven corruption");
|
||||
assert!(error.to_string().contains("incomplete deep scan result"));
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn deep_scan_results_preserve_disk_and_volume_failures() {
|
||||
for (status, expected_error) in [
|
||||
(CHECK_PART_DISK_NOT_FOUND, DiskError::DiskNotFound),
|
||||
(CHECK_PART_VOLUME_NOT_FOUND, DiskError::VolumeNotFound),
|
||||
] {
|
||||
let error = validate_deep_scan_results(&[CHECK_PART_SUCCESS, status], 2)
|
||||
.expect_err("an unavailable part cannot certify a healthy disk");
|
||||
assert_eq!(error, expected_error);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_has_part_err() {
|
||||
// Test checking for part errors
|
||||
|
||||
@@ -141,6 +141,7 @@ pub(in crate::set_disk::ops) async fn verify_written_bitrot_shards(
|
||||
mod tests {
|
||||
use super::super::object::hermetic_set_disks_support::hermetic_set_disks_for_pool_with_default_parity;
|
||||
use super::*;
|
||||
use crate::storage_api_contracts::object::ObjectIO as _;
|
||||
|
||||
async fn encode_streaming_shard(data: &[u8], shard_size: usize) -> Bytes {
|
||||
let mut writer = coding::BitrotWriter::new(Cursor::new(Vec::new()), shard_size, HashAlgorithm::HighwayHash256S);
|
||||
@@ -295,4 +296,19 @@ mod tests {
|
||||
assert!(err.to_string().contains("trailing data"));
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn no_parity_put_round_trip_large_stream() {
|
||||
let (_temp_dirs, disks, set_disks) = hermetic_set_disks_for_pool_with_default_parity(1, 0, 0).await;
|
||||
let bucket = "no-parity-put-round-trip";
|
||||
for disk in &disks {
|
||||
disk.make_volume(bucket).await.expect("bucket volume should be created");
|
||||
}
|
||||
let payload = vec![0x5a; 13 * 1024 * 1024 + 727_213];
|
||||
let mut reader = crate::set_disk::PutObjReader::from_vec(payload);
|
||||
set_disks
|
||||
.put_object(bucket, "object", &mut reader, &crate::set_disk::ObjectOptions::default())
|
||||
.await
|
||||
.expect("single-disk no-parity PUT should pass self-verify");
|
||||
}
|
||||
}
|
||||
|
||||
@@ -38,6 +38,7 @@ use crate::disk::DiskAPI;
|
||||
use crate::disk::local::{DELETE_DATA_DIR_MARKER_PREFIX, metadata_less_part_file};
|
||||
use crate::io_support::bitrot::object_mmap_read_enabled;
|
||||
use crate::storage_api_contracts::namespace::NamespaceLocking as _;
|
||||
use rustfs_common::mrf_channel::MrfDeleteMarkerPurge;
|
||||
use rustfs_common::trace_bus::{TraceEvent, TraceFunc, TraceKind, trace_emit};
|
||||
use tracing::trace;
|
||||
|
||||
@@ -55,6 +56,13 @@ pub(crate) struct HealedObjectAbsence {
|
||||
pub removed: bool,
|
||||
}
|
||||
|
||||
/// Conditions under which a missing object may become a storage-owned proof.
|
||||
#[derive(Clone, Copy)]
|
||||
pub(crate) struct AbsenceProofRequest<'a> {
|
||||
pub retirement: Option<&'a MarkerRetirementContext<'a>>,
|
||||
pub allow_unversioned: bool,
|
||||
}
|
||||
|
||||
fn is_unavailable_heal_rpc(error: &DiskError) -> bool {
|
||||
let DiskError::Io(error) = error else {
|
||||
return false;
|
||||
@@ -846,7 +854,7 @@ impl SetDisks {
|
||||
);
|
||||
|
||||
let disks = self.get_disks_internal().await;
|
||||
let bucket_heal_scope = crate::store::bucket_heal_scope(bucket);
|
||||
let bucket_heal_scope = crate::store::bucket_heal_scope_for_object(bucket, object);
|
||||
if let Some(scope) = &bucket_heal_scope {
|
||||
scope.check()?;
|
||||
}
|
||||
@@ -971,12 +979,25 @@ impl SetDisks {
|
||||
result.parity_blocks = result.disk_count - read_quorum as usize;
|
||||
result.data_blocks = read_quorum as usize;
|
||||
|
||||
let ((mut online_disks, quorum_mod_time, quorum_etag), disk_len) = {
|
||||
let ((quorum_disks, quorum_mod_time, quorum_etag), disk_len) = {
|
||||
let disks = self.disks.read().await;
|
||||
let disk_len = disks.len();
|
||||
(Self::list_online_disks(&disks, &parts_metadata, &errs, read_quorum as usize), disk_len)
|
||||
};
|
||||
|
||||
// A deep heal must inspect every reachable member, not only the
|
||||
// quorum that agreed on the canonical metadata. A returning disk
|
||||
// can legitimately carry an older xl.meta (or an explicit-version
|
||||
// absence) while still being the exact stale target that needs
|
||||
// repair. Restricting the repair set to quorum members silently
|
||||
// turns that stale member into `verified_healthy`/`unchanged`
|
||||
// (backlog#2612, #2596). Normal scans retain the quorum fast path.
|
||||
let mut online_disks = if opts.scan_mode == HealScanMode::Deep {
|
||||
self.get_disks_internal().await
|
||||
} else {
|
||||
quorum_disks
|
||||
};
|
||||
|
||||
trace!(
|
||||
event = EVENT_SET_DISK_HEAL,
|
||||
component = LOG_COMPONENT_ECSTORE,
|
||||
@@ -1449,7 +1470,12 @@ impl SetDisks {
|
||||
})
|
||||
.transpose()
|
||||
.map_err(DiskError::from)?;
|
||||
let till_offset = erasure.shard_file_offset(0, part.size, part.size);
|
||||
// This reader covers the whole encoded shard. A range
|
||||
// offset can exclude the final encoded byte when the
|
||||
// logical part ends inside a data block, allowing a
|
||||
// one-byte tail truncation to evade deep Heal.
|
||||
let till_offset = usize::try_from(erasure.shard_file_size(part.size as i64))
|
||||
.map_err(|_| DiskError::FileCorrupt)?;
|
||||
let use_mmap_read = object_mmap_read_enabled();
|
||||
|
||||
let mut readers = Vec::with_capacity(latest_disks.len());
|
||||
@@ -1887,13 +1913,26 @@ impl SetDisks {
|
||||
.zip(&errs)
|
||||
.any(|(info, error)| error.is_none() && info.is_canonical_delete_marker())
|
||||
{
|
||||
let cleanup = match Self::retired_marker_candidate(&parts_metadata, &errs) {
|
||||
Ok(marker) => {
|
||||
self.remove_retired_marker(retirement, bucket, object, marker, opts, &disks)
|
||||
.await
|
||||
}
|
||||
Err(error) => Err(error),
|
||||
};
|
||||
// A current-generation delete marker can be left behind on
|
||||
// an inaccessible member after an acknowledged S3 DELETE.
|
||||
// That case has no bucket-retirement record, so evaluate its
|
||||
// stricter same-incarnation proof before the retired-marker
|
||||
// path. The proof requires an exact marker identity, at
|
||||
// least one replica already absent, and every target online.
|
||||
let cleanup =
|
||||
match Self::same_incarnation_marker_candidate(&parts_metadata, &errs, retirement.current_incarnation) {
|
||||
Ok(marker) => {
|
||||
self.remove_current_marker(retirement, bucket, object, marker, opts, &disks)
|
||||
.await
|
||||
}
|
||||
Err(_) => match Self::retired_marker_candidate(&parts_metadata, &errs) {
|
||||
Ok(marker) => {
|
||||
self.remove_retired_marker(retirement, bucket, object, marker, opts, &disks)
|
||||
.await
|
||||
}
|
||||
Err(error) => Err(error),
|
||||
},
|
||||
};
|
||||
let absent = cleanup.is_ok();
|
||||
let item = self
|
||||
.dangling_heal_result(FileInfo::default(), &errs, bucket, object, version_id, absent)
|
||||
@@ -2007,6 +2046,41 @@ impl SetDisks {
|
||||
candidate.ok_or_else(|| DiskError::retired_marker_deferred("no exact marker candidate"))
|
||||
}
|
||||
|
||||
fn same_incarnation_marker_candidate(
|
||||
metadata: &[FileInfo],
|
||||
errors: &[Option<DiskError>],
|
||||
current_incarnation: Option<Uuid>,
|
||||
) -> disk::error::Result<rustfs_filemeta::MetaDeleteMarker> {
|
||||
let defer = DiskError::retired_marker_deferred;
|
||||
let Some(current) = current_incarnation.filter(|id| !id.is_nil()) else {
|
||||
return Err(defer("current bucket incarnation is unavailable"));
|
||||
};
|
||||
let mut candidate = None;
|
||||
let mut missing_replica = false;
|
||||
for (info, error) in metadata.iter().zip(errors) {
|
||||
match error {
|
||||
Some(DiskError::FileNotFound | DiskError::FileVersionNotFound) => {
|
||||
missing_replica = true;
|
||||
continue;
|
||||
}
|
||||
Some(_) => return Err(defer("not every replica is readable")),
|
||||
None => {}
|
||||
}
|
||||
if !info.is_canonical_delete_marker() || info.delete_marker_incarnation() != Some(current) {
|
||||
return Err(defer("marker is not a canonical current-generation marker"));
|
||||
}
|
||||
let marker = rustfs_filemeta::MetaDeleteMarker::from(info.clone());
|
||||
if candidate.as_ref().is_some_and(|previous| previous != &marker) {
|
||||
return Err(defer("surviving marker identities conflict"));
|
||||
}
|
||||
candidate = Some(marker);
|
||||
}
|
||||
if !missing_replica {
|
||||
return Err(defer("marker is present on every replica"));
|
||||
}
|
||||
candidate.ok_or_else(|| defer("no exact current-generation marker candidate"))
|
||||
}
|
||||
|
||||
async fn remove_retired_marker(
|
||||
&self,
|
||||
context: &MarkerRetirementContext<'_>,
|
||||
@@ -2045,6 +2119,72 @@ impl SetDisks {
|
||||
if context.lifecycle_guard.is_lock_lost() {
|
||||
return Err(defer("bucket lifecycle fence was lost"));
|
||||
}
|
||||
let result = self
|
||||
.remove_marker_exact_under_fence(bucket, object, marker, version, disks)
|
||||
.await;
|
||||
if context.lifecycle_guard.is_lock_lost() {
|
||||
return Err(defer("bucket lifecycle fence was lost"));
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
async fn remove_current_marker(
|
||||
&self,
|
||||
context: &MarkerRetirementContext<'_>,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
marker: rustfs_filemeta::MetaDeleteMarker,
|
||||
opts: &HealOpts,
|
||||
disks: &[Option<DiskStore>],
|
||||
) -> disk::error::Result<()> {
|
||||
let defer = DiskError::retired_marker_deferred;
|
||||
if opts.dry_run || !opts.remove {
|
||||
return Err(defer("cleanup requires remove=true and dry_run=false"));
|
||||
}
|
||||
if disks.len() != self.set_drive_count || disks.iter().any(Option::is_none) {
|
||||
return Err(defer("every target disk must be online"));
|
||||
}
|
||||
let Some(current) = context.current_incarnation.filter(|id| !id.is_nil()) else {
|
||||
return Err(defer("current bucket incarnation is unavailable"));
|
||||
};
|
||||
if context.lifecycle_guard.is_lock_lost() {
|
||||
return Err(defer("bucket lifecycle fence was lost"));
|
||||
}
|
||||
let incarnation = marker.into_fileinfo(bucket, object, false)?.delete_marker_incarnation();
|
||||
if incarnation != Some(current) {
|
||||
return Err(defer("marker is not from the current bucket incarnation"));
|
||||
}
|
||||
let Some(version) = marker.version_id.filter(|id| !id.is_nil()) else {
|
||||
return Err(defer("cleanup requires an explicit non-null version"));
|
||||
};
|
||||
let marker_identity = marker.stable_identity();
|
||||
let marker_bytes = marker.marshal_msg().map_err(DiskError::other)?;
|
||||
let Some(purge) = MrfDeleteMarkerPurge::new(current, current, marker_identity, marker_bytes) else {
|
||||
return Err(defer("marker purge proof payload is invalid"));
|
||||
};
|
||||
if !self.has_marker_purge_receipt(bucket, object, version, &purge).await {
|
||||
return Err(defer("same-incarnation marker lacks an acknowledged delete receipt"));
|
||||
}
|
||||
let result = self
|
||||
.remove_marker_exact_under_fence(bucket, object, marker, version, disks)
|
||||
.await;
|
||||
if context.lifecycle_guard.is_lock_lost() {
|
||||
return Err(defer("bucket lifecycle fence was lost"));
|
||||
}
|
||||
if result.is_ok() {
|
||||
self.consume_marker_purge_receipt(bucket, object, version, &purge).await;
|
||||
}
|
||||
result
|
||||
}
|
||||
|
||||
async fn remove_marker_exact_under_fence(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
marker: rustfs_filemeta::MetaDeleteMarker,
|
||||
version: Uuid,
|
||||
disks: &[Option<DiskStore>],
|
||||
) -> disk::error::Result<()> {
|
||||
let request = FileInfo {
|
||||
volume: bucket.to_owned(),
|
||||
name: object.to_owned(),
|
||||
@@ -2085,13 +2225,14 @@ impl SetDisks {
|
||||
let same_targets = selected.len() == disks.len() && selected.iter().zip(disks).all(|(current, original)| {
|
||||
matches!((current, original), (Some(current), Some(original)) if std::sync::Arc::ptr_eq(current, original))
|
||||
});
|
||||
if context.lifecycle_guard.is_lock_lost()
|
||||
|| !same_targets
|
||||
if !same_targets
|
||||
|| !errors
|
||||
.iter()
|
||||
.all(|error| matches!(error, Some(DiskError::FileNotFound | DiskError::FileVersionNotFound)))
|
||||
{
|
||||
return Err(defer("complete marker absence could not be verified under the original fence"));
|
||||
return Err(DiskError::retired_marker_deferred(
|
||||
"complete marker absence could not be verified under the original target set",
|
||||
));
|
||||
}
|
||||
self.invalidate_get_object_metadata_cache(bucket, object).await;
|
||||
Ok(())
|
||||
@@ -3012,8 +3153,18 @@ impl crate::storage_api_contracts::heal::HealOperations for SetDisks {
|
||||
version_id: &str,
|
||||
opts: &HealOpts,
|
||||
) -> Result<(HealResultItem, Option<Error>)> {
|
||||
self.heal_object_with_absence(bucket, object, version_id, opts, &mut None)
|
||||
.await
|
||||
self.heal_object_with_absence(
|
||||
bucket,
|
||||
object,
|
||||
version_id,
|
||||
opts,
|
||||
&mut None,
|
||||
AbsenceProofRequest {
|
||||
retirement: None,
|
||||
allow_unversioned: false,
|
||||
},
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
#[tracing::instrument(skip(self))]
|
||||
@@ -3098,8 +3249,9 @@ impl SetDisks {
|
||||
version_id: &str,
|
||||
opts: &HealOpts,
|
||||
absence: &mut Option<HealedObjectAbsence>,
|
||||
proof: AbsenceProofRequest<'_>,
|
||||
) -> Result<(HealResultItem, Option<Error>)> {
|
||||
self.heal_object_with_retirement(bucket, object, version_id, opts, absence, None)
|
||||
self.heal_object_with_retirement(bucket, object, version_id, opts, absence, proof)
|
||||
.await
|
||||
}
|
||||
|
||||
@@ -3110,7 +3262,7 @@ impl SetDisks {
|
||||
version_id: &str,
|
||||
opts: &HealOpts,
|
||||
absence: &mut Option<HealedObjectAbsence>,
|
||||
retirement: Option<&MarkerRetirementContext<'_>>,
|
||||
proof: AbsenceProofRequest<'_>,
|
||||
) -> Result<(HealResultItem, Option<Error>)> {
|
||||
*absence = None;
|
||||
let _write_lock_guard = if !opts.no_lock {
|
||||
@@ -3176,7 +3328,7 @@ impl SetDisks {
|
||||
.await;
|
||||
// Check the lease after the final await before publishing the proof.
|
||||
if !opts.dry_run
|
||||
&& !version_id.is_empty()
|
||||
&& (proof.allow_unversioned || !version_id.is_empty())
|
||||
&& !disks.is_empty()
|
||||
&& disks.iter().all(Option::is_some)
|
||||
&& errs
|
||||
@@ -3205,7 +3357,7 @@ impl SetDisks {
|
||||
ExplicitVersionHeal {
|
||||
opts: &inner_opts,
|
||||
allow_regeneration: true,
|
||||
retirement,
|
||||
retirement: proof.retirement,
|
||||
},
|
||||
absence,
|
||||
)
|
||||
@@ -3225,7 +3377,7 @@ impl SetDisks {
|
||||
ExplicitVersionHeal {
|
||||
opts: &inner_opts,
|
||||
allow_regeneration: true,
|
||||
retirement,
|
||||
retirement: proof.retirement,
|
||||
},
|
||||
absence,
|
||||
)
|
||||
@@ -3277,7 +3429,7 @@ impl SetDisks {
|
||||
#[cfg(test)]
|
||||
mod heal_result_report_tests {
|
||||
use super::{
|
||||
DanglingCheckPartsFailure, DanglingDeleteFailure, DanglingDeleteSafety, ReadRepairCommitFingerprint,
|
||||
AbsenceProofRequest, DanglingCheckPartsFailure, DanglingDeleteFailure, DanglingDeleteSafety, ReadRepairCommitFingerprint,
|
||||
ReadRepairPauseScope, SetDisks, heal_writer_error_summary,
|
||||
};
|
||||
use super::{HEAL_RENAME_INCOMPLETE, HealRenameFailureScope, HealWriterFailureScope};
|
||||
@@ -3288,7 +3440,9 @@ mod heal_result_report_tests {
|
||||
use crate::disk::{DiskAPI as _, DiskOption, DiskStore, RUSTFS_META_TMP_BUCKET, ReadOptions, STORAGE_FORMAT_FILE, new_disk};
|
||||
use crate::error::Error;
|
||||
use crate::object_api::{ObjectOptions, PutObjReader};
|
||||
use crate::set_disk::ops::object::hermetic_set_disks_support::hermetic_set_disks_isolated;
|
||||
use crate::set_disk::ops::object::hermetic_set_disks_support::{
|
||||
hermetic_set_disks_for_pool_with_default_parity_isolated, hermetic_set_disks_isolated,
|
||||
};
|
||||
use crate::storage_api_contracts::bucket::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions};
|
||||
use crate::storage_api_contracts::heal::HealOperations as _;
|
||||
use crate::storage_api_contracts::object::{ObjectIO as _, ObjectOperations as _};
|
||||
@@ -4069,6 +4223,634 @@ mod heal_result_report_tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn deep_heal_rebuilds_a_stale_current_delete_marker_metadata_replica() {
|
||||
let (_temp_dirs, disks, set) = hermetic_set_disks_isolated(4).await;
|
||||
let bucket = "deep-heal-stale-current-marker";
|
||||
let object = "object.bin";
|
||||
for disk in &disks {
|
||||
disk.make_volume(bucket).await.expect("test bucket should be created");
|
||||
}
|
||||
|
||||
let versioned = ObjectOptions {
|
||||
no_lock: true,
|
||||
versioned: true,
|
||||
..Default::default()
|
||||
};
|
||||
let original = set
|
||||
.put_object(bucket, object, &mut PutObjReader::from_vec(vec![0x41; 1024]), &versioned)
|
||||
.await
|
||||
.expect("versioned source object should be written");
|
||||
let stale_metadata = disks[0]
|
||||
.read_version("", bucket, object, "", &ReadOptions::default())
|
||||
.await
|
||||
.expect("pre-marker metadata should be readable");
|
||||
|
||||
let marker = set
|
||||
.delete_object(bucket, object, versioned)
|
||||
.await
|
||||
.expect("versioned delete should create a current marker");
|
||||
let marker_version = marker.version_id.expect("delete marker should have a version id");
|
||||
|
||||
// Simulate a member that was offline while the acknowledged marker was
|
||||
// committed on the other members: its physical root still contains the
|
||||
// pre-delete metadata and the original data version.
|
||||
let mut stale_metadata = stale_metadata;
|
||||
stale_metadata.fresh = true;
|
||||
disks[0]
|
||||
.write_metadata("", bucket, object, stale_metadata)
|
||||
.await
|
||||
.expect("stale member metadata should be restored for the fixture");
|
||||
assert_eq!(
|
||||
disks[0]
|
||||
.read_version("", bucket, object, &marker_version.to_string(), &ReadOptions::default())
|
||||
.await
|
||||
.expect_err("stale member must not contain the current marker")
|
||||
.to_string(),
|
||||
"file version not found"
|
||||
);
|
||||
|
||||
let (result, error) = set
|
||||
.heal_object(
|
||||
bucket,
|
||||
object,
|
||||
&marker_version.to_string(),
|
||||
&HealOpts {
|
||||
no_lock: true,
|
||||
scan_mode: HealScanMode::Deep,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("deep heal should report the stale marker result");
|
||||
|
||||
assert!(error.is_none(), "deep heal should repair stale marker metadata: {error:?}");
|
||||
assert_eq!(result.drives_healed(), Some(1));
|
||||
let repaired = disks[0]
|
||||
.read_version("", bucket, object, &marker_version.to_string(), &ReadOptions::default())
|
||||
.await
|
||||
.expect("rejoined member should contain the marker");
|
||||
assert!(repaired.deleted, "repaired metadata must remain a delete marker");
|
||||
assert_eq!(repaired.version_id, Some(marker_version));
|
||||
assert!(original.version_id.is_some(), "source object should have a version id");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn deep_heal_rebuilds_near_tail_truncated_part() {
|
||||
let (temp_dirs, disks, set) = hermetic_set_disks_isolated(4).await;
|
||||
let bucket = "deep-heal-near-tail-truncation";
|
||||
let object = "object.bin";
|
||||
for disk in &disks {
|
||||
disk.make_volume(bucket).await.expect("bucket volume should be created");
|
||||
}
|
||||
|
||||
let expected_payload = vec![0x6d; 1024 * 1024];
|
||||
set.put_object(
|
||||
bucket,
|
||||
object,
|
||||
&mut PutObjReader::from_vec(expected_payload.clone()),
|
||||
&ObjectOptions {
|
||||
no_lock: true,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("source object should be written before shard truncation");
|
||||
let source = disks[2]
|
||||
.read_version("", bucket, object, "", &ReadOptions::default())
|
||||
.await
|
||||
.expect("source metadata should be readable");
|
||||
let data_dir = source.data_dir.expect("non-inline source should have a data directory");
|
||||
let truncated_part = temp_dirs[1]
|
||||
.path()
|
||||
.join(bucket)
|
||||
.join(object)
|
||||
.join(data_dir.to_string())
|
||||
.join("part.1");
|
||||
let original_len = tokio::fs::metadata(&truncated_part)
|
||||
.await
|
||||
.expect("target shard should exist")
|
||||
.len();
|
||||
assert!(original_len > 1, "test shard must be large enough to truncate");
|
||||
let file = tokio::fs::OpenOptions::new()
|
||||
.write(true)
|
||||
.open(&truncated_part)
|
||||
.await
|
||||
.expect("target shard should be writable");
|
||||
file.set_len(original_len - 1)
|
||||
.await
|
||||
.expect("target shard should be truncated");
|
||||
|
||||
let mut reader = set
|
||||
.get_object_reader(bucket, object, None, Default::default(), &ObjectOptions::default())
|
||||
.await
|
||||
.expect("GET should remain readable after a one-byte shard truncation");
|
||||
let mut read_back = Vec::new();
|
||||
tokio::io::copy(&mut reader, &mut read_back)
|
||||
.await
|
||||
.expect("GET should reconstruct the truncated shard through EC");
|
||||
assert_eq!(read_back, expected_payload, "EC GET must preserve the object bytes");
|
||||
|
||||
let (result, error) = set
|
||||
.heal_object(
|
||||
bucket,
|
||||
object,
|
||||
"",
|
||||
&HealOpts {
|
||||
no_lock: true,
|
||||
scan_mode: HealScanMode::Deep,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("deep heal should finish after a one-byte tail truncation");
|
||||
|
||||
assert!(error.is_none(), "deep heal should recover the truncated shard: {error:?}");
|
||||
assert_eq!(result.drives_healed(), Some(1));
|
||||
assert_eq!(result.before.drives[1].state, DriveState::Corrupt.to_string());
|
||||
assert_eq!(
|
||||
tokio::fs::metadata(&truncated_part)
|
||||
.await
|
||||
.expect("repaired shard should exist")
|
||||
.len(),
|
||||
original_len,
|
||||
"deep heal must restore the complete shard length"
|
||||
);
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
struct FailedIntegrityReadTransport {
|
||||
error: DiskError,
|
||||
reads: std::sync::atomic::AtomicUsize,
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
impl crate::cluster::rpc::internode_data_transport::InternodeDataTransport for FailedIntegrityReadTransport {
|
||||
async fn open_read(
|
||||
&self,
|
||||
_request: crate::cluster::rpc::internode_data_transport::ReadStreamRequest,
|
||||
) -> crate::disk::error::Result<crate::disk::FileReader> {
|
||||
self.reads.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
|
||||
let error = match &self.error {
|
||||
DiskError::FileCorrupt => DiskError::from(rustfs_rio::new_test_remote_file_corrupt_http_io_error()),
|
||||
DiskError::FileNotFound => DiskError::from(rustfs_rio::new_test_remote_file_not_found_http_io_error()),
|
||||
DiskError::VolumeNotFound => DiskError::from(rustfs_rio::new_test_remote_volume_not_found_http_io_error()),
|
||||
error => error.clone(),
|
||||
};
|
||||
Err(error)
|
||||
}
|
||||
|
||||
async fn open_write(
|
||||
&self,
|
||||
_request: crate::cluster::rpc::internode_data_transport::WriteStreamRequest,
|
||||
) -> crate::disk::error::Result<crate::disk::FileWriter> {
|
||||
panic!("an integrity scan must not write remote data");
|
||||
}
|
||||
|
||||
async fn open_walk_dir(
|
||||
&self,
|
||||
_request: crate::cluster::rpc::internode_data_transport::WalkDirStreamRequest,
|
||||
) -> crate::disk::error::Result<crate::disk::FileReader> {
|
||||
panic!("an object integrity scan must not walk remote directories");
|
||||
}
|
||||
|
||||
fn name(&self) -> &'static str {
|
||||
"failed-integrity-read"
|
||||
}
|
||||
|
||||
fn capabilities(&self) -> crate::cluster::rpc::internode_data_transport::InternodeDataTransportCapabilities {
|
||||
crate::cluster::rpc::internode_data_transport::InternodeDataTransportCapabilities::tcp_http()
|
||||
}
|
||||
}
|
||||
|
||||
async fn assert_remote_integrity_read_failure(expected_error: DiskError) {
|
||||
use crate::disk::{CHECK_PART_FILE_CORRUPT, CHECK_PART_FILE_NOT_FOUND, CHECK_PART_SUCCESS, CHECK_PART_UNKNOWN};
|
||||
use crate::object_api::ShardIntegrityWriteMode;
|
||||
|
||||
let (dirs, disks, set) = hermetic_set_disks_for_pool_with_default_parity_isolated(4, 0, 2).await;
|
||||
let bucket = "remote-integrity-read-failure";
|
||||
let object = "object.bin";
|
||||
for disk in &disks {
|
||||
disk.make_volume(bucket).await.expect("create fixture bucket");
|
||||
}
|
||||
set.put_object(
|
||||
bucket,
|
||||
object,
|
||||
&mut PutObjReader::from_vec(vec![0x6d; 1024 * 1024 + 123]),
|
||||
&ObjectOptions {
|
||||
shard_integrity_write_mode: Some(ShardIntegrityWriteMode::Protected),
|
||||
no_lock: true,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("write protected external fixture");
|
||||
|
||||
let mut files = Vec::new();
|
||||
let mut snapshots = Vec::new();
|
||||
for (dir, disk) in dirs.iter().zip(&disks) {
|
||||
let file = disk
|
||||
.read_version("", bucket, object, "", &ReadOptions::default())
|
||||
.await
|
||||
.expect("read fixture metadata");
|
||||
assert!(file.data.is_none(), "the remote test must open an external shard");
|
||||
let object_dir = dir.path().join(bucket).join(object);
|
||||
let part_path = object_dir
|
||||
.join(file.data_dir.expect("external shard directory").to_string())
|
||||
.join("part.1");
|
||||
for path in [part_path, object_dir.join("xl.meta")] {
|
||||
let bytes = tokio::fs::read(&path).await.expect("snapshot fixture before scan");
|
||||
snapshots.push((path, bytes));
|
||||
}
|
||||
files.push(file);
|
||||
}
|
||||
let transport = Arc::new(FailedIntegrityReadTransport {
|
||||
error: expected_error.clone(),
|
||||
reads: std::sync::atomic::AtomicUsize::new(0),
|
||||
});
|
||||
let endpoint = Endpoint {
|
||||
url: url::Url::parse("http://remote-integrity.invalid:9000/data/disk0").expect("fixture endpoint"),
|
||||
is_local: false,
|
||||
pool_idx: 0,
|
||||
set_idx: 0,
|
||||
disk_idx: 0,
|
||||
};
|
||||
let remote = crate::cluster::rpc::RemoteDisk::new(
|
||||
&endpoint,
|
||||
&DiskOption {
|
||||
cleanup: false,
|
||||
health_check: false,
|
||||
},
|
||||
transport.clone(),
|
||||
)
|
||||
.await
|
||||
.expect("construct remote disk with failing data transport");
|
||||
let mut verification_disks: Vec<_> = disks.iter().cloned().map(Some).collect();
|
||||
verification_disks[0] = Some(Arc::new(crate::disk::Disk::Remote(Box::new(remote))));
|
||||
let mut statuses = [(0, vec![CHECK_PART_UNKNOWN; disks.len()])].into_iter().collect();
|
||||
let result = crate::io_support::shard_integrity::verify_deep_parts(
|
||||
&files,
|
||||
&verification_disks,
|
||||
&files[0],
|
||||
bucket,
|
||||
object,
|
||||
&mut statuses,
|
||||
)
|
||||
.await;
|
||||
assert!(transport.reads.load(std::sync::atomic::Ordering::Relaxed) > 0);
|
||||
if matches!(expected_error, DiskError::FileCorrupt | DiskError::FileNotFound) {
|
||||
result.expect("typed remote payload damage must identify a repairable shard");
|
||||
let expected_status = if expected_error == DiskError::FileCorrupt {
|
||||
CHECK_PART_FILE_CORRUPT
|
||||
} else {
|
||||
CHECK_PART_FILE_NOT_FOUND
|
||||
};
|
||||
assert_eq!(statuses[&0][0], expected_status);
|
||||
assert!(statuses[&0][1..].iter().all(|status| *status == CHECK_PART_SUCCESS));
|
||||
let (heal, metadata, reason) = super::should_heal_object_on_disk(&None, &[statuses[&0][0]], &files[0], &files[0]);
|
||||
assert!(heal, "remote payload damage must schedule repair: {expected_error:?}");
|
||||
assert!(!metadata);
|
||||
let expected_reason = if expected_error == DiskError::FileCorrupt {
|
||||
DiskError::FileCorrupt
|
||||
} else {
|
||||
DiskError::PartMissingOrCorrupt
|
||||
};
|
||||
assert_eq!(reason, Some(expected_reason));
|
||||
} else {
|
||||
let error = result.expect_err("unavailable payload storage must abort health verification");
|
||||
if expected_error.is_internode_http_status(500) {
|
||||
assert!(error.is_internode_http_status(500), "preserve the original transport error: {error:?}");
|
||||
} else {
|
||||
assert_eq!(error, expected_error, "preserve the original storage failure");
|
||||
}
|
||||
assert_eq!(statuses[&0][0], CHECK_PART_UNKNOWN, "failed verification cannot certify the shard");
|
||||
}
|
||||
for (path, before) in snapshots {
|
||||
assert_eq!(tokio::fs::read(path).await.expect("read fixture after scan"), before);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn deep_scan_remote_payload_damage_schedules_repair() {
|
||||
for error in [DiskError::FileCorrupt, DiskError::FileNotFound] {
|
||||
assert_remote_integrity_read_failure(error).await;
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn deep_scan_remote_storage_failure_cannot_certify_healthy() {
|
||||
for error in [
|
||||
DiskError::from(rustfs_rio::new_test_internode_http_io_error(
|
||||
rustfs_rio::InternodeHttpErrorKind::HttpStatus(reqwest::StatusCode::INTERNAL_SERVER_ERROR),
|
||||
)),
|
||||
DiskError::DiskNotFound,
|
||||
DiskError::VolumeNotFound,
|
||||
DiskError::FileAccessDenied,
|
||||
] {
|
||||
assert_remote_integrity_read_failure(error).await;
|
||||
}
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
#[tokio::test]
|
||||
async fn deep_heal_legacy_verification_error_cannot_certify_healthy() {
|
||||
use crate::object_api::ShardIntegrityWriteMode;
|
||||
|
||||
let (dirs, disks, set) = hermetic_set_disks_isolated(4).await;
|
||||
let bucket = "legacy-deep-scan-verification-error";
|
||||
let object = "object.bin";
|
||||
for disk in &disks {
|
||||
disk.make_volume(bucket).await.expect("create legacy fixture bucket");
|
||||
}
|
||||
set.put_object(
|
||||
bucket,
|
||||
object,
|
||||
&mut PutObjReader::from_vec(vec![0x6d; 1024 * 1024 + 123]),
|
||||
&ObjectOptions {
|
||||
shard_integrity_write_mode: Some(ShardIntegrityWriteMode::Legacy),
|
||||
no_lock: true,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("write external legacy fixture");
|
||||
|
||||
let mut files = Vec::new();
|
||||
let mut part_paths = Vec::new();
|
||||
let mut preserved = Vec::new();
|
||||
for (index, (dir, disk)) in dirs.iter().zip(&disks).enumerate() {
|
||||
let file = disk
|
||||
.read_version("", bucket, object, "", &ReadOptions::default())
|
||||
.await
|
||||
.expect("read legacy fixture metadata");
|
||||
assert!(file.data.is_none(), "legacy verification must open the physical shard");
|
||||
assert!(file.parts.iter().all(|part| part.integrity.is_none()));
|
||||
let object_dir = dir.path().join(bucket).join(object);
|
||||
let part = object_dir
|
||||
.join(file.data_dir.expect("external legacy shard directory").to_string())
|
||||
.join("part.1");
|
||||
let metadata = object_dir.join("xl.meta");
|
||||
let bytes = tokio::fs::read(&metadata).await.expect("snapshot legacy metadata");
|
||||
preserved.push((metadata, bytes));
|
||||
if index != 0 {
|
||||
let bytes = tokio::fs::read(&part).await.expect("snapshot healthy legacy shard");
|
||||
preserved.push((part.clone(), bytes));
|
||||
}
|
||||
part_paths.push(part);
|
||||
files.push(file);
|
||||
}
|
||||
let broken_part = &part_paths[0];
|
||||
tokio::fs::remove_file(broken_part)
|
||||
.await
|
||||
.expect("remove fixture shard before symlink fault");
|
||||
std::os::unix::fs::symlink("part.1", broken_part).expect("create self-referential shard symlink");
|
||||
let open_error = tokio::fs::File::open(broken_part)
|
||||
.await
|
||||
.expect_err("self-referential symlink must fail to open");
|
||||
assert_eq!(open_error.raw_os_error(), Some(libc::ELOOP));
|
||||
|
||||
let verification_error = disks[0]
|
||||
.verify_file(bucket, object, &files[0])
|
||||
.await
|
||||
.expect_err("legacy disk verifier must reject the symlink path");
|
||||
assert_eq!(verification_error, DiskError::InvalidPath);
|
||||
let error = set
|
||||
.heal_object(
|
||||
bucket,
|
||||
object,
|
||||
"",
|
||||
&HealOpts {
|
||||
no_lock: true,
|
||||
scan_mode: HealScanMode::Deep,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect_err("an unverified legacy part must fail the complete heal operation");
|
||||
assert_eq!(error, DiskError::InvalidPath, "complete heal must preserve the verifier's path error");
|
||||
assert_eq!(
|
||||
tokio::fs::read_link(broken_part)
|
||||
.await
|
||||
.expect("failed heal must preserve the symlink fault"),
|
||||
std::path::Path::new("part.1")
|
||||
);
|
||||
for (path, before) in preserved {
|
||||
assert_eq!(tokio::fs::read(path).await.expect("read unchanged legacy fixture"), before);
|
||||
}
|
||||
}
|
||||
|
||||
async fn assert_h2_truncated_shards(
|
||||
mode: crate::object_api::ShardIntegrityWriteMode,
|
||||
coding_indexes: &[usize],
|
||||
retained_len: fn(usize, usize) -> usize,
|
||||
) {
|
||||
use crate::object_api::ShardIntegrityWriteMode;
|
||||
|
||||
let (temp_dirs, disks, set) = hermetic_set_disks_for_pool_with_default_parity_isolated(16, 0, 4).await;
|
||||
let bucket = "deep-heal-h2-truncation";
|
||||
let object = "object.bin";
|
||||
for disk in &disks {
|
||||
disk.make_volume(bucket).await.expect("create bucket volume");
|
||||
}
|
||||
let expected_payload: Vec<_> = (0..5 * 1024 * 1024 + 123)
|
||||
.map(|index| u8::try_from((index / 4096 + index) % 251).expect("payload byte fits"))
|
||||
.collect();
|
||||
set.put_object(
|
||||
bucket,
|
||||
object,
|
||||
&mut PutObjReader::from_vec(expected_payload.clone()),
|
||||
&ObjectOptions {
|
||||
shard_integrity_write_mode: Some(mode),
|
||||
no_lock: true,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("commit all source shards");
|
||||
|
||||
let mut files = Vec::new();
|
||||
let mut paths = Vec::new();
|
||||
let mut original_shards = Vec::new();
|
||||
let mut original_metadata = Vec::new();
|
||||
let mut damaged = Vec::new();
|
||||
for (slot, disk) in disks.iter().enumerate() {
|
||||
let source = disk
|
||||
.read_version("", bucket, object, "", &ReadOptions::default())
|
||||
.await
|
||||
.expect("read source metadata");
|
||||
assert_eq!((source.erasure.data_blocks, source.erasure.parity_blocks), (12, 4));
|
||||
assert_eq!(source.parts[0].integrity.is_some(), mode == ShardIntegrityWriteMode::Protected);
|
||||
let directory = temp_dirs[slot].path().join(bucket).join(object);
|
||||
let path = directory
|
||||
.join(source.data_dir.expect("external shard data directory").to_string())
|
||||
.join("part.1");
|
||||
let original = tokio::fs::read(&path).await.expect("read original shard");
|
||||
original_metadata.push(tokio::fs::read(directory.join("xl.meta")).await.expect("read xl.meta"));
|
||||
if coding_indexes.contains(&source.erasure.index) {
|
||||
let erasure = crate::erasure::coding::Erasure::try_new_with_options(
|
||||
12,
|
||||
4,
|
||||
source.erasure.block_size,
|
||||
source.uses_legacy_checksum,
|
||||
)
|
||||
.expect("valid erasure layout");
|
||||
let stripe_size = erasure.shard_size() + source.erasure.get_checksum_info(1).algorithm.size();
|
||||
let keep = retained_len(original.len(), stripe_size);
|
||||
assert!(keep < original.len(), "fixture must truncate the shard");
|
||||
tokio::fs::OpenOptions::new()
|
||||
.write(true)
|
||||
.open(&path)
|
||||
.await
|
||||
.expect("open target shard")
|
||||
.set_len(u64::try_from(keep).expect("shard length fits"))
|
||||
.await
|
||||
.expect("truncate target shard");
|
||||
damaged.push((slot, keep));
|
||||
}
|
||||
paths.push(path);
|
||||
original_shards.push(original);
|
||||
files.push(source);
|
||||
}
|
||||
assert_eq!(damaged.len(), coding_indexes.len());
|
||||
if mode == ShardIntegrityWriteMode::Protected {
|
||||
let mut statuses = [(0, vec![crate::disk::CHECK_PART_UNKNOWN; disks.len()])]
|
||||
.into_iter()
|
||||
.collect();
|
||||
let online_disks: Vec<_> = disks.iter().cloned().map(Some).collect();
|
||||
crate::io_support::shard_integrity::verify_deep_parts(
|
||||
&files,
|
||||
&online_disks,
|
||||
&files[0],
|
||||
bucket,
|
||||
object,
|
||||
&mut statuses,
|
||||
)
|
||||
.await
|
||||
.expect("verify protected shards directly");
|
||||
for (slot, status) in statuses[&0].iter().enumerate() {
|
||||
let expected = if damaged.iter().any(|(index, _)| *index == slot) {
|
||||
crate::disk::CHECK_PART_FILE_CORRUPT
|
||||
} else {
|
||||
crate::disk::CHECK_PART_SUCCESS
|
||||
};
|
||||
assert_eq!(*status, expected, "protected shard {slot}");
|
||||
}
|
||||
}
|
||||
|
||||
let opts = HealOpts {
|
||||
no_lock: true,
|
||||
scan_mode: HealScanMode::Deep,
|
||||
..Default::default()
|
||||
};
|
||||
let (dry_run, error) = set
|
||||
.heal_object(bucket, object, "", &HealOpts { dry_run: true, ..opts })
|
||||
.await
|
||||
.expect("dry-run should report corruption");
|
||||
assert!(error.is_none(), "dry-run error: {error:?}");
|
||||
assert_eq!(dry_run.drives_healed(), Some(0));
|
||||
for (slot, keep) in &damaged {
|
||||
assert_eq!(dry_run.after.drives[*slot].state, DriveState::Corrupt.to_string());
|
||||
assert_eq!(
|
||||
tokio::fs::read(&paths[*slot]).await.expect("dry-run preserved shard"),
|
||||
original_shards[*slot][..*keep]
|
||||
);
|
||||
}
|
||||
|
||||
let recoverable = damaged.len() <= 4;
|
||||
if recoverable {
|
||||
let mut reader = set
|
||||
.get_object_reader(bucket, object, None, Default::default(), &ObjectOptions::default())
|
||||
.await
|
||||
.expect("degraded GET should remain readable");
|
||||
let mut read_back = Vec::new();
|
||||
tokio::io::copy(&mut reader, &mut read_back)
|
||||
.await
|
||||
.expect("read complete degraded GET");
|
||||
assert_eq!(read_back, expected_payload);
|
||||
}
|
||||
let (result, error) = set
|
||||
.heal_object(bucket, object, "", &opts)
|
||||
.await
|
||||
.expect("heal should report outcome");
|
||||
if recoverable {
|
||||
assert!(error.is_none(), "heal error: {error:?}");
|
||||
assert_eq!(result.drives_healed(), Some(damaged.len()));
|
||||
for (slot, _) in &damaged {
|
||||
assert_eq!(result.before.drives[*slot].state, DriveState::Corrupt.to_string());
|
||||
}
|
||||
} else {
|
||||
assert_eq!(error, Some(DiskError::ErasureReadQuorum));
|
||||
assert_eq!(result.drives_healed(), Some(0));
|
||||
}
|
||||
for (slot, path) in paths.iter().enumerate() {
|
||||
let expected = if !recoverable && let Some((_, keep)) = damaged.iter().find(|(index, _)| *index == slot) {
|
||||
&original_shards[slot][..*keep]
|
||||
} else {
|
||||
original_shards[slot].as_slice()
|
||||
};
|
||||
assert_eq!(
|
||||
tokio::fs::read(path).await.expect("read shard after heal"),
|
||||
expected,
|
||||
"physical shard {slot}"
|
||||
);
|
||||
if !recoverable {
|
||||
let metadata_path = temp_dirs[slot].path().join(bucket).join(object).join("xl.meta");
|
||||
assert_eq!(tokio::fs::read(metadata_path).await.expect("preserved metadata"), original_metadata[slot]);
|
||||
}
|
||||
}
|
||||
if recoverable {
|
||||
let mut reader = set
|
||||
.get_object_reader(bucket, object, None, Default::default(), &ObjectOptions::default())
|
||||
.await
|
||||
.expect("GET after repair");
|
||||
let mut read_back = Vec::new();
|
||||
tokio::io::copy(&mut reader, &mut read_back)
|
||||
.await
|
||||
.expect("read repaired object");
|
||||
assert_eq!(read_back, expected_payload);
|
||||
let (healthy, error) = set.heal_object(bucket, object, "", &opts).await.expect("second deep scan");
|
||||
assert!(error.is_none());
|
||||
assert_eq!(healthy.drives_healed(), Some(0), "repair must converge");
|
||||
assert_eq!(healthy.integrity_verified, mode == ShardIntegrityWriteMode::Protected);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn deep_heal_rebuilds_h2_near_tail_truncated_part() {
|
||||
use crate::object_api::ShardIntegrityWriteMode::{Legacy, Protected};
|
||||
for mode in [Legacy, Protected] {
|
||||
for coding_index in [1, 13] {
|
||||
assert_h2_truncated_shards(mode, &[coding_index], |length, _| length - 1).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn deep_heal_rebuilds_h2_half_truncated_part() {
|
||||
use crate::object_api::ShardIntegrityWriteMode::{Legacy, Protected};
|
||||
for mode in [Legacy, Protected] {
|
||||
for coding_index in [1, 13] {
|
||||
assert_h2_truncated_shards(mode, &[coding_index], |length, _| length / 2).await;
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn deep_heal_rebuilds_h2_stripe_boundary_truncated_parts_at_read_quorum() {
|
||||
use crate::object_api::ShardIntegrityWriteMode::{Legacy, Protected};
|
||||
for mode in [Legacy, Protected] {
|
||||
assert_h2_truncated_shards(mode, &[1, 2, 13, 16], |_, stripe| stripe).await;
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn deep_heal_rejects_h2_truncated_parts_below_read_quorum() {
|
||||
use crate::object_api::ShardIntegrityWriteMode::{Legacy, Protected};
|
||||
for mode in [Legacy, Protected] {
|
||||
assert_h2_truncated_shards(mode, &[1, 2, 3, 13, 16], |length, _| length / 2).await;
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn deep_heal_inline_bitrot_rebuilds_physical_data_and_parity_shards() {
|
||||
use crate::storage_api_contracts::range::HTTPRangeSpec;
|
||||
@@ -5385,7 +6167,17 @@ mod heal_result_report_tests {
|
||||
let failure = DanglingDeleteFailure::install(bucket, object, 0, DiskError::FaultyDisk);
|
||||
let mut proof = None;
|
||||
let (_, error) = set
|
||||
.heal_object_with_absence(bucket, object, &version.to_string(), &opts, &mut proof)
|
||||
.heal_object_with_absence(
|
||||
bucket,
|
||||
object,
|
||||
&version.to_string(),
|
||||
&opts,
|
||||
&mut proof,
|
||||
AbsenceProofRequest {
|
||||
retirement: None,
|
||||
allow_unversioned: false,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("heal should return a per-object failure");
|
||||
assert!(
|
||||
@@ -5401,7 +6193,17 @@ mod heal_result_report_tests {
|
||||
);
|
||||
drop(failure);
|
||||
let (result, error) = set
|
||||
.heal_object_with_absence(bucket, object, &version.to_string(), &opts, &mut proof)
|
||||
.heal_object_with_absence(
|
||||
bucket,
|
||||
object,
|
||||
&version.to_string(),
|
||||
&opts,
|
||||
&mut proof,
|
||||
AbsenceProofRequest {
|
||||
retirement: None,
|
||||
allow_unversioned: false,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("retry should execute cleanup");
|
||||
assert!(error.is_none(), "retry should complete: {error:?}");
|
||||
@@ -5416,7 +6218,17 @@ mod heal_result_report_tests {
|
||||
.all(|drive| drive.state == DriveState::Missing.to_string())
|
||||
);
|
||||
let (_, _) = set
|
||||
.heal_object_with_absence(bucket, object, &version.to_string(), &opts, &mut proof)
|
||||
.heal_object_with_absence(
|
||||
bucket,
|
||||
object,
|
||||
&version.to_string(),
|
||||
&opts,
|
||||
&mut proof,
|
||||
AbsenceProofRequest {
|
||||
retirement: None,
|
||||
allow_unversioned: false,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("already absent replay should execute");
|
||||
assert!(!proof.expect("exact already-absent replay must remain provable").removed);
|
||||
|
||||
@@ -6715,7 +6715,7 @@ fn remote_version_state_writer_enabled() -> bool {
|
||||
}
|
||||
|
||||
fn remote_version_state_writer_fleet_proof() -> Option<RemoteVersionStateFleetProofToken> {
|
||||
transaction_fencing_fleet_proof(remote_version_state_writer_requested())
|
||||
crate::services::notification_sys::acquire_remote_version_state_writer_fleet_proof()
|
||||
}
|
||||
|
||||
fn remote_version_state_writer_requested() -> bool {
|
||||
@@ -7571,7 +7571,11 @@ impl SetDisks {
|
||||
let fi = &transported;
|
||||
let disks = self.disk_inventory().await;
|
||||
let namespace_owner = (!is_meta_bucketname(bucket)).then(|| self.ctx.begin_namespace_commit());
|
||||
let write_quorum = disks.len() / 2 + 1;
|
||||
// Quorum is a property of the configured set, not of the current
|
||||
// online snapshot. A decommission/offline refresh may temporarily
|
||||
// shorten the snapshot; deriving quorum from it would turn a
|
||||
// quorum-minus-one delete into an apparent success.
|
||||
let write_quorum = self.set_drive_count / 2 + 1;
|
||||
let rollback_dir = Uuid::new_v4();
|
||||
|
||||
let mut futures = Vec::with_capacity(disks.len());
|
||||
@@ -7699,10 +7703,11 @@ impl SetDisks {
|
||||
drop(namespace_owner);
|
||||
if quorum_result.is_ok()
|
||||
&& errs.iter().any(Option::is_some)
|
||||
&& let Some(purge) = delete_marker_purge
|
||||
&& let Some(purge) = delete_marker_purge.as_ref()
|
||||
&& let Some(version) = fi.version_id.filter(|version| !version.is_nil())
|
||||
{
|
||||
let _ = self.persist_delete_marker_purge(bucket, object, version, purge).await;
|
||||
let _ = self.persist_marker_purge_receipt(bucket, object, version, purge).await;
|
||||
let _ = self.persist_delete_marker_purge(bucket, object, version, purge.clone()).await;
|
||||
}
|
||||
// An explicit purge can carry deleted=true for the existing marker.
|
||||
// It must not create a repair intent that could reintroduce that marker.
|
||||
|
||||
@@ -645,6 +645,17 @@ impl ECStore {
|
||||
.as_ref()
|
||||
.and_then(|info| info.created)
|
||||
.unwrap_or(OffsetDateTime::UNIX_EPOCH);
|
||||
// A sidecar left by an interrupted legacy migration is the
|
||||
// bucket's published incarnation; keep it rather than minting
|
||||
// one that would invalidate fences taken on the stored value.
|
||||
// A retired incarnation is residue of a deleted bucket and
|
||||
// must not come back.
|
||||
let store = metadata_sys::object_store_in(&self.ctx).await?;
|
||||
if let Some(stored) = crate::bucket::metadata::load_bucket_incarnation(store.clone(), bucket).await?
|
||||
&& !crate::bucket::retirement::is_retired(store, bucket, stored).await?
|
||||
{
|
||||
metadata.bucket_incarnation_id = stored;
|
||||
}
|
||||
} else if !metadata.bucket_incarnation_sidecar && !metadata.bucket_incarnation_id.is_nil() {
|
||||
return Err(Error::other(format!(
|
||||
"bucket incarnation sidecar is missing for new-format metadata: {bucket}"
|
||||
@@ -2881,6 +2892,57 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
/// rustfs/rustfs#8003: a force-create (site replication replay, admin
|
||||
/// import) against a bucket whose legacy migration stopped after the
|
||||
/// sidecar write must persist `.metadata.bin` under the stored incarnation
|
||||
/// rather than fail closed or replace the published identity.
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
#[serial]
|
||||
async fn issue_8003_force_create_adopts_existing_incarnation_sidecar() {
|
||||
let (disk_paths, ecstore) = setup_bucket_delete_test_env().await;
|
||||
let bucket = format!("bucket-8003-sidecar-{}", Uuid::new_v4().simple());
|
||||
|
||||
ecstore
|
||||
.make_bucket(&bucket, &MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("bucket should be created");
|
||||
let stored = metadata_sys::get_in(&ecstore.ctx, &bucket)
|
||||
.await
|
||||
.expect("metadata should load")
|
||||
.bucket_incarnation_id;
|
||||
assert!(!stored.is_nil());
|
||||
|
||||
let metadata_path = format!("{BUCKET_META_PREFIX}/{bucket}/{}", crate::bucket::metadata::BUCKET_METADATA_FILE);
|
||||
crate::config::com::delete_config(ecstore.clone(), &metadata_path)
|
||||
.await
|
||||
.expect("simulate the interrupted migration by removing only .metadata.bin");
|
||||
assert!(!any_disk_path_exists(&disk_paths, format!("{RUSTFS_META_BUCKET}/{metadata_path}")).await);
|
||||
metadata_sys::remove_bucket_metadata_in(&ecstore.ctx, &bucket)
|
||||
.await
|
||||
.expect("drop the cached copy so the next read hits disk");
|
||||
|
||||
ecstore
|
||||
.make_bucket(
|
||||
&bucket,
|
||||
&MakeBucketOptions {
|
||||
force_create: true,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("force create must repair the sidecar-only bucket");
|
||||
|
||||
let (repaired, persisted) = metadata_sys::get_config_from_disk_with_presence_in(&ecstore.ctx, &bucket)
|
||||
.await
|
||||
.expect("repaired metadata should load");
|
||||
assert!(persisted, "force create must persist .metadata.bin again");
|
||||
assert!(repaired.bucket_incarnation_sidecar);
|
||||
assert_eq!(
|
||||
repaired.bucket_incarnation_id, stored,
|
||||
"the repair must keep the incarnation the sidecar already published"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread")]
|
||||
#[serial]
|
||||
async fn make_bucket_seeds_new_bucket_durability_override() {
|
||||
|
||||
@@ -58,6 +58,26 @@ pub(crate) fn bucket_heal_scope(bucket: &str) -> Option<Arc<BucketHealScope>> {
|
||||
.flatten()
|
||||
}
|
||||
|
||||
/// Resolve only the two canonical bucket records; other internal paths carry no bucket authority.
|
||||
pub(super) fn bucket_metadata_owner(object: &str) -> Option<&str> {
|
||||
use crate::bucket::metadata::{BUCKET_INCARNATION_FILE, BUCKET_METADATA_FILE};
|
||||
let (prefix, rest) = object.split_once('/')?;
|
||||
let (bucket, file) = rest.split_once('/')?;
|
||||
(prefix == disk::BUCKET_META_PREFIX
|
||||
&& matches!(file, BUCKET_METADATA_FILE | BUCKET_INCARNATION_FILE)
|
||||
&& check_valid_bucket_name_strict(bucket).is_ok())
|
||||
.then_some(bucket)
|
||||
}
|
||||
|
||||
pub(crate) fn bucket_heal_scope_for_object(bucket: &str, object: &str) -> Option<Arc<BucketHealScope>> {
|
||||
let owner = if bucket == RUSTFS_META_BUCKET {
|
||||
bucket_metadata_owner(object)?
|
||||
} else {
|
||||
bucket
|
||||
};
|
||||
bucket_heal_scope(owner)
|
||||
}
|
||||
|
||||
/// Storage-owned proof for the exact version and every selected erasure location.
|
||||
/// This is an in-process result, never reconstructed from admin drive telemetry.
|
||||
#[derive(Debug)]
|
||||
@@ -296,6 +316,24 @@ impl ECStore {
|
||||
.await
|
||||
}
|
||||
|
||||
pub async fn heal_mrf_object_at_incarnation(
|
||||
self: &Arc<Self>,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
version_id: &str,
|
||||
expected: uuid::Uuid,
|
||||
opts: &HealOpts,
|
||||
) -> Result<HealObjectStorageResult> {
|
||||
let object = object.to_owned();
|
||||
let version_id = version_id.to_owned();
|
||||
self.run_bucket_heal_at_incarnation(bucket, expected, opts, |store, bucket, opts| async move {
|
||||
store
|
||||
.heal_object_with_authoritative_absence_proof(&bucket, &object, &version_id, &opts)
|
||||
.await
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
async fn acquire_heal_format_fence(
|
||||
&self,
|
||||
) -> Result<(
|
||||
@@ -404,6 +442,97 @@ impl ECStore {
|
||||
Ok(Arc::ptr_eq(&target_set, &pool.get_disks_by_key(POOL_META_NAME)))
|
||||
}
|
||||
|
||||
/// Restore the bucket configuration and incarnation before retiring a replacement marker.
|
||||
pub async fn heal_replacement_bucket_metadata(
|
||||
self: &Arc<Self>,
|
||||
bucket: &str,
|
||||
opts: &HealOpts,
|
||||
targets: &[String],
|
||||
) -> Result<()> {
|
||||
use crate::bucket::metadata::{BUCKET_INCARNATION_FILE, BUCKET_METADATA_FILE};
|
||||
use crate::bucket::metadata_sys::acquire_bucket_metadata_transaction_read_lock_in;
|
||||
|
||||
let (Some(pool_index), Some(set_index)) = (opts.pool, opts.set) else {
|
||||
return Err(Error::PreconditionFailed);
|
||||
};
|
||||
if targets.is_empty() || opts.dry_run || opts.no_lock || check_valid_bucket_name_strict(bucket).is_err() {
|
||||
return Err(Error::PreconditionFailed);
|
||||
}
|
||||
let pool = self
|
||||
.pools
|
||||
.get(pool_index)
|
||||
.ok_or_else(|| invalid_heal_pool_index(pool_index, self.pools.len()))?;
|
||||
let target_set = pool.get_disks_for_heal_object(BUCKET_METADATA_FILE, opts)?;
|
||||
if targets.iter().any(|target| {
|
||||
target_set
|
||||
.set_endpoints
|
||||
.iter()
|
||||
.filter(|endpoint| endpoint.to_string() == *target)
|
||||
.count()
|
||||
!= 1
|
||||
}) {
|
||||
return Err(Error::PreconditionFailed);
|
||||
}
|
||||
let objects: Vec<_> = [BUCKET_METADATA_FILE, BUCKET_INCARNATION_FILE]
|
||||
.into_iter()
|
||||
.map(|file| format!("{}/{bucket}/{file}", disk::BUCKET_META_PREFIX))
|
||||
.filter(|object| Arc::ptr_eq(&target_set, &pool.get_disks_by_key(object)))
|
||||
.collect();
|
||||
if objects.is_empty() {
|
||||
return Ok(());
|
||||
}
|
||||
|
||||
// Read the persisted identity without lazy migration: a replacement
|
||||
// must never turn lost configuration into newly fabricated defaults.
|
||||
let incarnation = self.bucket_incarnation_id_from_disk(bucket).await?;
|
||||
let targets = targets.to_vec();
|
||||
self.run_bucket_heal_at_incarnation(bucket, incarnation, opts, move |store, bucket, opts| async move {
|
||||
// Match config writers: lifecycle, metadata transaction, object locks.
|
||||
// Keep the transaction stable through target readback, including when
|
||||
// the caller is cancelled and the storage owner finishes its write.
|
||||
let transaction = acquire_bucket_metadata_transaction_read_lock_in(&store.ctx, &bucket).await?;
|
||||
for object in objects {
|
||||
let scope = bucket_heal_scope(&bucket).ok_or(Error::PreconditionFailed)?;
|
||||
scope.check()?;
|
||||
if transaction.is_lock_lost() {
|
||||
return Err(Error::PreconditionFailed);
|
||||
}
|
||||
// Config objects occupy one pool, unlike pool.bin. Require a
|
||||
// successful all-pool lookup before accepting another pool's ownership.
|
||||
if !store.single_pool() {
|
||||
let (owner, _) = store
|
||||
.get_pool_info_for_delete_marker(RUSTFS_META_BUCKET, &object, &ObjectOptions::default())
|
||||
.await?;
|
||||
if owner.index != pool_index {
|
||||
continue;
|
||||
}
|
||||
}
|
||||
let metadata_opts = HealOpts { remove: false, ..opts };
|
||||
let (result, error) = store.heal_object(RUSTFS_META_BUCKET, &object, "", &metadata_opts).await?;
|
||||
if let Some(error) = error {
|
||||
return Err(error);
|
||||
}
|
||||
for target in &targets {
|
||||
let mut drives = result.after.drives.iter().filter(|drive| &drive.endpoint == target);
|
||||
if !drives.next().is_some_and(|drive| drive.state == "ok") || drives.next().is_some() {
|
||||
return Err(Error::PreconditionFailed);
|
||||
}
|
||||
}
|
||||
if !store
|
||||
.replacement_targets_have_version(RUSTFS_META_BUCKET, &object, "", pool_index, set_index, &targets)
|
||||
.await?
|
||||
{
|
||||
return Err(Error::PreconditionFailed);
|
||||
}
|
||||
}
|
||||
if transaction.is_lock_lost() {
|
||||
return Err(Error::PreconditionFailed);
|
||||
}
|
||||
Ok(())
|
||||
})
|
||||
.await
|
||||
}
|
||||
|
||||
/// Return every live erasure set selected by an object-heal scope.
|
||||
pub async fn heal_erasure_set_scopes(&self, opts: &HealOpts) -> Result<Vec<(usize, usize)>> {
|
||||
let pools = self.get_pools_for_heal_object(opts)?;
|
||||
@@ -684,8 +813,18 @@ impl ECStore {
|
||||
version_id: &str,
|
||||
opts: &HealOpts,
|
||||
) -> Result<(HealResultItem, Option<Error>)> {
|
||||
self.handle_heal_object_with_absence(bucket, object, version_id, opts, &mut None, None)
|
||||
.await
|
||||
self.handle_heal_object_with_absence(
|
||||
bucket,
|
||||
object,
|
||||
version_id,
|
||||
opts,
|
||||
&mut None,
|
||||
crate::set_disk::AbsenceProofRequest {
|
||||
retirement: None,
|
||||
allow_unversioned: false,
|
||||
},
|
||||
)
|
||||
.await
|
||||
}
|
||||
|
||||
pub async fn heal_object_with_proof(
|
||||
@@ -695,7 +834,34 @@ impl ECStore {
|
||||
version_id: &str,
|
||||
opts: &HealOpts,
|
||||
) -> Result<HealObjectStorageResult> {
|
||||
if opts.dry_run || opts.no_lock || version_id.is_empty() || super::utils::is_reserved_or_invalid_bucket(bucket, false) {
|
||||
self.heal_object_with_absence_proof(bucket, object, version_id, opts, false)
|
||||
.await
|
||||
}
|
||||
|
||||
async fn heal_object_with_authoritative_absence_proof(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
version_id: &str,
|
||||
opts: &HealOpts,
|
||||
) -> Result<HealObjectStorageResult> {
|
||||
self.heal_object_with_absence_proof(bucket, object, version_id, opts, true)
|
||||
.await
|
||||
}
|
||||
|
||||
async fn heal_object_with_absence_proof(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
version_id: &str,
|
||||
opts: &HealOpts,
|
||||
allow_unversioned_absence: bool,
|
||||
) -> Result<HealObjectStorageResult> {
|
||||
if opts.dry_run
|
||||
|| opts.no_lock
|
||||
|| (!allow_unversioned_absence && version_id.is_empty())
|
||||
|| super::utils::is_reserved_or_invalid_bucket(bucket, false)
|
||||
{
|
||||
let (item, error) = self.handle_heal_object(bucket, object, version_id, opts).await?;
|
||||
return Ok(HealObjectStorageResult {
|
||||
item,
|
||||
@@ -738,7 +904,17 @@ impl ECStore {
|
||||
};
|
||||
let mut proofs = None;
|
||||
let (item, mut error) = self
|
||||
.handle_heal_object_with_absence(bucket, object, version_id, opts, &mut proofs, Some(&retirement))
|
||||
.handle_heal_object_with_absence(
|
||||
bucket,
|
||||
object,
|
||||
version_id,
|
||||
opts,
|
||||
&mut proofs,
|
||||
crate::set_disk::AbsenceProofRequest {
|
||||
retirement: Some(&retirement),
|
||||
allow_unversioned: allow_unversioned_absence,
|
||||
},
|
||||
)
|
||||
.await?;
|
||||
// Read the authoritative incarnation only for an absence candidate.
|
||||
// The lifecycle guard has pinned it throughout the storage operation.
|
||||
@@ -780,7 +956,7 @@ impl ECStore {
|
||||
version_id: &str,
|
||||
opts: &HealOpts,
|
||||
absence: &mut Option<Vec<crate::set_disk::HealedObjectAbsence>>,
|
||||
retirement: Option<&crate::bucket::retirement::MarkerRetirementContext<'_>>,
|
||||
proof: crate::set_disk::AbsenceProofRequest<'_>,
|
||||
) -> Result<(HealResultItem, Option<Error>)> {
|
||||
trace!(
|
||||
event = EVENT_HEAL_OBJECT_STARTED,
|
||||
@@ -871,7 +1047,7 @@ impl ECStore {
|
||||
}
|
||||
#[cfg(test)]
|
||||
crate::core::pools::notify_decommission_external_heal_operation_started(store_id);
|
||||
pool.heal_object_with_absence(bucket, &pool_object, version_id, &opts, retirement)
|
||||
pool.heal_object_with_absence(bucket, &pool_object, version_id, &opts, proof)
|
||||
.await
|
||||
}
|
||||
});
|
||||
@@ -895,7 +1071,7 @@ impl ECStore {
|
||||
move |opts| async move {
|
||||
#[cfg(test)]
|
||||
crate::core::pools::notify_decommission_external_heal_operation_started(store_id);
|
||||
pool.heal_object_with_absence(bucket, &pool_object, version_id, &opts, retirement)
|
||||
pool.heal_object_with_absence(bucket, &pool_object, version_id, &opts, proof)
|
||||
.await
|
||||
},
|
||||
));
|
||||
@@ -1280,6 +1456,16 @@ mod tests {
|
||||
.await
|
||||
.is_err()
|
||||
);
|
||||
for file in [".metadata.bin", ".bucket-incarnation"] {
|
||||
let metadata = format!("buckets/{bucket}/{file}");
|
||||
assert!(
|
||||
store
|
||||
.rename_local_data_at_incarnation(&disk_ref, (&bucket, object), &fi, (RUSTFS_META_BUCKET, &metadata), old)
|
||||
.await
|
||||
.is_err(),
|
||||
"stale remote heal must not replace the successor's {file}"
|
||||
);
|
||||
}
|
||||
assert!(store.heal_bucket_at_incarnation(&bucket, old, &opts).await.is_err());
|
||||
assert!(
|
||||
store
|
||||
@@ -1678,6 +1864,194 @@ mod tests {
|
||||
(temp_dir, store, shutdown)
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn bucket_metadata_heal_owner_rejects_path_aliases_and_unrelated_records() {
|
||||
for file in [".metadata.bin", ".bucket-incarnation"] {
|
||||
assert_eq!(bucket_metadata_owner(&format!("buckets/example/{file}")), Some("example"));
|
||||
}
|
||||
for path in [
|
||||
"/buckets/example/.metadata.bin",
|
||||
"buckets/../.metadata.bin",
|
||||
"buckets/example/../.metadata.bin",
|
||||
"buckets/example/.metadata.bin/extra",
|
||||
"buckets/example//.metadata.bin",
|
||||
"buckets/example/.metadata.bin.old",
|
||||
"config/iam/example/.metadata.bin",
|
||||
"buckets/example/usage.json",
|
||||
] {
|
||||
assert_eq!(bucket_metadata_owner(path), None, "unexpected bucket authority for {path}");
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn replacement_bucket_metadata_respects_pool_ownership_and_required_records() {
|
||||
let (root, store, shutdown) = multi_pool_heal_store().await;
|
||||
let bucket = "replacement-bucket-metadata";
|
||||
for invalid_bucket in ["MixedCase", "../example", RUSTFS_META_BUCKET] {
|
||||
assert!(matches!(
|
||||
store
|
||||
.heal_replacement_bucket_metadata(
|
||||
invalid_bucket,
|
||||
&HealOpts {
|
||||
pool: Some(0),
|
||||
set: Some(0),
|
||||
..Default::default()
|
||||
},
|
||||
&[root.path().join("pool0-disk0").to_string_lossy().into_owned()],
|
||||
)
|
||||
.await,
|
||||
Err(Error::PreconditionFailed)
|
||||
));
|
||||
}
|
||||
store
|
||||
.make_bucket(bucket, &MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("create bucket and its persisted records");
|
||||
let mut owned = 0;
|
||||
for pool in 0..2 {
|
||||
let target = root.path().join(format!("pool{pool}-disk0"));
|
||||
let records: Vec<_> = [".metadata.bin", ".bucket-incarnation"]
|
||||
.into_iter()
|
||||
.map(|file| {
|
||||
let path = target.join(RUSTFS_META_BUCKET).join("buckets").join(bucket).join(file);
|
||||
let exists = path.join("xl.meta").exists();
|
||||
if exists {
|
||||
std::fs::remove_dir_all(&path).expect("remove replacement metadata shard");
|
||||
owned += 1;
|
||||
}
|
||||
(path, exists)
|
||||
})
|
||||
.collect();
|
||||
store
|
||||
.heal_replacement_bucket_metadata(
|
||||
bucket,
|
||||
&HealOpts {
|
||||
pool: Some(pool),
|
||||
set: Some(0),
|
||||
recreate: true,
|
||||
..Default::default()
|
||||
},
|
||||
&[target.to_string_lossy().into_owned()],
|
||||
)
|
||||
.await
|
||||
.expect("repair owned records and accept authoritative ownership in the other pool");
|
||||
for (path, existed) in records {
|
||||
assert_eq!(path.join("xl.meta").exists(), existed, "wrong placement for {}", path.display());
|
||||
}
|
||||
}
|
||||
assert_eq!(owned, 2, "each required record must have exactly one owning pool");
|
||||
let object = format!("buckets/{bucket}/.metadata.bin");
|
||||
let owner = store
|
||||
.get_pool_idx_existing_with_opts(RUSTFS_META_BUCKET, &object, &ObjectOptions::default())
|
||||
.await
|
||||
.expect("persisted metadata owner");
|
||||
let disk = store.pools[owner].disk_set[0].disks.read().await[0]
|
||||
.clone()
|
||||
.expect("metadata target disk");
|
||||
let fi = disk
|
||||
.read_version(
|
||||
"",
|
||||
RUSTFS_META_BUCKET,
|
||||
&object,
|
||||
"",
|
||||
&disk::ReadOptions {
|
||||
read_data: true,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("inline metadata to repair through target-side admission");
|
||||
assert!(fi.data.is_some(), "fixture must contain an inline metadata shard");
|
||||
let target_path = root.path().join(format!("pool{owner}-disk0"));
|
||||
std::fs::remove_dir_all(target_path.join(RUSTFS_META_BUCKET).join(&object)).expect("remove remote target shard");
|
||||
let incarnation = store
|
||||
.bucket_incarnation_id_from_disk(bucket)
|
||||
.await
|
||||
.expect("current incarnation");
|
||||
store
|
||||
.rename_local_data_at_incarnation(
|
||||
&disk.endpoint().to_string(),
|
||||
(disk::RUSTFS_META_TMP_BUCKET, "metadata-heal-control"),
|
||||
&fi,
|
||||
(RUSTFS_META_BUCKET, &object),
|
||||
incarnation,
|
||||
)
|
||||
.await
|
||||
.expect("the target-side RPC admission must accept current bucket metadata repair");
|
||||
assert!(target_path.join(RUSTFS_META_BUCKET).join(&object).join("xl.meta").exists());
|
||||
let (configuration, _) = read_config_no_lock_preserve_empty_with_metadata(store.clone(), &object)
|
||||
.await
|
||||
.expect("read the current bucket configuration");
|
||||
save_config(store.clone(), &object, configuration)
|
||||
.await
|
||||
.expect("publish a newer config revision in the same bucket incarnation");
|
||||
let committed =
|
||||
std::fs::read(target_path.join(RUSTFS_META_BUCKET).join(&object).join("xl.meta")).expect("new target configuration");
|
||||
assert!(
|
||||
store
|
||||
.rename_local_data_at_incarnation(
|
||||
&disk.endpoint().to_string(),
|
||||
(disk::RUSTFS_META_TMP_BUCKET, "delayed-metadata-heal"),
|
||||
&fi,
|
||||
(RUSTFS_META_BUCKET, &object),
|
||||
incarnation,
|
||||
)
|
||||
.await
|
||||
.is_err(),
|
||||
"a delayed repair must not overwrite a newer config in the same incarnation"
|
||||
);
|
||||
assert_eq!(
|
||||
std::fs::read(target_path.join(RUSTFS_META_BUCKET).join(&object).join("xl.meta")).expect("retained configuration"),
|
||||
committed,
|
||||
"rejected repair must leave the newer configuration unchanged"
|
||||
);
|
||||
for pool in 0..2 {
|
||||
for disk in 0..4 {
|
||||
let path = root
|
||||
.path()
|
||||
.join(format!("pool{pool}-disk{disk}"))
|
||||
.join(RUSTFS_META_BUCKET)
|
||||
.join("buckets")
|
||||
.join(bucket)
|
||||
.join(".metadata.bin");
|
||||
if path.exists() {
|
||||
std::fs::remove_dir_all(path).expect("remove every copy of required bucket configuration");
|
||||
}
|
||||
}
|
||||
}
|
||||
metadata_sys::remove_bucket_metadata_in(&store.ctx, bucket)
|
||||
.await
|
||||
.expect("remove the cached configuration so repair cannot rely on it");
|
||||
assert!(
|
||||
store
|
||||
.heal_replacement_bucket_metadata(
|
||||
bucket,
|
||||
&HealOpts {
|
||||
pool: Some(0),
|
||||
set: Some(0),
|
||||
recreate: true,
|
||||
..Default::default()
|
||||
},
|
||||
&[root.path().join("pool0-disk0").to_string_lossy().into_owned()],
|
||||
)
|
||||
.await
|
||||
.is_err(),
|
||||
"total metadata loss must not be accepted as ownership in another pool"
|
||||
);
|
||||
for pool in 0..2 {
|
||||
assert!(
|
||||
!root
|
||||
.path()
|
||||
.join(format!("pool{pool}-disk0"))
|
||||
.join(RUSTFS_META_BUCKET)
|
||||
.join(&object)
|
||||
.exists(),
|
||||
"missing required configuration must not be recreated with defaults"
|
||||
);
|
||||
}
|
||||
shutdown.cancel();
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
#[serial_test::serial]
|
||||
async fn absence_proof_requires_every_selected_pool() {
|
||||
|
||||
@@ -4090,7 +4090,7 @@ mod tests {
|
||||
.collect::<Vec<_>>();
|
||||
for disk in &disks {
|
||||
disk.close().await.expect("fault injection should stop per-disk monitoring");
|
||||
disk.force_runtime_state_for_test(crate::disk::health_state::RuntimeDriveHealthState::Offline);
|
||||
disk.force_offline_for_test();
|
||||
}
|
||||
|
||||
// Sets has an independent endpoint monitor that renews missing slots.
|
||||
@@ -4127,7 +4127,7 @@ mod tests {
|
||||
.collect::<Vec<_>>();
|
||||
for disk in &disks {
|
||||
disk.close().await.expect("fault injection should stop per-disk monitoring");
|
||||
disk.force_runtime_state_for_test(crate::disk::health_state::RuntimeDriveHealthState::Offline);
|
||||
disk.force_offline_for_test();
|
||||
}
|
||||
set.connect_disks().await;
|
||||
for disk in &disks {
|
||||
@@ -10052,9 +10052,18 @@ mod tests {
|
||||
recovered_transition_version.clone(),
|
||||
)))
|
||||
.await;
|
||||
let transaction_stats = recover_transition_transaction_records(store.clone(), 100, None)
|
||||
let transaction_stats = {
|
||||
let _proof = crate::services::notification_sys::install_current_remote_version_state_fleet_proof_for_test();
|
||||
temp_env::async_with_vars(
|
||||
[
|
||||
(rustfs_config::ENV_TIER_REMOTE_VERSION_STATE_WRITE, Some("true")),
|
||||
(rustfs_config::ENV_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED, Some("true")),
|
||||
],
|
||||
recover_transition_transaction_records(store.clone(), 100, None),
|
||||
)
|
||||
.await
|
||||
.expect("transition recovery should advance and consume the migrated transaction before completion");
|
||||
.expect("transition recovery should advance and consume the migrated transaction before completion")
|
||||
};
|
||||
backend.set_transition_candidate_probe_override(None).await;
|
||||
assert_eq!(
|
||||
(
|
||||
@@ -21938,9 +21947,18 @@ mod tests {
|
||||
expected_removes.push((transaction.remote_object, put_version));
|
||||
}
|
||||
|
||||
let stats = recover_transition_transaction_records(store.clone(), 100, None)
|
||||
let stats = {
|
||||
let _proof = crate::services::notification_sys::install_current_remote_version_state_fleet_proof_for_test();
|
||||
temp_env::async_with_vars(
|
||||
[
|
||||
(rustfs_config::ENV_TIER_REMOTE_VERSION_STATE_WRITE, Some("true")),
|
||||
(rustfs_config::ENV_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED, Some("true")),
|
||||
],
|
||||
recover_transition_transaction_records(store.clone(), 100, None),
|
||||
)
|
||||
.await
|
||||
.expect("transition transaction recovery should run");
|
||||
.expect("transition transaction recovery should run")
|
||||
};
|
||||
|
||||
assert_eq!((stats.scanned, stats.recovered, stats.retained, stats.failed), (3, 3, 0, 0));
|
||||
assert_eq!(transition_transaction_record_count(store.clone()).await, 0);
|
||||
|
||||
@@ -66,7 +66,7 @@ use std::future::Future;
|
||||
use std::path::{Path, PathBuf};
|
||||
use std::sync::{
|
||||
Arc,
|
||||
atomic::{AtomicU64, Ordering},
|
||||
atomic::{AtomicBool, AtomicU64, Ordering},
|
||||
};
|
||||
use std::time::{Duration, SystemTime, UNIX_EPOCH};
|
||||
use tokio::io::duplex;
|
||||
@@ -341,6 +341,18 @@ pub struct ListPathOptions {
|
||||
pub cursor_generation: Option<String>,
|
||||
pub walkdir_timeout: Option<Duration>,
|
||||
pub walkdir_stall_timeout: Option<Duration>,
|
||||
/// Shared by the collector and raw producers to preserve bounded-walk
|
||||
/// completion when filtering removes all entries from a batch.
|
||||
pub producer_limit_reached: Option<Arc<AtomicBool>>,
|
||||
}
|
||||
|
||||
fn ensure_producer_limit_state(options: &mut ListPathOptions) -> Arc<AtomicBool> {
|
||||
let state = options
|
||||
.producer_limit_reached
|
||||
.clone()
|
||||
.unwrap_or_else(|| Arc::new(AtomicBool::new(false)));
|
||||
options.producer_limit_reached = Some(state.clone());
|
||||
state
|
||||
}
|
||||
|
||||
async fn can_skip_hidden_prefix_check(options: &ListPathOptions) -> bool {
|
||||
@@ -4304,6 +4316,7 @@ impl ECStore {
|
||||
// cancel channel
|
||||
let cancel = CancellationToken::new();
|
||||
let _cancel_guard = cancel.clone().drop_guard();
|
||||
ensure_producer_limit_state(&mut o);
|
||||
|
||||
let (err_tx, mut err_rx) = broadcast::channel::<Arc<Error>>(1);
|
||||
|
||||
@@ -5047,6 +5060,15 @@ async fn gather_results(
|
||||
}
|
||||
}
|
||||
|
||||
// A producer can close its stream after exhausting its own scan budget
|
||||
// before this collector reaches its output limit. In that case the input
|
||||
// channel closing is not authoritative EOF: the caller must advertise a
|
||||
// continuation page even when every remaining entry was filtered out.
|
||||
let producer_limit_reached = opts
|
||||
.producer_limit_reached
|
||||
.as_ref()
|
||||
.is_some_and(|reached| reached.load(Ordering::Acquire));
|
||||
|
||||
// finish not full, return eof
|
||||
let filtered = scanned_entries.saturating_sub(candidate_entries);
|
||||
if let Some(started) = gather_started {
|
||||
@@ -5083,7 +5105,7 @@ async fn gather_results(
|
||||
o: MetaCacheEntries(entries),
|
||||
..Default::default()
|
||||
}),
|
||||
err: Some(rustfs_filemeta::Error::Unexpected),
|
||||
err: (!producer_limit_reached).then_some(rustfs_filemeta::Error::Unexpected),
|
||||
})
|
||||
.await
|
||||
.is_err()
|
||||
@@ -5734,6 +5756,7 @@ impl Sets {
|
||||
let (err_tx, mut err_rx) = broadcast::channel::<Arc<Error>>(1);
|
||||
let (sender, recv) = mpsc::channel(o.limit as usize);
|
||||
|
||||
ensure_producer_limit_state(&mut o);
|
||||
let sets = self.clone();
|
||||
let opts = o.clone();
|
||||
let cancel_rx1 = cancel.clone();
|
||||
@@ -6777,6 +6800,7 @@ impl SetDisks {
|
||||
let (err_tx, mut err_rx) = broadcast::channel::<Arc<Error>>(1);
|
||||
let (sender, recv) = mpsc::channel(o.limit as usize);
|
||||
|
||||
ensure_producer_limit_state(&mut o);
|
||||
let set = self.clone();
|
||||
let opts = o.clone();
|
||||
let cancel_rx1 = cancel.clone();
|
||||
@@ -7098,6 +7122,7 @@ impl SetDisks {
|
||||
forward_to: opts.marker,
|
||||
min_disks: raw_min_disks,
|
||||
per_disk_limit: limit,
|
||||
producer_limit_reached: opts.producer_limit_reached.clone(),
|
||||
// A foreground listing is bounded by lack of drive progress (the walk
|
||||
// stall timeout) and by the page limit, never by how long a healthy
|
||||
// walk takes — a large prefix on slow media is not a fault (#4644).
|
||||
@@ -7328,6 +7353,7 @@ mod test {
|
||||
};
|
||||
use std::collections::{HashMap, HashSet};
|
||||
use std::path::PathBuf;
|
||||
use std::sync::atomic::AtomicBool;
|
||||
use std::sync::{Arc, Mutex};
|
||||
use std::time::Duration;
|
||||
use tokio::sync::mpsc;
|
||||
@@ -8347,6 +8373,45 @@ mod test {
|
||||
assert!(!cancel.is_cancelled());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn list_path_gather_results_preserves_bounded_producer_after_filtering() {
|
||||
let (entry_tx, entry_rx) = mpsc::channel(4);
|
||||
let (result_tx, mut result_rx) = mpsc::channel(1);
|
||||
let cancel = CancellationToken::new();
|
||||
let producer_limit_reached = Arc::new(AtomicBool::new(true));
|
||||
|
||||
entry_tx
|
||||
.send(test_meta_entry("outside-prefix"))
|
||||
.await
|
||||
.expect("filtered test entry should be queued");
|
||||
drop(entry_tx);
|
||||
|
||||
let handle = tokio::spawn(gather_results(
|
||||
cancel,
|
||||
ListPathOptions {
|
||||
bucket: "bucket".to_owned(),
|
||||
prefix: "requested/".to_owned(),
|
||||
limit: 8,
|
||||
incl_deleted: true,
|
||||
producer_limit_reached: Some(producer_limit_reached),
|
||||
..Default::default()
|
||||
},
|
||||
entry_rx,
|
||||
result_tx,
|
||||
));
|
||||
|
||||
let result = result_rx.recv().await.expect("bounded producer result should be delivered");
|
||||
assert!(result.entries.expect("entries should be present").entries().is_empty());
|
||||
assert!(result.err.is_none(), "bounded producer must not be reported as EOF");
|
||||
assert_eq!(
|
||||
handle
|
||||
.await
|
||||
.expect("gather task should not panic")
|
||||
.expect("gather should succeed"),
|
||||
GatherResultsState::InputClosed
|
||||
);
|
||||
}
|
||||
|
||||
/// A-1 guard (rustfs/backlog#1306): pin that a *successful* send is never
|
||||
/// misclassified as `ConsumerGone`. With the receiver alive, gather_results
|
||||
/// must return the real drain state and deliver the payload carrying the
|
||||
@@ -9678,6 +9743,131 @@ mod test {
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn empty_delimiter_listing_of_residue_ancestor_hides_and_purges_whole_tree() {
|
||||
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
|
||||
use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
|
||||
|
||||
let (dirs, store) = isolated_store_over_temp_disks().await;
|
||||
let bucket = "listing-purge-ancestor-bucket";
|
||||
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
|
||||
store
|
||||
.make_bucket(bucket, &MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("bucket should be created with authoritative metadata");
|
||||
let data_dir = uuid::Uuid::new_v4();
|
||||
let transaction = uuid::Uuid::new_v4();
|
||||
for dir in &dirs {
|
||||
let residue = dir
|
||||
.path()
|
||||
.join(bucket)
|
||||
.join("metrics/kubelet/2026/08/28/23/74992556388248657933757.parquet")
|
||||
.join(data_dir.to_string());
|
||||
tokio::fs::create_dir_all(&residue)
|
||||
.await
|
||||
.expect("committed delete residue should be created");
|
||||
tokio::fs::write(residue.join("part.1"), b"stale")
|
||||
.await
|
||||
.expect("stale part should be written");
|
||||
tokio::fs::write(
|
||||
residue.join(format!("{}{}", crate::disk::local::DELETE_DATA_DIR_MARKER_PREFIX, transaction)),
|
||||
[],
|
||||
)
|
||||
.await
|
||||
.expect("committed delete marker should be written");
|
||||
}
|
||||
|
||||
// Browsing an ancestor of the deleted key must not show the empty
|
||||
// date folders, and the empty result reclaims the whole residue tree
|
||||
// in one pass instead of one level per listing.
|
||||
let result = store
|
||||
.clone()
|
||||
.list_objects_generic(bucket, "metrics/", None, Some("/".to_owned()), 1000, false)
|
||||
.await
|
||||
.expect("delimiter listing should succeed");
|
||||
assert!(result.objects.is_empty());
|
||||
assert!(result.prefixes.is_empty(), "ancestors of delete residue must not surface as prefixes");
|
||||
for dir in &dirs {
|
||||
assert!(
|
||||
!dir.path().join(bucket).join("metrics").exists(),
|
||||
"the empty delimiter listing should reclaim the committed delete residue tree under it"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn empty_delimiter_listing_of_mixed_residue_ancestor_reclaims_only_committed_subtree() {
|
||||
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
|
||||
use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
|
||||
|
||||
let (dirs, store) = isolated_store_over_temp_disks().await;
|
||||
let bucket = "listing-purge-mixed-bucket";
|
||||
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
|
||||
store
|
||||
.make_bucket(bucket, &MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("bucket should be created with authoritative metadata");
|
||||
let committed_dir = uuid::Uuid::new_v4();
|
||||
let unmarked_dir = uuid::Uuid::new_v4();
|
||||
let transaction = uuid::Uuid::new_v4();
|
||||
for dir in &dirs {
|
||||
let committed = dir
|
||||
.path()
|
||||
.join(bucket)
|
||||
.join("metrics/cpu/2026/08/28/22/a.parquet")
|
||||
.join(committed_dir.to_string());
|
||||
tokio::fs::create_dir_all(&committed)
|
||||
.await
|
||||
.expect("committed delete residue should be created");
|
||||
tokio::fs::write(committed.join("part.1"), b"stale")
|
||||
.await
|
||||
.expect("stale part should be written");
|
||||
tokio::fs::write(
|
||||
committed.join(format!("{}{}", crate::disk::local::DELETE_DATA_DIR_MARKER_PREFIX, transaction)),
|
||||
[],
|
||||
)
|
||||
.await
|
||||
.expect("committed delete marker should be written");
|
||||
|
||||
// Residue left by a build that never wrote delete markers, or a
|
||||
// PUT still streaming its parts: indistinguishable, so never purged.
|
||||
let unmarked = dir
|
||||
.path()
|
||||
.join(bucket)
|
||||
.join("metrics/kubelet/2026/08/28/23/74992556388248657933757.parquet")
|
||||
.join(unmarked_dir.to_string());
|
||||
tokio::fs::create_dir_all(&unmarked)
|
||||
.await
|
||||
.expect("unmarked residue should be created");
|
||||
tokio::fs::write(unmarked.join("part.1"), b"stale")
|
||||
.await
|
||||
.expect("stale part should be written");
|
||||
}
|
||||
|
||||
let result = store
|
||||
.clone()
|
||||
.list_objects_generic(bucket, "metrics/", None, Some("/".to_owned()), 1000, false)
|
||||
.await
|
||||
.expect("delimiter listing should succeed");
|
||||
assert!(result.objects.is_empty());
|
||||
assert!(result.prefixes.is_empty(), "neither residue subtree may surface as a prefix");
|
||||
for dir in &dirs {
|
||||
let metrics = dir.path().join(bucket).join("metrics");
|
||||
assert!(
|
||||
!metrics.join("cpu").exists(),
|
||||
"the committed subtree must be reclaimed even though a sibling subtree is blocked"
|
||||
);
|
||||
assert!(
|
||||
metrics
|
||||
.join("kubelet/2026/08/28/23/74992556388248657933757.parquet")
|
||||
.join(unmarked_dir.to_string())
|
||||
.join("part.1")
|
||||
.exists(),
|
||||
"residue without a committed marker must survive the purge"
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn list_objects_index_provider_state_uses_lifecycle_active_generation() {
|
||||
let provider = ListObjectsIndexProviderState::walker_key_only();
|
||||
|
||||
@@ -426,8 +426,8 @@ mod bucket_fence;
|
||||
pub(crate) use bucket::await_bucket_namespace_operation;
|
||||
pub use bucket_fence::BucketIncarnationFenceGuard;
|
||||
mod heal;
|
||||
pub(crate) use heal::bucket_heal_scope;
|
||||
pub use heal::{HealObjectAbsenceProof, HealObjectStorageResult};
|
||||
pub(crate) use heal::{bucket_heal_scope, bucket_heal_scope_for_object};
|
||||
mod heal_walk;
|
||||
pub use heal_walk::HealWalkVersion;
|
||||
mod init;
|
||||
|
||||
@@ -9124,6 +9124,100 @@ mod tests {
|
||||
assert_eq!(dirs.len(), 16);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
|
||||
async fn current_marker_delete_ack_removes_stale_rejoined_copy() {
|
||||
let bucket = "current-marker-stale-rejoin";
|
||||
let object = "marker.bin";
|
||||
let ctx = Arc::new(crate::runtime::instance::InstanceContext::new());
|
||||
let (_dirs, original) = make_local_set_disks_with_ctx(16, 4, ctx.clone()).await;
|
||||
let store = Arc::new(new_prepared_reader_test_store_with_ctx(&[original], ctx).await);
|
||||
crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
|
||||
store
|
||||
.handle_make_bucket(
|
||||
bucket,
|
||||
&MakeBucketOptions {
|
||||
versioning_enabled: true,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("create versioned bucket");
|
||||
let marker = store
|
||||
.delete_object(
|
||||
bucket,
|
||||
object,
|
||||
ObjectOptions {
|
||||
versioned: true,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("create current marker")
|
||||
.version_id
|
||||
.expect("marker has explicit version");
|
||||
let current = store
|
||||
.bucket_incarnation_id_from_disk(bucket)
|
||||
.await
|
||||
.expect("bucket incarnation");
|
||||
let set = &store.pools[0].disk_set[0];
|
||||
let before = set.disks.read().await.clone();
|
||||
for slot in 12..16 {
|
||||
set.disks.write().await[slot] = None;
|
||||
}
|
||||
store
|
||||
.delete_object(
|
||||
bucket,
|
||||
object,
|
||||
ObjectOptions {
|
||||
versioned: true,
|
||||
version_id: Some(marker.to_string()),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("acknowledged delete should remove the marker from the online quorum");
|
||||
*set.disks.write().await = before.clone();
|
||||
|
||||
for disk in before.iter().take(12).flatten() {
|
||||
assert!(matches!(
|
||||
disk.read_version("", bucket, object, &marker.to_string(), &crate::disk::ReadOptions::default())
|
||||
.await,
|
||||
Err(crate::disk::error::DiskError::FileVersionNotFound | crate::disk::error::DiskError::FileNotFound)
|
||||
));
|
||||
}
|
||||
for disk in before.iter().skip(12).flatten() {
|
||||
assert!(
|
||||
disk.read_version("", bucket, object, &marker.to_string(), &crate::disk::ReadOptions::default())
|
||||
.await
|
||||
.is_ok(),
|
||||
"offline member must retain the stale current marker"
|
||||
);
|
||||
}
|
||||
let healed = store
|
||||
.heal_object_at_incarnation(
|
||||
bucket,
|
||||
object,
|
||||
&marker.to_string(),
|
||||
current,
|
||||
&rustfs_heal_contracts::heal_channel::HealOpts {
|
||||
remove: true,
|
||||
scan_mode: rustfs_heal_contracts::heal_channel::HealScanMode::Deep,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("stale current marker heal should complete");
|
||||
assert!(healed.error.is_none(), "stale current marker heal: {:?}", healed.error);
|
||||
assert!(healed.absence.as_ref().is_some_and(|proof| proof.removed));
|
||||
for disk in before.iter().flatten() {
|
||||
assert!(matches!(
|
||||
disk.read_version("", bucket, object, &marker.to_string(), &crate::disk::ReadOptions::default())
|
||||
.await,
|
||||
Err(crate::disk::error::DiskError::FileVersionNotFound | crate::disk::error::DiskError::FileNotFound)
|
||||
));
|
||||
}
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn multipool_delete_marker_stays_with_existing_versions() {
|
||||
let bucket = "multipool-marker-routing";
|
||||
|
||||
@@ -161,14 +161,52 @@ impl ECStore {
|
||||
"incarnation-bound rename requires namespace locking and a non-nil identity",
|
||||
));
|
||||
}
|
||||
let bucket = if destination.0 == RUSTFS_META_BUCKET {
|
||||
super::heal::bucket_metadata_owner(destination.1).ok_or(DiskError::FileAccessDenied)?
|
||||
} else {
|
||||
destination.0
|
||||
};
|
||||
let guard = Arc::new(
|
||||
self.acquire_bucket_incarnation_fence(destination.0, expected)
|
||||
self.acquire_bucket_incarnation_fence(bucket, expected)
|
||||
.await
|
||||
.map_err(|error| DiskError::other(error.to_string()))?,
|
||||
);
|
||||
if guard.is_lock_lost() {
|
||||
return Err(DiskError::other("bucket heal incarnation fence was lost"));
|
||||
}
|
||||
let transaction = if destination.0 == RUSTFS_META_BUCKET {
|
||||
// The receiver must retain the config fence after an RPC timeout.
|
||||
// A delayed request must also reject a newer config in the same
|
||||
// bucket incarnation before publishing its older repair payload.
|
||||
let transaction = crate::bucket::metadata_sys::acquire_bucket_metadata_transaction_read_lock_in(&self.ctx, bucket)
|
||||
.await
|
||||
.map_err(|error| DiskError::other(error.to_string()))?;
|
||||
let (current, _) = self
|
||||
.get_pool_info_for_delete_marker(
|
||||
destination.0,
|
||||
destination.1,
|
||||
&ObjectOptions {
|
||||
no_lock: true,
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.map_err(|error| DiskError::other(error.to_string()))?;
|
||||
let current = current.object_info;
|
||||
if transaction.is_lock_lost()
|
||||
|| guard.is_lock_lost()
|
||||
|| fi.mod_time.is_none()
|
||||
|| current.mod_time != fi.mod_time
|
||||
|| current.data_dir != fi.data_dir
|
||||
|| current.size != fi.size
|
||||
|| current.etag != fi.get_etag()
|
||||
{
|
||||
return Err(DiskError::FileAccessDenied);
|
||||
}
|
||||
Some(transaction)
|
||||
} else {
|
||||
None
|
||||
};
|
||||
rename_local_data_with_ctx(
|
||||
&self.ctx,
|
||||
disk_ref,
|
||||
@@ -176,7 +214,7 @@ impl ECStore {
|
||||
fi,
|
||||
destination,
|
||||
RenameDataGuards {
|
||||
external_guard: Some(guard),
|
||||
external_guard: Some(Arc::new((guard, transaction))),
|
||||
..Default::default()
|
||||
},
|
||||
)
|
||||
|
||||
@@ -907,6 +907,21 @@ impl ErasureSetHealer {
|
||||
}
|
||||
|
||||
if failed_objects == 0 && skipped_objects == 0 && failed_buckets == 0 {
|
||||
let targets = if self.pool_metadata_target_endpoints.is_empty() {
|
||||
self.target_endpoints.as_ref()
|
||||
} else {
|
||||
self.pool_metadata_target_endpoints.as_ref()
|
||||
};
|
||||
if self.replacement_task_id.is_some() || (!self.heal_opts.dry_run && self.heal_opts.recreate && !targets.is_empty()) {
|
||||
self.verify_replacement_identity_fence("bucket metadata").await?;
|
||||
// Recheck even resumed buckets: their user-object cursor does not
|
||||
// prove that the replacement holds the internal bucket records.
|
||||
for bucket in buckets {
|
||||
self.storage
|
||||
.heal_replacement_bucket_metadata(bucket, &self.heal_opts, targets)
|
||||
.await?;
|
||||
}
|
||||
}
|
||||
self.heal_replacement_pool_metadata(
|
||||
set_disk_id,
|
||||
&mut ErasureSetPassCounters {
|
||||
@@ -2267,6 +2282,8 @@ mod resume_loop_tests {
|
||||
list_include_lifecycle_object_info: Mutex<Vec<bool>>,
|
||||
replacement_target_identity_sequences: Mutex<VecDeque<Vec<ReplacementTargetIdentity>>>,
|
||||
pool_metadata_placement: Mutex<Option<ReplacementCommitEvidence>>,
|
||||
bucket_metadata_calls: Mutex<Vec<String>>,
|
||||
bucket_metadata_failure: AtomicBool,
|
||||
fail_listing: AtomicBool,
|
||||
fail_listing_buckets: Mutex<HashSet<String>>,
|
||||
}
|
||||
@@ -2462,6 +2479,13 @@ mod resume_loop_tests {
|
||||
None => Ok(true),
|
||||
}
|
||||
}
|
||||
async fn heal_replacement_bucket_metadata(&self, bucket: &str, _opts: &HealOpts, _targets: &[String]) -> Result<()> {
|
||||
self.bucket_metadata_calls.lock().unwrap().push(bucket.to_owned());
|
||||
if self.bucket_metadata_failure.load(Ordering::SeqCst) {
|
||||
return Err(Error::Storage(EcstoreError::PreconditionFailed));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
async fn replacement_targets_have_version(
|
||||
&self,
|
||||
_bucket: &str,
|
||||
@@ -3058,6 +3082,18 @@ mod resume_loop_tests {
|
||||
)
|
||||
.with_replacement_targets(vec!["replacement-a".to_string()], Some(replacement_task_id.clone()));
|
||||
|
||||
env.storage.bucket_metadata_failure.store(true, Ordering::SeqCst);
|
||||
assert!(healer.heal_erasure_set(&["b".to_string()], "pool_0_set_0").await.is_err());
|
||||
let incomplete = ResumeManager::load_replacement_intent(env.healer.disk.clone(), &replacement_task_id)
|
||||
.await
|
||||
.expect("failed metadata repair must retain replacement intent")
|
||||
.get_state()
|
||||
.await;
|
||||
assert!(!incomplete.completed);
|
||||
assert_ne!(incomplete.replacement_phase, crate::heal::resume::ReplacementPhase::Verified);
|
||||
assert!(env.storage.calls().is_empty(), "metadata failure must stop completion before pool.bin");
|
||||
env.storage.bucket_metadata_failure.store(false, Ordering::SeqCst);
|
||||
|
||||
let error = healer
|
||||
.heal_erasure_set(&["b".to_string()], "pool_0_set_0")
|
||||
.await
|
||||
@@ -3073,6 +3109,11 @@ mod resume_loop_tests {
|
||||
assert_eq!(state.replacement_phase, crate::heal::resume::ReplacementPhase::Intent);
|
||||
assert_eq!(state.retry_count, 1);
|
||||
assert_eq!(env.storage.calls(), vec![(POOL_META_NAME.to_string(), None)]);
|
||||
assert_eq!(
|
||||
*env.storage.bucket_metadata_calls.lock().unwrap(),
|
||||
vec!["b", "b"],
|
||||
"resuming a completed user scan must retry bucket metadata"
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
@@ -72,6 +72,41 @@ const LEGACY_ROOT_HEAL_PATH: &str = ".";
|
||||
const MAX_RECOVERABLE_HEAL_RETRIES: u32 = 3;
|
||||
const MAX_RECOVERABLE_HEAL_RETRY_DELAY: Duration = Duration::from_secs(30);
|
||||
const RESUME_GC_INTERVAL: Duration = Duration::from_secs(60 * 60);
|
||||
const BEST_EFFORT_QUEUE_RESERVE_PERCENT: usize = 10;
|
||||
const BEST_EFFORT_SOURCE_COUNT: usize = 3;
|
||||
|
||||
fn is_best_effort_source(source: HealRequestSource) -> bool {
|
||||
matches!(
|
||||
source,
|
||||
HealRequestSource::Scanner | HealRequestSource::AutoHeal | HealRequestSource::ReadRepair
|
||||
)
|
||||
}
|
||||
|
||||
fn best_effort_source_quota_index(source: HealRequestSource) -> Option<usize> {
|
||||
match source {
|
||||
HealRequestSource::Scanner => Some(0),
|
||||
HealRequestSource::ReadRepair => Some(1),
|
||||
HealRequestSource::AutoHeal => Some(2),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn best_effort_queue_reserve(queue_capacity: usize) -> usize {
|
||||
match queue_capacity {
|
||||
0..=2 => 0,
|
||||
capacity => (capacity.saturating_mul(BEST_EFFORT_QUEUE_RESERVE_PERCENT) / 100).max(BEST_EFFORT_SOURCE_COUNT),
|
||||
}
|
||||
}
|
||||
|
||||
fn best_effort_source_reserves(queue_capacity: usize) -> [usize; BEST_EFFORT_SOURCE_COUNT] {
|
||||
let reserve = best_effort_queue_reserve(queue_capacity);
|
||||
if reserve == 0 {
|
||||
return [0; BEST_EFFORT_SOURCE_COUNT];
|
||||
}
|
||||
let base = reserve / BEST_EFFORT_SOURCE_COUNT;
|
||||
let remainder = reserve % BEST_EFFORT_SOURCE_COUNT;
|
||||
[base + usize::from(remainder > 0), base + usize::from(remainder > 1), base]
|
||||
}
|
||||
|
||||
// Admission/scheduler outcomes for per-object requests (Object/Metadata/
|
||||
// ECDecode) log via demote_to_debug_when! — MRF, autoheal, and scanner
|
||||
@@ -965,11 +1000,7 @@ struct HealQueueContext<'a> {
|
||||
|
||||
impl HealManager {
|
||||
fn classify_full_admission(request: &HealRequest, config: &HealConfig) -> HealAdmissionResult {
|
||||
let best_effort_source = matches!(
|
||||
request.source,
|
||||
HealRequestSource::Scanner | HealRequestSource::AutoHeal | HealRequestSource::ReadRepair
|
||||
);
|
||||
if best_effort_source || (request.priority == HealPriority::Low && config.low_priority_drop_when_full) {
|
||||
if request.priority == HealPriority::Low && config.low_priority_drop_when_full {
|
||||
HealAdmissionResult::Dropped(HealAdmissionDropReason::QueueFull)
|
||||
} else {
|
||||
HealAdmissionResult::Full
|
||||
@@ -980,32 +1011,6 @@ impl HealManager {
|
||||
queue_len.saturating_mul(100).checked_div(queue_capacity).unwrap_or(0)
|
||||
}
|
||||
|
||||
fn classify_pressure_admission(
|
||||
request: &HealRequest,
|
||||
queue_len: usize,
|
||||
queue_capacity: usize,
|
||||
) -> Option<HealAdmissionResult> {
|
||||
if request.force_start || queue_capacity == 0 {
|
||||
return None;
|
||||
}
|
||||
|
||||
let queue_usage_pct = Self::queue_usage_pct(queue_len, queue_capacity);
|
||||
if queue_usage_pct < 80 {
|
||||
return None;
|
||||
}
|
||||
|
||||
match request.source {
|
||||
HealRequestSource::ReadRepair => Some(HealAdmissionResult::Dropped(HealAdmissionDropReason::PolicyDropped)),
|
||||
HealRequestSource::Scanner if request.priority == HealPriority::Low => {
|
||||
Some(HealAdmissionResult::Dropped(HealAdmissionDropReason::PolicyDropped))
|
||||
}
|
||||
HealRequestSource::AutoHeal if queue_usage_pct >= 95 => {
|
||||
Some(HealAdmissionResult::Dropped(HealAdmissionDropReason::PolicyDropped))
|
||||
}
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn duplicate_admission_for_request(request: &HealRequest, config: &HealConfig) -> HealAdmissionResult {
|
||||
if request.priority == HealPriority::Low && !config.low_priority_merge_enable {
|
||||
HealAdmissionResult::Dropped(HealAdmissionDropReason::PolicyDropped)
|
||||
@@ -1156,6 +1161,8 @@ impl HealManager {
|
||||
publish_heal_queue_length(queue);
|
||||
let queue_capacity = config.queue_size;
|
||||
let per_object_request = request.heal_type.is_per_object();
|
||||
let best_effort_reserve = best_effort_queue_reserve(queue_capacity);
|
||||
let best_effort_source_reserves = best_effort_source_reserves(queue_capacity);
|
||||
|
||||
if queue_len >= queue_capacity && !request.force_start {
|
||||
if Self::can_displace_queued_work(&request)
|
||||
@@ -1243,27 +1250,55 @@ impl HealManager {
|
||||
return HealAdmissionDecision::new(admission);
|
||||
}
|
||||
|
||||
if let Some(admission) = Self::classify_pressure_admission(&request, queue_len, queue_capacity) {
|
||||
Self::record_admission_metric(request.source, admission, context);
|
||||
if let HealAdmissionResult::Dropped(reason) = admission {
|
||||
debug!(
|
||||
target: "rustfs::heal::manager",
|
||||
event = EVENT_HEAL_QUEUE_ADMISSION,
|
||||
component = LOG_COMPONENT_HEAL,
|
||||
subsystem = LOG_SUBSYSTEM_MANAGER,
|
||||
request_id = %request.id,
|
||||
priority = ?request.priority,
|
||||
source = request.source.as_str(),
|
||||
context,
|
||||
queue_len,
|
||||
queue_capacity,
|
||||
queue_usage_pct = Self::queue_usage_pct(queue_len, queue_capacity),
|
||||
reason = reason.as_str(),
|
||||
result = "dropped_pressure",
|
||||
"Heal queue request dropped under pressure"
|
||||
);
|
||||
}
|
||||
return HealAdmissionDecision::new(admission);
|
||||
if !is_best_effort_source(request.source)
|
||||
&& request.priority <= HealPriority::Normal
|
||||
&& !request.force_start
|
||||
&& best_effort_reserve > 0
|
||||
&& queue_len >= queue_capacity.saturating_sub(best_effort_reserve)
|
||||
{
|
||||
Self::record_admission_metric(request.source, HealAdmissionResult::Full, context);
|
||||
demote_to_debug_when!(per_object_request, warn, target: "rustfs::heal::manager", {
|
||||
event = EVENT_HEAL_QUEUE_ADMISSION,
|
||||
component = LOG_COMPONENT_HEAL,
|
||||
subsystem = LOG_SUBSYSTEM_MANAGER,
|
||||
request_id = %request.id,
|
||||
priority = ?request.priority,
|
||||
source = request.source.as_str(),
|
||||
context,
|
||||
queue_len,
|
||||
queue_capacity,
|
||||
reserved_for_best_effort = best_effort_reserve,
|
||||
result = "rejected_for_reserved_capacity",
|
||||
"Normal heal request deferred to preserve best-effort capacity"
|
||||
});
|
||||
return HealAdmissionDecision::new(HealAdmissionResult::Full);
|
||||
}
|
||||
|
||||
if is_best_effort_source(request.source)
|
||||
&& request.priority <= HealPriority::Normal
|
||||
&& !request.force_start
|
||||
&& best_effort_reserve > 0
|
||||
&& queue_len >= queue_capacity.saturating_sub(best_effort_reserve)
|
||||
&& let Some(source_index) = best_effort_source_quota_index(request.source)
|
||||
&& queue.best_effort_source_count(request.source) >= best_effort_source_reserves[source_index]
|
||||
{
|
||||
Self::record_admission_metric(request.source, HealAdmissionResult::Full, context);
|
||||
demote_to_debug_when!(per_object_request, warn, target: "rustfs::heal::manager", {
|
||||
event = EVENT_HEAL_QUEUE_ADMISSION,
|
||||
component = LOG_COMPONENT_HEAL,
|
||||
subsystem = LOG_SUBSYSTEM_MANAGER,
|
||||
request_id = %request.id,
|
||||
priority = ?request.priority,
|
||||
source = request.source.as_str(),
|
||||
context,
|
||||
queue_len,
|
||||
queue_capacity,
|
||||
reserved_for_best_effort = best_effort_reserve,
|
||||
source_quota = best_effort_source_reserves[source_index],
|
||||
result = "rejected_for_source_quota",
|
||||
"Best-effort source deferred to preserve per-source capacity"
|
||||
});
|
||||
return HealAdmissionDecision::new(HealAdmissionResult::Full);
|
||||
}
|
||||
|
||||
if queue_capacity > 0 {
|
||||
|
||||
@@ -34,6 +34,33 @@ pub(super) struct PriorityHealQueue {
|
||||
pub(super) sequence: u64,
|
||||
/// Deduplication index for queued requests
|
||||
pub(super) dedup_keys: HashMap<String, DedupKeyEntry>,
|
||||
pub(super) best_effort_queued: usize,
|
||||
pub(super) best_effort_queued_by_source: [usize; 3],
|
||||
/// Number of MRF/Normal tasks dispatched since a best-effort task ran.
|
||||
/// This is kept with the queue so event-driven scheduler wakeups cannot
|
||||
/// reset the fairness budget between cycles.
|
||||
pub(super) mrf_normal_dispatches_since_best_effort: usize,
|
||||
pub(super) last_best_effort_source: Option<HealRequestSource>,
|
||||
}
|
||||
|
||||
const MRF_NORMAL_DISPATCH_QUANTUM: usize = 4;
|
||||
|
||||
fn best_effort_source_index(source: HealRequestSource) -> Option<usize> {
|
||||
match source {
|
||||
HealRequestSource::Scanner => Some(0),
|
||||
HealRequestSource::ReadRepair => Some(1),
|
||||
HealRequestSource::AutoHeal => Some(2),
|
||||
_ => None,
|
||||
}
|
||||
}
|
||||
|
||||
fn best_effort_source_order(last: Option<HealRequestSource>) -> [usize; 3] {
|
||||
match last {
|
||||
Some(HealRequestSource::Scanner) => [1, 2, 0],
|
||||
Some(HealRequestSource::ReadRepair) => [2, 0, 1],
|
||||
Some(HealRequestSource::AutoHeal) => [0, 1, 2],
|
||||
_ => [0, 1, 2],
|
||||
}
|
||||
}
|
||||
|
||||
/// Wrapper for heap items to implement proper ordering
|
||||
@@ -264,6 +291,10 @@ impl PriorityHealQueue {
|
||||
heap: BinaryHeap::new(),
|
||||
sequence: 0,
|
||||
dedup_keys: HashMap::new(),
|
||||
best_effort_queued: 0,
|
||||
best_effort_queued_by_source: [0; 3],
|
||||
mrf_normal_dispatches_since_best_effort: 0,
|
||||
last_best_effort_source: None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -271,9 +302,11 @@ impl PriorityHealQueue {
|
||||
self.heap.len()
|
||||
}
|
||||
|
||||
#[cfg(test)]
|
||||
pub(super) fn pop_next(&mut self) -> Option<HealRequest> {
|
||||
self.heap.pop().map(|item| {
|
||||
Self::decrement_or_remove_dedup_key(&mut self.dedup_keys, &item.dedup_key);
|
||||
self.remove_best_effort_count(item.request.source);
|
||||
item.request
|
||||
})
|
||||
}
|
||||
@@ -301,12 +334,19 @@ impl PriorityHealQueue {
|
||||
})
|
||||
.refcount += 1;
|
||||
self.sequence += 1;
|
||||
let source = request.source;
|
||||
self.heap.push(PriorityQueueItem {
|
||||
priority: request.priority,
|
||||
sequence: self.sequence,
|
||||
dedup_key: key,
|
||||
request,
|
||||
});
|
||||
if is_best_effort_source(source) {
|
||||
self.best_effort_queued += 1;
|
||||
if let Some(index) = best_effort_source_index(source) {
|
||||
self.best_effort_queued_by_source[index] += 1;
|
||||
}
|
||||
}
|
||||
QueuePushOutcome::Accepted
|
||||
}
|
||||
|
||||
@@ -356,6 +396,7 @@ impl PriorityHealQueue {
|
||||
|
||||
let displaced = displaced.map(|item| {
|
||||
Self::decrement_or_remove_dedup_key(&mut self.dedup_keys, &item.dedup_key);
|
||||
self.remove_best_effort_count(item.request.source);
|
||||
self.refresh_dedup_representative(&item.dedup_key);
|
||||
item.request
|
||||
});
|
||||
@@ -400,6 +441,7 @@ impl PriorityHealQueue {
|
||||
pub(super) fn pop(&mut self) -> Option<HealRequest> {
|
||||
self.heap.pop().map(|item| {
|
||||
Self::decrement_or_remove_dedup_key(&mut self.dedup_keys, &item.dedup_key);
|
||||
self.remove_best_effort_count(item.request.source);
|
||||
item.request
|
||||
})
|
||||
}
|
||||
@@ -429,12 +471,115 @@ impl PriorityHealQueue {
|
||||
(
|
||||
selected.map(|item| {
|
||||
Self::decrement_or_remove_dedup_key(&mut self.dedup_keys, &item.dedup_key);
|
||||
self.remove_best_effort_count(item.request.source);
|
||||
item.request
|
||||
}),
|
||||
skipped,
|
||||
)
|
||||
}
|
||||
|
||||
pub(super) fn pop_runnable_with_fairness<F, G>(&mut self, can_run: F, skip_label: G) -> (Option<HealRequest>, Vec<String>)
|
||||
where
|
||||
F: Fn(&HealRequest) -> bool,
|
||||
G: Fn(&HealRequest) -> Option<String>,
|
||||
{
|
||||
if self.mrf_normal_dispatches_since_best_effort < MRF_NORMAL_DISPATCH_QUANTUM {
|
||||
let (chosen, skipped) = self.pop_runnable_with_skips(can_run, skip_label);
|
||||
if let Some(request) = chosen.as_ref() {
|
||||
self.record_dispatch(request);
|
||||
}
|
||||
return (chosen, skipped);
|
||||
}
|
||||
|
||||
if self.best_effort_queued == 0 {
|
||||
self.mrf_normal_dispatches_since_best_effort = 0;
|
||||
let (chosen, skipped) = self.pop_runnable_with_skips(can_run, skip_label);
|
||||
if let Some(request) = chosen.as_ref() {
|
||||
self.record_dispatch(request);
|
||||
}
|
||||
return (chosen, skipped);
|
||||
}
|
||||
|
||||
let mut items = std::mem::take(&mut self.heap).into_vec();
|
||||
let mut skipped = Vec::new();
|
||||
let mut first_runnable: Option<usize> = None;
|
||||
let mut best_effort_candidates: [Option<usize>; 3] = [None; 3];
|
||||
for (index, item) in items.iter().enumerate() {
|
||||
if !can_run(&item.request) {
|
||||
if let Some(label) = skip_label(&item.request) {
|
||||
skipped.push(label);
|
||||
}
|
||||
continue;
|
||||
}
|
||||
if first_runnable.is_none_or(|current| item.cmp(&items[current]).is_gt()) {
|
||||
first_runnable = Some(index);
|
||||
}
|
||||
let Some(source_index) = best_effort_source_index(item.request.source) else {
|
||||
continue;
|
||||
};
|
||||
if best_effort_candidates[source_index].is_none_or(|current| {
|
||||
item.priority > items[current].priority
|
||||
|| (item.priority == items[current].priority && item.sequence < items[current].sequence)
|
||||
}) {
|
||||
best_effort_candidates[source_index] = Some(index);
|
||||
}
|
||||
}
|
||||
|
||||
let chosen_index = first_runnable.filter(|index| {
|
||||
let item = &items[*index];
|
||||
item.priority == HealPriority::Normal
|
||||
&& matches!(item.request.source, HealRequestSource::Mrf | HealRequestSource::Internal)
|
||||
});
|
||||
let best_effort_priority = best_effort_candidates
|
||||
.into_iter()
|
||||
.flatten()
|
||||
.map(|index| items[index].priority)
|
||||
.max();
|
||||
let chosen_index = chosen_index
|
||||
.and_then(|_| {
|
||||
let priority = best_effort_priority?;
|
||||
best_effort_source_order(self.last_best_effort_source)
|
||||
.into_iter()
|
||||
.find_map(|source| best_effort_candidates[source].filter(|index| items[*index].priority == priority))
|
||||
})
|
||||
.or(first_runnable);
|
||||
let chosen = chosen_index.map(|index| items.swap_remove(index));
|
||||
self.heap = BinaryHeap::from(items);
|
||||
|
||||
let chosen = chosen.map(|item| {
|
||||
let request = item.request;
|
||||
Self::decrement_or_remove_dedup_key(&mut self.dedup_keys, &item.dedup_key);
|
||||
self.remove_best_effort_count(request.source);
|
||||
self.record_dispatch(&request);
|
||||
request
|
||||
});
|
||||
(chosen, skipped)
|
||||
}
|
||||
|
||||
fn record_dispatch(&mut self, request: &HealRequest) {
|
||||
if is_best_effort_source(request.source) {
|
||||
self.mrf_normal_dispatches_since_best_effort = 0;
|
||||
self.last_best_effort_source = Some(request.source);
|
||||
} else if matches!(request.source, HealRequestSource::Mrf | HealRequestSource::Internal)
|
||||
&& request.priority == HealPriority::Normal
|
||||
{
|
||||
self.mrf_normal_dispatches_since_best_effort = self.mrf_normal_dispatches_since_best_effort.saturating_add(1);
|
||||
}
|
||||
}
|
||||
|
||||
fn remove_best_effort_count(&mut self, source: HealRequestSource) {
|
||||
if is_best_effort_source(source) {
|
||||
self.best_effort_queued = self.best_effort_queued.saturating_sub(1);
|
||||
if let Some(index) = best_effort_source_index(source) {
|
||||
self.best_effort_queued_by_source[index] = self.best_effort_queued_by_source[index].saturating_sub(1);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
pub(super) fn best_effort_source_count(&self, source: HealRequestSource) -> usize {
|
||||
best_effort_source_index(source).map_or(0, |index| self.best_effort_queued_by_source[index])
|
||||
}
|
||||
|
||||
fn restore_deferred_items(&mut self, deferred: Vec<PriorityQueueItem>) {
|
||||
if deferred.is_empty() {
|
||||
return;
|
||||
@@ -590,6 +735,7 @@ impl PriorityHealQueue {
|
||||
if removed.is_none() && item.request.id == request_id {
|
||||
let key = item.dedup_key.clone();
|
||||
Self::decrement_or_remove_dedup_key(&mut self.dedup_keys, &key);
|
||||
self.remove_best_effort_count(item.request.source);
|
||||
affected_key = Some(key);
|
||||
removed = Some(item.request);
|
||||
} else {
|
||||
@@ -615,6 +761,7 @@ impl PriorityHealQueue {
|
||||
while let Some(item) = self.heap.pop() {
|
||||
if should_remove(&item.request) {
|
||||
Self::decrement_or_remove_dedup_key(&mut self.dedup_keys, &item.dedup_key);
|
||||
self.remove_best_effort_count(item.request.source);
|
||||
affected_keys.push(item.dedup_key);
|
||||
removed.push(item.request);
|
||||
} else {
|
||||
|
||||
@@ -150,7 +150,7 @@ impl HealManager {
|
||||
for _ in 0..available_slots {
|
||||
let selected_request = if config.set_bulkhead_enable || mainline_pressure.is_some() {
|
||||
let max_concurrent_per_set = config.max_concurrent_per_set;
|
||||
let (selected_request, skipped_sets) = queue.pop_runnable_with_skips(
|
||||
let (selected_request, skipped_sets) = queue.pop_runnable_with_fairness(
|
||||
|request| {
|
||||
let set_allowed = !config.set_bulkhead_enable
|
||||
|| can_schedule_request(request, &running_per_set, max_concurrent_per_set);
|
||||
@@ -164,7 +164,7 @@ impl HealManager {
|
||||
}
|
||||
selected_request
|
||||
} else {
|
||||
queue.pop_next()
|
||||
queue.pop_runnable_with_fairness(|_| true, |_| None).0
|
||||
};
|
||||
|
||||
if let Some(mut request) = selected_request {
|
||||
|
||||
@@ -1488,6 +1488,164 @@ fn test_priority_queue_ordering() {
|
||||
assert_eq!(queue.len(), 0);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_priority_queue_fairness_gives_best_effort_bounded_service() {
|
||||
let mut queue = PriorityHealQueue::new();
|
||||
for index in 0..5 {
|
||||
assert_eq!(
|
||||
queue.push(bucket_request(&format!("mrf-{index}"), HealPriority::Normal, HealRequestSource::Mrf)),
|
||||
QueuePushOutcome::Accepted
|
||||
);
|
||||
}
|
||||
assert_eq!(
|
||||
queue.push(bucket_request("scanner", HealPriority::Low, HealRequestSource::Scanner)),
|
||||
QueuePushOutcome::Accepted
|
||||
);
|
||||
|
||||
for _ in 0..4 {
|
||||
let (request, skipped) = queue.pop_runnable_with_fairness(|_| true, |_| None);
|
||||
assert!(skipped.is_empty());
|
||||
assert_eq!(
|
||||
request.expect("MRF request should run before the fairness quantum").source,
|
||||
HealRequestSource::Mrf
|
||||
);
|
||||
}
|
||||
|
||||
let (request, skipped) = queue.pop_runnable_with_fairness(|_| true, |_| None);
|
||||
assert!(skipped.is_empty());
|
||||
assert_eq!(
|
||||
request
|
||||
.expect("best-effort work must get a bounded service opportunity")
|
||||
.source,
|
||||
HealRequestSource::Scanner
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_priority_queue_fairness_covers_internal_normal_background_work() {
|
||||
let mut queue = PriorityHealQueue::new();
|
||||
for index in 0..5 {
|
||||
assert_eq!(
|
||||
queue.push(bucket_request(
|
||||
&format!("internal-{index}"),
|
||||
HealPriority::Normal,
|
||||
HealRequestSource::Internal
|
||||
)),
|
||||
QueuePushOutcome::Accepted
|
||||
);
|
||||
}
|
||||
assert_eq!(
|
||||
queue.push(bucket_request("scanner", HealPriority::Low, HealRequestSource::Scanner)),
|
||||
QueuePushOutcome::Accepted
|
||||
);
|
||||
|
||||
for _ in 0..4 {
|
||||
let (request, _) = queue.pop_runnable_with_fairness(|_| true, |_| None);
|
||||
assert_eq!(request.expect("internal background work should run").source, HealRequestSource::Internal);
|
||||
}
|
||||
let (request, _) = queue.pop_runnable_with_fairness(|_| true, |_| None);
|
||||
assert_eq!(
|
||||
request.expect("Scanner should get a service opportunity").source,
|
||||
HealRequestSource::Scanner
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_priority_queue_fairness_round_robins_best_effort_sources() {
|
||||
let mut queue = PriorityHealQueue::new();
|
||||
for index in 0..8 {
|
||||
assert_eq!(
|
||||
queue.push(bucket_request(&format!("mrf-{index}"), HealPriority::Normal, HealRequestSource::Mrf)),
|
||||
QueuePushOutcome::Accepted
|
||||
);
|
||||
}
|
||||
for (bucket, source) in [
|
||||
("scanner", HealRequestSource::Scanner),
|
||||
("read-repair", HealRequestSource::ReadRepair),
|
||||
("auto-heal", HealRequestSource::AutoHeal),
|
||||
] {
|
||||
assert_eq!(queue.push(bucket_request(bucket, HealPriority::Low, source)), QueuePushOutcome::Accepted);
|
||||
}
|
||||
|
||||
for _ in 0..4 {
|
||||
let (request, _) = queue.pop_runnable_with_fairness(|_| true, |_| None);
|
||||
assert_eq!(request.expect("first MRF quantum should run").source, HealRequestSource::Mrf);
|
||||
}
|
||||
let (request, _) = queue.pop_runnable_with_fairness(|_| true, |_| None);
|
||||
assert_eq!(
|
||||
request.expect("Scanner should get the first best-effort turn").source,
|
||||
HealRequestSource::Scanner
|
||||
);
|
||||
|
||||
for _ in 0..4 {
|
||||
let (request, _) = queue.pop_runnable_with_fairness(|_| true, |_| None);
|
||||
assert_eq!(request.expect("second MRF quantum should run").source, HealRequestSource::Mrf);
|
||||
}
|
||||
let (request, _) = queue.pop_runnable_with_fairness(|_| true, |_| None);
|
||||
assert_eq!(
|
||||
request.expect("ReadRepair should get the next best-effort turn").source,
|
||||
HealRequestSource::ReadRepair
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_priority_queue_fairness_keeps_best_effort_priority_order() {
|
||||
let mut queue = PriorityHealQueue::new();
|
||||
for index in 0..4 {
|
||||
assert_eq!(
|
||||
queue.push(bucket_request(&format!("mrf-{index}"), HealPriority::Normal, HealRequestSource::Mrf)),
|
||||
QueuePushOutcome::Accepted
|
||||
);
|
||||
}
|
||||
assert_eq!(
|
||||
queue.push(bucket_request("scanner", HealPriority::Low, HealRequestSource::Scanner)),
|
||||
QueuePushOutcome::Accepted
|
||||
);
|
||||
assert_eq!(
|
||||
queue.push(bucket_request("read-repair", HealPriority::Normal, HealRequestSource::ReadRepair)),
|
||||
QueuePushOutcome::Accepted
|
||||
);
|
||||
|
||||
for _ in 0..4 {
|
||||
let (request, _) = queue.pop_runnable_with_fairness(|_| true, |_| None);
|
||||
assert_eq!(request.expect("MRF request should run").source, HealRequestSource::Mrf);
|
||||
}
|
||||
let (request, _) = queue.pop_runnable_with_fairness(|_| true, |_| None);
|
||||
assert_eq!(
|
||||
request.expect("higher-priority best-effort work should run first").source,
|
||||
HealRequestSource::ReadRepair
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_priority_queue_fairness_skips_blocked_best_effort_source() {
|
||||
let mut queue = PriorityHealQueue::new();
|
||||
for index in 0..4 {
|
||||
assert_eq!(
|
||||
queue.push(bucket_request(&format!("mrf-{index}"), HealPriority::Normal, HealRequestSource::Mrf)),
|
||||
QueuePushOutcome::Accepted
|
||||
);
|
||||
}
|
||||
assert_eq!(
|
||||
queue.push(bucket_request("scanner", HealPriority::Low, HealRequestSource::Scanner)),
|
||||
QueuePushOutcome::Accepted
|
||||
);
|
||||
assert_eq!(
|
||||
queue.push(bucket_request("read-repair", HealPriority::Low, HealRequestSource::ReadRepair)),
|
||||
QueuePushOutcome::Accepted
|
||||
);
|
||||
|
||||
for _ in 0..4 {
|
||||
let (request, _) = queue.pop_runnable_with_fairness(|_| true, |_| None);
|
||||
assert_eq!(request.expect("MRF request should run").source, HealRequestSource::Mrf);
|
||||
}
|
||||
let (request, _) = queue.pop_runnable_with_fairness(|request| request.source != HealRequestSource::Scanner, |_| None);
|
||||
assert_eq!(
|
||||
request.expect("runnable ReadRepair should bypass blocked Scanner").source,
|
||||
HealRequestSource::ReadRepair
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_priority_queue_fifo_same_priority() {
|
||||
let mut queue = PriorityHealQueue::new();
|
||||
@@ -4500,7 +4658,7 @@ async fn test_displacing_registered_mrf_task_drops_notice_ownership() {
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_submit_heal_request_drops_read_repair_under_pressure() {
|
||||
async fn test_submit_heal_request_admits_read_repair_under_pressure() {
|
||||
let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage);
|
||||
let manager = HealManager::new_without_root_recovery_for_test(
|
||||
storage,
|
||||
@@ -4510,7 +4668,7 @@ async fn test_submit_heal_request_drops_read_repair_under_pressure() {
|
||||
}),
|
||||
);
|
||||
|
||||
for index in 0..8 {
|
||||
for index in 0..7 {
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request(
|
||||
@@ -4529,12 +4687,12 @@ async fn test_submit_heal_request_drops_read_repair_under_pressure() {
|
||||
.await
|
||||
.expect("read repair admission should return a result");
|
||||
|
||||
assert_eq!(admission, HealAdmissionResult::Dropped(HealAdmissionDropReason::PolicyDropped));
|
||||
assert_eq!(admission, HealAdmissionResult::Accepted);
|
||||
assert_eq!(manager.get_queue_length().await, 8);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_submit_heal_request_drops_low_scanner_under_pressure() {
|
||||
async fn test_submit_heal_request_admits_low_scanner_under_pressure() {
|
||||
let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage);
|
||||
let manager = HealManager::new_without_root_recovery_for_test(
|
||||
storage,
|
||||
@@ -4544,7 +4702,7 @@ async fn test_submit_heal_request_drops_low_scanner_under_pressure() {
|
||||
}),
|
||||
);
|
||||
|
||||
for index in 0..8 {
|
||||
for index in 0..7 {
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request(
|
||||
@@ -4563,10 +4721,220 @@ async fn test_submit_heal_request_drops_low_scanner_under_pressure() {
|
||||
.await
|
||||
.expect("scanner admission should return a result");
|
||||
|
||||
assert_eq!(admission, HealAdmissionResult::Dropped(HealAdmissionDropReason::PolicyDropped));
|
||||
assert_eq!(admission, HealAdmissionResult::Accepted);
|
||||
assert_eq!(manager.get_queue_length().await, 8);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_mrf_admission_preserves_best_effort_queue_reserve() {
|
||||
let manager = HealManager::new_without_root_recovery_for_test(
|
||||
Arc::new(MockStorage),
|
||||
Some(HealConfig {
|
||||
queue_size: 10,
|
||||
..HealConfig::default()
|
||||
}),
|
||||
);
|
||||
|
||||
for index in 0..7 {
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request(&format!("mrf-{index}"), HealPriority::Normal, HealRequestSource::Mrf,))
|
||||
.await
|
||||
.expect("MRF request should be admitted before the reserve threshold"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
}
|
||||
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request("mrf-deferred", HealPriority::Normal, HealRequestSource::Mrf))
|
||||
.await
|
||||
.expect("MRF admission should return a typed result"),
|
||||
HealAdmissionResult::Full
|
||||
);
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request("internal-deferred", HealPriority::Normal, HealRequestSource::Internal))
|
||||
.await
|
||||
.expect("ordinary background admission should return a typed result"),
|
||||
HealAdmissionResult::Full
|
||||
);
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request("scanner-reserved", HealPriority::Low, HealRequestSource::Scanner))
|
||||
.await
|
||||
.expect("Scanner should consume the reserved slot"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request("read-repair-reserved", HealPriority::Low, HealRequestSource::ReadRepair))
|
||||
.await
|
||||
.expect("ReadRepair should consume its reserved slot"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request("auto-heal-reserved", HealPriority::Low, HealRequestSource::AutoHeal))
|
||||
.await
|
||||
.expect("AutoHeal should consume its reserved slot"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
assert_eq!(manager.get_queue_length().await, 10);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_small_queue_keeps_per_source_best_effort_slots() {
|
||||
let manager = HealManager::new_without_root_recovery_for_test(
|
||||
Arc::new(MockStorage),
|
||||
Some(HealConfig {
|
||||
queue_size: 9,
|
||||
..HealConfig::default()
|
||||
}),
|
||||
);
|
||||
|
||||
for index in 0..6 {
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request(&format!("mrf-{index}"), HealPriority::Normal, HealRequestSource::Mrf))
|
||||
.await
|
||||
.expect("MRF request should be admitted before the per-source reserve"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
}
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request("mrf-deferred", HealPriority::Normal, HealRequestSource::Mrf))
|
||||
.await
|
||||
.expect("MRF admission should return a typed result"),
|
||||
HealAdmissionResult::Full
|
||||
);
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request("scanner-reserved", HealPriority::Low, HealRequestSource::Scanner))
|
||||
.await
|
||||
.expect("Scanner should consume its reserved slot"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request("read-repair-reserved", HealPriority::Low, HealRequestSource::ReadRepair))
|
||||
.await
|
||||
.expect("ReadRepair should consume its reserved slot"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request("auto-heal-reserved", HealPriority::Low, HealRequestSource::AutoHeal))
|
||||
.await
|
||||
.expect("AutoHeal should consume its reserved slot"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_queue_size_four_keeps_mrf_and_each_best_effort_source() {
|
||||
let manager = HealManager::new_without_root_recovery_for_test(
|
||||
Arc::new(MockStorage),
|
||||
Some(HealConfig {
|
||||
queue_size: 4,
|
||||
..HealConfig::default()
|
||||
}),
|
||||
);
|
||||
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request("mrf", HealPriority::Normal, HealRequestSource::Mrf))
|
||||
.await
|
||||
.expect("MRF should retain one queue slot"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request("scanner", HealPriority::Low, HealRequestSource::Scanner))
|
||||
.await
|
||||
.expect("Scanner should use its source slot"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request("read-repair", HealPriority::Low, HealRequestSource::ReadRepair))
|
||||
.await
|
||||
.expect("ReadRepair should use its source slot"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request("auto-heal", HealPriority::Low, HealRequestSource::AutoHeal))
|
||||
.await
|
||||
.expect("AutoHeal should use its source slot"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_high_priority_mrf_bypasses_best_effort_reserve() {
|
||||
let manager = HealManager::new_without_root_recovery_for_test(
|
||||
Arc::new(MockStorage),
|
||||
Some(HealConfig {
|
||||
queue_size: 10,
|
||||
..HealConfig::default()
|
||||
}),
|
||||
);
|
||||
|
||||
for index in 0..7 {
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request(&format!("mrf-{index}"), HealPriority::Normal, HealRequestSource::Mrf))
|
||||
.await
|
||||
.expect("normal MRF request should be admitted before the reserve"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
}
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request("urgent-mrf", HealPriority::Urgent, HealRequestSource::Mrf))
|
||||
.await
|
||||
.expect("urgent MRF admission should return a typed result"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_high_priority_best_effort_bypasses_source_quota() {
|
||||
let manager = HealManager::new_without_root_recovery_for_test(
|
||||
Arc::new(MockStorage),
|
||||
Some(HealConfig {
|
||||
queue_size: 10,
|
||||
..HealConfig::default()
|
||||
}),
|
||||
);
|
||||
|
||||
for index in 0..7 {
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request(&format!("mrf-{index}"), HealPriority::Normal, HealRequestSource::Mrf))
|
||||
.await
|
||||
.expect("normal MRF request should be admitted before the reserve"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
}
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request("scanner-low", HealPriority::Low, HealRequestSource::Scanner))
|
||||
.await
|
||||
.expect("low Scanner request should be admitted into its source slot"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request("scanner-high", HealPriority::High, HealRequestSource::Scanner))
|
||||
.await
|
||||
.expect("high Scanner retry should bypass its low-priority quota"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_submit_heal_request_accepts_admin_high_under_pressure() {
|
||||
let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage);
|
||||
@@ -4578,7 +4946,7 @@ async fn test_submit_heal_request_accepts_admin_high_under_pressure() {
|
||||
}),
|
||||
);
|
||||
|
||||
for index in 0..8 {
|
||||
for index in 0..7 {
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request(
|
||||
@@ -4598,7 +4966,56 @@ async fn test_submit_heal_request_accepts_admin_high_under_pressure() {
|
||||
.expect("admin admission should return a result");
|
||||
|
||||
assert_eq!(admission, HealAdmissionResult::Accepted);
|
||||
assert_eq!(manager.get_queue_length().await, 9);
|
||||
assert_eq!(manager.get_queue_length().await, 8);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_admin_normal_displaces_best_effort_when_queue_is_full() {
|
||||
let manager = HealManager::new_without_root_recovery_for_test(
|
||||
Arc::new(MockStorage),
|
||||
Some(HealConfig {
|
||||
queue_size: 10,
|
||||
..HealConfig::default()
|
||||
}),
|
||||
);
|
||||
|
||||
for index in 0..7 {
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request(&format!("mrf-{index}"), HealPriority::Normal, HealRequestSource::Mrf))
|
||||
.await
|
||||
.expect("MRF request should be queued"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
}
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request("scanner", HealPriority::Low, HealRequestSource::Scanner))
|
||||
.await
|
||||
.expect("Scanner request should be queued"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request("read-repair", HealPriority::Low, HealRequestSource::ReadRepair))
|
||||
.await
|
||||
.expect("ReadRepair request should be queued"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request("auto-heal", HealPriority::Low, HealRequestSource::AutoHeal))
|
||||
.await
|
||||
.expect("AutoHeal request should be queued"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
assert_eq!(
|
||||
manager
|
||||
.submit_heal_request(bucket_request("admin", HealPriority::Normal, HealRequestSource::Admin))
|
||||
.await
|
||||
.expect("admin admission should return a typed result"),
|
||||
HealAdmissionResult::Accepted
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
|
||||
@@ -532,6 +532,19 @@ pub trait HealStorageAPI: Send + Sync {
|
||||
Err(Error::other("storage does not support incarnation-bound object healing"))
|
||||
}
|
||||
|
||||
/// Durable MRF repair may request an authoritative unversioned absence proof.
|
||||
async fn heal_mrf_object_at_incarnation(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
version_id: Option<&str>,
|
||||
expected: Uuid,
|
||||
opts: &HealOpts,
|
||||
) -> Result<HealStorageObjectResult> {
|
||||
self.heal_object_at_incarnation(bucket, object, version_id, expected, opts)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Heal object using ecstore
|
||||
async fn heal_object(
|
||||
&self,
|
||||
@@ -578,6 +591,11 @@ pub trait HealStorageAPI: Send + Sync {
|
||||
Err(Error::other("replacement pool metadata placement is unsupported"))
|
||||
}
|
||||
|
||||
/// Heal and physically verify bucket configuration on its replacement targets.
|
||||
async fn heal_replacement_bucket_metadata(&self, _bucket: &str, _opts: &HealOpts, _targets: &[String]) -> Result<()> {
|
||||
Err(Error::Storage(StorageError::PreconditionFailed))
|
||||
}
|
||||
|
||||
/// Read target-specific physical evidence for one replacement version.
|
||||
///
|
||||
/// This is only used by automatic replacement healing after the normal
|
||||
@@ -967,6 +985,24 @@ impl HealStorageAPI for ECStoreHealStorage {
|
||||
.await)
|
||||
}
|
||||
|
||||
async fn heal_mrf_object_at_incarnation(
|
||||
&self,
|
||||
bucket: &str,
|
||||
object: &str,
|
||||
version_id: Option<&str>,
|
||||
expected: Uuid,
|
||||
opts: &HealOpts,
|
||||
) -> Result<HealStorageObjectResult> {
|
||||
let result = self
|
||||
.ecstore
|
||||
.heal_mrf_object_at_incarnation(bucket, object, version_id.unwrap_or_default(), expected, opts)
|
||||
.await
|
||||
.map_err(|error| incarnation_storage_error(bucket, expected, error))?;
|
||||
Ok(self
|
||||
.object_result_with_receipt(bucket, object, version_id, opts, result, Some(expected))
|
||||
.await)
|
||||
}
|
||||
|
||||
async fn get_object_meta(&self, bucket: &str, object: &str) -> Result<Option<HealObjectInfo>> {
|
||||
debug!(
|
||||
target: "rustfs::heal::storage",
|
||||
@@ -1564,6 +1600,13 @@ impl HealStorageAPI for ECStoreHealStorage {
|
||||
.map_err(Error::Storage)
|
||||
}
|
||||
|
||||
async fn heal_replacement_bucket_metadata(&self, bucket: &str, opts: &HealOpts, targets: &[String]) -> Result<()> {
|
||||
self.ecstore
|
||||
.heal_replacement_bucket_metadata(bucket, opts, targets)
|
||||
.await
|
||||
.map_err(Error::Storage)
|
||||
}
|
||||
|
||||
async fn replacement_targets_have_version(
|
||||
&self,
|
||||
bucket: &str,
|
||||
|
||||
@@ -298,6 +298,18 @@ impl HealTask {
|
||||
"Heal object stage entered"
|
||||
);
|
||||
let object_size = result.object_size as u64;
|
||||
let ok_drive_state = DriveState::Ok.to_string();
|
||||
if self.source == HealRequestSource::Admin
|
||||
&& !self.options.dry_run
|
||||
&& !result.after.drives.is_empty()
|
||||
&& result.after.drives.iter().any(|drive| drive.state != ok_drive_state)
|
||||
{
|
||||
return Err(Error::TaskExecutionFailed {
|
||||
message: format!(
|
||||
"Heal left one or more drives unhealthy for {bucket}/{object}; retry after the missing drives are restored"
|
||||
),
|
||||
});
|
||||
}
|
||||
debug!(
|
||||
target: "rustfs::heal::task",
|
||||
event = EVENT_HEAL_OBJECT_RESULT,
|
||||
@@ -417,6 +429,10 @@ impl HealTask {
|
||||
|
||||
/// Recreate missing object (for EC decode scenarios)
|
||||
async fn recreate_missing_object(&self, bucket: &str, object: &str, version_id: Option<&str>) -> Result<()> {
|
||||
if self.source == HealRequestSource::Mrf {
|
||||
return self.recreate_missing_mrf_object(bucket, object, version_id).await;
|
||||
}
|
||||
|
||||
debug!(
|
||||
target: "rustfs::heal::task",
|
||||
event = EVENT_HEAL_OBJECT_STAGE,
|
||||
@@ -517,4 +533,65 @@ impl HealTask {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Durable MRF responsibilities may complete only with an exact storage proof.
|
||||
async fn recreate_missing_mrf_object(&self, bucket: &str, object: &str, version_id: Option<&str>) -> Result<()> {
|
||||
let heal_opts = HealOpts {
|
||||
recursive: false,
|
||||
dry_run: self.options.dry_run,
|
||||
remove: false,
|
||||
recreate: true,
|
||||
scan_mode: HealScanMode::Deep,
|
||||
update_parity: true,
|
||||
no_lock: self.options.no_lock,
|
||||
read_repair: false,
|
||||
pool: self.options.pool_index,
|
||||
set: self.options.set_index,
|
||||
};
|
||||
let mut expected = self.outcome_identity(bucket, object, version_id, self.options.pool_index, self.options.set_index);
|
||||
let bucket_incarnation_id = self
|
||||
.outcome_bucket_incarnation_id(bucket, self.options.dry_run)
|
||||
.await?
|
||||
.ok_or_else(|| Error::TaskExecutionFailed {
|
||||
message: format!("Missing bucket incarnation for durable MRF repair {bucket}/{object}"),
|
||||
})?;
|
||||
expected.bucket_incarnation_id = Some(bucket_incarnation_id);
|
||||
|
||||
let storage_result = self
|
||||
.await_with_control(self.storage.heal_mrf_object_at_incarnation(
|
||||
bucket,
|
||||
object,
|
||||
version_id,
|
||||
bucket_incarnation_id,
|
||||
&heal_opts,
|
||||
))
|
||||
.await?;
|
||||
if let Some(error) = storage_result.error {
|
||||
return Err(Error::TaskExecutionFailed {
|
||||
message: format!("Failed to recreate missing object {bucket}/{object}: {error}"),
|
||||
});
|
||||
}
|
||||
|
||||
let object_size = storage_result.item.object_size as u64;
|
||||
let authoritatively_absent = matches!(
|
||||
storage_result.receipt.as_ref().map(|receipt| &receipt.disposition),
|
||||
Some(HealObjectDisposition::AuthoritativelyAbsent)
|
||||
);
|
||||
if !self.record_verified_storage_receipt(expected, storage_result.receipt).await {
|
||||
return Err(Error::TaskExecutionFailed {
|
||||
message: format!("Missing exact storage proof for durable MRF repair {bucket}/{object}"),
|
||||
});
|
||||
}
|
||||
|
||||
{
|
||||
let mut progress = self.progress.write().await;
|
||||
if authoritatively_absent {
|
||||
progress.update_object_progress(1, 0, 0, 1, 0);
|
||||
} else {
|
||||
progress.update_object_progress(1, 1, 0, 0, object_size);
|
||||
}
|
||||
}
|
||||
self.record_result_item(storage_result.item).await;
|
||||
Ok(())
|
||||
}
|
||||
}
|
||||
|
||||
@@ -3471,6 +3471,40 @@ async fn test_prefix_heal_lists_and_repairs_objects_under_prefix() {
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn admin_object_heal_fails_when_drive_remains_unhealthy() {
|
||||
let storage = Arc::new(MockStorage::default());
|
||||
storage.heal_object_outcomes.lock().expect("outcomes").insert(
|
||||
"object-a".to_string(),
|
||||
VecDeque::from([MockHealObjectOutcome::UnavailableDrive(DriveState::Missing)]),
|
||||
);
|
||||
let mut request = HealRequest::new(
|
||||
HealType::Object {
|
||||
bucket: "bucket-a".to_string(),
|
||||
object: "object-a".to_string(),
|
||||
version_id: None,
|
||||
},
|
||||
HealOptions {
|
||||
recreate_missing: false,
|
||||
timeout: None,
|
||||
..Default::default()
|
||||
},
|
||||
HealPriority::Normal,
|
||||
);
|
||||
request.source = HealRequestSource::Admin;
|
||||
let task = HealTask::from_request(request, storage);
|
||||
|
||||
let error = task
|
||||
.execute()
|
||||
.await
|
||||
.expect_err("admin heal must not finish with an unhealthy drive");
|
||||
assert!(error.to_string().contains("left one or more drives unhealthy"));
|
||||
let outcome = task.get_outcome().await;
|
||||
assert_eq!(outcome.counters.failed, 1);
|
||||
assert_eq!(outcome.counters.unknown, 0);
|
||||
assert_eq!(outcome.execution, HealExecutionOutcome::CompletedWithErrors);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_data_usage_cache_lock_timeout_does_not_fail_object_heal() {
|
||||
let storage = Arc::new(MockStorage::default());
|
||||
@@ -3855,6 +3889,67 @@ async fn test_heal_recreate_scanner_non_dir_not_found_fails() {
|
||||
assert!(matches!(task.get_status().await, HealTaskStatus::Failed { .. }));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn mrf_recreate_missing_object_records_exact_absence_receipt_with_scope() {
|
||||
let incarnation = Uuid::new_v4();
|
||||
let storage = Arc::new(MockStorage {
|
||||
object_exists: Mutex::new(Some(false)),
|
||||
bucket_incarnation_id: Mutex::new(Some(incarnation)),
|
||||
heal_object_receipts: Mutex::new(HashMap::from([(
|
||||
"deleted.bin".to_string(),
|
||||
VecDeque::from([HealObjectReceipt {
|
||||
identity: HealObjectIdentity {
|
||||
kind: HealObjectKind::Object,
|
||||
bucket: "bucket-a".to_string(),
|
||||
object: "deleted.bin".to_string(),
|
||||
version_id: None,
|
||||
bucket_incarnation_id: Some(incarnation),
|
||||
pool_index: Some(2),
|
||||
set_index: Some(3),
|
||||
},
|
||||
disposition: HealObjectDisposition::AuthoritativelyAbsent,
|
||||
}]),
|
||||
)])),
|
||||
..Default::default()
|
||||
});
|
||||
let mut request = HealRequest::new(
|
||||
HealType::Object {
|
||||
bucket: "bucket-a".to_string(),
|
||||
object: "deleted.bin".to_string(),
|
||||
version_id: None,
|
||||
},
|
||||
HealOptions {
|
||||
recreate_missing: true,
|
||||
pool_index: Some(2),
|
||||
set_index: Some(3),
|
||||
timeout: None,
|
||||
..Default::default()
|
||||
},
|
||||
HealPriority::Normal,
|
||||
);
|
||||
request.source = HealRequestSource::Mrf;
|
||||
let task = HealTask::from_request(request, storage.clone());
|
||||
|
||||
task.execute()
|
||||
.await
|
||||
.expect("a complete MRF absence receipt must complete the task");
|
||||
|
||||
let opts = storage.object_heal_opts.lock().unwrap()[0];
|
||||
assert_eq!(opts.pool, Some(2));
|
||||
assert_eq!(opts.set, Some(3));
|
||||
assert!(opts.recreate);
|
||||
assert_eq!(opts.scan_mode, HealScanMode::Deep);
|
||||
let outcome = task.get_outcome().await;
|
||||
assert_eq!(outcome.counters.unchanged, 1);
|
||||
assert_eq!(outcome.counters.unknown, 0);
|
||||
assert_eq!(outcome.objects.len(), 1);
|
||||
assert_eq!(outcome.objects[0].disposition, HealObjectDisposition::AuthoritativelyAbsent);
|
||||
assert_eq!(
|
||||
(outcome.objects[0].identity.pool_index, outcome.objects[0].identity.set_index),
|
||||
(Some(2), Some(3))
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn test_heal_scanner_missing_object_without_recreate_probes_storage() {
|
||||
let storage = Arc::new(MockStorage {
|
||||
|
||||
@@ -36,6 +36,7 @@ use rustfs_heal::heal::{
|
||||
};
|
||||
use rustfs_heal_contracts::heal_channel::{HealOpts, HealScanMode};
|
||||
use serial_test::serial;
|
||||
use sha2::{Digest, Sha256};
|
||||
use std::{
|
||||
path::{Path, PathBuf},
|
||||
sync::Arc,
|
||||
@@ -722,6 +723,91 @@ mod serial_tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
#[serial]
|
||||
async fn test_recursive_deep_heal_reports_stale_delete_marker_repair() {
|
||||
let (disk_paths, ecstore, storage) = heal_env().await;
|
||||
let manager = HealManager::new(
|
||||
storage.clone(),
|
||||
Some(HealConfig {
|
||||
heal_interval: Duration::from_millis(1),
|
||||
..Default::default()
|
||||
}),
|
||||
);
|
||||
manager.start().await.expect("heal manager should start");
|
||||
|
||||
let bucket = "b5-stale-delete-marker-recursive";
|
||||
let object = "obj.bin";
|
||||
create_versioned_bucket(&ecstore, bucket).await;
|
||||
put_unversioned(&ecstore, bucket, "control/object.bin", &versioned_test_data(42)).await;
|
||||
let historical_data = versioned_test_data(43);
|
||||
let historical = put_versioned(&ecstore, bucket, object, &historical_data).await;
|
||||
let target = &disk_paths[1];
|
||||
let target_meta = xl_meta_path(&object_dir(target, bucket, object));
|
||||
let stale_meta = std::fs::read(&target_meta).expect("target xl.meta must exist before marker creation");
|
||||
let stale_digest = Sha256::digest(&stale_meta);
|
||||
let marker = put_delete_marker(&ecstore, bucket, object).await;
|
||||
std::fs::write(&target_meta, &stale_meta).expect("restore stale target xl.meta");
|
||||
|
||||
let request = HealRequest::new(
|
||||
HealType::Bucket {
|
||||
bucket: bucket.to_string(),
|
||||
},
|
||||
HealOptions {
|
||||
recursive: true,
|
||||
scan_mode: HealScanMode::Deep,
|
||||
pool_index: Some(0),
|
||||
set_index: Some(0),
|
||||
..Default::default()
|
||||
},
|
||||
HealPriority::Normal,
|
||||
);
|
||||
let task_id = request.id.clone();
|
||||
assert!(
|
||||
manager
|
||||
.submit_heal_request(request)
|
||||
.await
|
||||
.expect("submit recursive heal")
|
||||
.is_admitted()
|
||||
);
|
||||
wait_for_task(&manager, &task_id, Duration::from_secs(60)).await;
|
||||
|
||||
let report = manager.get_task_report(&task_id).await.expect("completed task report");
|
||||
let outcome = report.outcome.expect("recursive task must have an outcome");
|
||||
assert_eq!(outcome.counters.failed, 0);
|
||||
assert_eq!(outcome.counters.unknown, 0);
|
||||
assert_eq!(outcome.counters.processed, 3);
|
||||
assert_eq!(outcome.counters.healed, 1, "delete marker repair must be counted as healed");
|
||||
assert_eq!(outcome.counters.unchanged, 2);
|
||||
|
||||
let repaired_meta = std::fs::read(&target_meta).expect("repaired target xl.meta must exist");
|
||||
assert_ne!(
|
||||
Sha256::digest(&repaired_meta),
|
||||
stale_digest,
|
||||
"deep heal must replace stale rejoined metadata"
|
||||
);
|
||||
|
||||
let marker_info = physical_version(target, bucket, object, &marker);
|
||||
assert!(
|
||||
marker_info.deleted && marker_info.is_latest,
|
||||
"target metadata must converge to delete-marker latest"
|
||||
);
|
||||
let historical_info = physical_version(target, bucket, object, &historical);
|
||||
assert!(!historical_info.is_latest, "historical version must no longer be marked latest");
|
||||
assert_eq!(
|
||||
read_version(&ecstore, bucket, object, &historical).await,
|
||||
historical_data,
|
||||
"historical version must remain readable after marker convergence"
|
||||
);
|
||||
let marker_receipt = outcome
|
||||
.objects
|
||||
.iter()
|
||||
.find(|item| item.identity.version_id.as_deref() == Some(marker.as_str()))
|
||||
.expect("deep heal must emit a receipt for the repaired delete marker");
|
||||
assert_eq!(marker_receipt.disposition, HealObjectDisposition::Repaired);
|
||||
manager.stop().await.expect("heal manager should stop");
|
||||
}
|
||||
|
||||
/// A missing xl.meta must produce an exact marker repair receipt, while a
|
||||
/// healthy replay must prove metadata health without claiming payload integrity.
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
|
||||
@@ -717,6 +717,49 @@ mod absence_receipt_regressions {
|
||||
assert_versions(&store, bucket, &old, ¤t).await;
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
|
||||
#[serial]
|
||||
async fn unversioned_missing_object_produces_fenced_absence_receipt() {
|
||||
let bucket = "absence-receipt-unversioned";
|
||||
let object = "deleted.bin";
|
||||
let (_paths, store, storage) = heal_env_n(16).await;
|
||||
store
|
||||
.make_bucket(bucket, &MakeBucketOptions::default())
|
||||
.await
|
||||
.expect("create unversioned bucket");
|
||||
let incarnation = storage
|
||||
.admit_bucket_incarnation(bucket)
|
||||
.await
|
||||
.expect("admit bucket incarnation for the durable proof");
|
||||
|
||||
let result = storage
|
||||
.heal_mrf_object_at_incarnation(
|
||||
bucket,
|
||||
object,
|
||||
None,
|
||||
incarnation,
|
||||
&HealOpts {
|
||||
pool: Some(0),
|
||||
set: Some(0),
|
||||
..deep_heal_opts()
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("fenced absence probe should return a storage result");
|
||||
assert!(
|
||||
result.error.is_none(),
|
||||
"complete unversioned absence must not remain an error: {:?}",
|
||||
result.error
|
||||
);
|
||||
let receipt = result.receipt.expect("complete unversioned absence needs a receipt");
|
||||
assert_eq!(receipt.disposition, HealObjectDisposition::AuthoritativelyAbsent);
|
||||
assert_eq!(receipt.identity.bucket, bucket);
|
||||
assert_eq!(receipt.identity.object, object);
|
||||
assert!(receipt.identity.version_id.is_none());
|
||||
assert_eq!((receipt.identity.pool_index, receipt.identity.set_index), (Some(0), Some(0)));
|
||||
assert_eq!(receipt.identity.bucket_incarnation_id, Some(incarnation));
|
||||
}
|
||||
|
||||
#[test]
|
||||
#[serial]
|
||||
fn historical_absence_receipt_bucket_outcome_matches_c06() {
|
||||
|
||||
@@ -12,7 +12,7 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
//! Regression for replacement healing requiring pool.bin on a non-owning set.
|
||||
//! Replacement healing must restore internal records only on their owning sets.
|
||||
#![recursion_limit = "256"]
|
||||
|
||||
use http::HeaderMap;
|
||||
@@ -101,6 +101,31 @@ async fn replacement_pool_metadata_follows_real_two_set_placement() {
|
||||
storage.replacement_pool_metadata_required(&opts).is_err(),
|
||||
"invalid scope must fail closed"
|
||||
);
|
||||
assert!(
|
||||
storage
|
||||
.heal_replacement_bucket_metadata(bucket, &opts, &[paths[0].to_string_lossy().into_owned()])
|
||||
.await
|
||||
.is_err(),
|
||||
"bucket metadata repair must reject an incomplete or invalid replacement scope"
|
||||
);
|
||||
}
|
||||
|
||||
for targets in [vec![], vec![paths[4].to_string_lossy().into_owned()]] {
|
||||
assert!(
|
||||
storage
|
||||
.heal_replacement_bucket_metadata(
|
||||
bucket,
|
||||
&HealOpts {
|
||||
pool: Some(0),
|
||||
set: Some(0),
|
||||
..Default::default()
|
||||
},
|
||||
&targets,
|
||||
)
|
||||
.await
|
||||
.is_err(),
|
||||
"empty or wrong-set targets must be rejected before repairing metadata"
|
||||
);
|
||||
}
|
||||
|
||||
let mut owning_set = None;
|
||||
@@ -120,6 +145,17 @@ async fn replacement_pool_metadata_follows_real_two_set_placement() {
|
||||
assert!(owning_set.replace(set).is_none(), "only one set owns pool.bin");
|
||||
std::fs::remove_dir_all(&metadata_path).unwrap();
|
||||
}
|
||||
let bucket_records: Vec<_> = [".metadata.bin", ".bucket-incarnation"]
|
||||
.into_iter()
|
||||
.map(|file| {
|
||||
let path = target_path.join(RUSTFS_META_BUCKET).join("buckets").join(bucket).join(file);
|
||||
let existed = path.join("xl.meta").exists();
|
||||
if existed {
|
||||
std::fs::remove_dir_all(&path).expect("remove bucket metadata shard from replacement");
|
||||
}
|
||||
(path, existed)
|
||||
})
|
||||
.collect();
|
||||
// Select a user key using the real placement algorithm, independently
|
||||
// of the metadata-scope decision under test.
|
||||
let key = (0..1000)
|
||||
@@ -191,6 +227,14 @@ async fn replacement_pool_metadata_follows_real_two_set_placement() {
|
||||
"completed repair must clear its healing marker"
|
||||
);
|
||||
assert_eq!(metadata_path.join("xl.meta").exists(), owns_metadata);
|
||||
for (path, owned) in bucket_records {
|
||||
assert_eq!(
|
||||
path.join("xl.meta").exists(),
|
||||
owned,
|
||||
"replacement must restore each owned bucket record without creating a wrong-set copy: {}",
|
||||
path.display()
|
||||
);
|
||||
}
|
||||
for (relative, bytes) in original_parts {
|
||||
assert_eq!(std::fs::read(object_path.join(relative)).unwrap(), bytes, "reconstructed shard differs");
|
||||
}
|
||||
|
||||
@@ -90,6 +90,82 @@ async fn partial_write_persistence_failure_is_reported_and_retained_for_retry()
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn unversioned_deleted_partial_write_is_discharged_by_an_absence_proof() {
|
||||
const STACK_SIZE: usize = 8 * 1024 * 1024;
|
||||
std::thread::Builder::new()
|
||||
.name("mrf-partial-write-absence".to_owned())
|
||||
.stack_size(STACK_SIZE)
|
||||
.spawn(|| {
|
||||
let runtime = tokio::runtime::Builder::new_current_thread()
|
||||
.thread_stack_size(STACK_SIZE)
|
||||
.enable_all()
|
||||
.build()
|
||||
.expect("partial-write absence runtime should build");
|
||||
runtime.block_on(unversioned_deleted_partial_write_is_discharged_by_an_absence_proof_inner());
|
||||
})
|
||||
.expect("partial-write absence test thread should spawn")
|
||||
.join()
|
||||
.expect("partial-write absence test thread should finish");
|
||||
}
|
||||
|
||||
async fn unversioned_deleted_partial_write_is_discharged_by_an_absence_proof_inner() {
|
||||
use rustfs_common::mrf_channel::{MrfScope, persist_partial_write_intent};
|
||||
|
||||
temp_env::async_with_vars([("RUSTFS_HEAL_MRF_ENABLE", Some("true"))], async {
|
||||
let root = tempfile::tempdir().expect("partial-write absence fixture directory");
|
||||
let env = TestECStoreEnv::builder().disk_count(16).base_dir(root.path()).build().await;
|
||||
env.make_bucket("partial-absence", false).await;
|
||||
let mut coordinator_pool = env.endpoint_pools.as_ref()[0].clone();
|
||||
let mut endpoints = coordinator_pool.endpoints.as_ref().to_vec();
|
||||
for endpoint in endpoints.iter_mut().skip(4) {
|
||||
endpoint.is_local = false;
|
||||
}
|
||||
coordinator_pool.endpoints = Endpoints::from(endpoints);
|
||||
init_local_disks(EndpointServerPools::from(vec![coordinator_pool]))
|
||||
.await
|
||||
.expect("coordinator journal disks");
|
||||
|
||||
let manager = manager(&env);
|
||||
mrf_queue::spawn_mrf_consumer(manager.clone());
|
||||
for (object, version_id) in [
|
||||
("deleted-unversioned.bin", None),
|
||||
("deleted-versioned.bin", Some(uuid::Uuid::new_v4())),
|
||||
] {
|
||||
persist_partial_write_intent(
|
||||
"partial-absence",
|
||||
object,
|
||||
version_id,
|
||||
MrfScope {
|
||||
pool_index: 0,
|
||||
set_index: 0,
|
||||
},
|
||||
)
|
||||
.await
|
||||
.expect("durable partial-write responsibility must commit before scheduling");
|
||||
assert!(snapshot_contains(object).await, "committed responsibility must exist before repair runs");
|
||||
}
|
||||
|
||||
manager.start().await.expect("MRF scheduler should start");
|
||||
for object in ["deleted-unversioned.bin", "deleted-versioned.bin"] {
|
||||
assert!(
|
||||
wait_until(|| async { !snapshot_contains(object).await }).await,
|
||||
"complete absence proof must discharge the durable responsibility"
|
||||
);
|
||||
}
|
||||
assert!(
|
||||
wait_until(|| async {
|
||||
let snapshot = manager.operations_snapshot().await;
|
||||
snapshot.queue_length == 0 && snapshot.active_tasks == 0
|
||||
})
|
||||
.await,
|
||||
"discharged absence repair must leave no queued work"
|
||||
);
|
||||
manager.stop().await.expect("absence manager should stop");
|
||||
})
|
||||
.await;
|
||||
}
|
||||
|
||||
async fn wait_until<F: FnMut() -> Fut, Fut: Future<Output = bool>>(mut probe: F) -> bool {
|
||||
tokio::time::timeout(Duration::from_secs(60), async {
|
||||
loop {
|
||||
|
||||
@@ -69,6 +69,15 @@ pub fn record_remote_lock_rpc_timeout(peer: &str, op: &'static str) {
|
||||
counter!("rustfs_remote_lock_rpc_timeouts_total", "peer" => peer.to_string(), "op" => op).increment(1);
|
||||
}
|
||||
|
||||
/// Record a lock acquisition rejected by the per-peer request breaker or
|
||||
/// in-flight admission limit.
|
||||
#[inline(always)]
|
||||
pub fn record_remote_lock_request_suppressed(peer: &str, op: &'static str, reason: &'static str) {
|
||||
use metrics::counter;
|
||||
counter!("rustfs_remote_lock_request_suppressed_total", "peer" => peer.to_string(), "op" => op, "reason" => reason)
|
||||
.increment(1);
|
||||
}
|
||||
|
||||
/// Record the cached lock channel to `peer` being evicted after an RPC failure.
|
||||
#[inline(always)]
|
||||
pub fn record_remote_lock_channel_eviction(peer: &str, trigger: &'static str) {
|
||||
@@ -229,6 +238,7 @@ mod tests {
|
||||
record_early_release();
|
||||
record_contention_event();
|
||||
record_remote_lock_rpc_timeout("http://peer:9000", "lock");
|
||||
record_remote_lock_request_suppressed("http://peer:9000", "lock", "breaker_open");
|
||||
record_remote_lock_channel_eviction("http://peer:9000", "timeout");
|
||||
record_remote_lock_channel_eviction_suppressed("http://peer:9000", "peer_recently_served");
|
||||
record_remote_lock_rpc_detached("lock", "detached");
|
||||
@@ -251,6 +261,7 @@ mod tests {
|
||||
"rustfs_lock_early_releases",
|
||||
"rustfs_lock_contentions",
|
||||
"rustfs_remote_lock_rpc_timeouts_total",
|
||||
"rustfs_remote_lock_request_suppressed_total",
|
||||
"rustfs_remote_lock_channel_evictions_total",
|
||||
"rustfs_remote_lock_channel_evictions_suppressed_total",
|
||||
"rustfs_remote_lock_rpc_detached_total",
|
||||
|
||||
@@ -241,7 +241,9 @@ impl RuleValidate for LifecycleRule {
|
||||
return Err(std::io::Error::other(ERR_LIFECYCLE_INVALID_DEL_MARKER_EXPIRATION_DAYS));
|
||||
}
|
||||
// Rule must have at least one action
|
||||
let has_expiration = self.expiration.is_some();
|
||||
let has_expiration = self.expiration.as_ref().is_some_and(|expiration| {
|
||||
expiration.days.is_some() || expiration.date.is_some() || expiration.expired_object_delete_marker.is_some()
|
||||
});
|
||||
let has_transition = self.transitions.as_ref().is_some_and(|t| !t.is_empty());
|
||||
// `NewerNoncurrentVersions` on its own is a MinIO extension, not an AWS
|
||||
// form: it keeps the newest N noncurrent versions and expires the rest
|
||||
@@ -4455,6 +4457,19 @@ mod tests {
|
||||
.expect_err("a negative retention count on a transition must be rejected");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn validate_rejects_empty_expiration_action() {
|
||||
let mut rule = rule_with_filter(LifecycleRuleFilter::default());
|
||||
rule.expiration = Some(LifecycleExpiration::default());
|
||||
|
||||
let err = config_with_rules(vec![rule])
|
||||
.validate(&ObjectLockConfiguration::default())
|
||||
.await
|
||||
.expect_err("an empty Expiration object must not count as an action");
|
||||
|
||||
assert_eq!(err.to_string(), ERR_LIFECYCLE_RULE_MUST_HAVE_ACTION);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn zero_newer_noncurrent_versions_means_no_count_constraint() {
|
||||
// Zero carries no constraint, matching how the batch limit path has
|
||||
|
||||
+146
-24
@@ -13,12 +13,11 @@
|
||||
// limitations under the License.
|
||||
|
||||
use std::collections::HashMap;
|
||||
use std::hash::{Hash, Hasher};
|
||||
use std::sync::Arc;
|
||||
use std::sync::atomic::{AtomicBool, Ordering};
|
||||
use std::time::{Duration, SystemTime};
|
||||
use tokio::sync::RwLock;
|
||||
use tokio::time::Instant;
|
||||
use tokio::time::{Instant, timeout};
|
||||
|
||||
use crate::{
|
||||
FastLockGuard, GlobalLockManager, LockClient, LockId, LockInfo, LockManager, LockMetadata, LockPriority, LockRequest,
|
||||
@@ -131,35 +130,52 @@ impl LocalClient {
|
||||
self.manager.clone().unwrap_or_else(crate::get_global_lock_manager)
|
||||
}
|
||||
|
||||
/// Get the shard index for a given lock ID
|
||||
fn get_shard_index(&self, lock_id: &LockId) -> usize {
|
||||
let mut hasher = std::collections::hash_map::DefaultHasher::new();
|
||||
lock_id.hash(&mut hasher);
|
||||
(hasher.finish() as usize) & self.shard_mask
|
||||
/// Guard entries for one resource share a shard so expiration checks do not
|
||||
/// need to take every shard's write lock on the acquisition hot path.
|
||||
fn get_resource_shard_index(&self, resource: &crate::ObjectKey) -> usize {
|
||||
resource.shard_index(self.shard_mask)
|
||||
}
|
||||
|
||||
/// Get the shard for a given lock ID
|
||||
fn get_shard(&self, lock_id: &LockId) -> &Arc<RwLock<HashMap<LockId, LocalGuardEntry>>> {
|
||||
let index = self.get_shard_index(lock_id);
|
||||
let index = self.get_resource_shard_index(&lock_id.resource);
|
||||
&self.guard_storage[index]
|
||||
}
|
||||
|
||||
async fn reclaim_expired_guards_for_resource(&self, resource: &crate::ObjectKey) -> usize {
|
||||
let expired_entries = Self::extract_expired_guards(&self.guard_storage, Some(resource)).await;
|
||||
let shard = &self.guard_storage[self.get_resource_shard_index(resource)];
|
||||
let has_expired = {
|
||||
let guards = shard.read().await;
|
||||
guards
|
||||
.iter()
|
||||
.any(|(lock_id, entry)| &lock_id.resource == resource && entry.is_expired())
|
||||
};
|
||||
if !has_expired {
|
||||
return 0;
|
||||
}
|
||||
|
||||
let expired_entries = {
|
||||
let mut guards = shard.write().await;
|
||||
guards
|
||||
.extract_if(|lock_id, entry| &lock_id.resource == resource && entry.is_expired())
|
||||
.map(|(_, entry)| entry)
|
||||
.collect::<Vec<_>>()
|
||||
};
|
||||
Self::release_reclaimed_guards(expired_entries, Some(resource))
|
||||
}
|
||||
|
||||
async fn extract_expired_guards(storage: &GuardStorage, resource: Option<&crate::ObjectKey>) -> Vec<LocalGuardEntry> {
|
||||
async fn extract_expired_guards(storage: &GuardStorage) -> Vec<LocalGuardEntry> {
|
||||
let mut expired_entries = Vec::new();
|
||||
for shard in storage.iter() {
|
||||
let has_expired = {
|
||||
let guards = shard.read().await;
|
||||
guards.values().any(LocalGuardEntry::is_expired)
|
||||
};
|
||||
if !has_expired {
|
||||
continue;
|
||||
}
|
||||
let mut guards = shard.write().await;
|
||||
expired_entries.extend(
|
||||
guards
|
||||
.extract_if(|lock_id, entry| {
|
||||
resource.is_none_or(|resource| &lock_id.resource == resource) && entry.is_expired()
|
||||
})
|
||||
.map(|(_, entry)| entry),
|
||||
);
|
||||
expired_entries.extend(guards.extract_if(|_, entry| entry.is_expired()).map(|(_, entry)| entry));
|
||||
}
|
||||
expired_entries
|
||||
}
|
||||
@@ -197,7 +213,7 @@ impl LocalClient {
|
||||
let Some(storage) = storage.upgrade() else {
|
||||
break;
|
||||
};
|
||||
let expired_entries = Self::extract_expired_guards(&storage, None).await;
|
||||
let expired_entries = Self::extract_expired_guards(&storage).await;
|
||||
Self::release_reclaimed_guards(expired_entries, None);
|
||||
}
|
||||
});
|
||||
@@ -214,11 +230,24 @@ impl Default for LocalClient {
|
||||
impl LockClient for LocalClient {
|
||||
async fn acquire_lock(&self, request: &LockRequest) -> Result<LockResponse> {
|
||||
self.ensure_reaper();
|
||||
let lock_manager = self.get_lock_manager();
|
||||
let reclaimed_before_acquire = self.reclaim_expired_guards_for_resource(&request.resource).await;
|
||||
if request.lock_id.resource != request.resource {
|
||||
return Ok(LockResponse::failure(
|
||||
"Lock id resource does not match the requested resource",
|
||||
Duration::ZERO,
|
||||
));
|
||||
}
|
||||
let acquire_deadline = Instant::now()
|
||||
.checked_add(request.acquire_timeout)
|
||||
.unwrap_or_else(Instant::now);
|
||||
let lock_manager = self.get_lock_manager();
|
||||
let initial_remaining = acquire_deadline.saturating_duration_since(Instant::now());
|
||||
let reclaimed_before_acquire = if initial_remaining.is_zero() {
|
||||
0
|
||||
} else {
|
||||
timeout(initial_remaining, self.reclaim_expired_guards_for_resource(&request.resource))
|
||||
.await
|
||||
.unwrap_or(0)
|
||||
};
|
||||
|
||||
let build_lock_request = |acquire_timeout| match request.lock_type {
|
||||
LockType::Exclusive => crate::ObjectLockRequest::new_write(request.resource.clone(), request.owner.clone())
|
||||
@@ -262,7 +291,11 @@ impl LockClient for LocalClient {
|
||||
return Ok(LockResponse::success(lock_info, Duration::ZERO));
|
||||
}
|
||||
Err(crate::fast_lock::LockResult::Timeout) => {
|
||||
if !retried_after_reclaim && self.reclaim_expired_guards_for_resource(&request.resource).await > 0 {
|
||||
if !retried_after_reclaim {
|
||||
let remaining = acquire_deadline.saturating_duration_since(Instant::now());
|
||||
if !remaining.is_zero() {
|
||||
let _ = timeout(remaining, self.reclaim_expired_guards_for_resource(&request.resource)).await;
|
||||
}
|
||||
retried_after_reclaim = true;
|
||||
continue;
|
||||
}
|
||||
@@ -272,7 +305,11 @@ impl LockClient for LocalClient {
|
||||
current_owner,
|
||||
current_mode,
|
||||
}) => {
|
||||
if !retried_after_reclaim && self.reclaim_expired_guards_for_resource(&request.resource).await > 0 {
|
||||
if !retried_after_reclaim {
|
||||
let remaining = acquire_deadline.saturating_duration_since(Instant::now());
|
||||
if !remaining.is_zero() {
|
||||
let _ = timeout(remaining, self.reclaim_expired_guards_for_resource(&request.resource)).await;
|
||||
}
|
||||
retried_after_reclaim = true;
|
||||
continue;
|
||||
}
|
||||
@@ -290,8 +327,11 @@ impl LockClient for LocalClient {
|
||||
|
||||
async fn release(&self, lock_id: &LockId) -> Result<bool> {
|
||||
let shard = self.get_shard(lock_id);
|
||||
let mut guards = shard.write().await;
|
||||
if let Some(guard) = guards.remove(lock_id) {
|
||||
let guard = {
|
||||
let mut guards = shard.write().await;
|
||||
guards.remove(lock_id)
|
||||
};
|
||||
if let Some(guard) = guard {
|
||||
// Guard automatically releases the lock when dropped
|
||||
drop(guard.guard);
|
||||
Ok(true)
|
||||
@@ -418,6 +458,88 @@ mod tests {
|
||||
panic!("lock guard was not reaped before test deadline");
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn acquire_does_not_wait_on_unrelated_guard_shards() {
|
||||
let manager = Arc::new(GlobalLockManager::new());
|
||||
let client = LocalClient::with_manager_and_reaper_interval(manager, Duration::from_secs(60));
|
||||
client.reaper_started.store(true, Ordering::Release);
|
||||
let resource = crate::ObjectKey::new("bucket", "resource-shard-hot-path");
|
||||
let target_shard = client.get_resource_shard_index(&resource);
|
||||
|
||||
let mut unrelated_writes = Vec::new();
|
||||
for (index, shard) in client.guard_storage.iter().enumerate() {
|
||||
if index != target_shard {
|
||||
unrelated_writes.push(shard.write().await);
|
||||
}
|
||||
}
|
||||
|
||||
let lock_request = request(resource, "owner", Duration::from_secs(30));
|
||||
let response = tokio::time::timeout(Duration::from_millis(100), client.acquire_lock(&lock_request))
|
||||
.await
|
||||
.expect("an acquire must not sweep unrelated guard shards")
|
||||
.expect("local acquire should return a response");
|
||||
assert!(response.success);
|
||||
|
||||
drop(unrelated_writes);
|
||||
assert!(client.release(&lock_request.lock_id).await.unwrap());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn reclaim_expired_guard_only_writes_the_resource_shard() {
|
||||
let manager = Arc::new(GlobalLockManager::new());
|
||||
let client = LocalClient::with_manager_and_reaper_interval(manager, Duration::from_secs(60));
|
||||
client.reaper_started.store(true, Ordering::Release);
|
||||
let resource = crate::ObjectKey::new("bucket", "resource-shard-expired");
|
||||
let target_shard = client.get_resource_shard_index(&resource);
|
||||
let expired = request(resource.clone(), "expired-owner", Duration::ZERO);
|
||||
assert!(client.acquire_lock(&expired).await.unwrap().success);
|
||||
|
||||
let mut unrelated_writes = Vec::new();
|
||||
for (index, shard) in client.guard_storage.iter().enumerate() {
|
||||
if index != target_shard {
|
||||
unrelated_writes.push(shard.write().await);
|
||||
}
|
||||
}
|
||||
|
||||
let replacement = request(resource, "replacement-owner", Duration::from_secs(30));
|
||||
let response = tokio::time::timeout(Duration::from_millis(100), client.acquire_lock(&replacement))
|
||||
.await
|
||||
.expect("expired guard reclaim must only write the resource shard")
|
||||
.expect("local acquire should return a response");
|
||||
assert!(response.success);
|
||||
|
||||
drop(unrelated_writes);
|
||||
assert!(client.release(&replacement.lock_id).await.unwrap());
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn acquire_deadline_includes_resource_shard_reclaim_wait() {
|
||||
let manager = Arc::new(GlobalLockManager::new());
|
||||
let client = LocalClient::with_manager_and_reaper_interval(manager, Duration::from_secs(60));
|
||||
client.reaper_started.store(true, Ordering::Release);
|
||||
let resource = crate::ObjectKey::new("bucket", "resource-shard-deadline");
|
||||
let target_shard = client.get_resource_shard_index(&resource);
|
||||
let _target_write = client.guard_storage[target_shard].write().await;
|
||||
let lock_request = request(resource, "owner", Duration::from_secs(30)).with_acquire_timeout(Duration::from_millis(10));
|
||||
|
||||
let response = tokio::time::timeout(Duration::from_millis(100), client.acquire_lock(&lock_request))
|
||||
.await
|
||||
.expect("acquire should return after its reclaim budget expires")
|
||||
.expect("local acquire should return a response");
|
||||
assert!(!response.success);
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn acquire_rejects_a_lock_id_for_another_resource() {
|
||||
let client = LocalClient::with_manager(Arc::new(GlobalLockManager::new()));
|
||||
let mut lock_request = request(crate::ObjectKey::new("bucket", "requested"), "owner", Duration::from_secs(30));
|
||||
lock_request.lock_id.resource = crate::ObjectKey::new("bucket", "other");
|
||||
|
||||
let response = client.acquire_lock(&lock_request).await.unwrap();
|
||||
assert!(!response.success);
|
||||
assert!(response.error.unwrap().contains("does not match"));
|
||||
}
|
||||
|
||||
#[tokio::test(flavor = "current_thread")]
|
||||
async fn expired_guard_is_reaped_without_resource_reacquire() {
|
||||
let manager = Arc::new(GlobalLockManager::new());
|
||||
|
||||
@@ -53,7 +53,9 @@ impl NotifyBucketConfigManager {
|
||||
self.rule_engine.has_subscriber(bucket, event).await
|
||||
}
|
||||
|
||||
pub async fn load_bucket_notification_config(
|
||||
/// Validates that the runtime can accept `cfg` for `bucket` without
|
||||
/// mutating any state.
|
||||
pub async fn validate_bucket_notification_config(
|
||||
&self,
|
||||
bucket: &str,
|
||||
cfg: &BucketNotificationConfig,
|
||||
@@ -98,6 +100,11 @@ impl NotifyBucketConfigManager {
|
||||
);
|
||||
}
|
||||
|
||||
Ok(())
|
||||
}
|
||||
|
||||
/// Publishes an already validated `cfg` to the runtime rule state.
|
||||
pub async fn apply_bucket_notification_config(&self, bucket: &str, cfg: &BucketNotificationConfig) {
|
||||
self.subscriber_view.apply_bucket_config(bucket, cfg);
|
||||
self.rule_engine.set_bucket_rules(bucket, cfg.get_rules_map().clone()).await;
|
||||
info!(
|
||||
@@ -109,6 +116,15 @@ impl NotifyBucketConfigManager {
|
||||
rule_count = cfg.get_rules_map().inner().len(),
|
||||
"notify bucket config state"
|
||||
);
|
||||
}
|
||||
|
||||
pub async fn load_bucket_notification_config(
|
||||
&self,
|
||||
bucket: &str,
|
||||
cfg: &BucketNotificationConfig,
|
||||
) -> Result<(), NotificationError> {
|
||||
self.validate_bucket_notification_config(bucket, cfg).await?;
|
||||
self.apply_bucket_notification_config(bucket, cfg).await;
|
||||
Ok(())
|
||||
}
|
||||
|
||||
@@ -122,7 +138,7 @@ impl NotifyBucketConfigManager {
|
||||
mod tests {
|
||||
use super::NotifyBucketConfigManager;
|
||||
use crate::{
|
||||
BucketNotificationConfig, integration::NotificationMetrics,
|
||||
BucketNotificationConfig, NotificationError, integration::NotificationMetrics,
|
||||
notification_system_subscriber::NotificationSystemSubscriberView, notifier::EventNotifier, rule_engine::NotifyRuleEngine,
|
||||
};
|
||||
use rustfs_s3_types::EventName;
|
||||
@@ -156,4 +172,37 @@ mod tests {
|
||||
manager.remove_bucket_notification_config("bucket").await;
|
||||
assert!(!manager.subscriber_view.has_subscriber("bucket", &EventName::ObjectCreatedPut));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn validate_bucket_notification_config_rejects_missing_targets_without_applying_rules() {
|
||||
let manager = build_manager();
|
||||
let target_id = TargetID::new("primary".to_string(), "webhook".to_string());
|
||||
let mut cfg = BucketNotificationConfig::new("us-east-1");
|
||||
cfg.add_rule(&[EventName::ObjectCreatedPut], "*".to_string(), target_id);
|
||||
|
||||
let err = manager
|
||||
.validate_bucket_notification_config("bucket", &cfg)
|
||||
.await
|
||||
.expect_err("validation must fail when the runtime has no notify targets");
|
||||
assert!(matches!(err, NotificationError::Configuration(_)));
|
||||
assert!(!manager.has_subscriber("bucket", &EventName::ObjectCreatedPut).await);
|
||||
assert!(!manager.subscriber_view.has_subscriber("bucket", &EventName::ObjectCreatedPut));
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn failed_validation_keeps_previous_bucket_rules() {
|
||||
let manager = build_manager();
|
||||
let target_id = TargetID::new("primary".to_string(), "webhook".to_string());
|
||||
let mut cfg = BucketNotificationConfig::new("us-east-1");
|
||||
cfg.add_rule(&[EventName::ObjectCreatedPut], "*".to_string(), target_id);
|
||||
manager.apply_bucket_notification_config("bucket", &cfg).await;
|
||||
assert!(manager.has_subscriber("bucket", &EventName::ObjectCreatedPut).await);
|
||||
|
||||
let err = manager
|
||||
.validate_bucket_notification_config("bucket", &cfg)
|
||||
.await
|
||||
.expect_err("validation must fail when the runtime has no notify targets");
|
||||
assert!(matches!(err, NotificationError::Configuration(_)));
|
||||
assert!(manager.has_subscriber("bucket", &EventName::ObjectCreatedPut).await);
|
||||
}
|
||||
}
|
||||
|
||||
+37
-10
@@ -239,6 +239,42 @@ pub mod notifier_global {
|
||||
.await
|
||||
}
|
||||
|
||||
fn bucket_notification_config_for_rules(
|
||||
region: &str,
|
||||
event_rules: &[(Vec<EventName>, String, String, Vec<TargetID>)],
|
||||
) -> BucketNotificationConfig {
|
||||
let mut bucket_config = BucketNotificationConfig::new(region);
|
||||
|
||||
for (event_names, prefix, suffix, target_ids) in event_rules {
|
||||
// Use `new_pattern` to construct a matching pattern
|
||||
let pattern = crate::rules::pattern::new_pattern(Some(prefix.as_str()), Some(suffix.as_str()));
|
||||
|
||||
for target_id in target_ids {
|
||||
bucket_config.add_rule(event_names, pattern.clone(), target_id.clone());
|
||||
}
|
||||
}
|
||||
|
||||
bucket_config
|
||||
}
|
||||
|
||||
/// Checks that the runtime can accept `event_rules` for `region` without
|
||||
/// publishing them. Callers use this to fail before persisting a bucket
|
||||
/// notification configuration the notify subsystem would reject.
|
||||
pub async fn validate_event_specific_rules(
|
||||
bucket_name: &str,
|
||||
region: &str,
|
||||
event_rules: &[(Vec<EventName>, String, String, Vec<TargetID>)],
|
||||
) -> Result<(), NotificationError> {
|
||||
let bucket_config = bucket_notification_config_for_rules(region, event_rules);
|
||||
|
||||
// Get global NotificationSystem instance
|
||||
let notification_sys = notification_system().ok_or(NotificationError::Lifecycle(LifecycleError::NotInitialized))?;
|
||||
|
||||
notification_sys
|
||||
.validate_bucket_notification_config(bucket_name, &bucket_config)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Dynamically add notification rules according to different event types.
|
||||
///
|
||||
/// # Parameter
|
||||
@@ -256,16 +292,7 @@ pub mod notifier_global {
|
||||
region: &str,
|
||||
event_rules: &[(Vec<EventName>, String, String, Vec<TargetID>)],
|
||||
) -> Result<(), NotificationError> {
|
||||
let mut bucket_config = BucketNotificationConfig::new(region);
|
||||
|
||||
for (event_names, prefix, suffix, target_ids) in event_rules {
|
||||
// Use `new_pattern` to construct a matching pattern
|
||||
let pattern = crate::rules::pattern::new_pattern(Some(prefix.as_str()), Some(suffix.as_str()));
|
||||
|
||||
for target_id in target_ids {
|
||||
bucket_config.add_rule(event_names, pattern.clone(), target_id.clone());
|
||||
}
|
||||
}
|
||||
let bucket_config = bucket_notification_config_for_rules(region, event_rules);
|
||||
|
||||
// Get global NotificationSystem instance
|
||||
let notification_sys = notification_system().ok_or(NotificationError::Lifecycle(LifecycleError::NotInitialized))?;
|
||||
|
||||
@@ -372,6 +372,19 @@ impl NotificationSystem {
|
||||
self.services.config_manager.lifecycle().is_converged()
|
||||
}
|
||||
|
||||
/// Validates a bucket notification configuration against the runtime targets
|
||||
/// without publishing it, so callers can reject a request before persisting.
|
||||
pub async fn validate_bucket_notification_config(
|
||||
&self,
|
||||
bucket: &str,
|
||||
cfg: &BucketNotificationConfig,
|
||||
) -> Result<(), NotificationError> {
|
||||
self.services
|
||||
.bucket_config_manager
|
||||
.validate_bucket_notification_config(bucket, cfg)
|
||||
.await
|
||||
}
|
||||
|
||||
/// Loads the bucket notification configuration
|
||||
pub async fn load_bucket_notification_config(
|
||||
&self,
|
||||
|
||||
@@ -65,7 +65,7 @@ rustfs-config = { workspace = true, features = ["constants"] }
|
||||
rustfs-io-metrics = { workspace = true }
|
||||
rustfs-utils = { workspace = true, features = ["os"] }
|
||||
futures = { workspace = true }
|
||||
tokio = { workspace = true, features = ["sync", "time", "fs", "rt-multi-thread"] }
|
||||
tokio = { workspace = true, features = ["sync", "time", "fs", "rt-multi-thread","macros"] }
|
||||
tokio-util = { workspace = true }
|
||||
tracing = { workspace = true }
|
||||
uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics"] }
|
||||
|
||||
@@ -89,6 +89,18 @@ fn rust_log_requests_verbose(rust_log: &str) -> bool {
|
||||
})
|
||||
}
|
||||
|
||||
/// Returns `true` when the effective logging configuration selects a verbose
|
||||
/// (`debug`/`trace`) level.
|
||||
///
|
||||
/// Follows the same precedence as [`build_env_filter`]: `RUST_LOG` overrides
|
||||
/// `logger_level` when it is set.
|
||||
pub(super) fn effective_verbose_logging(logger_level: &str) -> bool {
|
||||
match env::var("RUST_LOG").ok().filter(|value| !value.trim().is_empty()) {
|
||||
Some(rust_log) => rust_log_requests_verbose(&rust_log),
|
||||
None => is_verbose_level(logger_level),
|
||||
}
|
||||
}
|
||||
|
||||
fn should_suppress_noisy_crates(logger_level: &str, default_level: Option<&str>, rust_log: Option<&str>) -> bool {
|
||||
if let Some(level) = default_level {
|
||||
return !is_verbose_level(level);
|
||||
|
||||
@@ -36,7 +36,7 @@ use crate::cleaner::LogCleaner;
|
||||
use crate::cleaner::types::{CompressionAlgorithm, FileMatchMode};
|
||||
use crate::config::OtelConfig;
|
||||
use crate::global::{METRIC_LOG_CLEANER_RUN_FAILURES_TOTAL, METRIC_LOG_CLEANER_RUNS_TOTAL, set_observability_metric_enabled};
|
||||
use crate::telemetry::filter::{build_env_filter, pyroscope_log_filter};
|
||||
use crate::telemetry::filter::{build_env_filter, effective_verbose_logging, pyroscope_log_filter};
|
||||
use crate::telemetry::rolling::{RollingAppender, Rotation};
|
||||
use metrics::counter;
|
||||
use rustfs_config::observability::{
|
||||
@@ -44,10 +44,11 @@ use rustfs_config::observability::{
|
||||
DEFAULT_OBS_LOG_COMPRESSION_ALGORITHM, DEFAULT_OBS_LOG_DELETE_EMPTY_FILES, DEFAULT_OBS_LOG_DRY_RUN,
|
||||
DEFAULT_OBS_LOG_GZIP_COMPRESSION_LEVEL, DEFAULT_OBS_LOG_MATCH_MODE, DEFAULT_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES,
|
||||
DEFAULT_OBS_LOG_MAX_TOTAL_SIZE_BYTES, DEFAULT_OBS_LOG_MIN_FILE_AGE_SECONDS, DEFAULT_OBS_LOG_PARALLEL_COMPRESS,
|
||||
DEFAULT_OBS_LOG_PARALLEL_WORKERS, DEFAULT_OBS_LOG_ZSTD_COMPRESSION_LEVEL, DEFAULT_OBS_LOG_ZSTD_FALLBACK_TO_GZIP,
|
||||
DEFAULT_OBS_LOG_ZSTD_WORKERS,
|
||||
DEFAULT_OBS_LOG_PARALLEL_WORKERS, DEFAULT_OBS_LOG_SPAN_EVENTS, DEFAULT_OBS_LOG_ZSTD_COMPRESSION_LEVEL,
|
||||
DEFAULT_OBS_LOG_ZSTD_FALLBACK_TO_GZIP, DEFAULT_OBS_LOG_ZSTD_WORKERS, ENV_OBS_LOG_SPAN_EVENTS,
|
||||
};
|
||||
use rustfs_config::{APP_NAME, DEFAULT_LOG_KEEP_FILES, DEFAULT_LOG_ROTATION_TIME};
|
||||
use rustfs_utils::get_env_str;
|
||||
use serde_json::Value as JsonValue;
|
||||
use std::sync::Arc;
|
||||
use std::{collections::BTreeMap, fmt};
|
||||
@@ -78,6 +79,37 @@ pub(super) fn resolve_file_stdout_mirror(configured: Option<bool>, is_production
|
||||
configured.unwrap_or(!is_production)
|
||||
}
|
||||
|
||||
/// Resolves the span lifecycle events emitted by the JSON log layers.
|
||||
///
|
||||
/// Defaults to [`FmtSpan::NONE`]. A record per span lifecycle multiplies log
|
||||
/// volume on object hot paths and, combined with the ancestor span list,
|
||||
/// produced multi-megabyte lines that journald truncated (backlog#2642).
|
||||
/// Operators opt back in with `RUSTFS_OBS_LOG_SPAN_EVENTS=close|full`.
|
||||
pub(super) fn resolve_span_events() -> FmtSpan {
|
||||
match get_env_str(ENV_OBS_LOG_SPAN_EVENTS, DEFAULT_OBS_LOG_SPAN_EVENTS)
|
||||
.trim()
|
||||
.to_ascii_lowercase()
|
||||
.as_str()
|
||||
{
|
||||
"close" => FmtSpan::CLOSE,
|
||||
"full" => FmtSpan::FULL,
|
||||
"none" | "" => FmtSpan::NONE,
|
||||
unknown => {
|
||||
// An unrecognised value must not silently re-enable span events.
|
||||
eprintln!("{STDERR_WARNING_PREFIX} obs: unknown {ENV_OBS_LOG_SPAN_EVENTS}={unknown:?}; falling back to none");
|
||||
FmtSpan::NONE
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Whether JSON log records embed the full ancestor span list.
|
||||
///
|
||||
/// `with_span_list(true)` re-serializes every ancestor span on each record, so
|
||||
/// it is reserved for verbose levels where that context earns its volume.
|
||||
pub(super) fn resolve_span_list(logger_level: &str) -> bool {
|
||||
effective_verbose_logging(logger_level)
|
||||
}
|
||||
|
||||
#[cfg(unix)]
|
||||
pub(super) fn validate_stdout_sink(file_appender: &RollingAppender) -> Result<(), TelemetryError> {
|
||||
use std::os::fd::AsFd;
|
||||
@@ -222,7 +254,12 @@ where
|
||||
request_id
|
||||
}
|
||||
|
||||
pub(super) fn build_json_log_layer<S, W>(writer: W, enable_ansi: bool, span_events: FmtSpan) -> impl tracing_subscriber::Layer<S>
|
||||
pub(super) fn build_json_log_layer<S, W>(
|
||||
writer: W,
|
||||
enable_ansi: bool,
|
||||
span_events: FmtSpan,
|
||||
span_list: bool,
|
||||
) -> impl tracing_subscriber::Layer<S>
|
||||
where
|
||||
S: Subscriber + for<'span> LookupSpan<'span>,
|
||||
W: for<'writer> tracing_subscriber::fmt::MakeWriter<'writer> + Send + Sync + 'static,
|
||||
@@ -239,7 +276,7 @@ where
|
||||
.json()
|
||||
.flatten_event(true)
|
||||
.with_current_span(true)
|
||||
.with_span_list(true)
|
||||
.with_span_list(span_list)
|
||||
.with_span_events(span_events)
|
||||
.map_event_format(RequestIdJsonFormat::new)
|
||||
}
|
||||
@@ -311,11 +348,12 @@ pub(super) fn init_local_logging(
|
||||
fn init_stdout_only(_config: &OtelConfig, logger_level: &str, is_production: bool) -> Result<OtelGuard, TelemetryError> {
|
||||
let env_filter = build_env_filter(logger_level, None);
|
||||
let (nb, guard) = tracing_appender::non_blocking(std::io::stdout());
|
||||
let span_events = if is_production { FmtSpan::CLOSE } else { FmtSpan::FULL };
|
||||
let span_events = resolve_span_events();
|
||||
let span_list = resolve_span_list(logger_level);
|
||||
|
||||
// Keep stdout formatting JSON-shaped even in local-only mode so operators
|
||||
// can ship the same log schema to external collectors if needed.
|
||||
let fmt_layer = build_json_log_layer(nb, std::io::stdout().is_terminal(), span_events);
|
||||
let fmt_layer = build_json_log_layer(nb, std::io::stdout().is_terminal(), span_events, span_list);
|
||||
|
||||
tracing_subscriber::registry()
|
||||
.with(env_filter)
|
||||
@@ -413,11 +451,12 @@ fn init_file_logging_internal(
|
||||
|
||||
// ── 4. Build subscriber layers ────────────────────────────────────────────
|
||||
let env_filter = build_env_filter(logger_level, None);
|
||||
let span_events = if is_production { FmtSpan::CLOSE } else { FmtSpan::FULL };
|
||||
let span_events = resolve_span_events();
|
||||
let span_list = resolve_span_list(logger_level);
|
||||
|
||||
// File output stays machine-readable and free of ANSI sequences so the
|
||||
// resulting files are safe to parse or ship to log processors.
|
||||
let file_layer = build_json_log_layer(non_blocking, false, span_events.clone());
|
||||
let file_layer = build_json_log_layer(non_blocking, false, span_events.clone(), span_list);
|
||||
|
||||
// Optional stdout mirror: enabled explicitly via `log_stdout_enabled`, or
|
||||
// unconditionally in non-production environments so developers still see
|
||||
@@ -425,7 +464,10 @@ fn init_file_logging_internal(
|
||||
let (stdout_layer, stdout_guard) = if resolve_file_stdout_mirror(config.log_stdout_enabled, is_production) {
|
||||
let (stdout_nb, stdout_guard) = tracing_appender::non_blocking(std::io::stdout());
|
||||
let enable_color = std::io::stdout().is_terminal();
|
||||
(Some(build_json_log_layer(stdout_nb, enable_color, span_events)), Some(stdout_guard))
|
||||
(
|
||||
Some(build_json_log_layer(stdout_nb, enable_color, span_events, span_list)),
|
||||
Some(stdout_guard),
|
||||
)
|
||||
} else {
|
||||
(None, None)
|
||||
};
|
||||
@@ -773,7 +815,7 @@ mod tests {
|
||||
|
||||
fn render_json_log(enable_ansi: bool) -> (String, Value) {
|
||||
let writer = SharedWriter::default();
|
||||
let layer = build_json_log_layer(writer.clone(), enable_ansi, FmtSpan::NONE);
|
||||
let layer = build_json_log_layer(writer.clone(), enable_ansi, FmtSpan::NONE, false);
|
||||
let subscriber = Registry::default().with(layer);
|
||||
|
||||
tracing::subscriber::with_default(subscriber, || {
|
||||
@@ -793,9 +835,32 @@ mod tests {
|
||||
(raw, parsed)
|
||||
}
|
||||
|
||||
fn render_event_in_nested_spans(depth: usize, span_list: bool) -> String {
|
||||
let writer = SharedWriter::default();
|
||||
let layer = build_json_log_layer(writer.clone(), false, FmtSpan::NONE, span_list);
|
||||
let subscriber = Registry::default().with(layer);
|
||||
|
||||
tracing::subscriber::with_default(subscriber, || {
|
||||
let spans = (0..depth)
|
||||
.map(|level| tracing::info_span!("layer", depth = level, payload = "x".repeat(1024)))
|
||||
.collect::<Vec<_>>();
|
||||
let _guards = spans.iter().map(tracing::Span::enter).collect::<Vec<_>>();
|
||||
info!(component = "obs_contract_test", "nested event");
|
||||
});
|
||||
|
||||
String::from_utf8(
|
||||
writer
|
||||
.inner
|
||||
.lock()
|
||||
.expect("shared writer lock should not be poisoned")
|
||||
.clone(),
|
||||
)
|
||||
.expect("log output should be valid UTF-8")
|
||||
}
|
||||
|
||||
fn render_json_log_with_request_span() -> Value {
|
||||
let writer = SharedWriter::default();
|
||||
let layer = build_json_log_layer(writer.clone(), false, FmtSpan::NONE);
|
||||
let layer = build_json_log_layer(writer.clone(), false, FmtSpan::NONE, false);
|
||||
let subscriber = Registry::default().with(layer);
|
||||
|
||||
tracing::subscriber::with_default(subscriber, || {
|
||||
@@ -818,7 +883,7 @@ mod tests {
|
||||
|
||||
fn render_json_log_with_recovery_monitor_child_span() -> Value {
|
||||
let writer = SharedWriter::default();
|
||||
let layer = build_json_log_layer(writer.clone(), false, FmtSpan::NONE);
|
||||
let layer = build_json_log_layer(writer.clone(), false, FmtSpan::NONE, true);
|
||||
let subscriber = Registry::default().with(layer);
|
||||
|
||||
tracing::subscriber::with_default(subscriber, || {
|
||||
@@ -1075,6 +1140,10 @@ mod tests {
|
||||
assert_eq!(parsed["request-id"], Value::String("req-123".to_string()));
|
||||
assert_eq!(parsed["message"], Value::String("inside request span".to_string()));
|
||||
assert_eq!(parsed["span"]["request_id"], Value::String("req-123".to_string()));
|
||||
assert!(
|
||||
parsed.get("spans").is_none(),
|
||||
"ancestor span list must stay out of records when span_list is disabled: {parsed}"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -1112,4 +1181,65 @@ mod tests {
|
||||
|
||||
assert_eq!(resolve_log_cleanup_file_pattern(&config, "rustfs.log"), "rustfs.log");
|
||||
}
|
||||
|
||||
/// Regression for backlog#2642: records must not embed the ancestor span
|
||||
/// list, so their size stays flat as the enclosing span chain deepens.
|
||||
#[test]
|
||||
fn record_size_does_not_grow_with_span_depth_when_span_list_is_disabled() {
|
||||
let shallow = render_event_in_nested_spans(1, false);
|
||||
let deep = render_event_in_nested_spans(32, false);
|
||||
|
||||
assert!(
|
||||
deep.len() <= shallow.len() * 2,
|
||||
"ancestor spans leaked into the record: shallow={} deep={}",
|
||||
shallow.len(),
|
||||
deep.len()
|
||||
);
|
||||
|
||||
// The flag is what keeps it bounded: re-enabling the list must show the
|
||||
// growth the default avoids.
|
||||
let deep_with_list = render_event_in_nested_spans(32, true);
|
||||
assert!(
|
||||
deep_with_list.len() > deep.len(),
|
||||
"enabling the span list must reintroduce per-ancestor growth"
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn span_events_default_to_none_and_parse_overrides() {
|
||||
temp_env::with_var(ENV_OBS_LOG_SPAN_EVENTS, None::<&str>, || {
|
||||
assert_eq!(resolve_span_events(), FmtSpan::NONE);
|
||||
});
|
||||
temp_env::with_var(ENV_OBS_LOG_SPAN_EVENTS, Some("close"), || {
|
||||
assert_eq!(resolve_span_events(), FmtSpan::CLOSE);
|
||||
});
|
||||
temp_env::with_var(ENV_OBS_LOG_SPAN_EVENTS, Some("FULL"), || {
|
||||
assert_eq!(resolve_span_events(), FmtSpan::FULL);
|
||||
});
|
||||
temp_env::with_var(ENV_OBS_LOG_SPAN_EVENTS, Some("verbose"), || {
|
||||
assert_eq!(
|
||||
resolve_span_events(),
|
||||
FmtSpan::NONE,
|
||||
"an unrecognised value must not re-enable span events"
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn span_list_is_reserved_for_verbose_levels() {
|
||||
temp_env::with_var("RUST_LOG", None::<&str>, || {
|
||||
assert!(!resolve_span_list("info"), "info records must not embed the ancestor span list");
|
||||
assert!(!resolve_span_list("warn"));
|
||||
assert!(!resolve_span_list("error"));
|
||||
assert!(resolve_span_list("debug"));
|
||||
assert!(resolve_span_list("trace"));
|
||||
});
|
||||
|
||||
temp_env::with_var("RUST_LOG", Some("debug"), || {
|
||||
assert!(
|
||||
resolve_span_list("info"),
|
||||
"RUST_LOG=debug overrides the base level and should keep span context"
|
||||
);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
@@ -68,7 +68,7 @@ use std::{fs, io::IsTerminal, time::Duration};
|
||||
use tracing::{info, warn};
|
||||
use tracing_error::ErrorLayer;
|
||||
use tracing_opentelemetry::{MetricsLayer, OpenTelemetryLayer};
|
||||
use tracing_subscriber::{fmt::format::FmtSpan, layer::SubscriberExt, util::SubscriberInitExt};
|
||||
use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt};
|
||||
|
||||
const GET_OBJECT_DURATION_HISTOGRAM_METRICS: &[&str] = &[
|
||||
"rustfs_io_get_object_request_duration_seconds",
|
||||
@@ -184,7 +184,8 @@ pub(super) fn init_observability_http(
|
||||
// active, the OpenTelemetry bridge is the authoritative sink for
|
||||
// `tracing` events unless local file logging is needed as a fallback.
|
||||
}
|
||||
let span_events = if is_production { FmtSpan::CLOSE } else { FmtSpan::FULL };
|
||||
let span_events = crate::telemetry::local::resolve_span_events();
|
||||
let span_list = crate::telemetry::local::resolve_span_list(logger_level);
|
||||
// ── Case 2: File Logging
|
||||
// If a log directory is configured and OTLP log export is unavailable, use
|
||||
// the same rolling-file behavior as the local-only telemetry backend.
|
||||
@@ -221,7 +222,7 @@ pub(super) fn init_observability_http(
|
||||
crate::telemetry::local::validate_stdout_sink(&file_appender)?;
|
||||
|
||||
let (non_blocking, guard) = tracing_appender::non_blocking(file_appender);
|
||||
let file_layer = build_json_log_layer(non_blocking, false, span_events.clone());
|
||||
let file_layer = build_json_log_layer(non_blocking, false, span_events.clone(), span_list);
|
||||
let cleanup_handle = spawn_cleanup_task(config, log_directory, log_filename, keep_files);
|
||||
Ok((file_layer, guard, cleanup_handle, rotation_str))
|
||||
})();
|
||||
@@ -267,7 +268,7 @@ pub(super) fn init_observability_http(
|
||||
if force_stdout_logging || crate::telemetry::local::resolve_file_stdout_mirror(config.log_stdout_enabled, is_production) {
|
||||
let (stdout_nb, stdout_g) = tracing_appender::non_blocking(std::io::stdout());
|
||||
stdout_guard = Some(stdout_g);
|
||||
stdout_layer_opt = Some(build_json_log_layer(stdout_nb, std::io::stdout().is_terminal(), span_events));
|
||||
stdout_layer_opt = Some(build_json_log_layer(stdout_nb, std::io::stdout().is_terminal(), span_events, span_list));
|
||||
}
|
||||
let local_file_fallback_enabled = file_layer_opt.is_some();
|
||||
let stdout_mirror_enabled = stdout_guard.is_some();
|
||||
|
||||
@@ -57,6 +57,7 @@ const EXCESSIVE_EMPTY_CHUNKS_ERROR: &str = "HTTP body returned too many empty ch
|
||||
pub const INTERNODE_DISK_ERROR_HEADER: &str = "x-rustfs-disk-error";
|
||||
pub const INTERNODE_FILE_NOT_FOUND: &str = "file-not-found";
|
||||
pub const INTERNODE_VOLUME_NOT_FOUND: &str = "volume-not-found";
|
||||
pub const INTERNODE_FILE_CORRUPT: &str = "file-corrupt";
|
||||
|
||||
#[derive(Debug, Clone, Copy, Eq, PartialEq)]
|
||||
pub enum InternodeHttpErrorKind {
|
||||
@@ -185,6 +186,7 @@ pub struct InternodeHttpError {
|
||||
enum RemoteDiskErrorKind {
|
||||
FileNotFound,
|
||||
VolumeNotFound,
|
||||
FileCorrupt,
|
||||
}
|
||||
|
||||
impl std::fmt::Debug for InternodeHttpError {
|
||||
@@ -215,6 +217,10 @@ impl InternodeHttpError {
|
||||
self.remote_disk_error == Some(RemoteDiskErrorKind::VolumeNotFound)
|
||||
}
|
||||
|
||||
pub fn is_remote_file_corrupt(&self) -> bool {
|
||||
self.remote_disk_error == Some(RemoteDiskErrorKind::FileCorrupt)
|
||||
}
|
||||
|
||||
fn new(kind: InternodeHttpErrorKind, context: InternodeHttpRequestContext) -> Self {
|
||||
Self {
|
||||
kind,
|
||||
@@ -328,6 +334,20 @@ pub fn new_test_remote_volume_not_found_http_io_error() -> io::Error {
|
||||
.into_io_error()
|
||||
}
|
||||
|
||||
#[doc(hidden)]
|
||||
pub fn new_test_remote_file_corrupt_http_io_error() -> io::Error {
|
||||
InternodeHttpError::with_remote_disk_error(
|
||||
InternodeHttpErrorKind::HttpStatus(reqwest::StatusCode::INTERNAL_SERVER_ERROR),
|
||||
InternodeHttpRequestContext {
|
||||
method: "GET".to_string(),
|
||||
target: READ_FILE_STREAM_PATH.to_string(),
|
||||
operation: Some(INTERNODE_OPERATION_READ_FILE_STREAM),
|
||||
},
|
||||
RemoteDiskErrorKind::FileCorrupt,
|
||||
)
|
||||
.into_io_error()
|
||||
}
|
||||
|
||||
fn add_root_certificates_from_der(builder: reqwest::ClientBuilder, certs_der: &[Vec<u8>]) -> reqwest::ClientBuilder {
|
||||
let mut b = builder;
|
||||
for der in certs_der {
|
||||
@@ -850,6 +870,7 @@ fn classify_http_response(
|
||||
let remote_disk_error = match headers.get(INTERNODE_DISK_ERROR_HEADER).and_then(|value| value.to_str().ok()) {
|
||||
Some(INTERNODE_FILE_NOT_FOUND) => Some(RemoteDiskErrorKind::FileNotFound),
|
||||
Some(INTERNODE_VOLUME_NOT_FOUND) => Some(RemoteDiskErrorKind::VolumeNotFound),
|
||||
Some(INTERNODE_FILE_CORRUPT) => Some(RemoteDiskErrorKind::FileCorrupt),
|
||||
_ => None,
|
||||
};
|
||||
ClassifiedHttpResponse { kind, remote_disk_error }
|
||||
@@ -2047,6 +2068,7 @@ mod tests {
|
||||
assert_eq!(INTERNODE_DISK_ERROR_HEADER, "x-rustfs-disk-error");
|
||||
assert_eq!(INTERNODE_FILE_NOT_FOUND, "file-not-found");
|
||||
assert_eq!(INTERNODE_VOLUME_NOT_FOUND, "volume-not-found");
|
||||
assert_eq!(INTERNODE_FILE_CORRUPT, "file-corrupt");
|
||||
}
|
||||
|
||||
#[test]
|
||||
@@ -2090,6 +2112,48 @@ mod tests {
|
||||
assert!(wrong_operation.remote_disk_error.is_none());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn classify_http_response_scopes_corrupt_disk_error_to_read_failures() {
|
||||
let mut headers = HeaderMap::new();
|
||||
headers.insert(INTERNODE_DISK_ERROR_HEADER, INTERNODE_FILE_CORRUPT.parse().expect("valid error token"));
|
||||
let classified = classify_http_response(
|
||||
reqwest::StatusCode::INTERNAL_SERVER_ERROR,
|
||||
&headers,
|
||||
Some(INTERNODE_OPERATION_READ_FILE_STREAM),
|
||||
);
|
||||
assert_eq!(classified.remote_disk_error, Some(RemoteDiskErrorKind::FileCorrupt));
|
||||
for status in [reqwest::StatusCode::NOT_FOUND, reqwest::StatusCode::SERVICE_UNAVAILABLE] {
|
||||
assert!(
|
||||
classify_http_response(status, &headers, Some(INTERNODE_OPERATION_READ_FILE_STREAM))
|
||||
.remote_disk_error
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
for operation in [
|
||||
None,
|
||||
Some(INTERNODE_OPERATION_WALK_DIR),
|
||||
Some(INTERNODE_OPERATION_PUT_FILE_STREAM),
|
||||
] {
|
||||
assert!(
|
||||
classify_http_response(reqwest::StatusCode::INTERNAL_SERVER_ERROR, &headers, operation)
|
||||
.remote_disk_error
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
headers.insert(INTERNODE_DISK_ERROR_HEADER, "unknown-disk-error".parse().expect("valid unknown token"));
|
||||
for headers in [&headers, &HeaderMap::new()] {
|
||||
assert!(
|
||||
classify_http_response(
|
||||
reqwest::StatusCode::INTERNAL_SERVER_ERROR,
|
||||
headers,
|
||||
Some(INTERNODE_OPERATION_READ_FILE_STREAM),
|
||||
)
|
||||
.remote_disk_error
|
||||
.is_none()
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Default)]
|
||||
struct TestState {
|
||||
head_count: Arc<AtomicUsize>,
|
||||
@@ -2158,6 +2222,16 @@ mod tests {
|
||||
let addr = listener.local_addr().expect("listener local address should be available");
|
||||
let app = Router::new()
|
||||
.route("/stream", get(get_stream).head(reject_head).put(accept_put))
|
||||
.route(
|
||||
READ_FILE_STREAM_PATH,
|
||||
get(|| async {
|
||||
(
|
||||
StatusCode::INTERNAL_SERVER_ERROR,
|
||||
[(INTERNODE_DISK_ERROR_HEADER, INTERNODE_FILE_CORRUPT)],
|
||||
"read file err file corrupt",
|
||||
)
|
||||
}),
|
||||
)
|
||||
.route(WALK_DIR_PATH, get(get_stream))
|
||||
.route("/reject-put", get(get_stream).put(reject_put))
|
||||
.route("/stall", get(get_stalling_stream))
|
||||
@@ -2172,6 +2246,44 @@ mod tests {
|
||||
Some((format!("http://{addr}/stream"), handle))
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn http_readers_preserve_remote_file_corruption() {
|
||||
let (url, server) = start_test_server(TestState::default())
|
||||
.await
|
||||
.expect("corruption regression server must bind");
|
||||
let url = format!("{}{READ_FILE_STREAM_PATH}", url.strip_suffix("/stream").expect("test server URL suffix"));
|
||||
let byte_error = HttpReader::new(url.clone(), Method::GET, HeaderMap::new(), None)
|
||||
.await
|
||||
.err()
|
||||
.expect("byte reader must reject corrupt shard response");
|
||||
let chunk_error = HttpChunkReader::new_with_stall_timeout(url, Method::GET, HeaderMap::new(), None, None)
|
||||
.await
|
||||
.err()
|
||||
.expect("chunk reader must reject corrupt shard response");
|
||||
for error in [byte_error, chunk_error] {
|
||||
let source = error
|
||||
.get_ref()
|
||||
.and_then(|source| source.downcast_ref::<InternodeHttpError>())
|
||||
.expect("HTTP error must retain typed internode source");
|
||||
assert!(source.is_remote_file_corrupt());
|
||||
assert!(!source.is_remote_file_not_found());
|
||||
assert!(!source.is_remote_volume_not_found());
|
||||
assert_eq!(
|
||||
source.kind(),
|
||||
InternodeHttpErrorKind::HttpStatus(reqwest::StatusCode::INTERNAL_SERVER_ERROR)
|
||||
);
|
||||
let cloned = clone_internode_http_io_error(&error).expect("typed transport error must be cloneable");
|
||||
assert!(
|
||||
cloned
|
||||
.get_ref()
|
||||
.and_then(|source| source.downcast_ref::<InternodeHttpError>())
|
||||
.expect("clone must retain typed internode source")
|
||||
.is_remote_file_corrupt()
|
||||
);
|
||||
}
|
||||
server.abort();
|
||||
}
|
||||
|
||||
struct BlockedH2Server {
|
||||
url: String,
|
||||
shutdown: tokio::sync::oneshot::Sender<()>,
|
||||
|
||||
@@ -83,13 +83,11 @@ impl ProviderVersionCapabilities {
|
||||
if tier_type.eq_ignore_ascii_case("s3")
|
||||
|| tier_type.eq_ignore_ascii_case("rustfs")
|
||||
|| tier_type.eq_ignore_ascii_case("minio")
|
||||
|| tier_type.eq_ignore_ascii_case("r2")
|
||||
|| tier_type.eq_ignore_ascii_case("wasabi")
|
||||
{
|
||||
let list_object_versions = tier_type.eq_ignore_ascii_case("s3")
|
||||
|| tier_type.eq_ignore_ascii_case("rustfs")
|
||||
|| tier_type.eq_ignore_ascii_case("minio")
|
||||
|| tier_type.eq_ignore_ascii_case("r2");
|
||||
|| tier_type.eq_ignore_ascii_case("minio");
|
||||
Self {
|
||||
raw_version_header: Some(X_AMZ_VERSION_ID),
|
||||
bucket_versioning_state: list_object_versions,
|
||||
@@ -101,6 +99,17 @@ impl ProviderVersionCapabilities {
|
||||
},
|
||||
exact_get_delete: true,
|
||||
}
|
||||
} else if tier_type.eq_ignore_ascii_case("r2") {
|
||||
// R2 exposes an x-amz-version-id on some PUT responses, but it does
|
||||
// not provide routable S3 object versions. GET/DELETE must therefore
|
||||
// use the unversioned object path and must not call version APIs.
|
||||
Self {
|
||||
raw_version_header: None,
|
||||
bucket_versioning_state: false,
|
||||
list_object_versions: false,
|
||||
conditional_create: ConditionalCreateCapability::IfNoneMatchStar,
|
||||
exact_get_delete: false,
|
||||
}
|
||||
} else if tier_type.eq_ignore_ascii_case("aliyun") {
|
||||
Self {
|
||||
raw_version_header: Some(X_OSS_VERSION_ID),
|
||||
@@ -210,8 +219,6 @@ mod tests {
|
||||
("RustFS", "x-amz-version-id"),
|
||||
("minio", "x-amz-version-id"),
|
||||
("MinIO", "x-amz-version-id"),
|
||||
("r2", "x-amz-version-id"),
|
||||
("R2", "x-amz-version-id"),
|
||||
("wasabi", "x-amz-version-id"),
|
||||
("Wasabi", "x-amz-version-id"),
|
||||
("aliyun", "x-oss-version-id"),
|
||||
@@ -255,6 +262,26 @@ mod tests {
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn r2_ignores_non_routable_put_version_header() {
|
||||
let mut headers = HeaderMap::new();
|
||||
headers.insert("x-amz-version-id", HeaderValue::from_static("r2-response-token"));
|
||||
let capabilities = ProviderVersionCapabilities::for_tier_type("r2");
|
||||
|
||||
assert_eq!(
|
||||
capabilities
|
||||
.raw_version_id(&headers)
|
||||
.expect("R2 header parsing should succeed"),
|
||||
None
|
||||
);
|
||||
assert_eq!(
|
||||
capabilities
|
||||
.remote_version(&headers, BucketVersioningState::Disabled)
|
||||
.expect("R2 uses unversioned routing"),
|
||||
RemoteVersion::Disabled
|
||||
);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn provider_version_missing_header_is_unknown_until_bucket_state_is_known() {
|
||||
let headers = HeaderMap::new();
|
||||
@@ -280,7 +307,7 @@ mod tests {
|
||||
("s3", true, true, ConditionalCreateCapability::IfNoneMatchStar, true),
|
||||
("rustfs", true, true, ConditionalCreateCapability::Unsupported, true),
|
||||
("minio", true, true, ConditionalCreateCapability::Unsupported, true),
|
||||
("r2", true, true, ConditionalCreateCapability::IfNoneMatchStar, true),
|
||||
("r2", false, false, ConditionalCreateCapability::IfNoneMatchStar, false),
|
||||
("wasabi", false, false, ConditionalCreateCapability::Unsupported, true),
|
||||
("aliyun", false, false, ConditionalCreateCapability::Unsupported, true),
|
||||
("tencent", false, false, ConditionalCreateCapability::Unsupported, true),
|
||||
|
||||
@@ -34,7 +34,7 @@ metrics = { workspace = true }
|
||||
rmp-serde = { workspace = true }
|
||||
rustfs-heal-contracts = { workspace = true }
|
||||
serde = { workspace = true, features = ["derive"] }
|
||||
tokio = { workspace = true, features = ["sync"] }
|
||||
tokio = { workspace = true, features = ["sync","rt"] }
|
||||
|
||||
[dev-dependencies]
|
||||
serde_json = { workspace = true }
|
||||
|
||||
@@ -910,11 +910,26 @@ impl DataUsageCache {
|
||||
if self.info.leader_epoch > leader_epoch {
|
||||
return DataUsageCachePrepareOutcome::RejectedNewerLeader;
|
||||
}
|
||||
// A checkpoint is a durable traversal frontier, not a publication
|
||||
// result. A newer leader may adopt it only when the frontier is
|
||||
// independently fenced by the bucket identity and has a validated
|
||||
// cursor/coverage receipt. Caches without that proof still take the
|
||||
// normal rebuild path; this keeps an old, incomplete writer from
|
||||
// authorizing a new leader to skip namespace coverage.
|
||||
let cross_epoch_checkpoint = self.info.leader_epoch < leader_epoch
|
||||
&& self.info.next_cycle == next_cycle
|
||||
&& self.info.scan_identity == Some(identity)
|
||||
&& (self.validated_scan_frontier().is_some()
|
||||
|| self.validated_raw_enumeration_cursor().is_some()
|
||||
|| self.validated_raw_enumeration_page_index().is_some());
|
||||
let handoff_frontier = cross_epoch_checkpoint
|
||||
.then(|| self.validated_scan_frontier().map(str::to_owned))
|
||||
.flatten();
|
||||
let reusable = identity.is_valid()
|
||||
&& name != DATA_USAGE_ROOT
|
||||
&& self.info.name == name
|
||||
&& self.info.source == Some(source)
|
||||
&& self.info.leader_epoch == leader_epoch
|
||||
&& (self.info.leader_epoch == leader_epoch || cross_epoch_checkpoint)
|
||||
&& self.info.cache_key_format == DATA_USAGE_CACHE_KEY_FORMAT
|
||||
&& self.info.scan_identity == Some(identity)
|
||||
&& self.info.tier_registry_generation == Some(identity.tier_registry_generation)
|
||||
@@ -974,6 +989,15 @@ impl DataUsageCache {
|
||||
self.info.tier_registry_generation = Some(identity.tier_registry_generation);
|
||||
self.info.scan_identity = Some(identity);
|
||||
self.info.snapshot_complete = false;
|
||||
if let Some(frontier) = handoff_frontier
|
||||
&& let Ok(digest) = self.coverage_prefix_digest(&frontier)
|
||||
&& let Some(receipt) = self.info.scan_coverage_receipt.as_mut()
|
||||
{
|
||||
// Migrate a legacy epoch-bound receipt while the old epoch is
|
||||
// still available for validation. New receipts deliberately do
|
||||
// not bind to the process-local leader epoch.
|
||||
receipt.digest = digest;
|
||||
}
|
||||
if let Some(progress) = &mut self.info.scan_progress {
|
||||
progress.requested_plan = scan_plan_digest;
|
||||
} else {
|
||||
@@ -998,19 +1022,53 @@ impl DataUsageCache {
|
||||
}
|
||||
|
||||
fn coverage_prefix_digest(&self, through: &str) -> Result<[u8; 32], serde_json::Error> {
|
||||
self.coverage_prefix_digest_with_epoch(through, None)
|
||||
}
|
||||
|
||||
fn legacy_coverage_prefix_digest(&self, through: &str) -> Result<[u8; 32], serde_json::Error> {
|
||||
self.coverage_prefix_digest_with_epoch(through, Some(self.info.leader_epoch))
|
||||
}
|
||||
|
||||
fn coverage_prefix_digest_with_epoch(
|
||||
&self,
|
||||
through: &str,
|
||||
legacy_leader_epoch: Option<u64>,
|
||||
) -> Result<[u8; 32], serde_json::Error> {
|
||||
let mut writer = CheckpointDigestWriter(Sha256::new());
|
||||
serde_json::to_writer(
|
||||
&mut writer,
|
||||
&(
|
||||
&self.info.name,
|
||||
self.info.scan_identity,
|
||||
self.info.source,
|
||||
self.info.leader_epoch,
|
||||
self.info.cache_key_format,
|
||||
self.info.scan_progress.map(|progress| progress.started_plan),
|
||||
through,
|
||||
),
|
||||
)?;
|
||||
if let Some(leader_epoch) = legacy_leader_epoch {
|
||||
// Keep the old tuple available for one-way migration of caches
|
||||
// written before leader handoff recovery was introduced.
|
||||
serde_json::to_writer(
|
||||
&mut writer,
|
||||
&(
|
||||
&self.info.name,
|
||||
self.info.scan_identity,
|
||||
self.info.source,
|
||||
leader_epoch,
|
||||
self.info.cache_key_format,
|
||||
self.info.scan_progress.map(|progress| progress.started_plan),
|
||||
through,
|
||||
),
|
||||
)?;
|
||||
} else {
|
||||
// The receipt must survive a scanner leader handoff. The bucket
|
||||
// identity and publication epoch below fence the namespace; the
|
||||
// process-local leader epoch is intentionally excluded because it
|
||||
// changes during a safe handoff. The domain version also keeps
|
||||
// old epoch-bound receipts from colliding with this digest.
|
||||
serde_json::to_writer(
|
||||
&mut writer,
|
||||
&(
|
||||
2u8,
|
||||
&self.info.name,
|
||||
self.info.scan_identity,
|
||||
self.info.source,
|
||||
self.info.cache_key_format,
|
||||
self.info.scan_progress.map(|progress| progress.started_plan),
|
||||
through,
|
||||
),
|
||||
)?;
|
||||
}
|
||||
let mut prefix = self
|
||||
.cache
|
||||
.iter()
|
||||
@@ -1049,7 +1107,8 @@ impl DataUsageCache {
|
||||
.strip_prefix(&self.info.name)
|
||||
.is_some_and(|suffix| suffix.starts_with('/'))
|
||||
&& self.find(&receipt.through).is_some()
|
||||
&& self.coverage_prefix_digest(&receipt.through).ok() == Some(receipt.digest))
|
||||
&& (self.coverage_prefix_digest(&receipt.through).ok() == Some(receipt.digest)
|
||||
|| self.legacy_coverage_prefix_digest(&receipt.through).ok() == Some(receipt.digest)))
|
||||
.then_some(receipt.through.as_str())
|
||||
}
|
||||
|
||||
|
||||
@@ -135,6 +135,15 @@ impl DataUsageCache {
|
||||
Ok(DataUsageCacheRevisions { main, backup })
|
||||
}
|
||||
|
||||
pub(crate) async fn read_revisions<S: ScannerObjectIO>(store: Arc<S>, name: &str) -> StorageResult<DataUsageCacheRevisions> {
|
||||
let main_path = path_join_buf(&[BUCKET_META_PREFIX, name]);
|
||||
let backup_name = format!("{name}.bkp");
|
||||
let backup_path = path_join_buf(&[BUCKET_META_PREFIX, &backup_name]);
|
||||
let main = read_config_revision(store.clone(), &main_path).await?;
|
||||
let backup = Some(read_config_revision(store, &backup_path).await?);
|
||||
Ok(DataUsageCacheRevisions { main, backup })
|
||||
}
|
||||
|
||||
async fn load_cache<S: ScannerObjectIO>(store: Arc<S>, name: &str) -> StorageResult<DataUsageCacheLoadResult> {
|
||||
let mut last_retryable = None;
|
||||
|
||||
|
||||
@@ -2033,6 +2033,59 @@ fn data_usage_cache_prepare_for_scan_fences_leader_epochs() {
|
||||
assert!(cache.cache.is_empty());
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn prepare_bucket_checkpoint_migrates_legacy_epoch_bound_receipt() {
|
||||
let identity = valid_scan_identity();
|
||||
let source = DataUsageCacheSource::new(1, 2);
|
||||
let mut cache = DataUsageCache {
|
||||
info: DataUsageCacheInfo {
|
||||
name: "bucket".to_string(),
|
||||
next_cycle: 8,
|
||||
leader_epoch: 1,
|
||||
source: Some(source),
|
||||
cache_key_format: DATA_USAGE_CACHE_KEY_FORMAT,
|
||||
scan_identity: Some(identity),
|
||||
tier_registry_generation: Some(identity.tier_registry_generation),
|
||||
scan_progress: Some(DataUsageScanProgress {
|
||||
started_plan: TEST_PLAN_DIGEST,
|
||||
requested_plan: TEST_PLAN_DIGEST,
|
||||
}),
|
||||
scan_resume_after: Some("bucket/a".to_string()),
|
||||
scan_checkpoint: Some(DataUsageScanCheckpoint::new(
|
||||
"bucket/a".to_string(),
|
||||
DataUsageScanCheckpointReason::Objects,
|
||||
)),
|
||||
..Default::default()
|
||||
},
|
||||
..Default::default()
|
||||
};
|
||||
cache.replace("bucket", "", DataUsageEntry::default());
|
||||
cache.replace(
|
||||
"bucket/a",
|
||||
"bucket",
|
||||
DataUsageEntry {
|
||||
objects: 1,
|
||||
..Default::default()
|
||||
},
|
||||
);
|
||||
let legacy_digest = cache
|
||||
.legacy_coverage_prefix_digest("bucket/a")
|
||||
.expect("legacy receipt digest");
|
||||
cache.info.scan_coverage_receipt = Some(DataUsageScanCoverageReceipt {
|
||||
through: "bucket/a".to_string(),
|
||||
digest: legacy_digest,
|
||||
});
|
||||
assert_eq!(cache.validated_scan_frontier(), Some("bucket/a"));
|
||||
|
||||
assert_eq!(
|
||||
cache.prepare_bucket_checkpoint("bucket", 8, 2, source, TEST_PLAN_DIGEST, identity),
|
||||
DataUsageCachePrepareOutcome::Reused
|
||||
);
|
||||
assert_eq!(cache.info.leader_epoch, 2);
|
||||
assert_eq!(cache.validated_scan_frontier(), Some("bucket/a"));
|
||||
assert_ne!(cache.info.scan_coverage_receipt.expect("migrated receipt").digest, legacy_digest);
|
||||
}
|
||||
|
||||
#[test]
|
||||
fn test_data_usage_cache_mutations_update_in_place() {
|
||||
let mut cache = DataUsageCache {
|
||||
|
||||
@@ -170,7 +170,7 @@ fn cached_remote_scanner_fence_matches(
|
||||
.is_some_and(|cached| cached.matches(requested_cycle, requested_leader_epoch, now)))
|
||||
}
|
||||
|
||||
async fn validate_remote_scanner_request_fence_with_store(
|
||||
pub(crate) async fn validate_remote_scanner_request_fence_with_store(
|
||||
requested_cycle: u64,
|
||||
requested_leader_epoch: u64,
|
||||
store: Arc<impl ScannerObjectIO>,
|
||||
|
||||
@@ -16,9 +16,9 @@
|
||||
use crate::RUSTFS_META_BUCKET;
|
||||
use crate::scanner_budget::{ScannerCycleBudget, ScannerCycleBudgetConfig};
|
||||
use crate::scanner_io::{
|
||||
DataUsageCacheReuseOptions, DataUsageCacheScanState, ScannerDiskScanOptions, ScannerDiskScanOutcome, ScannerIODisk,
|
||||
acquire_scanner_cache_locks, cache_root_entry_info, current_cache_root_or_prepare_with_generation,
|
||||
scanner_set_disk_inventory,
|
||||
DataUsageCacheReuseOptions, DataUsageCacheScanState, ScannerCheckpointPersistContext, ScannerCheckpointPersistResult,
|
||||
ScannerDiskScanOptions, ScannerDiskScanOutcome, ScannerIODisk, acquire_scanner_cache_locks, cache_root_entry_info,
|
||||
current_cache_root_or_prepare_with_generation, persist_scanner_checkpoint, scanner_set_disk_inventory,
|
||||
};
|
||||
use crate::storage_api::owner::NS_SCANNER_PROTOCOL_VERSION;
|
||||
use crate::{
|
||||
@@ -43,7 +43,7 @@ use std::sync::{
|
||||
};
|
||||
use std::time::Duration;
|
||||
use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt};
|
||||
use tokio::sync::{Notify, OwnedSemaphorePermit};
|
||||
use tokio::sync::{Notify, OwnedSemaphorePermit, mpsc};
|
||||
use tokio::time::{Instant, MissedTickBehavior};
|
||||
use tokio_util::sync::CancellationToken;
|
||||
use uuid::Uuid;
|
||||
@@ -709,7 +709,7 @@ async fn scan_and_persist_local_bucket(
|
||||
}
|
||||
})?;
|
||||
let mut cache = DataUsageCache::default();
|
||||
let revisions = cache.load_with_revisions(set.clone(), &cache_name).await.map_err(|err| {
|
||||
let mut revisions = cache.load_with_revisions(set.clone(), &cache_name).await.map_err(|err| {
|
||||
RemoteScannerServerError::worker(format!("remote namespace scanner cache load or revision lookup failed: {err}"))
|
||||
})?;
|
||||
// Remote workers use the same cycle-frozen registry as `scan_data_folder`.
|
||||
@@ -779,6 +779,7 @@ async fn scan_and_persist_local_bucket(
|
||||
cache.info.skip_healing = skip_healing;
|
||||
|
||||
let set_disks = scanner_set_disk_inventory(set.as_ref()).await;
|
||||
let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel::<DataUsageCache>(1);
|
||||
let scan_ctx = ctx.child_token();
|
||||
let scan = ScannerIODisk::nsscanner_disk(
|
||||
disk.clone(),
|
||||
@@ -790,6 +791,7 @@ async fn scan_and_persist_local_bucket(
|
||||
ScannerDiskScanOptions {
|
||||
scan_mode,
|
||||
prefix_scan_scope: None,
|
||||
checkpoint_tx: Some(checkpoint_tx),
|
||||
},
|
||||
);
|
||||
tokio::pin!(scan);
|
||||
@@ -797,6 +799,7 @@ async fn scan_and_persist_local_bucket(
|
||||
tokio::pin!(fence_watch);
|
||||
let mut lock_watch = tokio::time::interval(NS_SCANNER_LOCK_POLL_INTERVAL);
|
||||
lock_watch.set_missed_tick_behavior(MissedTickBehavior::Delay);
|
||||
let mut checkpoint_channel_closed = false;
|
||||
let outcome = loop {
|
||||
tokio::select! {
|
||||
result = &mut scan => {
|
||||
@@ -823,6 +826,53 @@ async fn scan_and_persist_local_bucket(
|
||||
));
|
||||
}
|
||||
}
|
||||
checkpoint = checkpoint_rx.recv(), if !checkpoint_channel_closed => {
|
||||
let Some(checkpoint) = checkpoint else {
|
||||
checkpoint_channel_closed = true;
|
||||
continue;
|
||||
};
|
||||
if guard.is_lock_lost() {
|
||||
scan_ctx.cancel();
|
||||
let _ = tokio::time::timeout(NS_SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT, scan.as_mut()).await;
|
||||
return Err(RemoteScannerServerError::worker(
|
||||
"remote namespace scanner cache lock was lost before checkpoint save",
|
||||
));
|
||||
}
|
||||
match persist_scanner_checkpoint(
|
||||
set.clone(),
|
||||
ScannerCheckpointPersistContext {
|
||||
ctx: &scan_ctx,
|
||||
expected_publication_epoch,
|
||||
cycle: next_cycle,
|
||||
leader_epoch,
|
||||
},
|
||||
&cache_name,
|
||||
&checkpoint,
|
||||
&mut revisions,
|
||||
)
|
||||
.await
|
||||
{
|
||||
ScannerCheckpointPersistResult::Saved => {
|
||||
if guard.is_lock_lost() {
|
||||
scan_ctx.cancel();
|
||||
let _ = tokio::time::timeout(NS_SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT, scan.as_mut()).await;
|
||||
return Err(RemoteScannerServerError::retry_bucket(
|
||||
"remote namespace scanner cache lock was lost after checkpoint save",
|
||||
));
|
||||
}
|
||||
}
|
||||
ScannerCheckpointPersistResult::FenceChanged => {
|
||||
scan_ctx.cancel();
|
||||
let _ = tokio::time::timeout(NS_SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT, scan.as_mut()).await;
|
||||
return Err(RemoteScannerServerError::retry_bucket(
|
||||
"remote namespace scanner cache fence changed during checkpoint save",
|
||||
));
|
||||
}
|
||||
ScannerCheckpointPersistResult::Failed(_error) => {
|
||||
checkpoint_channel_closed = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
result = &mut fence_watch => {
|
||||
scan_ctx.cancel();
|
||||
let _ = tokio::time::timeout(NS_SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT, scan.as_mut()).await;
|
||||
|
||||
@@ -1778,9 +1778,7 @@ where
|
||||
mark_scan_cycle_idle(cycle_info, &mut cycle_metrics_guard).await;
|
||||
return ScannerCycleOutcome::Failed;
|
||||
}
|
||||
BackgroundHealInfoReadStatus::ErasureSd
|
||||
| BackgroundHealInfoReadStatus::Loaded
|
||||
| BackgroundHealInfoReadStatus::Missing => {}
|
||||
BackgroundHealInfoReadStatus::Loaded | BackgroundHealInfoReadStatus::Missing => {}
|
||||
}
|
||||
let mut background_heal_info = background_heal_read.info;
|
||||
let background_heal_epoch = background_heal_read.expected_epoch;
|
||||
|
||||
@@ -24,14 +24,15 @@ use crate::data_usage_define::DataUsageCacheRevision;
|
||||
use crate::storage_api::ScannerStorage;
|
||||
use crate::storage_api::owner::ObjectIO as _;
|
||||
use crate::storage_api::owner::{
|
||||
MAX_SCANNER_PAUSE_BACKLOG_BYTES, ScannerPauseBacklogRetirementPlan, ScannerPauseBacklogRetirementReplica,
|
||||
register_scanner_pause_backlog_retirement_planner,
|
||||
MAX_SCANNER_PAUSE_BACKLOG_BYTES, ScannerPauseBacklogRetirementError, ScannerPauseBacklogRetirementPlan,
|
||||
ScannerPauseBacklogRetirementReplica, register_scanner_pause_backlog_retirement_planner,
|
||||
};
|
||||
use crate::{BUCKET_META_PREFIX, ECStore, EcstoreError, RUSTFS_META_BUCKET, ScannerObjectOptions, SetDisks};
|
||||
use futures::future::join_all;
|
||||
use http::HeaderMap;
|
||||
use serde::{Deserialize, Serialize};
|
||||
use std::collections::{BTreeSet, HashMap};
|
||||
use std::fmt;
|
||||
use std::sync::{Arc, LazyLock, RwLock};
|
||||
use std::time::{Duration, SystemTime, UNIX_EPOCH};
|
||||
use tokio::io::AsyncReadExt;
|
||||
@@ -996,10 +997,77 @@ fn scanner_pause_backlog_replica_ids(replicas: &[ScannerPauseBacklogReplica]) ->
|
||||
ids
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
enum ScannerPauseBacklogCommitSelectionError {
|
||||
ConflictingMaximumMembershipCommitProofs,
|
||||
}
|
||||
|
||||
impl fmt::Display for ScannerPauseBacklogCommitSelectionError {
|
||||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||
f.write_str("scanner pause backlog has conflicting maximum-membership commit proofs")
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Clone, Debug)]
|
||||
enum ScannerPauseBacklogSelectionError {
|
||||
RetryableHandoffConflict,
|
||||
AuthorityConflict(String),
|
||||
InvalidRecord(String),
|
||||
}
|
||||
|
||||
impl fmt::Display for ScannerPauseBacklogSelectionError {
|
||||
fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
|
||||
match self {
|
||||
Self::RetryableHandoffConflict => {
|
||||
f.write_str("scanner pause backlog has conflicting maximum-membership commit proofs")
|
||||
}
|
||||
Self::AuthorityConflict(reason) | Self::InvalidRecord(reason) => f.write_str(reason),
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
impl From<ScannerPauseBacklogCommitSelectionError> for ScannerPauseBacklogSelectionError {
|
||||
fn from(_err: ScannerPauseBacklogCommitSelectionError) -> Self {
|
||||
Self::RetryableHandoffConflict
|
||||
}
|
||||
}
|
||||
|
||||
fn scanner_pause_backlog_retirement_selection_error(
|
||||
err: ScannerPauseBacklogSelectionError,
|
||||
) -> ScannerPauseBacklogRetirementError {
|
||||
match err {
|
||||
ScannerPauseBacklogSelectionError::RetryableHandoffConflict => {
|
||||
ScannerPauseBacklogRetirementError::HandoffConflict { reason: err.to_string() }
|
||||
}
|
||||
ScannerPauseBacklogSelectionError::AuthorityConflict(reason) => {
|
||||
ScannerPauseBacklogRetirementError::AuthorityConflict { reason }
|
||||
}
|
||||
ScannerPauseBacklogSelectionError::InvalidRecord(reason) => ScannerPauseBacklogRetirementError::InvalidRecord { reason },
|
||||
}
|
||||
}
|
||||
|
||||
fn scanner_pause_backlog_retirement_commit_error(
|
||||
err: ScannerPauseBacklogCommitSelectionError,
|
||||
) -> ScannerPauseBacklogRetirementError {
|
||||
scanner_pause_backlog_retirement_handoff_conflict(err.to_string())
|
||||
}
|
||||
|
||||
fn scanner_pause_backlog_retirement_handoff_conflict(reason: impl Into<String>) -> ScannerPauseBacklogRetirementError {
|
||||
ScannerPauseBacklogRetirementError::HandoffConflict { reason: reason.into() }
|
||||
}
|
||||
|
||||
fn scanner_pause_backlog_retirement_authority_conflict(reason: impl Into<String>) -> ScannerPauseBacklogRetirementError {
|
||||
ScannerPauseBacklogRetirementError::AuthorityConflict { reason: reason.into() }
|
||||
}
|
||||
|
||||
fn scanner_pause_backlog_retirement_invalid_record(reason: impl Into<String>) -> ScannerPauseBacklogRetirementError {
|
||||
ScannerPauseBacklogRetirementError::InvalidRecord { reason: reason.into() }
|
||||
}
|
||||
|
||||
fn select_scanner_pause_backlog_commit(
|
||||
replicas: &[ScannerPauseBacklogReplica],
|
||||
replica_ids: &[ScannerPauseBacklogReplicaId],
|
||||
) -> Result<Option<ScannerPauseBacklogCommitRecord>, String> {
|
||||
) -> Result<Option<ScannerPauseBacklogCommitRecord>, ScannerPauseBacklogCommitSelectionError> {
|
||||
let current_ids = replica_ids.iter().copied().collect::<BTreeSet<_>>();
|
||||
let replicas_by_id = replicas
|
||||
.iter()
|
||||
@@ -1031,15 +1099,19 @@ fn select_scanner_pause_backlog_commit(
|
||||
let Some(max_membership_len) = valid.iter().map(|committed| committed.replicas.len()).max() else {
|
||||
return Ok(None);
|
||||
};
|
||||
let mut largest = valid
|
||||
let largest = valid
|
||||
.into_iter()
|
||||
.filter(|committed| committed.replicas.len() == max_membership_len);
|
||||
let Some(mut selected) = largest.next() else {
|
||||
let largest = largest.collect::<Vec<_>>();
|
||||
let Some(mut selected) = largest.first().cloned() else {
|
||||
return Ok(None);
|
||||
};
|
||||
for committed in largest {
|
||||
if largest.iter().any(|committed| committed.ledger != selected.ledger) {
|
||||
return Err(ScannerPauseBacklogCommitSelectionError::ConflictingMaximumMembershipCommitProofs);
|
||||
}
|
||||
for committed in largest.into_iter().skip(1) {
|
||||
if committed.ledger != selected.ledger {
|
||||
return Err("scanner pause backlog has conflicting maximum-membership commit proofs".to_string());
|
||||
unreachable!("conflicting maximum-membership commit proofs were checked above");
|
||||
}
|
||||
if committed.replicas < selected.replicas {
|
||||
selected = committed;
|
||||
@@ -1048,21 +1120,26 @@ fn select_scanner_pause_backlog_commit(
|
||||
Ok(Some(selected))
|
||||
}
|
||||
|
||||
fn select_scanner_pause_backlog_replicas(replicas: Vec<ScannerPauseBacklogReplica>) -> Result<LoadedScannerPauseBacklog, String> {
|
||||
fn select_scanner_pause_backlog_replicas(
|
||||
replicas: Vec<ScannerPauseBacklogReplica>,
|
||||
) -> Result<LoadedScannerPauseBacklog, ScannerPauseBacklogSelectionError> {
|
||||
if replicas.is_empty() {
|
||||
return Err("scanner pause backlog has no storage replicas".to_string());
|
||||
return Err(ScannerPauseBacklogSelectionError::AuthorityConflict(
|
||||
"scanner pause backlog has no storage replicas".to_string(),
|
||||
));
|
||||
}
|
||||
for replica in &replicas {
|
||||
if let ScannerPauseBacklogReplicaState::FutureSchema(version) = &replica.state {
|
||||
return Err(format!(
|
||||
return Err(ScannerPauseBacklogSelectionError::InvalidRecord(format!(
|
||||
"scanner pause backlog pool {} set {} uses future schema {version}",
|
||||
replica.id.pool_index, replica.id.set_index
|
||||
));
|
||||
)));
|
||||
}
|
||||
}
|
||||
|
||||
let replica_ids = scanner_pause_backlog_replica_ids(&replicas);
|
||||
let authoritative_commit = select_scanner_pause_backlog_commit(&replicas, &replica_ids)?;
|
||||
let authoritative_commit =
|
||||
select_scanner_pause_backlog_commit(&replicas, &replica_ids).map_err(ScannerPauseBacklogSelectionError::from)?;
|
||||
let stable_consensus = if replicas.iter().all(|replica| {
|
||||
matches!(
|
||||
&replica.state,
|
||||
@@ -1093,17 +1170,21 @@ fn select_scanner_pause_backlog_replicas(replicas: Vec<ScannerPauseBacklogReplic
|
||||
}
|
||||
_ => unreachable!("replica state was checked above"),
|
||||
};
|
||||
return Err(format!(
|
||||
let reason = format!(
|
||||
"scanner pause backlog pool {} set {} is unavailable: {reason}",
|
||||
replica.id.pool_index, replica.id.set_index
|
||||
));
|
||||
);
|
||||
return Err(match &replica.state {
|
||||
ScannerPauseBacklogReplicaState::Invalid(_) => ScannerPauseBacklogSelectionError::InvalidRecord(reason),
|
||||
_ => ScannerPauseBacklogSelectionError::AuthorityConflict(reason),
|
||||
});
|
||||
}
|
||||
match &stable_consensus {
|
||||
Ok(stable) => stable.clone(),
|
||||
Err(()) => {
|
||||
return Err(
|
||||
return Err(ScannerPauseBacklogSelectionError::AuthorityConflict(
|
||||
"scanner pause backlog has neither a surviving membership commit nor a stable rollback point".to_string(),
|
||||
);
|
||||
));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -1192,7 +1273,7 @@ pub fn register_scanner_pause_backlog_retirement() {
|
||||
fn decode_scanner_pause_backlog_retirement_replicas(
|
||||
source_pool_index: usize,
|
||||
replicas: &[ScannerPauseBacklogRetirementReplica],
|
||||
) -> Result<Vec<ScannerPauseBacklogReplica>, String> {
|
||||
) -> Result<Vec<ScannerPauseBacklogReplica>, ScannerPauseBacklogRetirementError> {
|
||||
let mut ids = BTreeSet::new();
|
||||
let mut decoded = Vec::with_capacity(replicas.len());
|
||||
let mut has_source = false;
|
||||
@@ -1202,19 +1283,27 @@ fn decode_scanner_pause_backlog_retirement_replicas(
|
||||
set_index: replica.set_index,
|
||||
};
|
||||
if !ids.insert(id) {
|
||||
return Err("scanner pause backlog retirement has duplicate replica membership".to_string());
|
||||
return Err(scanner_pause_backlog_retirement_invalid_record(
|
||||
"scanner pause backlog retirement has duplicate replica membership",
|
||||
));
|
||||
}
|
||||
let state = match replica.data.as_deref() {
|
||||
Some(data) if data.len() <= MAX_SCANNER_PAUSE_BACKLOG_BYTES as usize => {
|
||||
let state = decode_scanner_pause_backlog_ledger(data);
|
||||
if !matches!(state, ScannerPauseBacklogReplicaState::Valid(_)) {
|
||||
return Err("scanner pause backlog retirement found an invalid or unsupported native record".to_string());
|
||||
return Err(scanner_pause_backlog_retirement_invalid_record(
|
||||
"scanner pause backlog retirement found an invalid or unsupported native record",
|
||||
));
|
||||
}
|
||||
has_source |= replica.pool_index == source_pool_index;
|
||||
state
|
||||
}
|
||||
None => ScannerPauseBacklogReplicaState::Missing,
|
||||
_ => return Err("scanner pause backlog retirement has an oversized replica".to_string()),
|
||||
_ => {
|
||||
return Err(scanner_pause_backlog_retirement_invalid_record(
|
||||
"scanner pause backlog retirement has an oversized replica",
|
||||
));
|
||||
}
|
||||
};
|
||||
decoded.push(ScannerPauseBacklogReplica {
|
||||
id,
|
||||
@@ -1223,7 +1312,9 @@ fn decode_scanner_pause_backlog_retirement_replicas(
|
||||
});
|
||||
}
|
||||
if !has_source {
|
||||
return Err("scanner pause backlog retirement has no native source record".to_string());
|
||||
return Err(scanner_pause_backlog_retirement_invalid_record(
|
||||
"scanner pause backlog retirement has no native source record",
|
||||
));
|
||||
}
|
||||
Ok(decoded)
|
||||
}
|
||||
@@ -1231,12 +1322,14 @@ fn decode_scanner_pause_backlog_retirement_replicas(
|
||||
fn verify_scanner_pause_backlog_retirement(
|
||||
source_pool_index: usize,
|
||||
replicas: &[ScannerPauseBacklogRetirementReplica],
|
||||
) -> Result<(), String> {
|
||||
) -> Result<(), ScannerPauseBacklogRetirementError> {
|
||||
let mut decoded = decode_scanner_pause_backlog_retirement_replicas(source_pool_index, replicas)?;
|
||||
if decoded.iter().any(|replica| {
|
||||
replica.id.pool_index != source_pool_index && matches!(replica.state, ScannerPauseBacklogReplicaState::Missing)
|
||||
}) {
|
||||
return Err("scanner pause backlog retirement has a missing surviving replica".to_string());
|
||||
return Err(scanner_pause_backlog_retirement_authority_conflict(
|
||||
"scanner pause backlog retirement has a missing surviving replica",
|
||||
));
|
||||
}
|
||||
// Earlier entries may already have removed a source sibling after a
|
||||
// successful handoff. It contributes no stored authority to final cleanup.
|
||||
@@ -1246,15 +1339,17 @@ fn verify_scanner_pause_backlog_retirement(
|
||||
.filter(|replica| replica.id.pool_index != source_pool_index)
|
||||
.cloned()
|
||||
.collect::<Vec<_>>();
|
||||
let selected = select_scanner_pause_backlog_replicas(surviving)?;
|
||||
let selected = select_scanner_pause_backlog_replicas(surviving).map_err(scanner_pause_backlog_retirement_selection_error)?;
|
||||
if !selected.durable || !selected.stable_matches_ledger {
|
||||
return Err(
|
||||
"scanner pause backlog retirement requires stable authority on the complete surviving membership".to_string(),
|
||||
);
|
||||
return Err(scanner_pause_backlog_retirement_authority_conflict(
|
||||
"scanner pause backlog retirement requires stable authority on the complete surviving membership",
|
||||
));
|
||||
}
|
||||
let before = select_scanner_pause_backlog_replicas(decoded)?;
|
||||
let before = select_scanner_pause_backlog_replicas(decoded).map_err(scanner_pause_backlog_retirement_selection_error)?;
|
||||
if !before.durable || before.ledger != selected.ledger {
|
||||
return Err("scanner pause backlog retirement would change the native ledger authority".to_string());
|
||||
return Err(scanner_pause_backlog_retirement_authority_conflict(
|
||||
"scanner pause backlog retirement would change the native ledger authority",
|
||||
));
|
||||
}
|
||||
Ok(())
|
||||
}
|
||||
@@ -1262,7 +1357,7 @@ fn verify_scanner_pause_backlog_retirement(
|
||||
fn plan_scanner_pause_backlog_retirement(
|
||||
source_pool_index: usize,
|
||||
replicas: &[ScannerPauseBacklogRetirementReplica],
|
||||
) -> Result<Option<ScannerPauseBacklogRetirementPlan>, String> {
|
||||
) -> Result<Option<ScannerPauseBacklogRetirementPlan>, ScannerPauseBacklogRetirementError> {
|
||||
// Missing entries stay in the native selection. Omitting them could turn
|
||||
// an incomplete old cohort into a fabricated stable consensus.
|
||||
let decoded = decode_scanner_pause_backlog_retirement_replicas(source_pool_index, replicas)?;
|
||||
@@ -1272,12 +1367,16 @@ fn plan_scanner_pause_backlog_retirement(
|
||||
.cloned()
|
||||
.collect::<Vec<_>>();
|
||||
if surviving.is_empty() {
|
||||
return Err("scanner pause backlog retirement has no surviving membership".to_string());
|
||||
return Err(scanner_pause_backlog_retirement_authority_conflict(
|
||||
"scanner pause backlog retirement has no surviving membership",
|
||||
));
|
||||
}
|
||||
let all_ids = scanner_pause_backlog_replica_ids(&decoded);
|
||||
let surviving_ids = scanner_pause_backlog_replica_ids(&surviving);
|
||||
let full_commit = select_scanner_pause_backlog_commit(&decoded, &all_ids)?;
|
||||
let surviving_commit = select_scanner_pause_backlog_commit(&surviving, &surviving_ids)?;
|
||||
let full_commit =
|
||||
select_scanner_pause_backlog_commit(&decoded, &all_ids).map_err(scanner_pause_backlog_retirement_commit_error)?;
|
||||
let surviving_commit =
|
||||
select_scanner_pause_backlog_commit(&surviving, &surviving_ids).map_err(scanner_pause_backlog_retirement_commit_error)?;
|
||||
let selected = select_scanner_pause_backlog_replicas(surviving.clone());
|
||||
let full_selection = select_scanner_pause_backlog_replicas(decoded.clone());
|
||||
if full_commit.is_none()
|
||||
@@ -1293,15 +1392,20 @@ fn plan_scanner_pause_backlog_retirement(
|
||||
if record.committed.as_ref().is_some_and(|committed|
|
||||
committed.replicas.iter().any(|id| !all_ids.contains(id))))
|
||||
}) {
|
||||
return Err("scanner pause backlog bootstrap has an unread native commit member".to_string());
|
||||
return Err(scanner_pause_backlog_retirement_invalid_record(
|
||||
"scanner pause backlog bootstrap has an unread native commit member",
|
||||
));
|
||||
}
|
||||
let ledger = claim_scanner_pause_backlog_writer(&all.ledger, unix_now())?;
|
||||
let ledger = claim_scanner_pause_backlog_writer(&all.ledger, unix_now())
|
||||
.map_err(scanner_pause_backlog_retirement_invalid_record)?;
|
||||
let committed = commit_scanner_pause_backlog_record(None, &ledger, &surviving_ids);
|
||||
let stable = stable_scanner_pause_backlog_record(&ledger, committed.committed.as_ref(), &surviving_ids);
|
||||
return Ok(Some(ScannerPauseBacklogRetirementPlan {
|
||||
seed_record: None,
|
||||
commit_record: encode_scanner_pause_backlog_record(&committed)?,
|
||||
stable_record: encode_scanner_pause_backlog_record(&stable)?,
|
||||
commit_record: encode_scanner_pause_backlog_record(&committed)
|
||||
.map_err(scanner_pause_backlog_retirement_invalid_record)?,
|
||||
stable_record: encode_scanner_pause_backlog_record(&stable)
|
||||
.map_err(scanner_pause_backlog_retirement_invalid_record)?,
|
||||
}));
|
||||
}
|
||||
let ledger = if let Some(committed) = &full_commit {
|
||||
@@ -1309,7 +1413,9 @@ fn plan_scanner_pause_backlog_retirement(
|
||||
.as_ref()
|
||||
.is_some_and(|current| current.ledger != committed.ledger)
|
||||
{
|
||||
return Err("scanner pause backlog retirement found a different surviving native commit".to_string());
|
||||
return Err(scanner_pause_backlog_retirement_authority_conflict(
|
||||
"scanner pause backlog retirement found a different surviving native commit",
|
||||
));
|
||||
}
|
||||
if let Ok(current) = &selected
|
||||
&& current.durable
|
||||
@@ -1324,20 +1430,28 @@ fn plan_scanner_pause_backlog_retirement(
|
||||
if record.committed.as_ref() == Some(committed))
|
||||
});
|
||||
if !acknowledged_full_commit {
|
||||
return Err("scanner pause backlog retirement would replace surviving stable authority".to_string());
|
||||
return Err(scanner_pause_backlog_retirement_authority_conflict(
|
||||
"scanner pause backlog retirement would replace surviving stable authority",
|
||||
));
|
||||
}
|
||||
}
|
||||
// Use the same native selector that validates the full commit, rather
|
||||
// than inferring authority from source epoch or physical object time.
|
||||
full_selection?.ledger
|
||||
full_selection
|
||||
.map_err(scanner_pause_backlog_retirement_selection_error)?
|
||||
.ledger
|
||||
} else {
|
||||
// An interrupted cohort switch can invalidate the old full commit
|
||||
// while all survivors still have its stable rollback point. Source
|
||||
// stable fields need not match, but any valid conflicting commit above
|
||||
// must have been rejected before this recovery path.
|
||||
let current = selected.as_ref().map_err(|err| err.clone())?;
|
||||
let current = selected
|
||||
.as_ref()
|
||||
.map_err(|err| scanner_pause_backlog_retirement_selection_error(err.clone()))?;
|
||||
if !current.durable {
|
||||
return Err("scanner pause backlog retirement has no proven native authority".to_string());
|
||||
return Err(scanner_pause_backlog_retirement_authority_conflict(
|
||||
"scanner pause backlog retirement has no proven native authority",
|
||||
));
|
||||
}
|
||||
if decoded
|
||||
.iter()
|
||||
@@ -1348,7 +1462,9 @@ fn plan_scanner_pause_backlog_retirement(
|
||||
&& record.committed.as_ref().is_none_or(|committed| committed.ledger != current.ledger))
|
||||
})
|
||||
{
|
||||
return Err("scanner pause backlog retirement has unrelated source stable authority".to_string());
|
||||
return Err(scanner_pause_backlog_retirement_authority_conflict(
|
||||
"scanner pause backlog retirement has unrelated source stable authority",
|
||||
));
|
||||
}
|
||||
current.ledger.clone()
|
||||
};
|
||||
@@ -1370,19 +1486,23 @@ fn plan_scanner_pause_backlog_retirement(
|
||||
.as_ref()
|
||||
.filter(|committed| committed.ledger == ledger)
|
||||
.or_else(|| surviving_commit.as_ref().filter(|committed| committed.ledger == ledger))
|
||||
.ok_or_else(|| "scanner pause backlog retirement cannot seed without a native commit proof".to_string())?;
|
||||
Some(encode_scanner_pause_backlog_record(&stable_scanner_pause_backlog_record(
|
||||
&ledger,
|
||||
Some(authority),
|
||||
&surviving_ids,
|
||||
))?)
|
||||
.ok_or_else(|| {
|
||||
scanner_pause_backlog_retirement_authority_conflict(
|
||||
"scanner pause backlog retirement cannot seed without a native commit proof",
|
||||
)
|
||||
})?;
|
||||
Some(
|
||||
encode_scanner_pause_backlog_record(&stable_scanner_pause_backlog_record(&ledger, Some(authority), &surviving_ids))
|
||||
.map_err(scanner_pause_backlog_retirement_invalid_record)?,
|
||||
)
|
||||
};
|
||||
let committed = commit_scanner_pause_backlog_record(Some(&ledger), &ledger, &surviving_ids);
|
||||
let stable = stable_scanner_pause_backlog_record(&ledger, committed.committed.as_ref(), &surviving_ids);
|
||||
Ok(Some(ScannerPauseBacklogRetirementPlan {
|
||||
seed_record,
|
||||
commit_record: encode_scanner_pause_backlog_record(&committed)?,
|
||||
stable_record: encode_scanner_pause_backlog_record(&stable)?,
|
||||
commit_record: encode_scanner_pause_backlog_record(&committed)
|
||||
.map_err(scanner_pause_backlog_retirement_invalid_record)?,
|
||||
stable_record: encode_scanner_pause_backlog_record(&stable).map_err(scanner_pause_backlog_retirement_invalid_record)?,
|
||||
}))
|
||||
}
|
||||
|
||||
@@ -1432,7 +1552,7 @@ where
|
||||
return Err("scanner pause backlog has no surviving storage replicas".to_string());
|
||||
}
|
||||
let replicas = join_all(writable.into_iter().map(read_scanner_pause_backlog_replica)).await;
|
||||
select_scanner_pause_backlog_replicas(replicas)
|
||||
select_scanner_pause_backlog_replicas(replicas).map_err(|err| err.to_string())
|
||||
}
|
||||
|
||||
async fn write_scanner_pause_backlog_record<S>(
|
||||
@@ -3010,7 +3130,10 @@ mod tests {
|
||||
];
|
||||
let blocked = verify_scanner_pause_backlog_retirement(0, &replicas)
|
||||
.expect_err("a complete commit still needs its native stabilization barrier");
|
||||
assert!(blocked.contains("stable authority"), "{blocked}");
|
||||
assert!(
|
||||
matches!(blocked, ScannerPauseBacklogRetirementError::AuthorityConflict { .. }),
|
||||
"a commit without the native stabilization barrier is an authority conflict: {blocked}"
|
||||
);
|
||||
|
||||
let stable = replica_record_for_members(&new, &new, &targets);
|
||||
replicas[1] = retirement_replica(targets[0], &stable);
|
||||
@@ -3032,7 +3155,10 @@ mod tests {
|
||||
new.claim_writer(100).unwrap();
|
||||
prepare_scanner_pause_backlog_persist(&mut new, 100).unwrap();
|
||||
let stable = replica_record_for_members(&new, &new, &targets);
|
||||
for source_sets in [2, 3] {
|
||||
// Equal-size competing source proofs are a transient handoff conflict
|
||||
// the native writer can still resolve. A strictly larger source proof
|
||||
// is durable authority that must never be discarded.
|
||||
for (source_sets, authority_conflict) in [(2, false), (3, true)] {
|
||||
let sources = (0..source_sets).map(|set| replica_id(0, set)).collect::<Vec<_>>();
|
||||
let source_record = replica_record_for_members(&old, &old, &sources);
|
||||
let mut replicas = sources
|
||||
@@ -3040,8 +3166,13 @@ mod tests {
|
||||
.map(|id| retirement_replica(*id, &source_record))
|
||||
.collect::<Vec<_>>();
|
||||
replicas.extend(targets.iter().map(|id| retirement_replica(*id, &stable)));
|
||||
verify_scanner_pause_backlog_retirement(0, &replicas)
|
||||
let error = verify_scanner_pause_backlog_retirement(0, &replicas)
|
||||
.expect_err("all source sets must be consulted before discarding a competing or larger native proof");
|
||||
match (authority_conflict, &error) {
|
||||
(true, ScannerPauseBacklogRetirementError::AuthorityConflict { .. })
|
||||
| (false, ScannerPauseBacklogRetirementError::HandoffConflict { .. }) => {}
|
||||
_ => panic!("competing source proofs must keep their typed classification: {error}"),
|
||||
}
|
||||
assert!(plan_scanner_pause_backlog_retirement(0, &replicas).is_err());
|
||||
}
|
||||
}
|
||||
@@ -3068,7 +3199,10 @@ mod tests {
|
||||
let error = plan_scanner_pause_backlog_retirement(0, &replicas)
|
||||
.err()
|
||||
.expect("bootstrap must not turn an unread old cohort into a missing member");
|
||||
assert!(error.contains("unread"), "{error}");
|
||||
let ScannerPauseBacklogRetirementError::InvalidRecord { reason } = &error else {
|
||||
panic!("an unread old cohort member is an invalid record, not a missing member: {error}");
|
||||
};
|
||||
assert!(reason.contains("unread"), "{reason}");
|
||||
|
||||
let new = durable_ledger(100);
|
||||
let source_record = replica_record_for_members(&old, &old, &[source]);
|
||||
@@ -3079,7 +3213,10 @@ mod tests {
|
||||
let error = plan_scanner_pause_backlog_retirement(0, &replicas)
|
||||
.err()
|
||||
.expect("an independent source-only proof must not overwrite stable surviving authority");
|
||||
assert!(error.contains("stable authority"), "{error}");
|
||||
assert!(
|
||||
matches!(error, ScannerPauseBacklogRetirementError::AuthorityConflict { .. }),
|
||||
"an independent source-only proof must be an authority conflict: {error}"
|
||||
);
|
||||
}
|
||||
|
||||
#[derive(Clone, Copy)]
|
||||
|
||||
@@ -28,7 +28,6 @@ pub struct BackgroundHealInfo {
|
||||
|
||||
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
|
||||
pub(super) enum BackgroundHealInfoReadStatus {
|
||||
ErasureSd,
|
||||
Loaded,
|
||||
Missing,
|
||||
Blocked,
|
||||
@@ -66,15 +65,6 @@ pub(super) async fn read_background_heal_info_with_epoch<S>(storeapi: Arc<S>) ->
|
||||
where
|
||||
S: ScannerStorage,
|
||||
{
|
||||
// Skip for ErasureSD setup
|
||||
if storeapi.setup_is_erasure_sd().await {
|
||||
return BackgroundHealInfoRead {
|
||||
info: BackgroundHealInfo::default(),
|
||||
expected_epoch: None,
|
||||
status: BackgroundHealInfoReadStatus::ErasureSd,
|
||||
};
|
||||
}
|
||||
|
||||
let expected_epoch = scanner_publication_epoch(storeapi.clone()).await;
|
||||
if expected_epoch.is_none() {
|
||||
return BackgroundHealInfoRead {
|
||||
@@ -146,11 +136,6 @@ pub(super) async fn save_background_heal_info_for_epoch<S>(
|
||||
) where
|
||||
S: ScannerStorage,
|
||||
{
|
||||
// Skip for ErasureSD setup
|
||||
if storeapi.setup_is_erasure_sd().await {
|
||||
return;
|
||||
}
|
||||
|
||||
// Serialize to JSON
|
||||
let data = match serde_json::to_vec(&info) {
|
||||
Ok(data) => data,
|
||||
|
||||
@@ -91,6 +91,9 @@ const DATA_SCANNER_FORCE_COMPACT_AT_FOLDERS: usize = 250_000;
|
||||
const SCANNER_LIST_PATH_RAW_STALL_TIMEOUT: Duration = Duration::from_secs(60);
|
||||
const SCANNER_ENTRY_PROGRESS_BATCH: u64 = 32;
|
||||
const SCANNER_ENTRY_PROGRESS_INTERVAL: Duration = Duration::from_secs(30);
|
||||
const SCANNER_CHECKPOINT_OBJECT_INTERVAL: u64 = 1024;
|
||||
const SCANNER_CHECKPOINT_MIN_INTERVAL: Duration = Duration::from_secs(5);
|
||||
const SCANNER_CHECKPOINT_INTERVAL: Duration = Duration::from_secs(60);
|
||||
const SCANNER_RAW_ENUMERATION_PAGE_ENTRY_LIMIT: usize = 128;
|
||||
const SCANNER_RAW_ENUMERATION_PAGE_BUILD_BUDGET: usize = 1;
|
||||
// Erasure data directories contain direct part.N files; keep namespace probes bounded.
|
||||
@@ -723,7 +726,11 @@ pub struct FolderScanner {
|
||||
disks_quorum: usize,
|
||||
|
||||
updates: Option<mpsc::Sender<DataUsageEntry>>,
|
||||
checkpoint_tx: Option<mpsc::Sender<DataUsageCache>>,
|
||||
last_update: SystemTime,
|
||||
checkpoint_objects: u64,
|
||||
last_checkpoint_objects: u64,
|
||||
last_checkpoint_at: Instant,
|
||||
|
||||
update_current_path: UpdateCurrentPathFn,
|
||||
|
||||
@@ -1204,6 +1211,58 @@ impl FolderScanner {
|
||||
(progress.cursor(), progress.page_index())
|
||||
}
|
||||
|
||||
fn peek_raw_enumeration_resume_state(&self) -> (Option<DataUsageRawEnumerationCursor>, Option<RawEnumerationPageIndex>) {
|
||||
self.raw_enumeration_progress
|
||||
.iter()
|
||||
.max_by_key(|progress| progress.checkpointable_entry_count())
|
||||
.map_or((None, None), |progress| (progress.cursor(), progress.page_index()))
|
||||
}
|
||||
|
||||
fn maybe_send_checkpoint(&mut self) {
|
||||
let Some(checkpoint_tx) = self.checkpoint_tx.as_ref() else { return };
|
||||
let elapsed = self.last_checkpoint_at.elapsed();
|
||||
if self.new_cache.info.scan_progress.is_none()
|
||||
|| elapsed < SCANNER_CHECKPOINT_MIN_INTERVAL
|
||||
|| (self.checkpoint_objects.saturating_sub(self.last_checkpoint_objects) < SCANNER_CHECKPOINT_OBJECT_INTERVAL
|
||||
&& elapsed < SCANNER_CHECKPOINT_INTERVAL)
|
||||
{
|
||||
return;
|
||||
}
|
||||
if self.new_cache.root().is_none() && self.raw_enumeration_progress.is_empty() {
|
||||
return;
|
||||
}
|
||||
|
||||
let mut snapshot = self.new_cache.clone();
|
||||
snapshot.info.last_update = Some(SystemTime::now());
|
||||
snapshot.info.snapshot_complete = false;
|
||||
let (cursor, page_index) = self.peek_raw_enumeration_resume_state();
|
||||
if cursor.is_some() || page_index.is_some() {
|
||||
snapshot.info.scan_raw_enumeration_cursor = cursor;
|
||||
snapshot.info.scan_raw_enumeration_page_index = page_index;
|
||||
snapshot.info.scan_resume_after = None;
|
||||
snapshot.info.scan_checkpoint = None;
|
||||
snapshot.info.scan_coverage_receipt = None;
|
||||
} else if snapshot.seal_scan_frontier(self.coverage_frontier.as_deref()).is_err() {
|
||||
return;
|
||||
}
|
||||
if snapshot.root().is_none()
|
||||
&& snapshot.info.scan_raw_enumeration_cursor.is_none()
|
||||
&& snapshot.info.scan_raw_enumeration_page_index.is_none()
|
||||
{
|
||||
return;
|
||||
}
|
||||
if snapshot.info.scan_raw_enumeration_cursor.is_none()
|
||||
&& snapshot.info.scan_raw_enumeration_page_index.is_none()
|
||||
&& snapshot.validated_scan_frontier().is_none()
|
||||
{
|
||||
return;
|
||||
}
|
||||
if checkpoint_tx.try_send(snapshot).is_ok() {
|
||||
self.last_checkpoint_objects = self.checkpoint_objects;
|
||||
self.last_checkpoint_at = Instant::now();
|
||||
}
|
||||
}
|
||||
|
||||
fn carry_forward_old_children(&mut self, parent_hash: &DataUsageHash, entry: &mut DataUsageEntry) {
|
||||
if entry.compacted {
|
||||
// Compacted entries store child totals directly; child links would be flattened twice.
|
||||
@@ -1592,6 +1651,7 @@ impl FolderScanner {
|
||||
}
|
||||
}
|
||||
self.record_raw_enumeration_entry(&folder.name, &file_name);
|
||||
self.maybe_send_checkpoint();
|
||||
let is_storage_format_entry = file_name == STORAGE_FORMAT_FILE;
|
||||
|
||||
let file_path = entry.path().to_string_lossy().to_string();
|
||||
@@ -1746,6 +1806,14 @@ impl FolderScanner {
|
||||
continue;
|
||||
}
|
||||
|
||||
// Do not start another metadata read while foreground work is
|
||||
// active. The post-object timer protects the next request only
|
||||
// after the read has already been dispatched; this admission
|
||||
// point keeps the scanner from extending a single-disk I/O
|
||||
// burst across foreground requests.
|
||||
if crate::workload_admission::foreground_workload_activity() > 0 {
|
||||
self.sleeper.sleep_folder().await;
|
||||
}
|
||||
let timer = self.sleeper.timer();
|
||||
|
||||
let heal_enabled = this_hash.mod_alt(
|
||||
@@ -1887,6 +1955,8 @@ impl FolderScanner {
|
||||
into.objects += 1;
|
||||
object_count += 1;
|
||||
self.budget.record_object_scanned();
|
||||
self.checkpoint_objects = self.checkpoint_objects.saturating_add(1);
|
||||
self.maybe_send_checkpoint();
|
||||
|
||||
timer.sleep().await;
|
||||
|
||||
@@ -2180,6 +2250,7 @@ impl FolderScanner {
|
||||
|
||||
into.add_child(&h);
|
||||
self.record_completed_child(&folder_item.name, dst.failed_objects == 0);
|
||||
self.maybe_send_checkpoint();
|
||||
// We scanned a folder, optionally send update.
|
||||
self.update_cache.delete_recursive(&h);
|
||||
self.update_cache.copy_with_children(&self.new_cache, &h, &folder_item.parent);
|
||||
@@ -2578,6 +2649,7 @@ impl FolderScanner {
|
||||
self.update_cache.delete_recursive(&h);
|
||||
self.update_cache.copy_with_children(&self.new_cache, &h, &folder_item.parent);
|
||||
self.send_update().await;
|
||||
self.maybe_send_checkpoint();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -2668,7 +2740,7 @@ pub async fn scan_data_folder(
|
||||
scan_mode: HealScanMode,
|
||||
sleeper: DynamicSleeper,
|
||||
) -> Result<DataUsageCache, ScannerError> {
|
||||
scan_data_folder_scoped(ctx, budget, disks, local_disk, cache, updates, scan_mode, sleeper, None).await
|
||||
scan_data_folder_scoped(ctx, budget, disks, local_disk, cache, updates, scan_mode, sleeper, None, None).await
|
||||
}
|
||||
|
||||
#[allow(clippy::too_many_arguments)]
|
||||
@@ -2682,6 +2754,7 @@ pub(crate) async fn scan_data_folder_scoped(
|
||||
scan_mode: HealScanMode,
|
||||
sleeper: DynamicSleeper,
|
||||
prefix_scan_scope: Option<ScannerBucketPrefixScanScope>,
|
||||
checkpoint_tx: Option<mpsc::Sender<DataUsageCache>>,
|
||||
) -> Result<DataUsageCache, ScannerError> {
|
||||
use crate::data_usage_define::DATA_USAGE_ROOT;
|
||||
|
||||
@@ -2740,7 +2813,11 @@ pub(crate) async fn scan_data_folder_scoped(
|
||||
disks,
|
||||
disks_quorum,
|
||||
updates,
|
||||
checkpoint_tx,
|
||||
last_update: SystemTime::UNIX_EPOCH,
|
||||
checkpoint_objects: 0,
|
||||
last_checkpoint_objects: 0,
|
||||
last_checkpoint_at: Instant::now(),
|
||||
update_current_path,
|
||||
budget: budget.clone(),
|
||||
skip_heal,
|
||||
|
||||
@@ -339,7 +339,11 @@ async fn build_test_scanner() -> (FolderScanner, std::path::PathBuf) {
|
||||
disks: Vec::new(),
|
||||
disks_quorum: 0,
|
||||
updates: None,
|
||||
checkpoint_tx: None,
|
||||
last_update: SystemTime::UNIX_EPOCH,
|
||||
checkpoint_objects: 0,
|
||||
last_checkpoint_objects: 0,
|
||||
last_checkpoint_at: Instant::now(),
|
||||
update_current_path,
|
||||
budget: ScannerCycleBudget::new(&CancellationToken::new(), Default::default()),
|
||||
skip_heal: Arc::new(AtomicBool::new(false)),
|
||||
@@ -366,6 +370,55 @@ async fn build_test_scanner() -> (FolderScanner, std::path::PathBuf) {
|
||||
(scanner, temp_dir)
|
||||
}
|
||||
|
||||
#[tokio::test]
|
||||
async fn periodic_checkpoint_emits_at_object_threshold_without_wall_clock_wait() {
|
||||
let (mut scanner, temp_dir) = build_test_scanner().await;
|
||||
let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel(1);
|
||||
scanner.checkpoint_tx = Some(checkpoint_tx);
|
||||
scanner.checkpoint_objects = SCANNER_CHECKPOINT_OBJECT_INTERVAL;
|
||||
scanner.last_checkpoint_at = Instant::now()
|
||||
.checked_sub(SCANNER_CHECKPOINT_MIN_INTERVAL)
|
||||
.expect("test instant subtraction");
|
||||
scanner.new_cache.info.name = "bucket".to_string();
|
||||
scanner.new_cache.info.scan_progress = Some(crate::DataUsageScanProgress {
|
||||
started_plan: crate::DataUsageScanPlanDigest([1; 32]),
|
||||
requested_plan: crate::DataUsageScanPlanDigest([1; 32]),
|
||||
});
|
||||
scanner.new_cache.info.source = Some(crate::DataUsageCacheSource::new(0, 0));
|
||||
scanner.new_cache.info.scan_identity = Some(crate::DataUsageScanIdentity {
|
||||
version: 1,
|
||||
bucket_incarnation: Uuid::from_u128(1),
|
||||
set_layout: crate::DataUsageScanPlanDigest([2; 32]),
|
||||
publication_epoch: 1,
|
||||
tier_registry_generation: 0,
|
||||
scan_mode: HealScanMode::Normal,
|
||||
});
|
||||
scanner.new_cache.replace("bucket", "", DataUsageEntry::default());
|
||||
scanner.new_cache.replace("bucket/a", "bucket", DataUsageEntry::default());
|
||||
scanner.coverage_frontier = Some("bucket/a".to_string());
|
||||
scanner.new_cache.info.scan_checkpoint = Some(crate::DataUsageScanCheckpoint::new(
|
||||
"bucket/a".to_string(),
|
||||
crate::DataUsageScanCheckpointReason::Objects,
|
||||
));
|
||||
scanner.new_cache.info.scan_resume_after = Some("bucket/a".to_string());
|
||||
scanner
|
||||
.new_cache
|
||||
.seal_scan_frontier(Some("bucket/a"))
|
||||
.expect("fixture frontier");
|
||||
assert_eq!(scanner.new_cache.validated_scan_frontier(), Some("bucket/a"));
|
||||
|
||||
scanner.maybe_send_checkpoint();
|
||||
|
||||
let checkpoint = checkpoint_rx.try_recv().expect("object threshold emits a bounded checkpoint");
|
||||
assert_eq!(checkpoint.info.name, "bucket");
|
||||
assert!(!checkpoint.info.snapshot_complete);
|
||||
assert_eq!(scanner.last_checkpoint_objects, SCANNER_CHECKPOINT_OBJECT_INTERVAL);
|
||||
assert!(checkpoint_rx.try_recv().is_err(), "checkpoint queue remains bounded");
|
||||
tokio::fs::remove_dir_all(temp_dir)
|
||||
.await
|
||||
.expect("remove test scanner directory");
|
||||
}
|
||||
|
||||
struct TestGuard {
|
||||
temp_dir: Option<std::path::PathBuf>,
|
||||
}
|
||||
|
||||
@@ -33,6 +33,7 @@ fn scan_options(scan_mode: HealScanMode) -> ScannerDiskScanOptions {
|
||||
ScannerDiskScanOptions {
|
||||
scan_mode,
|
||||
prefix_scan_scope: None,
|
||||
checkpoint_tx: None,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -382,14 +383,31 @@ fn checkpoint_fixture_identity_changes_and_future_state_fail_closed() {
|
||||
assert!(next.info.scan_checkpoint.is_none());
|
||||
assert!(!next.info.snapshot_complete);
|
||||
}
|
||||
for (source, epoch) in [(crate::DataUsageCacheSource::new(1, 0), 7), (SOURCE, 8)] {
|
||||
let mut next = cache.clone();
|
||||
assert_eq!(
|
||||
next.prepare_bucket_checkpoint("bucket", 11, epoch, source, PLAN, identity),
|
||||
crate::DataUsageCachePrepareOutcome::Reset
|
||||
);
|
||||
assert!(next.cache.is_empty());
|
||||
}
|
||||
let mut source_mismatch = cache.clone();
|
||||
assert_eq!(
|
||||
source_mismatch.prepare_bucket_checkpoint("bucket", 11, 7, crate::DataUsageCacheSource::new(1, 0), PLAN, identity,),
|
||||
crate::DataUsageCachePrepareOutcome::Reset
|
||||
);
|
||||
assert!(source_mismatch.cache.is_empty());
|
||||
let mut handed_off = cache.clone();
|
||||
assert_eq!(
|
||||
handed_off.prepare_bucket_checkpoint("bucket", 11, 8, SOURCE, PLAN, identity),
|
||||
crate::DataUsageCachePrepareOutcome::Reused
|
||||
);
|
||||
assert_eq!(handed_off.info.leader_epoch, 8);
|
||||
assert_eq!(handed_off.validated_scan_frontier(), Some("bucket/static"));
|
||||
assert_eq!(retained(&handed_off), 3);
|
||||
|
||||
let mut unverified = cache.clone();
|
||||
unverified.info.scan_resume_after = None;
|
||||
unverified.info.scan_checkpoint = None;
|
||||
unverified.info.scan_coverage_receipt = None;
|
||||
assert_eq!(
|
||||
unverified.prepare_bucket_checkpoint("bucket", 11, 8, SOURCE, PLAN, identity),
|
||||
crate::DataUsageCachePrepareOutcome::Reset,
|
||||
"a cross-epoch cache without a durable frontier proof must rebuild"
|
||||
);
|
||||
assert!(unverified.cache.is_empty());
|
||||
for (cycle, epoch, expected) in [
|
||||
(10, 7, crate::DataUsageCachePrepareOutcome::RejectedNewerCycle),
|
||||
(11, 6, crate::DataUsageCachePrepareOutcome::RejectedNewerLeader),
|
||||
@@ -870,6 +888,7 @@ async fn run_checkpoint_fixture(change_digest: bool) {
|
||||
ScannerDiskScanOptions {
|
||||
scan_mode: HealScanMode::Normal,
|
||||
prefix_scan_scope: None,
|
||||
checkpoint_tx: None,
|
||||
},
|
||||
)
|
||||
.await
|
||||
@@ -951,6 +970,7 @@ async fn run_checkpoint_fixture(change_digest: bool) {
|
||||
ScannerDiskScanOptions {
|
||||
scan_mode: HealScanMode::Normal,
|
||||
prefix_scan_scope: None,
|
||||
checkpoint_tx: None,
|
||||
},
|
||||
)
|
||||
.await;
|
||||
|
||||
@@ -1293,6 +1293,7 @@ pub trait ScannerIOCache: Send + Sync + Debug + 'static {
|
||||
pub struct ScannerDiskScanOptions {
|
||||
pub scan_mode: HealScanMode,
|
||||
pub prefix_scan_scope: Option<ScannerBucketPrefixScanScope>,
|
||||
pub checkpoint_tx: Option<mpsc::Sender<DataUsageCache>>,
|
||||
}
|
||||
|
||||
#[async_trait::async_trait]
|
||||
@@ -1608,8 +1609,8 @@ use dirty_usage::*;
|
||||
use guards::*;
|
||||
|
||||
pub(crate) use cache::{
|
||||
DataUsageCacheReuseOptions, DataUsageCacheScanState, acquire_scanner_cache_locks,
|
||||
current_cache_root_or_prepare_with_generation,
|
||||
DataUsageCacheReuseOptions, DataUsageCacheScanState, ScannerCheckpointPersistContext, ScannerCheckpointPersistResult,
|
||||
acquire_scanner_cache_locks, current_cache_root_or_prepare_with_generation, persist_scanner_checkpoint,
|
||||
};
|
||||
pub use dirty_usage::{
|
||||
ScannerDirtyUsageAckError, ScannerDirtyUsageBucket, ScannerDirtyUsageClearObserver, ScannerDirtyUsageMutationObserver,
|
||||
|
||||
@@ -13,6 +13,7 @@
|
||||
// limitations under the License.
|
||||
/// scanner cache locks and the cache snapshot persist/publish path.
|
||||
use super::*;
|
||||
use crate::{ScannerConfigObjectDelete, ScannerObjectIO};
|
||||
|
||||
pub(crate) fn scanner_cache_lock_resource(cache_name: &str, source: DataUsageCacheSource) -> String {
|
||||
let lock_name = format!("{SCANNER_CACHE_LOCK_SUFFIX}.pool-{}.set-{}", source.pool_index, source.set_index);
|
||||
@@ -23,6 +24,118 @@ pub(crate) fn scanner_cache_lock_timeout() -> Duration {
|
||||
Duration::from_secs(rustfs_utils::get_env_u64("RUSTFS_LOCK_ACQUIRE_TIMEOUT", 5))
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
pub(crate) enum ScannerCheckpointPersistResult {
|
||||
Saved,
|
||||
FenceChanged,
|
||||
Failed(StorageError),
|
||||
}
|
||||
|
||||
pub(crate) struct ScannerCheckpointPersistContext<'a> {
|
||||
pub(crate) ctx: &'a CancellationToken,
|
||||
pub(crate) expected_publication_epoch: u64,
|
||||
pub(crate) cycle: u64,
|
||||
pub(crate) leader_epoch: u64,
|
||||
}
|
||||
|
||||
const CHECKPOINT_FOREGROUND_QUIET_WAIT: Duration = Duration::from_secs(1);
|
||||
|
||||
async fn wait_for_checkpoint_foreground_quiet(ctx: &CancellationToken) -> bool {
|
||||
let deadline = tokio::time::Instant::now() + CHECKPOINT_FOREGROUND_QUIET_WAIT;
|
||||
loop {
|
||||
if crate::workload_admission::foreground_workload_activity() == 0 {
|
||||
return true;
|
||||
}
|
||||
|
||||
let remaining = deadline.saturating_duration_since(tokio::time::Instant::now());
|
||||
if remaining.is_zero() {
|
||||
return false;
|
||||
}
|
||||
|
||||
let backoff = Duration::from_millis(
|
||||
crate::workload_admission::foreground_workload_activity()
|
||||
.saturating_mul(10)
|
||||
.min(250),
|
||||
)
|
||||
.max(Duration::from_millis(10))
|
||||
.min(remaining);
|
||||
tokio::select! {
|
||||
_ = ctx.cancelled() => return false,
|
||||
_ = tokio::time::sleep(backoff) => {}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/// Persist one bounded checkpoint and refresh its CAS revisions.
|
||||
///
|
||||
/// Local and remote workers share the same publication/leader fencing and
|
||||
/// revision-refresh contract; only their lock/cancellation handling remains
|
||||
/// at the caller because those guards have different concrete types.
|
||||
pub(crate) async fn persist_scanner_checkpoint<S>(
|
||||
store: Arc<S>,
|
||||
context: ScannerCheckpointPersistContext<'_>,
|
||||
cache_name: &str,
|
||||
checkpoint: &DataUsageCache,
|
||||
revisions: &mut DataUsageCacheRevisions,
|
||||
) -> ScannerCheckpointPersistResult
|
||||
where
|
||||
S: ScannerObjectIO + ScannerConfigObjectDelete,
|
||||
{
|
||||
let foreground_quiet = wait_for_checkpoint_foreground_quiet(context.ctx).await;
|
||||
if !foreground_quiet && context.ctx.is_cancelled() {
|
||||
return ScannerCheckpointPersistResult::FenceChanged;
|
||||
}
|
||||
if !foreground_quiet {
|
||||
debug!(
|
||||
target: "rustfs::scanner::io",
|
||||
event = EVENT_SCANNER_CACHE_PERSIST_STATE,
|
||||
component = LOG_COMPONENT_SCANNER,
|
||||
subsystem = LOG_SUBSYSTEM_IO,
|
||||
cache_name,
|
||||
state = "checkpoint_foreground_wait_expired",
|
||||
"Scanner checkpoint foreground quiet wait expired; preserving bounded progress"
|
||||
);
|
||||
}
|
||||
|
||||
if crate::remote_scanner::validate_remote_scanner_request_fence_with_store(context.cycle, context.leader_epoch, store.clone())
|
||||
.await
|
||||
.is_err()
|
||||
{
|
||||
return ScannerCheckpointPersistResult::FenceChanged;
|
||||
}
|
||||
if scanner_publication_admission_for_epoch(store.clone(), context.expected_publication_epoch)
|
||||
.await
|
||||
.is_none()
|
||||
{
|
||||
return ScannerCheckpointPersistResult::FenceChanged;
|
||||
}
|
||||
|
||||
if let Err(error) = checkpoint
|
||||
.save_with_revisions_for_epoch(store.clone(), cache_name, revisions, context.expected_publication_epoch)
|
||||
.await
|
||||
{
|
||||
return ScannerCheckpointPersistResult::Failed(error);
|
||||
}
|
||||
|
||||
if crate::remote_scanner::validate_remote_scanner_request_fence_with_store(context.cycle, context.leader_epoch, store.clone())
|
||||
.await
|
||||
.is_err()
|
||||
|| scanner_publication_admission_for_epoch(store.clone(), context.expected_publication_epoch)
|
||||
.await
|
||||
.is_none()
|
||||
{
|
||||
return ScannerCheckpointPersistResult::FenceChanged;
|
||||
}
|
||||
|
||||
match DataUsageCache::read_revisions(store, cache_name).await {
|
||||
Ok(next_revisions) => {
|
||||
*revisions = next_revisions;
|
||||
ScannerCheckpointPersistResult::Saved
|
||||
}
|
||||
Err(error) => ScannerCheckpointPersistResult::Failed(error),
|
||||
}
|
||||
}
|
||||
|
||||
#[derive(Debug)]
|
||||
pub(crate) struct ScannerCacheLockGuards {
|
||||
scoped: NamespaceLockGuard,
|
||||
|
||||
@@ -981,7 +981,7 @@ impl ScannerIOCache for SetDisks {
|
||||
};
|
||||
|
||||
let mut cache = DataUsageCache::default();
|
||||
let revisions = match cache.load_with_revisions(store_clone_clone.clone(), &cache_name).await {
|
||||
let mut revisions = match cache.load_with_revisions(store_clone_clone.clone(), &cache_name).await {
|
||||
Ok(revisions) => revisions,
|
||||
Err(e) => {
|
||||
record_failed_dirty_bucket(&failed_dirty_buckets_clone, &bucket.name).await;
|
||||
@@ -1145,6 +1145,7 @@ impl ScannerIOCache for SetDisks {
|
||||
);
|
||||
|
||||
let before = cache.info.last_update;
|
||||
let (checkpoint_tx, mut checkpoint_rx) = mpsc::channel::<DataUsageCache>(1);
|
||||
|
||||
let scan_ctx = ctx_clone.child_token();
|
||||
let scan = disk.clone().nsscanner_disk(
|
||||
@@ -1156,11 +1157,13 @@ impl ScannerIOCache for SetDisks {
|
||||
ScannerDiskScanOptions {
|
||||
scan_mode,
|
||||
prefix_scan_scope,
|
||||
checkpoint_tx: Some(checkpoint_tx),
|
||||
},
|
||||
);
|
||||
tokio::pin!(scan);
|
||||
let mut lock_watch = tokio::time::interval(SCANNER_CACHE_LOCK_POLL_INTERVAL);
|
||||
lock_watch.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
|
||||
let mut checkpoint_channel_closed = false;
|
||||
let scan_result = loop {
|
||||
tokio::select! {
|
||||
result = &mut scan => break result,
|
||||
@@ -1171,6 +1174,63 @@ impl ScannerIOCache for SetDisks {
|
||||
break Err(Error::other("scanner bucket cache lock was lost during bucket scan"));
|
||||
}
|
||||
}
|
||||
checkpoint = checkpoint_rx.recv(), if !checkpoint_channel_closed => {
|
||||
let Some(checkpoint) = checkpoint else {
|
||||
checkpoint_channel_closed = true;
|
||||
continue;
|
||||
};
|
||||
if cache_guard.is_lock_lost() {
|
||||
scan_ctx.cancel();
|
||||
await_scanner_disk_shutdown(scan.as_mut()).await;
|
||||
break Err(Error::other("scanner bucket cache lock was lost before checkpoint save"));
|
||||
}
|
||||
if ctx_clone.is_cancelled() {
|
||||
scan_ctx.cancel();
|
||||
await_scanner_disk_shutdown(scan.as_mut()).await;
|
||||
break Err(Error::other("scanner leader fence changed before checkpoint save"));
|
||||
}
|
||||
match persist_scanner_checkpoint(
|
||||
store_clone_clone.clone(),
|
||||
ScannerCheckpointPersistContext {
|
||||
ctx: &ctx_clone,
|
||||
expected_publication_epoch: expected_publication_epoch_clone,
|
||||
cycle: want_cycle,
|
||||
leader_epoch,
|
||||
},
|
||||
cache_name.as_str(),
|
||||
&checkpoint,
|
||||
&mut revisions,
|
||||
)
|
||||
.await
|
||||
{
|
||||
ScannerCheckpointPersistResult::Saved => {
|
||||
if cache_guard.is_lock_lost() || ctx_clone.is_cancelled() {
|
||||
scan_ctx.cancel();
|
||||
await_scanner_disk_shutdown(scan.as_mut()).await;
|
||||
break Err(Error::other("scanner bucket cache fence changed after checkpoint save"));
|
||||
}
|
||||
}
|
||||
ScannerCheckpointPersistResult::FenceChanged => {
|
||||
scan_ctx.cancel();
|
||||
await_scanner_disk_shutdown(scan.as_mut()).await;
|
||||
break Err(Error::other("scanner bucket cache fence changed during checkpoint save"));
|
||||
}
|
||||
ScannerCheckpointPersistResult::Failed(error) => {
|
||||
error!(
|
||||
target: "rustfs::scanner::io",
|
||||
event = EVENT_SCANNER_CACHE_PERSIST_STATE,
|
||||
component = LOG_COMPONENT_SCANNER,
|
||||
subsystem = LOG_SUBSYSTEM_IO,
|
||||
bucket = %bucket.name,
|
||||
cache_name = %cache_name,
|
||||
state = "periodic_checkpoint_save_failed",
|
||||
error = %error,
|
||||
"Scanner periodic checkpoint save failed"
|
||||
);
|
||||
checkpoint_channel_closed = true;
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
};
|
||||
if let Some(expected) = checkpoint_identity
|
||||
|
||||
@@ -159,6 +159,7 @@ impl ScannerIODisk for Disk {
|
||||
let ScannerDiskScanOptions {
|
||||
scan_mode,
|
||||
prefix_scan_scope,
|
||||
checkpoint_tx,
|
||||
} = options;
|
||||
let done_drive = Metrics::time(Metric::ScanBucketDrive);
|
||||
let drive_start = std::time::Instant::now();
|
||||
@@ -224,6 +225,7 @@ impl ScannerIODisk for Disk {
|
||||
scan_mode,
|
||||
SCANNER_SLEEPER.clone(),
|
||||
prefix_scan_scope,
|
||||
checkpoint_tx,
|
||||
)
|
||||
.await;
|
||||
|
||||
|
||||
@@ -29,8 +29,8 @@ pub(crate) use s3s::dto::{
|
||||
pub(crate) use s3s::dto::{ExpirationStatus as EcstoreExpirationStatus, LifecycleRule as EcstoreLifecycleRule};
|
||||
|
||||
pub(crate) use rustfs_ecstore::api::data_usage::{
|
||||
MAX_SCANNER_PAUSE_BACKLOG_BYTES, ScannerPauseBacklogRetirementPlan, ScannerPauseBacklogRetirementReplica,
|
||||
register_scanner_pause_backlog_retirement_planner,
|
||||
MAX_SCANNER_PAUSE_BACKLOG_BYTES, ScannerPauseBacklogRetirementError, ScannerPauseBacklogRetirementPlan,
|
||||
ScannerPauseBacklogRetirementReplica, register_scanner_pause_backlog_retirement_planner,
|
||||
};
|
||||
#[cfg(test)]
|
||||
pub(crate) use rustfs_ecstore::api::data_usage::{NativeScannerPauseBacklogWriteFault, SourceCleanupDeleteBarrier};
|
||||
@@ -143,8 +143,8 @@ pub(crate) type EcstoreHealResultItem = <EcstoreStore as storage_contracts::Heal
|
||||
|
||||
pub(crate) mod owner {
|
||||
pub(crate) use super::{
|
||||
MAX_SCANNER_PAUSE_BACKLOG_BYTES, ScannerPauseBacklogRetirementPlan, ScannerPauseBacklogRetirementReplica,
|
||||
register_scanner_pause_backlog_retirement_planner,
|
||||
MAX_SCANNER_PAUSE_BACKLOG_BYTES, ScannerPauseBacklogRetirementError, ScannerPauseBacklogRetirementPlan,
|
||||
ScannerPauseBacklogRetirementReplica, register_scanner_pause_backlog_retirement_planner,
|
||||
};
|
||||
#[cfg(test)]
|
||||
pub(crate) use super::{NativeScannerPauseBacklogWriteFault, SourceCleanupDeleteBarrier};
|
||||
|
||||
+45
-4
@@ -208,10 +208,10 @@ pub fn is_local_host(host: Host<&str>, port: u16, local_port: u16) -> std::io::R
|
||||
Host::Domain(domain) => {
|
||||
let ips = resolve_domain(domain)?.into_iter().collect::<Vec<_>>();
|
||||
|
||||
ips.iter().any(|ip| local_set.contains(ip))
|
||||
ips.iter().any(|ip| is_local_ip(&local_set, ip))
|
||||
}
|
||||
Host::Ipv4(ip) => local_set.contains(&IpAddr::V4(ip)),
|
||||
Host::Ipv6(ip) => local_set.contains(&IpAddr::V6(ip)),
|
||||
Host::Ipv4(ip) => is_local_ip(&local_set, &IpAddr::V4(ip)),
|
||||
Host::Ipv6(ip) => is_local_ip(&local_set, &IpAddr::V6(ip)),
|
||||
};
|
||||
|
||||
if port > 0 {
|
||||
@@ -221,6 +221,31 @@ pub fn is_local_host(host: Host<&str>, port: u16, local_port: u16) -> std::io::R
|
||||
Ok(is_local_host)
|
||||
}
|
||||
|
||||
/// Whether `ip` belongs to this host: it is either configured on a local
|
||||
/// interface or, where the platform routes the full loopback space, a loopback
|
||||
/// address.
|
||||
fn is_local_ip(local_set: &HashSet<IpAddr>, ip: &IpAddr) -> bool {
|
||||
if local_set.contains(ip) {
|
||||
return true;
|
||||
}
|
||||
is_loopback_local(ip)
|
||||
}
|
||||
|
||||
#[cfg(not(windows))]
|
||||
fn is_loopback_local(ip: &IpAddr) -> bool {
|
||||
// Unix kernels route and bind the entire RFC 1122 loopback space
|
||||
// (127.0.0.0/8 plus ::1), so any loopback address is reachable on this
|
||||
// host even when only 127.0.0.1 is configured on `lo` (e.g. 127.0.5.9).
|
||||
ip.is_loopback()
|
||||
}
|
||||
|
||||
#[cfg(windows)]
|
||||
fn is_loopback_local(_ip: &IpAddr) -> bool {
|
||||
// Windows only allows 127.0.0.1 by default, so loopback acceptance there
|
||||
// is limited to the interface-configured addresses already checked above.
|
||||
false
|
||||
}
|
||||
|
||||
fn get_custom_dns_resolver() -> Option<Arc<DynDnsResolver>> {
|
||||
match CUSTOM_DNS_RESOLVER.read() {
|
||||
Ok(guard) => guard.clone(),
|
||||
@@ -516,7 +541,16 @@ mod test {
|
||||
#[test]
|
||||
fn test_check_local_server_addr() {
|
||||
// Test valid local addresses
|
||||
let valid_cases = ["localhost:54321", "127.0.0.1:9000", "0.0.0.0:9000", "[::1]:8080", "::1:8080"];
|
||||
let valid_cases = [
|
||||
"localhost:54321",
|
||||
"127.0.0.1:9000",
|
||||
"0.0.0.0:9000",
|
||||
"[::1]:8080",
|
||||
"::1:8080",
|
||||
// Unix binds the whole 127.0.0.0/8 loopback range; Windows only allows 127.0.0.1.
|
||||
#[cfg(not(windows))]
|
||||
"127.0.5.9:9000",
|
||||
];
|
||||
|
||||
for addr in valid_cases {
|
||||
let result = check_local_server_addr(addr);
|
||||
@@ -557,6 +591,13 @@ mod test {
|
||||
let ipv4_loopback = Host::Ipv4(Ipv4Addr::new(127, 0, 0, 1));
|
||||
assert!(is_local_host(ipv4_loopback, 0, 0).unwrap());
|
||||
|
||||
// Any 127.0.0.0/8 address is loopback on Unix; Windows only allows 127.0.0.1.
|
||||
#[cfg(not(windows))]
|
||||
{
|
||||
let ipv4_loopback_octet = Host::Ipv4(Ipv4Addr::new(127, 0, 5, 9));
|
||||
assert!(is_local_host(ipv4_loopback_octet, 0, 0).unwrap());
|
||||
}
|
||||
|
||||
let ipv6_loopback = Host::Ipv6(Ipv6Addr::new(0, 0, 0, 0, 0, 0, 0, 1));
|
||||
assert!(is_local_host(ipv6_loopback, 0, 0).unwrap());
|
||||
|
||||
|
||||
@@ -39,7 +39,7 @@ allow-git = [
|
||||
# Official s3s repository. Temporarily pinned to the merged generic REST
|
||||
# SigV4 payload-checksum fix until it is available in a crates.io release.
|
||||
# owner: marshawcoco review: 2026-10
|
||||
# "https://github.com/s3s-project/s3s.git",
|
||||
"https://github.com/s3s-project/s3s.git",
|
||||
# RustFS fork carrying presigned expiry and constant-time authentication fixes.
|
||||
# owner: rustfs-maintainers review: 2026-10
|
||||
# "https://github.com/rustfs/s3s.git",
|
||||
|
||||
@@ -18,6 +18,11 @@ operators should start with:
|
||||
| [Replication object size limits](operations/replication-object-size-limits.md) | Multipart routing, large-object limits, and retry characteristics. |
|
||||
| [Replication outbound transport](operations/replication-outbound-transport.md) | Integrity headers, generic target behavior, and transport knobs. |
|
||||
|
||||
For the erasure-coded cluster lifecycle (planning, parity and `EC:0`,
|
||||
expansion, rebalance, decommission, heal, drive replacement, restart
|
||||
recovery, and the `rc` CLI mapping), start with
|
||||
[Cluster and erasure-coding lifecycle operations](operations/cluster-lifecycle-operations.md).
|
||||
|
||||
For persisted administrator bucket tasks and bucket recreation, see
|
||||
[Bucket heal recovery](operations/bucket-heal-recovery.md).
|
||||
|
||||
@@ -36,3 +41,7 @@ drive ownership, snapshot freshness, counter queries, and rolling upgrades.
|
||||
|
||||
For optional shard commitments, see [Independent shard integrity rollout](operations/shard-integrity-rollout.md):
|
||||
activation, legacy repair results, multipart mode changes, and rollback limits.
|
||||
|
||||
For crates.io publication of workspace crates, see
|
||||
[Workspace Cargo Publish](operations/cargo-publish-workspace.md): dependency
|
||||
ordering, dry-run, publish, and failure handling.
|
||||
|
||||
@@ -17,6 +17,7 @@ This document describes the algorithm as implemented on `main`. The *invariant*
|
||||
- [placement-repair-invariants.md](placement-repair-invariants.md) — owns object-to-**set** placement (which erasure set a key lands in), per-set readiness/lock quorum, scanner/heal admission, and the behavior-change gates.
|
||||
- [ecstore-layout-boundary.md](ecstore-layout-boundary.md) — owns the ecstore module ownership map, `FormatV3` set-ordering and disk-UUID-position invariants, and where the erasure engine physically lives.
|
||||
- [decommission-compatibility.md](decommission-compatibility.md) — owns moving encoded objects between pools (decommission/rebalance) and its persisted `PoolMeta` contract.
|
||||
- [../operations/cluster-lifecycle-operations.md](../operations/cluster-lifecycle-operations.md) — owns the operator runbook (planning, `EC:0` consequences, expansion, rebalance, decommission, heal, drive replacement, restart recovery); this spec stays normative for the rules it cites.
|
||||
- [../operations/tier-ilm-debugging.md](../operations/tier-ilm-debugging.md) — owns the ILM/tier runtime runbook, the dual internal-metadata-key table, the defensive binary-UUID read pattern, and the `xl.meta` inspection tooling (`dump_fileinfo` / `dump_versions`).
|
||||
- [AGENTS.md](../../AGENTS.md) "Cross-Cutting Domain Invariants" — dual internal metadata keys, defensive UUID reads, unversioned tier buckets.
|
||||
- [../../ARCHITECTURE.md](../../ARCHITECTURE.md) — crate roles (`ecstore`, `filemeta`, the `rustfs-erasure-codec` codec fork, `heal`, `scanner`).
|
||||
|
||||
@@ -41,7 +41,7 @@ Every route asserted in `rustfs/src/admin/route_registration_test.rs` is registe
|
||||
| Pools decommission / cancel / clear | 部分兼容 | `rustfs/src/admin/handlers/pools.rs` returns `NotImplemented` when endpoints are not initialized (single-pool or uninitialized clusters). |
|
||||
| `/v3/top/drives`, `/v3/top/net` | 缺失 | Only `/v3/top/locks` is registered (`rustfs/src/admin/handlers/diagnostics.rs`). |
|
||||
| Bucket / site replication per-object diff | 缺失 | `/v3/replicationmetrics` and site-replication status exist; no diff endpoint. |
|
||||
| MRF (most-recent-failures) replication metrics breakdown | 缺失 | Only the generic `/v3/metrics` stream and replication metrics wire (`rustfs/src/admin/replication_metrics_wire.rs`). |
|
||||
| MRF (most-recent-failures) replication metrics breakdown | 缺失 | Only the generic `/v3/realtime` stream and replication metrics wire (`rustfs/src/admin/replication_metrics_wire.rs`). |
|
||||
|
||||
Formerly-missing families that are now registered and therefore not exceptions: `/v3/healthinfo`, `/v3/obdinfo`, `/v3/force-unlock`, `/v3/top/locks`, `/v3/speedtest*`, `/v3/log`, `/v3/trace`, `/v3/profile`, `/v3/profiling/*`, `/v3/idp/{ldap|openid}/*`, `/v3/idp-config/*`.
|
||||
|
||||
|
||||
@@ -43,6 +43,9 @@ an unknown or unsupported peer-health snapshot degrades readiness with
|
||||
- A blocked pool metadata writer degrades node and cluster-write readiness with
|
||||
`pool_meta_write_blocked`. Metadata save-gate inspection is bounded to 100 ms;
|
||||
contention reports `pool_metadata_check_timeout` without installing a block.
|
||||
Node readiness keeps the last confirmed writable observation for the normal
|
||||
readiness-cache TTL during a transient inspection timeout, while preserving a
|
||||
confirmed block and failing closed when no fresh observation exists.
|
||||
- The authenticated cluster snapshot extends its existing node-local metadata
|
||||
gate inspection with safe reason, failure phase, and original block time. It
|
||||
distinguishes timeout from a block and changes no admission or recovery
|
||||
@@ -76,7 +79,7 @@ For a healthy IAM and metadata writer in a four-node, one-drive-per-node EC 2+2
|
||||
| 1 | false | false | true | 503 / false |
|
||||
| All restored | true | true | true | 200 / true |
|
||||
|
||||
The node path reads local disk-handle health and reuses the same reachable-host observation as its lock dependency, including the existing `RUSTFS_HEALTH_READINESS_CACHE_TTL_MS` cache. Only `Online` drives count; a reachable host with a `Returning` drive does not yet prove data I/O has recovered. It does not call cluster `storage_info`, local `disk_info`, or add disk-info RPCs. The entire storage inventory snapshot has a separate 100 ms wait budget; expiry reports `storage_readiness_check_timeout` and fails closed. Pool metadata inspection retains its own 100 ms budget. These observations are not an atomic cluster snapshot and do not bypass the existing lock-probe timing or cache policy.
|
||||
The node path reads local disk-handle health and reuses the same reachable-host observation as its lock dependency, including the existing `RUSTFS_HEALTH_READINESS_CACHE_TTL_MS` cache. Only `Online` drives count; a reachable host with a `Returning` drive does not yet prove data I/O has recovered. It does not call cluster `storage_info`, local `disk_info`, or add disk-info RPCs. The entire storage inventory snapshot has a separate 100 ms wait budget; expiry reports `storage_readiness_check_timeout` and fails closed. Pool metadata inspection retains its own 100 ms budget; a timeout is counted by `rustfs_pool_metadata_check_timeouts_total` and uses the last confirmed node-local gate state only within the same cache TTL. These observations are not an atomic cluster snapshot and do not bypass the existing lock-probe timing or cache policy.
|
||||
|
||||
The new fields are additive. Their absence in an older response is not evidence of storage quorum. Minimal responses still contain only the existing top-level fields, liveness remains dependency-independent, and HEAD responses remain bodyless.
|
||||
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user