Compare commits

...

28 Commits

Author SHA1 Message Date
cxymds cafdca1413 test(rpc): verify delete inventory through the owning disk endpoint 2026-11-20 14:44:08 +08:00
cxymds 0b37085741 feat(admin): expose guarded delete residue inspection and disposal 2026-11-20 14:42:26 +08:00
cxymds 463d62dc2a fix(ecstore): bind legacy disposal to exact physical evidence 2026-11-20 14:42:25 +08:00
cxymds bf1915234b feat(heal): page through physical delete residue by bucket 2026-11-20 14:32:18 +08:00
cxymds 004d962b0e fix(rpc): retain cancelled internode streams on release 1.0.7
(cherry picked from commit 7deb30d8c84e913730f52ff8c323b7ce1b83482a)
2026-11-20 14:29:52 +08:00
Hauser Zhi 74d8e698df fix(ecstore): back off blocked delete recovery per key
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
(cherry picked from commit 98ce924ae0)
2026-11-20 14:20:41 +08:00
cxymds 811dfdf0c5 fix(scanner): backport usage and recovery fixes to 1.0.7 (#9531)
* fix(scanner): retain the sole online drive for usage scans

* fix(admin): require measured usage before overlaying memory totals

* fix(usage): distinguish carried totals and quota reporting holds

* fix(quota): reconcile directory objects in their storage shard

* fix(tier): admit names that scanner accounting can represent

* fix(lifecycle): bound immediate scans after configuration changes

* fix(replication): stamp completed resyncs and compare target tags

* fix(replication): verify replica tags before completing a resync

* fix(scanner): stop faulted source walks before publishing usage

* style: format release scanner backports

* fix(ecstore): publish local reconnects under a shared slot fence

* fix(ecstore): keep RPC routes available while a set waits for renewal

* test(scanner): account for rejected metadata symlink paths
2026-11-20 11:43:32 +08:00
cxymds 56d87c8d0c fix(scanner): resume wide directory scans on release 1.0.7 (#9523)
* fix(scanner): resume wide directory scans on release 1.0.7

Persist physical directory progress across budget boundaries and worker or leader changes. Store large checkpoints in independently recoverable pages and expose the incomplete scan stage. Preserve the release branch disk-rebinding checks and snapshot publication fences.

* fix(storage): keep registered local disks active during startup

Distributed initialization can pass the registered local disk handle directly into Sets. Close only a distinct bootstrap handle when replacing it with the registered one, preserving the retirement token of the active disk. Cover both reused and superseded handles through real disk-map initialization.

* fix(scanner): stabilize distributed usage watch coverage

Derive mutation-watch owners from the deployment and configured node endpoint while preserving dirty snapshot, acknowledgement, and publication identities.

Keep complete producer witnesses independent of coordinator-local dirty history. Fence scanner configuration cache publication through the owning bucket's witness, including periodic refresh after missed peer notifications, while preserving unchanged refreshes and unrelated buckets.

Add shared-deployment, coordinator-history, signed RPC, configuration refresh, and multi-cycle regression coverage.
2026-11-20 09:19:36 +08:00
cxymds 3d7c0cd26a fix(scanner): merge post-release usage fixes into 1.0.7
Carry the six scanner, usage and admin snapshot backports from release/1.0.6. Keep metadata read observability inside I/O admission and preserve local drive retirement handling from #9512.
2026-11-19 14:45:14 +08:00
Chris An dada3276a3 fix(storage): backport scanner recovery and ordered shard reads (#9512)
* fix(scanner): recover interrupted local drive scans

(cherry picked from commit f18171d5cb)

* fix(scanner): bound initial local drive binding

(cherry picked from commit e56ce9d5e8)

* fix(ecstore): preserve DualInFlight stripe order in ErasureDecodeReader

In ErasureDecodeReader, poll_fill_result() previously extended
self.prefetched_bufs with lookahead stripes (queued_buffers, stripe
2k+1) before returning the primary stripe (buf, stripe 2k) to
poll_prefetch(). When a DualInFlight fill completed while output_buf
still had unconsumed bytes, poll_prefetch() pushed stripe 2k onto the
back of self.prefetched_bufs after stripe 2k+1, swapping adjacent
(2k, 2k+1) stripes on multi-block GetObject reads.

Return (buf, queued_buffers) from poll_fill_result() without mutating
self.prefetched_bufs, and extend self.prefetched_bufs with
queued_buffers in poll_prefetch() only after buf has been installed in
output_buf or pushed to self.prefetched_bufs. Also account for
non-empty self.prefetched_bufs when checking output_has_remaining and
deferring prefetch errors, and add a unit regression test.

(cherry picked from commit cd1e92921b)

* refactor(ecstore): name completed erasure stripes

(cherry picked from commit 9a3043fbb0)

* test(ecstore): keep completed stripes debuggable in tests

(cherry picked from commit f8662ec759)

---------

Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: Jinhao Hu <jinhaohu@google.com>
2026-11-19 04:47:17 +08:00
cxymds b77323f24b chore(scanner): integrate current 1.0.6 release updates 2026-11-16 09:10:36 +08:00
overtrue de9009d882 ci: use the larger macOS release runner
Backport the reviewed runner change from 3a23122104 (#9260) after preview.1 reached the job timeout.
2026-11-16 02:02:29 +08:00
overtrue 94d54f8f55 fix(clippy): backport Rust 1.100 lint fixes
Backport the applicable changes from 0f904a170e (#9377). Preserve release multipart guards and existing scanner behavior, and omit tests for features absent from the release branch.
2026-11-15 21:27:43 +08:00
overtrue 01edeb0eab chore(release): prepare 1.0.6 2026-11-15 19:40:53 +08:00
cxymds 5433d2f947 test(admin): bind snapshot fixtures to current bucket identities
(cherry picked from commit 73895b37e8)
2026-11-15 15:47:37 +08:00
cxymds f4dcda08e6 feat(scanner): observe bucket reuse and production proof gates
(cherry picked from commit 32f7f70a29)
2026-11-15 15:47:01 +08:00
cxymds b17fb942ce fix(usage): validate cached statistics against bucket identity
(cherry picked from commit 5d32827af5)
2026-11-15 15:47:01 +08:00
cxymds 0e054ec619 fix(scanner): cap superseded retries at the nominal cycle
(cherry picked from commit 04ef836aa3)
2026-11-15 15:38:59 +08:00
cxymds 89f99245ec fix(scanner): preserve baselines during backup catchup
Require decoded-primary provenance before accepting an unchanged baseline after its backup catches up. Keep current-revision CAS and publication fences, and align the real checkpoint fixture with the bounded-cycle carry contract.

(cherry picked from commit 03850c1cd6)
2026-11-15 15:38:59 +08:00
cxymds 409c8aa8af feat(scanner): expose scan work and reuse observations
(cherry picked from commit 654be96186)
2026-11-15 15:38:59 +08:00
cxymds dd2713687c fix(scanner): retire obsolete work without clearing responsibility 2026-11-14 16:14:54 +08:00
cxymds 4304e1b3ed test(scanner): preserve production drain for real checkpoints 2026-11-14 16:14:54 +08:00
overtrue 5b331fad73 fix(metrics): use iterator loops with Rust 1.100 2026-11-13 23:58:53 +08:00
overtrue 130a19de8f fix(admin): reuse S3 facade in MRF backport 2026-11-13 23:42:53 +08:00
overtrue 3e882ea0cf fix(heal): backport MRF management for 1.0.6 (#9399)
Backport PR #9399 at d57684d6e5 and adapt the scanner and admin interfaces to the 1.0.5 baseline.
2026-11-13 23:37:17 +08:00
overtrue 517ea37eb2 fix(scanner): backport usage convergence to 1.0.6 (#9397)
Backport PR #9397 at 222bdf83ee without main merge commits.
2026-11-13 23:09:12 +08:00
cxymds 6d0832b992 fix(heal): retain legacy MRF responsibility after physical repair (#9391)
(cherry picked from commit ce9d0b153e)
2026-11-13 23:08:37 +08:00
Chris An 34e6449ed9 revert(release): defer certification rollout for 1.0.5 (#9254)
Revert e5563e6004 only on release/1.0.5. Its trusted verifier workflow and immutable-candidate integration are not available yet. Restore the previous publication path under the approved release policy; retain the completed product regression and preview acceptance requirements.
2026-11-09 22:44:06 -05:00
361 changed files with 31443 additions and 8588 deletions
-77
View File
@@ -1,77 +0,0 @@
{
"schema_version": 1,
"plan_id": "rustfs-release-validation",
"description": "Versioned certification plan for every public RustFS release exit (rustfs/backlog#2662, R01). preview, prerelease and stable all require this explicit plan. Scripts/release/release_gate.py fail-closes publication on any missing, duplicate, stale, substituted or non-PASS required item.",
"release_types": ["preview", "prerelease", "stable"],
"targets": [
{"os": "linux", "arch": "x86_64", "abi": "musl"},
{"os": "linux", "arch": "aarch64", "abi": "musl"},
{"os": "linux", "arch": "x86_64", "abi": "gnu"},
{"os": "linux", "arch": "aarch64", "abi": "gnu"},
{"os": "macos", "arch": "aarch64", "abi": "none"},
{"os": "windows", "arch": "x86_64", "abi": "msvc"}
],
"targets_basis": "Platform matrix of .github/workflows/build.yml (prepare-platform-matrix). gnu and macOS targets build with the pyroscope feature; musl and Windows targets build with no extra features.",
"features": ["pyroscope"],
"trusted_workflows": [
{"repository": "rustfs/s3chaos", "workflow_id": "release-gate.yml"}
],
"trusted_workflows_basis": "The shared evidence envelope producer lives in rustfs/s3chaos (G01, fault/release_gate.rs) and its plan/live split workflow is .github/workflows/release-gate.yml there. Only results from these identities may enter a release certification; PR-uploaded JSON never gains release authority. R02 must extend this list when additional verifier workflows land, never widen it to untrusted sources.",
"lanes": [
{
"lane": "identity",
"harness_repository": "rustfs/s3chaos",
"harness_sha": "9a48f55c1d6d58ce63a8fb1ea7317e024f32f11d",
"harness_basis": "Fixed G01 evidence-envelope producer revision (rustfs/s3chaos PR #102 head recorded in rustfs/backlog#2658). R02 re-pins this when the live verifier ships.",
"timeout_seconds": 1800,
"environment_capabilities": ["process-exec"],
"cases": ["artifact-identity", "startup"],
"applies_to": {
"release_types": ["preview", "prerelease", "stable"],
"platforms": "all"
},
"applies_basis": "Artifact identity and startup checks are mandatory for every release type, preview included, on every shipped target (rustfs/backlog#2662 frozen design item 5)."
},
{
"lane": "upgrade",
"harness_repository": "rustfs/s3chaos",
"harness_sha": "9a48f55c1d6d58ce63a8fb1ea7317e024f32f11d",
"harness_basis": "Same fixed envelope producer as the identity lane; the workflow SHA and the product build SHA are bound separately and need not be equal.",
"timeout_seconds": 5400,
"environment_capabilities": ["process-exec", "previous-release-artifact"],
"cases": [
"direct-upgrade",
"mixed-version-upgrade",
"bucket-config-upgrade",
"bucket-config-rollback",
"multipart-layout-upgrade",
"multipart-layout-baseline"
],
"cases_basis": "Case IDs of the pinned previous-release matrix in .github/workflows/e2e-upgrade.yml (source.version/asset/sha256 and the case list mapped there).",
"applies_to": {
"release_types": ["prerelease", "stable"],
"platforms": [{"os": "linux", "arch": "x86_64", "abi": "gnu"}]
},
"applies_basis": "preview releases use the pre-declared smaller set (identity lane only). The upgrade lane is only exercised on linux-x86_64-gnu, the platform of the pinned source assets."
}
],
"upgrade_sources": {
"basis": "Exactly the two previous-release sources pinned in .github/workflows/e2e-upgrade.yml are exercised today (source.version / source.asset / source.sha256). No other N-1/N-2 compatibility is claimed, and these digests are not inferred from tags.",
"sources": [
{
"version": "1.0.1",
"role": "N-1",
"artifact_name": "rustfs-linux-x86_64-gnu-v1.0.1.zip",
"artifact_sha256": "9742e183cb0dfb0d48fa909797af572fe47b7f827569036baa4156e0d3958ee5"
},
{
"version": "1.0.0-rc.5",
"role": "N-2",
"artifact_name": "rustfs-linux-x86_64-gnu-v1.0.0-rc.5.zip",
"artifact_sha256": "3ee8df71e8edcfada533be452c4135868f697bc515460ae97b027313eade7a3d"
}
],
"rollback_supported": false,
"rollback_basis": "Only the bucket-config-rollback case (rollback_to_previous_release_reads_current_bucket_metadata) is exercised. Full data-plane rollback is not certified and is never a default claim."
}
}
+6 -312
View File
@@ -193,7 +193,7 @@ jobs:
{"target_id":"linux-aarch64-musl","os":"ubicloud-standard-16-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
{"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
{"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-16-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
{"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""},
{"target_id":"macos-aarch64","os":"macos-latest-xlarge","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""},
{"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":"","glibc":""}
]}'
@@ -814,18 +814,13 @@ jobs:
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
# Determine upload path based on build type. Release and prerelease
# builds only stage their bytes here: the official
# artifacts/rustfs/release/ path is written solely by the
# certification-gated promote-release-artifacts job, from these same
# staged objects. Development builds keep the uncertified dev channel
# and never touch official or latest channels.
# Determine upload path based on build type
if [[ "$BUILD_TYPE" == "development" ]]; then
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/dev/"
echo "📤 Uploading development build to R2 dev directory"
else
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}/"
echo "📤 Staging release build for certification-gated promotion"
R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/release/"
echo "📤 Uploading release build to R2 release directory"
fi
# Upload all rustfs zip files to R2 using S3-compatible API
@@ -874,221 +869,6 @@ jobs:
echo "This run does not trigger Docker publishing."
fi
# Aggregate the parent candidate batch. This job always collects results and
# retains its evidence, including on failure, and every publication job
# below requires its success. Missing, stale, substituted or non-PASS
# evidence fails closed here: only the trusted verification sources
# (rustfs/s3chaos release-gate envelopes, rustfs/backlog#2658) can satisfy
# it, so a PR-uploaded JSON set never gains release authority.
release-certification:
name: Release Certification
needs: [ build-check, build-rustfs ]
if: ${{ !cancelled() && needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') }}
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: read
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download all build artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: ./artifacts
pattern: rustfs-*
merge-multiple: true
- name: Write candidate and collect trusted validation results
env:
# Cross-repo read of the trusted verification run. Falls back to the
# workflow token, which cannot read rustfs/s3chaos; a missing token
# therefore fail-closes instead of certifying.
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
PRODUCT_COMMIT: ${{ github.sha }}
VERSION: ${{ needs.build-check.outputs.version }}
BUILD_TYPE: ${{ needs.build-check.outputs.build_type }}
FEATURES: ${{ needs.build-check.outputs.build_type == 'development' && '' || 'pyroscope' }}
shell: bash
run: |
set -euo pipefail
mkdir -p results release-gate-output
PLAN_SHA256=$(python3 -c "import hashlib,json,sys; sys.path.insert(0,'scripts/release'); import release_gate; print(release_gate.canonical_sha256(json.load(open('.config/release-validation-plan.json'))))")
python3 - "$PLAN_SHA256" > candidate.json <<'PY'
import hashlib, json, os, sys, zipfile
plan_sha256 = sys.argv[1]
profile = {"release": "stable", "prerelease": "prerelease", "preview": "preview"}.get(os.environ["BUILD_TYPE"])
if profile is None:
raise SystemExit("release certification only covers preview, prerelease and stable builds")
def sha256(path):
digest = hashlib.sha256()
with open(path, "rb") as handle:
for chunk in iter(lambda: handle.read(1024 * 1024), b""):
digest.update(chunk)
return digest.hexdigest()
deliverables = []
for name in sorted(os.listdir("artifacts")):
if not name.startswith("rustfs-") or not name.endswith(".zip"):
continue
path = os.path.join("artifacts", name)
members = zipfile.ZipFile(path).namelist()
binary_name = "rustfs.exe" if "rustfs.exe" in members else "rustfs"
with zipfile.ZipFile(path) as archive:
binary_bytes = archive.read(binary_name)
import tempfile
with tempfile.NamedTemporaryFile() as handle:
handle.write(binary_bytes)
handle.flush()
binary_sha = sha256(handle.name)
deliverables.append({
"kind": "zip",
"name": name,
"location": "https://github.com/rustfs/rustfs/releases/download/{}/{}".format(os.environ["VERSION"], name),
"sha256": sha256(path),
"size": os.path.getsize(path),
"binary_sha256": binary_sha,
})
if not deliverables:
raise SystemExit("no built deliverables found to certify")
candidate = {
"schema_version": 1,
"product_commit": os.environ["PRODUCT_COMMIT"],
"version": os.environ["VERSION"],
"release_profile": profile,
"features": [f for f in os.environ.get("FEATURES", "").split(",") if f],
"plan_sha256": plan_sha256,
"validation_batch": {"run_id": os.environ["GITHUB_RUN_ID"], "run_attempt": int(os.environ["GITHUB_RUN_ATTEMPT"])},
"deliverables": deliverables,
}
json.dump(candidate, sys.stdout, indent=2, sort_keys=True)
print()
PY
# Trusted verification evidence only: the release-gate envelopes of
# rustfs/s3chaos for this exact product commit. Absence blocks below.
TRUSTED_REPO="rustfs/s3chaos"
TRUSTED_WORKFLOW="release-gate.yml"
TRUSTED_RUN=$(gh api "repos/${TRUSTED_REPO}/actions/workflows/${TRUSTED_WORKFLOW}/runs?head_sha=${PRODUCT_COMMIT}&status=success&per_page=1" --jq '.workflow_runs[0].id // empty' || true)
if [[ -z "$TRUSTED_RUN" ]]; then
echo "❌ No trusted verification run of ${TRUSTED_REPO}/${TRUSTED_WORKFLOW} for ${PRODUCT_COMMIT}; refusing to certify" >&2
else
gh run download "$TRUSTED_RUN" -R "$TRUSTED_REPO" -n release-validation-evidence -D results || \
echo "❌ Trusted verification run ${TRUSTED_RUN} published no release-validation-evidence artifact" >&2
fi
python3 scripts/release/release_gate.py validate \
--plan .config/release-validation-plan.json \
--candidate candidate.json \
--results results \
--output release-gate-output
- name: Upload certification evidence
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: release-certification
path: |
candidate.json
results/
release-gate-output/
if-no-files-found: warn
retention-days: 30
# The only writer of the official R2 release path. Promotion re-checks the
# exact bytes about to be published against the passed certification list
# and copies the staged objects byte-for-byte: no rebuild-after-verify.
promote-release-artifacts:
name: Promote Release Artifacts
needs: [ build-check, build-rustfs, release-certification ]
# Target-channel lock for the official R2 release path.
concurrency:
group: release-channel-r2-release
cancel-in-progress: false
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
actions: read
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
AWS_EC2_METADATA_DISABLED: true
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download build and certification artifacts
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: ./artifacts
pattern: rustfs-*
merge-multiple: true
- name: Download certification evidence
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: release-certification
path: ./certification
- name: Verify and promote staged release artifacts
shell: bash
run: |
set -euo pipefail
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
echo "❌ R2 credentials missing; refusing to promote to the official channel" >&2
exit 1
fi
for zip_file in artifacts/rustfs-*.zip; do
[[ -f "$zip_file" ]] || continue
name="$(basename "$zip_file")"
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$zip_file" \
--artifact-name "$name"
done
if ! command -v aws >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
fi
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
STAGING_PATH="s3://${R2_BUCKET}/artifacts/rustfs/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}/"
RELEASE_PATH="s3://${R2_BUCKET}/artifacts/rustfs/release/"
for zip_file in artifacts/rustfs-*.zip; do
[[ -f "$zip_file" ]] || continue
name="$(basename "$zip_file")"
staged="$(mktemp)"
aws s3 cp "${STAGING_PATH}${name}" "$staged" --endpoint-url "$R2_ENDPOINT" --only-show-errors
staged_sha="$(sha256sum "$staged" | awk '{print $1}')"
built_sha="$(sha256sum "$zip_file" | awk '{print $1}')"
if [[ "$staged_sha" != "$built_sha" ]]; then
echo "❌ Staged object ${name} does not match the verified bytes" >&2
exit 1
fi
aws s3 cp "$staged" "${RELEASE_PATH}${name}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
rm -f "$staged"
echo "✅ Promoted ${name} (${built_sha})"
done
# Create GitHub Release for every valid release tag, including previews
create-release:
name: Create GitHub Release
@@ -1248,10 +1028,6 @@ jobs:
update-latest-version:
name: Update Latest Version
needs: [ build-check, publish-release ]
# Target-channel lock for the latest.json version pointer.
concurrency:
group: release-channel-oss-latest
cancel-in-progress: false
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest
timeout-minutes: 30
@@ -1292,45 +1068,6 @@ jobs:
rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
chmod +x "$OSSUTIL_BIN"
# Candidate/pointer ordering: an older candidate must never overwrite
# a newer pointer, even if its promotion job finishes later.
CURRENT_LATEST="${RUNNER_TEMP}/current-latest.json"
if "$OSSUTIL_BIN" cp oss://rustfs-version/latest.json "$CURRENT_LATEST" 2>/dev/null; then
python3 - "$CURRENT_LATEST" "$VERSION" <<'PY'
import json
import re
import sys
def key(version):
match = re.match(r"^(\d+)\.(\d+)\.(\d+)(?:-([0-9A-Za-z.-]+))?$", version)
if not match:
raise SystemExit(3)
core = tuple(int(part) for part in match.groups()[:3])
prerelease = match.group(4)
if prerelease is None:
return core + (1,)
return core + (0, tuple(int(p) if p.isdigit() else p for p in re.split(r"[.-]", prerelease)))
try:
current = json.load(open(sys.argv[1], encoding="utf-8")).get("version", "")
proposed = sys.argv[2]
if current and key(current) > key(proposed):
raise SystemExit(1)
except SystemExit:
raise
except (OSError, ValueError, TypeError):
raise SystemExit(3)
PY
guard_status=$?
if [[ "$guard_status" -eq 1 ]]; then
echo "Current latest.json points at a newer version; refusing to downgrade" >&2
exit 0
elif [[ "$guard_status" -ne 0 ]]; then
echo "Existing latest.json is unparsable; refusing to overwrite it blindly" >&2
exit 1
fi
fi
# Create latest.json
cat > latest.json << EOF
{
@@ -1347,59 +1084,16 @@ jobs:
echo "✅ Updated latest.json for ${RELEASE_TYPE} release $VERSION"
# Publish release (remove draft status). Requires the release-certification
# aggregate: without a complete legal success there is no public write.
# Publish release (remove draft status)
publish-release:
name: Publish Release
needs: [ build-check, create-release, upload-release-assets, release-certification ]
# Target-channel lock: concurrent releases serialize on this channel so an
# older candidate cannot interleave its publish with a newer one.
concurrency:
group: release-channel-github-release
cancel-in-progress: false
needs: [ build-check, create-release, upload-release-assets ]
if: needs.build-check.outputs.should_publish == 'true' && startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: write
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download certification evidence
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: release-certification
path: ./certification
- name: Verify release assets against certification
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
set -euo pipefail
TAG="${{ needs.build-check.outputs.version }}"
mkdir -p "$RUNNER_TEMP/promotion-assets"
gh release download "$TAG" --dir "$RUNNER_TEMP/promotion-assets" --pattern '*.zip'
found=0
for zip_file in "$RUNNER_TEMP"/promotion-assets/*.zip; do
[[ -f "$zip_file" ]] || continue
found=1
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$zip_file" \
--artifact-name "$(basename "$zip_file")"
done
if [[ "$found" -ne 1 ]]; then
echo "No release zip assets to verify against the certification" >&2
exit 1
fi
- name: Publish release
env:
GH_TOKEN: ${{ github.token }}
+2 -182
View File
@@ -329,75 +329,13 @@ jobs:
echo " - Create latest: $create_latest"
echo " - Source ref: $source_ref"
# Consume the trusted release certification for the exact version this
# workflow wraps and verify the certified input digests. A manual dispatch
# without that certification fails here and pushes no official tag; only
# results from the trusted verification sources carry release authority.
release-certification:
name: Release Certification
needs: build-check
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
ref: ${{ needs.build-check.outputs.source_ref }}
- name: Download and verify release certification
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
VERSION: ${{ needs.build-check.outputs.version }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification
run_id="$TRIGGERING_RUN_ID"
if [[ -z "$run_id" ]]; then
run_id="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$VERSION" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
if [[ -z "$run_id" ]]; then
echo "No certified Build and Release run found for ${VERSION}; failing closed" >&2
exit 1
fi
if ! gh run download "$run_id" -n release-certification -D certification; then
echo "Build run ${run_id} published no release-certification artifact; failing closed" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
# Input digest binding: every zip this image wraps must be the
# certified byte-for-byte content, never a mutable latest pointer.
while IFS=$'\t' read -r name location expected; do
echo "Verifying certified input ${name}"
actual="$(curl -fsSL "$location" | sha256sum | awk '{print $1}')"
if [[ "$actual" != "$expected" ]]; then
echo "Certified input ${name} does not match its recorded digest" >&2
exit 1
fi
done < <(jq -r '.deliverables[] | select(.name | contains("linux")) | [.name, .location, .sha256] | @tsv' certification/candidate.json)
# Build multi-arch Docker images
# Strategy: Build images using pre-built binaries from dl.rustfs.com
# Supports both release and dev channel binaries based on build context
# Only runs when should_build is true (which includes workflow success check)
build-docker:
name: Build Docker Images
needs: [ build-check, release-certification ]
needs: build-check
if: needs.build-check.outputs.should_build == 'true'
runs-on: ubuntu-latest
timeout-minutes: 60
@@ -505,14 +443,6 @@ jobs:
# Output tags
echo "tags=$TAGS" >> "$GITHUB_OUTPUT"
# Staging identity for this run's promotion: official tags are only
# created after the staged image verifies against the certification,
# and they point at this same digest (no rebuild-after-verify).
STAGING_TAG="staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${VARIANT_SUFFIX}"
STAGING_TAGS="${{ env.REGISTRY_DOCKERHUB }}:${STAGING_TAG},${{ env.REGISTRY_GHCR }}:${STAGING_TAG},${{ env.REGISTRY_QUAY }}:${STAGING_TAG}"
echo "staging_tags=$STAGING_TAGS" >> "$GITHUB_OUTPUT"
echo "staging_tag=$STAGING_TAG" >> "$GITHUB_OUTPUT"
# Generate labels
LABELS="org.opencontainers.image.title=RustFS"
LABELS="$LABELS,org.opencontainers.image.description=RustFS distributed object storage system"
@@ -537,9 +467,7 @@ jobs:
file: ${{ matrix.file }}
platforms: ${{ env.DOCKER_PLATFORMS }}
push: ${{ needs.build-check.outputs.should_push == 'true' }}
# Push to staging tags only: official tags are created by the
# certification-gated promotion step below from this same digest.
tags: ${{ steps.meta.outputs.staging_tags }}
tags: ${{ steps.meta.outputs.tags }}
labels: ${{ steps.meta.outputs.labels }}
# No layer cache. This build compiles nothing — it downloads a
# release zip and runs apk/apt — so the cache could only save the
@@ -565,114 +493,6 @@ jobs:
no-cache: false
pull: true
# A later-built wrapper: the staged image must complete this platform's
# own binary identity verification before any official tag is written.
# It may not inherit an unverified package's conclusion.
- name: Verify staged image against certification
if: needs.build-check.outputs.should_push == 'true'
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
VERSION: ${{ needs.build-check.outputs.version }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification
run_id="$TRIGGERING_RUN_ID"
if [[ -z "$run_id" ]]; then
run_id="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$VERSION" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
gh run download "$run_id" -n release-certification -D certification
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
case "${{ matrix.variant }}" in
musl) libc="musl" ;;
glibc) libc="gnu" ;;
esac
for platform in linux/amd64 linux/arm64; do
case "$platform" in
linux/amd64) arch_sub="x86_64" ;;
linux/arm64) arch_sub="aarch64" ;;
esac
docker pull --platform "$platform" "$STAGING_REF"
container="$(docker create --platform "$platform" "$STAGING_REF")"
docker cp "$container:/usr/bin/rustfs" "$RUNNER_TEMP/staged-rustfs"
docker rm "$container" >/dev/null
actual="$(sha256sum "$RUNNER_TEMP/staged-rustfs" | awk '{print $1}')"
expected="$(jq -r --arg needle "linux-${arch_sub}-${libc}" \
'[.deliverables[] | select(.name | contains($needle)) | .binary_sha256][0] // empty' \
certification/candidate.json)"
if [[ -z "$expected" || "$actual" != "$expected" ]]; then
echo "Staged image binary for ${platform} does not match the certified binary digest" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$RUNNER_TEMP/staged-rustfs" \
--artifact-role binary
done
- name: Promote staged image to official tags
if: needs.build-check.outputs.should_push == 'true'
shell: bash
run: |
set -euo pipefail
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
IFS=',' read -r -a official_tags <<< "${{ steps.meta.outputs.tags }}"
for tag in "${official_tags[@]}"; do
docker buildx imagetools create -t "$tag" "$STAGING_REF"
echo "Promoted $tag -> $STAGING_REF"
done
# Downstream consumers (Helm) verify this record instead of inheriting
# an unverified wrapper's conclusion.
- name: Record image promotion
if: needs.build-check.outputs.should_push == 'true'
shell: bash
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
VERSION: ${{ needs.build-check.outputs.version }}
run: |
set -euo pipefail
mkdir -p certification
run_id="$TRIGGERING_RUN_ID"
if [[ -z "$run_id" ]]; then
run_id="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$VERSION" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
gh run download "$run_id" -n release-certification -D certification
STAGING_REF="${{ env.REGISTRY_DOCKERHUB }}:staging-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}${{ matrix.suffix }}"
digest="$(docker buildx imagetools inspect "$STAGING_REF" --format '{{json .Manifest}}' | jq -r '.digest')"
candidate_sha="$(python3 -c "import json,hashlib,sys; sys.path.insert(0,'scripts/release'); import release_gate; print(release_gate.canonical_sha256(json.load(open('certification/candidate.json'))))")"
jq -n \
--arg version "$VERSION" \
--arg digest "$digest" \
--arg staging_ref "$STAGING_REF" \
--arg candidate_sha256 "$candidate_sha" \
--arg tags "${{ steps.meta.outputs.tags }}" \
--slurpfile candidate certification/candidate.json \
'{schema_version: 1, version: $version, image_digest: $digest, staging_ref: $staging_ref,
promoted_tags: ($tags | split(",")), candidate_sha256: $candidate_sha256,
binary_sha256s: ($candidate[0].deliverables | map(.binary_sha256) | unique)}' \
> image-promotion-record.json
- name: Upload image promotion record
if: needs.build-check.outputs.should_push == 'true'
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: image-promotion-record${{ matrix.suffix }}
path: image-promotion-record.json
if-no-files-found: error
retention-days: 30
# Note: Manifest creation is no longer needed as we only build one variant
# Multi-arch manifests are automatically created by docker/build-push-action
-93
View File
@@ -30,102 +30,9 @@ permissions:
contents: read
jobs:
# Consume the same trusted release certification as every other publication
# exit and verify the image promotion record of the Docker workflow. The
# chart wraps no binary of its own, so it may only reference an image whose
# platform verification already passed; it must not inherit an unverified
# package's conclusion. A run without that evidence fails closed here and
# never reaches the chart repository.
release-certification:
runs-on: ubuntu-latest
timeout-minutes: 30
permissions:
contents: read
actions: read
outputs:
version: ${{ steps.version.outputs.raw_tag }}
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Normalize release version
id: version
env:
RAW_INPUT: ${{ github.event.inputs.version }}
RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
run: |
set -euo pipefail
if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
RAW="$RAW_INPUT"
else
RAW="$RAW_BRANCH"
fi
case "$RAW" in
refs/tags/*) RAW_TAG="${RAW#refs/tags/}" ;;
*) RAW_TAG="$RAW" ;;
esac
echo "raw_tag=$RAW_TAG" >> "$GITHUB_OUTPUT"
- name: Download and verify release certification and image record
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
TRIGGERING_RUN_ID: ${{ github.event.workflow_run.id }}
RAW_TAG: ${{ steps.version.outputs.raw_tag }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification image-record
build_run="$TRIGGERING_RUN_ID"
if [[ -z "$build_run" ]]; then
build_run="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/build.yml/runs" \
-f branch="$RAW_TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
fi
if [[ -z "$build_run" ]]; then
echo "No certified Build and Release run found for ${RAW_TAG}; failing closed" >&2
exit 1
fi
if ! gh run download "$build_run" -n release-certification -D certification; then
echo "Build run ${build_run} published no release-certification artifact; failing closed" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
docker_run="$(gh api --method GET \
"repos/${GITHUB_REPOSITORY}/actions/workflows/docker.yml/runs" \
-f branch="$RAW_TAG" -f status=success -F per_page=1 \
--jq '.workflow_runs[0].id // empty' || true)"
if [[ -z "$docker_run" ]] || ! gh run download "$docker_run" -n image-promotion-record -D image-record; then
echo "No image promotion record for ${RAW_TAG}; a chart may not wrap an unverified image" >&2
exit 1
fi
candidate_sha="$(python3 -c "import json,hashlib,sys; sys.path.insert(0,'scripts/release'); import release_gate; print(release_gate.canonical_sha256(json.load(open('certification/candidate.json'))))")"
record_sha="$(jq -r '.candidate_sha256 // empty' image-record/image-promotion-record.json)"
if [[ "$record_sha" != "$candidate_sha" ]]; then
echo "Image promotion record does not belong to this certification" >&2
exit 1
fi
overlap="$(jq -r --slurpfile candidate certification/candidate.json \
'[.binary_sha256s[]] - ($candidate[0].deliverables | map(.binary_sha256)) | length' \
image-record/image-promotion-record.json)"
if [[ "$overlap" -ne 0 ]]; then
echo "Image promotion record carries binary digests outside the certification" >&2
exit 1
fi
build-helm-package:
runs-on: ubuntu-latest
timeout-minutes: 30
needs: [ release-certification ]
if: |
(github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
(
+14 -210
View File
@@ -225,57 +225,10 @@ jobs:
echo " Build run ID: $BUILD_RUN_ID"
echo " Build run number: $RUN_NUMBER"
# Consume the trusted release certification of the resolved build run and
# verify the certified input digests before any package is wrapped. A run
# without that certification fails closed here; packages built later must
# complete their own verification and cannot inherit an unverified
# conclusion.
release-certification:
name: Release Certification
needs: resolve
if: needs.resolve.result == 'success'
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: read
actions: read
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download and verify release certification
env:
GH_TOKEN: ${{ secrets.RELEASE_VALIDATION_TOKEN || github.token }}
BUILD_RUN_ID: ${{ needs.resolve.outputs.build_run_id }}
BUILD_TYPE: ${{ needs.resolve.outputs.build_type }}
shell: bash
run: |
set -euo pipefail
mkdir -p certification
if [[ "$BUILD_TYPE" == "development" ]]; then
# Development artifacts are explicitly uncertified and never touch
# official or latest channels; only the staging/dev paths remain.
jq -n '{schema_version: 1, certified: false, reason: "development artifacts are explicitly uncertified"}' \
> certification/development-uncertified.json
exit 0
fi
if ! gh run download "$BUILD_RUN_ID" -n release-certification -D certification; then
echo "Build run ${BUILD_RUN_ID} published no release-certification artifact; failing closed" >&2
exit 1
fi
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact-sha256 "$(jq -r '.deliverables[0].sha256' certification/candidate.json)" \
--artifact-name "$(jq -r '.deliverables[0].name' certification/candidate.json)"
# Build DEB and RPM packages for each architecture and libc variant
package:
name: Package (${{ matrix.arch }}-${{ matrix.libc }})
needs: [ resolve, release-certification ]
needs: resolve
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
strategy:
@@ -541,55 +494,6 @@ jobs:
echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
echo "✅ RPM built: $RPM_FILE"
- name: Download certification evidence
if: needs.resolve.outputs.build_type != 'development'
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: release-certification
path: ./certification
# Later-built wrapper verification: the binary inside each package must
# match the certified binary digest for this platform before anything is
# uploaded, and the package itself stays on staging until promotion.
- name: Verify wrapped binary against certification
if: needs.resolve.outputs.build_type != 'development'
shell: bash
env:
DEB_FILE: ${{ steps.deb.outputs.deb_file }}
RPM_FILE: ${{ steps.rpm.outputs.rpm_file }}
ARTIFACT_NAME: ${{ matrix.artifact_name }}
run: |
set -euo pipefail
expected="$(jq -r --arg needle "$ARTIFACT_NAME" \
'[.deliverables[] | select(.name | contains($needle)) | .binary_sha256][0] // empty' \
certification/candidate.json)"
if [[ -z "$expected" ]]; then
echo "No certified binary digest recorded for ${ARTIFACT_NAME}" >&2
exit 1
fi
if ! command -v rpm2cpio >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y rpm2cpio
fi
if ! command -v cpio >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y cpio
fi
for package_file in "$DEB_FILE" "$RPM_FILE"; do
case "$package_file" in
*.deb)
dpkg-deb --fsys-tarfile "$package_file" | tar -xOf - ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
*.rpm)
rpm2cpio "$package_file" | cpio -i --to-stdout ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
esac
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$RUNNER_TEMP/wrapped-rustfs" \
--artifact-role binary
done
- name: Upload packages to artifacts
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
@@ -627,14 +531,10 @@ jobs:
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
# Build-phase writes stay on staging for release types: the official
# packages/release and packages/latest paths are written only by the
# certification-gated promote-packages job, from these same staged
# objects. Development packages keep the uncertified dev channel.
if [[ "$BUILD_TYPE" == "development" ]]; then
R2_PREFIX="artifacts/rustfs/packages/dev"
else
R2_PREFIX="artifacts/rustfs/packages/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}"
R2_PREFIX="artifacts/rustfs/packages/release"
fi
R2_PATH="s3://${R2_BUCKET}/${R2_PREFIX}/"
@@ -649,6 +549,18 @@ jobs:
echo "✅ Upload complete"
# Also upload as latest for release/prerelease
if [[ "$BUILD_TYPE" == "release" || "$BUILD_TYPE" == "prerelease" ]]; then
LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "Uploading latest: $(basename "$f")"
aws s3 cp "$f" "$LATEST_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
fi
done
echo "✅ Latest packages updated"
fi
- name: Upload packages to GitHub Release
if: needs.resolve.outputs.tag != ''
env:
@@ -702,114 +614,6 @@ jobs:
echo "✅ GitHub release assets updated"
# The only writer of the official packages/release and packages/latest R2
# paths. Promotion re-checks each package's wrapped binary against the
# passed certification list and copies the staged objects byte-for-byte:
# no rebuild-after-verify, no mutable latest resolution downstream.
promote-packages:
name: Promote Packages
needs: [ resolve, package, release-certification ]
# Target-channel lock for the official package paths.
concurrency:
group: release-channel-r2-packages
cancel-in-progress: false
if: needs.resolve.outputs.tag != '' && (needs.resolve.outputs.build_type == 'release' || needs.resolve.outputs.build_type == 'prerelease')
runs-on: ubuntu-latest # gh CLI required; the sm-standard-2 fleet does not ship it
timeout-minutes: 30
permissions:
contents: read
actions: read
env:
R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
R2_BUCKET: ${{ secrets.R2_BUCKET }}
AWS_EC2_METADATA_DISABLED: true
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Download packages and certification
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
path: ./packages
pattern: packages-*
merge-multiple: true
- name: Download certification evidence
uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
with:
name: release-certification
path: ./certification
- name: Verify and promote staged packages
shell: bash
run: |
set -euo pipefail
if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
echo "❌ R2 credentials missing; refusing to promote to the official channel" >&2
exit 1
fi
STAGING_PREFIX="s3://${R2_BUCKET}/artifacts/rustfs/packages/staging/${GITHUB_RUN_ID}/${GITHUB_RUN_ATTEMPT}/"
RELEASE_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/release/"
LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
if ! command -v aws >/dev/null 2>&1; then
sudo apt-get update && sudo apt-get install -y awscli
fi
export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
export AWS_DEFAULT_REGION="auto"
if ! command -v rpm2cpio >/dev/null 2>&1; then
sudo apt-get install -y rpm2cpio
fi
if ! command -v cpio >/dev/null 2>&1; then
sudo apt-get install -y cpio
fi
promoted=0
for package_file in packages/*.deb packages/*.rpm; do
[[ -f "$package_file" ]] || continue
name="$(basename "$package_file")"
case "$package_file" in
*.deb)
dpkg-deb --fsys-tarfile "$package_file" | tar -xOf - ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
*.rpm)
rpm2cpio "$package_file" | cpio -i --to-stdout ./usr/bin/rustfs > "$RUNNER_TEMP/wrapped-rustfs"
;;
esac
python3 scripts/release/release_gate.py verify-promotion \
--plan .config/release-validation-plan.json \
--candidate certification/candidate.json \
--summary certification/release-gate-output \
--artifact "$RUNNER_TEMP/wrapped-rustfs" \
--artifact-role binary
staged="$(mktemp)"
aws s3 cp "${STAGING_PREFIX}${name}" "$staged" --endpoint-url "$R2_ENDPOINT" --only-show-errors
staged_sha="$(sha256sum "$staged" | awk '{print $1}')"
built_sha="$(sha256sum "$package_file" | awk '{print $1}')"
if [[ "$staged_sha" != "$built_sha" ]]; then
echo "❌ Staged package ${name} does not match the verified bytes" >&2
exit 1
fi
aws s3 cp "$staged" "${RELEASE_PATH}${name}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
aws s3 cp "$staged" "${LATEST_PATH}${name}" --endpoint-url "$R2_ENDPOINT" --only-show-errors
rm -f "$staged"
promoted=$((promoted + 1))
echo "✅ Promoted ${name} (${built_sha})"
done
if [[ "$promoted" -eq 0 ]]; then
echo "No staged packages found to promote" >&2
exit 1
fi
# Summary
summary:
name: Summary
Generated
+58 -52
View File
@@ -4136,7 +4136,7 @@ checksum = "d0881ea181b1df73ff77ffaaf9c7544ecc11e82fba9b5f27b262a3c73a332555"
[[package]]
name = "e2e_test"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"anyhow",
"async-compression",
@@ -9520,7 +9520,7 @@ dependencies = [
[[package]]
name = "rustfs"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"aes-gcm",
"anyhow",
@@ -9680,7 +9680,7 @@ dependencies = [
[[package]]
name = "rustfs-audit"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"const-str",
"futures",
@@ -9702,7 +9702,7 @@ dependencies = [
[[package]]
name = "rustfs-checksums"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"base64-simd",
"bytes",
@@ -9718,7 +9718,7 @@ dependencies = [
[[package]]
name = "rustfs-common"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"arc-swap",
"hotpath",
@@ -9733,7 +9733,7 @@ dependencies = [
[[package]]
name = "rustfs-concurrency"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"hotpath",
"insta",
@@ -9746,7 +9746,7 @@ dependencies = [
[[package]]
name = "rustfs-config"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"const-str",
"hotpath",
@@ -9756,7 +9756,7 @@ dependencies = [
[[package]]
name = "rustfs-connect-symbol-catalog"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"hex-simd",
"object 0.40.0",
@@ -9768,7 +9768,7 @@ dependencies = [
[[package]]
name = "rustfs-credentials"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"base64-simd",
"hmac 0.13.0",
@@ -9782,7 +9782,7 @@ dependencies = [
[[package]]
name = "rustfs-crypto"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"aes-gcm",
"argon2",
@@ -9803,17 +9803,18 @@ dependencies = [
[[package]]
name = "rustfs-data-usage"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"hotpath",
"rmp-serde",
"rustfs-filemeta",
"serde",
"serde_json",
]
[[package]]
name = "rustfs-ecstore"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"arc-swap",
"async-channel",
@@ -9951,7 +9952,7 @@ dependencies = [
[[package]]
name = "rustfs-extension-schema"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"hotpath",
"serde",
@@ -9961,7 +9962,7 @@ dependencies = [
[[package]]
name = "rustfs-filemeta"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"arc-swap",
"base64-simd",
@@ -9990,7 +9991,7 @@ dependencies = [
[[package]]
name = "rustfs-heal"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"async-trait",
"base64-simd",
@@ -10031,7 +10032,7 @@ dependencies = [
[[package]]
name = "rustfs-heal-contracts"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"serde",
"serde_json",
@@ -10041,7 +10042,7 @@ dependencies = [
[[package]]
name = "rustfs-iam"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"arc-swap",
"async-trait",
@@ -10091,7 +10092,7 @@ dependencies = [
[[package]]
name = "rustfs-io-core"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"bytes",
"hotpath",
@@ -10104,7 +10105,7 @@ dependencies = [
[[package]]
name = "rustfs-io-metrics"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"criterion",
"hotpath",
@@ -10168,7 +10169,7 @@ dependencies = [
[[package]]
name = "rustfs-keystone"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"bytes",
"futures",
@@ -10194,7 +10195,7 @@ dependencies = [
[[package]]
name = "rustfs-kms"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"aes-gcm",
"anyhow",
@@ -10244,14 +10245,14 @@ dependencies = [
[[package]]
name = "rustfs-license"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"thiserror 2.0.21",
]
[[package]]
name = "rustfs-lifecycle"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"async-trait",
"hotpath",
@@ -10273,7 +10274,7 @@ dependencies = [
[[package]]
name = "rustfs-lock"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"async-trait",
"compact_str",
@@ -10297,7 +10298,7 @@ dependencies = [
[[package]]
name = "rustfs-log-analyzer"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"chrono",
"flate2",
@@ -10316,7 +10317,7 @@ dependencies = [
[[package]]
name = "rustfs-madmin"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"hotpath",
"http 1.5.0",
@@ -10354,7 +10355,7 @@ dependencies = [
[[package]]
name = "rustfs-notify"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"arc-swap",
"async-trait",
@@ -10389,7 +10390,7 @@ dependencies = [
[[package]]
name = "rustfs-object-capacity"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"criterion",
"futures",
@@ -10408,7 +10409,7 @@ dependencies = [
[[package]]
name = "rustfs-object-data-cache"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"bytes",
"criterion",
@@ -10425,7 +10426,7 @@ dependencies = [
[[package]]
name = "rustfs-obs"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"chrono",
"crossbeam-channel",
@@ -10488,7 +10489,7 @@ dependencies = [
[[package]]
name = "rustfs-policy"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"async-trait",
"base64-simd",
@@ -10519,7 +10520,7 @@ dependencies = [
[[package]]
name = "rustfs-protocols"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"async-compression",
"async-trait",
@@ -10582,7 +10583,7 @@ dependencies = [
[[package]]
name = "rustfs-protos"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"flatbuffers",
"hotpath",
@@ -10601,13 +10602,15 @@ dependencies = [
"tonic",
"tonic-prost",
"tonic-prost-build",
"tower",
"tracing",
"tracing-subscriber",
"uuid",
]
[[package]]
name = "rustfs-replication"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"byteorder",
"bytes",
@@ -10625,7 +10628,7 @@ dependencies = [
[[package]]
name = "rustfs-rio"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"aes-gcm",
"arc-swap",
@@ -10671,7 +10674,7 @@ dependencies = [
[[package]]
name = "rustfs-rio-v2"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"aes-gcm",
"bytes",
@@ -10694,7 +10697,7 @@ dependencies = [
[[package]]
name = "rustfs-s3-client"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"base64-simd",
"bytes",
@@ -10738,7 +10741,7 @@ dependencies = [
[[package]]
name = "rustfs-s3-ops"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"hotpath",
"rustfs-s3-types",
@@ -10746,7 +10749,7 @@ dependencies = [
[[package]]
name = "rustfs-s3-types"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"hotpath",
"serde",
@@ -10755,7 +10758,7 @@ dependencies = [
[[package]]
name = "rustfs-s3select-api"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"arc-swap",
"async-compression",
@@ -10796,7 +10799,7 @@ dependencies = [
[[package]]
name = "rustfs-s3select-query"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"async-trait",
"bytes",
@@ -10818,7 +10821,7 @@ dependencies = [
[[package]]
name = "rustfs-scanner"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"async-trait",
"bytes",
@@ -10851,7 +10854,9 @@ dependencies = [
"rustfs-storage-api",
"rustfs-test-utils",
"rustfs-utils",
"rustix",
"s3s",
"same-file",
"serde",
"serde_json",
"serial_test",
@@ -10865,11 +10870,12 @@ dependencies = [
"tracing",
"tracing-subscriber",
"uuid",
"winapi-util",
]
[[package]]
name = "rustfs-scanner-metrics"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"chrono",
"jiff",
@@ -10884,7 +10890,7 @@ dependencies = [
[[package]]
name = "rustfs-security-governance"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"hotpath",
"thiserror 2.0.21",
@@ -10892,7 +10898,7 @@ dependencies = [
[[package]]
name = "rustfs-signer"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"base64-simd",
"bytes",
@@ -10911,7 +10917,7 @@ dependencies = [
[[package]]
name = "rustfs-storage-api"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"async-trait",
"hotpath",
@@ -10926,7 +10932,7 @@ dependencies = [
[[package]]
name = "rustfs-targets"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"arc-swap",
"async-nats",
@@ -10980,7 +10986,7 @@ dependencies = [
[[package]]
name = "rustfs-test-utils"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"hotpath",
"rustfs-data-usage",
@@ -10996,7 +11002,7 @@ dependencies = [
[[package]]
name = "rustfs-tls-runtime"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"arc-swap",
"hotpath",
@@ -11035,7 +11041,7 @@ dependencies = [
[[package]]
name = "rustfs-trusted-proxies"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"async-trait",
"axum",
@@ -11072,7 +11078,7 @@ dependencies = [
[[package]]
name = "rustfs-utils"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"aws-lc-rs",
"base64-simd",
@@ -11115,7 +11121,7 @@ dependencies = [
[[package]]
name = "rustfs-zip"
version = "1.0.5"
version = "1.0.6"
dependencies = [
"async-compression",
"futures",
+51 -51
View File
@@ -75,7 +75,7 @@ edition = "2024"
license = "Apache-2.0"
repository = "https://github.com/rustfs/rustfs"
rust-version = "1.99.0"
version = "1.0.5"
version = "1.0.6"
homepage = "https://rustfs.com"
description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -92,56 +92,56 @@ redundant_clone = "warn"
[workspace.dependencies]
# RustFS Internal Crates
rustfs = { path = "./rustfs", version = "1.0.5" }
rustfs-heal = { path = "crates/heal", version = "1.0.5" }
rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.5" }
rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.5" }
rustfs-audit = { path = "crates/audit", version = "1.0.5" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.5" }
rustfs-common = { path = "crates/common", version = "1.0.5" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.5" }
rustfs-config = { path = "./crates/config", version = "1.0.5" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.5" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.5" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.5" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.5" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.5" }
rustfs-iam = { path = "crates/iam", version = "1.0.5" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.5" }
rustfs-license = { path = "crates/license", version = "1.0.5" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.5" }
rustfs-kms = { path = "crates/kms", version = "1.0.5" }
rustfs-lock = { path = "crates/lock", version = "1.0.5" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.5" }
rustfs-notify = { path = "crates/notify", version = "1.0.5" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.5" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.5" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.5" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.5", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.5" }
rustfs-obs = { path = "crates/obs", version = "1.0.5" }
rustfs-policy = { path = "crates/policy", version = "1.0.5" }
rustfs-protos = { path = "crates/protos", version = "1.0.5" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.5" }
rustfs-replication = { path = "crates/replication", version = "1.0.5" }
rustfs-rio = { path = "crates/rio", version = "1.0.5" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.5" }
rustfs-s3-client = { path = "crates/s3-client", version = "1.0.5" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.5" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.5" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.5" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.5" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.5" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.5" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.5" }
rustfs-signer = { path = "crates/signer", version = "1.0.5" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.5" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.5" }
rustfs-targets = { path = "crates/targets", version = "1.0.5" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.5" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.5" }
rustfs-utils = { path = "crates/utils", version = "1.0.5" }
rustfs-zip = { path = "./crates/zip", version = "1.0.5" }
rustfs = { path = "./rustfs", version = "1.0.6" }
rustfs-heal = { path = "crates/heal", version = "1.0.6" }
rustfs-heal-contracts = { path = "crates/heal-contracts", version = "1.0.6" }
rustfs-scanner-metrics = { path = "crates/scanner-metrics", version = "1.0.6" }
rustfs-audit = { path = "crates/audit", version = "1.0.6" }
rustfs-checksums = { path = "crates/checksums", version = "1.0.6" }
rustfs-common = { path = "crates/common", version = "1.0.6" }
rustfs-data-usage = { path = "crates/data-usage", version = "1.0.6" }
rustfs-config = { path = "./crates/config", version = "1.0.6" }
rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.6" }
rustfs-credentials = { path = "crates/credentials", version = "1.0.6" }
rustfs-crypto = { path = "crates/crypto", version = "1.0.6" }
rustfs-ecstore = { path = "crates/ecstore", version = "1.0.6" }
rustfs-filemeta = { path = "crates/filemeta", version = "1.0.6" }
rustfs-iam = { path = "crates/iam", version = "1.0.6" }
rustfs-keystone = { path = "crates/keystone", version = "1.0.6" }
rustfs-license = { path = "crates/license", version = "1.0.6" }
rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.6" }
rustfs-kms = { path = "crates/kms", version = "1.0.6" }
rustfs-lock = { path = "crates/lock", version = "1.0.6" }
rustfs-madmin = { path = "crates/madmin", version = "1.0.6" }
rustfs-notify = { path = "crates/notify", version = "1.0.6" }
rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.6" }
rustfs-io-core = { path = "crates/io-core", version = "1.0.6" }
rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.6" }
rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.6", default-features = false }
rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.6" }
rustfs-obs = { path = "crates/obs", version = "1.0.6" }
rustfs-policy = { path = "crates/policy", version = "1.0.6" }
rustfs-protos = { path = "crates/protos", version = "1.0.6" }
rustfs-protocols = { path = "crates/protocols", version = "1.0.6" }
rustfs-replication = { path = "crates/replication", version = "1.0.6" }
rustfs-rio = { path = "crates/rio", version = "1.0.6" }
rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.6" }
rustfs-s3-client = { path = "crates/s3-client", version = "1.0.6" }
rustfs-s3-types = { path = "crates/s3-types", version = "1.0.6" }
rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.6" }
rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.6" }
rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.6" }
rustfs-scanner = { path = "crates/scanner", version = "1.0.6" }
rustfs-security-governance = { path = "crates/security-governance", version = "1.0.6" }
rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.6" }
rustfs-signer = { path = "crates/signer", version = "1.0.6" }
rustfs-storage-api = { path = "crates/storage-api", version = "1.0.6" }
rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.6" }
rustfs-targets = { path = "crates/targets", version = "1.0.6" }
rustfs-test-utils = { path = "crates/test-utils", version = "1.0.6" }
rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.6" }
rustfs-utils = { path = "crates/utils", version = "1.0.6" }
rustfs-zip = { path = "./crates/zip", version = "1.0.6" }
# Async Runtime and Networking
async-channel = "2.5.0"
+11
View File
@@ -121,6 +121,11 @@ when restoring `/rustfs/console` for a custom OEM build.
- `RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS` (canonical)
- `RUSTFS_SCANNER_CYCLE_MAX_OBJECTS` (canonical)
- `RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES` (canonical)
- `RUSTFS_SCANNER_BUCKET_QUANTUM_DURATION_SECS` (canonical)
- `RUSTFS_SCANNER_BUCKET_QUANTUM_OBJECTS` (canonical)
- `RUSTFS_SCANNER_BUCKET_QUANTUM_DIRECTORIES` (canonical)
- `RUSTFS_SCANNER_BUCKET_QUANTUM_ENTRIES` (canonical)
- `RUSTFS_SCANNER_BUCKET_QUANTUM_NAME_BYTES` (canonical)
- `RUSTFS_SCANNER_MAX_CONCURRENT_SET_SCANS` (canonical)
- `RUSTFS_SCANNER_MAX_CONCURRENT_DISK_SCANS` (canonical)
- `RUSTFS_SCANNER_WALKERS_PER_DISK` (canonical)
@@ -130,6 +135,12 @@ Scanner cycle budget controls:
- When `RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS` is unset, the finite default is 1800 seconds (30 minutes), matching the scanner benchmark guidance.
- An explicit `0` preserves the compatibility behavior of an unbounded runtime budget. Object and directory budgets likewise remain unbounded when explicitly set to `0`.
Scanner bucket quantum controls:
- `RUSTFS_SCANNER_BUCKET_QUANTUM_DURATION_SECS`, `RUSTFS_SCANNER_BUCKET_QUANTUM_OBJECTS`, `RUSTFS_SCANNER_BUCKET_QUANTUM_DIRECTORIES`, `RUSTFS_SCANNER_BUCKET_QUANTUM_ENTRIES`, and `RUSTFS_SCANNER_BUCKET_QUANTUM_NAME_BYTES` cap one bucket admission. Their persistent keys are `scanner.bucket_quantum_duration`, `scanner.bucket_quantum_objects`, `scanner.bucket_quantum_directories`, `scanner.bucket_quantum_entries`, and `scanner.bucket_quantum_name_bytes`; duration is in whole seconds and the remaining keys are work counts or bytes.
- All five defaults are `0` for controlled rollout. A positive value enables that dimension; `0` disables that dimension, and all five zero values disable the bucket quantum. Parent cycle limits remain in force. Invalid unsigned integers and durations outside the platform timer range are rejected; environment overrides take precedence but cannot hide invalid persisted values. Admin scanner status reports each effective value and source under `runtime_config.bucket_quantum_*`, using `bucket_quantum_duration_seconds` for duration.
- Disabled defaults are an initial rollout state, not completion of usage-convergence acceptance. Production defaults require matched-topology A/B measurements and the remote progress/fairness checks in [Scanner runtime controls](../../docs/operations/scanner-runtime-controls.md#bucket-quantum-rollout-and-acceptance). These controls do not certify partial usage or acknowledge pending dirty work.
Scanner recovery retry controls:
- `RUSTFS_SCANNER_RECOVERY_BACKOFF_CAP_SECS` caps the wait between automatic startup recovery retries. The persistent admin key is `scanner.recovery_backoff_cap`; the environment takes precedence over the persistent value, and the unset default remains 600 seconds (10 minutes). Values must be positive integer seconds within the platform timer range; `0`, malformed values, and overflow are rejected. A valid environment override does not hide an invalid persistent value from `admin config set` validation.
+50
View File
@@ -44,6 +44,21 @@ pub const SCANNER_CYCLE_MAX_OBJECTS: &str = "cycle_max_objects";
/// Scanner config key capping directories entered by one cycle.
pub const SCANNER_CYCLE_MAX_DIRECTORIES: &str = "cycle_max_directories";
/// Per-bucket admission limit in seconds; zero disables this dimension.
pub const SCANNER_BUCKET_QUANTUM_DURATION: &str = "bucket_quantum_duration";
/// Per-bucket admission limit in objects; zero disables this dimension.
pub const SCANNER_BUCKET_QUANTUM_OBJECTS: &str = "bucket_quantum_objects";
/// Per-bucket admission limit in directories; zero disables this dimension.
pub const SCANNER_BUCKET_QUANTUM_DIRECTORIES: &str = "bucket_quantum_directories";
/// Per-bucket admission limit in raw entries; zero disables this dimension.
pub const SCANNER_BUCKET_QUANTUM_ENTRIES: &str = "bucket_quantum_entries";
/// Per-bucket admission limit in raw entry-name bytes; zero disables this dimension.
pub const SCANNER_BUCKET_QUANTUM_NAME_BYTES: &str = "bucket_quantum_name_bytes";
/// Scanner config key setting the periodic bitrot scan cycle in seconds.
pub const SCANNER_BITROT_CYCLE: &str = "bitrot_cycle";
@@ -93,6 +108,11 @@ pub const SCANNER_KEYS: &[&str] = &[
SCANNER_CYCLE_MAX_DURATION,
SCANNER_CYCLE_MAX_OBJECTS,
SCANNER_CYCLE_MAX_DIRECTORIES,
SCANNER_BUCKET_QUANTUM_DURATION,
SCANNER_BUCKET_QUANTUM_OBJECTS,
SCANNER_BUCKET_QUANTUM_DIRECTORIES,
SCANNER_BUCKET_QUANTUM_ENTRIES,
SCANNER_BUCKET_QUANTUM_NAME_BYTES,
SCANNER_BITROT_CYCLE,
SCANNER_IDLE_MODE,
SCANNER_CACHE_SAVE_TIMEOUT,
@@ -141,6 +161,36 @@ pub const ENV_SCANNER_CYCLE_MAX_OBJECTS: &str = "RUSTFS_SCANNER_CYCLE_MAX_OBJECT
/// - Example: `export RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES=100000`
pub const ENV_SCANNER_CYCLE_MAX_DIRECTORIES: &str = "RUSTFS_SCANNER_CYCLE_MAX_DIRECTORIES";
/// Per-bucket admission limit in seconds; zero disables this dimension.
pub const ENV_SCANNER_BUCKET_QUANTUM_DURATION_SECS: &str = "RUSTFS_SCANNER_BUCKET_QUANTUM_DURATION_SECS";
/// Disabled until controlled-fleet acceptance selects production limits.
pub const DEFAULT_SCANNER_BUCKET_QUANTUM_DURATION_SECS: u64 = 0;
/// Per-bucket admission limit in objects; zero disables this dimension.
pub const ENV_SCANNER_BUCKET_QUANTUM_OBJECTS: &str = "RUSTFS_SCANNER_BUCKET_QUANTUM_OBJECTS";
/// Disabled until controlled-fleet acceptance selects production limits.
pub const DEFAULT_SCANNER_BUCKET_QUANTUM_OBJECTS: u64 = 0;
/// Per-bucket admission limit in directories; zero disables this dimension.
pub const ENV_SCANNER_BUCKET_QUANTUM_DIRECTORIES: &str = "RUSTFS_SCANNER_BUCKET_QUANTUM_DIRECTORIES";
/// Disabled until controlled-fleet acceptance selects production limits.
pub const DEFAULT_SCANNER_BUCKET_QUANTUM_DIRECTORIES: u64 = 0;
/// Per-bucket admission limit in raw entries; zero disables this dimension.
pub const ENV_SCANNER_BUCKET_QUANTUM_ENTRIES: &str = "RUSTFS_SCANNER_BUCKET_QUANTUM_ENTRIES";
/// Disabled until controlled-fleet acceptance selects production limits.
pub const DEFAULT_SCANNER_BUCKET_QUANTUM_ENTRIES: u64 = 0;
/// Per-bucket admission limit in raw entry-name bytes; zero disables this dimension.
pub const ENV_SCANNER_BUCKET_QUANTUM_NAME_BYTES: &str = "RUSTFS_SCANNER_BUCKET_QUANTUM_NAME_BYTES";
/// Disabled until controlled-fleet acceptance selects production limits.
pub const DEFAULT_SCANNER_BUCKET_QUANTUM_NAME_BYTES: u64 = 0;
/// Environment variable that selects the scanner speed preset.
/// Valid values: `fastest`, `fast`, `default`, `slow`, `slowest`.
/// Controls the sleep factor, maximum sleep duration, and cycle interval.
+3
View File
@@ -39,5 +39,8 @@ serde = { workspace = true, features = ["derive"] }
rmp-serde = { workspace = true }
rustfs-filemeta = { workspace = true }
[dev-dependencies]
serde_json.workspace = true
[lib]
doctest = false
+110 -12
View File
@@ -66,6 +66,12 @@ pub const LEGACY_DATA_USAGE_OBJECT_NAME: &str = ".usage.json";
/// stale tier names from growing persisted per-tier maps without bound.
pub const UNKNOWN_TIER: &str = "UNKNOWN_TIER";
/// Whether a remote tier name can be represented in scanner accounting.
/// Local storage classes are checked separately by the configuration owner.
pub fn is_scannable_tier_name(name: &str) -> bool {
!name.is_empty() && name.len() <= 256 && !name.bytes().any(|byte| byte.is_ascii_control()) && name != UNKNOWN_TIER
}
/// Returns true when `existing_last_update` is ahead of `now` by more than
/// [`USAGE_LAST_UPDATE_FUTURE_TOLERANCE`], i.e. the persisted timestamp cannot be
/// trusted for staleness comparisons and a fresh snapshot save must be allowed.
@@ -474,6 +480,12 @@ pub struct BucketTargetUsageInfo {
#[derive(Debug, Default, Clone, Serialize, Deserialize, PartialEq, Eq)]
pub struct BucketUsageInfo {
pub size: u64,
/// Provenance of this value. Missing and unknown states remain unproven.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub usage_value_state: Option<crate::BucketUsageValueState>,
/// Certification of this exact value, independent of aggregation time.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub usage_certification: Option<crate::BucketUsageCertification>,
/// Bucket identity certified by every set contributing to this snapshot.
#[serde(default, skip_serializing_if = "Option::is_none")]
pub bucket_incarnation: Option<String>,
@@ -580,6 +592,18 @@ pub struct DataUsageInfo {
/// which last-known-good generation contributed to the view.
#[serde(default, skip_serializing_if = "Vec::is_empty")]
pub usage_snapshot_set_states: Vec<DataUsageSnapshotSetState>,
/// Complete mixed observations for admin display only. Legacy bucket maps
/// and quota consumers continue to use the canonical values above.
#[serde(
default,
skip_serializing_if = "HashMap::is_empty",
deserialize_with = "crate::usage_observation::deserialize_bucket_observations"
)]
pub usage_snapshot_bucket_observations: HashMap<String, crate::DataUsageBucketObservation>,
/// Response-only marker: a projected admin view must never be persisted or
/// admitted as a quota/cache baseline, including after serialization.
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
pub usage_snapshot_admin_projection: bool,
/// Buckets whose dirty-usage generation moved while this complete
/// snapshot was being walked. Their `buckets_usage` values are the walk
/// result, which the cycle could not certify as current; the next cycle
@@ -709,7 +733,8 @@ pub fn observed_data_usage_is_newer(observed: &DataUsageInfo, authoritative: &Da
/// This applies the structural, exact-baseline, and generation checks of
/// [`observed_data_usage_is_newer`] without retaining the baseline's bucket maps.
pub fn observed_data_usage_is_newer_than_identity(observed: &DataUsageInfo, authoritative: DataUsageSnapshotIdentity) -> bool {
observed.usage_snapshot_converged == Some(false)
!observed.usage_snapshot_admin_projection
&& observed.usage_snapshot_converged == Some(false)
&& (observed.is_complete_bucket_usage_snapshot() || observed.is_valid_partial_snapshot())
&& observed.usage_snapshot_authoritative_baseline == Some(authoritative)
&& data_usage_identity_is_newer(observed.snapshot_identity(), authoritative)
@@ -1210,23 +1235,66 @@ impl Serialize for DataUsageEntry {
where
S: serde::Serializer,
{
self.serialize_fields(serializer, &self.children, &self.replication_stats, &self.all_tier_stats)
}
}
struct OrderedUsageEntry<'a>(&'a DataUsageEntry);
impl Serialize for OrderedUsageEntry<'_> {
fn serialize<S: serde::Serializer>(&self, serializer: S) -> Result<S::Ok, S::Error> {
let mut children: Vec<_> = self.0.children.iter().collect();
children.sort_unstable();
// MessagePack decoders accept struct fields as positional sequences.
// Include the nested target and tier maps in stable order as well.
let replication = self.0.replication_stats.as_ref().map(|stats| {
let ReplicationAllStats {
targets,
replica_size,
replica_count,
} = stats;
let targets: std::collections::BTreeMap<_, _> = targets.iter().collect();
(targets, replica_size, replica_count)
});
let tiers = self.0.all_tier_stats.as_ref().map(|stats| {
let AllTierStats { tiers } = stats;
let tiers: std::collections::BTreeMap<_, _> = tiers.iter().collect();
(tiers,)
});
self.0.serialize_fields(serializer, &children, &replication, &tiers)
}
}
impl DataUsageEntry {
/// Stable MessagePack encoding for independently reusable scanner checkpoint pages.
pub fn ordered_encoding(&self) -> impl Serialize + '_ {
OrderedUsageEntry(self)
}
fn serialize_fields<S: serde::Serializer, C: Serialize, R: Serialize, T: Serialize>(
&self,
serializer: S,
children: &C,
replication: &R,
tiers: &T,
) -> Result<S::Ok, S::Error> {
// Keep entries map-encoded so older readers can ignore fields appended
// by newer scanner versions during rolling upgrades. The derived
// (array) encoding made any appended field a decode error for them.
let mut state = serializer.serialize_map(Some(
11 + usize::from(self.unknown_tier_stats.is_some()) + usize::from(self.tier_accounting_proof.is_some()),
))?;
state.serialize_entry("children", &self.children)?;
state.serialize_entry("children", children)?;
state.serialize_entry("size", &self.size)?;
state.serialize_entry("objects", &self.objects)?;
state.serialize_entry("versions", &self.versions)?;
state.serialize_entry("delete_markers", &self.delete_markers)?;
state.serialize_entry("obj_sizes", &self.obj_sizes)?;
state.serialize_entry("obj_versions", &self.obj_versions)?;
state.serialize_entry("replication_stats", &self.replication_stats)?;
state.serialize_entry("replication_stats", replication)?;
state.serialize_entry("compacted", &self.compacted)?;
state.serialize_entry("failed_objects", &self.failed_objects)?;
state.serialize_entry("all_tier_stats", &self.all_tier_stats)?;
state.serialize_entry("all_tier_stats", tiers)?;
// Keep the legacy no-unknown shape byte-for-byte stable. Once unknown
// accounting exists, append its map field before the optional proof.
if let Some(unknown_tier_stats) = self.unknown_tier_stats.as_ref() {
@@ -1764,12 +1832,7 @@ impl DataUsageCache {
let mut found = HashSet::new();
found.insert(top);
mark(self, &top_e, &mut found);
self.cache.retain(|k, _| {
if !found.contains(k) {
return false;
}
true
});
self.cache.retain(|k, _| found.contains(k));
}
pub fn reduce_children_of(&mut self, path: &DataUsageHash, limit: usize, compact_self: bool) {
@@ -2029,7 +2092,8 @@ impl DataUsageInfo {
/// Whether this snapshot authoritatively covers every reported bucket.
pub fn is_complete_bucket_usage_snapshot(&self) -> bool {
!self.usage_snapshot_bootstrap_pending
!self.usage_snapshot_admin_projection
&& !self.usage_snapshot_bootstrap_pending
&& self.usage_snapshot_complete
&& self.last_update.is_some()
&& u64::try_from(self.buckets_usage.len()).ok() == Some(self.buckets_count)
@@ -2039,7 +2103,8 @@ impl DataUsageInfo {
/// admin display. Partial data is accepted only with unique set states,
/// a plan digest for every state, and at least one usable generation.
pub fn is_valid_partial_snapshot(&self) -> bool {
if self.usage_snapshot_bootstrap_pending
if self.usage_snapshot_admin_projection
|| self.usage_snapshot_bootstrap_pending
|| !self.usage_snapshot_partial
|| self.usage_snapshot_converged != Some(false)
|| self.last_update.is_none()
@@ -2077,9 +2142,11 @@ impl DataUsageInfo {
Ok(name) => name,
Err(_) => return,
};
self.usage_snapshot_bucket_observations.remove(&bucket_name);
// Update bucket statistics
if let Some(bucket_usage) = self.buckets_usage.get_mut(&bucket_name) {
bucket_usage.mark_observed();
bucket_usage.size += meta_object.size as u64;
bucket_usage.objects_count += 1;
bucket_usage.versions_count += 1; // Simplified: assume 1 version per object
@@ -2132,6 +2199,7 @@ impl DataUsageInfo {
Ok(name) => name,
Err(_) => return,
};
self.usage_snapshot_bucket_observations.remove(&bucket_name);
// Calculate accurate statistics from all versions
let mut total_size = 0u64;
@@ -2160,6 +2228,7 @@ impl DataUsageInfo {
// Update bucket statistics
if let Some(bucket_usage) = self.buckets_usage.get_mut(&bucket_name) {
bucket_usage.mark_observed();
bucket_usage.size += total_size;
bucket_usage.objects_count += 1;
bucket_usage.versions_count += versions_count;
@@ -2340,6 +2409,9 @@ impl BucketUsageInfo {
/// Add size summary to this bucket usage
/// Merge another BucketUsageInfo into this one
pub fn merge(&mut self, other: &BucketUsageInfo) {
// Adding contributions produces a new value, not either input's
// whole-bucket certificate. The scanner may certify the final sum.
self.mark_observed();
if self.bucket_incarnation != other.bucket_incarnation {
self.bucket_incarnation = None;
}
@@ -2505,6 +2577,32 @@ mod tests {
entry
}
#[test]
fn ordered_checkpoint_entry_round_trips_and_stays_identical_after_reload() {
let mut entry = DataUsageEntry::default();
let mut replication = ReplicationAllStats {
replica_size: 998,
replica_count: 3,
..Default::default()
};
let mut tiers = AllTierStats::default();
for index in 1..=32 {
let key = format!("source-{index}");
entry.children.insert(key.clone());
replication.targets.insert(key.clone(), ReplicationTargetUsage::default());
tiers.tiers.insert(key, TierStats::default());
}
entry.replication_stats = Some(replication);
entry.all_tier_stats = Some(tiers);
let bytes = rmp_serde::to_vec(&entry.ordered_encoding()).expect("encode ordered checkpoint");
let reloaded: DataUsageEntry = rmp_serde::from_slice(&bytes).expect("read with the existing codec");
assert_eq!(reloaded.children, entry.children);
let replicas = reloaded.replication_stats.as_ref().expect("replication stats");
assert_eq!((replicas.targets.len(), replicas.replica_size, replicas.replica_count), (32, 998, 3));
assert_eq!(reloaded.all_tier_stats.as_ref().expect("tier stats").tiers.len(), 32);
assert_eq!(rmp_serde::to_vec(&reloaded.ordered_encoding()).expect("encode reloaded entry"), bytes);
}
#[test]
fn tier_stats_survive_entry_merge() {
let mut left = tier_entry(
+2
View File
@@ -13,5 +13,7 @@
// limitations under the License.
pub mod data_usage;
pub mod usage_observation;
pub use data_usage::*;
pub use usage_observation::*;
+329
View File
@@ -0,0 +1,329 @@
// Copyright 2026 RustFS Team
// Licensed under the Apache License, Version 2.0.
//! Diagnostic provenance and response-only projections of complete bucket observations.
use crate::{BucketUsageInfo, DataUsageInfo, DataUsageSnapshotIdentity};
use serde::{Deserialize, Serialize};
use std::{collections::HashMap, time::SystemTime};
pub const DATA_USAGE_BUCKET_OBSERVATION_VERSION: u16 = 1;
/// Provenance of the returned bucket value, independent of quota authority.
#[derive(Clone, Copy, Debug, Deserialize, Serialize, PartialEq, Eq)]
#[serde(rename_all = "snake_case")]
pub enum BucketUsageValueState {
Certified,
Carried,
Observed,
#[serde(other)]
Unknown,
}
/// A certification of this exact bucket value, never its publication timestamp.
#[derive(Clone, Copy, Debug, Deserialize, Serialize, PartialEq, Eq)]
pub struct BucketUsageCertification {
pub epoch: u64,
pub cycle: u64,
pub certified_at: SystemTime,
}
/// Immutable provenance of one complete traversal contribution.
///
/// The scanner validates its storage-owned sweep identity before constructing
/// this public projection. Equal start/request plans are allowed when a fresh
/// set contributes alongside another set's mixed traversal.
#[derive(Clone, Debug, Deserialize, Serialize, PartialEq, Eq)]
pub struct DataUsageBucketObservationSource {
pub pool_index: u64,
pub set_index: u64,
pub bucket_incarnation: String,
pub scanner_epoch: u64,
pub scanner_cycle: u64,
pub scan_plan_digest: [u8; 32],
pub started_plan: [u8; 32],
pub requested_plan: [u8; 32],
pub completed_at: SystemTime,
}
/// An additive companion; legacy bucket maps never contain this mixed value.
#[derive(Clone, Debug, Deserialize, Serialize, PartialEq, Eq)]
pub struct DataUsageBucketObservation {
pub version: u16,
pub usage: BucketUsageInfo,
/// Exact enclosing candidate identity. Rewrapping preserves source times.
pub snapshot_identity: DataUsageSnapshotIdentity,
/// Oldest contribution time, so a newer set cannot hide an older value.
pub completed_at: SystemTime,
/// Strictly ordered, complete coverage of the enclosing live set inventory.
pub sources: Vec<DataUsageBucketObservationSource>,
}
/// Companion evolution must not make an otherwise usable canonical snapshot
/// unreadable. Unsupported or malformed diagnostic entries never grant proof.
pub(crate) fn deserialize_bucket_observations<'de, D>(
deserializer: D,
) -> Result<HashMap<String, DataUsageBucketObservation>, D::Error>
where
D: serde::Deserializer<'de>,
{
#[derive(Deserialize)]
#[serde(untagged)]
enum Entry {
Observation(Box<DataUsageBucketObservation>),
Ignored(serde::de::IgnoredAny),
}
Ok(HashMap::<String, Entry>::deserialize(deserializer)?
.into_iter()
.filter_map(|(bucket, entry)| match entry {
Entry::Observation(observation) if observation.version == DATA_USAGE_BUCKET_OBSERVATION_VERSION => {
Some((bucket, *observation))
}
Entry::Observation(_) | Entry::Ignored(_) => None,
})
.collect())
}
impl BucketUsageInfo {
/// Attach a certificate only after the caller proves complete bucket coverage.
pub fn certify(&mut self, certification: BucketUsageCertification) {
self.usage_value_state = Some(BucketUsageValueState::Certified);
self.usage_certification = Some(certification);
}
/// Preserve the certificate when carrying the exact prior bucket value.
pub fn mark_carried(&mut self) {
self.normalize_usage_provenance();
self.usage_value_state = Some(BucketUsageValueState::Carried);
}
/// The value changed without a complete bucket certificate.
pub fn mark_observed(&mut self) {
self.usage_value_state = Some(BucketUsageValueState::Observed);
self.usage_certification = None;
}
/// Future or legacy provenance cannot authorize a retained certificate.
pub fn normalize_usage_provenance(&mut self) {
if !matches!(
self.usage_value_state,
Some(BucketUsageValueState::Certified | BucketUsageValueState::Carried)
) {
self.usage_certification = None;
}
if self.usage_value_state == Some(BucketUsageValueState::Certified) && self.usage_certification.is_none() {
self.usage_value_state = Some(BucketUsageValueState::Unknown);
}
}
}
impl DataUsageBucketObservation {
/// Verify the public provenance against the enclosing candidate and current
/// storage-owned incarnation. This never grants publication or ACK authority.
pub fn matches_snapshot(&self, snapshot: &DataUsageInfo, incarnation: &str) -> bool {
let Some(root_generation) = snapshot.scanner_epoch.zip(snapshot.scanner_cycle) else {
return false;
};
if self.version != DATA_USAGE_BUCKET_OBSERVATION_VERSION
|| snapshot.usage_snapshot_admin_projection
|| self.snapshot_identity != snapshot.snapshot_identity()
|| incarnation.is_empty()
|| self.usage.bucket_incarnation.as_deref() != Some(incarnation)
|| self.sources.is_empty()
|| self.usage.usage_certification.is_some()
|| self.usage.usage_value_state != Some(BucketUsageValueState::Observed)
|| self.sources.iter().map(|source| source.completed_at).min() != Some(self.completed_at)
{
return false;
}
let mut previous = None;
let mut plan = None;
for state in &snapshot.usage_snapshot_set_states {
let key = (state.pool_index, state.set_index);
if previous.is_some_and(|previous| previous >= key)
|| state.scan_plan_digest.is_none()
|| plan.is_some_and(|plan| Some(plan) != state.scan_plan_digest)
{
return false;
}
previous = Some(key);
plan = state.scan_plan_digest;
}
let mut sources = self.sources.iter();
for state in snapshot.usage_snapshot_set_states.iter().filter(|state| !state.tombstone) {
let Some(source) = sources.next() else {
return false;
};
let Some(set_generation) = state.scanner_epoch.zip(state.scanner_cycle) else {
return false;
};
if (source.pool_index, source.set_index) != (state.pool_index, state.set_index)
|| source.bucket_incarnation != incarnation
|| Some(source.scan_plan_digest) != state.scan_plan_digest
|| (source.scanner_epoch, source.scanner_cycle) > set_generation
|| set_generation > root_generation
{
return false;
}
}
sources.next().is_none()
}
}
impl DataUsageInfo {
/// Hide values that cannot be bound to the current storage-owned identity.
///
/// `incarnations` contains canonical non-nil identities read from storage;
/// a missing entry means identity was unavailable. Neither legacy values
/// without identity nor metadata read failures establish confirmed zero.
/// Changed responses are display-only and cannot authorize quota or writes.
/// If retained counters overflow, all values remain unknown.
/// Returns the number of canonical values removed.
pub fn retain_current_bucket_usage_for_admin(&mut self, incarnations: &HashMap<String, String>) -> usize {
let original_count = self.buckets_usage.len();
let original_observation_count = self.usage_snapshot_bucket_observations.len();
let matches_current = |bucket: &String, usage: &BucketUsageInfo| {
incarnations.get(bucket).is_some_and(|current| {
!current.is_empty()
&& current != "00000000-0000-0000-0000-000000000000"
&& usage.bucket_incarnation.as_ref() == Some(current)
})
};
self.buckets_usage.retain(|bucket, usage| matches_current(bucket, usage));
self.usage_snapshot_bucket_observations.retain(|bucket, observation| {
matches_current(bucket, &observation.usage)
&& observation
.sources
.iter()
.all(|source| incarnations.get(bucket) == Some(&source.bucket_incarnation))
});
if original_count == self.buckets_usage.len()
&& original_observation_count == self.usage_snapshot_bucket_observations.len()
{
return 0;
}
let totals = self.buckets_usage.values().try_fold([0_u64; 4], |total, usage| {
Some([
total[0].checked_add(usage.size)?,
total[1].checked_add(usage.objects_count)?,
total[2].checked_add(usage.versions_count)?,
total[3].checked_add(usage.delete_markers_count)?,
])
});
let [size, objects, versions, markers] = totals.unwrap_or_else(|| {
self.buckets_usage.clear();
self.usage_snapshot_bucket_observations.clear();
[0; 4]
});
self.bucket_sizes = self
.buckets_usage
.iter()
.map(|(bucket, usage)| (bucket.clone(), usage.size))
.collect();
self.buckets_count = u64::try_from(self.buckets_usage.len()).unwrap_or(u64::MAX);
self.objects_total_size = size;
self.objects_total_count = objects;
self.versions_total_count = versions;
self.delete_markers_total_count = markers;
self.replication_info.clear();
self.tier_stats = None;
self.unknown_tier_stats = None;
self.usage_snapshot_complete = false;
self.usage_snapshot_partial = true;
self.usage_snapshot_converged = Some(false);
self.usage_snapshot_admin_projection = true;
original_count.saturating_sub(self.buckets_usage.len())
}
/// Overlay validated companions on an owned admin response only.
///
/// `incarnations` must come from current storage metadata, not the companion.
/// The projection is marked explicitly so persistence and quota boundaries
/// can reject accidental reuse. Invalid companions leave canonical values
/// unchanged. Counter overflow rejects the whole projection.
pub fn project_bucket_observations_for_admin(&mut self, incarnations: &HashMap<String, String>) -> usize {
if self.usage_snapshot_admin_projection
|| (!self.is_complete_bucket_usage_snapshot() && !self.is_valid_partial_snapshot())
{
return 0;
}
let observations =
self.usage_snapshot_bucket_observations
.iter()
.filter(|(bucket, observation)| {
if self.usage_snapshot_complete
&& (!self.usage_snapshot_pending_buckets.iter().any(|pending| pending == *bucket)
|| u64::try_from(self.usage_snapshot_pending_buckets.len()).ok()
!= Some(self.usage_snapshot_pending_bucket_count))
{
return false;
}
if self
.buckets_usage
.get(*bucket)
.and_then(|usage| usage.usage_certification)
.is_some_and(|certificate| {
observation.sources.iter().all(|source| {
(source.scanner_epoch, source.scanner_cycle) <= (certificate.epoch, certificate.cycle)
})
})
{
return false;
}
incarnations.get(*bucket).is_some_and(|incarnation| {
self.buckets_usage
.get(*bucket)
.and_then(|usage| usage.bucket_incarnation.as_ref())
.is_none_or(|canonical| canonical == incarnation)
&& observation.matches_snapshot(self, incarnation)
})
})
.collect::<HashMap<_, _>>();
if observations.is_empty() {
return 0;
}
let mut values = self
.buckets_usage
.iter()
.filter(|(bucket, _)| !observations.contains_key(bucket))
.map(|(_, usage)| usage)
.chain(observations.values().map(|observation| &observation.usage));
let totals = values.try_fold([0_u64; 4], |total, usage| {
Some([
total[0].checked_add(usage.size)?,
total[1].checked_add(usage.objects_count)?,
total[2].checked_add(usage.versions_count)?,
total[3].checked_add(usage.delete_markers_count)?,
])
});
let Some([size, objects, versions, markers]) = totals else {
return 0;
};
let projected = observations.len();
for (bucket, observation) in observations {
self.buckets_usage.insert(bucket.clone(), observation.usage.clone());
}
self.bucket_sizes = self
.buckets_usage
.iter()
.map(|(bucket, usage)| (bucket.clone(), usage.size))
.collect();
self.buckets_count = u64::try_from(self.buckets_usage.len()).unwrap_or(u64::MAX);
self.objects_total_size = size;
self.objects_total_count = objects;
self.versions_total_count = versions;
self.delete_markers_total_count = markers;
self.replication_info.clear();
self.tier_stats = None;
self.unknown_tier_stats = None;
self.usage_snapshot_complete = false;
self.usage_snapshot_partial = true;
self.usage_snapshot_converged = Some(false);
self.usage_snapshot_observational = true;
self.usage_snapshot_admin_projection = true;
projected
}
}
#[cfg(test)]
mod tests;
@@ -0,0 +1,430 @@
// Copyright 2026 RustFS Team
// Licensed under the Apache License, Version 2.0.
use super::*;
use crate::DataUsageSnapshotSetState;
use std::time::Duration;
const INCARNATION: &str = "11111111-1111-4111-8111-111111111111";
fn time(seconds: u64) -> SystemTime {
SystemTime::UNIX_EPOCH + Duration::from_secs(seconds)
}
fn usage(size: u64) -> BucketUsageInfo {
BucketUsageInfo {
size,
objects_count: size,
versions_count: size,
bucket_incarnation: Some(INCARNATION.to_string()),
..Default::default()
}
}
fn snapshot() -> DataUsageInfo {
let mut main = usage(10);
main.certify(BucketUsageCertification {
epoch: 2,
cycle: 3,
certified_at: time(30),
});
main.mark_carried();
let mut snapshot = DataUsageInfo {
last_update: Some(time(90)),
scanner_epoch: Some(2),
scanner_cycle: Some(9),
buckets_count: 1,
buckets_usage: HashMap::from([("bucket".to_string(), main)]),
bucket_sizes: HashMap::from([("bucket".to_string(), 10)]),
usage_snapshot_complete: true,
usage_snapshot_converged: Some(false),
usage_snapshot_pending_buckets: vec!["bucket".to_string()],
usage_snapshot_pending_bucket_count: 1,
usage_snapshot_set_states: (0..2)
.map(|set_index| DataUsageSnapshotSetState {
pool_index: 0,
set_index,
scanner_epoch: Some(2),
scanner_cycle: Some(9),
scan_plan_digest: Some([1; 32]),
complete: true,
..Default::default()
})
.collect(),
..Default::default()
};
snapshot.calculate_totals();
let mut observed = usage(25);
observed.mark_observed();
let observation = DataUsageBucketObservation {
version: DATA_USAGE_BUCKET_OBSERVATION_VERSION,
usage: observed,
snapshot_identity: snapshot.snapshot_identity(),
completed_at: time(40),
sources: (0..2)
.map(|set_index| DataUsageBucketObservationSource {
pool_index: 0,
set_index,
bucket_incarnation: INCARNATION.to_string(),
scanner_epoch: 2,
scanner_cycle: 4 + set_index,
scan_plan_digest: [1; 32],
started_plan: [2; 32],
requested_plan: [3; 32],
completed_at: time(40 + set_index),
})
.collect(),
};
snapshot
.usage_snapshot_bucket_observations
.insert("bucket".to_string(), observation);
snapshot
}
fn incarnations() -> HashMap<String, String> {
HashMap::from([("bucket".to_string(), INCARNATION.to_string())])
}
#[test]
fn provenance_is_additive_and_unknown_states_do_not_keep_certificates() {
let legacy = usage(10);
let bytes = serde_json::to_vec(&legacy).expect("encode a legacy bucket without provenance");
let text = std::str::from_utf8(&bytes).expect("JSON is UTF-8");
assert!(!text.contains("usage_value_state"));
assert!(!text.contains("usage_certification"));
let decoded: BucketUsageInfo = serde_json::from_slice(&bytes).expect("read legacy bucket");
assert_eq!(decoded.usage_value_state, None);
assert_eq!(decoded.usage_certification, None);
let mut wire = serde_json::to_value(legacy).expect("encode future bucket fixture");
wire["usage_value_state"] = serde_json::json!("future_state");
wire["usage_certification"] =
serde_json::json!({"epoch":2,"cycle":3,"certified_at":{"secs_since_epoch":30,"nanos_since_epoch":0}});
let mut future: BucketUsageInfo = serde_json::from_value(wire).expect("unknown diagnostic enum remains readable");
future.normalize_usage_provenance();
assert_eq!(future.usage_value_state, Some(BucketUsageValueState::Unknown));
assert_eq!(future.usage_certification, None);
}
#[test]
fn carrying_preserves_time_but_adding_a_source_invalidates_the_certificate() {
let mut bucket = snapshot().buckets_usage.remove("bucket").expect("canonical bucket");
bucket.mark_carried();
assert_eq!(bucket.usage_certification.map(|certification| certification.certified_at), Some(time(30)));
bucket.merge(&usage(1));
assert_eq!(bucket.size, 11);
assert_eq!(bucket.usage_value_state, Some(BucketUsageValueState::Observed));
assert_eq!(bucket.usage_certification, None);
}
#[test]
fn direct_object_accumulation_does_not_reuse_the_old_certificate() {
let mut simple = snapshot();
simple.add_object("bucket/object", &rustfs_filemeta::MetaObject::default());
assert_eq!(simple.buckets_usage["bucket"].usage_value_state, Some(BucketUsageValueState::Observed));
assert_eq!(simple.buckets_usage["bucket"].usage_certification, None);
assert!(simple.usage_snapshot_bucket_observations.is_empty());
let mut versioned = snapshot();
versioned.add_object_from_file_meta("bucket/object", &rustfs_filemeta::FileMeta::default());
assert_eq!(versioned.buckets_usage["bucket"].usage_value_state, Some(BucketUsageValueState::Observed));
assert_eq!(versioned.buckets_usage["bucket"].usage_certification, None);
assert!(versioned.usage_snapshot_bucket_observations.is_empty());
}
#[test]
fn future_or_malformed_companions_do_not_break_the_canonical_reader() {
let canonical = snapshot();
for entry in [
serde_json::json!({"version": 2, "future_payload": {"new": true}}),
serde_json::json!({"version": 1, "usage": "broken"}),
] {
let mut wire = serde_json::to_value(&canonical).expect("canonical JSON");
wire["usage_snapshot_bucket_observations"]["bucket"] = entry;
let decoded: DataUsageInfo = serde_json::from_value(wire).expect("diagnostics cannot break canonical data");
assert_eq!(decoded.buckets_usage, canonical.buckets_usage);
assert!(decoded.usage_snapshot_bucket_observations.is_empty());
assert!(decoded.is_complete_bucket_usage_snapshot());
}
}
#[test]
fn complete_companion_projects_only_the_admin_copy_and_keeps_measurement_times() {
let canonical = snapshot();
let canonical_bytes = serde_json::to_vec(&canonical).expect("encode canonical candidate");
let mut response = canonical.clone();
assert_eq!(response.project_bucket_observations_for_admin(&incarnations()), 1);
assert_eq!(response.buckets_usage["bucket"].size, 25);
assert_eq!(response.buckets_usage["bucket"].usage_value_state, Some(BucketUsageValueState::Observed));
assert_eq!(response.buckets_usage["bucket"].usage_certification, None);
assert_eq!(response.objects_total_size, 25);
assert_eq!(response.bucket_sizes["bucket"], 25);
assert_eq!(response.last_update, canonical.last_update);
assert_eq!(response.usage_snapshot_bucket_observations["bucket"].completed_at, time(40));
assert!(response.usage_snapshot_admin_projection);
assert!(response.usage_snapshot_partial);
assert!(response.usage_snapshot_observational);
assert_eq!(response.usage_snapshot_converged, Some(false));
assert!(!response.is_complete_bucket_usage_snapshot());
assert!(
!response.is_valid_partial_snapshot(),
"response-only values cannot be a storage candidate"
);
assert_eq!(serde_json::to_vec(&canonical).expect("reencode original"), canonical_bytes);
assert_eq!(canonical.buckets_usage["bucket"].size, 10);
assert_eq!(
canonical.buckets_usage["bucket"]
.usage_certification
.map(|certificate| certificate.certified_at),
Some(time(30))
);
assert_eq!(response.project_bucket_observations_for_admin(&incarnations()), 0);
}
#[test]
fn invalid_companions_preserve_the_canonical_bucket() {
for invalid in 0..11 {
let mut candidate = snapshot();
let observation = candidate
.usage_snapshot_bucket_observations
.get_mut("bucket")
.expect("companion");
match invalid {
0 => observation.version += 1,
1 => observation.snapshot_identity.scanner_cycle = Some(8),
2 => {
observation.sources.pop();
}
3 => observation.sources[1].set_index = 0,
4 => observation.sources[1].scanner_cycle = 10,
5 => observation.sources[1].scan_plan_digest = [9; 32],
6 => observation.sources[1].bucket_incarnation = "other".to_string(),
7 => observation.completed_at = time(41),
8 => observation.usage.certify(BucketUsageCertification {
epoch: 2,
cycle: 9,
certified_at: time(90),
}),
9 => candidate.usage_snapshot_set_states[1].scanner_cycle = None,
10 => {
candidate
.buckets_usage
.get_mut("bucket")
.expect("canonical")
.bucket_incarnation = Some("recreated".to_string())
}
_ => unreachable!("bounded fixture variants"),
}
let before = candidate.clone();
assert_eq!(
candidate.project_bucket_observations_for_admin(&incarnations()),
0,
"invalid case {invalid}"
);
assert_eq!(candidate, before, "invalid case {invalid} changed canonical data");
}
}
#[test]
fn a_fresh_set_can_contribute_without_refreshing_an_older_mixed_set() {
let mut candidate = snapshot();
let observation = candidate
.usage_snapshot_bucket_observations
.get_mut("bucket")
.expect("companion");
observation.sources[1].started_plan = observation.sources[1].requested_plan;
observation.sources[1].completed_at = time(90);
assert_eq!(candidate.project_bucket_observations_for_admin(&incarnations()), 1);
assert_eq!(candidate.usage_snapshot_bucket_observations["bucket"].completed_at, time(40));
}
#[test]
fn companion_only_partial_keeps_legacy_maps_empty_until_admin_projection() {
let mut candidate = snapshot();
candidate.buckets_usage.clear();
candidate.bucket_sizes.clear();
candidate.buckets_count = 0;
candidate.calculate_totals();
candidate.usage_snapshot_complete = false;
candidate.usage_snapshot_partial = true;
candidate.usage_snapshot_observational = true;
for state in &mut candidate.usage_snapshot_set_states {
state.complete = false;
}
assert!(candidate.is_valid_partial_snapshot());
let wire = serde_json::to_value(&candidate).expect("encode companion-only observation");
assert_eq!(wire["buckets_usage"], serde_json::json!({}));
assert_eq!(wire["bucket_sizes"], serde_json::json!({}));
assert_eq!(wire["objects_total_size"], 0);
assert_eq!(candidate.project_bucket_observations_for_admin(&incarnations()), 1);
assert_eq!(candidate.objects_total_size, 25);
assert!(candidate.usage_snapshot_partial);
assert!(!candidate.usage_snapshot_complete);
}
#[test]
fn projection_rejects_counter_overflow_without_partial_mutation() {
let mut candidate = snapshot();
candidate.buckets_usage.insert("cold".to_string(), usage(u64::MAX));
candidate.buckets_count = 2;
let before = candidate.clone();
assert_eq!(candidate.project_bucket_observations_for_admin(&incarnations()), 0);
assert_eq!(candidate, before);
}
#[test]
fn a_newer_canonical_certificate_and_missing_pending_prevent_old_overlay() {
for newer_certificate in [false, true] {
let mut candidate = snapshot();
if newer_certificate {
candidate
.buckets_usage
.get_mut("bucket")
.expect("canonical")
.certify(BucketUsageCertification {
epoch: 2,
cycle: 9,
certified_at: time(90),
});
} else {
candidate.usage_snapshot_pending_buckets.clear();
candidate.usage_snapshot_pending_bucket_count = 0;
}
assert_eq!(candidate.project_bucket_observations_for_admin(&incarnations()), 0);
assert_eq!(candidate.buckets_usage["bucket"].size, 10);
}
}
#[test]
fn provenance_and_companions_change_candidate_bytes_and_round_trip_as_maps() {
let canonical = snapshot();
let original = serde_json::to_vec(&canonical).expect("encode candidate");
let mut changed = canonical.clone();
changed
.buckets_usage
.get_mut("bucket")
.expect("bucket")
.usage_certification
.as_mut()
.expect("certificate")
.certified_at = time(31);
assert_ne!(serde_json::to_vec(&changed).expect("encode changed certificate"), original);
let mut changed = canonical.clone();
changed
.usage_snapshot_bucket_observations
.get_mut("bucket")
.expect("observation")
.sources[0]
.started_plan = [8; 32];
assert_ne!(serde_json::to_vec(&changed).expect("encode changed source"), original);
let named = rmp_serde::to_vec_named(&canonical).expect("encode named map");
let restored: DataUsageInfo = rmp_serde::from_slice(&named).expect("read named map with additive fields");
assert_eq!(restored, canonical);
}
#[test]
fn current_identity_filter_preserves_an_unchanged_complete_snapshot() {
let mut candidate = snapshot();
let original = candidate.clone();
assert_eq!(candidate.retain_current_bucket_usage_for_admin(&incarnations()), 0);
assert_eq!(candidate, original);
assert!(candidate.is_complete_bucket_usage_snapshot());
}
#[test]
fn current_identity_filter_hides_unbound_values_without_confirming_zero() {
for identity in [
None,
Some(""),
Some("corrupt"),
Some("00000000-0000-0000-0000-000000000000"),
Some("22222222-2222-4222-8222-222222222222"),
] {
let mut candidate = snapshot();
candidate.usage_snapshot_bucket_observations.clear();
candidate
.buckets_usage
.get_mut("bucket")
.expect("canonical")
.bucket_incarnation = identity.map(str::to_string);
assert_eq!(
candidate.retain_current_bucket_usage_for_admin(&incarnations()),
1,
"identity {identity:?}"
);
assert!(!candidate.buckets_usage.contains_key("bucket"));
assert!(!candidate.bucket_sizes.contains_key("bucket"));
assert_eq!(candidate.usage_snapshot_converged, Some(false));
assert!(candidate.usage_snapshot_admin_projection);
assert!(!candidate.is_complete_bucket_usage_snapshot());
assert!(!candidate.is_valid_partial_snapshot());
}
for current in ["", "00000000-0000-0000-0000-000000000000"] {
let mut candidate = snapshot();
candidate
.buckets_usage
.get_mut("bucket")
.expect("canonical")
.bucket_incarnation = Some(current.to_string());
assert_eq!(
candidate.retain_current_bucket_usage_for_admin(&HashMap::from([("bucket".to_string(), current.to_string())])),
1
);
assert!(candidate.usage_snapshot_admin_projection);
}
let mut candidate = snapshot();
assert_eq!(candidate.retain_current_bucket_usage_for_admin(&HashMap::new()), 1);
assert!(candidate.usage_snapshot_bucket_observations.is_empty());
assert_eq!(candidate.usage_snapshot_converged, Some(false));
}
#[test]
fn current_identity_filter_keeps_healthy_buckets_and_recalculates_partial_totals() {
let mut candidate = snapshot();
candidate.buckets_usage.insert("cold".to_string(), usage(7));
candidate.buckets_count = 2;
candidate.bucket_sizes.insert("cold".to_string(), 7);
candidate.calculate_totals();
candidate.replication_info.insert("target".to_string(), Default::default());
candidate.tier_stats = Some(Default::default());
candidate.unknown_tier_stats = Some(Default::default());
assert_eq!(
candidate.retain_current_bucket_usage_for_admin(&HashMap::from([("cold".to_string(), INCARNATION.to_string())])),
1
);
assert_eq!(candidate.buckets_usage.len(), 1);
assert_eq!(candidate.buckets_usage["cold"].size, 7);
assert_eq!(candidate.bucket_sizes, HashMap::from([("cold".to_string(), 7)]));
assert_eq!(
(
candidate.buckets_count,
candidate.objects_total_size,
candidate.objects_total_count,
candidate.versions_total_count
),
(1, 7, 7, 7)
);
assert!(candidate.replication_info.is_empty());
assert!(candidate.tier_stats.is_none());
assert!(candidate.unknown_tier_stats.is_none());
assert!(candidate.usage_snapshot_admin_projection);
assert_eq!(candidate.usage_snapshot_converged, Some(false));
}
#[test]
fn current_identity_filter_overflow_leaves_totals_unknown() {
let mut candidate = snapshot();
candidate.buckets_usage.insert("large".to_string(), usage(u64::MAX));
candidate.buckets_usage.insert("small".to_string(), usage(1));
candidate.buckets_count = 3;
let current = HashMap::from([
("large".to_string(), INCARNATION.to_string()),
("small".to_string(), INCARNATION.to_string()),
]);
assert_eq!(candidate.retain_current_bucket_usage_for_admin(&current), 3);
assert!(candidate.buckets_usage.is_empty());
assert!(candidate.bucket_sizes.is_empty());
assert_eq!(candidate.usage_snapshot_converged, Some(false));
assert!(candidate.usage_snapshot_admin_projection);
assert!(!candidate.is_complete_bucket_usage_snapshot());
}
@@ -766,9 +766,10 @@ impl Run {
|| async {
let mut ready = true;
let mut observations = Vec::with_capacity(4);
for node in 0..4 {
assert!(self.dist.cluster.nodes.len() >= 4);
for (node, cluster_node) in self.dist.cluster.nodes.iter().enumerate().take(4) {
match http
.get(format!("{}/health/ready", self.dist.cluster.nodes[node].url))
.get(format!("{}/health/ready", cluster_node.url))
.timeout(Duration::from_secs(3))
.send()
.await
@@ -1332,14 +1332,15 @@ mod tests {
.collect();
assert_eq!(confirmed.len(), 1);
let mut receivers = Vec::new();
for drive in 0..phase.disks.get(1).map_or(0, Vec::len) {
let receiver_endpoints = phase.disks.get(1).map_or(&[][..], |disks| &phase.endpoints[1][..disks.len()]);
for endpoint in receiver_endpoints {
for cas in [prepares[1], commits[1]] {
let matching: Vec<_> = events[1]
.iter()
.copied()
.filter(|event| {
event["kind"] == "receiver"
&& event["disk"] == phase.endpoints[1][drive]
&& event["disk"] == *endpoint
&& event["dst_volume"] == ".rustfs.sys"
&& event["dst_path"] == "pool.bin"
&& event["etag"] == cas["etag"]
+3
View File
@@ -96,6 +96,9 @@ mod replication_target_matrix_test;
#[cfg(test)]
mod data_usage_test;
#[cfg(test)]
mod scanner_wide_directory_test;
// KMS-specific test modules
#[cfg(test)]
mod kms;
@@ -22,10 +22,11 @@ use rustfs_protos::{
models::PingBodyBuilder,
proto_gen::node_service::{
BatchGenerallyLockRequest, BatchGenerallyLockResponse, BatchReadVersionRequest, BatchReadVersionResponse,
GenerallyLockRequest, GenerallyLockResponse, GenerallyLockResult, PingRequest, PingResponse,
ScannerPublicationLeaseReleaseRequest, ScannerPublicationLeaseReleaseResponse, ScannerPublicationLeaseRequest,
ScannerPublicationLeaseResponse, SnapshotLeaseMutationResponse, SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest,
SnapshotLeaseRequest, SnapshotLeaseResponse, node_service_server::NodeService,
GenerallyLockRequest, GenerallyLockResponse, GenerallyLockResult, MrfLifecycleRequest, MrfLifecycleResponse, PingRequest,
PingResponse, ScannerPublicationLeaseReleaseRequest, ScannerPublicationLeaseReleaseResponse,
ScannerPublicationLeaseRequest, ScannerPublicationLeaseResponse, SnapshotLeaseMutationResponse,
SnapshotLeaseReleaseRequest, SnapshotLeaseRenewRequest, SnapshotLeaseRequest, SnapshotLeaseResponse,
node_service_server::NodeService,
},
};
use std::pin::Pin;
@@ -995,6 +996,10 @@ impl NodeService for MinimalLockNodeService {
Err(Status::unimplemented("lock-only test server"))
}
async fn mrf_lifecycle(&self, _request: Request<MrfLifecycleRequest>) -> Result<Response<MrfLifecycleResponse>, Status> {
Err(Status::unimplemented("lock-only test server"))
}
async fn background_heal_status(
&self,
_request: Request<rustfs_protos::proto_gen::node_service::BackgroundHealStatusRequest>,
@@ -0,0 +1,208 @@
// Copyright 2026 RustFS Team
// Licensed under the Apache License, Version 2.0.
use std::collections::BTreeMap;
use std::path::Path;
use std::time::{Duration, SystemTime};
use aws_sdk_s3::{Client, primitives::ByteStream};
use futures::{StreamExt, TryStreamExt, stream};
use rustfs_data_usage::{BucketUsageValueState, DataUsageInfo};
use serde::{Deserialize, de::DeserializeOwned};
use tokio::time::{Instant, sleep, timeout};
use crate::common::{RustFSTestEnvironment, signed_request};
use crate::storage_api::scanner_peer_bucket::{BUCKET_META_PREFIX, RUSTFS_META_BUCKET};
type TestResult<T = ()> = Result<T, Box<dyn std::error::Error + Send + Sync>>;
const BUCKET: &str = "wide-directory-usage";
const WIDTH: usize = 512;
// The per-disk token bucket allows an initial burst of one second. Even the
// first pass can read at most (1 + 3) * 96 = 384 of these 512 root entries.
const SCAN_ENV: &[(&str, &str)] = &[
("RUST_LOG", "warn,rustfs::scanner::io=debug,rustfs::scanner::data_usage=debug"),
("RUSTFS_SCANNER_ENABLED", "true"),
("RUSTFS_SCANNER_START_DELAY_SECS", "0"),
("RUSTFS_SCANNER_CYCLE", "1"),
("RUSTFS_SCANNER_CYCLE_MAX_DURATION_SECS", "3"),
("RUSTFS_SCANNER_SPEED", "fastest"),
("RUSTFS_SCANNER_DISK_IO_BUDGET_IOPS", "96"),
("RUSTFS_SCANNER_DISK_IO_BUDGET_BYTES_PER_SEC", "off"),
];
#[derive(Debug, Deserialize)]
struct Status {
metrics: Metrics,
disk_io_budget: Vec<DiskBudget>,
}
#[derive(Debug, Deserialize)]
struct Metrics {
current_cycle_active: bool,
active_paths: Vec<String>,
cycle_max_duration_seconds: f64,
partial_cycles_runtime: u64,
}
#[derive(Debug, Deserialize)]
struct DiskBudget {
iops: Option<u64>,
}
async fn admin<T: DeserializeOwned>(env: &RustFSTestEnvironment, resource: &str) -> TestResult<T> {
let url = format!("{}/rustfs/admin/v3/{resource}", env.url);
let response = signed_request(http::Method::GET, &url, &env.access_key, &env.secret_key, None, None).await?;
Ok(response.error_for_status()?.json().await?)
}
async fn inventory(client: &Client) -> TestResult<BTreeMap<String, u64>> {
let mut pages = client.list_objects_v2().bucket(BUCKET).max_keys(37).into_paginator().send();
let mut sizes = BTreeMap::new();
while let Some(page) = pages.next().await {
for object in page?.contents() {
let key = object.key().ok_or("S3 inventory omitted object key")?;
let size = u64::try_from(object.size().ok_or("S3 inventory omitted object size")?)?;
assert!(sizes.insert(key.to_owned(), size).is_none(), "duplicate inventory key {key}");
}
}
Ok(sizes)
}
async fn scan_status(env: &RustFSTestEnvironment) -> TestResult<Status> {
let status: Status = admin(env, "scanner/status").await?;
assert_eq!(status.metrics.cycle_max_duration_seconds, 3.0, "effective cycle budget");
assert!(status.disk_io_budget.iter().all(|disk| disk.iops == Some(96)), "effective disk budget");
Ok(status)
}
#[tokio::test(flavor = "multi_thread")]
async fn certifies_wide_directory_after_mid_scan_process_restart() -> TestResult {
let mut env = RustFSTestEnvironment::new().await?;
let temporary_log = tempfile::NamedTempFile::new()?;
let server_log = env
.capture_log_path
.get_or_insert_with(|| temporary_log.path().display().to_string())
.clone();
let bucket_metadata = Path::new(&env.temp_dir)
.join(RUSTFS_META_BUCKET)
.join(BUCKET_META_PREFIX)
.join(BUCKET);
// Seed a cold bucket without allowing any earlier, larger-budget sweep to count it.
env.start_rustfs_server_without_cleanup_with_env(&[("RUSTFS_SCANNER_ENABLED", "false")])
.await?;
let client = env.create_s3_client();
client.create_bucket().bucket(BUCKET).send().await?;
stream::iter(0..WIDTH)
.map(Ok)
.try_for_each_concurrent(12, |i| {
let client = &client;
async move {
client
.put_object()
.bucket(BUCKET)
.key(format!("object-{i:064x}"))
.body(ByteStream::from(vec![42; 1 + i % 67]))
.send()
.await
.map(|_| ())
}
})
.await?;
let expected = inventory(&client).await?;
assert_eq!(expected.len(), WIDTH);
env.restart_server_preserving_data(vec![], SCAN_ENV).await?;
let first_pid = env.process.as_ref().ok_or("missing scanner process")?.id();
timeout(Duration::from_secs(60), async {
loop {
let status = scan_status(&env).await?;
if status.metrics.partial_cycles_runtime >= 1
&& bucket_metadata.join(".usage-cache.bin").join("xl.meta").is_file()
&& bucket_metadata.join(".usage-cache.bin.pages").is_dir()
&& status.metrics.current_cycle_active
&& status
.metrics
.active_paths
.iter()
.any(|path| path.split('/').any(|part| part == BUCKET))
{
let log = tokio::fs::read_to_string(&server_log).await?;
if !log
.lines()
.any(|line| line.contains("partial_saved_not_published") && line.contains(BUCKET))
{
sleep(Duration::from_millis(200)).await;
continue;
}
assert!(!status.disk_io_budget.is_empty(), "scanner never admitted disk I/O");
break Ok::<_, Box<dyn std::error::Error + Send + Sync>>(());
}
sleep(Duration::from_millis(200)).await;
}
})
.await
.map_err(|_| "scanner did not re-enter the bucket after exhausting the fixed runtime budget")??;
let partial: DataUsageInfo = admin(&env, "datausageinfo").await?;
assert!(
partial
.buckets_usage
.get(BUCKET)
.is_none_or(|value| value.usage_certification.is_none()),
"the first process certified the bucket before the injected crash"
);
env.stop_server();
// The terminal save has returned and the old process can no longer write.
assert!(
bucket_metadata.join(".usage-cache.bin").join("xl.meta").is_file()
&& bucket_metadata.join(".usage-cache.bin.pages").is_dir(),
"the interrupted process must have committed a paged checkpoint"
);
let restart_log_offset = usize::try_from(tokio::fs::metadata(&server_log).await?.len())?;
let restarted_at = SystemTime::now();
env.start_rustfs_server_without_cleanup_with_env(SCAN_ENV).await?;
let second_pid = env.process.as_ref().ok_or("missing restarted scanner process")?.id();
assert_ne!(first_pid, second_pid);
let deadline = Instant::now() + Duration::from_secs(180);
loop {
let usage: DataUsageInfo = admin(&env, "datausageinfo").await?;
let status = scan_status(&env).await?;
if let Some(bucket) = usage.buckets_usage.get(BUCKET)
&& bucket.usage_value_state == Some(BucketUsageValueState::Certified)
{
let log = tokio::fs::read_to_string(&server_log).await?;
let first_admission = log
.get(restart_log_offset..)
.ok_or("server log truncated across restart")?
.lines()
.filter_map(|line| serde_json::from_str::<serde_json::Value>(line).ok())
.find(|record| record["state"] == "bucket_checkpoint_prepared" && record["bucket"] == BUCKET)
.ok_or("no checkpoint admission in the restarted process")?;
assert_eq!(first_admission["resume_position"], true, "restarted scanner discarded the checkpoint");
assert!(usage.is_complete_bucket_usage_snapshot());
let certificate = bucket
.usage_certification
.ok_or("missing certificate for the measured bucket")?;
assert!(certificate.certified_at >= restarted_at, "stale pre-restart certificate");
assert!(status.metrics.partial_cycles_runtime > 0, "new process must cross a cycle deadline");
assert!(!status.disk_io_budget.is_empty(), "restarted scanner never admitted disk I/O");
assert_eq!(inventory(&client).await?, expected, "durable S3 inventory changed across restart");
let count = u64::try_from(expected.len())?;
let bytes = expected.values().sum::<u64>();
assert_eq!((bucket.objects_count, bucket.size), (count, bytes));
assert_eq!((usage.objects_total_count, usage.objects_total_size), (count, bytes));
eprintln!(
"PIDs {first_pid}/{second_pid}, runtime-limited cycles since restart {}, certificate epoch {} cycle {}, S3 and admin {} objects/{} bytes",
status.metrics.partial_cycles_runtime, certificate.epoch, certificate.cycle, count, bytes
);
return Ok(());
}
assert!(
Instant::now() < deadline,
"no certified bucket after restart: {status:?}, value={:?}",
usage.buckets_usage.get(BUCKET)
);
sleep(Duration::from_millis(500)).await;
}
}
+43 -9
View File
@@ -440,8 +440,8 @@ pub mod data_usage {
DATA_USAGE_CACHE_NAME, apply_bucket_usage_memory_overlay, compute_bucket_usage,
init_compression_total_memory_from_backend, invalidate_admin_data_usage_snapshot_cache,
invalidate_data_usage_snapshot_cache, live_bucket_usage_computations, load_admin_data_usage_from_backend_cached,
load_bucket_usage_from_backend_cached, load_compression_total_from_memory, load_data_usage_from_backend,
load_data_usage_from_backend_cached, quota_object_size, record_bucket_delete_marker_memory,
load_admin_data_usage_with_memory_overlay, load_bucket_usage_from_backend_cached, load_compression_total_from_memory,
load_data_usage_from_backend, load_data_usage_from_backend_cached, quota_object_size, record_bucket_delete_marker_memory,
record_bucket_object_delete_memory, record_bucket_object_version_write_memory, record_bucket_object_write_memory,
record_bucket_object_write_unknown_previous_memory, record_compression_total_memory,
refresh_bucket_usage_from_object_layer, refresh_versioned_bucket_usage_from_object_layer,
@@ -553,6 +553,10 @@ pub mod runtime {
global_lock_client, global_lock_clients, global_tier_config_mgr, local_disk_map_read, object_store_handle, region,
rustfs_port, setup_is_dist_erasure, setup_is_erasure, setup_is_erasure_sd, transition_state_handle,
};
pub use crate::runtime::usage_watch::{
SCANNER_USAGE_WATCH_COVERAGE_VERSION, SCANNER_USAGE_WATCH_LEASE_MS, ScannerUsageWatchKnown, ScannerUsageWatchObservation,
ScannerUsageWatchRequest,
};
}
pub mod layout {
@@ -636,6 +640,30 @@ pub mod rio {
}
pub mod rpc {
#[cfg(feature = "test-util")]
pub use crate::cluster::rpc::internode_data_transport::{
InternodeDataTransport, InternodeDataTransportCapabilities, NsScannerCapabilityRequest, NsScannerStreamRequest,
ReadStreamRequest, WalkDirStreamRequest, WriteStreamRequest,
};
#[cfg(feature = "test-util")]
pub mod test_util {
/// Build a remote disk using a controlled transport and no health monitor.
///
/// # Errors
/// Returns the normal remote-disk construction error for the supplied endpoint.
pub async fn remote_disk_with_transport(
endpoint: &crate::layout::endpoint::Endpoint,
transport: std::sync::Arc<dyn crate::cluster::rpc::internode_data_transport::InternodeDataTransport>,
) -> crate::disk::error::Result<crate::disk::DiskStore> {
let options = crate::disk::DiskOption {
cleanup: false,
health_check: false,
};
let disk = crate::cluster::rpc::remote_disk::RemoteDisk::new(endpoint, &options, transport).await?;
Ok(std::sync::Arc::new(crate::disk::Disk::Remote(Box::new(disk))))
}
}
pub use crate::cluster::rpc::http_auth::with_rpc_background_request;
pub use crate::cluster::rpc::{
AuthenticatedChannel, KMS_SIGNAL_SUBSYSTEM, LocalPeerS3Client, MAX_NETWORK_PROBE_BYTES, MAX_NETWORK_PROBE_DURATION,
@@ -648,13 +676,16 @@ pub mod rpc {
gen_tonic_replay_scope_headers, gen_tonic_signature_headers, gen_tonic_signature_interceptor,
node_service_time_out_client, node_service_time_out_client_no_auth, normalize_tonic_rpc_audience,
rpc_auth_is_resource_exhausted, rpc_background_admission, set_tonic_canonical_body_digest, sign_ns_scanner_capability,
sign_ns_scanner_capability_with_checkpoint_coverage, sign_ns_scanner_capability_with_tier_registry_generation,
sign_put_file_capability, sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers,
tonic_rpc_auth_failure_reason, verify_ns_scanner_capability, verify_ns_scanner_capability_with_checkpoint_coverage,
verify_ns_scanner_capability_with_tier_registry_generation, verify_put_file_auth_trailer, verify_put_file_capability,
verify_rpc_signature, verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest,
verify_tonic_mutation_body_digest, verify_tonic_mutation_body_digest_reject_unsigned, verify_tonic_rpc_response_proof,
verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap,
sign_ns_scanner_capability_with_checkpoint_coverage, sign_ns_scanner_capability_with_durable_progress,
sign_ns_scanner_capability_with_tier_registry_generation, sign_ns_scanner_capability_with_usage_capabilities,
sign_ns_scanner_usage_watch_capability, sign_put_file_capability, sign_tonic_rpc_response_proof,
tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason,
verify_ns_scanner_capability, verify_ns_scanner_capability_with_checkpoint_coverage,
verify_ns_scanner_capability_with_durable_progress, verify_ns_scanner_capability_with_tier_registry_generation,
verify_ns_scanner_capability_with_usage_capabilities, verify_ns_scanner_usage_watch_capability,
verify_put_file_auth_trailer, verify_put_file_capability, verify_rpc_signature, verify_tonic_boot_epoch_response,
verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest, verify_tonic_mutation_body_digest_reject_unsigned,
verify_tonic_rpc_response_proof, verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap,
};
}
@@ -697,6 +728,9 @@ pub mod store_list {
pub mod storage {
pub use crate::core::pools::{HealLifecycleExpiryContext, POOL_META_NAME};
pub use crate::store::HealWalkVersion;
pub use crate::store::delete_recovery_inventory::{
DeleteResidueInventoryPage, DeleteResidueSource, DeleteResidueSourceStatus,
};
pub use crate::store::{
BootstrapLocalTarget, ECStore, SCANNER_PUBLICATION_LEASE_TTL_MS, ScannerDataMovementPauseStatus, all_local_disk,
all_local_disk_path, find_local_disk_by_ref, find_local_disk_by_ref_for_rpc, init_local_disks,
+114 -4
View File
@@ -26,6 +26,8 @@ use crate::bucket::target::BucketTargetType;
use crate::bucket::target::{self, BucketTarget, BucketTargets, Credentials};
use crate::bucket::versioning_sys::BucketVersioningSys;
use crate::runtime::sources as runtime_sources;
use aws_sdk_s3::config::interceptors::BeforeDeserializationInterceptorContextRef;
use aws_sdk_s3::config::{ConfigBag, Intercept, RuntimeComponents};
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::error::SdkError;
use aws_sdk_s3::operation::complete_multipart_upload::CompleteMultipartUploadOutput;
@@ -53,8 +55,9 @@ use http::{HeaderMap, HeaderName, HeaderValue, StatusCode};
use reqwest::Client as HttpClient;
use rustfs_utils::http::{
AMZ_BUCKET_REPLICATION_STATUS, AMZ_OBJECT_LOCK_BYPASS_GOVERNANCE, AMZ_OBJECT_LOCK_LEGAL_HOLD, AMZ_OBJECT_LOCK_MODE,
AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, AMZ_OBJECT_TAGGING_LOWER, AMZ_STORAGE_CLASS, AMZ_WEBSITE_REDIRECT_LOCATION, is_amz_header,
is_minio_header, is_rustfs_header, is_standard_header, is_storageclass_header,
AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, AMZ_OBJECT_TAGGING, AMZ_OBJECT_TAGGING_LOWER, AMZ_STORAGE_CLASS,
AMZ_TAGGING_DIRECTIVE_LOWER, AMZ_WEBSITE_REDIRECT_LOCATION, is_amz_header, is_minio_header, is_rustfs_header,
is_standard_header, is_storageclass_header,
};
use rustfs_utils::http::{
SUFFIX_FORCE_DELETE, SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_ETAG, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_PROXY_REQUEST,
@@ -132,6 +135,36 @@ impl From<&BucketTarget> for RemoteS3EndpointSpec {
pub type HeadObjectSdkError = Box<SdkError<HeadObjectError>>;
/// Captures the `x-amz-tagging` response header of a replication HEAD, which
/// the SDK drops while deserializing `HeadObjectOutput`.
#[derive(Clone, Debug, Default)]
struct HeadObjectTaggingCapture(Arc<std::sync::Mutex<Option<String>>>);
impl HeadObjectTaggingCapture {
fn take(&self) -> Option<String> {
self.0.lock().unwrap_or_else(std::sync::PoisonError::into_inner).take()
}
}
impl Intercept for HeadObjectTaggingCapture {
fn name(&self) -> &'static str {
"HeadObjectTaggingCapture"
}
fn read_before_deserialization(
&self,
context: &BeforeDeserializationInterceptorContextRef<'_>,
_runtime_components: &RuntimeComponents,
_cfg: &mut ConfigBag,
) -> Result<(), aws_sdk_s3::error::BoxError> {
// Overwrite on every attempt so a retried HEAD reports the tags of the
// response that was actually deserialized.
*self.0.lock().unwrap_or_else(std::sync::PoisonError::into_inner) =
context.response().headers().get(AMZ_OBJECT_TAGGING_LOWER).map(str::to_string);
Ok(())
}
}
/// Whether an edited bucket target still addresses the same remote service
/// (endpoint, bucket, path style, TLS and identity), so a verdict learned
/// about that service stays valid across the edit.
@@ -2089,12 +2122,19 @@ impl TargetClient {
// object with an identical ETag, and the worker concludes the object
// already converged — so it never actually replicates it.
insert_header(&mut headers, SUFFIX_SOURCE_PROXY_REQUEST, "false");
self.client
// Ask for the tag set itself (MinIO `x-amz-tagging-directive: ACCESS`):
// a plain HEAD reports only the tag count, which cannot tell the
// replica's tags from the source's.
headers.insert(HeaderName::from_static(AMZ_TAGGING_DIRECTIVE_LOWER), HeaderValue::from_static("ACCESS"));
let target_tagging = HeadObjectTaggingCapture::default();
let mut output = self
.client
.head_object()
.bucket(bucket)
.key(object)
.set_version_id(resolve_read_api_version_id(version_id))
.customize()
.interceptor(target_tagging.clone())
.map_request(move |mut req| {
for (k, v) in headers.clone().into_iter() {
if let Some(key_str) = k.map(|k| k.as_str().to_string()) {
@@ -2106,7 +2146,17 @@ impl TargetClient {
})
.send()
.await
.map_err(Box::new)
.map_err(Box::new)?;
// The SDK's typed output has no field for the tag set; carry it where
// the replication comparison reads it. A target that ignores the
// directive leaves the key absent, which compares as "tags unknown".
if let Some(tagging) = target_tagging.take() {
output
.metadata
.get_or_insert_default()
.insert(AMZ_OBJECT_TAGGING.to_string(), tagging);
}
Ok(output)
}
/// Candidate replicas by content identity on a target that mints its own
@@ -4542,6 +4592,66 @@ mod tests {
);
}
#[derive(Clone, Debug)]
struct TaggingHeadConnector {
target_tagging: Option<&'static str>,
tagging_directives: Arc<std::sync::Mutex<Vec<Option<String>>>>,
}
impl SmithyHttpConnector for TaggingHeadConnector {
fn call(&self, request: HttpRequest) -> HttpConnectorFuture {
self.tagging_directives
.lock()
.expect("recorded directive lock should not be poisoned")
.push(request.headers().get(AMZ_TAGGING_DIRECTIVE_LOWER).map(str::to_string));
let mut response = HttpResponse::new(
aws_smithy_runtime_api::http::StatusCode::try_from(200_u16).expect("200 should be a valid response status"),
SdkBody::empty(),
);
response.headers_mut().insert("x-amz-meta-owner", "team-a");
// Both targets expose the standard count, including the older target
// that does not implement the optional tag-set response header.
response.headers_mut().insert(rustfs_utils::http::AMZ_TAG_COUNT, "2");
if let Some(tagging) = self.target_tagging {
response.headers_mut().insert(AMZ_OBJECT_TAGGING_LOWER, tagging);
}
HttpConnectorFuture::ready(Ok(response))
}
}
#[tokio::test]
async fn head_object_requests_and_reports_the_target_tag_set() {
for target_tagging in [Some("env=prod&team=a"), None] {
let tagging_directives = Arc::new(std::sync::Mutex::new(Vec::new()));
let connector = SharedHttpConnector::new(TaggingHeadConnector {
target_tagging,
tagging_directives: Arc::clone(&tagging_directives),
});
let http_client = http_client_fn(move |_settings, _components| connector.clone());
let (mut client, _) = recording_target_client();
client.client = Arc::new(s3_client_for_test(443, Some(http_client)));
let head = client
.head_object("target-bucket", "object", None)
.await
.expect("canned HEAD should succeed");
assert_eq!(
*tagging_directives
.lock()
.expect("recorded directive lock should not be poisoned"),
vec![Some("ACCESS".to_string())],
"the replication HEAD must ask the target for its tag set"
);
assert_eq!(head.tag_count(), Some(2));
let metadata = head.metadata.expect("user metadata should be parsed");
assert_eq!(metadata.get("owner").map(String::as_str), Some("team-a"));
// Absent means "tags unknown": the comparison then keeps treating a
// tagged replica as different instead of trusting the count alone.
assert_eq!(metadata.get(AMZ_OBJECT_TAGGING).map(String::as_str), target_tagging);
}
}
#[tokio::test]
async fn put_object_sends_source_version_id_query_to_target() {
// MinIO reads the replicated version only from the `versionId` query
+299 -22
View File
@@ -27,6 +27,7 @@ use crate::disk::RUSTFS_META_BUCKET;
use crate::error::{Error, Result, is_err_bucket_not_found, is_err_strict_volume_not_found};
use crate::object_api::ObjectOptions;
use crate::runtime::sources as runtime_sources;
use crate::runtime::usage_watch::UsageWatchMutationGuard;
use crate::storage_api_contracts::heal::HealOperations as _;
use crate::storage_api_contracts::namespace::NamespaceLocking as _;
use crate::store::{ECStore, await_bucket_namespace_operation};
@@ -270,7 +271,12 @@ pub async fn init_bucket_metadata_sys(api: Arc<ECStore>, buckets: Vec<String>) {
let sys = Arc::new(RwLock::new(sys));
instance_ctx.init_bucket_metadata_sys(sys.clone());
{
// Installing a whole cache can also withdraw buckets absent from its
// initial listing. Fence replacement of the instance's previous cache.
let _watch = instance_ctx.usage_watches.begin(None);
instance_ctx.init_bucket_metadata_sys(sys.clone());
}
if is_dist_erasure {
start_refresh_buckets_metadata_loop(sys);
@@ -1955,6 +1961,46 @@ pub(crate) async fn has_authoritative_never_versioned_state(bucket: &str) -> Res
bucket_meta_sys.has_authoritative_never_versioned_state(bucket).await
}
/// Hold the shared bucket config transaction through an operator's inspection and
/// discard. Read the authoritative disk snapshot under that lock without migrating
/// historical metadata or accepting a stale notification cache.
pub(crate) async fn legacy_discard_config_guard_in(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
incarnation: Uuid,
) -> Result<(rustfs_lock::NamespaceLockGuard, bool)> {
let sys = bucket_metadata_sys_of(ctx)?.read().await.clone();
let guard =
acquire_metadata_transaction_guard(sys.object_store(), bucket, "delete_residue_config", rustfs_lock::LockType::Shared)
.await?;
// The caller already holds the bucket lifecycle fence. The config transaction
// guard excludes writers while this scoped read and the eventual discard run.
let authority = sys.read_authoritative_metadata_from_disk_under_guard(bucket, &guard).await?;
let eligible = match authority {
BucketMetadataAuthority::Authoritative(metadata)
if metadata.bucket_incarnation_sidecar && metadata.bucket_incarnation_id == incarnation =>
{
authoritative_metadata_never_versioned(bucket, &metadata)?
&& matches!(
object_lock_config_state_from_authoritative_metadata(&metadata)?,
ObjectLockConfigState::ConfirmedAbsent
)
}
_ => false,
};
if guard.is_lock_lost() {
return Err(Error::PreconditionFailed);
}
Ok((guard, eligible))
}
fn authoritative_metadata_never_versioned(bucket: &str, metadata: &BucketMetadata) -> Result<bool> {
if let Some(raw_len) = metadata.xml_config_unreadable_len(super::metadata::BUCKET_VERSIONING_CONFIG) {
return Err(unreadable_config_error(bucket, super::metadata::BUCKET_VERSIONING_CONFIG, raw_len));
}
Ok(metadata.versioning_config.is_none() && metadata.versioning_config_xml.is_empty())
}
pub(crate) async fn has_authoritative_never_versioned_state_in(
ctx: &crate::runtime::instance::InstanceContext,
bucket: &str,
@@ -2063,6 +2109,37 @@ pub(crate) fn current_local_meta_mutation_gen() -> u64 {
LOCAL_META_MUTATION_GEN.load(std::sync::atomic::Ordering::Acquire)
}
// These are the cached inputs used by the scanner's lifecycle, retention and
// replication decisions. Raw bytes distinguish absent/unreadable records;
// typed values also cover publications of already-parsed metadata. A periodic
// reload of the same content or an unrelated config edit preserves the watch.
fn same_scanner_configuration(previous: &BucketMetadata, next: &BucketMetadata) -> bool {
fn target_resets(metadata: &BucketMetadata) -> impl Iterator<Item = (&str, &str, Option<OffsetDateTime>)> {
metadata.bucket_target_config.iter().flat_map(|targets| {
targets
.targets
.iter()
.map(|target| (target.arn.as_str(), target.reset_id.as_str(), target.reset_before_date))
})
}
previous.bucket_incarnation_id == next.bucket_incarnation_id
&& previous.bucket_incarnation_sidecar == next.bucket_incarnation_sidecar
&& previous.bucket_creation_committed == next.bucket_creation_committed
&& previous.lock_enabled == next.lock_enabled
&& previous.lifecycle_config_xml == next.lifecycle_config_xml
&& previous.lifecycle_config == next.lifecycle_config
&& previous.object_lock_config_xml == next.object_lock_config_xml
&& previous.object_lock_config == next.object_lock_config
&& previous.versioning_config_xml == next.versioning_config_xml
&& previous.versioning_config == next.versioning_config
&& previous.replication_config_xml == next.replication_config_xml
&& previous.replication_config == next.replication_config
&& previous.bucket_targets_config_json == next.bucket_targets_config_json
&& previous.bucket_targets_unreadable() == next.bucket_targets_unreadable()
&& target_resets(previous).eq(target_resets(next))
}
type LocalAuthorityCache = Arc<StdMutex<HashMap<String, (u64, Arc<BucketMetadata>)>>>;
#[derive(Debug, Clone)]
@@ -2147,6 +2224,33 @@ impl BucketMetadataSys {
self.api.upgrade()
}
/// The bucket publish lock precedes the metadata map write lock, then the
/// instance's watch registry. A caller keeps the returned guard until
/// dependent caches (including replication targets) have also been updated.
fn replace_cached_metadata(
&self,
map: &mut HashMap<String, Arc<BucketMetadata>>,
bucket: &str,
next: Option<Arc<BucketMetadata>>,
) -> (bool, Option<UsageWatchMutationGuard>) {
let changed = match (map.get(bucket), next.as_deref()) {
(Some(previous), Some(next)) => !same_scanner_configuration(previous, next),
(None, None) => false,
_ => true,
};
let watch = if changed {
self.object_store_if_live()
.map(|store| store.ctx.usage_watches.begin(Some(bucket)))
} else {
None
};
let previous = match next {
Some(next) => map.insert(bucket.to_owned(), next),
None => map.remove(bucket),
};
(previous.is_some(), watch)
}
/// Whether the backing `ECStore` has been dropped.
///
/// Used by `InstanceContext::clear_bucket_metadata_sys_if_stale` during
@@ -2354,7 +2458,7 @@ impl BucketMetadataSys {
let _publish_guard = self
.lock_metadata_publish(bucket, namespace_guard, "stale bucket metadata removal")
.await?;
let removed = self.metadata_map.write().await.remove(bucket).is_some();
let (removed, _watch) = self.replace_cached_metadata(&mut *self.metadata_map.write().await, bucket, None);
self.fabricated_metadata.write().await.remove(bucket);
self.missing_buckets.insert(bucket.to_string(), ()).await;
if removed {
@@ -2393,7 +2497,8 @@ impl BucketMetadataSys {
let _publish_guard = self
.lock_metadata_publish(bucket, namespace_guard, "initial bucket metadata publish")
.await?;
self.metadata_map.write().await.insert(bucket.to_string(), Arc::clone(&bm));
let (_, _watch) =
self.replace_cached_metadata(&mut *self.metadata_map.write().await, bucket, Some(Arc::clone(&bm)));
self.fabricated_metadata.write().await.remove(bucket);
self.missing_buckets.invalidate(bucket).await;
sync_bucket_target_sys(bucket, &bm).await;
@@ -2438,7 +2543,7 @@ impl BucketMetadataSys {
if !unchanged {
return Ok(());
}
let removed = map.remove(bucket).is_some();
let (removed, _watch) = self.replace_cached_metadata(&mut map, bucket, None);
drop(map);
self.fabricated_metadata.write().await.insert(bucket.to_string());
self.missing_buckets.invalidate(bucket).await;
@@ -2462,7 +2567,7 @@ impl BucketMetadataSys {
if !unchanged {
return Ok(());
}
map.insert(bucket.to_string(), Arc::clone(&metadata));
let (_, _watch) = self.replace_cached_metadata(&mut map, bucket, Some(Arc::clone(&metadata)));
drop(map);
self.fabricated_metadata.write().await.remove(bucket);
self.missing_buckets.invalidate(bucket).await;
@@ -2491,7 +2596,7 @@ impl BucketMetadataSys {
let publish_lock = self.metadata_publish_lock(&bucket);
let _publish_guard = publish_lock.lock().await;
let mut map = self.metadata_map.write().await;
map.insert(bucket.clone(), bm.clone());
let (_, _watch) = self.replace_cached_metadata(&mut map, &bucket, Some(Arc::clone(&bm)));
drop(map);
self.fabricated_metadata.write().await.remove(&bucket);
self.missing_buckets.invalidate(&bucket).await;
@@ -2512,7 +2617,7 @@ impl BucketMetadataSys {
let publish_lock = self.metadata_publish_lock(bucket);
let _publish_guard = publish_lock.lock().await;
let mut map = self.metadata_map.write().await;
let removed = map.remove(bucket).is_some();
let (removed, _watch) = self.replace_cached_metadata(&mut map, bucket, None);
drop(map);
let removed_fabricated = self.fabricated_metadata.write().await.remove(bucket);
self.missing_buckets.insert(bucket.to_string(), ()).await;
@@ -2528,6 +2633,7 @@ impl BucketMetadataSys {
async fn _reset(&mut self) {
bump_local_meta_mutation_gen();
let mut map = self.metadata_map.write().await;
let _watch = self.object_store_if_live().map(|store| store.ctx.usage_watches.begin(None));
map.clear();
drop(map);
self.fabricated_metadata.write().await.clear();
@@ -2849,7 +2955,7 @@ impl BucketMetadataSys {
if let Some(current) = map.get(bucket) {
return Ok((Arc::clone(current), true));
}
map.insert(bucket.to_string(), bm.clone());
let (_, _watch) = self.replace_cached_metadata(&mut map, bucket, Some(Arc::clone(&bm)));
drop(map);
self.fabricated_metadata.write().await.remove(bucket);
self.missing_buckets.invalidate(bucket).await;
@@ -2905,11 +3011,7 @@ impl BucketMetadataSys {
return Ok(false);
};
if let Some(raw_len) = metadata.xml_config_unreadable_len(super::metadata::BUCKET_VERSIONING_CONFIG) {
return Err(unreadable_config_error(bucket, super::metadata::BUCKET_VERSIONING_CONFIG, raw_len));
}
Ok(metadata.versioning_config.is_none() && metadata.versioning_config_xml.is_empty())
authoritative_metadata_never_versioned(bucket, &metadata)
}
pub async fn get_bucket_policy(&self, bucket: &str) -> Result<(BucketPolicy, OffsetDateTime)> {
@@ -3167,10 +3269,8 @@ impl BucketMetadataSys {
let _publish_guard = self
.lock_metadata_publish(bucket, &namespace_guard, "bucket creation commitment publish")
.await?;
self.metadata_map
.write()
.await
.insert(bucket.to_string(), Arc::clone(&metadata));
let (_, _watch) =
self.replace_cached_metadata(&mut *self.metadata_map.write().await, bucket, Some(Arc::clone(&metadata)));
self.fabricated_metadata.write().await.remove(bucket);
self.missing_buckets.invalidate(bucket).await;
Ok(BucketMetadataAuthority::Authoritative(metadata))
@@ -3225,7 +3325,7 @@ impl BucketMetadataSys {
let _publish_guard = self
.lock_metadata_publish(bucket, &namespace_guard, "missing legacy bucket metadata publish")
.await?;
self.metadata_map.write().await.remove(bucket);
let (_, _watch) = self.replace_cached_metadata(&mut *self.metadata_map.write().await, bucket, None);
self.fabricated_metadata.write().await.remove(bucket);
self.missing_buckets.insert(bucket.to_string(), ()).await;
return Ok(BucketMetadataAuthority::MissingBucket);
@@ -3296,10 +3396,8 @@ impl BucketMetadataSys {
let _publish_guard = self
.lock_metadata_publish(bucket, &namespace_guard, "legacy bucket metadata publish")
.await?;
self.metadata_map
.write()
.await
.insert(bucket.to_string(), Arc::clone(&metadata));
let (_, _watch) =
self.replace_cached_metadata(&mut *self.metadata_map.write().await, bucket, Some(Arc::clone(&metadata)));
self.fabricated_metadata.write().await.remove(bucket);
self.missing_buckets.invalidate(bucket).await;
sync_bucket_target_sys(bucket, &metadata).await;
@@ -5545,6 +5643,185 @@ mod tests {
(dirs, node_a, node_b)
}
#[test]
fn usage_watch_metadata_comparison_covers_scanner_inputs_and_skips_unrelated_config() {
let original = BucketMetadata::new("scanner-config");
for config in [
BUCKET_LIFECYCLE_CONFIG,
BUCKET_VERSIONING_CONFIG,
OBJECT_LOCK_CONFIG,
BUCKET_REPLICATION_CONFIG,
] {
let (_, bytes) = NEW_WRITER_CONFIGS
.iter()
.find(|(file, _)| *file == config)
.expect("scanner config fixture");
let mut parsed = original.clone();
parsed.update_config(config, bytes.to_vec()).expect("valid scanner config");
parsed.parse_all_configs().expect("parse scanner config");
assert!(!same_scanner_configuration(&original, &parsed), "{config} must fence the watch");
let mut unreadable = parsed.clone();
match config {
BUCKET_LIFECYCLE_CONFIG => unreadable.lifecycle_config = None,
BUCKET_VERSIONING_CONFIG => unreadable.versioning_config = None,
OBJECT_LOCK_CONFIG => unreadable.object_lock_config = None,
BUCKET_REPLICATION_CONFIG => unreadable.replication_config = None,
_ => unreachable!(),
}
assert!(
!same_scanner_configuration(&parsed, &unreadable),
"{config} parser state must fence the watch"
);
}
let mut changed = original.clone();
changed.lock_enabled = !original.lock_enabled;
assert!(!same_scanner_configuration(&original, &changed));
let mut changed = original.clone();
changed.bucket_target_config = Some(BucketTargets {
targets: vec![BucketTarget {
arn: "arn:replica".into(),
reset_id: "resync".into(),
..Default::default()
}],
});
assert!(
!same_scanner_configuration(&original, &changed),
"in-memory replication resync targets changed"
);
let mut unrelated = original.clone();
unrelated.tagging_config_xml = b"<Tagging/>".to_vec();
unrelated.lifecycle_config_updated_at = OffsetDateTime::UNIX_EPOCH;
assert!(same_scanner_configuration(&original, &unrelated));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn periodic_metadata_refresh_fences_usage_watch_after_missed_peer_notification() {
use crate::bucket::versioning::VersioningApi as _;
use crate::layout::endpoints::EndpointServerPools;
use crate::runtime::instance::InstanceContext;
use crate::runtime::usage_watch::{ScannerUsageWatchKnown, ScannerUsageWatchRequest};
use crate::store::init_local_disks_with_instance_ctx;
async fn observe(store: &ECStore, request: &ScannerUsageWatchRequest) -> ScannerUsageWatchKnown {
store.scanner_usage_watch(std::slice::from_ref(request)).await[0]
.known
.clone()
.expect("a healthy peer must return a bucket watch")
}
let (_dirs, writer) = isolated_store_over_temp_disks().await;
let endpoints = EndpointServerPools::from(vec![writer.pools[0].endpoints.clone()]);
let peer_ctx = Arc::new(InstanceContext::new());
init_local_disks_with_instance_ctx(&peer_ctx, endpoints.clone())
.await
.expect("initialize the second instance's disks");
let peer = ECStore::new_with_instance_ctx(
"127.0.0.1:0".parse().expect("test address"),
endpoints,
CancellationToken::new(),
peer_ctx,
)
.await
.expect("open the same deployment with an independent instance");
assert_eq!(writer.id, peer.id);
assert_ne!(writer.scanner_usage_watch_session_id(), peer.scanner_usage_watch_session_id());
init_bucket_metadata_sys(writer.clone(), Vec::new()).await;
let bucket = "missed-config-notification";
writer
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
let other = "unchanged-watch-bucket";
writer
.make_bucket(other, &MakeBucketOptions::default())
.await
.expect("create control bucket");
init_bucket_metadata_sys(peer.clone(), vec![bucket.to_owned(), other.to_owned()]).await;
let peer_sys = peer.ctx.bucket_metadata_sys().expect("peer metadata cache");
let writer_sys = writer.ctx.bucket_metadata_sys().expect("writer metadata cache");
assert!(!Arc::ptr_eq(&peer_sys, &writer_sys));
let (initial, _) = peer_sys
.read()
.await
.get_config(bucket)
.await
.expect("cache original metadata on peer");
assert!(initial.versioning_config_xml.is_empty());
let request = ScannerUsageWatchRequest {
bucket: bucket.into(),
bucket_incarnation: peer
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("bucket incarnation"),
watch_id: None,
session_id: None,
release: false,
};
let control = ScannerUsageWatchRequest {
bucket: other.into(),
bucket_incarnation: peer
.bucket_incarnation_id_from_disk(other)
.await
.expect("control incarnation"),
..request.clone()
};
let first = observe(&peer, &request).await;
let unaffected = observe(&peer, &control).await;
let enabled = br#"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>"#;
writer_sys
.read()
.await
.update(bucket, BUCKET_VERSIONING_CONFIG, enabled.to_vec())
.await
.expect("persist versioning");
assert!(
writer_sys
.read()
.await
.get_versioning_config(bucket)
.await
.expect("writer versioning")
.0
.enabled()
);
assert!(Arc::ptr_eq(
&initial,
&peer_sys.read().await.get(bucket).await.expect("stale cached snapshot")
));
assert!(observe(&peer, &request).await.unchanged_since(&first), "the peer missed the notification");
refresh_buckets_metadata_once(peer_sys.clone()).await;
let installed = peer_sys.read().await.get(bucket).await.expect("refreshed cached snapshot");
assert!(installed.versioning_config.as_ref().is_some_and(|config| config.enabled()));
let refreshed = observe(&peer, &request).await;
assert!(
!refreshed.unchanged_since(&first),
"installing changed scanner configuration must fence that peer's old watch"
);
assert!(observe(&peer, &control).await.unchanged_since(&unaffected));
refresh_buckets_metadata_once(peer_sys.clone()).await;
assert!(
observe(&peer, &request).await.unchanged_since(&refreshed),
"an unchanged periodic reload must preserve progress"
);
assert!(peer_sys.read().await.remove(bucket).await);
assert!(!observe(&peer, &request).await.unchanged_since(&refreshed));
assert!(observe(&peer, &control).await.unchanged_since(&unaffected));
peer_sys.write().await._reset().await;
let old_control = ScannerUsageWatchRequest {
watch_id: Some(unaffected.watch_id),
session_id: Some(unaffected.session_id),
..control
};
assert!(
peer.scanner_usage_watch(&[old_control]).await[0].known.is_none(),
"reset retires all cached policy proofs"
);
}
#[tokio::test]
#[serial]
async fn disk_incarnation_read_detects_stale_cache_until_peer_reload() {
+3 -1
View File
@@ -544,8 +544,10 @@ mod tests {
] {
let first: Error = make_error("buckets/first/.metadata.bin".into()).into();
let second: Error = make_error("buckets/second/.metadata.bin".into()).into();
let cloned = first.clone();
assert_eq!(first, second, "record paths must not fragment error grouping");
assert_eq!(first.clone(), second, "cloning must preserve error grouping");
assert_eq!(cloned, second, "cloning must preserve error grouping");
assert_eq!(std::io::Error::from(cloned).to_string(), message);
let io_error = std::io::Error::from(first);
let detail = io_error
+10 -10
View File
@@ -19,7 +19,7 @@ use super::{
use crate::bucket::metadata_sys::{
BucketMetadataSys, update, update_if_incarnation, update_quota_with_sys_if_incarnation_and_revision,
};
use crate::data_usage::{get_bucket_usage_memory, reload_unknown_bucket_usage};
use crate::data_usage::{get_bucket_usage_for_statistics, get_bucket_usage_memory, reload_unknown_bucket_usage};
use crate::error::StorageError;
use rustfs_config::QUOTA_CONFIG_FILE;
use rustfs_scanner_metrics::metrics::Metric;
@@ -96,7 +96,7 @@ impl QuotaChecker {
// reported as absent instead of failing the write (rustfs/backlog#2904).
// The snapshot protocol still admits from this value and fails closed.
let current_usage = if uses_durable_reservations {
self.report_only_usage(bucket).await
Self::report_usage(bucket, get_bucket_usage_memory(bucket).await)
} else {
Some(self.get_real_time_usage(bucket).await?)
};
@@ -235,8 +235,12 @@ impl QuotaChecker {
Ok(updated_at)
}
/// Return quota configuration with optional scanner usage, independently
/// of the usage requirements enforced by write admission.
/// Return the configured limit and best-effort usage for read-only statistics.
/// Single-process reports reflect known deletes before quota reconciliation;
/// distributed reports read the persisted scanner baseline.
///
/// # Errors
/// Returns an error for a missing bucket or unreadable quota configuration.
pub async fn get_quota_stats(&self, bucket: &str) -> Result<(BucketQuota, Option<u64>), QuotaError> {
// If bucket doesn't exist, return ConfigNotFound error
if !self.bucket_exists(bucket).await {
@@ -246,16 +250,12 @@ impl QuotaChecker {
}
let quota = self.get_quota_config(bucket).await?;
let current_usage = self.report_only_usage(bucket).await;
let current_usage = Self::report_usage(bucket, get_bucket_usage_for_statistics(bucket).await);
Ok((quota, current_usage))
}
/// Best-effort usage for statistics and durable admission reports.
/// Missing scanner coverage is unknown, not an error or a zero-sized bucket.
async fn report_only_usage(&self, bucket: &str) -> Option<u64> {
// The plain lookup: a report must not wait for an early snapshot reload.
let usage = get_bucket_usage_memory(bucket).await;
fn report_usage(bucket: &str, usage: Option<u64>) -> Option<u64> {
if usage.is_none() {
metrics::counter!(METRIC_QUOTA_USAGE_REPORT_UNAVAILABLE_TOTAL).increment(1);
debug!(
@@ -20,6 +20,8 @@ use crate::test_metrics::CapturingRecorder;
use rustfs_data_usage::{BucketUsageInfo, DataUsageInfo};
use std::time::SystemTime;
mod native_compatibility;
async fn stop_reporting_fixture(store: &crate::store::ECStore) {
if let Some(cancel) = store.ctx.background_cancel_token() {
cancel.cancel();
@@ -90,6 +92,10 @@ async fn install_reporting_snapshot(
#[tokio::test(flavor = "current_thread")]
#[serial]
async fn durable_quota_reporting_retires_degraded_floors_without_changing_ledger_admission() {
assert_durable_quota_reporting_preserves_admission().await;
}
async fn assert_durable_quota_reporting_preserves_admission() {
let recorder = CapturingRecorder::default();
let _recorder = metrics::set_default_local_recorder(&recorder);
for protocol in [1, 2] {
@@ -196,6 +202,11 @@ async fn durable_quota_reporting_retires_degraded_floors_without_changing_ledger
.expect("fixture cache cleanup");
crate::data_usage::invalidate_data_usage_snapshot_cache().await;
stop_reporting_fixture(&store).await;
for dir in _dirs {
let path = dir.path().to_path_buf();
dir.close().expect("remove durable quota reporting fixture");
assert!(!path.exists());
}
}
}
}
@@ -203,6 +214,10 @@ async fn durable_quota_reporting_retires_degraded_floors_without_changing_ledger
#[tokio::test]
#[serial]
async fn snapshot_quota_admission_retains_degraded_floors_until_migration() {
assert_snapshot_quota_preserves_degraded_floors().await;
}
async fn assert_snapshot_quota_preserves_degraded_floors() {
for source in ["missing", "legacy", "incomplete", "observed"] {
let (_dirs, store, _) = isolated_durable_quota_store().await;
let bucket = format!("quota-compat-{}", Uuid::new_v4().simple());
@@ -233,12 +248,21 @@ async fn snapshot_quota_admission_retains_degraded_floors_until_migration() {
.expect("fixture cache cleanup");
crate::data_usage::invalidate_data_usage_snapshot_cache().await;
stop_reporting_fixture(&store).await;
for dir in _dirs {
let path = dir.path().to_path_buf();
dir.close().expect("remove legacy quota reporting fixture");
assert!(!path.exists());
}
}
}
#[tokio::test(flavor = "current_thread")]
#[serial]
async fn quota_without_limit_keeps_the_zero_usage_io_fast_path() {
assert_unlimited_quota_preserves_fast_path().await;
}
async fn assert_unlimited_quota_preserves_fast_path() {
let (_dirs, store, sys) = isolated_durable_quota_store().await;
let bucket = format!("quota-unlimited-{}", Uuid::new_v4().simple());
create_bucket_with_quota_json(
@@ -259,6 +283,11 @@ async fn quota_without_limit_keeps_the_zero_usage_io_fast_path() {
assert_eq!(recorder.counter_value(METRIC_QUOTA_USAGE_REPORT_UNAVAILABLE_TOTAL, &[]), 0);
assert_eq!(crate::data_usage::reload_unknown_bucket_usage_calls_for_test(&bucket), 0);
stop_reporting_fixture(&store).await;
for dir in _dirs {
let path = dir.path().to_path_buf();
dir.close().expect("remove unlimited quota reporting fixture");
assert!(!path.exists());
}
}
#[tokio::test]
@@ -324,6 +353,41 @@ async fn quota_statistics_remain_available_without_a_complete_scanner_snapshot()
}
}
#[tokio::test]
#[serial]
async fn quota_statistics_exclude_local_deletes_while_snapshot_admission_retains_them() {
let (_dirs, store, _) = isolated_durable_quota_store().await;
let bucket = format!("quota-delete-report-{}", Uuid::new_v4().simple());
create_bucket_with_quota_json(&store, &bucket, legacy_quota_json(100)).await;
// Keep the snapshot-protocol control from initiating a background migration.
let metadata_sys = BucketMetadataSys::new(store.clone());
let (metadata, _) = metadata_sys.get_config(&bucket).await.expect("load quota metadata");
metadata_sys.set(bucket.clone(), metadata).await;
let checker = QuotaChecker::new(Arc::new(RwLock::new(metadata_sys)));
install_reporting_snapshot(store.clone(), &bucket, "complete", 60).await;
crate::data_usage::record_bucket_object_write_memory(&bucket, None, 40).await;
for (deleted, expected) in [(60, 40), (40, 0)] {
crate::data_usage::record_bucket_object_delete_memory(&bucket, deleted, true).await;
let (quota, usage) = checker.get_quota_stats(&bucket).await.expect("read quota statistics");
assert_eq!(quota.quota, Some(100));
assert_eq!(usage, Some(expected), "read-only quota GETs must report the surviving bytes");
let check = checker
.check_quota(&bucket, QuotaOperation::PutObject, 1)
.await
.expect("check snapshot quota");
assert_eq!(check.current_usage, Some(100));
assert!(!check.allowed, "reading statistics must not release held bytes for writes");
}
crate::data_usage::prepare_bucket_usage_for_namespace_change(&bucket, None)
.await
.expect("fixture cache cleanup");
crate::data_usage::invalidate_data_usage_snapshot_cache().await;
stop_reporting_fixture(&store).await;
}
#[tokio::test]
async fn quota_statistics_preserve_configuration_errors() {
let (_dirs, store, _) = isolated_durable_quota_store().await;
@@ -0,0 +1,265 @@
// Copyright 2026 RustFS Team
// SPDX-License-Identifier: Apache-2.0
use super::*;
use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
use rustfs_data_usage::{
DATA_USAGE_BUCKET_OBSERVATION_VERSION, DataUsageBucketObservation, DataUsageBucketObservationSource,
DataUsageSnapshotSetState,
};
#[tokio::test(flavor = "current_thread")]
#[serial]
async fn usage_quota_modes_preserve_actual_admission_decisions() {
let (dirs, store, sys) = isolated_durable_quota_store().await;
let bucket = format!("usage-no-quota-{}", Uuid::new_v4().simple());
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("unconfigured quota bucket");
let check = QuotaChecker::new(sys)
.check_quota(&bucket, QuotaOperation::PutObject, u64::MAX)
.await
.expect("no quota");
assert!(check.allowed && check.quota_limit.is_none());
store
.put_object(
&bucket,
"allowed",
&mut PutObjReader::from_vec(vec![0x5a; 512]),
&ObjectOptions::default(),
)
.await
.expect("real no-quota write");
stop_reporting_fixture(&store).await;
let paths = dirs.iter().map(|dir| dir.path().to_path_buf()).collect::<Vec<_>>();
drop(store);
drop(dirs);
assert!(paths.iter().all(|path| !path.exists()));
assert_unlimited_quota_preserves_fast_path().await;
assert_snapshot_quota_preserves_degraded_floors().await;
assert_durable_quota_reporting_preserves_admission().await;
println!(
"usage_assertion: {}",
serde_json::json!({
"case_id": "USAGE-19", "assertion_id": "USAGE-19-A01", "status": "PASS",
"evidence": {
"quota_modes": ["unconfigured", "unlimited", "legacy-snapshot", "durable-v1", "durable-v2"],
"durable_snapshot_sources": ["missing", "legacy", "incomplete", "observed", "complete"],
"durable_storage_decisions_per_fixture": ["reject-8192", "accept-512", "accept-3584", "reject-1"],
"legacy_missing_baseline_fails_closed": true,
"legacy_grounded_baselines": ["legacy", "incomplete", "observed"]
}
})
);
println!(
"usage_cleanup: {}",
serde_json::json!({"case_id": "USAGE-19", "status": "PASS", "fixtures_removed": true})
);
}
async fn persist_companion_fixture(
store: Arc<crate::store::ECStore>,
bucket: &str,
primary: bool,
companion_size: u64,
) -> (String, Vec<u8>) {
let incarnation = store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("real companion bucket identity");
let completed_at = SystemTime::UNIX_EPOCH + std::time::Duration::from_secs(20);
let mut snapshot = DataUsageInfo {
last_update: Some(completed_at),
scanner_epoch: Some(2),
scanner_cycle: Some(3),
usage_snapshot_complete: primary,
usage_snapshot_partial: !primary,
usage_snapshot_observational: true,
usage_snapshot_converged: Some(false),
usage_snapshot_authoritative_baseline: Some(DataUsageInfo::default().snapshot_identity()),
usage_snapshot_pending_buckets: vec![bucket.to_owned()],
usage_snapshot_pending_bucket_count: 1,
usage_snapshot_set_states: vec![DataUsageSnapshotSetState {
pool_index: 0,
set_index: 0,
scanner_epoch: Some(2),
scanner_cycle: Some(3),
scan_plan_digest: Some([1; 32]),
complete: primary,
..Default::default()
}],
..Default::default()
};
if primary {
snapshot.buckets_count = 1;
snapshot.buckets_usage.insert(
bucket.into(),
BucketUsageInfo {
size: 1000,
objects_count: 1,
versions_count: 1,
bucket_incarnation: Some(incarnation.to_string()),
..Default::default()
},
);
snapshot.bucket_sizes.insert(bucket.into(), 1000);
snapshot.calculate_totals();
}
let mut usage = BucketUsageInfo {
size: companion_size,
objects_count: 1,
versions_count: 1,
bucket_incarnation: Some(incarnation.to_string()),
..Default::default()
};
usage.mark_observed();
snapshot.usage_snapshot_bucket_observations.insert(
bucket.into(),
DataUsageBucketObservation {
version: DATA_USAGE_BUCKET_OBSERVATION_VERSION,
usage,
snapshot_identity: snapshot.snapshot_identity(),
completed_at,
sources: vec![DataUsageBucketObservationSource {
pool_index: 0,
set_index: 0,
bucket_incarnation: incarnation.to_string(),
scanner_epoch: 2,
scanner_cycle: 3,
scan_plan_digest: [1; 32],
started_plan: [2; 32],
requested_plan: [3; 32],
completed_at,
}],
},
);
let mut admin = snapshot.clone();
assert_eq!(
admin.project_bucket_observations_for_admin(&HashMap::from([(bucket.to_owned(), incarnation.to_string())])),
1,
"the companion must be valid, not discarded as malformed"
);
assert_eq!(admin.buckets_usage[bucket].size, companion_size);
let path = format!(
"{}/{}",
crate::disk::BUCKET_META_PREFIX,
rustfs_data_usage::DATA_USAGE_OBSERVED_OBJECT_NAME
);
let bytes = serde_json::to_vec(&snapshot).expect("encode valid companion snapshot");
crate::config::com::save_config(store, &path, bytes.clone())
.await
.expect("persist companion beside canonical fields");
crate::data_usage::invalidate_data_usage_snapshot_cache().await;
(path, bytes)
}
#[tokio::test(flavor = "current_thread")]
#[serial]
async fn usage_companions_do_not_expand_legacy_fallback_or_durable_admission() {
let mut decisions = Vec::new();
for protocol in [0, 1, 2] {
for (primary, companion_size) in [(true, 0), (true, 1_000_000), (false, 1_000_000)] {
let (dirs, store, sys) = isolated_durable_quota_store().await;
let bucket = format!("usage-companion-quota-{}", Uuid::new_v4().simple());
let quota = match protocol {
0 => legacy_quota_json(4096),
1 => serde_json::to_vec(&BucketQuota::new(Some(4096))).unwrap(),
_ => serde_json::to_vec(&BucketQuota::new_sharded(4096)).unwrap(),
};
create_bucket_with_quota_json(&store, &bucket, quota).await;
let (path, bytes) = persist_companion_fixture(store.clone(), &bucket, primary, companion_size).await;
let checker = if protocol == 0 {
QuotaChecker::new(Arc::new(RwLock::new(BucketMetadataSys::new(store.clone()))))
} else {
QuotaChecker::new(sys)
};
if protocol == 0 {
let small = checker.check_quota(&bucket, QuotaOperation::PutObject, 512).await;
if primary {
let small = small.expect("grounded primary fallback");
assert!(small.allowed && !small.uses_durable_reservations);
assert_eq!(small.current_usage, Some(1000));
assert!(
!checker
.check_quota(&bucket, QuotaOperation::PutObject, 4096)
.await
.expect("legacy limit")
.allowed
);
} else {
assert!(
matches!(small, Err(QuotaError::UsageUnavailable { .. })),
"a companion cannot invent a legacy baseline"
);
}
} else {
assert!(
checker
.check_quota(&bucket, QuotaOperation::PutObject, 512)
.await
.expect("durable checker")
.uses_durable_reservations
);
if protocol == 1 {
assert!(matches!(
store
.put_object(
&bucket,
"bootstrap",
&mut PutObjReader::from_vec(vec![0x5a; 512]),
&ObjectOptions::default()
)
.await,
Err(StorageError::SlowDown)
));
crate::bucket::quota::reservation::retry_quota_reconciliation_for_test(store.clone(), &bucket)
.await
.expect("real v1 ledger bootstrap");
}
store
.put_object(&bucket, "fits", &mut PutObjReader::from_vec(vec![0x5a; 4096]), &ObjectOptions::default())
.await
.expect("ledger admits exactly the physical quota regardless of companion");
assert!(
matches!(
store
.put_object(&bucket, "exceeds", &mut PutObjReader::from_vec(vec![0x5a; 1]), &ObjectOptions::default())
.await,
Err(StorageError::QuotaExceeded { .. })
),
"a low companion cannot expand the ledger"
);
}
assert_eq!(
crate::config::com::read_config(store.clone(), &path)
.await
.expect("retained scanner evidence"),
bytes
);
decisions
.push(serde_json::json!({"protocol": protocol, "primary_baseline": primary, "companion_size": companion_size}));
crate::data_usage::prepare_bucket_usage_for_namespace_change(&bucket, None)
.await
.expect("clear fixture caches");
crate::data_usage::invalidate_data_usage_snapshot_cache().await;
stop_reporting_fixture(&store).await;
let paths = dirs.iter().map(|dir| dir.path().to_path_buf()).collect::<Vec<_>>();
drop(store);
drop(dirs);
assert!(paths.iter().all(|path| !path.exists()));
}
}
println!(
"usage_assertion: {}",
serde_json::json!({
"case_id": "USAGE-19", "assertion_id": "USAGE-19-A02", "status": "PASS",
"evidence": {"validated_companion_fixtures": decisions, "legacy_primary_bytes": 1000, "durable_limit_bytes": 4096, "companion_only_legacy_admission": "UsageUnavailable"}
})
);
println!(
"usage_cleanup: {}",
serde_json::json!({"case_id": "USAGE-19", "status": "PASS", "fixtures_removed": true})
);
}
@@ -1602,6 +1602,7 @@ fn shard_object(ledger_object: &str, shard_index: u16, bucket_incarnation: Uuid,
}
fn shard_index(object: &str) -> u16 {
let object = rustfs_utils::path::encode_dir_object_ref(object);
let digest = Sha256::digest(object.as_bytes());
u16::from_be_bytes([digest[0], digest[1]]) % SHARDED_LEDGER_COUNT
}
@@ -3306,6 +3307,35 @@ mod tests {
);
}
#[tokio::test]
#[serial]
async fn quota_shard_reconciliation_matches_the_directory_object_write() {
use crate::object_api::PutObjReader;
use crate::storage_api_contracts::object::ObjectIO;
let (_directories, store) = metadata_sys::test_support::isolated_store_over_temp_disks().await;
metadata_sys::init_bucket_metadata_sys(Arc::clone(&store), Vec::new()).await;
let bucket = "quota-directory-marker";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create test bucket");
let listed_name = "photos/";
let stored_name = rustfs_utils::path::encode_dir_object(listed_name);
let mut body = PutObjReader::from_vec(vec![1, 2, 3]);
store
.put_object(bucket, listed_name, &mut body, &ObjectOptions::default())
.await
.expect("write a directory object with billable bytes");
let usages = exact_shard_usages(&store, bucket)
.await
.expect("reconcile the version listing");
assert_eq!(usages.iter().sum::<u64>(), 3);
assert_eq!(usages[usize::from(shard_index(&stored_name))], 3);
assert_eq!(shard_index(listed_name), shard_index(&stored_name));
}
#[tokio::test]
#[serial]
async fn sharded_reservation_persists_a_populated_lazy_shard_before_refill() {
@@ -52,9 +52,9 @@ use super::replication_resync_boundary::{
#[cfg(test)]
use super::replication_resync_boundary::{RESYNC_META_FORMAT, RESYNC_META_VERSION, WIRE_ZERO_TIME_UNIX};
use super::replication_storage_boundary::{
AdvancedGetOptions, EcstoreObjectOperations, GetObjectReader, HTTPPreconditions, HTTPRangeSpec, ObjectInfo, ObjectOptions,
ObjectToDelete, ReplicationObjectIO, ReplicationStatusWritebackCondition, ReplicationStatusWritebackMode, ReplicationStorage,
StatObjectOptions, StorageObjectInfoOrErr, WalkOptions,
EcstoreObjectOperations, GetObjectReader, HTTPPreconditions, HTTPRangeSpec, ObjectInfo, ObjectOptions, ObjectToDelete,
ReplicationObjectIO, ReplicationStatusWritebackCondition, ReplicationStatusWritebackMode, ReplicationStorage,
StorageObjectInfoOrErr, WalkOptions,
};
#[cfg(test)]
use super::replication_storage_boundary::{NamespaceLockFence, NamespaceLockSignalTestFence, ReplicationDeletedObject};
@@ -67,8 +67,8 @@ use super::replication_target_boundary::{
replication_complete_multipart_options, replication_delete_marker_purge_remove_options, replication_delete_remove_options,
replication_force_delete_remove_options, replication_object_is_ssec_encrypted, replication_put_object_header_size,
replication_put_object_options, replication_source_etag_is_comparable, replication_target_head_is_newer_null_version,
resolve_read_api_version_id, ssec_passthrough_evidence_present, ssec_passthrough_gate, version_identity_capability_from_put,
version_identity_drifted,
replication_target_tags_match, resolve_read_api_version_id, ssec_passthrough_evidence_present, ssec_passthrough_gate,
version_identity_capability_from_put, version_identity_drifted,
};
use super::replication_versioning_boundary::ReplicationVersioningStore;
use super::runtime_boundary as runtime_sources;
@@ -87,9 +87,9 @@ use metrics::counter;
use rmp_serde;
use rustfs_s3_types::EventName;
use rustfs_utils::http::{
AMZ_OBJECT_LOCK_LEGAL_HOLD, AMZ_OBJECT_LOCK_MODE, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, AMZ_TAGGING_DIRECTIVE,
SUFFIX_REPLICATION_RESET, SUFFIX_REPLICATION_STATUS, SUFFIX_REPLICATION_TARGET_VERSION_ARN_PREFIX, has_internal_suffix,
insert_str, replication_target_versions,
AMZ_OBJECT_LOCK_LEGAL_HOLD, AMZ_OBJECT_LOCK_MODE, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, SUFFIX_REPLICATION_RESET,
SUFFIX_REPLICATION_RESET_ARN_PREFIX, SUFFIX_REPLICATION_STATUS, SUFFIX_REPLICATION_TARGET_VERSION_ARN_PREFIX,
has_internal_suffix, insert_str, replication_target_versions,
};
use rustfs_utils::{DEFAULT_SIP_HASH_KEY, get_env_usize, sip_hash};
#[cfg(test)]
@@ -850,7 +850,8 @@ fn user_defined_value<'a>(user_defined: &'a HashMap<String, String>, name: &str)
/// difference (retention is only ever extended here — shortening or clearing
/// it needs a governance bypass the replication client does not hold, so the
/// replica keeps its longer window and the difference is logged). Heal and
/// resync attempts only touch what the HEAD proves diverged.
/// resync attempts apply known differences. An existing-object reset also
/// reconciles a nonempty tag set whose values the target HEAD omits.
async fn sync_drifted_replica_metadata(
tgt_client: &TargetClient,
object: &str,
@@ -863,10 +864,15 @@ async fn sync_drifted_replica_metadata(
let mut applied = false;
let source_tags = ReplicationTagFilter::decode_tags_to_map(&source.user_tags);
let source_tag_count = i32::try_from(source_tags.len()).unwrap_or(i32::MAX);
if op_type == ReplicationType::Metadata || replica.tag_count.unwrap_or_default() != source_tag_count {
let replica_tag_count = replica.tag_count.unwrap_or_default();
let update_tags = match op_type {
ReplicationType::Metadata => true,
ReplicationType::ExistingObject => !replication_target_tags_match(&source_tags, replica_tag_count, replica.metadata()),
_ => usize::try_from(replica_tag_count) != Ok(source_tags.len()),
};
if update_tags {
if source_tags.is_empty() {
if replica.tag_count.unwrap_or_default() > 0 {
if replica_tag_count > 0 || op_type == ReplicationType::ExistingObject {
tgt_client
.delete_object_tagging(&tgt_client.bucket, object, version_id.clone())
.await
@@ -4116,6 +4122,17 @@ fn replication_status_writeback_mode(state_update_needed: bool) -> ReplicationSt
}
}
/// Source-object metadata a replication worker publishes with its terminal
/// status.
#[derive(Clone, Copy, Default)]
struct ReplicationStatusWriteback<'a> {
replication_internal: Option<&'a String>,
/// `(arn, version id the target assigned)` for targets minting their own ids.
target_version_ledger: &'a [(String, String)],
/// `(arn, resync stamp)` for targets this attempt resynced.
resync_stamps: &'a [(String, String)],
}
/// Publish a worker's status with a storage-enforced compare-and-set token.
/// `put_object_metadata` checks the token while it owns the object write lock,
/// avoiding both a read/write race and any need to hold a hot object lock
@@ -4123,12 +4140,16 @@ fn replication_status_writeback_mode(state_update_needed: bool) -> ReplicationSt
fn replication_status_writeback_options(
roi: &ReplicateObjectInfo,
replication_lock_guard: &rustfs_lock::NamespaceLockGuard,
new_replication_internal: Option<&String>,
target_version_ledger: &[(String, String)],
writeback: ReplicationStatusWriteback<'_>,
mode: ReplicationStatusWritebackMode,
) -> ObjectOptions {
let ReplicationStatusWriteback {
replication_internal,
target_version_ledger,
resync_stamps,
} = writeback;
let mut eval_metadata = HashMap::new();
if let Some(status) = new_replication_internal {
if let Some(status) = replication_internal {
insert_str(&mut eval_metadata, SUFFIX_REPLICATION_STATUS, status.clone());
}
// One durable key per drifting target: the version id it assigned to this
@@ -4141,6 +4162,12 @@ fn replication_status_writeback_options(
assigned_version_id.clone(),
);
}
// The per-target existing-object resync stamp. Without it the object keeps
// answering "not yet resynced" for the target's reset id, and the scanner
// re-queues it on every cycle.
for (arn, stamp) in resync_stamps {
insert_str(&mut eval_metadata, &format!("{SUFFIX_REPLICATION_RESET_ARN_PREFIX}{arn}"), stamp.clone());
}
let mut write_opts = ObjectOptions {
version_id: roi.version_id.map(|version_id| version_id.to_string()),
eval_metadata: Some(eval_metadata),
@@ -4162,13 +4189,11 @@ async fn persist_replication_state_if_current<S: ReplicationStorage>(
roi: &ReplicateObjectInfo,
storage: &Arc<S>,
replication_lock_guard: &rustfs_lock::NamespaceLockGuard,
new_replication_internal: Option<&String>,
target_version_ledger: &[(String, String)],
writeback: ReplicationStatusWriteback<'_>,
mode: ReplicationStatusWritebackMode,
object_info: &mut ObjectInfo,
) -> Result<ReplicationStatePersistOutcome> {
let write_opts =
replication_status_writeback_options(roi, replication_lock_guard, new_replication_internal, target_version_ledger, mode);
let write_opts = replication_status_writeback_options(roi, replication_lock_guard, writeback, mode);
match storage.put_object_metadata(&roi.bucket, &roi.name, &write_opts).await {
Ok(updated) => {
*object_info = updated;
@@ -4334,6 +4359,12 @@ pub(crate) async fn replicate_object_with_outcome<S: ReplicationStorage>(
.filter(|target| !target.is_empty())
.filter_map(|target| Some((target.arn.clone(), target.target_version_id.clone()?)))
.collect();
let resync_stamps: Vec<(String, String)> = rinfos
.targets
.iter()
.filter(|target| !target.is_empty() && !target.resync_timestamp.is_empty())
.map(|target| (target.arn.clone(), target.resync_timestamp.clone()))
.collect();
let state_update_needed = roi.replication_status_internal != new_replication_internal
|| rinfos.replication_resynced()
|| !target_version_ledger.is_empty();
@@ -4343,8 +4374,11 @@ pub(crate) async fn replicate_object_with_outcome<S: ReplicationStorage>(
&roi,
&storage,
&obj_lock_guard,
new_replication_internal.as_ref(),
&target_version_ledger,
ReplicationStatusWriteback {
replication_internal: new_replication_internal.as_ref(),
target_version_ledger: &target_version_ledger,
resync_stamps: &resync_stamps,
},
writeback_mode,
&mut object_info,
)
@@ -5005,8 +5039,6 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
return rinfo;
}
let _sopts = replicate_all_stat_options(&object_info, &bucket, &tgt_client);
let Some((replication_action, object_info)) = resolve_replicate_all_action(
ReplicateAllActionContext {
siblings: &storage,
@@ -5090,6 +5122,13 @@ impl ReplicateObjectInfoExt for ReplicateObjectInfo {
return rinfo;
}
// The transfer is this target's resync of the object. Stamp it like the
// already-converged outcome above, or the next scanner cycle admits the
// object for the same reset id again.
if self.op_type == ReplicationType::ExistingObject && !tgt_client.reset_id.is_empty() {
apply_replication_resync_timestamp(&mut rinfo, &tgt_client.reset_id);
}
rinfo
}
@@ -5238,34 +5277,6 @@ fn note_replicate_all_target_bucket_empty(bucket: &str, tgt_client: &TargetClien
});
}
/// Build the stat options for the target metadata comparison, logging (without
/// failing) when the tagging directive header cannot be set.
fn replicate_all_stat_options(object_info: &ObjectInfo, bucket: &str, tgt_client: &TargetClient) -> StatObjectOptions {
let mut sopts = StatObjectOptions {
version_id: object_info.version_id.map(|v| v.to_string()).unwrap_or_default(),
internal: AdvancedGetOptions {
replication_proxy_request: "false".to_string(),
..Default::default()
},
..Default::default()
};
if let Err(err) = sopts.set(AMZ_TAGGING_DIRECTIVE, "ACCESS") {
debug!(
event = EVENT_RESYNC_RUNTIME_SKIPPED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION_RESYNC,
bucket = %bucket,
arn = %tgt_client.arn,
error = %err,
reason = "tagging_directive_header_invalid",
"Skipped replication tagging directive header detail"
);
}
sopts
}
/// Record a failed payload transfer: mark the outcome FAILED, log the target
/// operation failure, and take the target offline when the error is a network
/// failure.
@@ -5465,7 +5476,15 @@ async fn resolve_replicate_all_action(
// records the sync without running the PUT.
replication_action = ReplicationAction::None;
} else {
replication_action = if replica_matches {
// A current-version fallback may omit its version ID. Avoid
// mutating an unpinned version's tags or certifying stale tags.
replication_action = if replica_matches
&& (roi.op_type != ReplicationType::ExistingObject
|| replication_target_tags_match(
&ReplicationTagFilter::decode_tags_to_map(&object_info.user_tags),
oi.tag_count.unwrap_or_default(),
oi.metadata(),
)) {
ReplicationAction::None
} else {
ReplicationAction::All
@@ -6395,8 +6414,7 @@ mod tests {
let opts = replication_status_writeback_options(
&ReplicateObjectInfo::default(),
&guard,
None,
&[],
ReplicationStatusWriteback::default(),
ReplicationStatusWritebackMode::ValidateOnly,
);
forced_lost.store(true, std::sync::atomic::Ordering::Release);
@@ -6415,6 +6433,76 @@ mod tests {
);
}
#[tokio::test]
async fn terminal_writeback_persists_resync_stamp_that_ends_existing_object_resync() {
let lock = rustfs_lock::NamespaceLock::new(
"replication-status-writeback-resync-stamp".to_string(),
Arc::new(rustfs_lock::LocalClient::new()),
);
let guard = lock
.get_write_lock(
rustfs_lock::ObjectKey::new("bucket", "/[replicate]/object"),
"worker-a",
std::time::Duration::from_secs(2),
)
.await
.expect("replication lock should be acquired");
let arn = "arn:rustfs:replication::target:bucket";
let reset_before = OffsetDateTime::now_utc();
let mut rinfo = ReplicatedTargetInfo {
arn: arn.to_string(),
replication_status: ReplicationStatusType::Completed,
..Default::default()
};
apply_replication_resync_timestamp(&mut rinfo, "reset-1");
let resync_decision = |user_defined: &HashMap<String, String>, reset_id: &str| {
super::super::replication_object_decision_boundary::resync_target_for_object(
&super::super::replication_object_decision_boundary::ReplicationResyncTargetObject {
mod_time: Some(reset_before - time::Duration::hours(1)),
user_defined,
},
arn,
reset_id,
Some(reset_before),
ReplicationStatusType::Completed,
)
.replicate
};
let persisted = |resync_stamps: &[(String, String)]| {
replication_status_writeback_options(
&ReplicateObjectInfo::default(),
&guard,
ReplicationStatusWriteback {
resync_stamps,
..Default::default()
},
ReplicationStatusWritebackMode::Update,
)
.eval_metadata
.expect("writeback always carries evaluated metadata")
};
let unstamped = persisted(&[]);
assert!(
resync_decision(&unstamped, "reset-1"),
"an object older than the reset date stays admitted until it carries the stamp"
);
let stamped = persisted(&[(rinfo.arn.clone(), rinfo.resync_timestamp.clone())]);
let suffix = format!("{SUFFIX_REPLICATION_RESET_ARN_PREFIX}{arn}");
for prefix in [
rustfs_utils::http::RUSTFS_INTERNAL_PREFIX,
rustfs_utils::http::MINIO_INTERNAL_PREFIX,
] {
assert_eq!(stamped.get(&format!("{prefix}{suffix}")), Some(&rinfo.resync_timestamp));
}
assert!(
!resync_decision(&stamped, "reset-1"),
"the persisted stamp must end the resync for the same reset id"
);
assert!(resync_decision(&stamped, "reset-2"), "a new reset id must admit the object again");
}
#[test]
fn unavailable_object_target_is_persisted_as_failed() {
let arn = "arn:object-target";
@@ -8058,43 +8146,28 @@ mod tests {
assert!(purge_backoff_active(&dobj));
}
#[tokio::test]
async fn metadata_update_on_a_drifted_target_uses_the_tagging_api_instead_of_a_put() {
let (endpoint, server) = spawn_scripted_target_server(3, |line| {
if line.starts_with("HEAD ") && line.contains(&format!("versionId={DRIFTED_ASSIGNED_VERSION_ID}")) {
format!(
"HTTP/1.1 200 OK\r\nETag: \"{DRIFTED_ETAG}\"\r\nContent-Length: 4\r\nx-amz-version-id: {DRIFTED_ASSIGNED_VERSION_ID}\r\nx-amz-tagging-count: 1\r\nLast-Modified: Sun, 06 Sep 2026 10:00:00 GMT\r\nConnection: close\r\n\r\n"
)
} else if line.starts_with("HEAD ") {
empty_response("404 Not Found")
} else if line.starts_with("PUT ")
&& line.contains("tagging")
&& line.contains(&format!("versionId={DRIFTED_ASSIGNED_VERSION_ID}"))
{
empty_response("200 OK")
} else {
empty_response("500 Unexpected")
}
});
let target = test_target_client(endpoint);
async fn resolve_tagged_test_replica(
target: &Arc<TargetClient>,
op_type: ReplicationType,
has_ledger: bool,
) -> (Option<ReplicationAction>, ReplicatedTargetInfo) {
let (mut roi, mut object_info) = drifted_roi_and_object();
roi.op_type = ReplicationType::Metadata;
roi.op_type = op_type;
roi.user_tags = "phase=after".to_string();
object_info.user_tags = Arc::new("phase=after".to_string());
let mut user_defined = HashMap::new();
insert_str(
&mut user_defined,
&format!("{SUFFIX_REPLICATION_TARGET_VERSION_ARN_PREFIX}{}", target.arn),
DRIFTED_ASSIGNED_VERSION_ID.to_string(),
);
object_info.user_defined = Arc::new(user_defined);
let mut rinfo = replicate_all_target_info(&roi, &target);
object_info.user_tags = Arc::new(roi.user_tags.clone());
if has_ledger {
insert_str(
Arc::make_mut(&mut object_info.user_defined),
&format!("{SUFFIX_REPLICATION_TARGET_VERSION_ARN_PREFIX}{}", target.arn),
DRIFTED_ASSIGNED_VERSION_ID.to_string(),
);
}
let mut rinfo = replicate_all_target_info(&roi, target);
let action = resolve_replicate_all_action(
ReplicateAllActionContext {
siblings: &NoSiblingLedger,
roi: &roi,
tgt_client: &target,
tgt_client: target,
bucket: &roi.bucket,
object: &roi.name,
start_time: OffsetDateTime::now_utc(),
@@ -8104,24 +8177,93 @@ mod tests {
&mut rinfo,
)
.await;
(action.map(|(action, _)| action), rinfo)
}
assert!(
matches!(action, Some((ReplicationAction::None, _))),
"metadata is applied in place; the PUT transport must not run: {action:?}"
);
assert!(rinfo.error.is_none(), "{:?}", rinfo.error);
assert!(
rinfo.target_version_id.is_none(),
"an id the ledger already records must not trigger another metadata writeback"
);
async fn assert_versioned_tagging_sync(
op_type: ReplicationType,
tag_header: Option<&'static str>,
expect_write: bool,
put_status: &'static str,
) {
let wrote = Arc::new(AtomicBool::new(false));
let server_wrote = Arc::clone(&wrote);
let (endpoint, server) = spawn_scripted_target_server(2 + usize::from(expect_write), move |line| {
let assigned = line.contains(&format!("versionId={DRIFTED_ASSIGNED_VERSION_ID}"));
if line.starts_with("HEAD ") && assigned {
let tags = tag_header
.map(|value| format!("x-amz-tagging: {value}\r\n"))
.unwrap_or_default();
format!(
"HTTP/1.1 200 OK\r\nETag: \"{DRIFTED_ETAG}\"\r\nContent-Length: 4\r\nx-amz-version-id: {DRIFTED_ASSIGNED_VERSION_ID}\r\nx-amz-tagging-count: 1\r\n{tags}Last-Modified: Sun, 06 Sep 2026 10:00:00 GMT\r\nConnection: close\r\n\r\n"
)
} else if line.starts_with("HEAD ") {
empty_response("404 Not Found")
} else if line.starts_with("PUT ") && line.contains("tagging") && assigned {
server_wrote.store(true, Ordering::Release);
empty_response(put_status)
} else {
empty_response("400 Unexpected")
}
});
let mut target = test_target_client(endpoint);
Arc::get_mut(&mut target).expect("unshared test target").reset_id = "reset-1".to_string();
let (action, rinfo) = resolve_tagged_test_replica(&target, op_type, true).await;
assert_eq!(wrote.load(Ordering::Acquire), expect_write, "{tag_header:?}");
if put_status == "200 OK" {
assert_eq!(action, Some(ReplicationAction::None), "the payload PUT must not run");
assert!(rinfo.error.is_none(), "{:?}", rinfo.error);
assert!(rinfo.target_version_id.is_none(), "the ledger already records the replica version");
} else {
assert_eq!(action, None, "a failed tag write must stop before the caller can mark the reset complete");
assert_eq!(rinfo.replication_status, ReplicationStatusType::Failed);
assert!(rinfo.error.is_some() && rinfo.resync_timestamp.is_empty());
}
let seen = server.join().expect("test HTTP server should finish");
assert_eq!(seen.len(), 3, "HEAD by source id, HEAD by ledger id, PutObjectTagging: {seen:?}");
assert!(
seen[1].starts_with("HEAD ") && seen[1].contains(DRIFTED_ASSIGNED_VERSION_ID),
"{}",
seen[1]
);
assert!(seen[2].starts_with("PUT ") && seen[2].contains("tagging"), "{}", seen[2]);
assert_eq!(seen.len(), 2 + usize::from(expect_write));
}
#[tokio::test]
async fn metadata_update_on_a_drifted_target_uses_the_tagging_api_instead_of_a_put() {
assert_versioned_tagging_sync(ReplicationType::Metadata, None, true, "200 OK").await;
}
#[tokio::test]
async fn existing_object_resync_verifies_drifted_replica_tags_before_stamping() {
for (tag_header, write, status) in [
(Some("phase=after"), false, "200 OK"),
(Some("phase=before"), true, "200 OK"),
(None, true, "200 OK"),
(Some("phase=before"), true, "403 Forbidden"),
] {
assert_versioned_tagging_sync(ReplicationType::ExistingObject, tag_header, write, status).await;
}
}
#[tokio::test]
async fn existing_object_resync_does_not_stamp_unaddressed_replica_with_stale_tags() {
let (endpoint, server) = spawn_scripted_target_server(2, |line| {
if line.starts_with("HEAD ") && line.contains("versionId") {
empty_response("400 Bad Request")
} else if line.starts_with("HEAD ") {
format!(
"HTTP/1.1 200 OK\r\nETag: \"{DRIFTED_ETAG}\"\r\nContent-Length: 4\r\nx-amz-tagging-count: 1\r\nx-amz-tagging: phase=before\r\nConnection: close\r\n\r\n"
)
} else {
empty_response("400 Unexpected")
}
});
let target = test_target_client(endpoint);
let (action, rinfo) = resolve_tagged_test_replica(&target, ReplicationType::ExistingObject, false).await;
assert_eq!(action, Some(ReplicationAction::All));
assert!(rinfo.resync_timestamp.is_empty());
assert_eq!(server.join().expect("test HTTP server should finish").len(), 2);
}
#[test]
@@ -33,7 +33,6 @@ pub(crate) use crate::storage_api_contracts::object::{
};
pub(crate) use crate::storage_api_contracts::range::HTTPRangeSpec;
pub(crate) use rustfs_replication::{DeletedObject as ReplicationDeletedObject, ObjectToDelete as ReplicationObjectToDelete};
pub(crate) use rustfs_s3_client::api_get_options::{AdvancedGetOptions, StatObjectOptions};
type ListObjectsV2Info = StorageListObjectsV2Info<ObjectInfo>;
type ListObjectVersionsInfo = StorageListObjectVersionsInfo<ObjectInfo>;
@@ -45,6 +45,7 @@ pub(crate) use crate::bucket::bucket_target_sys::{
pub(crate) use crate::bucket::target::BucketTarget;
pub(crate) use crate::bucket::target::BucketTargets;
pub use rustfs_replication::SsecPassthroughCapability;
pub(crate) use rustfs_replication::object::replication_target_tags_match;
pub use rustfs_replication::{ObjectLockIntegrity, object_lock_put_integrity};
pub(crate) use rustfs_replication::{
SsecPassthroughGate, is_replication_target_offline_error, ssec_passthrough_gate, version_identity_drifted,
+270 -20
View File
@@ -139,6 +139,8 @@ const REPLAY_CACHE_AUTO_MEMORY_PERCENT: u64 = 13;
const REPLAY_CACHE_METADATA_BYTES: u64 = 1024 * 1024;
const REPLAY_CACHE_TABLE_OVERHEAD_BYTES: u64 = 512;
const NS_SCANNER_CAPABILITY_AUTH_DOMAIN: &[u8] = b"rustfs-ns-scanner-capability-v3";
const NS_SCANNER_COMPLETE_OBSERVATION_AUTH_DOMAIN: &[u8] = b"rustfs-ns-scanner-complete-observation-v1";
const NS_SCANNER_DURABLE_PROGRESS_AUTH_DOMAIN: &[u8] = b"rustfs-ns-scanner-durable-progress-v1";
const NS_SCANNER_CHECKPOINT_COVERAGE_AUTH_DOMAIN: &[u8] = b"rustfs-ns-scanner-checkpoint-coverage-v1";
const NS_SCANNER_TIER_REGISTRY_GENERATION_AUTH_DOMAIN: &[u8] = b"rustfs-ns-scanner-tier-registry-generation-v1";
pub const TONIC_RPC_PREFIX: &str = "/node_service.NodeService";
@@ -795,6 +797,8 @@ fn update_ns_scanner_capability_mac(
server_epoch: Uuid,
supports_tier_registry_generation: bool,
supports_checkpoint_coverage: bool,
supports_durable_progress: bool,
supports_complete_observation: bool,
) {
mac.update(NS_SCANNER_CAPABILITY_AUTH_DOMAIN);
mac.update(&NS_SCANNER_PROTOCOL_VERSION.to_be_bytes());
@@ -809,14 +813,22 @@ fn update_ns_scanner_capability_mac(
if supports_checkpoint_coverage {
mac.update(NS_SCANNER_CHECKPOINT_COVERAGE_AUTH_DOMAIN);
}
if supports_durable_progress {
mac.update(NS_SCANNER_DURABLE_PROGRESS_AUTH_DOMAIN);
}
if supports_complete_observation {
mac.update(NS_SCANNER_COMPLETE_OBSERVATION_AUTH_DOMAIN);
}
}
fn generate_ns_scanner_capability_proof(
fn generate_ns_scanner_capability_proof_with_observation(
secret: &str,
challenge: Uuid,
server_epoch: Uuid,
supports_tier_registry_generation: bool,
supports_checkpoint_coverage: bool,
supports_durable_progress: bool,
supports_complete_observation: bool,
) -> std::io::Result<Vec<u8>> {
if challenge.is_nil() || server_epoch.is_nil() {
return Err(std::io::Error::other("Invalid namespace scanner capability scope"));
@@ -829,17 +841,22 @@ fn generate_ns_scanner_capability_proof(
server_epoch,
supports_tier_registry_generation,
supports_checkpoint_coverage,
supports_durable_progress,
supports_complete_observation,
);
Ok(mac.finalize().into_bytes().to_vec())
}
fn verify_ns_scanner_capability_proof(
#[allow(clippy::too_many_arguments)]
fn verify_ns_scanner_capability_proof_with_observation(
secret: &str,
challenge: Uuid,
server_epoch: Uuid,
proof: &[u8],
supports_tier_registry_generation: bool,
supports_checkpoint_coverage: bool,
supports_durable_progress: bool,
supports_complete_observation: bool,
) -> std::io::Result<()> {
if challenge.is_nil() || server_epoch.is_nil() {
return Err(std::io::Error::other("Invalid namespace scanner capability scope"));
@@ -852,11 +869,35 @@ fn verify_ns_scanner_capability_proof(
server_epoch,
supports_tier_registry_generation,
supports_checkpoint_coverage,
supports_durable_progress,
supports_complete_observation,
);
mac.verify_slice(proof)
.map_err(|_| std::io::Error::new(std::io::ErrorKind::PermissionDenied, "Invalid namespace scanner capability proof"))
}
fn generate_ns_scanner_capability_proof(
secret: &str,
challenge: Uuid,
server_epoch: Uuid,
tier: bool,
checkpoint: bool,
progress: bool,
) -> std::io::Result<Vec<u8>> {
generate_ns_scanner_capability_proof_with_observation(secret, challenge, server_epoch, tier, checkpoint, progress, false)
}
fn verify_ns_scanner_capability_proof(
secret: &str,
challenge: Uuid,
server_epoch: Uuid,
proof: &[u8],
tier: bool,
checkpoint: bool,
progress: bool,
) -> std::io::Result<()> {
verify_ns_scanner_capability_proof_with_observation(secret, challenge, server_epoch, proof, tier, checkpoint, progress, false)
}
pub fn sign_ns_scanner_capability(challenge: Uuid, server_epoch: Uuid) -> std::io::Result<Vec<u8>> {
sign_ns_scanner_capability_with_tier_registry_generation(challenge, server_epoch, false)
}
@@ -878,6 +919,22 @@ pub fn sign_ns_scanner_capability_with_checkpoint_coverage(
server_epoch: Uuid,
supports_tier_registry_generation: bool,
supports_checkpoint_coverage: bool,
) -> std::io::Result<Vec<u8>> {
sign_ns_scanner_capability_with_durable_progress(
challenge,
server_epoch,
supports_tier_registry_generation,
supports_checkpoint_coverage,
false,
)
}
pub fn sign_ns_scanner_capability_with_durable_progress(
challenge: Uuid,
server_epoch: Uuid,
supports_tier_registry_generation: bool,
supports_checkpoint_coverage: bool,
supports_durable_progress: bool,
) -> std::io::Result<Vec<u8>> {
generate_ns_scanner_capability_proof(
&get_shared_secret()?,
@@ -885,6 +942,7 @@ pub fn sign_ns_scanner_capability_with_checkpoint_coverage(
server_epoch,
supports_tier_registry_generation,
supports_checkpoint_coverage,
supports_durable_progress,
)
}
@@ -909,6 +967,24 @@ pub fn verify_ns_scanner_capability_with_checkpoint_coverage(
proof: &[u8],
supports_tier_registry_generation: bool,
supports_checkpoint_coverage: bool,
) -> std::io::Result<()> {
verify_ns_scanner_capability_with_durable_progress(
challenge,
server_epoch,
proof,
supports_tier_registry_generation,
supports_checkpoint_coverage,
false,
)
}
pub fn verify_ns_scanner_capability_with_durable_progress(
challenge: Uuid,
server_epoch: Uuid,
proof: &[u8],
supports_tier_registry_generation: bool,
supports_checkpoint_coverage: bool,
supports_durable_progress: bool,
) -> std::io::Result<()> {
verify_ns_scanner_capability_proof(
&get_shared_secret()?,
@@ -917,6 +993,74 @@ pub fn verify_ns_scanner_capability_with_checkpoint_coverage(
proof,
supports_tier_registry_generation,
supports_checkpoint_coverage,
supports_durable_progress,
)
}
pub fn sign_ns_scanner_capability_with_usage_capabilities(
challenge: Uuid,
server_epoch: Uuid,
tier: bool,
checkpoint: bool,
progress: bool,
observation: bool,
) -> std::io::Result<Vec<u8>> {
generate_ns_scanner_capability_proof_with_observation(
&get_shared_secret()?,
challenge,
server_epoch,
tier,
checkpoint,
progress,
observation,
)
}
fn ns_scanner_usage_watch_capability_mac(secret: &str, challenge: Uuid, server_epoch: Uuid) -> std::io::Result<HmacSha256> {
if challenge.is_nil() || server_epoch.is_nil() {
return Err(std::io::Error::other("Invalid namespace scanner usage watch capability scope"));
}
let mut mac =
<HmacSha256 as KeyInit>::new_from_slice(secret.as_bytes()).map_err(|_| std::io::Error::other("Invalid RPC HMAC key"))?;
mac.update(b"rustfs-ns-scanner-usage-watch-capability-v1\0");
mac.update(&NS_SCANNER_PROTOCOL_VERSION.to_be_bytes());
mac.update(challenge.as_bytes());
mac.update(server_epoch.as_bytes());
Ok(mac)
}
/// Sign the independent worker capability to validate bucket watches before
/// reusing or durably publishing a complete scanner cache.
pub fn sign_ns_scanner_usage_watch_capability(challenge: Uuid, server_epoch: Uuid) -> std::io::Result<Vec<u8>> {
Ok(ns_scanner_usage_watch_capability_mac(&get_shared_secret()?, challenge, server_epoch)?
.finalize()
.into_bytes()
.to_vec())
}
pub fn verify_ns_scanner_usage_watch_capability(challenge: Uuid, server_epoch: Uuid, proof: &[u8]) -> std::io::Result<()> {
ns_scanner_usage_watch_capability_mac(&get_shared_secret()?, challenge, server_epoch)?
.verify_slice(proof)
.map_err(|_| std::io::Error::other("Invalid namespace scanner usage watch capability proof"))
}
pub fn verify_ns_scanner_capability_with_usage_capabilities(
challenge: Uuid,
server_epoch: Uuid,
proof: &[u8],
tier: bool,
checkpoint: bool,
progress: bool,
observation: bool,
) -> std::io::Result<()> {
verify_ns_scanner_capability_proof_with_observation(
&get_shared_secret()?,
challenge,
server_epoch,
proof,
tier,
checkpoint,
progress,
observation,
)
}
@@ -2135,13 +2279,119 @@ mod tests {
let secret = "test-scanner-capability-secret";
let challenge = Uuid::new_v4();
let server_epoch = Uuid::new_v4();
let proof = generate_ns_scanner_capability_proof(secret, challenge, server_epoch, false, false)
let proof = generate_ns_scanner_capability_proof(secret, challenge, server_epoch, false, false, false)
.expect("capability proof should be generated");
assert!(verify_ns_scanner_capability_proof(secret, challenge, server_epoch, &proof, false, false).is_ok());
assert!(verify_ns_scanner_capability_proof(secret, Uuid::new_v4(), server_epoch, &proof, false, false).is_err());
assert!(verify_ns_scanner_capability_proof(secret, challenge, Uuid::new_v4(), &proof, false, false).is_err());
assert!(verify_ns_scanner_capability_proof("different-secret", challenge, server_epoch, &proof, false, false).is_err());
assert!(verify_ns_scanner_capability_proof(secret, challenge, server_epoch, &proof, false, false, false).is_ok());
assert!(verify_ns_scanner_capability_proof(secret, Uuid::new_v4(), server_epoch, &proof, false, false, false).is_err());
assert!(verify_ns_scanner_capability_proof(secret, challenge, Uuid::new_v4(), &proof, false, false, false).is_err());
assert!(
verify_ns_scanner_capability_proof("different-secret", challenge, server_epoch, &proof, false, false, false).is_err()
);
}
#[test]
fn namespace_scanner_watch_capability_is_independent_and_rejects_replay() {
let _ = rustfs_credentials::set_global_rpc_secret("usage-watch-capability-test-secret".into());
let challenge = Uuid::new_v4();
let epoch = Uuid::new_v4();
let proof = sign_ns_scanner_usage_watch_capability(challenge, epoch).expect("watch capability");
verify_ns_scanner_usage_watch_capability(challenge, epoch, &proof).expect("exact watch capability scope");
assert!(verify_ns_scanner_usage_watch_capability(Uuid::new_v4(), epoch, &proof).is_err());
assert!(verify_ns_scanner_usage_watch_capability(challenge, Uuid::new_v4(), &proof).is_err());
for progress in [false, true] {
for observation in [false, true] {
let other =
sign_ns_scanner_capability_with_usage_capabilities(challenge, epoch, true, true, progress, observation)
.expect("independent older capabilities");
assert!(verify_ns_scanner_usage_watch_capability(challenge, epoch, &other).is_err());
assert!(
verify_ns_scanner_capability_with_usage_capabilities(
challenge,
epoch,
&proof,
true,
true,
progress,
observation
)
.is_err()
);
}
}
}
#[test]
fn namespace_scanner_observation_and_progress_scopes_are_independent() {
let challenge = Uuid::new_v4();
let epoch = Uuid::new_v4();
let secret = "independent-usage-capabilities";
for progress in [false, true] {
for observation in [false, true] {
let proof = generate_ns_scanner_capability_proof_with_observation(
secret,
challenge,
epoch,
true,
true,
progress,
observation,
)
.expect("proof");
assert!(
verify_ns_scanner_capability_proof_with_observation(
secret,
challenge,
epoch,
&proof,
true,
true,
progress,
observation
)
.is_ok()
);
assert!(
verify_ns_scanner_capability_proof_with_observation(
secret,
challenge,
epoch,
&proof,
true,
true,
!progress,
observation
)
.is_err()
);
assert!(
verify_ns_scanner_capability_proof_with_observation(
secret,
challenge,
epoch,
&proof,
true,
true,
progress,
!observation
)
.is_err()
);
}
}
}
#[test]
fn namespace_scanner_capability_proof_binds_durable_progress_independently() {
let challenge = Uuid::new_v4();
let epoch = Uuid::new_v4();
let secret = "durable-progress-test";
for supported in [false, true] {
let proof = generate_ns_scanner_capability_proof(secret, challenge, epoch, true, true, supported).expect("proof");
assert!(verify_ns_scanner_capability_proof(secret, challenge, epoch, &proof, true, true, supported).is_ok());
assert!(verify_ns_scanner_capability_proof(secret, challenge, epoch, &proof, true, true, !supported).is_err());
assert!(verify_ns_scanner_capability_proof(secret, Uuid::new_v4(), epoch, &proof, true, true, supported).is_err());
}
}
#[test]
@@ -2149,16 +2399,16 @@ mod tests {
let secret = "checkpoint-capability-test-secret";
let challenge = Uuid::new_v4();
let server_epoch = Uuid::new_v4();
let proof = generate_ns_scanner_capability_proof(secret, challenge, server_epoch, true, true)
let proof = generate_ns_scanner_capability_proof(secret, challenge, server_epoch, true, true, false)
.expect("checkpoint capability proof");
assert!(verify_ns_scanner_capability_proof(secret, challenge, server_epoch, &proof, true, true).is_ok());
assert!(verify_ns_scanner_capability_proof(secret, challenge, server_epoch, &proof, true, false).is_err());
assert!(verify_ns_scanner_capability_proof(secret, challenge, server_epoch, &proof, false, true).is_err());
let legacy = generate_ns_scanner_capability_proof(secret, challenge, server_epoch, true, false)
assert!(verify_ns_scanner_capability_proof(secret, challenge, server_epoch, &proof, true, true, false).is_ok());
assert!(verify_ns_scanner_capability_proof(secret, challenge, server_epoch, &proof, true, false, false).is_err());
assert!(verify_ns_scanner_capability_proof(secret, challenge, server_epoch, &proof, false, true, false).is_err());
let legacy = generate_ns_scanner_capability_proof(secret, challenge, server_epoch, true, false, false)
.expect("legacy generation capability proof");
assert!(verify_ns_scanner_capability_proof(secret, challenge, server_epoch, &legacy, true, true).is_err());
assert!(verify_ns_scanner_capability_proof(secret, challenge, server_epoch, &[], true, true).is_err());
assert!(verify_ns_scanner_capability_proof(secret, Uuid::new_v4(), server_epoch, &proof, true, true).is_err());
assert!(verify_ns_scanner_capability_proof(secret, challenge, server_epoch, &legacy, true, true, false).is_err());
assert!(verify_ns_scanner_capability_proof(secret, challenge, server_epoch, &[], true, true, false).is_err());
assert!(verify_ns_scanner_capability_proof(secret, Uuid::new_v4(), server_epoch, &proof, true, true, false).is_err());
}
#[test]
@@ -2166,14 +2416,14 @@ mod tests {
let secret = "test-scanner-capability-secret";
let challenge = Uuid::new_v4();
let server_epoch = Uuid::new_v4();
let proof = generate_ns_scanner_capability_proof(secret, challenge, server_epoch, true, false)
let proof = generate_ns_scanner_capability_proof(secret, challenge, server_epoch, true, false, false)
.expect("generation capability proof should be generated");
assert!(verify_ns_scanner_capability_proof(secret, challenge, server_epoch, &proof, true, false).is_ok());
assert!(verify_ns_scanner_capability_proof(secret, challenge, server_epoch, &proof, false, false).is_err());
let legacy = generate_ns_scanner_capability_proof(secret, challenge, server_epoch, false, false)
assert!(verify_ns_scanner_capability_proof(secret, challenge, server_epoch, &proof, true, false, false).is_ok());
assert!(verify_ns_scanner_capability_proof(secret, challenge, server_epoch, &proof, false, false, false).is_err());
let legacy = generate_ns_scanner_capability_proof(secret, challenge, server_epoch, false, false, false)
.expect("legacy capability proof should be generated");
assert!(verify_ns_scanner_capability_proof(secret, challenge, server_epoch, &legacy, true, false).is_err());
assert!(verify_ns_scanner_capability_proof(secret, challenge, server_epoch, &legacy, true, false, false).is_err());
}
/// Security regression for GHSA-r5qv-rc46-hv8q (internode RPC fail-closed,
@@ -14,14 +14,16 @@
use crate::cluster::rpc::{
build_auth_headers, build_put_file_auth_trailer, verify_ns_scanner_capability_with_checkpoint_coverage,
verify_ns_scanner_capability_with_tier_registry_generation, verify_put_file_capability,
verify_ns_scanner_capability_with_durable_progress, verify_ns_scanner_capability_with_tier_registry_generation,
verify_ns_scanner_capability_with_usage_capabilities, verify_put_file_capability,
};
use crate::disk::error::{Error, Result};
use crate::disk::{FileReader, FileWriter};
use crate::storage_api_contracts::internode::{
NS_SCANNER_BODY_SHA256_QUERY, NS_SCANNER_CAPABILITY_CHALLENGE_QUERY, NS_SCANNER_CHECKPOINT_COVERAGE_QUERY,
NS_SCANNER_CYCLE_QUERY, NS_SCANNER_LEADER_EPOCH_QUERY, NS_SCANNER_PROTOCOL_VERSION, NS_SCANNER_PROTOCOL_VERSION_QUERY,
NS_SCANNER_REQUEST_ID_QUERY, NS_SCANNER_SERVER_EPOCH_QUERY, NS_SCANNER_SESSION_ID_QUERY, NS_SCANNER_SESSION_SEQUENCE_QUERY,
NS_SCANNER_COMPLETE_OBSERVATION_QUERY, NS_SCANNER_CYCLE_QUERY, NS_SCANNER_DURABLE_PROGRESS_QUERY,
NS_SCANNER_LEADER_EPOCH_QUERY, NS_SCANNER_PROTOCOL_VERSION, NS_SCANNER_PROTOCOL_VERSION_QUERY, NS_SCANNER_REQUEST_ID_QUERY,
NS_SCANNER_SERVER_EPOCH_QUERY, NS_SCANNER_SESSION_ID_QUERY, NS_SCANNER_SESSION_SEQUENCE_QUERY,
NS_SCANNER_TIER_REGISTRY_GENERATION_QUERY, NsScannerCapabilityResponse, PUT_FILE_AUTH_QUERY, PUT_FILE_AUTH_V1,
PUT_FILE_CAPABILITY_CHALLENGE_QUERY, PUT_FILE_CAPABILITY_QUERY, PUT_FILE_CAPABILITY_VERSION, PUT_FILE_NONCE_QUERY,
PUT_FILE_SERVER_EPOCH_QUERY, PutFileCapabilityResponse, WALK_DIR_BODY_SHA256_QUERY, WALK_DIR_STREAM_COMPLETION_QUERY,
@@ -267,6 +269,9 @@ pub struct NsScannerCapabilityRequest {
pub endpoint: String,
pub supports_tier_registry_generation: bool,
pub supports_checkpoint_coverage: bool,
pub supports_durable_progress: bool,
pub supports_complete_observation: bool,
pub supports_usage_watch: bool,
}
/// Data-plane stream opener used by `RemoteDisk`.
@@ -307,6 +312,9 @@ pub trait InternodeDataTransport: Send + Sync + std::fmt::Debug {
proof: Vec::new(),
supports_tier_registry_generation: None,
supports_checkpoint_coverage: None,
supports_durable_progress: None,
supports_complete_observation: None,
usage_watch_proof: None,
})
}
// Interface facet nobody calls yet: every transport implements both, but no
@@ -401,11 +409,53 @@ impl InternodeDataTransport for TcpHttpInternodeDataTransport {
}
async fn probe_ns_scanner_capability(&self, request: NsScannerCapabilityRequest) -> Result<NsScannerCapabilityResponse> {
let request = if request.supports_usage_watch {
match self.probe_ns_scanner_capability_once(&request).await {
Ok(response) => return Ok(response),
Err(error) if ns_scanner_capability_error_allows_legacy(&error) => NsScannerCapabilityRequest {
supports_usage_watch: false,
..request
},
Err(error) => return Err(error),
}
} else {
request
};
let request = if request.supports_complete_observation {
match self.probe_ns_scanner_capability_once(&request).await {
Ok(response) => return Ok(response),
Err(error) if ns_scanner_capability_error_allows_legacy(&error) => NsScannerCapabilityRequest {
supports_complete_observation: false,
supports_usage_watch: false,
..request
},
Err(error) => return Err(error),
}
} else {
request
};
let request = if request.supports_durable_progress {
match self.probe_ns_scanner_capability_once(&request).await {
Ok(response) => return Ok(response),
Err(error) if ns_scanner_capability_error_allows_legacy(&error) => NsScannerCapabilityRequest {
supports_durable_progress: false,
supports_complete_observation: false,
supports_usage_watch: false,
..request
},
Err(error) => return Err(error),
}
} else {
request
};
let request = if request.supports_checkpoint_coverage {
match self.probe_ns_scanner_capability_once(&request).await {
Ok(response) => return Ok(response),
Err(error) if ns_scanner_capability_error_allows_legacy(&error) => NsScannerCapabilityRequest {
supports_checkpoint_coverage: false,
supports_durable_progress: false,
supports_complete_observation: false,
supports_usage_watch: false,
..request
},
Err(error) => return Err(error),
@@ -425,6 +475,9 @@ impl InternodeDataTransport for TcpHttpInternodeDataTransport {
endpoint: request.endpoint.clone(),
supports_tier_registry_generation: false,
supports_checkpoint_coverage: false,
supports_durable_progress: false,
supports_complete_observation: false,
supports_usage_watch: false,
};
match self.probe_ns_scanner_capability_once(&legacy_request).await {
Ok(mut response) => {
@@ -482,13 +535,62 @@ impl TcpHttpInternodeDataTransport {
if response.version != NS_SCANNER_PROTOCOL_VERSION || response.server_epoch.is_nil() {
return Err(Error::other("incompatible remote namespace scanner capability response"));
}
if let Err(err) = verify_ns_scanner_capability_with_checkpoint_coverage(
if request.supports_usage_watch {
if let Some(proof) = response.usage_watch_proof.as_deref() {
crate::cluster::rpc::verify_ns_scanner_usage_watch_capability(challenge, response.server_epoch, proof)
.map_err(|_| Error::other("remote namespace scanner usage watch capability authentication failed"))?;
}
} else {
response.usage_watch_proof = None;
}
if let Err(err) = verify_ns_scanner_capability_with_usage_capabilities(
challenge,
response.server_epoch,
&response.proof,
request.supports_tier_registry_generation,
request.supports_checkpoint_coverage,
request.supports_durable_progress,
request.supports_complete_observation,
) {
if request.supports_complete_observation
&& verify_ns_scanner_capability_with_durable_progress(
challenge,
response.server_epoch,
&response.proof,
request.supports_tier_registry_generation,
request.supports_checkpoint_coverage,
request.supports_durable_progress,
)
.is_ok()
{
normalize_ns_scanner_capability_response(
&mut response,
request.supports_tier_registry_generation,
request.supports_checkpoint_coverage,
request.supports_durable_progress,
false,
);
return Ok(response);
}
if request.supports_durable_progress
&& verify_ns_scanner_capability_with_checkpoint_coverage(
challenge,
response.server_epoch,
&response.proof,
request.supports_tier_registry_generation,
request.supports_checkpoint_coverage,
)
.is_ok()
{
normalize_ns_scanner_capability_response(
&mut response,
request.supports_tier_registry_generation,
request.supports_checkpoint_coverage,
false,
false,
);
return Ok(response);
}
if request.supports_checkpoint_coverage
&& verify_ns_scanner_capability_with_tier_registry_generation(
challenge,
@@ -498,7 +600,13 @@ impl TcpHttpInternodeDataTransport {
)
.is_ok()
{
normalize_ns_scanner_capability_response(&mut response, request.supports_tier_registry_generation, false);
normalize_ns_scanner_capability_response(
&mut response,
request.supports_tier_registry_generation,
false,
false,
false,
);
return Ok(response);
}
// A permissive older peer can ignore the additive marker and
@@ -508,6 +616,8 @@ impl TcpHttpInternodeDataTransport {
if request.supports_tier_registry_generation && ns_scanner_capability_legacy_proof_is_valid(challenge, &response) {
response.supports_tier_registry_generation = None;
response.supports_checkpoint_coverage = None;
response.supports_durable_progress = None;
response.supports_complete_observation = None;
return Ok(response);
}
return Err(Error::other(format!("remote namespace scanner capability authentication failed: {err}")));
@@ -521,6 +631,8 @@ impl TcpHttpInternodeDataTransport {
&mut response,
request.supports_tier_registry_generation,
request.supports_checkpoint_coverage,
request.supports_durable_progress,
request.supports_complete_observation,
);
Ok(response)
}
@@ -855,9 +967,13 @@ fn normalize_ns_scanner_capability_response(
response: &mut NsScannerCapabilityResponse,
requested_generation_support: bool,
requested_checkpoint_coverage: bool,
requested_durable_progress: bool,
requested_complete_observation: bool,
) {
response.supports_tier_registry_generation = requested_generation_support.then_some(true);
response.supports_checkpoint_coverage = requested_checkpoint_coverage.then_some(true);
response.supports_durable_progress = requested_durable_progress.then_some(true);
response.supports_complete_observation = requested_complete_observation.then_some(true);
}
fn ns_scanner_capability_legacy_proof_is_valid(challenge: Uuid, response: &NsScannerCapabilityResponse) -> bool {
@@ -890,7 +1006,7 @@ fn build_ns_scanner_url(request: &NsScannerStreamRequest) -> String {
fn build_ns_scanner_capability_url(request: &NsScannerCapabilityRequest, challenge: Uuid) -> String {
format!(
"{}{}?{}={}&{}={}{}{}",
"{}{}?{}={}&{}={}{}{}{}{}{}",
request.endpoint,
NS_SCANNER_PATH,
NS_SCANNER_PROTOCOL_VERSION_QUERY,
@@ -906,6 +1022,21 @@ fn build_ns_scanner_capability_url(request: &NsScannerCapabilityRequest, challen
format!("&{}=true", NS_SCANNER_CHECKPOINT_COVERAGE_QUERY)
} else {
String::new()
},
if request.supports_durable_progress {
format!("&{}=true", NS_SCANNER_DURABLE_PROGRESS_QUERY)
} else {
String::new()
},
if request.supports_complete_observation {
format!("&{}=true", NS_SCANNER_COMPLETE_OBSERVATION_QUERY)
} else {
String::new()
},
if request.supports_usage_watch {
"&ns_scanner_usage_watch=true".to_string()
} else {
String::new()
}
)
}
@@ -1252,6 +1383,9 @@ mod tests {
endpoint: "http://node1:9000".to_string(),
supports_tier_registry_generation: false,
supports_checkpoint_coverage: false,
supports_durable_progress: false,
supports_complete_observation: false,
supports_usage_watch: false,
})
.await
.expect_err("legacy transport should report namespace scanner as unsupported");
@@ -1278,6 +1412,106 @@ mod tests {
assert!(matches!(open_err, Error::MethodNotAllowed));
}
#[tokio::test]
async fn namespace_scanner_watch_capability_preserves_older_capabilities_and_rejects_stale_proof() {
crate::runtime::sources::ensure_test_rpc_secret();
for peer in ["current", "strict-old", "missing-watch", "stale-watch"] {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
.await
.expect("watch capability listener");
let address = listener.local_addr().expect("listener address");
let epoch = Uuid::new_v4();
let server = tokio::spawn(async move {
let requests = if peer == "strict-old" { 2 } else { 1 };
for index in 0..requests {
let (mut socket, _) = listener.accept().await.expect("watch capability request");
let mut bytes = Vec::new();
while !bytes.windows(4).any(|window| window == b"\r\n\r\n") {
let mut chunk = [0; 1024];
let read = socket.read(&mut chunk).await.expect("request headers");
assert!(read > 0 && bytes.len() < 16 * 1024);
bytes.extend_from_slice(&chunk[..read]);
}
let headers = std::str::from_utf8(&bytes).expect("HTTP headers");
let target = headers.split_whitespace().nth(1).expect("target");
let url = reqwest::Url::parse(&format!("http://{address}{target}")).expect("request URL");
let query: HashMap<_, _> = url.query_pairs().into_owned().collect();
assert_eq!(query.contains_key("ns_scanner_usage_watch"), index == 0);
assert!(query.contains_key(NS_SCANNER_DURABLE_PROGRESS_QUERY));
assert!(query.contains_key(NS_SCANNER_COMPLETE_OBSERVATION_QUERY));
if peer == "strict-old" && index == 0 {
socket
.write_all(b"HTTP/1.1 400 Bad Request\r\ncontent-length: 0\r\nconnection: close\r\n\r\n")
.await
.expect("old parser rejects only the new field");
continue;
}
let challenge = query[NS_SCANNER_CAPABILITY_CHALLENGE_QUERY].parse().expect("challenge");
let proof = crate::cluster::rpc::sign_ns_scanner_capability_with_usage_capabilities(
challenge, epoch, true, true, true, true,
)
.expect("prior capability proof");
let usage_watch_proof = match peer {
"current" => Some(
crate::cluster::rpc::sign_ns_scanner_usage_watch_capability(challenge, epoch).expect("watch proof"),
),
"stale-watch" => Some(
crate::cluster::rpc::sign_ns_scanner_usage_watch_capability(Uuid::new_v4(), epoch)
.expect("stale watch proof"),
),
_ => None,
};
let body = rmp_serde::to_vec_named(&NsScannerCapabilityResponse {
version: NS_SCANNER_PROTOCOL_VERSION,
server_epoch: epoch,
proof,
supports_tier_registry_generation: Some(true),
supports_checkpoint_coverage: Some(true),
supports_durable_progress: Some(true),
supports_complete_observation: Some(true),
usage_watch_proof,
})
.expect("capability body");
socket
.write_all(
format!("HTTP/1.1 200 OK\r\ncontent-length: {}\r\nconnection: close\r\n\r\n", body.len()).as_bytes(),
)
.await
.expect("response headers");
socket.write_all(&body).await.expect("response body");
}
requests
});
let response = tokio::time::timeout(
Duration::from_secs(10),
TcpHttpInternodeDataTransport.probe_ns_scanner_capability(NsScannerCapabilityRequest {
endpoint: format!("http://{address}"),
supports_tier_registry_generation: true,
supports_checkpoint_coverage: true,
supports_durable_progress: true,
supports_complete_observation: true,
supports_usage_watch: true,
}),
)
.await
.expect("bounded probe");
if peer == "stale-watch" {
assert!(
response
.expect_err("stale response must fail authentication")
.to_string()
.contains("authentication failed")
);
} else {
let response = response.expect("verified capability result");
assert_eq!(response.usage_watch_proof.is_some(), peer == "current");
assert_eq!(response.supports_durable_progress, Some(true));
assert_eq!(response.supports_complete_observation, Some(true));
}
assert_eq!(server.await.expect("capability server"), if peer == "strict-old" { 2 } else { 1 });
}
}
#[tokio::test]
async fn namespace_scanner_checkpoint_capability_authenticates_new_and_legacy_peers() {
crate::runtime::sources::ensure_test_rpc_secret();
@@ -1330,6 +1564,9 @@ mod tests {
supports_tier_registry_generation: Some(true),
// Strip a supported bit, or forge it for an older peer.
supports_checkpoint_coverage: (peer != "current").then_some(true),
supports_durable_progress: None,
supports_complete_observation: None,
usage_watch_proof: None,
})
.expect("capability response");
socket
@@ -1348,6 +1585,9 @@ mod tests {
endpoint: format!("http://{address}"),
supports_tier_registry_generation: true,
supports_checkpoint_coverage: true,
supports_durable_progress: false,
supports_complete_observation: false,
supports_usage_watch: false,
}),
)
.await
@@ -1369,6 +1609,206 @@ mod tests {
}
}
#[tokio::test]
async fn namespace_scanner_durable_progress_capability_authenticates_new_and_legacy_peers() {
crate::runtime::sources::ensure_test_rpc_secret();
for peer in ["current", "strict-old", "permissive-old", "invalid-proof", "missing-proof"] {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
.await
.expect("capability listener");
let address = listener.local_addr().expect("capability listener address");
let server_epoch = Uuid::new_v4();
let server = tokio::spawn(async move {
let requests = if peer == "strict-old" { 2 } else { 1 };
for index in 0..requests {
let (mut socket, _) = listener.accept().await.expect("capability request");
let mut bytes = Vec::new();
while !bytes.windows(4).any(|window| window == b"\r\n\r\n") {
let mut chunk = [0; 1024];
let read = socket.read(&mut chunk).await.expect("capability headers");
assert!(read > 0 && bytes.len() < 16 * 1024, "bounded complete capability headers");
bytes.extend_from_slice(&chunk[..read]);
}
let request = std::str::from_utf8(&bytes).expect("HTTP headers");
let target = request.split_whitespace().nth(1).expect("HTTP request target");
let url = reqwest::Url::parse(&format!("http://{address}{target}")).expect("capability URL");
let query: HashMap<_, _> = url.query_pairs().into_owned().collect();
assert_eq!(query.get(NS_SCANNER_TIER_REGISTRY_GENERATION_QUERY).map(String::as_str), Some("true"));
assert_eq!(query.contains_key(NS_SCANNER_DURABLE_PROGRESS_QUERY), index == 0);
if peer == "strict-old" && index == 0 {
socket
.write_all(b"HTTP/1.1 400 Bad Request\r\ncontent-length: 0\r\nconnection: close\r\n\r\n")
.await
.expect("legacy parser rejection");
continue;
}
let challenge = query[NS_SCANNER_CAPABILITY_CHALLENGE_QUERY].parse().expect("challenge UUID");
let proof = match peer {
"invalid-proof" => vec![0; 32],
"missing-proof" => Vec::new(),
_ => crate::cluster::rpc::sign_ns_scanner_capability_with_durable_progress(
challenge,
server_epoch,
true,
true,
peer == "current",
)
.expect("server capability proof"),
};
let body = rmp_serde::to_vec_named(&NsScannerCapabilityResponse {
version: NS_SCANNER_PROTOCOL_VERSION,
server_epoch,
proof,
supports_tier_registry_generation: Some(true),
// Strip a supported bit, or forge it for an older peer.
supports_checkpoint_coverage: Some(true),
supports_durable_progress: (peer != "current").then_some(true),
supports_complete_observation: None,
usage_watch_proof: None,
})
.expect("capability response");
socket
.write_all(
format!("HTTP/1.1 200 OK\r\ncontent-length: {}\r\nconnection: close\r\n\r\n", body.len()).as_bytes(),
)
.await
.expect("capability response headers");
socket.write_all(&body).await.expect("capability response body");
}
requests
});
let response = tokio::time::timeout(
Duration::from_secs(10),
TcpHttpInternodeDataTransport.probe_ns_scanner_capability(NsScannerCapabilityRequest {
endpoint: format!("http://{address}"),
supports_tier_registry_generation: true,
supports_checkpoint_coverage: true,
supports_durable_progress: true,
supports_complete_observation: false,
supports_usage_watch: false,
}),
)
.await
.expect("bounded capability probe");
if peer.ends_with("proof") {
assert!(
response
.expect_err("unverified capability must fail closed")
.to_string()
.contains("authentication failed")
);
} else {
let response = response.expect("authenticated capability");
assert_eq!(response.server_epoch, server_epoch);
assert_eq!(response.supports_tier_registry_generation, Some(true));
assert_eq!(response.supports_checkpoint_coverage, Some(true));
assert_eq!(response.supports_durable_progress, (peer == "current").then_some(true));
}
assert_eq!(server.await.expect("capability server"), if peer == "strict-old" { 2 } else { 1 });
}
}
#[tokio::test]
async fn namespace_scanner_complete_observation_capability_authenticates_new_and_legacy_peers() {
crate::runtime::sources::ensure_test_rpc_secret();
for peer in ["current", "strict-old", "permissive-old", "invalid-proof", "missing-proof"] {
let listener = tokio::net::TcpListener::bind("127.0.0.1:0")
.await
.expect("capability listener");
let address = listener.local_addr().expect("capability listener address");
let server_epoch = Uuid::new_v4();
let server = tokio::spawn(async move {
let requests = if peer == "strict-old" { 2 } else { 1 };
for index in 0..requests {
let (mut socket, _) = listener.accept().await.expect("capability request");
let mut bytes = Vec::new();
while !bytes.windows(4).any(|window| window == b"\r\n\r\n") {
let mut chunk = [0; 1024];
let read = socket.read(&mut chunk).await.expect("capability headers");
assert!(read > 0 && bytes.len() < 16 * 1024, "bounded complete capability headers");
bytes.extend_from_slice(&chunk[..read]);
}
let request = std::str::from_utf8(&bytes).expect("HTTP headers");
let target = request.split_whitespace().nth(1).expect("HTTP request target");
let url = reqwest::Url::parse(&format!("http://{address}{target}")).expect("capability URL");
let query: HashMap<_, _> = url.query_pairs().into_owned().collect();
assert_eq!(query.get(NS_SCANNER_TIER_REGISTRY_GENERATION_QUERY).map(String::as_str), Some("true"));
assert_eq!(query.contains_key(NS_SCANNER_COMPLETE_OBSERVATION_QUERY), index == 0);
if peer == "strict-old" && index == 0 {
socket
.write_all(b"HTTP/1.1 400 Bad Request\r\ncontent-length: 0\r\nconnection: close\r\n\r\n")
.await
.expect("legacy parser rejection");
continue;
}
let challenge = query[NS_SCANNER_CAPABILITY_CHALLENGE_QUERY].parse().expect("challenge UUID");
let proof = match peer {
"invalid-proof" => vec![0; 32],
"missing-proof" => Vec::new(),
_ => crate::cluster::rpc::sign_ns_scanner_capability_with_usage_capabilities(
challenge,
server_epoch,
true,
true,
true,
peer == "current",
)
.expect("server capability proof"),
};
let body = rmp_serde::to_vec_named(&NsScannerCapabilityResponse {
version: NS_SCANNER_PROTOCOL_VERSION,
server_epoch,
proof,
supports_tier_registry_generation: Some(true),
// Strip a supported bit, or forge it for an older peer.
supports_checkpoint_coverage: Some(true),
supports_durable_progress: Some(true),
supports_complete_observation: (peer != "current").then_some(true),
usage_watch_proof: None,
})
.expect("capability response");
socket
.write_all(
format!("HTTP/1.1 200 OK\r\ncontent-length: {}\r\nconnection: close\r\n\r\n", body.len()).as_bytes(),
)
.await
.expect("capability response headers");
socket.write_all(&body).await.expect("capability response body");
}
requests
});
let response = tokio::time::timeout(
Duration::from_secs(10),
TcpHttpInternodeDataTransport.probe_ns_scanner_capability(NsScannerCapabilityRequest {
endpoint: format!("http://{address}"),
supports_tier_registry_generation: true,
supports_checkpoint_coverage: true,
supports_durable_progress: true,
supports_complete_observation: true,
supports_usage_watch: false,
}),
)
.await
.expect("bounded capability probe");
if peer.ends_with("proof") {
assert!(
response
.expect_err("unverified capability must fail closed")
.to_string()
.contains("authentication failed")
);
} else {
let response = response.expect("authenticated capability");
assert_eq!(response.server_epoch, server_epoch);
assert_eq!(response.supports_tier_registry_generation, Some(true));
assert_eq!(response.supports_checkpoint_coverage, Some(true));
assert_eq!(response.supports_durable_progress, Some(true));
assert_eq!(response.supports_complete_observation, (peer == "current").then_some(true));
}
assert_eq!(server.await.expect("capability server"), if peer == "strict-old" { 2 } else { 1 });
}
}
#[tokio::test]
async fn walk_dir_restores_explicit_remote_missing_error() {
let _ = rustfs_credentials::set_global_rpc_secret("walk-dir-error-test-secret".to_string());
@@ -2112,6 +2552,9 @@ mod tests {
endpoint: "http://node1:9000".to_string(),
supports_tier_registry_generation: false,
supports_checkpoint_coverage: false,
supports_durable_progress: false,
supports_complete_observation: false,
supports_usage_watch: false,
},
challenge,
);
@@ -2132,6 +2575,9 @@ mod tests {
endpoint: "http://node1:9000".to_string(),
supports_tier_registry_generation: true,
supports_checkpoint_coverage: false,
supports_durable_progress: false,
supports_complete_observation: false,
supports_usage_watch: false,
},
challenge,
);
@@ -2171,13 +2617,16 @@ mod tests {
proof: Vec::new(),
supports_tier_registry_generation: None,
supports_checkpoint_coverage: None,
supports_durable_progress: None,
supports_complete_observation: None,
usage_watch_proof: None,
};
normalize_ns_scanner_capability_response(&mut response, true, true);
normalize_ns_scanner_capability_response(&mut response, true, true, false, false);
assert_eq!(response.supports_tier_registry_generation, Some(true));
response.supports_tier_registry_generation = Some(false);
normalize_ns_scanner_capability_response(&mut response, false, false);
normalize_ns_scanner_capability_response(&mut response, false, false, false, false);
assert_eq!(response.supports_tier_registry_generation, None);
}
@@ -2192,6 +2641,9 @@ mod tests {
.expect("placeholder proof should be generated"),
supports_tier_registry_generation: None,
supports_checkpoint_coverage: None,
supports_durable_progress: None,
supports_complete_observation: None,
usage_watch_proof: None,
};
// A proof bound to a different challenge cannot authorize the legacy
// fallback, even though the response has the expected shape.
+9 -6
View File
@@ -40,12 +40,15 @@ pub use http_auth::{
gen_tonic_signature_headers, normalize_tonic_rpc_audience, rpc_auth_is_resource_exhausted, rpc_background_admission,
set_tonic_canonical_body_digest, set_tonic_mutation_body_digest, set_tonic_rolling_canonical_body_digest,
set_tonic_rolling_mutation_body_digest, sign_ns_scanner_capability, sign_ns_scanner_capability_with_checkpoint_coverage,
sign_ns_scanner_capability_with_tier_registry_generation, sign_put_file_capability, sign_tonic_rpc_response_proof,
tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason, verify_ns_scanner_capability,
verify_ns_scanner_capability_with_checkpoint_coverage, verify_ns_scanner_capability_with_tier_registry_generation,
verify_put_file_auth_trailer, verify_put_file_capability, verify_rpc_signature, verify_tonic_boot_epoch_response,
verify_tonic_canonical_body_digest, verify_tonic_mutation_body_digest, verify_tonic_mutation_body_digest_reject_unsigned,
verify_tonic_rpc_response_proof, verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap,
sign_ns_scanner_capability_with_durable_progress, sign_ns_scanner_capability_with_tier_registry_generation,
sign_ns_scanner_capability_with_usage_capabilities, sign_ns_scanner_usage_watch_capability, sign_put_file_capability,
sign_tonic_rpc_response_proof, tonic_boot_epoch_challenge, tonic_boot_epoch_response_headers, tonic_rpc_auth_failure_reason,
verify_ns_scanner_capability, verify_ns_scanner_capability_with_checkpoint_coverage,
verify_ns_scanner_capability_with_durable_progress, verify_ns_scanner_capability_with_tier_registry_generation,
verify_ns_scanner_capability_with_usage_capabilities, verify_ns_scanner_usage_watch_capability, verify_put_file_auth_trailer,
verify_put_file_capability, verify_rpc_signature, verify_tonic_boot_epoch_response, verify_tonic_canonical_body_digest,
verify_tonic_mutation_body_digest, verify_tonic_mutation_body_digest_reject_unsigned, verify_tonic_rpc_response_proof,
verify_tonic_rpc_signature, verify_tonic_rpc_signature_with_bootstrap,
};
#[cfg(test)]
pub(crate) use internode_data_transport::TcpHttpInternodeDataTransport;
@@ -97,6 +97,7 @@ const PEER_REST_RECOVERY_MAX_ATTEMPTS: u32 = 60;
const PEER_REST_RECOVERY_MAX_BACKOFF: Duration = Duration::from_secs(30);
const SCANNER_ACTIVITY_MAX_MESSAGE_SIZE: usize = 1024;
const SCANNER_SCOPED_DIRTY_USAGE_STAGE_TIMEOUT: Duration = Duration::from_secs(5);
const SCANNER_USAGE_WATCH_PEER_CONCURRENCY: usize = 4;
/// Reserve time for the acquire response's network/clock uncertainty. The
/// server owns the real expiry; this local deadline is intentionally earlier
/// so a coordinator never starts a bounded persistence operation at the edge
@@ -298,6 +299,22 @@ fn scanner_scoped_dirty_usage_ack_payload(
Ok(payload)
}
/// Authentication binds a response to the complete current request before any
/// returned watch can be consumed. Semantic validation still rejects a signed
/// response for the wrong producer session, bucket, or token.
fn validate_scanner_usage_watch_response(
request: &rustfs_protos::proto_gen::node_service::ScannerUsageWatchRequest,
response: &rustfs_protos::proto_gen::node_service::ScannerUsageWatchResponse,
) -> Result<()> {
use rustfs_protos::usage_watch::{
canonical_usage_watch_request, canonical_usage_watch_response, validate_usage_watch_response,
};
let canonical = canonical_usage_watch_request(request).map_err(|error| Error::other(error.to_string()))?;
let body = canonical_usage_watch_response(&canonical, response).map_err(|error| Error::other(error.to_string()))?;
verify_tonic_rpc_response_proof(&body, response.response_proof.as_ref())?;
validate_usage_watch_response(request, response).map_err(|error| Error::other(error.to_string()))
}
fn scanner_scoped_dirty_usage_ack_response_matches(
request: &ScannerScopedDirtyUsageAckRequest,
response: &ScannerScopedDirtyUsageAckResponse,
@@ -741,6 +758,9 @@ pub struct PeerRestClient {
topology_member: String,
offline: Arc<AtomicBool>,
recovery_running: Arc<AtomicBool>,
/// All clones used by admission, keepalive and root verification share
/// this peer's bounded sender capacity.
scanner_usage_watch_admission: Arc<tokio::sync::Semaphore>,
}
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
@@ -900,9 +920,20 @@ impl PeerRestClient {
topology_member,
offline: Arc::new(AtomicBool::new(false)),
recovery_running: Arc::new(AtomicBool::new(false)),
scanner_usage_watch_admission: Arc::new(tokio::sync::Semaphore::new(SCANNER_USAGE_WATCH_PEER_CONCURRENCY)),
}
}
async fn acquire_usage_watch_send_permit(&self) -> Result<tokio::sync::OwnedSemaphorePermit> {
timeout(
SCANNER_SCOPED_DIRTY_USAGE_STAGE_TIMEOUT,
self.scanner_usage_watch_admission.clone().acquire_owned(),
)
.await
.map_err(|_| Error::other("usage watch sender admission timed out"))?
.map_err(|_| Error::other("usage watch sender admission is closed"))
}
fn parse_topology_host(peer_host_port: &str, grid_host: &str) -> Result<XHost> {
let url = url::Url::parse(grid_host).map_err(|_| Error::other("peer grid host is not a valid URL"))?;
if !matches!(url.scheme(), "http" | "https")
@@ -1639,6 +1670,51 @@ impl PeerRestClient {
Err(Error::NotImplemented)
}
/// Routes one authorized MRF operation without retrying ambiguous mutations.
/// `None` means the selected node predates this protocol; it is never an empty result.
pub async fn mrf_lifecycle(
&self,
topology: String,
node_id: String,
command: Vec<u8>,
timeout: std::time::Duration,
) -> Result<Option<Vec<u8>>> {
use rustfs_protos::proto_gen::node_service::MrfLifecycleRequest;
use rustfs_protos::{MRF_LIFECYCLE_MAX_COMMAND_BYTES, MRF_LIFECYCLE_MAX_MESSAGE_BYTES, MRF_LIFECYCLE_PROTOCOL_VERSION};
if command.len() > MRF_LIFECYCLE_MAX_COMMAND_BYTES {
return Err(Error::other("MRF command exceeds size limit"));
}
let payload = MrfLifecycleRequest {
protocol_version: MRF_LIFECYCLE_PROTOCOL_VERSION,
topology_fingerprint: topology,
node_id,
command: command.into(),
challenge: Uuid::new_v4().as_bytes().to_vec().into(),
};
let mut request = Request::new(payload.clone());
request.set_timeout(timeout);
set_tonic_mutation_body_digest(&mut request)?;
let mut client = self
.get_client()
.await?
.max_decoding_message_size(MRF_LIFECYCLE_MAX_MESSAGE_BYTES)
.max_encoding_message_size(MRF_LIFECYCLE_MAX_MESSAGE_BYTES);
let response = match client.mrf_lifecycle(request).await {
Ok(response) => response.into_inner(),
// RUSTFS_COMPAT_TODO(backlog-3177): old peers lack MRF management. Remove after every supported peer implements this RPC.
Err(status) if status.code() == tonic::Code::Unimplemented => return Ok(None),
Err(status) => return Err(status.into()),
};
let canonical = rustfs_protos::canonical_mrf_lifecycle_response_body(&payload, &response)
.map_err(|_| Error::other("MRF response is too large to authenticate"))?;
verify_tonic_rpc_response_proof(&canonical, &response.response_proof)
.map_err(|_| Error::other("MRF response authentication failed"))?;
if response.protocol_version != MRF_LIFECYCLE_PROTOCOL_VERSION {
return Err(Error::other("unsupported MRF response protocol"));
}
Ok(Some(response.result.to_vec()))
}
pub async fn background_heal_status(&self) -> Result<Option<Vec<u8>>> {
self.finalize_result(
async {
@@ -2429,6 +2505,129 @@ impl PeerRestClient {
.await
}
/// Independently negotiated bucket watches; an old peer returns an error
/// and callers retain the global-generation fallback for that bucket.
pub async fn scanner_usage_watch(
&self,
owner_id: Uuid,
mut entries: Vec<crate::runtime::usage_watch::ScannerUsageWatchRequest>,
) -> Result<Vec<crate::runtime::usage_watch::ScannerUsageWatchObservation>> {
use crate::runtime::usage_watch::{
SCANNER_USAGE_WATCH_COVERAGE_VERSION, SCANNER_USAGE_WATCH_LEASE_MS, ScannerUsageWatchKnown,
ScannerUsageWatchObservation,
};
use rustfs_protos::proto_gen::node_service::{ScannerUsageWatchEntry, ScannerUsageWatchRequest};
use rustfs_protos::usage_watch::*;
entries.sort_by(|left, right| left.bucket.cmp(&right.bucket));
if owner_id.is_nil() || entries.windows(2).any(|pair| pair[0].bucket == pair[1].bucket) {
return Err(Error::other("usage watch request contains duplicate buckets or an invalid owner"));
}
// Local queue pressure is not a network failure. Keep admission errors
// outside finalize_result so they cannot mark a healthy peer offline.
let _send_permit = self.acquire_usage_watch_send_permit().await?;
self.finalize_result(
async {
let mut client = super::client::scanner_control_time_out_client(
&self.grid_host,
TonicInterceptor::Signature(gen_tonic_signature_interceptor()),
)
.await?;
let mut observations = Vec::with_capacity(entries.len());
for batch in entries.chunks(USAGE_WATCH_MAX_ENTRIES) {
let payload = ScannerUsageWatchRequest {
challenge: Uuid::new_v4().as_bytes().to_vec().into(),
protocol_version: USAGE_WATCH_PROTOCOL_VERSION,
owner_id: owner_id.to_string(),
entries: batch
.iter()
.map(|entry| ScannerUsageWatchEntry {
bucket: entry.bucket.clone(),
bucket_incarnation: entry.bucket_incarnation.as_bytes().to_vec().into(),
watch_id: entry.watch_id.map_or_else(Bytes::new, |id| id.as_bytes().to_vec().into()),
release: entry.release,
session_id: entry.session_id.map_or_else(Bytes::new, |id| id.as_bytes().to_vec().into()),
})
.collect(),
};
let canonical = canonical_usage_watch_request(&payload).map_err(|error| Error::other(error.to_string()))?;
let mut request = Request::new(payload.clone());
set_tonic_canonical_body_digest(&mut request, &canonical)?;
let started = std::time::Instant::now();
let response = timeout(SCANNER_SCOPED_DIRTY_USAGE_STAGE_TIMEOUT, client.scanner_usage_watch(request))
.await
.map_err(|_| Error::other("usage watch request timed out"))??
.into_inner();
let budget_used = u64::try_from(started.elapsed().as_millis())
.unwrap_or(u64::MAX)
.saturating_add(250);
validate_scanner_usage_watch_response(&payload, &response)?;
let session = Uuid::from_slice(response.session_id.as_ref())
.map_err(|_| Error::other("invalid usage watch session"))?;
if response.protocol_version != USAGE_WATCH_PROTOCOL_VERSION
|| response.owner_id != payload.owner_id
|| session.is_nil()
|| response.entries.len() != payload.entries.len()
{
return Err(Error::other("usage watch response does not match request"));
}
for (requested, entry) in batch.iter().zip(response.entries) {
if entry.bucket != requested.bucket {
return Err(Error::other("usage watch response bucket order differs"));
}
let known = if entry.known {
let token = Uuid::from_slice(entry.watch_id.as_ref())
.map_err(|_| Error::other("invalid usage watch token"))?;
if token.is_nil()
|| entry.bucket_incarnation.as_ref() != requested.bucket_incarnation.as_bytes()
|| requested.watch_id.is_some_and(|expected| expected != token)
|| entry.revision == 0
|| entry.revision == u64::MAX
|| entry.inflight == u64::MAX
|| entry.coverage_version != SCANNER_USAGE_WATCH_COVERAGE_VERSION
|| entry.coverage_epoch == 0
|| entry.lease_remaining_ms == 0
|| entry.lease_remaining_ms > SCANNER_USAGE_WATCH_LEASE_MS
{
return Err(Error::other("usage watch response contains an invalid proof"));
}
let lease_remaining_ms = entry.lease_remaining_ms.saturating_sub(budget_used);
(lease_remaining_ms != 0).then_some(ScannerUsageWatchKnown {
owner_id,
session_id: session,
watch_id: token,
bucket_incarnation: requested.bucket_incarnation,
revision: entry.revision,
inflight: entry.inflight,
coverage_version: entry.coverage_version,
coverage_epoch: entry.coverage_epoch,
lease_remaining_ms,
})
} else {
if !entry.watch_id.is_empty()
|| !entry.bucket_incarnation.is_empty()
|| entry.revision != 0
|| entry.inflight != 0
|| entry.coverage_version != 0
|| entry.coverage_epoch != 0
|| entry.lease_remaining_ms != 0
{
return Err(Error::other("Unknown usage watch entry contains proof fields"));
}
None
};
observations.push(ScannerUsageWatchObservation {
bucket: entry.bucket,
known,
});
}
}
Ok(observations)
}
.await,
)
.await
}
pub async fn acknowledge_scanner_scoped_dirty_usage(
&self,
owner_id: String,
@@ -3263,6 +3462,74 @@ mod tests {
)
}
#[tokio::test]
async fn usage_watch_client_clones_share_receiver_sized_send_admission() {
let receiver = Arc::new(tokio::sync::Semaphore::new(4));
let occupied = (0..4)
.map(|_| receiver.clone().try_acquire_owned().expect("four receiver slots"))
.collect::<Vec<_>>();
assert!(receiver.try_acquire().is_err(), "an uncoordinated fifth request is rejected");
drop(occupied);
let client = test_peer_client();
let active = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let peak = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let tasks = (0..12).map(|_| {
// Simulate the shared client's independent admission, keeper and
// publication callers. Each uses the actual production gate.
let client = client.clone();
let receiver = receiver.clone();
let active = active.clone();
let peak = peak.clone();
tokio::spawn(async move {
let _send = client
.acquire_usage_watch_send_permit()
.await
.expect("bounded sender admission");
let _receive = receiver
.try_acquire_owned()
.expect("coexisting client clones must never send more than the receiver's four slots");
let count = active.fetch_add(1, Ordering::AcqRel) + 1;
peak.fetch_max(count, Ordering::AcqRel);
tokio::time::sleep(Duration::from_millis(10)).await;
active.fetch_sub(1, Ordering::AcqRel);
})
});
for result in futures::future::join_all(tasks).await {
result.expect("all queued callers finish");
}
assert_eq!(peak.load(Ordering::Acquire), 4);
assert_eq!(active.load(Ordering::Acquire), 0);
assert_eq!(client.scanner_usage_watch_admission.available_permits(), 4);
assert_eq!(receiver.available_permits(), 4);
}
#[tokio::test(start_paused = true)]
async fn usage_watch_sender_queue_timeout_does_not_mark_the_peer_offline() {
let client = test_peer_client();
let _occupied = client
.scanner_usage_watch_admission
.clone()
.acquire_many_owned(4)
.await
.unwrap();
let result = client
.scanner_usage_watch(
Uuid::from_u128(1),
vec![crate::runtime::usage_watch::ScannerUsageWatchRequest {
bucket: "bucket".into(),
bucket_incarnation: Uuid::from_u128(2),
watch_id: None,
session_id: None,
release: false,
}],
)
.await;
assert!(result.unwrap_err().to_string().contains("sender admission timed out"));
assert!(!client.offline.load(Ordering::Acquire));
assert!(!client.recovery_running.load(Ordering::Acquire));
}
fn decode_test_scanner_activity(response: ScannerActivityResponse) -> Result<ScannerPeerActivity> {
decode_scanner_activity_with_verifier(response, &[9; 16], |_canonical, proof| {
(proof == b"proof")
@@ -4338,6 +4605,112 @@ mod tests {
assert!(!client.offline.load(Ordering::Acquire));
}
#[test]
fn usage_watch_consumer_authenticates_the_current_request_and_rejects_replay() {
use rustfs_protos::proto_gen::node_service::{
ScannerUsageWatchEntry, ScannerUsageWatchRequest, ScannerUsageWatchResponse, ScannerUsageWatchResult,
};
use rustfs_protos::usage_watch::{canonical_usage_watch_request, canonical_usage_watch_response};
runtime_sources::ensure_test_rpc_secret();
let request = ScannerUsageWatchRequest {
challenge: Uuid::new_v4().as_bytes().to_vec().into(),
protocol_version: 1,
owner_id: Uuid::new_v4().to_string(),
entries: vec![ScannerUsageWatchEntry {
bucket: "usage-watch".into(),
bucket_incarnation: Uuid::new_v4().as_bytes().to_vec().into(),
watch_id: Uuid::new_v4().as_bytes().to_vec().into(),
session_id: Uuid::new_v4().as_bytes().to_vec().into(),
release: false,
}],
};
let sign = |request: &ScannerUsageWatchRequest, response: &mut ScannerUsageWatchResponse| {
let canonical = canonical_usage_watch_request(request).expect("valid request to sign");
let body = canonical_usage_watch_response(&canonical, response).expect("bounded canonical watch response");
response.response_proof = crate::cluster::rpc::sign_tonic_rpc_response_proof(&body)
.expect("real response HMAC")
.into();
};
let mut response = ScannerUsageWatchResponse {
protocol_version: 1,
owner_id: request.owner_id.clone(),
session_id: request.entries[0].session_id.clone(),
entries: vec![ScannerUsageWatchResult {
bucket: request.entries[0].bucket.clone(),
known: true,
bucket_incarnation: request.entries[0].bucket_incarnation.clone(),
watch_id: request.entries[0].watch_id.clone(),
revision: 7,
inflight: 0,
coverage_version: 1,
coverage_epoch: 3,
lease_remaining_ms: 60_000,
}],
response_proof: Bytes::new(),
};
sign(&request, &mut response);
validate_scanner_usage_watch_response(&request, &response).expect("current request accepts its signed response");
let mut rejected_replays = 0;
for field in 0..8 {
let mut current = request.clone();
match field {
0 => current.challenge = Uuid::new_v4().as_bytes().to_vec().into(),
1 => current.owner_id = Uuid::new_v4().to_string(),
2 => current.entries[0].bucket.push('x'),
3 => current.entries[0].bucket_incarnation = Uuid::new_v4().as_bytes().to_vec().into(),
4 => current.entries[0].watch_id = Uuid::new_v4().as_bytes().to_vec().into(),
5 => current.entries[0].session_id = Uuid::new_v4().as_bytes().to_vec().into(),
6 => current.entries[0].release = true,
_ => current.protocol_version += 1,
}
assert!(
validate_scanner_usage_watch_response(&current, &response).is_err(),
"a previously signed response must not authorize request mutation {field}"
);
rejected_replays += 1;
}
let mut tampered = response.clone();
tampered.entries[0].revision += 1;
assert!(validate_scanner_usage_watch_response(&request, &tampered).is_err());
let mut unsigned = response.clone();
unsigned.response_proof = Bytes::new();
assert!(validate_scanner_usage_watch_response(&request, &unsigned).is_err());
let mut rejected_signed_mismatches = 0;
for field in 0..5 {
let mut mismatched = response.clone();
match field {
0 => mismatched.owner_id = Uuid::new_v4().to_string(),
1 => mismatched.session_id = Uuid::new_v4().as_bytes().to_vec().into(),
2 => mismatched.entries[0].bucket_incarnation = Uuid::new_v4().as_bytes().to_vec().into(),
3 => mismatched.entries[0].watch_id = Uuid::new_v4().as_bytes().to_vec().into(),
_ => mismatched.entries[0].known = false,
}
sign(&request, &mut mismatched);
assert!(
validate_scanner_usage_watch_response(&request, &mismatched).is_err(),
"an authentic HMAC cannot replace requested watch identity or authorize malformed Unknown: {field}"
);
rejected_signed_mismatches += 1;
}
println!(
"usage_assertion: {}",
serde_json::json!({
"case_id": "USAGE-12", "assertion_id": "USAGE-12-A07", "status": "PASS",
"evidence": {"current_response_authenticated": true, "replayed_requests_rejected": rejected_replays,
"signed_identity_mismatches_rejected": rejected_signed_mismatches,
"tampered_response_rejected": true, "missing_hmac_rejected": true}
})
);
drop((request, response, tampered, unsigned));
println!(
"usage_cleanup: {}",
serde_json::json!({"case_id": "USAGE-12", "status": "PASS", "scope": "process-local"})
);
}
#[test]
fn heal_control_capability_proof_must_authenticate_exact_ack() {
runtime_sources::ensure_test_rpc_secret();
+57 -2
View File
@@ -1102,7 +1102,38 @@ impl RemoteDisk {
}
}
async fn recover_delete_transaction(
pub(crate) async fn probe_delete_residue_disk(&self, volume: &str) -> Result<Uuid> {
match self
.recover_delete_transaction(volume, DeleteRecoveryRpcOperation::ProbeLegacyDiscard)
.await
{
Ok(DeleteRecoveryRpcReply::LegacyDiscardDisk { disk_id }) if !disk_id.is_nil() => Ok(disk_id),
Ok(_) => Err(Error::FileCorrupt),
Err(Error::Io(source))
if crate::cluster::rpc::client::embedded_tonic_status(&source)
.is_some_and(|status| matches!(status.code(), tonic::Code::Unimplemented | tonic::Code::InvalidArgument)) =>
{
Err(Error::UnsupportedDisk)
}
Err(error) => Err(error),
}
}
pub(crate) async fn list_delete_residue_on_disk(
&self,
volume: &str,
disk_id: Uuid,
cursor: Option<DeleteRecoveryCursor>,
limit: u32,
) -> Result<DeleteRecoveryPage> {
let operation = DeleteRecoveryRpcOperation::ListResidueOnDisk { disk_id, cursor, limit };
match self.recover_delete_transaction(volume, operation).await? {
DeleteRecoveryRpcReply::ListedResidueOnDisk { disk_id: observed, page } if observed == disk_id => Ok(page),
_ => Err(Error::FileCorrupt),
}
}
pub(crate) async fn recover_delete_transaction(
&self,
volume: &str,
operation: DeleteRecoveryRpcOperation,
@@ -1115,6 +1146,7 @@ impl RemoteDisk {
}
_ => DELETE_RECOVERY_PROTOCOL_VERSION,
};
let bounded_inventory_page = matches!(&operation, DeleteRecoveryRpcOperation::ListResidueOnDisk { .. });
let payload = rmp_serde::to_vec_named(&operation)?;
if payload.len() > operation.max_wire_payload_bytes() {
return Err(Error::other("delete recovery operation exceeds payload limit"));
@@ -1142,6 +1174,9 @@ impl RemoteDisk {
} else {
self.get_client().await?
};
if bounded_inventory_page {
client = client.max_decoding_message_size(rustfs_protos::DELETE_RECOVERY_MAX_CONTROL_PAYLOAD_BYTES);
}
let mut request = Request::new(RecoverDeleteTransactionRequest {
disk: self.endpoint.to_string(),
volume: volume.to_owned(),
@@ -1221,6 +1256,15 @@ impl RemoteDisk {
}
pub(crate) async fn ns_scanner_server_epoch(&self) -> Result<Option<Uuid>> {
Ok(self
.ns_scanner_progress_capability()
.await?
.map(|capability| capability.server_epoch))
}
pub(crate) async fn ns_scanner_progress_capability(
&self,
) -> Result<Option<crate::storage_api_contracts::internode::NsScannerCapabilities>> {
if self.health.is_faulty() {
return Err(DiskError::FaultyDisk);
}
@@ -1228,6 +1272,9 @@ impl RemoteDisk {
endpoint: self.endpoint.grid_host(),
supports_tier_registry_generation: true,
supports_checkpoint_coverage: true,
supports_durable_progress: true,
supports_complete_observation: true,
supports_usage_watch: true,
});
let result = timeout(NS_SCANNER_CAPABILITY_PROBE_TIMEOUT, probe)
.await
@@ -1238,7 +1285,12 @@ impl RemoteDisk {
if response.supports_tier_registry_generation == Some(true)
&& response.supports_checkpoint_coverage == Some(true) =>
{
Ok(Some(response.server_epoch))
Ok(Some(crate::storage_api_contracts::internode::NsScannerCapabilities {
server_epoch: response.server_epoch,
supports_durable_progress: response.supports_durable_progress == Some(true),
supports_complete_observation: response.supports_complete_observation == Some(true),
supports_usage_watch: response.usage_watch_proof.is_some(),
}))
}
Ok(_) => Ok(None),
// RUSTFS_COMPAT_TODO(ns-scanner-rpc-v3): old peers and legacy transports lack the authenticated startup-epoch handshake. Remove after every supported peer implements namespace scanner protocol v3.
@@ -7065,6 +7117,9 @@ mod tests {
proof: Vec::new(),
supports_tier_registry_generation,
supports_checkpoint_coverage: *self.ns_scanner_checkpoint_support.lock().expect("checkpoint capability lock"),
supports_durable_progress: None,
supports_complete_observation: None,
usage_watch_proof: None,
})
}
+10
View File
@@ -314,6 +314,16 @@ mod tests {
assert_eq!(scanner_kvs.get(SCANNER_DELAY), "");
assert_eq!(scanner_kvs.get(SCANNER_MAX_WAIT), "");
assert_eq!(scanner_kvs.get(SCANNER_CYCLE_MAX_OBJECTS), "0");
for key in [
rustfs_config::SCANNER_BUCKET_QUANTUM_DURATION,
rustfs_config::SCANNER_BUCKET_QUANTUM_OBJECTS,
rustfs_config::SCANNER_BUCKET_QUANTUM_DIRECTORIES,
rustfs_config::SCANNER_BUCKET_QUANTUM_ENTRIES,
rustfs_config::SCANNER_BUCKET_QUANTUM_NAME_BYTES,
] {
assert!(rustfs_config::SCANNER_KEYS.contains(&key), "admin key {key} is registered");
assert_eq!(scanner_kvs.get(key), "0", "quantum dimension {key} stays disabled until rollout");
}
let heal_kvs = cfg
.get_value(HEAL_SUB_SYS, DEFAULT_DELIMITER)
+31
View File
@@ -25,6 +25,12 @@ use rustfs_config::{
SCANNER_MAX_CONCURRENT_SET_SCANS, SCANNER_MAX_WAIT, SCANNER_RECOVERY_BACKOFF_CAP, SCANNER_SPEED, SCANNER_START_DELAY,
SCANNER_WALKERS_PER_DISK, SCANNER_YIELD_EVERY_N_OBJECTS,
};
use rustfs_config::{
DEFAULT_SCANNER_BUCKET_QUANTUM_DIRECTORIES, DEFAULT_SCANNER_BUCKET_QUANTUM_DURATION_SECS,
DEFAULT_SCANNER_BUCKET_QUANTUM_ENTRIES, DEFAULT_SCANNER_BUCKET_QUANTUM_NAME_BYTES, DEFAULT_SCANNER_BUCKET_QUANTUM_OBJECTS,
SCANNER_BUCKET_QUANTUM_DIRECTORIES, SCANNER_BUCKET_QUANTUM_DURATION, SCANNER_BUCKET_QUANTUM_ENTRIES,
SCANNER_BUCKET_QUANTUM_NAME_BYTES, SCANNER_BUCKET_QUANTUM_OBJECTS,
};
use std::sync::LazyLock;
pub static DEFAULT_KVS: LazyLock<KVS> = LazyLock::new(|| {
@@ -69,6 +75,31 @@ pub static DEFAULT_KVS: LazyLock<KVS> = LazyLock::new(|| {
value: DEFAULT_SCANNER_CYCLE_MAX_DIRECTORIES.to_string(),
hidden_if_empty: false,
},
KV {
key: SCANNER_BUCKET_QUANTUM_DURATION.to_owned(),
value: DEFAULT_SCANNER_BUCKET_QUANTUM_DURATION_SECS.to_string(),
hidden_if_empty: false,
},
KV {
key: SCANNER_BUCKET_QUANTUM_OBJECTS.to_owned(),
value: DEFAULT_SCANNER_BUCKET_QUANTUM_OBJECTS.to_string(),
hidden_if_empty: false,
},
KV {
key: SCANNER_BUCKET_QUANTUM_DIRECTORIES.to_owned(),
value: DEFAULT_SCANNER_BUCKET_QUANTUM_DIRECTORIES.to_string(),
hidden_if_empty: false,
},
KV {
key: SCANNER_BUCKET_QUANTUM_ENTRIES.to_owned(),
value: DEFAULT_SCANNER_BUCKET_QUANTUM_ENTRIES.to_string(),
hidden_if_empty: false,
},
KV {
key: SCANNER_BUCKET_QUANTUM_NAME_BYTES.to_owned(),
value: DEFAULT_SCANNER_BUCKET_QUANTUM_NAME_BYTES.to_string(),
hidden_if_empty: false,
},
KV {
key: SCANNER_BITROT_CYCLE.to_owned(),
value: DEFAULT_SCANNER_BITROT_CYCLE_SECS.to_string(),
+4 -4
View File
@@ -20944,12 +20944,12 @@ impl ECStore {
let locator = parse_decommission_durable_ilm_receipt_path(&receipt_path)?;
let mut proof = None::<DecommissionDurableIlmReceipt>;
let mut nonterminal_receipt_found = false;
for pool_idx in 0..self.pools.len() {
for (pool_idx, pool) in self.pools.iter().enumerate() {
if pool_idx == source_pool_idx {
continue;
}
let data = match read_config_limited_preserve_empty(
self.pools[pool_idx].clone(),
pool.clone(),
&receipt_path,
DECOMMISSION_DURABLE_ILM_RECEIPT_MAX_SIZE,
)
@@ -28412,7 +28412,7 @@ mod tests {
for wrapped in [
err.clone(),
data_movement::data_movement_stage_error_for_test("decommission_object", "put_object", "b", "o", err.clone()),
with_decommission_entry_context("migrate_object", "b", "o", err.clone()),
with_decommission_entry_context("migrate_object", "b", "o", err),
] {
assert_eq!(
classify_decommission_capacity_failure(&wrapped),
@@ -40280,7 +40280,7 @@ mod pools_tests {
)
.expect("finalize records the completion");
let mut adopted = meta.clone();
let mut adopted = meta;
let plan = decommission_capacity_intent_adoption_plan(&adopted, 0, generation)
.expect("plan adoption")
.expect("the interrupted completion should be planned");
+62 -12
View File
@@ -222,33 +222,32 @@ impl Sets {
let mut set_endpoints = Vec::with_capacity(set_drive_count);
for j in 0..set_drive_count {
let idx = i * set_drive_count + j;
let mut disk = disks[idx].clone();
let endpoint = endpoints.endpoints.as_ref()[idx].clone();
set_endpoints.push(endpoint);
if disk.is_none() {
let Some(mut disk) = disks[idx].clone() else {
warn!("sets new set_drive {}-{} is none", i, j);
set_drive.push(None);
continue;
}
};
if disk.as_ref().unwrap().is_local() && instance_ctx.is_dist_erasure().await {
let local_disk = runtime_sources::local_disk_set_drive(&instance_ctx, pool_idx, i, j).await;
if local_disk.is_none() {
if disk.is_local() && instance_ctx.is_dist_erasure().await {
let Some(local_disk) = runtime_sources::local_disk_set_drive(&instance_ctx, pool_idx, i, j).await else {
warn!("sets new set_drive {}-{} local_disk is none", i, j);
set_drive.push(None);
continue;
};
// Format initialization can reuse the registered slot's handle.
if !Arc::ptr_eq(&disk, &local_disk) {
let _ = disk.close().await;
}
let _ = disk.as_ref().unwrap().close().await;
disk = local_disk;
}
let has_disk_id = disk.as_ref().unwrap().get_disk_id().await.unwrap_or_else(|err| {
let has_disk_id = disk.get_disk_id().await.unwrap_or_else(|err| {
if err == DiskError::UnformattedDisk {
error!("get_disk_id err {:?}", err);
} else {
@@ -259,7 +258,7 @@ impl Sets {
});
if let Some(_disk_id) = has_disk_id {
set_drive.push(disk);
set_drive.push(Some(disk));
} else {
error!("sets new set_drive {}-{} get_disk_id is none", i, j);
set_drive.push(None);
@@ -2317,6 +2316,57 @@ mod tests {
(format, pool_endpoints)
}
#[tokio::test]
async fn distributed_sets_only_retire_superseded_local_disk_handles() {
let (_dirs, format, fixture) = setup_heal_format_sets(2, false).await;
let mut pool = fixture.endpoints.clone();
let mut endpoints = pool.endpoints.as_ref().clone();
let mut peer = Endpoint::try_from("http://127.0.0.1:9001/data").expect("peer endpoint");
peer.set_pool_index(0);
peer.set_set_index(0);
peer.set_disk_index(2);
endpoints[2] = peer;
pool.endpoints = endpoints.clone().into();
let ctx = Arc::new(InstanceContext::new());
ctx.update_erasure_type(SetupType::DistErasure).await;
let options = DiskOption::default();
runtime_sources::initialize_local_disk_maps(&ctx, vec![pool.clone()].into(), &options)
.await
.expect("initialize distributed drive slots");
// Slot 0 reuses the registered handle. Removing only the endpoint
// route for slot 1 makes init_disks open a separate bootstrap handle.
ctx.local_disk_map().write().await.remove(&endpoints[1].to_string());
let local_endpoints = Endpoints::from(endpoints[..2].to_vec());
let (mut disks, errors) = crate::store::init_format::init_disks(&ctx, &local_endpoints, &options).await;
assert!(errors.iter().all(Option::is_none), "initialize local disks: {errors:?}");
let registered = runtime_sources::local_disk_set_drive(&ctx, 0, 0, 0)
.await
.expect("registered disk");
let replacement = runtime_sources::local_disk_set_drive(&ctx, 0, 0, 1)
.await
.expect("replacement disk");
let bootstrap = disks[1].as_ref().expect("bootstrap disk");
assert!(Arc::ptr_eq(&registered, disks[0].as_ref().expect("reused disk")));
assert!(!Arc::ptr_eq(&replacement, bootstrap));
let bootstrap_retired = bootstrap.local_retirement_token().expect("local bootstrap token");
disks.push(None); // The remote peer has not started.
let sets = Sets::new_with_instance_ctx(disks, &pool, &format, 0, 1, ctx)
.await
.expect("construct distributed sets");
let inventory = sets.disk_set[0].disk_inventory().await;
for (slot, expected) in [registered, replacement].into_iter().enumerate() {
let active = inventory[slot].as_ref().expect("active local disk");
assert!(Arc::ptr_eq(active, &expected));
assert!(
!active.local_retirement_token().expect("local disk token").is_cancelled(),
"set construction retired the active disk in slot {slot}"
);
}
assert!(bootstrap_retired.is_cancelled(), "superseded bootstrap handle must be retired");
}
// Phase 5 follow-up (backlog#1052): a pool built through the ctx-explicit
// constructor carries the caller's context through Sets AND every SetDisks,
// so nothing in the object graph silently binds to the process bootstrap.
+1 -1
View File
@@ -4242,7 +4242,7 @@ mod tests {
"compatible aliases for {suffix} should match"
);
let mut source_without_marker = source.clone();
let mut source_without_marker = source;
rustfs_utils::http::remove_str(Arc::make_mut(&mut source_without_marker.user_defined), suffix);
let mut target_with_generated_marker = source_without_marker.clone();
rustfs_utils::http::insert_str(
+213 -22
View File
@@ -56,6 +56,8 @@ pub const DATA_USAGE_ROOT: &str = SLASH_SEPARATOR;
const DATA_COMPRESSION_TOTAL_NAME: &str = ".compression.json";
pub const DATA_USAGE_CACHE_NAME: &str = ".usage-cache.bin";
const DATA_USAGE_CACHE_TTL_SECS: u64 = 30;
const ADMIN_USAGE_INCARNATION_CONCURRENCY: usize = 4;
const ADMIN_USAGE_INCARNATION_TIMEOUT: Duration = Duration::from_secs(2);
// A scanner save invalidates only the saving process's snapshot cache; every
// other node sees it only after reloading. A fail-closed usage lookup for a
// bucket the cached snapshot does not mention at all may reload a copy this
@@ -505,6 +507,7 @@ fn stale_data_usage_persist_reason_for_source(
/// Store data usage info to backend storage
#[instrument(skip(store))]
pub async fn store_data_usage_in_backend(data_usage_info: DataUsageInfo, store: Arc<ECStore>) -> Result<(), Error> {
reject_admin_usage_projection(&data_usage_info)?;
if data_usage_info.usage_snapshot_converged == Some(false) {
return Err(Error::other(
"nonconverged data usage observations cannot replace the quota-authoritative snapshot",
@@ -534,6 +537,7 @@ async fn save_data_usage_in_backend(
store: Arc<ECStore>,
expected_publication_epoch: u64,
) -> Result<(), Error> {
reject_admin_usage_projection(&data_usage_info)?;
let data =
serde_json::to_vec(&data_usage_info).map_err(|e| Error::other(format!("Failed to serialize data usage info: {e}")))?;
@@ -663,8 +667,9 @@ fn remove_bucket_usage_from_info(data_usage_info: &mut DataUsageInfo, bucket: &s
let removed_usage = data_usage_info.buckets_usage.remove(bucket).is_some();
let removed_size = data_usage_info.bucket_sizes.remove(bucket).is_some();
let removed_observation = data_usage_info.usage_snapshot_bucket_observations.remove(bucket).is_some();
if !removed_usage && !removed_size {
if !removed_usage && !removed_size && !removed_observation {
return false;
}
@@ -816,7 +821,9 @@ fn map_data_usage_metadata_read_error(err: Error, object: &str) -> Error {
}
fn data_usage_contains_bucket(data_usage_info: &DataUsageInfo, bucket: &str) -> bool {
data_usage_info.buckets_usage.contains_key(bucket) || data_usage_info.bucket_sizes.contains_key(bucket)
data_usage_info.buckets_usage.contains_key(bucket)
|| data_usage_info.bucket_sizes.contains_key(bucket)
|| data_usage_info.usage_snapshot_bucket_observations.contains_key(bucket)
}
fn advance_data_usage_last_update(data_usage_info: &mut DataUsageInfo) {
@@ -1137,14 +1144,26 @@ fn record_usage_snapshot_decode_failure(operation: &'static str, object: &str, e
}
fn parse_usage_snapshot(buf: &[u8]) -> Result<DataUsageInfo, Error> {
serde_json::from_slice(buf).map_err(|e| {
let mut info: DataUsageInfo = serde_json::from_slice(buf).map_err(|e| {
Error::other(format!(
"Failed to deserialize data usage info: {:?} at line {} column {}",
e.classify(),
e.line(),
e.column()
))
})
})?;
reject_admin_usage_projection(&info)?;
for usage in info.buckets_usage.values_mut() {
usage.normalize_usage_provenance();
}
Ok(info)
}
fn reject_admin_usage_projection(info: &DataUsageInfo) -> Result<(), Error> {
if info.usage_snapshot_admin_projection {
return Err(Error::other("admin usage projections cannot be persisted or used as a storage baseline"));
}
Ok(())
}
/// Decide which usage snapshot to serve: a primary usage object or its
@@ -1327,6 +1346,9 @@ async fn load_data_usage_from_backend_with_baseline(store: Arc<ECStore>) -> Resu
/// Fill quota-baseline gaps from an observed (nonconverged) snapshot without
/// overriding any bucket the authoritative baseline already covers.
fn backfill_degraded_baseline_from_observed(degraded_baseline: &mut HashMap<String, u64>, observed: &DataUsageInfo) {
if observed.usage_snapshot_admin_projection {
return;
}
for (bucket, usage) in &observed.buckets_usage {
degraded_baseline.entry(bucket.clone()).or_insert(usage.size);
}
@@ -1407,6 +1429,11 @@ fn select_admin_data_usage_snapshot(
}
fn merge_partial_observation_for_admin(mut authoritative: DataUsageInfo, observed: DataUsageInfo) -> DataUsageInfo {
// The response takes the observation's cycle below. Buckets absent from
// that observation retain their old value and original certificate.
for usage in authoritative.buckets_usage.values_mut() {
usage.mark_carried();
}
for (bucket, usage) in observed.buckets_usage {
authoritative.buckets_usage.insert(bucket, usage);
}
@@ -1419,6 +1446,7 @@ fn merge_partial_observation_for_admin(mut authoritative: DataUsageInfo, observe
authoritative.usage_snapshot_converged = Some(false);
authoritative.usage_snapshot_authoritative_baseline = observed.usage_snapshot_authoritative_baseline;
authoritative.usage_snapshot_set_states = observed.usage_snapshot_set_states;
authoritative.usage_snapshot_bucket_observations = observed.usage_snapshot_bucket_observations;
authoritative.usage_snapshot_bootstrap_pending = false;
authoritative.buckets_count = authoritative.buckets_usage.len() as u64;
authoritative.bucket_sizes = authoritative
@@ -1437,12 +1465,73 @@ async fn load_admin_data_usage_from_backend(store: Arc<ECStore>) -> Result<DataU
let (authoritative, source) = load_data_usage_snapshot(store.clone()).await?;
// For the one-shot admin view a failed observed read degrades to "no
// observation", same as before the read was fallible.
let observed = load_observed_data_usage_snapshot(store).await.ok().flatten();
let observed = load_observed_data_usage_snapshot(store.clone()).await.ok().flatten();
let (selected, selected_is_current_format) =
select_admin_data_usage_snapshot(authoritative, source.is_authoritative(), observed);
Ok(normalize_loaded_data_usage(selected, selected_is_current_format).await.0)
}
async fn project_admin_data_usage_for_current_buckets(
store: Arc<ECStore>,
mut response: DataUsageInfo,
with_memory_overlay: bool,
) -> DataUsageInfo {
use futures::StreamExt as _;
let snapshot_measured = response.is_complete_bucket_usage_snapshot() || response.is_valid_partial_snapshot();
let overlay_authoritative = with_memory_overlay && snapshot_measured && !runtime_sources::setup_is_dist_erasure().await;
let deadline = tokio::time::Instant::now() + ADMIN_USAGE_INCARNATION_TIMEOUT;
let mut incarnations = HashMap::new();
{
// Identity reads happen after releasing snapshot and refresh locks,
// including on cache hits. Metadata transaction reads linearize each
// bucket against creation/deletion without relying on peer notices.
// The common deadline cancels unfinished work and bounds added latency.
let mut buckets = response
.buckets_usage
.keys()
.chain(response.usage_snapshot_bucket_observations.keys())
.cloned()
.collect::<HashSet<_>>();
if overlay_authoritative {
// Copy only names while holding the memory lock, then release it
// before storage I/O. A later new entry has no identity proof and
// will remain unknown in this response.
buckets.extend(memory_cache().read().await.keys().cloned());
}
let mut identities = futures::stream::iter(buckets)
.map(move |bucket| {
let store = store.clone();
async move {
let identity = store.bucket_incarnation_id_from_disk(&bucket).await;
(bucket, identity)
}
})
.buffer_unordered(ADMIN_USAGE_INCARNATION_CONCURRENCY);
while tokio::time::Instant::now() < deadline {
let Ok(Some((bucket, identity))) = tokio::time::timeout_at(deadline, identities.next()).await else {
break;
};
if let Ok(incarnation) = identity
&& !incarnation.is_nil()
{
incarnations.insert(bucket, incarnation.to_string());
}
}
}
response.project_bucket_observations_for_admin(&incarnations);
if apply_bucket_usage_memory_overlay_if_authoritative(&mut response, overlay_authoritative).await {
response.usage_snapshot_complete = false;
response.usage_snapshot_partial = true;
response.usage_snapshot_converged = Some(false);
response.usage_snapshot_admin_projection = true;
response.replication_info.clear();
response.tier_stats = None;
response.unknown_tier_stats = None;
}
response.retain_current_bucket_usage_for_admin(&incarnations);
response
}
fn discard_incomplete_bucket_usage(data_usage_info: &mut DataUsageInfo) {
if !data_usage_info.is_complete_bucket_usage_snapshot() && !data_usage_info.usage_snapshot_partial {
data_usage_info.usage_snapshot_complete = false;
@@ -1512,7 +1601,7 @@ async fn normalize_loaded_data_usage(
&& !cfg.role.is_empty()
{
data_usage_info.replication_info.insert(
cfg.role.clone(),
cfg.role,
BucketTargetUsageInfo {
replication_failed_size: bui.replication_failed_size_v1,
replication_failed_count: bui.replication_failed_count_v1,
@@ -1629,11 +1718,7 @@ async fn project_data_usage_from_backend_cached<T>(
}
}
/// Load the freshest structurally complete snapshot for authenticated admin
/// observability. A scan raced by namespace activity may be selected here, but
/// never by [`load_data_usage_from_backend_cached`], which remains the
/// converged source for quota admission.
pub async fn load_admin_data_usage_from_backend_cached(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
async fn load_admin_data_usage_snapshot_cached(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
let ttl = Duration::from_secs(DATA_USAGE_CACHE_TTL_SECS);
loop {
@@ -1682,6 +1767,38 @@ pub async fn load_admin_data_usage_from_backend_cached(store: Arc<ECStore>) -> R
}
}
/// Load cached scanner statistics for authenticated admin observability.
///
/// Each response verifies bucket identities against current storage metadata,
/// including cache hits. Missing, failed, or timed-out identity reads leave that
/// bucket's usage absent (unknown). Legacy values without an incarnation stay
/// unknown until a scanner snapshot certifies their identity. Per-bucket reads
/// linearize independently; this is not an atomic namespace snapshot.
/// Identity reads have a shared deadline and do not outlive cancellation. The
/// unchanged cached baseline remains independent of quota and persistence.
///
/// # Errors
/// Returns the cached snapshot loading error when no baseline can be loaded.
pub async fn load_admin_data_usage_from_backend_cached(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
let response = load_admin_data_usage_snapshot_cached(store.clone()).await?;
Ok(project_admin_data_usage_for_current_buckets(store, response, false).await)
}
/// Load the admin view with the existing authoritative single-process overlay.
///
/// Only a complete or valid partial scanner snapshot can receive memory values.
/// The final identity check excludes statistics from retired or unverified buckets.
/// Distributed nodes continue to use scanner statistics without local totals.
/// Cancellation, unknown values, and per-bucket read semantics are the same as
/// [`load_admin_data_usage_from_backend_cached`]. No overlay is persisted.
///
/// # Errors
/// Returns the cached snapshot loading error when no baseline can be loaded.
pub async fn load_admin_data_usage_with_memory_overlay(store: Arc<ECStore>) -> Result<DataUsageInfo, Error> {
let response = load_admin_data_usage_snapshot_cached(store.clone()).await?;
Ok(project_admin_data_usage_for_current_buckets(store, response, true).await)
}
/// Invalidate the process-local persisted usage snapshot cache after a durable
/// scanner write.
pub async fn invalidate_data_usage_snapshot_cache() {
@@ -1725,7 +1842,7 @@ fn merge_snapshot(aggregated: &mut DataUsageInfo, mut snapshot: LocalUsageSnapsh
match aggregated.buckets_usage.entry(bucket.clone()) {
Entry::Occupied(mut entry) => entry.get_mut().merge(&usage),
Entry::Vacant(entry) => {
entry.insert(usage.clone());
entry.insert(usage);
}
}
@@ -2082,7 +2199,9 @@ where
fn apply_live_bucket_usage_to_response(data_usage_info: &mut DataUsageInfo, bucket: &str, usage: &BucketUsageInfo) {
let inserted_bucket = !data_usage_info.buckets_usage.contains_key(bucket);
data_usage_info.bucket_sizes.insert(bucket.to_string(), usage.size);
data_usage_info.buckets_usage.insert(bucket.to_string(), usage.clone());
let mut observed = usage.clone();
observed.mark_observed();
data_usage_info.buckets_usage.insert(bucket.to_string(), observed);
if inserted_bucket {
data_usage_info.usage_snapshot_complete = false;
}
@@ -2550,6 +2669,16 @@ pub async fn reload_unknown_bucket_usage(bucket: &str) -> Option<u64> {
}
pub async fn get_bucket_usage_memory(bucket: &str) -> Option<u64> {
cached_bucket_usage_size(bucket, |cached| cached.usage.size.saturating_add(cached.pending_negative_delta)).await
}
/// Read-only statistics show the post-delete value. The local hold on deleted
/// bytes is an admission precaution, not part of the bucket's measured size.
pub(crate) async fn get_bucket_usage_for_statistics(bucket: &str) -> Option<u64> {
cached_bucket_usage_size(bucket, |cached| cached.usage.size).await
}
async fn cached_bucket_usage_size(bucket: &str, local_size: impl FnOnce(&CachedBucketUsage) -> u64) -> Option<u64> {
// A process-local mutation cache is authoritative only when every request
// is handled by this process. In a distributed deployment it contains an
// absolute count derived from one node's requests, so applying it as the
@@ -2562,10 +2691,7 @@ pub async fn get_bucket_usage_memory(bucket: &str) -> Option<u64> {
update_usage_cache_if_needed().await;
let cache = memory_cache().read().await;
cache
.get(bucket)
.filter(|cached| cached.authoritative)
.map(|cached| cached.usage.size.saturating_add(cached.pending_negative_delta))
cache.get(bucket).filter(|cached| cached.authoritative).map(local_size)
}
async fn update_usage_cache_if_needed() {
@@ -2742,16 +2868,17 @@ async fn replace_bucket_usage_memory_from_info_if_generation(data_usage_info: &D
*cache = next_cache;
}
async fn apply_bucket_usage_memory_overlay_if_authoritative(data_usage_info: &mut DataUsageInfo, authoritative: bool) {
async fn apply_bucket_usage_memory_overlay_if_authoritative(data_usage_info: &mut DataUsageInfo, authoritative: bool) -> bool {
if !authoritative {
return;
return false;
}
let cache = memory_cache().read().await;
if cache.is_empty() {
return;
return false;
}
let was_complete = data_usage_info.usage_snapshot_complete;
let persisted_update = data_usage_info.last_update;
let mut changed = false;
let mut added_bucket = false;
@@ -2774,7 +2901,9 @@ async fn apply_bucket_usage_memory_overlay_if_authoritative(data_usage_info: &mu
}
added_bucket |= !data_usage_info.buckets_usage.contains_key(bucket);
data_usage_info.buckets_usage.insert(bucket.clone(), cached.usage.clone());
let mut observed = cached.usage.clone();
observed.mark_observed();
data_usage_info.buckets_usage.insert(bucket.clone(), observed);
data_usage_info.bucket_sizes.insert(bucket.clone(), cached.usage.size);
changed = true;
}
@@ -2786,6 +2915,7 @@ async fn apply_bucket_usage_memory_overlay_if_authoritative(data_usage_info: &mu
data_usage_info.buckets_count = data_usage_info.buckets_usage.len() as u64;
data_usage_info.calculate_totals();
}
changed || was_complete != data_usage_info.usage_snapshot_complete
}
pub async fn apply_bucket_usage_memory_overlay(data_usage_info: &mut DataUsageInfo) {
@@ -3024,6 +3154,7 @@ pub async fn init_compression_total_memory_from_backend(store: Arc<ECStore>) {
#[cfg(test)]
mod tests {
mod companion_projection;
mod snapshot_projection;
use super::*;
@@ -3678,6 +3809,23 @@ mod tests {
*cache_updating().write().await = false;
}
async fn bind_admin_usage_test_identities(store: &Arc<ECStore>, snapshot: &mut DataUsageInfo) {
crate::bucket::metadata_sys::init_bucket_metadata_sys(store.clone(), Vec::new()).await;
for (bucket, usage) in &mut snapshot.buckets_usage {
store
.make_bucket(bucket, &crate::storage_api_contracts::bucket::MakeBucketOptions::default())
.await
.expect("create a real bucket for current admin identity validation");
usage.bucket_incarnation = Some(
store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("read the storage-owned fixture identity")
.to_string(),
);
}
}
fn data_usage_info_for_test(bucket: &str, objects_count: u64, size: u64, last_update: SystemTime) -> DataUsageInfo {
let mut info = DataUsageInfo {
last_update: Some(last_update),
@@ -4019,6 +4167,16 @@ mod tests {
let mut authoritative = data_usage_info_for_test("cold", 152_318, 80 * 1024 * 1024 * 1024, baseline_time);
authoritative.scanner_epoch = Some(4);
authoritative.scanner_cycle = Some(10);
let cold_certificate = rustfs_data_usage::BucketUsageCertification {
epoch: 4,
cycle: 10,
certified_at: baseline_time,
};
authoritative
.buckets_usage
.get_mut("cold")
.expect("cold baseline")
.certify(cold_certificate);
authoritative.buckets_usage.insert(
"hot".to_string(),
BucketUsageInfo {
@@ -4080,6 +4238,7 @@ mod tests {
objects_count: 34,
versions_count: 34,
size: 8 * 1024 * 1024,
usage_value_state: Some(rustfs_data_usage::BucketUsageValueState::Observed),
..Default::default()
},
);
@@ -4095,6 +4254,13 @@ mod tests {
assert!(selected.is_valid_partial_snapshot());
assert_eq!(selected.usage_snapshot_converged, Some(false));
assert_eq!(selected.buckets_count, 2);
assert_eq!(selected.scanner_cycle, Some(11));
let cold = &selected.buckets_usage["cold"];
assert_eq!(cold.usage_value_state, Some(rustfs_data_usage::BucketUsageValueState::Carried));
assert_eq!(cold.usage_certification, Some(cold_certificate));
let hot = &selected.buckets_usage["hot"];
assert_eq!(hot.usage_value_state, Some(rustfs_data_usage::BucketUsageValueState::Observed));
assert_eq!(hot.usage_certification, None);
assert_eq!(
selected
.buckets_usage
@@ -4213,7 +4379,11 @@ mod tests {
use crate::config::com::save_config;
let (_dirs, store) = isolated_store_over_temp_disks().await;
let (authoritative, observed) = usage_snapshots_with_missing_observed_set();
let (mut authoritative, mut observed) = usage_snapshots_with_missing_observed_set();
bind_admin_usage_test_identities(&store, &mut authoritative).await;
for (bucket, usage) in &mut observed.buckets_usage {
usage.bucket_incarnation = authoritative.buckets_usage[bucket].bucket_incarnation.clone();
}
let primary_bytes = serde_json::to_vec(&authoritative).expect("encode complete usage");
let observed_bytes = serde_json::to_vec(&observed).expect("encode partial observation");
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), primary_bytes.clone())
@@ -4262,6 +4432,11 @@ mod tests {
reconciled.scanner_epoch = Some(4);
reconciled.scanner_cycle = Some(12);
reconciled.usage_snapshot_converged = Some(true);
reconciled
.buckets_usage
.get_mut("large-bucket")
.expect("reconciled bucket")
.bucket_incarnation = authoritative.buckets_usage["large-bucket"].bucket_incarnation.clone();
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
@@ -4783,6 +4958,12 @@ mod tests {
backup.scanner_epoch = Some(4);
backup.scanner_cycle = Some(10);
backup.usage_snapshot_converged = Some(true);
bind_admin_usage_test_identities(&store, &mut current).await;
backup
.buckets_usage
.get_mut("existing")
.expect("backup bucket")
.bucket_incarnation = current.buckets_usage["existing"].bucket_incarnation.clone();
for (path, info) in [
(DATA_USAGE_OBJ_NAME_PATH.as_str(), &current),
(DATA_USAGE_OBJ_BACKUP_PATH.as_str(), &backup),
@@ -4805,6 +4986,11 @@ mod tests {
after_delete.scanner_epoch = current.scanner_epoch;
after_delete.scanner_cycle = current.scanner_cycle;
after_delete.usage_snapshot_converged = Some(true);
after_delete
.buckets_usage
.get_mut("existing")
.expect("post-delete bucket")
.bucket_incarnation = current.buckets_usage["existing"].bucket_incarnation.clone();
let mut stale_observation =
data_usage_info_for_test("retired-incarnation", 99, 999, SystemTime::UNIX_EPOCH + Duration::from_secs(4));
stale_observation.scanner_epoch = Some(4);
@@ -4866,6 +5052,7 @@ mod tests {
authoritative.scanner_epoch = Some(7);
authoritative.scanner_cycle = Some(10);
authoritative.usage_snapshot_converged = Some(true);
bind_admin_usage_test_identities(&store, &mut authoritative).await;
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
@@ -4880,6 +5067,10 @@ mod tests {
info.scanner_cycle = Some(cycle);
info.usage_snapshot_converged = Some(false);
info.usage_snapshot_authoritative_baseline = Some(authoritative.snapshot_identity());
info.buckets_usage
.get_mut("bucket")
.expect("observed bucket")
.bucket_incarnation = authoritative.buckets_usage["bucket"].bucket_incarnation.clone();
info
};
let first = observation(7, 11, 200, 11, 1_100);
@@ -0,0 +1,558 @@
// Copyright 2026 RustFS Team
// Licensed under the Apache License, Version 2.0.
use super::*;
use rustfs_data_usage::{
BucketUsageCertification, BucketUsageValueState, DATA_USAGE_BUCKET_OBSERVATION_VERSION, DataUsageBucketObservation,
DataUsageBucketObservationSource, DataUsageSnapshotSetState,
};
mod native_compatibility;
fn companion_snapshot(bucket: &str, incarnation: uuid::Uuid) -> DataUsageInfo {
let completed_at = SystemTime::UNIX_EPOCH + Duration::from_secs(40);
let mut snapshot = data_usage_info_for_test(bucket, 10, 100, completed_at + Duration::from_secs(10));
snapshot.scanner_epoch = Some(2);
snapshot.scanner_cycle = Some(9);
snapshot.usage_snapshot_converged = Some(false);
snapshot.usage_snapshot_pending_buckets = vec![bucket.to_string()];
snapshot.usage_snapshot_pending_bucket_count = 1;
let canonical = snapshot.buckets_usage.get_mut(bucket).expect("canonical bucket");
canonical.bucket_incarnation = Some(incarnation.to_string());
canonical.certify(BucketUsageCertification {
epoch: 2,
cycle: 3,
certified_at: SystemTime::UNIX_EPOCH + Duration::from_secs(30),
});
canonical.mark_carried();
snapshot.usage_snapshot_set_states = vec![DataUsageSnapshotSetState {
pool_index: 0,
set_index: 0,
scanner_epoch: Some(2),
scanner_cycle: Some(9),
scan_plan_digest: Some([1; 32]),
complete: true,
..Default::default()
}];
let mut observed = BucketUsageInfo {
size: 250,
objects_count: 25,
versions_count: 25,
bucket_incarnation: Some(incarnation.to_string()),
..Default::default()
};
observed.mark_observed();
snapshot.usage_snapshot_bucket_observations.insert(
bucket.to_string(),
DataUsageBucketObservation {
version: DATA_USAGE_BUCKET_OBSERVATION_VERSION,
usage: observed,
snapshot_identity: snapshot.snapshot_identity(),
completed_at,
sources: vec![DataUsageBucketObservationSource {
pool_index: 0,
set_index: 0,
bucket_incarnation: incarnation.to_string(),
scanner_epoch: 2,
scanner_cycle: 4,
scan_plan_digest: [1; 32],
started_plan: [2; 32],
requested_plan: [3; 32],
completed_at,
}],
},
);
snapshot
}
#[test]
fn companion_cannot_backfill_quota_and_projection_is_rejected_at_storage_boundaries() {
let incarnation = uuid::Uuid::from_u128(31);
let mut snapshot = companion_snapshot("bucket", incarnation);
let mut canonical_quota = HashMap::new();
backfill_degraded_baseline_from_observed(&mut canonical_quota, &snapshot);
assert_eq!(canonical_quota.get("bucket"), Some(&100), "quota reads only the unchanged main map");
assert_eq!(
snapshot.project_bucket_observations_for_admin(&HashMap::from([("bucket".to_string(), incarnation.to_string())])),
1
);
let mut accidental_quota = HashMap::new();
backfill_degraded_baseline_from_observed(&mut accidental_quota, &snapshot);
assert!(accidental_quota.is_empty());
assert!(reject_admin_usage_projection(&snapshot).is_err());
assert!(certified_bucket_usage(&snapshot, "bucket", incarnation).is_none());
let wire = serde_json::to_vec(&snapshot).expect("admin response remains serializable");
assert!(parse_usage_snapshot(&wire).is_err(), "a serialized response is not a persisted snapshot");
}
#[test]
fn companion_only_observed_snapshot_does_not_create_a_legacy_quota_baseline() {
let incarnation = uuid::Uuid::from_u128(31);
let mut snapshot = companion_snapshot("bucket", incarnation);
snapshot.buckets_usage.clear();
snapshot.bucket_sizes.clear();
snapshot.buckets_count = 0;
snapshot.calculate_totals();
snapshot.usage_snapshot_complete = false;
snapshot.usage_snapshot_partial = true;
snapshot.usage_snapshot_observational = true;
snapshot.usage_snapshot_set_states[0].complete = false;
let decoded = parse_usage_snapshot(&serde_json::to_vec(&snapshot).expect("encode companion-only snapshot"))
.expect("read valid partial snapshot");
assert!(decoded.is_valid_partial_snapshot());
let mut baseline = HashMap::new();
backfill_degraded_baseline_from_observed(&mut baseline, &decoded);
assert!(baseline.is_empty(), "unknown usage must not become either zero or mixed quota usage");
println!(
"usage_assertion: {}",
serde_json::json!({
"case_id": "USAGE-08", "assertion_id": "USAGE-08-A02", "status": "PASS",
"evidence": {"primary_buckets": decoded.buckets_usage.len(), "legacy_bucket_sizes": decoded.bucket_sizes.len(), "quota_baseline_buckets": baseline.len(), "companion_buckets": decoded.usage_snapshot_bucket_observations.len(), "partial": decoded.is_valid_partial_snapshot()}
})
);
println!(
"usage_cleanup: {}",
serde_json::json!({"case_id": "USAGE-08", "status": "PASS", "scope": "process-local"})
);
}
#[test]
fn bucket_removal_discards_companions_even_without_a_canonical_entry() {
let mut snapshot = companion_snapshot("bucket", uuid::Uuid::from_u128(31));
snapshot.buckets_usage.clear();
snapshot.bucket_sizes.clear();
assert!(data_usage_contains_bucket(&snapshot, "bucket"));
assert!(remove_bucket_usage_from_info(&mut snapshot, "bucket"));
assert!(!data_usage_contains_bucket(&snapshot, "bucket"));
assert!(snapshot.usage_snapshot_bucket_observations.is_empty());
}
#[test]
fn partial_selection_replaces_value_provenance_together() {
let incarnation = uuid::Uuid::from_u128(31);
let authoritative = companion_snapshot("bucket", incarnation);
let mut observed = authoritative.clone();
observed.usage_snapshot_authoritative_baseline = Some(authoritative.snapshot_identity());
observed.scanner_cycle = Some(10);
observed.last_update = Some(SystemTime::UNIX_EPOCH + Duration::from_secs(60));
observed.usage_snapshot_complete = false;
observed.usage_snapshot_partial = true;
observed.usage_snapshot_observational = true;
observed.usage_snapshot_bucket_observations.clear();
let usage = observed.buckets_usage.get_mut("bucket").expect("observed main bucket");
usage.size = 175;
usage.mark_observed();
let (selected, _) = select_admin_data_usage_snapshot(authoritative, true, Some(observed));
assert_eq!(selected.buckets_usage["bucket"].size, 175);
assert_eq!(selected.buckets_usage["bucket"].usage_value_state, Some(BucketUsageValueState::Observed));
assert_eq!(selected.buckets_usage["bucket"].usage_certification, None);
assert!(
selected.usage_snapshot_bucket_observations.is_empty(),
"old root companions cannot cross a baseline change"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn companion_admin_cache_isolated_from_quota_memory_and_persistence() {
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::config::com::save_config;
use crate::storage_api_contracts::bucket::MakeBucketOptions;
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = "usage-companion";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create bucket for incarnation validation");
let incarnation = store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("trusted incarnation");
let canonical = companion_snapshot(bucket, incarnation);
let bytes = serde_json::to_vec(&canonical).expect("encode canonical root with companion");
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), bytes.clone())
.await
.expect("save canonical root");
clear_usage_memory_cache_for_test().await;
invalidate_data_usage_snapshot_cache().await;
let mut samples = Vec::new();
let mut rejected_writes = 0;
for _ in 0..2 {
let response = load_admin_data_usage_from_backend_cached(store.clone())
.await
.expect("read admin cache");
assert_eq!(response.buckets_usage[bucket].size, 250);
assert!(response.usage_snapshot_admin_projection);
replace_bucket_usage_memory_from_info(&response).await;
assert!(memory_cache().read().await.is_empty(), "response cannot seed quota memory");
let admin_bytes = response.buckets_usage[bucket].size;
let rejected = store_data_usage_in_backend(response, store.clone()).await.is_err();
assert!(rejected);
rejected_writes += usize::from(rejected);
let quota = load_data_usage_from_backend_cached(store.clone())
.await
.expect("read independent quota cache");
assert_eq!(quota.buckets_usage[bucket].size, 100);
assert!(!quota.usage_snapshot_admin_projection);
samples.push((admin_bytes, quota.buckets_usage[bucket].size));
invalidate_data_usage_snapshot_cache().await;
}
let persisted = read_config(store, DATA_USAGE_OBJ_NAME_PATH.as_str())
.await
.expect("read unchanged canonical root");
assert_eq!(persisted, bytes);
println!(
"usage_assertion: {}",
serde_json::json!({
"case_id": "USAGE-18", "assertion_id": "USAGE-18-A03", "status": "PASS",
"evidence": {"admin_and_quota_byte_samples": samples, "rejected_projected_writes": rejected_writes, "quota_memory_entries": memory_cache().read().await.len(), "persisted_canonical_unchanged": persisted == bytes}
})
);
let paths = _dirs.iter().map(|dir| dir.path().to_path_buf()).collect::<Vec<_>>();
drop(_dirs);
assert!(paths.iter().all(|path| !path.exists()));
println!(
"usage_cleanup: {}",
serde_json::json!({"case_id": "USAGE-18", "status": "PASS", "fixtures_removed": paths.iter().all(|path| !path.exists())})
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn blocked_companion_identity_stays_unknown_without_delaying_healthy_buckets() {
use crate::bucket::metadata_sys::{
acquire_metadata_transaction_guard, init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks,
};
use crate::config::com::save_config;
use crate::storage_api_contracts::bucket::MakeBucketOptions;
// Admin reads also feed spawned metrics collectors across crate boundaries.
fn assert_send<T: Send>(future: T) -> T {
future
}
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let blocked = "usage-blocked-identity";
let healthy = "usage-healthy-identity";
let mut snapshots = Vec::new();
for bucket in [blocked, healthy] {
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create companion identity fixture");
let incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("real incarnation");
snapshots.push(companion_snapshot(bucket, incarnation));
}
let mut canonical = snapshots.remove(0);
let other = snapshots.remove(0);
canonical.buckets_usage.extend(other.buckets_usage);
canonical.bucket_sizes.extend(other.bucket_sizes);
canonical
.usage_snapshot_bucket_observations
.extend(other.usage_snapshot_bucket_observations);
canonical.usage_snapshot_pending_buckets.push(healthy.to_string());
canonical.usage_snapshot_pending_bucket_count = 2;
canonical.buckets_count = 2;
canonical.calculate_totals();
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
serde_json::to_vec(&canonical).expect("encode both canonical values and companions"),
)
.await
.expect("persist usage fixture");
invalidate_data_usage_snapshot_cache().await;
let blocked_guard = acquire_metadata_transaction_guard(
store.clone(),
blocked,
"admin_identity_timeout_test",
rustfs_lock::LockType::Exclusive,
)
.await
.expect("hold the real bucket metadata writer lock");
let response = tokio::time::timeout(
ADMIN_USAGE_INCARNATION_TIMEOUT + Duration::from_secs(3),
assert_send(load_admin_data_usage_from_backend_cached(store.clone())),
)
.await
.expect("optional projection must return before the blocked metadata lock's long timeout")
.expect("canonical admin response stays available");
assert!(
!response.buckets_usage.contains_key(blocked),
"an unavailable current identity cannot bind the canonical value"
);
assert!(!response.bucket_sizes.contains_key(blocked));
assert_eq!(response.buckets_usage[healthy].size, 250);
assert_eq!(response.buckets_usage[healthy].usage_value_state, Some(BucketUsageValueState::Observed));
assert!(response.usage_snapshot_admin_projection);
assert!(
ADMIN_DATA_USAGE_SNAPSHOT_REFRESH
.get()
.expect("admin refresh mutex")
.try_lock()
.is_ok()
);
drop(blocked_guard);
let recovered = load_admin_data_usage_from_backend_cached(store.clone())
.await
.expect("refresh after lock recovery");
assert_eq!(
recovered.buckets_usage[blocked].size, 250,
"unavailable identity can recover on a cache hit"
);
assert_eq!(recovered.buckets_usage[healthy].size, 250);
let quota = load_data_usage_from_backend_cached(store)
.await
.expect("separate canonical quota read");
assert_eq!(quota.buckets_usage[blocked].size, 100);
assert_eq!(quota.buckets_usage[healthy].size, 100);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn cached_canonical_admin_usage_rechecks_current_identity_without_peer_invalidation() {
use crate::bucket::metadata::{load_bucket_incarnation, save_bucket_incarnation};
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::config::com::save_config;
use crate::storage_api_contracts::bucket::MakeBucketOptions;
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = "usage-recreated-cache";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create identity fixture");
let old_identity = store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("original identity");
let mut canonical = companion_snapshot(bucket, old_identity);
canonical.usage_snapshot_bucket_observations.clear();
canonical.usage_snapshot_converged = Some(true);
canonical.usage_snapshot_pending_buckets.clear();
canonical.usage_snapshot_pending_bucket_count = 0;
canonical
.buckets_usage
.get_mut(bucket)
.expect("canonical bucket")
.objects_count = 107;
canonical.calculate_totals();
let original_bytes = serde_json::to_vec(&canonical).expect("encode canonical old bucket");
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), original_bytes.clone())
.await
.expect("persist canonical root");
invalidate_data_usage_snapshot_cache().await;
let first = load_admin_data_usage_from_backend_cached(store.clone())
.await
.expect("prime raw admin cache");
assert_eq!(first.buckets_usage[bucket].objects_count, 107);
assert!(first.is_complete_bucket_usage_snapshot());
let cache_generation = admin_data_usage_snapshot_generation();
let new_identity = uuid::Uuid::new_v4();
// Simulate the storage-owned identity changing on another node while this
// process has not received its invalidation notice. Do not clear the cache.
save_bucket_incarnation(store.clone(), bucket, new_identity)
.await
.expect("publish changed disk identity");
assert_eq!(
load_bucket_incarnation(store.clone(), bucket)
.await
.expect("read new sidecar"),
Some(new_identity)
);
let second = load_admin_data_usage_from_backend_cached(store.clone())
.await
.expect("read raw cache with fresh identity");
assert_eq!(admin_data_usage_snapshot_generation(), cache_generation);
assert!(
!second.buckets_usage.contains_key(bucket),
"same name cannot retain the predecessor's 107"
);
assert!(!second.bucket_sizes.contains_key(bucket));
assert_eq!(second.usage_snapshot_converged, Some(false));
assert!(second.usage_snapshot_admin_projection);
assert!(reject_admin_usage_projection(&second).is_err());
assert!(certified_bucket_usage(&second, bucket, new_identity).is_none());
assert!(parse_usage_snapshot(&serde_json::to_vec(&second).expect("encode display response")).is_err());
let quota = load_data_usage_from_backend_cached(store.clone())
.await
.expect("independent quota baseline");
assert_eq!(quota.buckets_usage[bucket].objects_count, 107);
assert_eq!(
read_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str())
.await
.expect("unchanged persisted root"),
original_bytes
);
// Rebinding a cached raw baseline is transient: recovery does not need to
// reload or permanently blank a healthy bucket's statistics.
save_bucket_incarnation(store.clone(), bucket, old_identity)
.await
.expect("restore fixture identity");
let recovered = load_admin_data_usage_from_backend_cached(store)
.await
.expect("recover on another cache hit");
assert_eq!(recovered.buckets_usage[bucket].objects_count, 107);
assert!(recovered.is_complete_bucket_usage_snapshot());
assert_eq!(admin_data_usage_snapshot_generation(), cache_generation);
invalidate_data_usage_snapshot_cache().await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn admin_memory_overlay_requires_a_measured_scanner_baseline() {
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::config::com::save_config;
use crate::storage_api_contracts::bucket::MakeBucketOptions;
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
assert!(!runtime_sources::setup_is_dist_erasure().await);
let bucket = "usage-unmeasured-overlay";
store.make_bucket(bucket, &MakeBucketOptions::default()).await.unwrap();
let incarnation = store.bucket_incarnation_id_from_disk(bucket).await.unwrap();
let mut baseline = data_usage_info_for_test(bucket, 1, 100, SystemTime::UNIX_EPOCH);
let usage = baseline.buckets_usage.get_mut(bucket).unwrap();
usage.bucket_incarnation = Some(incarnation.to_string());
let memory_usage = BucketUsageInfo {
size: 150,
..usage.clone()
};
clear_usage_memory_cache_for_test().await;
replace_bucket_usage_memory_from_authoritative(bucket, memory_usage, SystemTime::now()).await;
// A missing snapshot and a mismatched bucket count both leave usage unknown.
for (buckets_count, expected_size) in [(None, None), (Some(2), None), (Some(1), Some(150))] {
if let Some(count) = buckets_count {
baseline.buckets_count = count;
save_config(store.clone(), DATA_USAGE_OBJ_NAME_PATH.as_str(), serde_json::to_vec(&baseline).unwrap())
.await
.unwrap();
}
invalidate_data_usage_snapshot_cache().await;
let response = load_admin_data_usage_with_memory_overlay(store.clone()).await.unwrap();
assert_eq!(
response.buckets_usage.get(bucket).map(|usage| usage.size),
expected_size,
"baseline count {buckets_count:?}"
);
assert_eq!(response.bucket_sizes.get(bucket).copied(), expected_size);
assert_eq!(response.objects_total_size, expected_size.unwrap_or_default());
assert_eq!(response.usage_snapshot_admin_projection, expected_size.is_some());
assert!(!response.is_complete_bucket_usage_snapshot());
assert!(!response.is_valid_partial_snapshot());
}
clear_usage_memory_cache_for_test().await;
invalidate_data_usage_snapshot_cache().await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn admin_memory_overlay_cannot_reintroduce_a_predecessor_identity() {
use crate::bucket::metadata::save_bucket_incarnation;
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::config::com::save_config;
use crate::storage_api_contracts::bucket::MakeBucketOptions;
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = "usage-recreated-overlay";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create overlay identity fixture");
let old_identity = store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("original identity");
let mut canonical = companion_snapshot(bucket, old_identity);
canonical.usage_snapshot_bucket_observations.clear();
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
serde_json::to_vec(&canonical).expect("encode canonical root"),
)
.await
.expect("persist overlay fixture");
invalidate_data_usage_snapshot_cache().await;
let cached_usage = canonical.buckets_usage[bucket].clone();
replace_bucket_usage_memory_from_authoritative(bucket, cached_usage, SystemTime::now()).await;
let new_identity = uuid::Uuid::new_v4();
save_bucket_incarnation(store.clone(), bucket, new_identity)
.await
.expect("publish a replacement identity");
let response = load_admin_data_usage_with_memory_overlay(store.clone())
.await
.expect("project after memory overlay");
assert!(!response.buckets_usage.contains_key(bucket));
assert_eq!(response.usage_snapshot_converged, Some(false));
assert!(response.usage_snapshot_admin_projection);
clear_usage_memory_cache_for_test().await;
invalidate_data_usage_snapshot_cache().await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn admin_overlay_flags_only_change_keeps_identity_bound_cold_usage() {
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::config::com::save_config;
use crate::storage_api_contracts::bucket::MakeBucketOptions;
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
assert!(
!runtime_sources::setup_is_dist_erasure().await,
"exercise the production single-process overlay"
);
let cold = "usage-overlay-cold";
let unverified = "usage-overlay-unverified";
for bucket in [cold, unverified] {
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create overlay fixture bucket");
}
let identity = store
.bucket_incarnation_id_from_disk(cold)
.await
.expect("cold bucket identity");
let mut canonical = companion_snapshot(cold, identity);
canonical.usage_snapshot_bucket_observations.clear();
canonical.usage_snapshot_converged = Some(true);
save_config(
store.clone(),
DATA_USAGE_OBJ_NAME_PATH.as_str(),
serde_json::to_vec(&canonical).expect("encode cold canonical value"),
)
.await
.expect("persist cold baseline");
clear_usage_memory_cache_for_test().await;
invalidate_data_usage_snapshot_cache().await;
memory_cache().write().await.insert(
unverified.to_string(),
cached_bucket_usage_from_backend(BucketUsageInfo::default(), SystemTime::UNIX_EPOCH, false),
);
record_bucket_object_write_memory(unverified, None, 42).await;
{
let cache = memory_cache().read().await;
assert!(cache[unverified].dirty);
assert!(!cache[unverified].authoritative);
}
let response = load_admin_data_usage_with_memory_overlay(store)
.await
.expect("identity-filtered flags-only overlay");
assert_eq!(response.buckets_usage[cold].size, 100);
assert_eq!(response.objects_total_size, 100);
assert_eq!(response.bucket_sizes[cold], 100);
assert!(!response.buckets_usage.contains_key(unverified));
assert!(response.usage_snapshot_admin_projection);
assert_eq!(response.usage_snapshot_converged, Some(false));
assert!(!response.is_complete_bucket_usage_snapshot());
assert!(!response.is_valid_partial_snapshot());
clear_usage_memory_cache_for_test().await;
invalidate_data_usage_snapshot_cache().await;
}
@@ -0,0 +1,96 @@
// Copyright 2026 RustFS Team
// SPDX-License-Identifier: Apache-2.0
use super::*;
// Freeze the pre-companion reader's primary fields. Serde's unknown-field
// behavior models the existing JSON reader without importing the new schema.
#[derive(serde::Deserialize)]
struct LegacyPrimarySnapshot {
buckets_usage: HashMap<String, LegacyPrimaryBucket>,
bucket_sizes: HashMap<String, u64>,
objects_total_count: u64,
objects_total_size: u64,
#[serde(default)]
usage_snapshot_complete: bool,
}
#[derive(serde::Deserialize)]
struct LegacyPrimaryBucket {
size: u64,
objects_count: u64,
}
#[test]
fn usage_legacy_reader_never_promotes_companion_values_into_primary_lkg() {
let incarnation = uuid::Uuid::from_u128(31);
let mut snapshot = companion_snapshot("bucket", incarnation);
let original_certificate = snapshot.buckets_usage["bucket"].usage_certification;
let mut decoded_sizes = Vec::new();
for cycle in 10..=12 {
snapshot.scanner_cycle = Some(cycle);
let identity = snapshot.snapshot_identity();
snapshot
.usage_snapshot_bucket_observations
.get_mut("bucket")
.unwrap()
.snapshot_identity = identity;
let bytes = serde_json::to_vec(&snapshot).expect("production persisted snapshot encoding");
let old: LegacyPrimarySnapshot = serde_json::from_slice(&bytes).expect("old reader tolerates additive fields");
assert!(old.usage_snapshot_complete);
assert_eq!(old.buckets_usage["bucket"].size, 100);
assert_eq!(old.buckets_usage["bucket"].objects_count, 10);
assert_eq!(old.bucket_sizes["bucket"], 100);
assert_eq!(old.objects_total_size, 100);
assert_eq!(old.objects_total_count, 10);
let current = parse_usage_snapshot(&bytes).expect("current reader accepts persisted primary plus companion");
assert_eq!(current.buckets_usage["bucket"].usage_certification, original_certificate);
assert_eq!(current.usage_snapshot_bucket_observations["bucket"].usage.size, 250);
decoded_sizes.push(old.buckets_usage["bucket"].size);
}
snapshot.buckets_usage.clear();
snapshot.bucket_sizes.clear();
snapshot.buckets_count = 0;
snapshot.calculate_totals();
snapshot.usage_snapshot_complete = false;
snapshot.usage_snapshot_partial = true;
snapshot.usage_snapshot_observational = true;
snapshot.usage_snapshot_set_states[0].complete = false;
let bytes = serde_json::to_vec(&snapshot).expect("persisted companion-only encoding");
let old: LegacyPrimarySnapshot = serde_json::from_slice(&bytes).expect("old reader handles the absent baseline");
assert!(old.buckets_usage.is_empty() && old.bucket_sizes.is_empty());
assert!(!old.usage_snapshot_complete);
assert_eq!((old.objects_total_count, old.objects_total_size), (0, 0));
let current = parse_usage_snapshot(&bytes).expect("valid current observational snapshot");
assert!(current.is_valid_partial_snapshot());
assert_eq!(current.usage_snapshot_bucket_observations["bucket"].usage.size, 250);
assert_eq!(
snapshot.project_bucket_observations_for_admin(&HashMap::from([("bucket".into(), incarnation.to_string())])),
1
);
let response_bytes = serde_json::to_vec(&snapshot).expect("admin response encoding");
assert!(
parse_usage_snapshot(&response_bytes).is_err(),
"a projected response cannot be written as a baseline"
);
println!(
"usage_assertion: {}",
serde_json::json!({
"case_id": "USAGE-16", "assertion_id": "USAGE-16-A01", "status": "PASS",
"evidence": {
"wire_format": "production JSON, frozen legacy named primary fields",
"legacy_primary_sizes_across_rounds": decoded_sizes,
"new_companion_size": 250,
"companion_only_legacy_primary_count": old.buckets_usage.len(),
"companion_only_legacy_complete": old.usage_snapshot_complete,
"projected_response_rejected_by_persistence_parser": true
}
})
);
println!(
"usage_cleanup: {}",
serde_json::json!({"case_id": "USAGE-16", "status": "PASS", "scope": "process-local"})
);
}
@@ -521,7 +521,7 @@ impl EndpointTopology {
}
};
topology.aliases.entry(host_port.clone()).or_insert(member_index);
topology.aliases.entry(host_port).or_insert(member_index);
topology.aliases.entry(ep.to_string()).or_insert(member_index);
if let Some(display_endpoint) = ep.url.host_str() {
match display_aliases.entry(display_endpoint.to_owned()) {
@@ -1033,16 +1033,26 @@ mod tests {
#[tokio::test]
#[serial]
async fn server_info_reads_complete_observation_without_promoting_authority() {
use crate::bucket::metadata_sys::test_support::isolated_store_over_temp_disks;
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::config::com::{read_config, save_config};
use crate::data_usage::{
DATA_USAGE_OBJ_NAME_PATH, DATA_USAGE_OBSERVED_OBJ_NAME_PATH, invalidate_data_usage_snapshot_cache,
load_data_usage_from_backend_cached,
};
use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
use rustfs_data_usage::{BucketUsageInfo, DataUsageInfo};
use std::time::{Duration, SystemTime};
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
store
.make_bucket("bucket", &MakeBucketOptions::default())
.await
.expect("create a real observed bucket");
let incarnation = store
.bucket_incarnation_id_from_disk("bucket")
.await
.expect("read current storage-owned incarnation");
let primary = DataUsageInfo {
last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1)),
scanner_epoch: Some(1),
@@ -1066,6 +1076,7 @@ mod tests {
versions_count: 4,
delete_markers_count: 1,
size: 6,
bucket_incarnation: Some(incarnation.to_string()),
..Default::default()
},
)]
@@ -1117,6 +1128,46 @@ mod tests {
invalidate_data_usage_snapshot_cache().await;
}
#[test]
fn identity_filtered_usage_remains_unknown_in_server_info() {
let mut info = rustfs_data_usage::DataUsageInfo {
last_update: Some(std::time::SystemTime::UNIX_EPOCH),
buckets_count: 1,
objects_total_count: 107,
objects_total_size: 1024,
buckets_usage: [(
"bucket".to_string(),
rustfs_data_usage::BucketUsageInfo {
objects_count: 107,
size: 1024,
bucket_incarnation: Some("11111111-1111-4111-8111-111111111111".to_string()),
..Default::default()
},
)]
.into_iter()
.collect(),
usage_snapshot_complete: true,
..Default::default()
};
let current =
std::collections::HashMap::from([("bucket".to_string(), "22222222-2222-4222-8222-222222222222".to_string())]);
assert_eq!(info.retain_current_bucket_usage_for_admin(&current), 1);
assert!(info.buckets_usage.is_empty());
assert!(info.usage_snapshot_admin_projection);
assert_eq!(info.usage_snapshot_converged, Some(false));
let mut buckets = rustfs_madmin::Buckets::default();
let mut objects = rustfs_madmin::Objects::default();
let mut versions = rustfs_madmin::Versions::default();
let mut delete_markers = rustfs_madmin::DeleteMarkers::default();
let mut usage = rustfs_madmin::Usage::default();
apply_data_usage_result(Ok(info), &mut buckets, &mut objects, &mut versions, &mut delete_markers, &mut usage);
assert_eq!(buckets.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
assert_eq!(objects.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
assert_eq!(versions.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
assert_eq!(delete_markers.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
assert_eq!(usage.error.as_deref(), Some(DATA_USAGE_UNAVAILABLE_ERROR));
}
#[test]
fn incomplete_data_usage_is_unavailable_in_server_info() {
let mut buckets = rustfs_madmin::Buckets::default();
+29 -1
View File
@@ -1623,6 +1623,8 @@ pub struct LocalDiskWrapper {
health_check: bool,
/// Cancellation token for monitoring tasks
cancel_token: CancellationToken,
/// Retirement belongs to this wrapper instance, not its monitoring policy.
retired: CancellationToken,
/// Disk ID for stale checking
disk_id: Arc<RwLock<Option<Uuid>>>,
/// Timeout policy for scanner-sensitive operations, loaded once on wrapper initialization.
@@ -1776,6 +1778,7 @@ impl LocalDiskWrapper {
physical_admission: Arc::new(crate::disk::io_admission::PhysicalDiskBinding::default()),
health_check: health_check && env_health_check,
cancel_token: CancellationToken::new(),
retired: CancellationToken::new(),
disk_id: Arc::new(RwLock::new(None)),
timeout_health_policy: get_drive_timeout_health_policy(),
};
@@ -1983,6 +1986,31 @@ impl LocalDiskWrapper {
self.cancel_token.cancel();
}
/// Retire a superseded wrapper and stop any monitoring tied to that attachment.
pub(crate) fn retire_handle(&self) {
self.retired.cancel();
self.cancel_token.cancel();
}
/// Check whether the scanner can still use this handle without issuing disk I/O.
///
/// # Errors
/// Returns `DiskNotFound` after retirement or `FaultyDisk` for an offline disk.
pub fn check_scanner_handle(&self) -> Result<()> {
if self.retired.is_cancelled() {
return Err(DiskError::DiskNotFound);
}
if self.health.is_faulty() {
return Err(DiskError::FaultyDisk);
}
Ok(())
}
/// Observe retirement without granting callers authority to retire this wrapper.
pub fn retirement_token(&self) -> CancellationToken {
self.retired.child_token()
}
fn spawn_recovery_monitor_if_needed(&self) {
if !self.health_check {
return;
@@ -2980,7 +3008,7 @@ impl DiskAPI for LocalDiskWrapper {
}
async fn close(&self) -> Result<()> {
self.stop_monitoring().await;
self.retire_handle();
self.disk.close().await
}
+1 -1
View File
@@ -1626,7 +1626,7 @@ mod tests {
for disk_error in disk_errors {
let original_message = disk_error.to_string();
let io_error: std::io::Error = disk_error.clone().into();
let io_error: std::io::Error = disk_error.into();
// The io::Error should contain the original error message
assert!(io_error.to_string().contains(&original_message));
+1
View File
@@ -175,6 +175,7 @@ fn is_windows_dir_access_denied(_path: &Path, _e: &io::Error) -> bool {
false
}
#[cfg(test)]
#[tracing::instrument(level = "debug", skip_all)]
pub async fn remove(path: impl AsRef<Path>) -> io::Result<()> {
let path = path.as_ref().to_path_buf();
+129 -30
View File
@@ -76,11 +76,11 @@ use crate::disk::{
error_conv::{to_access_error, to_file_error, to_unformatted_disk_error, to_volume_error},
format::FormatV3,
fs::{
O_APPEND, O_CREATE, O_RDONLY, O_WRONLY, access, cached_access, invalidate_bucket_cache, lstat, lstat_std, remove,
remove_all_std, remove_std, rename,
O_APPEND, O_CREATE, O_RDONLY, O_WRONLY, access, cached_access, invalidate_bucket_cache, lstat, lstat_std, remove_all_std,
remove_std, rename,
},
is_quota_mutation_fence_path, os,
os::{check_path_length, is_dir_not_empty_error, is_empty_dir, is_root_disk, rename_all, rename_all_ignore_missing_source},
os::{check_path_length, is_dir_not_empty_error, is_empty_dir, is_root_disk, rename_all_ignore_missing_source},
quota_mutation_fence_path,
};
use crate::erasure::coding::{self, bitrot_verify};
@@ -3339,11 +3339,7 @@ pub(crate) fn has_delete_version_test_failpoints() -> bool {
if os::prepared_publication_test_hooks::has_any_hooks() {
return true;
}
#[cfg(feature = "e2e-test-hooks")]
if std::env::var_os("RUSTFS_DELETE_RECOVERY_BARRIER_DIR").is_some() {
return true;
}
false
cfg!(feature = "e2e-test-hooks") && std::env::var_os("RUSTFS_DELETE_RECOVERY_BARRIER_DIR").is_some()
}
#[cfg(test)]
@@ -7587,6 +7583,7 @@ pub struct LocalDisk {
pub root: PathBuf,
publication_root: os::PublicationRoot,
multipart_proof_authority: OnceLock<crate::disk::multipart_source::ProofVerificationContext>,
pub(crate) usage_watch_binding: Arc<super::usage_watch::DiskUsageWatchBinding>,
multipart_index: Arc<multipart_index::IndexOwner>,
/// I/O root pinned to the mount instance that was opened while the disk
/// was initialized. On Linux this is `/proc/self/fd/<dirfd>/.`; resolving
@@ -8137,6 +8134,7 @@ impl LocalDisk {
root: root.clone(),
publication_root,
multipart_proof_authority: OnceLock::new(),
usage_watch_binding: Arc::new(super::usage_watch::DiskUsageWatchBinding::default()),
io_root: io_root.clone(),
#[cfg(target_os = "linux")]
mount_lease: mount_lease.clone(),
@@ -8980,6 +8978,7 @@ impl LocalDisk {
fi: FileInfo,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Result<()> {
let namespace_owner = self.usage_watch_binding.owner(volume, namespace_owner);
if volume == RUSTFS_META_BUCKET {
crate::bucket::metadata_sys::bump_local_meta_mutation_gen();
}
@@ -9189,6 +9188,7 @@ impl LocalDisk {
opts: DeleteOptions,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Vec<Option<Error>> {
let namespace_owner = self.usage_watch_binding.owner(volume, namespace_owner);
let concurrency = delete_version_batch_concurrency(&versions, &opts);
let mut errors = vec![None; versions.len()];
let mut pending = stream::iter(
@@ -9221,6 +9221,7 @@ impl LocalDisk {
opts: DeleteOptions,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Vec<Option<DiskError>> {
let namespace_owner = self.usage_watch_binding.owner(volume, namespace_owner);
let payload = super::GuardedDeleteVersionsPayload {
versions,
opts,
@@ -9256,6 +9257,8 @@ impl LocalDisk {
mut fi: FileInfo,
mutation: DeleteVersionMutation,
) -> Result<()> {
let mut mutation = mutation;
mutation.namespace_owner = self.usage_watch_binding.owner(volume, mutation.namespace_owner.take());
if volume == RUSTFS_META_BUCKET {
crate::bucket::metadata_sys::bump_local_meta_mutation_gen();
}
@@ -9878,7 +9881,12 @@ impl LocalDisk {
.await
.err()
} else {
match rename(&delete_path, &trash_path).await {
let renamed = if namespace_owner.is_some() {
os::rename_with_namespace_owner(delete_path, &trash_path, namespace_owner.clone()).await
} else {
rename(delete_path, &trash_path).await
};
match renamed {
Ok(()) => None,
Err(err)
if err.kind() == ErrorKind::NotFound && os::rename_source_is_missing(delete_path, &self.publication_root) =>
@@ -9930,6 +9938,7 @@ impl LocalDisk {
opt: DeleteOptions,
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Result<()> {
let namespace_owner = self.usage_watch_binding.owner(volume, namespace_owner);
if volume == RUSTFS_META_BUCKET {
crate::bucket::metadata_sys::bump_local_meta_mutation_gen();
}
@@ -10365,6 +10374,7 @@ impl LocalDisk {
Ok(())
}
#[cfg(test)]
async fn delete_file(&self, base_path: &Path, delete_path: &PathBuf, recursive: bool, immediate_purge: bool) -> Result<()> {
self.delete_file_with_namespace_owner(base_path, delete_path, recursive, immediate_purge, None, None)
.await
@@ -11642,6 +11652,15 @@ impl LocalDisk {
// Executes the given SyncMode verbatim: durability policy (tier gating,
// system-critical pinning) is resolved by callers via effective_durability.
async fn write_all_internal(&self, file_path: &Path, data: Bytes, sync: SyncMode, skip_parent: &Path) -> Result<()> {
let relative = file_path
.strip_prefix(self.io_root())
.or_else(|_| file_path.strip_prefix(&self.root));
let bucket = relative
.ok()
.and_then(|path| path.components().next())
.and_then(|component| component.as_os_str().to_str())
.unwrap_or("");
let usage_owner = self.usage_watch_binding.owner(bucket, None);
let skip_parent = if skip_parent.as_os_str().is_empty() {
self.io_root()
} else {
@@ -11660,6 +11679,7 @@ impl LocalDisk {
let root = self.publication_root.clone();
let operation = move |is_cancelled: &dyn Fn() -> bool| {
let _usage_owner = usage_owner;
if is_cancelled() {
return Err(std::io::Error::other("blocking namespace operation cancelled"));
}
@@ -14562,7 +14582,15 @@ impl DiskAPI for LocalDisk {
multipart_staging::reject_native_control_write(&self.root, self.io_root(), src_volume, src_path)?;
multipart_staging::reject_native_control_write(&self.root, self.io_root(), dst_volume, dst_path)?;
multipart_staging::reject_noncanonical_scope(&self.root, self.io_root(), src_volume, src_path)?;
let source_owner = self.multipart_source_owner(dst_volume, dst_path, true, false, None).await?;
let usage_owner = self.usage_watch_binding.owner(src_volume, None);
let usage_owner = if src_volume == dst_volume {
usage_owner
} else {
self.usage_watch_binding.owner(dst_volume, usage_owner)
};
let source_owner = self
.multipart_source_owner(dst_volume, dst_path, true, false, usage_owner)
.await?;
let admission = self.multipart_source_admission(dst_volume, dst_path, true, false)?;
let (src_volume_dir, src_file_path) = self.io_get_bucket_and_object_path(src_volume, src_path)?;
let (dst_volume_dir, dst_file_path) = self.io_get_bucket_and_object_path(dst_volume, dst_path)?;
@@ -14751,7 +14779,8 @@ impl DiskAPI for LocalDisk {
#[tracing::instrument(level = "trace", skip_all)]
async fn settle_part_transaction(&self, volume: &str, path: &str, action: PartTransactionAction) -> Result<()> {
let source_owner = self.multipart_source_owner(volume, path, true, false, None).await?;
let usage_owner = self.usage_watch_binding.owner(volume, None);
let source_owner = self.multipart_source_owner(volume, path, true, false, usage_owner).await?;
let admission = self.multipart_source_admission(volume, path, true, false)?;
let admission_source = admission.as_ref().map(|a| a.source().to_path_buf());
let txn_rel_path = crate::disk::part_transaction_path(path);
@@ -14902,7 +14931,15 @@ impl DiskAPI for LocalDisk {
multipart_staging::reject_native_control_write(&self.root, self.io_root(), src_volume, src_path)?;
multipart_staging::reject_native_control_write(&self.root, self.io_root(), dst_volume, dst_path)?;
multipart_staging::reject_noncanonical_scope(&self.root, self.io_root(), src_volume, src_path)?;
let source_owner = self.multipart_source_owner(dst_volume, dst_path, true, false, None).await?;
let usage_owner = self.usage_watch_binding.owner(src_volume, None);
let usage_owner = if src_volume == dst_volume {
usage_owner
} else {
self.usage_watch_binding.owner(dst_volume, usage_owner)
};
let source_owner = self
.multipart_source_owner(dst_volume, dst_path, true, false, usage_owner)
.await?;
multipart_staging::reject_reserved_anchor(dst_path)?;
let admission = self.multipart_source_admission(dst_volume, dst_path, true, false)?;
let admission_source = admission.as_ref().map(|a| a.source().to_path_buf());
@@ -15214,8 +15251,15 @@ impl DiskAPI for LocalDisk {
}
if !published_transaction_meta && let Some(parent) = src_file_for_cleanup.parent() {
self.delete_file(&src_volume_for_cleanup, &parent.to_path_buf(), false, false)
.await?;
self.delete_file_with_namespace_owner(
&src_volume_for_cleanup,
&parent.to_path_buf(),
false,
false,
source_owner,
None,
)
.await?;
}
Ok(())
@@ -15223,6 +15267,12 @@ impl DiskAPI for LocalDisk {
#[tracing::instrument(level = "trace", skip_all)]
async fn rename_file(&self, src_volume: &str, src_path: &str, dst_volume: &str, dst_path: &str) -> Result<()> {
let usage_owner = self.usage_watch_binding.owner(src_volume, None);
let usage_owner = if src_volume == dst_volume {
usage_owner
} else {
self.usage_watch_binding.owner(dst_volume, usage_owner)
};
multipart_staging::reject_native_control_write(&self.root, self.io_root(), src_volume, src_path)?;
multipart_staging::reject_native_control_write(&self.root, self.io_root(), dst_volume, dst_path)?;
if src_volume == RUSTFS_META_MULTIPART_BUCKET || dst_volume == RUSTFS_META_MULTIPART_BUCKET {
@@ -15280,14 +15330,21 @@ impl DiskAPI for LocalDisk {
// destination is the normal case when renaming a directory to a new
// location, so tolerate NotFound instead of aborting the whole rename
// (MinIO's RenameFile ignores osIsNotExist here).
if let Err(e) = remove(&dst_file_path).await
if let Err(e) = os::remove_with_owner(&dst_file_path, usage_owner.clone()).await
&& e.kind() != ErrorKind::NotFound
{
return Err(to_file_error(e).into());
}
}
rename_all(&src_file_path, &dst_file_path, &dst_volume_dir, &self.publication_root).await?;
os::rename_all_with_owner(
&src_file_path,
&dst_file_path,
&dst_volume_dir,
&self.publication_root,
usage_owner.clone(),
)
.await?;
// Both ends changed identity: the source path no longer exists and the
// destination now resolves to a different inode (backlog#1145).
@@ -15300,7 +15357,9 @@ impl DiskAPI for LocalDisk {
}
if let Some(parent) = src_file_path.parent() {
let _ = self.delete_file(&src_volume_dir, &parent.to_path_buf(), false, false).await;
let _ = self
.delete_file_with_namespace_owner(&src_volume_dir, &parent.to_path_buf(), false, false, usage_owner, None)
.await;
}
Ok(())
@@ -15308,6 +15367,10 @@ impl DiskAPI for LocalDisk {
#[tracing::instrument(level = "trace", skip_all)]
async fn create_file(&self, origvolume: &str, volume: &str, path: &str, _file_size: i64) -> Result<FileWriter> {
// Normal PUT streams into the internal staging volume and is covered
// by rename_data. A raw stream into a live user volume has no retained
// physical completion owner, so it revokes the watch capability.
self.usage_watch_binding.mark_uncovered_user_stream(volume);
multipart_staging::reject_native_control_write(&self.root, self.io_root(), volume, path)?;
multipart_staging::reject_reserved_anchor(path)?;
multipart_staging::reject_noncanonical_scope(&self.root, self.io_root(), volume, path)?;
@@ -15350,6 +15413,12 @@ impl DiskAPI for LocalDisk {
}
async fn rename_file_durable(&self, src_volume: &str, src_path: &str, dst_volume: &str, dst_path: &str) -> Result<()> {
let usage_owner = self.usage_watch_binding.owner(src_volume, None);
let usage_owner = if src_volume == dst_volume {
usage_owner
} else {
self.usage_watch_binding.owner(dst_volume, usage_owner)
};
let source = self.io_get_object_path(src_volume, src_path)?;
let destination = self.io_get_object_path(dst_volume, dst_path)?;
check_path_length(source.to_string_lossy().as_ref())?;
@@ -15360,9 +15429,11 @@ impl DiskAPI for LocalDisk {
return Err(DiskError::FileAccessDenied);
}
if durability.syncs_data_shards() {
let sync_owner = usage_owner.clone();
#[cfg(target_os = "linux")]
let mount_lease = self.mount_lease.clone();
spawn_blocking_with_disk_io_permit(None, move || {
let _usage_owner = sync_owner;
#[cfg(target_os = "linux")]
let _mount_lease = mount_lease;
os::sync_file(&source)
@@ -15373,7 +15444,7 @@ impl DiskAPI for LocalDisk {
}
self.rename_file(src_volume, src_path, dst_volume, dst_path).await?;
if durability.syncs_commit_metadata() {
os::fsync_dir(destination.parent().ok_or(DiskError::InvalidPath)?)
os::fsync_dir_with_owner(destination.parent().ok_or(DiskError::InvalidPath)?, usage_owner.clone())
.await
.map_err(to_file_error)?;
}
@@ -15383,6 +15454,7 @@ impl DiskAPI for LocalDisk {
#[tracing::instrument(level = "trace", skip_all)]
// async fn append_file(&self, volume: &str, path: &str, mut r: DuplexStream) -> Result<File> {
async fn append_file(&self, volume: &str, path: &str) -> Result<FileWriter> {
self.usage_watch_binding.mark_uncovered_user_stream(volume);
multipart_staging::reject_native_control_write(&self.root, self.io_root(), volume, path)?;
multipart_staging::reject_reserved_anchor(path)?;
multipart_staging::reject_noncanonical_scope(&self.root, self.io_root(), volume, path)?;
@@ -15732,6 +15804,7 @@ impl DiskAPI for LocalDisk {
} else {
self.multipart_source_owner(volume, path, true, true, None).await?
};
let owner = self.usage_watch_binding.owner(volume, owner);
self.move_to_trash_with_namespace_owner(&file_path, false, false, owner)
.await?;
@@ -15887,6 +15960,7 @@ impl DiskAPI for LocalDisk {
#[tracing::instrument(level = "trace", skip_all)]
async fn update_metadata(&self, volume: &str, path: &str, fi: FileInfo, opts: &UpdateMetadataOpts) -> Result<()> {
let status_only = super::usage_watch::status_only_metadata_update(&fi, opts);
multipart_staging::reject_native_control_write(&self.root, self.io_root(), volume, path)?;
pool_meta_native::reject_raw_path(&self.root, volume, path)?;
let object_path = self.get_object_path(volume, path)?;
@@ -15899,13 +15973,18 @@ impl DiskAPI for LocalDisk {
condition,
)
.await?;
let owner: Option<Arc<dyn Send + Sync>> = Some(authority.clone());
let owner = self.usage_watch_binding.owner(volume, Some(authority.clone()));
let owner: Option<Arc<dyn Send + Sync>> = Some(os::acquire_metadata_mutation_lease(&object_path, owner).await);
return self
.reconcile_transition_state_metadata(volume, path, fi.version_id, condition, owner, authority)
.await;
}
let source_owner = self.multipart_source_owner(volume, path, false, true, None).await?;
let source_owner = if status_only {
source_owner
} else {
self.usage_watch_binding.owner(volume, source_owner)
};
let namespace_owner: Option<Arc<dyn Send + Sync>> =
Some(os::acquire_metadata_mutation_lease(&object_path, source_owner).await);
if !fi.metadata.is_empty() {
@@ -16293,18 +16372,35 @@ impl DiskAPI for LocalDisk {
// non-recursive rmdir calls. A file that exists during the scan, or
// appears before its parent is removed, fails closed with
// VolumeNotEmpty. Only an explicit force delete removes recursively.
let res = if force_delete {
fs::remove_dir_all(&p).await
} else {
#[cfg(target_os = "linux")]
{
remove_empty_directory_tree_under_mount_lease(&self.mount_lease, volume, p.clone()).await
let usage_owner = self.usage_watch_binding.owner(volume, None);
let physical_path = p.clone();
let physical_volume = volume.to_owned();
let backend = Arc::clone(&self.io_backend);
#[cfg(target_os = "linux")]
let mount_lease = Arc::clone(&self.mount_lease);
let res = tokio::spawn(async move {
let _usage_owner = usage_owner;
let _volume_mutation_guard = _volume_mutation_guard;
let result = if force_delete {
fs::remove_dir_all(&physical_path).await
} else {
#[cfg(target_os = "linux")]
{
remove_empty_directory_tree_under_mount_lease(&mount_lease, &physical_volume, physical_path.clone()).await
}
#[cfg(not(target_os = "linux"))]
{
remove_empty_directory_tree(&physical_path).await
}
};
if result.is_ok() {
backend.invalidate_cached_fds_for_volume(&physical_volume);
invalidate_bucket_cache(&physical_path);
}
#[cfg(not(target_os = "linux"))]
{
remove_empty_directory_tree(&p).await
}
};
result
})
.await
.map_err(DiskError::from)?;
if let Err(err) = res {
let e: DiskError = classify_delete_volume_error(err);
@@ -16484,6 +16580,9 @@ async fn get_disk_info(drive_path: PathBuf) -> Result<(rustfs_utils::os::DiskInf
Ok((disk_info, root_drive))
}
#[cfg(all(test, not(windows)))]
mod usage_watch_tests;
#[cfg(test)]
mod test {
use super::commit::create_local_inline_rollback_backup;
+4
View File
@@ -500,6 +500,10 @@ impl LocalDisk {
dst_path: &str,
state: &mut RenameDataState,
) -> Result<RenameDataResp> {
state.namespace_owner = self.usage_watch_binding.owner(dst_volume, state.namespace_owner.take());
if src_volume != dst_volume {
state.namespace_owner = self.usage_watch_binding.owner(src_volume, state.namespace_owner.take());
}
super::multipart_staging::reject_noncanonical_scope(&self.root, self.io_root(), src_volume, src_path).inspect_err(
|_| {
state.preflight_rejection = Some(LocalRenamePreflightRejection(()));
@@ -1777,6 +1777,28 @@ impl LocalDisk {
})
}
/// Read the format through the owning disk's pinned mount, bypassing cached IDs.
pub(crate) async fn delete_residue_disk_id(&self) -> Result<Uuid> {
let bytes = self.read_format_json().await?;
let format = FormatV3::try_from(bytes.as_slice()).map_err(|_| DiskError::CorruptedBackend)?;
let id = format.erasure.this;
let (set, slot) = format.find_disk_index_by_disk_id(id)?;
if id.is_nil()
|| usize::try_from(self.endpoint.set_idx).ok() != Some(set)
|| usize::try_from(self.endpoint.disk_idx).ok() != Some(slot)
{
return Err(DiskError::InconsistentDisk);
}
Ok(id)
}
pub(crate) async fn require_delete_residue_disk_id(&self, expected: Uuid) -> Result<()> {
if expected.is_nil() || self.delete_residue_disk_id().await? != expected {
return Err(DiskError::OutdatedXLMeta);
}
Ok(())
}
pub(crate) async fn delete_recovery_settle(
&self,
volume: &str,
@@ -1827,6 +1849,7 @@ impl LocalDisk {
action: DeleteRecoveryAction,
owner: Option<Arc<dyn Send + Sync>>,
) -> Result<DeleteRecoveryInspection> {
let owner = self.usage_watch_binding.owner(volume, owner);
self.reject_multipart_delete_recovery_mutation(volume, object)?;
if let Ok(root) = self.io_get_object_path(volume, object) {
let _ = take_prepared_delete_receipt(&root.join(transaction.to_string()));
@@ -1966,7 +1989,13 @@ impl LocalDisk {
return Ok(observed);
}
}
if action == DeleteRecoveryAction::DiscardLegacy {
if matches!(
action,
DeleteRecoveryAction::DiscardLegacy | DeleteRecoveryAction::DiscardLegacyOnDisk { .. }
) {
if let DeleteRecoveryAction::DiscardLegacyOnDisk { disk_id } = action {
self.require_delete_residue_disk_id(disk_id).await?;
}
if observed.state != DeleteRecoveryState::LegacyAmbiguous || !observed.explicit_delete_safe {
return Err(DiskError::FileCorrupt);
}
@@ -2415,7 +2444,9 @@ impl LocalDisk {
}
}
DeleteRecoveryAction::AdoptCommit { .. } => return Err(DiskError::MethodNotAllowed),
DeleteRecoveryAction::DiscardLegacy => return Err(DiskError::MethodNotAllowed),
DeleteRecoveryAction::DiscardLegacy | DeleteRecoveryAction::DiscardLegacyOnDisk { .. } => {
return Err(DiskError::MethodNotAllowed);
}
}
self.delete_recovery_inspect_owned(volume, object, transaction, owner).await
}
@@ -2779,6 +2810,7 @@ impl LocalDisk {
namespace_owner: Option<Arc<dyn Send + Sync>>,
work_budget: Option<Arc<crate::object_api::DeleteRecoveryWorkBudget>>,
) -> Result<Option<(DeleteRecoveryJournal, bool, bool)>> {
let namespace_owner = self.usage_watch_binding.owner(volume, namespace_owner);
if transaction.is_nil() {
return Err(DiskError::InvalidPath);
}
@@ -2974,6 +3006,7 @@ impl LocalDisk {
namespace_owner: Option<Arc<dyn Send + Sync>>,
work_budget: Option<Arc<crate::object_api::DeleteRecoveryWorkBudget>>,
) -> Result<bool> {
let namespace_owner = self.usage_watch_binding.owner(volume, namespace_owner);
fn ignore_not_found(res: std::io::Result<()>) -> std::io::Result<()> {
match res {
Ok(()) => Ok(()),
@@ -0,0 +1,234 @@
// Copyright 2026 RustFS Team
// Licensed under the Apache License, Version 2.0.
use super::*;
use crate::runtime::usage_watch::{ScannerUsageWatchRequest, UsageWatchRegistry};
fn watch_request(bucket: &str) -> ScannerUsageWatchRequest {
ScannerUsageWatchRequest {
bucket: bucket.into(),
bucket_incarnation: Uuid::from_u128(2),
watch_id: None,
release: false,
session_id: None,
}
}
async fn fixture() -> (tempfile::TempDir, Arc<LocalDisk>, Arc<UsageWatchRegistry>) {
let root = tempfile::tempdir().expect("watch physical fixture directory");
let endpoint = Endpoint::try_from(root.path().to_string_lossy().as_ref()).expect("watch physical endpoint");
let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("watch physical disk"));
disk.make_volume("bucket").await.expect("watch test bucket");
let registry = Arc::new(UsageWatchRegistry::default());
disk.usage_watch_binding.bind(&registry);
(root, disk, registry)
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn usage_watch_stays_inflight_after_metadata_is_visible_and_waiter_is_canceled() {
let (_root, disk, registry) = fixture().await;
let target = disk
.io_get_object_path("bucket", "object")
.expect("target metadata directory");
let (entered, wait_for_enter) = tokio::sync::oneshot::channel();
let (release, hold) = std::sync::mpsc::channel::<()>();
let _hook = os::prepared_publication_test_hooks::install_at(
os::prepared_publication_test_hooks::Stage::DirFsync,
&target,
move || {
let _ = entered.send(());
let _ = hold.recv();
},
);
let writer_disk = Arc::clone(&disk);
let mut writer = tokio::spawn(async move {
let info =
super::test::test_file_info("object", Uuid::new_v4(), Some(Uuid::new_v4()), Some(Bytes::from_static(b"payload")));
writer_disk.write_metadata("", "bucket", "object", info).await
});
tokio::time::timeout(Duration::from_secs(10), wait_for_enter)
.await
.expect("reach physical fsync barrier")
.expect("barrier entered");
assert!(
target.join(STORAGE_FORMAT_FILE).exists(),
"metadata is already visible at the blocked fsync"
);
let during = registry
.observe(Uuid::from_u128(1), &watch_request("bucket"))
.expect("install during active physical mutation");
assert!(during.inflight > 0, "installation must include pre-existing writes");
writer.abort();
assert!((&mut writer).await.expect_err("cancel logical waiter").is_cancelled());
let canceled = registry
.observe(Uuid::from_u128(1), &watch_request("bucket"))
.expect("watch survives logical cancellation");
assert!(canceled.inflight > 0, "physical tail retains its guard");
drop(release);
let settled = tokio::time::timeout(Duration::from_secs(10), async {
loop {
let observation = registry
.observe(Uuid::from_u128(1), &watch_request("bucket"))
.expect("settlement proof");
if observation.inflight == 0 {
break observation;
}
tokio::task::yield_now().await;
}
})
.await
.expect("physical tail settles");
assert!(settled.revision > during.revision);
assert!(!settled.unchanged_since(&during));
println!(
"usage_assertion: {}",
serde_json::json!({
"case_id": "USAGE-12", "assertion_id": "USAGE-12-A03", "status": "PASS",
"evidence": {"inflight_at_install": during.inflight, "inflight_after_waiter_cancel": canceled.inflight, "inflight_after_physical_end": settled.inflight, "revision_at_install": during.revision, "revision_after_physical_end": settled.revision}
})
);
drop(disk);
drop(registry);
let root_path = _root.path().to_path_buf();
drop(_root);
assert!(!root_path.exists());
println!(
"usage_cleanup: {}",
serde_json::json!({"case_id": "USAGE-12", "status": "PASS", "fixture_removed": !root_path.exists()})
);
}
#[tokio::test]
async fn usage_watch_status_only_metadata_keeps_maintenance_semantics() {
let (_root, disk, registry) = fixture().await;
let info = super::test::test_file_info("object", Uuid::new_v4(), Some(Uuid::new_v4()), Some(Bytes::from_static(b"payload")));
disk.write_metadata("", "bucket", "object", info.clone())
.await
.expect("initial counted version");
let before = registry
.observe(Uuid::from_u128(1), &watch_request("bucket"))
.expect("idle watch");
let mut status = info.clone();
status.metadata.clear();
rustfs_utils::http::metadata_compat::insert_str(
&mut status.metadata,
rustfs_utils::http::SUFFIX_REPLICATION_STATUS,
"arn=COMPLETED;".into(),
);
disk.update_metadata("bucket", "object", status, &UpdateMetadataOpts::default())
.await
.expect("status-class metadata update");
let after = registry
.observe(Uuid::from_u128(1), &watch_request("bucket"))
.expect("status result watch");
assert!(after.unchanged_since(&before), "replication counters retain maintenance cadence");
let mut next = info;
next.version_id = Some(Uuid::new_v4());
disk.write_metadata("", "bucket", "object", next).await.expect("new version");
let changed = registry
.observe(Uuid::from_u128(1), &watch_request("bucket"))
.expect("new version watch");
assert!(changed.revision > after.revision);
assert_eq!(changed.inflight, 0);
}
#[tokio::test]
async fn usage_watch_raw_live_stream_revokes_coverage_and_internal_staging_does_not() {
let (_root, disk, registry) = fixture().await;
let before = registry
.observe(Uuid::from_u128(1), &watch_request("bucket"))
.expect("idle watch");
disk.usage_watch_binding.mark_uncovered_user_stream(RUSTFS_META_TMP_BUCKET);
assert!(
registry
.observe(Uuid::from_u128(1), &watch_request("bucket"))
.expect("internal staging stays covered")
.unchanged_since(&before)
);
let _stream = disk.create_file("bucket", "bucket", "raw-file", 0).await;
assert_eq!(registry.observe(Uuid::from_u128(1), &watch_request("bucket")), None);
assert!(!disk.usage_watch_binding.matches(&registry));
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn usage_watch_delete_paths_retains_ownership_through_canceled_trash_rename() {
let (_root, disk, registry) = fixture().await;
let target = disk.io_get_object_path("bucket", "delete-me").expect("delete source");
std::fs::write(&target, b"payload").expect("initial source");
let trash = disk.io_get_bucket_path(RUSTFS_META_TMP_DELETED_BUCKET).expect("trash volume");
std::fs::create_dir_all(&trash).expect("trash directory");
let before = registry
.observe(Uuid::from_u128(1), &watch_request("bucket"))
.expect("initial proof");
let (entered, wait_for_enter) = tokio::sync::oneshot::channel();
let (release, hold) = std::sync::mpsc::channel::<()>();
let _hook =
os::prepared_publication_test_hooks::install_at(os::prepared_publication_test_hooks::Stage::Rename, &target, move || {
let _ = entered.send(());
let _ = hold.recv();
});
let deleting_disk = Arc::clone(&disk);
let mut deleting = tokio::spawn(async move { deleting_disk.delete_paths("bucket", &["delete-me".into()]).await });
tokio::time::timeout(Duration::from_secs(10), wait_for_enter)
.await
.expect("reach raw trash rename")
.expect("rename entered");
deleting.abort();
assert!((&mut deleting).await.expect_err("logical waiter canceled").is_cancelled());
let during = registry
.observe(Uuid::from_u128(1), &watch_request("bucket"))
.expect("active proof");
assert!(during.inflight > 0, "trash syscall owns the watch after waiter cancellation");
assert!(during.revision > before.revision);
drop(release);
let after = tokio::time::timeout(Duration::from_secs(10), async {
loop {
let observation = registry
.observe(Uuid::from_u128(1), &watch_request("bucket"))
.expect("draining proof");
if observation.inflight == 0 {
break observation;
}
tokio::task::yield_now().await;
}
})
.await
.expect("physical rename settles");
assert!(!target.exists());
assert!(after.revision > during.revision);
assert!(!after.unchanged_since(&before));
}
#[tokio::test]
async fn owned_stale_destination_removal_preserves_file_directory_and_symlink_semantics() {
let (_root, disk, registry) = fixture().await;
let target = disk.io_get_object_path("bucket", "stale").expect("stale destination");
std::fs::write(&target, b"stale-file").expect("stale file");
os::remove_with_owner(&target, disk.usage_watch_binding.owner("bucket", None))
.await
.expect("remove stale file");
assert!(!target.exists());
std::fs::create_dir(&target).expect("stale directory");
os::remove_with_owner(&target, disk.usage_watch_binding.owner("bucket", None))
.await
.expect("remove empty directory");
assert!(!target.exists());
#[cfg(unix)]
{
let referent = disk.io_get_object_path("bucket", "retained").expect("symlink referent");
std::fs::write(&referent, b"retained").expect("retained file");
std::os::unix::fs::symlink(&referent, &target).expect("stale symlink");
os::remove_with_owner(&target, disk.usage_watch_binding.owner("bucket", None))
.await
.expect("remove symlink itself");
assert!(!target.exists());
assert_eq!(std::fs::read(referent).expect("referent remains"), b"retained");
}
assert_eq!(
registry
.observe(Uuid::from_u128(1), &watch_request("bucket"))
.expect("all work drained")
.inflight,
0
);
}
+107
View File
@@ -30,6 +30,7 @@ pub mod local;
pub mod multipart_index;
pub mod multipart_source;
pub mod os;
mod usage_watch;
pub(crate) use io_admission::memory as background_memory;
pub(crate) use io_admission::remote as remote_io_admission;
@@ -103,6 +104,7 @@ use sha2::Digest as _;
use std::{fmt::Debug, path::PathBuf, sync::Arc, time::Duration};
use time::OffsetDateTime;
use tokio::io::{AsyncRead, AsyncWrite};
use tokio_util::sync::CancellationToken;
use uuid::Uuid;
/// Independent admission and physical ownership for one disk rename.
@@ -324,6 +326,76 @@ impl Disk {
}
}
/// Identify the physical participant that supports identity-bound legacy discard.
pub async fn probe_delete_residue_disk(&self, volume: &str) -> Result<Uuid> {
match self {
Self::Local(disk) => disk.get_disk().delete_residue_disk_id().await,
Self::Remote(disk) => disk.probe_delete_residue_disk(volume).await,
}
}
/// Enumerate one page on the same physical participant verified before and after traversal.
pub async fn list_delete_residue_on_disk(
&self,
volume: &str,
expected: Uuid,
cursor: Option<DeleteRecoveryCursor>,
limit: u32,
) -> Result<DeleteRecoveryPage> {
if expected.is_nil() || !(1..=MAX_DELETE_RESIDUE_PAGE_SIZE).contains(&limit) {
return Err(Error::InvalidPath);
}
match self {
Self::Local(disk) => {
let local = disk.get_disk();
local.require_delete_residue_disk_id(expected).await?;
let page = disk.discover_delete_transactions(volume, None, cursor, limit).await?;
local.require_delete_residue_disk_id(expected).await?;
Ok(page)
}
Self::Remote(disk) => disk.list_delete_residue_on_disk(volume, expected, cursor, limit).await,
}
}
/// Bound the physical disk identity to the same owner that performs a single
/// bounded discovery or inspection. An old peer cannot substitute an unbound reply.
pub async fn read_delete_residue_on_disk(
&self,
volume: &str,
object: &str,
disk_id: Uuid,
transaction: Option<Uuid>,
max_entries: u32,
) -> Result<DeleteRecoveryRpcReply> {
match self {
Self::Local(disk) => {
disk.get_disk().require_delete_residue_disk_id(disk_id).await?;
match transaction {
Some(id) => disk
.inspect_delete_transaction_bounded(volume, object, id, max_entries)
.await
.map(DeleteRecoveryRpcReply::InspectedBounded),
None => disk
.discover_delete_transactions_bounded(volume, object, max_entries)
.await
.map(DeleteRecoveryRpcReply::DiscoveredBounded),
}
}
Self::Remote(disk) => {
disk.recover_delete_transaction(
volume,
DeleteRecoveryRpcOperation::ReadResidueOnDisk {
object: object.to_owned(),
disk_id,
transaction,
max_entries,
},
)
.await
}
}
}
/// A read-only probe; the physical delete repeats the same leaf checks.
pub async fn is_plain_orphan_data_dir(&self, volume: &str, path: &str) -> Result<bool> {
match self {
@@ -1386,6 +1458,16 @@ impl Disk {
}
impl Disk {
/// The progress flag is derived from a verified capability scope.
pub async fn ns_scanner_progress_capability(
&self,
) -> Result<Option<crate::storage_api_contracts::internode::NsScannerCapabilities>> {
match self {
Disk::Local(_) => Ok(None),
Disk::Remote(remote_disk) => remote_disk.ns_scanner_progress_capability().await,
}
}
pub async fn ns_scanner_server_epoch(&self) -> Result<Option<Uuid>> {
match self {
Disk::Local(_) => Ok(None),
@@ -1407,6 +1489,15 @@ impl Disk {
}
}
/// Observe a local wrapper's retirement. Remote handles have their own RPC lifecycle.
/// Cancelling the returned child token does not retire the underlying wrapper.
pub fn local_retirement_token(&self) -> Option<CancellationToken> {
match self {
Disk::Local(local_disk) => Some(local_disk.retirement_token()),
Disk::Remote(_) => None,
}
}
pub fn offline_duration_secs(&self) -> Option<u64> {
match self {
Disk::Local(local_disk) => local_disk.offline_duration_secs(),
@@ -2559,6 +2650,7 @@ pub enum DeleteRecoveryAction {
Rollback,
DiscardLegacy,
AdoptCommit { authority: Box<DeleteRecoveryJournal> },
DiscardLegacyOnDisk { disk_id: Uuid },
}
/// A complete inventory or a work-limit refusal. No public discovery cursor is allocated.
@@ -2571,6 +2663,7 @@ pub struct BoundedDeleteRecoveryDiscovery {
}
pub const DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES: u32 = 4096;
pub const MAX_DELETE_RESIDUE_PAGE_SIZE: u32 = 64;
pub const DELETE_RECOVERY_MAX_INSPECTION_ENTRIES: u32 = 65536;
#[derive(Debug, Clone, Serialize, Deserialize)]
@@ -2648,6 +2741,18 @@ pub enum DeleteRecoveryRpcOperation {
Exists {
object: String,
},
ReadResidueOnDisk {
object: String,
disk_id: Uuid,
transaction: Option<Uuid>,
max_entries: u32,
},
ProbeLegacyDiscard,
ListResidueOnDisk {
disk_id: Uuid,
cursor: Option<DeleteRecoveryCursor>,
limit: u32,
},
}
impl DeleteRecoveryRpcOperation {
@@ -2682,6 +2787,8 @@ pub enum DeleteRecoveryRpcReply {
AppliedVersion,
AppliedVersions(Vec<Option<u32>>),
Evidence { present: bool },
LegacyDiscardDisk { disk_id: Uuid },
ListedResidueOnDisk { disk_id: Uuid, page: DeleteRecoveryPage },
}
#[derive(Debug, Clone, Serialize, Deserialize)]
+2 -2
View File
@@ -2137,7 +2137,7 @@ mod multipart_proof_authority_tests {
assert_eq!(expected, snapshot);
assert_eq!(actual_round, round);
let publication = PublicationIdentity {
lineage: round.lineage.clone(),
lineage: round.lineage,
source_generation: snapshot.generation,
attempt: Uuid::new_v4(),
version_id: None,
@@ -2145,7 +2145,7 @@ mod multipart_proof_authority_tests {
modified_unix_nanos: nanos,
completion_identity: [6; 32],
};
snapshot.publication = Some(publication.clone());
snapshot.publication = Some(publication);
let response = MultipartSourceResponse::TerminalConfirmed(
MultipartSourceProof::issue(
&authority,
+18 -31
View File
@@ -384,14 +384,11 @@ pub(crate) mod prepared_publication_test_hooks {
}
pub(crate) fn has_any_hooks() -> bool {
if !BEFORE_PUBLICATION.lock().is_empty() {
return true;
}
#[cfg(test)]
if !RENAME_DESTINATIONS.lock().is_empty() || !HARDLINK_DESTINATIONS.lock().is_empty() {
return true;
}
false
!BEFORE_PUBLICATION.lock().is_empty()
}
#[cfg(test)]
@@ -3511,20 +3508,6 @@ pub async fn read_dir(path: impl AsRef<Path>, count: i32) -> std::io::Result<Vec
Ok(volumes)
}
#[tracing::instrument(level = "debug", skip_all)]
pub async fn rename_all(
src_file_path: impl AsRef<Path>,
dst_file_path: impl AsRef<Path>,
base_dir: impl AsRef<Path>,
publication_root: &PublicationRoot,
) -> Result<()> {
reliable_rename(src_file_path, dst_file_path.as_ref(), base_dir, publication_root)
.await
.map_err(to_file_error)?;
Ok(())
}
/// Retain mutation ownership through both metadata publication and its directory
/// barriers, even when the async caller stops waiting after the rename starts.
pub(crate) async fn rename_metadata_and_sync_with_lease(
@@ -3569,6 +3552,19 @@ pub(crate) async fn fsync_dir_with_owner(path: &Path, namespace_owner: Option<Ar
}
}
/// Preserve file, symlink, and empty-directory removal semantics while the
/// actual filesystem executor retains namespace ownership after cancellation.
pub(crate) async fn remove_with_owner(path: impl AsRef<Path>, namespace_owner: Option<Arc<dyn Send + Sync>>) -> io::Result<()> {
let path = path.as_ref().to_path_buf();
let lease = acquire_namespace_mutation_lease_with_owner(&path, namespace_owner).await;
run_blocking_namespace_operation(lease, move || {
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Remove, &path);
super::fs::remove_std(path)
})
.await
}
/// Retain namespace ownership in the actual filesystem executor after timeout.
pub(crate) async fn remove_file_with_owner(
path: impl AsRef<Path>,
@@ -4635,15 +4631,6 @@ pub(crate) fn rename_source_is_missing(src_file_path: &Path, _publication_root:
matches!(std::fs::symlink_metadata(src_file_path), Err(err) if err.kind() == io::ErrorKind::NotFound)
}
async fn reliable_rename(
src_file_path: impl AsRef<Path>,
dst_file_path: impl AsRef<Path>,
base_dir: impl AsRef<Path>,
publication_root: &PublicationRoot,
) -> io::Result<()> {
reliable_rename_inner(src_file_path, dst_file_path, base_dir, publication_root, true).await
}
async fn reliable_rename_inner(
src_file_path: impl AsRef<Path>,
dst_file_path: impl AsRef<Path>,
@@ -8658,7 +8645,7 @@ mod tests {
let dst_file_path = dst_file_path.as_ref();
let base_dir = base_dir.as_ref();
let publication_root = test_publication_root(&[src_file_path, dst_file_path, base_dir]);
super::rename_all(src_file_path, dst_file_path, base_dir, &publication_root).await
super::rename_all_with_owner(src_file_path, dst_file_path, base_dir, &publication_root, None).await
}
async fn rename_all_ignore_missing_source(
@@ -9503,7 +9490,7 @@ mod tests {
let configured_dst = configured_base.join("object");
std::fs::create_dir(&configured_base).expect("create bucket through configured root");
std::fs::write(&configured_src, b"configured").expect("write staged object through configured root");
super::rename_all(&configured_src, &configured_dst, &configured_base, &publication_root)
super::rename_all_with_owner(&configured_src, &configured_dst, &configured_base, &publication_root, None)
.await
.expect("publish a configured path relative to the pinned root");
assert_eq!(
@@ -9516,7 +9503,7 @@ mod tests {
let resolved_dst = resolved_base.join("object");
std::fs::create_dir(&resolved_base).expect("create bucket through resolved root");
std::fs::write(&resolved_src, b"resolved").expect("write staged object through resolved root");
super::rename_all(&resolved_src, &resolved_dst, &resolved_base, &publication_root)
super::rename_all_with_owner(&resolved_src, &resolved_dst, &resolved_base, &publication_root, None)
.await
.expect("publish a resolved path relative to the pinned root");
@@ -9936,7 +9923,7 @@ mod tests {
std::fs::write(&dst, b"uncommitted-metadata").expect("write metadata to replace");
let publication_root = PublicationRoot::new(temp_dir.path()).expect("open publication root above the object tree");
super::rename_all(&src, &dst, &object_dir, &publication_root)
super::rename_all_with_owner(&src, &dst, &object_dir, &publication_root, None)
.await
.expect("a rollback source below its destination parent must not conflict with its own guards");
+196
View File
@@ -0,0 +1,196 @@
// Copyright 2026 RustFS Team
// Licensed under the Apache License, Version 2.0.
//! Explicit instance binding at the physical disk boundary. Lock order is this
//! binding mutex, then the registry mutex; registry code never locks a disk.
use crate::runtime::usage_watch::{UsageWatchMutationGuard, UsageWatchRegistry};
use std::sync::{Arc, Mutex, Weak};
/// Existing-version replication status writes change neither size nor version
/// membership. Replication aggregates keep their existing maintenance cadence.
pub(crate) fn status_only_metadata_update(fi: &rustfs_filemeta::FileInfo, opts: &super::UpdateMetadataOpts) -> bool {
use rustfs_utils::http::*;
!opts.replace_user_metadata
&& opts.transition_reconcile.is_none()
&& !fi.metadata.is_empty()
&& fi.metadata.keys().all(|key| {
strip_internal_prefix_preserving_case(key).is_some_and(|suffix| {
[
SUFFIX_REPLICA_STATUS,
SUFFIX_REPLICA_TIMESTAMP,
SUFFIX_REPLICATION_STATUS,
SUFFIX_REPLICATION_TIMESTAMP,
SUFFIX_REPLICATION_GENERATION,
SUFFIX_REPLICATION_RESET,
SUFFIX_PURGESTATUS,
]
.iter()
.any(|known| suffix.eq_ignore_ascii_case(known))
})
})
}
#[derive(Default)]
struct BindingState {
registry: Option<Weak<UsageWatchRegistry>>,
unbound_inflight: u64,
prebinding_guard: Option<UsageWatchMutationGuard>,
conflicted: bool,
}
#[derive(Default)]
pub(crate) struct DiskUsageWatchBinding {
state: Mutex<BindingState>,
}
struct UnboundGuard(Arc<DiskUsageWatchBinding>);
impl Drop for UnboundGuard {
fn drop(&mut self) {
let mut state = self.0.state.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
if state.unbound_inflight != 0 {
state.unbound_inflight -= 1;
}
if state.unbound_inflight == 0 {
state.prebinding_guard = None;
}
}
}
impl DiskUsageWatchBinding {
pub(crate) fn mark_uncovered_user_stream(&self, bucket: &str) {
if crate::bucket::utils::is_meta_bucketname(bucket) {
return;
}
let mut state = self.state.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
state.conflicted = true;
if let Some(registry) = state.registry.as_ref().and_then(Weak::upgrade) {
registry.mark_coverage_unavailable();
}
}
pub(crate) fn bind(&self, registry: &Arc<UsageWatchRegistry>) {
let mut state = self.state.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
if let Some(existing) = state.registry.as_ref().and_then(Weak::upgrade) {
if !Arc::ptr_eq(&existing, registry) {
state.conflicted = true;
// A query may already have checked this binding before
// awaiting disk metadata. Never allow it to reinstall a
// proof after this permanently ambiguous ownership change.
existing.mark_coverage_unavailable();
registry.mark_coverage_unavailable();
}
return;
}
// A newly registered disk can have pre-registration physical tails.
// Keep the whole registry Unknown until those exact guards drain.
if state.unbound_inflight != 0 {
state.prebinding_guard = Some(registry.begin(None));
}
if state.conflicted {
registry.mark_coverage_unavailable();
}
state.registry = Some(Arc::downgrade(registry));
}
pub(crate) fn matches(&self, registry: &Arc<UsageWatchRegistry>) -> bool {
let state = self.state.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
!state.conflicted
&& state
.registry
.as_ref()
.and_then(Weak::upgrade)
.is_some_and(|bound| Arc::ptr_eq(&bound, registry))
}
pub(crate) fn owner(self: &Arc<Self>, bucket: &str, parent: Option<Arc<dyn Send + Sync>>) -> Option<Arc<dyn Send + Sync>> {
if crate::bucket::utils::is_meta_bucketname(bucket) {
return parent;
}
let mut state = self.state.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let guard: Arc<dyn Send + Sync> = if let Some(registry) = state.registry.as_ref().and_then(Weak::upgrade) {
Arc::new(registry.begin((!state.conflicted).then_some(bucket)))
} else {
match state.unbound_inflight.checked_add(1) {
Some(count) => state.unbound_inflight = count,
None => state.conflicted = true,
}
Arc::new(UnboundGuard(Arc::clone(self)))
};
Some(match parent {
Some(parent) => Arc::new((parent, guard)),
None => guard,
})
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::runtime::usage_watch::ScannerUsageWatchRequest;
use uuid::Uuid;
#[test]
fn binding_preserves_mutations_that_started_before_registration() {
let binding = Arc::new(DiskUsageWatchBinding::default());
let registry = Arc::new(UsageWatchRegistry::default());
let tail = binding.owner("bucket", None);
binding.bind(&registry);
assert!(binding.matches(&registry));
let request = ScannerUsageWatchRequest {
bucket: "bucket".into(),
bucket_incarnation: Uuid::from_u128(2),
watch_id: None,
release: false,
session_id: None,
};
assert_eq!(registry.observe(Uuid::from_u128(1), &request), None);
drop(tail);
assert_eq!(
registry
.observe(Uuid::from_u128(1), &request)
.expect("drained binding")
.inflight,
0
);
}
#[test]
fn composite_owner_and_instance_conflicts_fail_closed() {
let binding = Arc::new(DiskUsageWatchBinding::default());
let registry = Arc::new(UsageWatchRegistry::default());
binding.bind(&registry);
let request = ScannerUsageWatchRequest {
bucket: "bucket".into(),
bucket_incarnation: Uuid::from_u128(2),
watch_id: None,
release: false,
session_id: None,
};
registry.observe(Uuid::from_u128(1), &request).expect("initial watch");
let owner = binding.owner("bucket", None).expect("physical owner");
let nested = binding.owner("bucket", Some(owner));
assert_eq!(
registry
.observe(Uuid::from_u128(1), &request)
.expect("both operations active")
.inflight,
2
);
drop(nested);
assert_eq!(
registry
.observe(Uuid::from_u128(1), &request)
.expect("drained owners")
.inflight,
0
);
let other = Arc::new(UsageWatchRegistry::default());
binding.bind(&other);
assert!(!binding.matches(&registry));
assert!(!binding.matches(&other));
assert_eq!(registry.observe(Uuid::from_u128(1), &request), None);
assert_eq!(other.observe(Uuid::from_u128(1), &request), None);
}
}
+1 -1
View File
@@ -439,7 +439,7 @@ mod tests {
reconstruct_with(&engine, &mut shards).expect("restore missing data and parity");
assert_eq!(shards, expected, "every restored byte must match at width {}", shards.len());
let mut corrupt = expected.clone();
let mut corrupt = expected;
corrupt[0] = None;
corrupt[data_shards] = None;
corrupt[data_shards + 1].as_mut().expect("surplus parity present")[0] ^= 1;
@@ -61,6 +61,12 @@ struct FillRequest {
response: oneshot::Sender<FillResult>,
}
#[cfg_attr(test, derive(Debug))]
struct FilledStripes {
primary: Vec<u8>,
queued: VecDeque<Vec<u8>>,
}
struct FillResult {
result: io::Result<Option<Vec<u8>>>,
queued_buffers: VecDeque<Vec<u8>>,
@@ -293,7 +299,7 @@ where
Self::new_with_fill_policy_inner(source, engine, total_length, metrics_path, fill_policy)
}
fn poll_fill_result(&mut self, cx: &mut Context<'_>) -> Poll<io::Result<Option<Vec<u8>>>> {
fn poll_fill_result(&mut self, cx: &mut Context<'_>) -> Poll<io::Result<Option<FilledStripes>>> {
if self.fill.is_none() {
let fill_worker_tx = match self.fill_worker_tx() {
Ok(tx) => tx,
@@ -345,15 +351,6 @@ where
if let Some(deferred_error) = deferred_error {
self.prefetch_error = Some(deferred_error);
}
// Queued stripes are delivered to the client via `prefetched_bufs.pop_front()`
// in `poll_read` without touching `self.remaining`, so their bytes must be
// accounted for here. Otherwise, under multi-in-flight fill policies (e.g. the
// default `DualInFlight`), `remaining` never reaches 0 and a fully delivered
// multi-block object still terminates the GET with `LessData`. Buffers in
// `queued_buffers` come only from `Ok(true)` decodes, so they are non-empty and
// their total is bounded by `remaining - main_buf.len()`, ruling out underflow.
self.remaining -= queued_buffers.iter().map(Vec::len).sum::<usize>();
self.prefetched_bufs.extend(queued_buffers);
match result {
Ok(Some(buf)) => {
@@ -362,8 +359,17 @@ where
}
rustfs_io_metrics::record_get_object_reader_stripe(self.metrics_path);
rustfs_io_metrics::record_get_object_reader_bytes(self.metrics_path, buf.len());
// Queued stripes are delivered to the client via `prefetched_bufs.pop_front()`
// in `poll_read` without touching `self.remaining`, so their bytes must be
// accounted for here alongside the primary stripe. Buffers in `queued_buffers`
// come only from `Ok(true)` decodes, so they are non-empty and their total is
// bounded by `remaining - buf.len()`, ruling out underflow.
self.remaining -= buf.len();
Poll::Ready(Ok(Some(buf)))
self.remaining -= queued_buffers.iter().map(Vec::len).sum::<usize>();
Poll::Ready(Ok(Some(FilledStripes {
primary: buf,
queued: queued_buffers,
})))
}
Ok(None) => {
if self.remaining == 0 {
@@ -401,8 +407,8 @@ where
};
match fill {
Ok(Some(buf)) => {
let output_has_remaining = self.output_pos < self.output_buf.len();
Ok(Some(FilledStripes { primary: buf, queued })) => {
let output_has_remaining = self.output_pos < self.output_buf.len() || !self.prefetched_bufs.is_empty();
let mut queued_count = 0usize;
let mut queued_bytes = 0usize;
@@ -420,6 +426,7 @@ where
self.output_buf = buf;
self.output_pos = 0;
}
self.prefetched_bufs.extend(queued);
if !self.prefetched_bufs.is_empty() {
let total_prefetched_bytes = self.prefetched_bufs.iter().map(Vec::len).sum::<usize>();
@@ -451,7 +458,7 @@ where
Poll::Ready(Ok(()))
}
Err(err) => {
if self.output_pos < self.output_buf.len() {
if self.output_pos < self.output_buf.len() || !self.prefetched_bufs.is_empty() {
rustfs_io_metrics::record_get_object_reader_prefetch(self.metrics_path, GET_READER_PREFETCH_ERROR_DEFERRED);
self.prefetch_error = Some(err);
Poll::Ready(Ok(()))
@@ -1860,6 +1867,42 @@ mod tests {
.expect("dual inflight policy should prefetch two future stripes before current output drains");
}
#[tokio::test]
async fn erasure_decode_reader_dual_inflight_preserves_stripe_order_when_prefetch_completes_before_output_drains() {
let erasure = Erasure::new(4, 2, 16);
let data = (0..96u8).collect::<Vec<_>>();
let read_count = Arc::new(AtomicUsize::new(0));
let mut source = source_from_data(&erasure, &data, &[]);
source.read_count = Some(Arc::clone(&read_count));
let engine = LegacyEcDecodeEngine::new(erasure);
let mut reader = ErasureDecodeReader::new_with_fill_policy(
source,
engine,
data.len(),
GET_OBJECT_PATH_CODEC_STREAMING,
FillPolicy::DualInFlight,
)
.expect("reader should be constructed");
let mut decoded = Vec::with_capacity(data.len());
let mut chunk = [0u8; 5];
loop {
let n = reader.read(&mut chunk).await.expect("chunk read should succeed");
if n == 0 {
break;
}
decoded.extend_from_slice(&chunk[..n]);
// Yield between partial reads so the DualInFlight worker completes
// a 2-stripe fill (primary + queued_buffers) while output_buf still
// has unconsumed bytes.
for _ in 0..4 {
yield_now().await;
}
}
assert_eq!(decoded, data);
}
#[tokio::test]
async fn erasure_decode_reader_dual_inflight_reads_multi_block_object_to_end() {
// Regression for the byte-accounting bug: under DualInFlight each fill
+1 -1
View File
@@ -2002,7 +2002,7 @@ mod tests {
for storage_error in storage_errors {
let original_message = storage_error.to_string();
let io_error: std::io::Error = storage_error.clone().into();
let io_error: std::io::Error = storage_error.into();
// The io::Error should contain the original error message or info
assert!(io_error.to_string().contains(&original_message));
+1 -1
View File
@@ -391,7 +391,7 @@ impl PoolEndpointList {
e.get_mut().extend(host_ip_set.iter());
}
Entry::Vacant(e) => {
e.insert(host_ip_set.clone());
e.insert(host_ip_set);
}
}
}
+1 -1
View File
@@ -47,7 +47,7 @@ pub(crate) fn formats_to_drives_info(
before_drives.push(HealDriveInfo {
uuid,
endpoint: drive,
state: state.to_string(),
state,
});
}
before_drives
+16
View File
@@ -50,6 +50,7 @@ use crate::layout::endpoints::{EndpointServerPools, SetupType};
use crate::object_api::ObjectEncryptionResolver;
use crate::services::event_notification::EventNotifier;
use crate::services::tier::tier::TierConfigMgr;
use crate::store::delete_recovery_inventory::DeleteResidueCursors;
use crate::store::delete_recovery_runtime::DeleteRecoveryHints;
use rustfs_lock::{GlobalLockManager, get_global_lock_manager};
use s3s::region::Region;
@@ -99,6 +100,7 @@ pub(crate) struct ScannerPublicationLeaseEntry {
pub(crate) struct NamespaceCommitGuard {
ctx: Arc<InstanceContext>,
counted: bool,
_usage_watch: super::usage_watch::UsageWatchMutationGuard,
}
impl Drop for NamespaceCommitGuard {
@@ -365,8 +367,10 @@ pub struct InstanceContext {
scanner_publication_state: AtomicU8,
namespace_commits: AtomicU64,
namespace_commit_generation: AtomicU64,
pub(crate) usage_watches: Arc<super::usage_watch::UsageWatchRegistry>,
/// Resolves object-encryption material at the application boundary.
object_encryption_resolver: OnceLock<Arc<dyn ObjectEncryptionResolver>>,
pub(crate) delete_residue_cursors: DeleteResidueCursors,
delete_recovery_hints: DeleteRecoveryHints,
tier_delete_journal_recovery_stores: std::sync::Mutex<HashSet<Uuid>>,
#[cfg(test)]
@@ -444,7 +448,9 @@ impl InstanceContext {
scanner_publication_state: AtomicU8::new(SCANNER_PUBLICATION_STATE_UNKNOWN),
namespace_commits: AtomicU64::new(0),
namespace_commit_generation: AtomicU64::new(0),
usage_watches: Arc::new(super::usage_watch::UsageWatchRegistry::default()),
object_encryption_resolver: OnceLock::new(),
delete_residue_cursors: DeleteResidueCursors::default(),
delete_recovery_hints: DeleteRecoveryHints::default(),
tier_delete_journal_recovery_stores: std::sync::Mutex::new(HashSet::new()),
#[cfg(test)]
@@ -683,6 +689,15 @@ impl InstanceContext {
}
pub(crate) fn begin_namespace_commit(self: &Arc<Self>) -> Arc<NamespaceCommitGuard> {
self.begin_namespace_commit_scoped(None)
}
pub(crate) fn begin_namespace_commit_for_bucket(self: &Arc<Self>, bucket: &str) -> Arc<NamespaceCommitGuard> {
self.begin_namespace_commit_scoped(Some(bucket))
}
fn begin_namespace_commit_scoped(self: &Arc<Self>, bucket: Option<&str>) -> Arc<NamespaceCommitGuard> {
let usage_watch = self.usage_watches.begin(bucket);
let counted = self
.namespace_commits
.try_update(Ordering::AcqRel, Ordering::Acquire, |count| count.checked_add(1))
@@ -695,6 +710,7 @@ impl InstanceContext {
Arc::new(NamespaceCommitGuard {
ctx: Arc::clone(self),
counted,
_usage_watch: usage_watch,
})
}
+1
View File
@@ -17,3 +17,4 @@
pub(crate) mod global;
pub(crate) mod instance;
pub(crate) mod sources;
pub(crate) mod usage_watch;
+10 -11
View File
@@ -31,11 +31,10 @@ use crate::{
layout::endpoints::{EndpointServerPools, SetupType},
runtime::global::{
GLOBAL_BOOT_TIME, GLOBAL_LIFECYCLE_SYS, GLOBAL_LOCAL_NODE_NAME_FALLBACK, GLOBAL_ROOT_DISK_THRESHOLD,
TypeLocalDiskSetDrives, get_background_services_cancel_token, get_global_bucket_monitor, get_global_deployment_id,
get_global_endpoints, get_global_endpoints_opt, get_global_lock_client, get_global_lock_clients, get_global_region,
get_global_tier_config_mgr, global_rustfs_port, init_global_bucket_monitor, is_dist_erasure, is_erasure, is_erasure_sd,
is_first_cluster_node_local, resolve_object_store_handle, set_global_lock_client, set_global_lock_clients,
set_object_layer, update_erasure_type,
get_background_services_cancel_token, get_global_bucket_monitor, get_global_deployment_id, get_global_endpoints,
get_global_endpoints_opt, get_global_lock_client, get_global_lock_clients, get_global_region, get_global_tier_config_mgr,
global_rustfs_port, init_global_bucket_monitor, is_dist_erasure, is_erasure, is_erasure_sd, is_first_cluster_node_local,
resolve_object_store_handle, set_global_lock_client, set_global_lock_clients, set_object_layer, update_erasure_type,
},
services::batch_processor::{GlobalBatchProcessors, get_global_processors},
services::notification_sys::{NotificationSys, get_global_notification_sys},
@@ -410,10 +409,6 @@ pub(crate) fn local_disk_id_map_handle() -> Arc<RwLock<HashMap<Uuid, String>>> {
crate::runtime::global::current_ctx().local_disk_id_map()
}
pub(crate) fn local_disk_set_drives_handle() -> Arc<RwLock<TypeLocalDiskSetDrives>> {
crate::runtime::global::current_ctx().local_disk_set_drives()
}
pub(crate) fn tier_config_mgr_handle() -> Arc<RwLock<TierConfigMgr>> {
get_global_tier_config_mgr()
}
@@ -525,8 +520,11 @@ pub(crate) async fn record_local_disks(instance_ctx: &Arc<InstanceContext>, disk
let map = instance_ctx.local_disk_map();
let mut global_local_disk_map = map.write().await;
for disk in disks {
if let crate::disk::Disk::Local(wrapper) = disk.as_ref() {
wrapper.get_disk().usage_watch_binding.bind(&instance_ctx.usage_watches);
}
let path = disk.endpoint().to_string();
global_local_disk_map.insert(path, Some(disk.clone()));
global_local_disk_map.insert(path, Some(disk));
}
}
@@ -611,6 +609,7 @@ pub(crate) async fn initialize_local_disk_maps(
Err(err) => return Err(err.into()),
};
if let crate::disk::Disk::Local(wrapper) = disk.as_ref() {
wrapper.get_disk().usage_watch_binding.bind(&instance_ctx.usage_watches);
wrapper.get_disk().bind_multipart_proof_authority(&multipart_authority)?;
}
let path = disk.endpoint().to_string();
@@ -625,7 +624,7 @@ pub(crate) async fn initialize_local_disk_maps(
})?;
global_local_disk_map.insert(path, Some(disk.clone()));
global_set_drives[pool_idx][set_idx][disk_idx] = Some(disk.clone());
global_set_drives[pool_idx][set_idx][disk_idx] = Some(disk);
}
}
+416
View File
@@ -0,0 +1,416 @@
// Copyright 2026 RustFS Team
// Licensed under the Apache License, Version 2.0.
//! Bounded, same-session bucket mutation watches. The registry lock is a short
//! synchronous critical section shared by installation, mutation begin/end,
//! and observation. No I/O or await occurs while it is held.
use std::{
collections::HashMap,
sync::{Arc, Mutex, OnceLock},
time::{Duration, Instant},
};
use uuid::Uuid;
pub const SCANNER_USAGE_WATCH_COVERAGE_VERSION: u32 = rustfs_protos::usage_watch::USAGE_WATCH_COVERAGE_VERSION;
pub const SCANNER_USAGE_WATCH_MAX_ENTRIES: usize = 256;
pub const SCANNER_USAGE_WATCH_MAX_ACTIVE_BUCKETS: usize = 4096;
pub const SCANNER_USAGE_WATCH_MAX_NAME_BYTES: usize = 16 * 1024;
pub const SCANNER_USAGE_WATCH_MAX_ACTIVE_NAME_BYTES: usize = 256 * 1024;
pub const SCANNER_USAGE_WATCH_LEASE_MS: u64 = rustfs_protos::usage_watch::USAGE_WATCH_MAX_LEASE_MS;
/// A missing watch ID installs a new watch; a supplied ID can only refresh the
/// exact, still-live token. An expired token never silently creates a new one.
#[derive(Clone, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
pub struct ScannerUsageWatchRequest {
pub bucket: String,
pub bucket_incarnation: Uuid,
pub watch_id: Option<Uuid>,
/// Release only this exact producer session and token after publication.
pub release: bool,
pub session_id: Option<Uuid>,
}
#[derive(Clone, Debug, PartialEq, Eq, serde::Serialize, serde::Deserialize)]
pub struct ScannerUsageWatchKnown {
pub owner_id: Uuid,
pub session_id: Uuid,
pub watch_id: Uuid,
pub bucket_incarnation: Uuid,
pub revision: u64,
pub inflight: u64,
pub coverage_version: u32,
pub coverage_epoch: u64,
/// Duration from the receiver's observation, never a wall-clock deadline.
pub lease_remaining_ms: u64,
}
impl ScannerUsageWatchKnown {
pub fn is_well_formed(&self) -> bool {
!self.owner_id.is_nil()
&& !self.session_id.is_nil()
&& !self.watch_id.is_nil()
&& !self.bucket_incarnation.is_nil()
&& self.revision != 0
&& self.revision != u64::MAX
&& self.inflight != u64::MAX
&& self.coverage_version == SCANNER_USAGE_WATCH_COVERAGE_VERSION
&& self.coverage_epoch != 0
&& self.coverage_epoch != u64::MAX
&& self.lease_remaining_ms != 0
&& self.lease_remaining_ms <= SCANNER_USAGE_WATCH_LEASE_MS
}
/// Equality of stable proof fields; the caller must also respect its
/// conservative local lease deadline and validate the source/owner set.
pub fn unchanged_since(&self, earlier: &Self) -> bool {
self.is_well_formed()
&& earlier.is_well_formed()
&& self.owner_id == earlier.owner_id
&& self.session_id == earlier.session_id
&& self.watch_id == earlier.watch_id
&& self.bucket_incarnation == earlier.bucket_incarnation
&& self.revision == earlier.revision
&& self.inflight == 0
&& earlier.inflight == 0
&& self.coverage_version == SCANNER_USAGE_WATCH_COVERAGE_VERSION
&& self.coverage_version == earlier.coverage_version
&& self.coverage_epoch == earlier.coverage_epoch
&& self.lease_remaining_ms != 0
&& earlier.lease_remaining_ms != 0
}
}
#[derive(Clone, Debug, PartialEq, Eq)]
pub struct ScannerUsageWatchObservation {
pub bucket: String,
/// Missing means Unknown. It is never a revision-zero clean value.
pub known: Option<ScannerUsageWatchKnown>,
}
pub(crate) struct UsageWatchOwners {
pub(crate) deployment: Uuid,
pub(crate) local: Uuid,
pub(crate) peers: HashMap<String, Uuid>,
}
impl UsageWatchOwners {
pub(crate) fn from_topology(deployment: Uuid, topology: &crate::layout::endpoints::EndpointServerPools) -> Option<Self> {
use crate::layout::endpoint::EndpointType;
use sha2::{Digest, Sha256};
use std::collections::hash_map::Entry;
if deployment.is_nil() || topology.as_ref().is_empty() {
return None;
}
let mut has_paths = false;
let mut nodes = HashMap::new();
for pool in topology.as_ref() {
if pool.endpoints.as_ref().is_empty()
|| pool.set_count.checked_mul(pool.drives_per_set) != Some(pool.endpoints.as_ref().len())
{
return None;
}
for endpoint in pool.endpoints.as_ref() {
if endpoint.get_type() == EndpointType::Path {
if !endpoint.is_local {
return None;
}
has_paths = true;
continue;
}
let host = endpoint.host_port();
let grid = endpoint.grid_host();
if host.is_empty() || grid.is_empty() {
return None;
}
let route = (grid, endpoint.is_local);
match nodes.entry(host) {
Entry::Vacant(entry) => {
entry.insert(route);
}
Entry::Occupied(entry) if entry.get() == &route => {}
Entry::Occupied(_) => return None,
}
}
}
if has_paths {
return nodes.is_empty().then_some(Self {
deployment,
local: deployment,
peers: HashMap::new(),
});
}
let mut local = None;
let mut peers = HashMap::new();
for (host, (grid, is_local)) in nodes {
let mut hash = Sha256::new();
hash.update(b"rustfs-scanner-usage-watch-owner-v1");
hash.update(deployment.as_bytes());
hash.update(host.as_bytes());
let digest = hash.finalize();
let mut bytes = [0; 16];
bytes.copy_from_slice(&digest[..16]);
let owner = uuid::Builder::from_custom_bytes(bytes).into_uuid();
if is_local {
if local.replace(owner).is_some() {
return None;
}
} else if peers.insert(grid, owner).is_some() {
return None;
}
}
Some(Self {
deployment,
local: local?,
peers,
})
}
}
struct Watch {
owner: Uuid,
token: Uuid,
incarnation: Uuid,
revision: u64,
expires_at: Instant,
}
#[derive(Default)]
struct State {
watches: HashMap<Arc<str>, Watch>,
active: HashMap<Arc<str>, u64>,
watch_name_bytes: usize,
active_name_bytes: usize,
untracked: u64,
exhausted: bool,
coverage_epoch: u64,
}
pub(crate) struct UsageWatchRegistry {
pub(crate) owners: OnceLock<UsageWatchOwners>,
session: Uuid,
state: Mutex<State>,
}
enum Mutation {
Tracked(Arc<str>),
Untracked,
}
/// The physical mutation owner retains this value through all detached tails.
pub(crate) struct UsageWatchMutationGuard {
registry: Arc<UsageWatchRegistry>,
mutation: Mutation,
}
impl Default for UsageWatchRegistry {
fn default() -> Self {
Self {
owners: OnceLock::new(),
session: Uuid::new_v4(),
state: Mutex::new(State {
coverage_epoch: 1,
..Default::default()
}),
}
}
}
impl State {
fn invalidate_all(&mut self) {
self.watches.clear();
self.watch_name_bytes = 0;
match self.coverage_epoch.checked_add(1).filter(|epoch| *epoch != u64::MAX) {
Some(epoch) => self.coverage_epoch = epoch,
None => self.exhausted = true,
}
}
fn advance_bucket(&mut self, bucket: &str) {
if let Some(watch) = self.watches.get_mut(bucket) {
if let Some(next) = watch.revision.checked_add(1).filter(|next| *next != u64::MAX) {
watch.revision = next;
} else {
self.remove_watch(bucket);
}
}
}
fn remove_watch(&mut self, bucket: &str) {
if let Some((name, _)) = self.watches.remove_entry(bucket) {
self.watch_name_bytes -= name.len();
}
}
}
impl UsageWatchRegistry {
pub(crate) fn session_id(&self) -> Uuid {
self.session
}
pub(crate) fn begin(self: &Arc<Self>, bucket: Option<&str>) -> UsageWatchMutationGuard {
let mut state = self.state.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
// Internal metadata is outside the user bucket counters. Callers with
// an unknown user scope must pass None, which invalidates every watch.
let bucket = bucket.filter(|bucket| !bucket.is_empty() && bucket.len() <= 63);
let tracked = bucket.and_then(|bucket| {
if let Some((key, count)) = state.active.get_key_value(bucket) {
let key = Arc::clone(key);
let next = count.checked_add(1)?;
state.active.insert(Arc::clone(&key), next);
return Some(key);
}
if state.active.len() >= SCANNER_USAGE_WATCH_MAX_ACTIVE_BUCKETS
|| state.active_name_bytes + bucket.len() > SCANNER_USAGE_WATCH_MAX_ACTIVE_NAME_BYTES
{
return None;
}
let key: Arc<str> = Arc::from(bucket);
state.active_name_bytes += bucket.len();
state.active.insert(Arc::clone(&key), 1);
Some(key)
});
let mutation = match tracked {
Some(bucket) => {
state.advance_bucket(&bucket);
Mutation::Tracked(bucket)
}
None => {
state.invalidate_all();
match state.untracked.checked_add(1) {
Some(count) => state.untracked = count,
None => state.exhausted = true,
}
Mutation::Untracked
}
};
UsageWatchMutationGuard {
registry: Arc::clone(self),
mutation,
}
}
/// Replay and unsupported runtime mode transitions revoke tokens. They
/// never copy a durable generation into a same-session watch revision.
pub(crate) fn invalidate(&self) {
self.state
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.invalidate_all();
}
/// A write mode without a physical completion owner cannot establish
/// same-session coverage again. Keep the global fallback until restart.
pub(crate) fn mark_coverage_unavailable(&self) {
let mut state = self.state.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
state.exhausted = true;
state.invalidate_all();
}
pub(crate) fn observe(&self, owner: Uuid, request: &ScannerUsageWatchRequest) -> Option<ScannerUsageWatchKnown> {
self.observe_at(owner, request, Instant::now())
}
fn observe_at(&self, owner: Uuid, request: &ScannerUsageWatchRequest, now: Instant) -> Option<ScannerUsageWatchKnown> {
if owner.is_nil() || request.bucket_incarnation.is_nil() || request.bucket.is_empty() || request.bucket.len() > 63 {
return None;
}
let mut state = self.state.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
if request.release {
if request.session_id == Some(self.session)
&& state.watches.get(request.bucket.as_str()).is_some_and(|watch| {
watch.owner == owner
&& watch.incarnation == request.bucket_incarnation
&& request.watch_id == Some(watch.token)
})
{
state.remove_watch(&request.bucket);
}
// Release is resource retirement, never an assertion of cleanliness.
return None;
}
if request.session_id.is_some_and(|session| session != self.session) {
return None;
}
if state.exhausted || state.untracked != 0 {
return None;
}
// Sweeps are bounded by the fixed watch limit. Never accumulate tombstones.
state.watches.retain(|_, watch| watch.expires_at > now);
state.watch_name_bytes = state.watches.keys().map(|key| key.len()).sum();
let matches = state.watches.get(request.bucket.as_str()).is_some_and(|watch| {
watch.owner == owner
&& watch.incarnation == request.bucket_incarnation
&& request.watch_id.is_none_or(|id| id == watch.token)
});
if !matches {
if request.watch_id.is_some() {
return None;
}
state.remove_watch(&request.bucket);
if state.watches.len() >= SCANNER_USAGE_WATCH_MAX_ENTRIES
|| state.watch_name_bytes + request.bucket.len() > SCANNER_USAGE_WATCH_MAX_NAME_BYTES
{
return None;
}
let key: Arc<str> = Arc::from(request.bucket.as_str());
state.watch_name_bytes += key.len();
state.watches.insert(
key,
Watch {
owner,
token: Uuid::new_v4(),
incarnation: request.bucket_incarnation,
revision: 1,
expires_at: now + Duration::from_millis(SCANNER_USAGE_WATCH_LEASE_MS),
},
);
}
let inflight = state.active.get(request.bucket.as_str()).copied().unwrap_or(0);
let coverage_epoch = state.coverage_epoch;
let watch = state.watches.get_mut(request.bucket.as_str())?;
watch.expires_at = now + Duration::from_millis(SCANNER_USAGE_WATCH_LEASE_MS);
Some(ScannerUsageWatchKnown {
owner_id: owner,
session_id: self.session,
watch_id: watch.token,
bucket_incarnation: watch.incarnation,
revision: watch.revision,
inflight,
coverage_version: SCANNER_USAGE_WATCH_COVERAGE_VERSION,
coverage_epoch,
lease_remaining_ms: SCANNER_USAGE_WATCH_LEASE_MS,
})
}
}
impl Drop for UsageWatchMutationGuard {
fn drop(&mut self) {
let mut state = self.registry.state.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
match &self.mutation {
Mutation::Tracked(bucket) => {
// Advance before exposing inflight=0, including canceled and
// failed operations whose physical outcome may be uncertain.
state.advance_bucket(bucket);
if let Some(count) = state.active.get_mut(bucket) {
*count -= 1;
if *count == 0 {
state.active.remove(bucket);
state.active_name_bytes -= bucket.len();
}
} else {
state.exhausted = true;
state.invalidate_all();
}
}
Mutation::Untracked => {
if state.untracked != 0 {
state.untracked -= 1;
}
}
}
}
}
#[cfg(test)]
mod tests;
@@ -0,0 +1,321 @@
// Copyright 2026 RustFS Team
// Licensed under the Apache License, Version 2.0.
use super::*;
fn request(bucket: &str) -> ScannerUsageWatchRequest {
ScannerUsageWatchRequest {
bucket: bucket.into(),
bucket_incarnation: Uuid::from_u128(7),
watch_id: None,
release: false,
session_id: None,
}
}
fn query(registry: &UsageWatchRegistry, request: &ScannerUsageWatchRequest, at: Instant) -> ScannerUsageWatchKnown {
registry
.observe_at(Uuid::from_u128(1), request, at)
.expect("known bucket proof")
}
#[test]
fn watch_installation_detects_an_already_active_mutation() {
let registry = Arc::new(UsageWatchRegistry::default());
let now = Instant::now();
let mutation = registry.begin(Some("bucket"));
let before = query(&registry, &request("bucket"), now);
assert_eq!(before.inflight, 1);
drop(mutation);
let after = query(&registry, &request("bucket"), now);
assert_eq!(after.inflight, 0);
assert!(after.revision > before.revision);
assert!(!after.unchanged_since(&before));
}
#[test]
fn every_begin_and_physical_tail_end_advances_only_the_target_bucket() {
let registry = Arc::new(UsageWatchRegistry::default());
let now = Instant::now();
let before = query(&registry, &request("cold"), now);
let hot = registry.begin(Some("hot"));
assert!(query(&registry, &request("cold"), now).unchanged_since(&before));
drop(hot);
assert!(query(&registry, &request("cold"), now).unchanged_since(&before));
let owner = Arc::new(registry.begin(Some("cold")));
let physical_tail = Arc::clone(&owner);
let during = query(&registry, &request("cold"), now);
assert_eq!(during.revision, before.revision + 1);
assert_eq!(during.inflight, 1);
drop(owner);
assert_eq!(query(&registry, &request("cold"), now).inflight, 1);
drop(physical_tail);
let after = query(&registry, &request("cold"), now);
assert_eq!(after.revision, before.revision + 2);
assert_eq!(after.inflight, 0);
}
#[test]
fn expiry_reincarnation_and_restart_do_not_resurrect_old_tokens() {
let registry = UsageWatchRegistry::default();
let now = Instant::now();
let before = query(&registry, &request("bucket"), now);
let mut refresh = request("bucket");
refresh.watch_id = Some(before.watch_id);
let expired = now + Duration::from_millis(SCANNER_USAGE_WATCH_LEASE_MS);
assert_eq!(registry.observe_at(Uuid::from_u128(1), &refresh, expired), None);
let after = query(&registry, &request("bucket"), expired);
assert_ne!(after.watch_id, before.watch_id);
assert!(!after.unchanged_since(&before));
refresh.bucket_incarnation = Uuid::new_v4();
assert_eq!(registry.observe_at(Uuid::from_u128(1), &refresh, expired), None);
refresh.watch_id = None;
let recreated = query(&registry, &refresh, expired);
assert_ne!(recreated.watch_id, after.watch_id);
let restarted = UsageWatchRegistry::default();
let restart_proof = query(&restarted, &refresh, expired);
assert!(!restart_proof.unchanged_since(&recreated));
println!(
"usage_assertion: {}",
serde_json::json!({
"case_id": "USAGE-12", "assertion_id": "USAGE-12-A05", "status": "PASS",
"evidence": {"expired_token": before.watch_id, "replacement_token": after.watch_id, "lease_ms": SCANNER_USAGE_WATCH_LEASE_MS, "expired_proof_accepted": false}
})
);
println!(
"usage_assertion: {}",
serde_json::json!({
"case_id": "USAGE-12", "assertion_id": "USAGE-12-A06", "status": "PASS",
"evidence": {"previous_session": recreated.session_id, "new_session": restart_proof.session_id, "unchanged": restart_proof.unchanged_since(&recreated)}
})
);
println!(
"usage_cleanup: {}",
serde_json::json!({"case_id": "USAGE-12", "status": "PASS", "scope": "process-local"})
);
}
#[test]
fn watch_capacity_is_bounded_and_rejected_entries_are_unknown() {
let registry = UsageWatchRegistry::default();
let now = Instant::now();
for index in 0..SCANNER_USAGE_WATCH_MAX_ENTRIES {
query(&registry, &request(&format!("bucket-{index}")), now);
}
assert_eq!(registry.observe_at(Uuid::from_u128(1), &request("overflow"), now), None);
let state = registry.state.lock().expect("registry state");
assert_eq!(state.watches.len(), SCANNER_USAGE_WATCH_MAX_ENTRIES);
assert!(state.watch_name_bytes <= SCANNER_USAGE_WATCH_MAX_NAME_BYTES);
}
#[test]
fn active_mutation_overflow_revokes_all_proofs_until_untracked_work_ends() {
let registry = Arc::new(UsageWatchRegistry::default());
let now = Instant::now();
let old = query(&registry, &request("cold"), now);
let mut active = Vec::new();
for index in 0..SCANNER_USAGE_WATCH_MAX_ACTIVE_BUCKETS {
active.push(registry.begin(Some(&format!("active-{index}"))));
}
let overflow = registry.begin(Some("not-in-active-map"));
assert_eq!(registry.observe_at(Uuid::from_u128(1), &request("cold"), now), None);
{
let state = registry.state.lock().expect("registry state");
assert_eq!(state.active.len(), SCANNER_USAGE_WATCH_MAX_ACTIVE_BUCKETS);
assert!(state.active_name_bytes <= SCANNER_USAGE_WATCH_MAX_ACTIVE_NAME_BYTES);
}
drop(overflow);
let new = query(&registry, &request("cold"), now);
assert_ne!(new.watch_id, old.watch_id);
assert!(new.coverage_epoch > old.coverage_epoch);
drop(active);
println!(
"usage_assertion: {}",
serde_json::json!({
"case_id": "USAGE-12", "assertion_id": "USAGE-12-A04", "status": "PASS",
"evidence": {"active_capacity": SCANNER_USAGE_WATCH_MAX_ACTIVE_BUCKETS, "previous_token": old.watch_id, "replacement_token": new.watch_id, "previous_coverage": old.coverage_epoch, "new_coverage": new.coverage_epoch, "active_after_cleanup": registry.state.lock().expect("drained registry").active.len()}
})
);
println!(
"usage_cleanup: {}",
serde_json::json!({"case_id": "USAGE-12", "status": "PASS", "scope": "process-local"})
);
}
#[test]
fn unknown_scope_and_replay_invalidate_without_reusing_generation() {
let registry = Arc::new(UsageWatchRegistry::default());
let now = Instant::now();
let old = query(&registry, &request("bucket"), now);
let mutation = registry.begin(None);
assert_eq!(registry.observe_at(Uuid::from_u128(1), &request("bucket"), now), None);
drop(mutation);
let new = query(&registry, &request("bucket"), now);
assert_ne!(new.watch_id, old.watch_id);
registry.invalidate();
let replayed = query(&registry, &request("bucket"), now);
assert_ne!(replayed.watch_id, new.watch_id);
assert!(replayed.coverage_epoch > new.coverage_epoch);
}
#[test]
fn revision_exhaustion_invalidates_the_token_without_aba() {
let registry = Arc::new(UsageWatchRegistry::default());
let now = Instant::now();
let old = query(&registry, &request("bucket"), now);
registry
.state
.lock()
.expect("registry state")
.watches
.get_mut("bucket")
.expect("installed watch")
.revision = u64::MAX - 1;
let mutation = registry.begin(Some("bucket"));
let refresh = ScannerUsageWatchRequest {
watch_id: Some(old.watch_id),
..request("bucket")
};
assert_eq!(registry.observe_at(Uuid::from_u128(1), &refresh, now), None);
let during = query(&registry, &request("bucket"), now);
assert_ne!(during.watch_id, old.watch_id);
assert_eq!(during.inflight, 1);
drop(mutation);
assert_eq!(query(&registry, &request("bucket"), now).revision, during.revision + 1);
}
#[test]
fn coverage_epoch_exhaustion_stays_unknown_without_wrapping() {
let registry = UsageWatchRegistry::default();
let now = Instant::now();
query(&registry, &request("bucket"), now);
registry.state.lock().expect("registry state").coverage_epoch = u64::MAX - 1;
registry.invalidate();
assert_eq!(registry.observe_at(Uuid::from_u128(1), &request("bucket"), now), None);
registry.invalidate();
assert_eq!(registry.observe_at(Uuid::from_u128(1), &request("bucket"), now), None);
let state = registry.state.lock().expect("registry state");
assert!(state.exhausted);
assert!(state.watches.is_empty());
assert_eq!(state.coverage_epoch, u64::MAX - 1);
}
#[test]
fn proof_comparison_rejects_every_changed_identity_and_busy_endpoint() {
let registry = UsageWatchRegistry::default();
let original = query(&registry, &request("bucket"), Instant::now());
for field in 0..10 {
let mut changed = original.clone();
match field {
0 => changed.owner_id = Uuid::new_v4(),
1 => changed.session_id = Uuid::new_v4(),
2 => changed.watch_id = Uuid::new_v4(),
3 => changed.bucket_incarnation = Uuid::new_v4(),
4 => changed.revision += 1,
5 => changed.inflight = 1,
6 => changed.coverage_version += 1,
7 => changed.coverage_epoch += 1,
8 => changed.lease_remaining_ms = 0,
_ => {
changed.revision = 0;
changed.watch_id = Uuid::nil();
}
}
assert!(!changed.unchanged_since(&original));
}
}
#[test]
fn watch_contention_cost_preserves_all_mutation_boundaries() {
let registry = Arc::new(UsageWatchRegistry::default());
let before = query(&registry, &request("hot"), Instant::now());
let threads = 8_u64;
let writes_per_thread = 10_000_u64;
let started = Instant::now();
std::thread::scope(|scope| {
for _ in 0..threads {
let registry = Arc::clone(&registry);
scope.spawn(move || {
for _ in 0..writes_per_thread {
drop(registry.begin(Some("hot")));
}
});
}
});
let elapsed = started.elapsed();
let after = query(&registry, &request("hot"), Instant::now());
let writes = threads * writes_per_thread;
assert_eq!(after.revision, before.revision + 2 * writes);
assert_eq!(after.inflight, 0);
assert!(registry.state.lock().expect("drained registry").active.is_empty());
eprintln!(
"usage_watch_cost threads={threads} writes={writes} elapsed_us={} wall_ns_per_begin_end={}",
elapsed.as_micros(),
elapsed.as_nanos() / u128::from(writes)
);
}
#[test]
fn exact_release_preserves_other_sessions_tokens_and_active_mutations() {
let registry = Arc::new(UsageWatchRegistry::default());
let owner = Uuid::from_u128(1);
let now = Instant::now();
let before = query(&registry, &request("bucket"), now);
let release = ScannerUsageWatchRequest {
watch_id: Some(before.watch_id),
session_id: Some(before.session_id),
release: true,
..request("bucket")
};
for case in 0..5 {
let mut wrong = release.clone();
let mut wrong_owner = owner;
match case {
0 => wrong.watch_id = Some(Uuid::new_v4()),
1 => wrong.session_id = Some(Uuid::new_v4()),
2 => wrong.bucket_incarnation = Uuid::new_v4(),
3 => wrong.watch_id = None,
_ => wrong_owner = Uuid::new_v4(),
}
assert_eq!(registry.observe_at(wrong_owner, &wrong, now), None);
assert!(query(&registry, &request("bucket"), now).unchanged_since(&before));
}
let mutation = registry.begin(Some("bucket"));
assert_eq!(registry.observe_at(owner, &release, now), None);
assert!(registry.state.lock().expect("registry state").watches.is_empty());
let replacement = query(&registry, &request("bucket"), now);
assert_ne!(replacement.watch_id, before.watch_id);
assert_eq!(replacement.inflight, 1, "release cannot discard physical mutation ownership");
assert_eq!(registry.observe_at(owner, &release, now), None);
assert_eq!(query(&registry, &request("bucket"), now).watch_id, replacement.watch_id);
drop(mutation);
let after = query(&registry, &request("bucket"), now);
assert_eq!(after.inflight, 0);
assert!(after.revision > replacement.revision);
}
#[test]
fn completed_watch_release_admits_later_buckets_before_lease_expiry() {
let registry = UsageWatchRegistry::default();
let now = Instant::now();
let mut completed = Vec::new();
for index in 0..SCANNER_USAGE_WATCH_MAX_ENTRIES {
let req = request(&format!("bucket-{index:03}"));
let known = query(&registry, &req, now);
completed.push(ScannerUsageWatchRequest {
watch_id: Some(known.watch_id),
session_id: Some(known.session_id),
release: true,
..req
});
}
assert_eq!(registry.observe_at(Uuid::from_u128(1), &request("later-bucket"), now), None);
for release in completed {
assert_eq!(registry.observe_at(Uuid::from_u128(1), &release, now), None);
}
let state = registry.state.lock().expect("registry state");
assert_eq!(state.watches.len(), 0);
assert_eq!(state.watch_name_bytes, 0);
drop(state);
assert!(query(&registry, &request("later-bucket"), now).is_well_formed());
}
@@ -107,7 +107,7 @@ pub(super) async fn run(
result.state = ItemState::Prepared;
result.detail = None;
etag = checkpoint(api, job, &etag, guard).await?;
let mut file = match stage(api, job, index, &info, &opts, guard).await {
let staged = match verify_source(api, job, index, &info, &opts, guard).await {
Ok(file) => file,
Err(IntegrityError::Conflict) => {
job.results[index].state = ItemState::Stale;
@@ -125,6 +125,7 @@ pub(super) async fn run(
if job.request.mode == JobMode::Audit {
job.results[index].state = ItemState::Verified;
} else {
let mut file = staged.ok_or(IntegrityError::Invalid("missing migration payload"))?;
let snapshot = validate_destination(api, job).await?;
file.rewind().await?;
let raw = sha256_bytes(&expected)?;
@@ -231,30 +232,29 @@ async fn ensure_running(api: &Arc<ECStore>, job: &Job, guard: &rustfs_lock::Name
Ok(())
}
async fn stage(
async fn verify_source(
api: &Arc<ECStore>,
job: &Job,
index: usize,
info: &ObjectInfo,
opts: &ObjectOptions,
guard: &rustfs_lock::NamespaceLockGuard,
) -> Result<tokio::fs::File> {
) -> Result<Option<tokio::fs::File>> {
let mut reader = api
.get_object_reader(&job.bucket, &job.request.items[index].key, None, Default::default(), opts)
.await?;
if fingerprint(&reader.object_info) != fingerprint(info) {
return Err(IntegrityError::Conflict);
}
let std_file = tokio::task::spawn_blocking(tempfile::tempfile)
.await
.map_err(|_| IntegrityError::Invalid("staging task failed"))??;
let mut file = tokio::fs::File::from_std(std_file);
let mut file = staging_file(job.request.mode, tempfile::tempfile).await?;
let expected = job.results[index]
.expected_sha256
.as_deref()
.ok_or(IntegrityError::Invalid("missing expected digest"))?;
copy_verified(api, job, &mut reader.stream, Some(&mut file), info.size, expected, guard).await?;
file.flush().await?;
copy_verified(api, job, &mut reader.stream, file.as_mut(), info.size, expected, guard).await?;
if let Some(file) = file.as_mut() {
file.flush().await?;
}
let after = api.get_object_info(&job.bucket, &job.request.items[index].key, opts).await?;
if fingerprint(&after) != fingerprint(info) {
return Err(IntegrityError::Conflict);
@@ -262,6 +262,20 @@ async fn stage(
Ok(file)
}
async fn staging_file(
mode: JobMode,
create: impl FnOnce() -> std::io::Result<std::fs::File> + Send + 'static,
) -> Result<Option<tokio::fs::File>> {
// Migration must publish the exact bytes verified here. Audit has no second consumer.
if mode == JobMode::Audit {
return Ok(None);
}
let file = tokio::task::spawn_blocking(create)
.await
.map_err(|_| IntegrityError::Invalid("staging task failed"))??;
Ok(Some(tokio::fs::File::from_std(file)))
}
async fn copy_verified(
api: &Arc<ECStore>,
job: &Job,
@@ -466,3 +480,31 @@ impl tokio::io::AsyncRead for ThrottledFile {
}
}
}
#[cfg(test)]
mod staging_tests {
use super::*;
#[tokio::test]
async fn audit_does_not_need_a_writable_payload_staging_volume() {
let result = staging_file(JobMode::Audit, || Err(std::io::Error::other("staging volume full")))
.await
.expect("audit must not open staging storage");
assert!(result.is_none());
}
#[tokio::test]
async fn migration_requires_writable_payload_staging() {
assert!(
staging_file(JobMode::Migrate, || { Err(std::io::Error::other("staging volume full")) })
.await
.is_err()
);
let mut file = staging_file(JobMode::Migrate, tempfile::tempfile).await.unwrap().unwrap();
file.write_all(b"verified bytes").await.unwrap();
file.rewind().await.unwrap();
let mut bytes = Vec::new();
file.read_to_end(&mut bytes).await.unwrap();
assert_eq!(bytes, b"verified bytes");
}
}
@@ -160,6 +160,7 @@ fn to_madmin_scanner_metrics(metrics: rustfs_scanner_metrics::metrics::ScannerMe
last_cycle_lifecycle_transition_actions: metrics.last_cycle_lifecycle_transition_actions,
last_cycle_partial_source: metrics.last_cycle_partial_source,
last_cycle_partial_source_code: metrics.last_cycle_partial_source_code,
last_cycle_partial_stage: metrics.last_cycle_partial_stage,
last_cycle_duration_seconds: metrics.last_cycle_duration_seconds,
last_cycle_objects_scanned: metrics.last_cycle_objects_scanned,
last_cycle_directories_scanned: metrics.last_cycle_directories_scanned,
@@ -691,6 +692,7 @@ mod test {
current_started: chrono_to_jiff_timestamp(current_started),
last_cycle_partial_source: "usage".to_string(),
last_cycle_partial_source_code: 1,
last_cycle_partial_stage: Some("object_budget".into()),
cycle_timeout_total: 3,
cycle_recovery_required_total: 2,
cycle_last_progress_age: 17,
@@ -706,6 +708,7 @@ mod test {
assert_eq!(scanner.current_started, chrono_to_jiff_timestamp(current_started));
assert_eq!(scanner.last_cycle_partial_source, "usage");
assert_eq!(scanner.last_cycle_partial_source_code, 1);
assert_eq!(scanner.last_cycle_partial_stage.as_deref(), Some("object_budget"));
assert_eq!(scanner.cycle_timeout_total, 3);
assert_eq!(scanner.cycle_recovery_required_total, 2);
assert_eq!(scanner.cycle_last_progress_age, 17);
@@ -3038,7 +3038,7 @@ impl NotificationSys {
return Err(Error::other("scanner scoped dirty usage capability requires scoped acknowledgements"));
}
};
if by_host.insert(host.clone(), acknowledgement).is_some() {
if by_host.insert(host, acknowledgement).is_some() {
return Err(Error::other("duplicate scanner dirty usage acknowledgement target"));
}
}
@@ -3081,6 +3081,42 @@ impl NotificationSys {
Ok(true)
}
/// Query exactly the requested peers and buckets. Missing peers and old
/// protocol endpoints fail the query instead of fabricating clean entries.
pub async fn scanner_usage_watches(
&self,
requests: Vec<(String, uuid::Uuid, Vec<crate::runtime::usage_watch::ScannerUsageWatchRequest>)>,
) -> Result<Vec<(String, Vec<crate::runtime::usage_watch::ScannerUsageWatchObservation>)>> {
let clients = self
.peer_clients
.iter()
.flatten()
.map(|client| (client.grid_host.clone(), client.clone()))
.collect::<HashMap<_, _>>();
let mut seen = std::collections::HashSet::new();
let mut futures = Vec::with_capacity(requests.len());
for (host, owner, entries) in requests {
if !seen.insert(host.clone()) {
return Err(Error::other("duplicate usage watch peer"));
}
let client = clients
.get(&host)
.cloned()
.ok_or_else(|| Error::other("usage watch peer is not reachable"))?;
futures.push(async move {
let observations = timeout(SCANNER_ACTIVITY_PROBE_TIMEOUT, client.scanner_usage_watch(owner, entries))
.await
.map_err(|_| Error::other("usage watch peer timed out"))??;
Ok::<_, Error>((host, observations))
});
}
let mut results = Vec::with_capacity(futures.len());
for result in join_all(futures).await {
results.push(result?);
}
Ok(results)
}
pub async fn acknowledge_scanner_dirty_usage(&self, acknowledgements: Vec<ScannerDirtyUsageAcknowledgement>) -> Result<bool> {
let mut by_host = HashMap::with_capacity(acknowledgements.len());
for acknowledgement in acknowledgements {
@@ -3088,7 +3124,7 @@ impl NotificationSys {
ScannerDirtyUsageAcknowledgement::Generation { host, .. }
| ScannerDirtyUsageAcknowledgement::Scoped { host, .. } => host.clone(),
};
if by_host.insert(host.clone(), acknowledgement).is_some() {
if by_host.insert(host, acknowledgement).is_some() {
return Err(Error::other("duplicate scanner dirty usage acknowledgement target"));
}
}
@@ -802,7 +802,6 @@ impl ECStore {
let set = set.clone();
let rx = rx.clone();
let bucket = bucket.clone();
let entry_tasks = entry_tasks.clone();
let job = tokio::spawn(async move {
let list_rx = rx.clone();
+17
View File
@@ -3661,6 +3661,9 @@ impl TierConfigMgr {
}
}
}
if !rustfs_data_usage::is_scannable_tier_name(&tier_config.name) {
return Err(tier_invalid_config("Remote tier name cannot be represented in lifecycle accounting"));
}
self.ensure_generation_is_idle(&tier_config.name)?;
let tier_name = tier_config.name.clone();
if tier_name != tier_name.to_uppercase() {
@@ -8435,6 +8438,20 @@ mod tests {
assert_eq!(legacy_gcs.gcs.as_ref().expect("GCS payload should remain").name, "COLD-LEGACY");
}
#[tokio::test]
async fn add_rejects_names_that_scanner_cannot_account_for() {
let mut mgr = empty_mgr();
let long_name = "X".repeat(257);
for name in ["COLD\n", rustfs_data_usage::UNKNOWN_TIER, long_name.as_str()] {
let err = mgr
.add(build_s3_tier(name), true)
.await
.expect_err("tier name must fail admission");
assert_eq!(err.code, ERR_TIER_INVALID_CONFIG.code);
assert!(mgr.tiers.is_empty());
}
}
#[tokio::test]
async fn test_add_rejects_non_uppercase_name() {
let mut mgr = empty_mgr();
@@ -8144,9 +8144,10 @@ mod tests {
assert_eq!(calls.total(disk_call_counters::KIND_DISK_SHARD_OPEN), 4);
assert_eq!(setup.available_shards(), 4);
assert_eq!(setup.deferred_shards(), 4);
for index in 4..8 {
assert!(setup.attempted.len() >= 8);
for (index, attempted) in setup.attempted.iter().enumerate().take(8).skip(4) {
assert_eq!(calls.for_disk(disk_call_counters::KIND_DISK_SHARD_OPEN, index), 0);
assert!(!setup.attempted[index], "parity must stay unopened until needed");
assert!(!attempted, "parity must stay unopened until needed");
}
let (output, error) = local_batch_quorum_decode(setup, files[0].erasure.block_size).await;
assert!(error.is_none(), "healthy decode failed: {error:?}");
+18 -8
View File
@@ -5611,6 +5611,20 @@ impl SetDisks {
metadata_sys::find_bucket_incarnation_id_in(&self.ctx, bucket).await
}
/// Confirm deletion of this exact generation using immutable retirement evidence.
/// Missing directories or unavailable bucket metadata do not prove retirement.
///
/// # Errors
/// Returns an error when the instance owner or retirement record is unavailable,
/// malformed, or bound to a different deployment or generation.
pub async fn is_bucket_generation_retired(&self, bucket: &str, incarnation: Uuid) -> Result<bool> {
let store = metadata_sys::object_store_in(&self.ctx).await?;
if !Arc::ptr_eq(&store.ctx, &self.ctx) {
return Err(Error::other("bucket retirement metadata belongs to another instance"));
}
crate::bucket::retirement::is_retired(store, bucket, incarnation).await
}
/// Admit one short scanner cache publication under this set's instance
/// movement fence. The caller must hold the returned guard through its
/// final conditional cache write; no scan-round work belongs under it.
@@ -6045,14 +6059,10 @@ fn should_use_inline_fast_path_with_plan(
// a stale/corrupt marker cannot route an oversized payload into the
// in-memory decoder. The independent direct-memory reader remains capped
// at 128 KiB below.
if !plan.is_plain()
|| shape.object_size > usize::try_from(INLINE_FAST_PATH_MAX_OBJECT_SIZE).expect("inline fast path limit fits usize")
|| fi.erasure.data_blocks == 0
|| fi.erasure.block_size == 0
{
return false;
}
true
plan.is_plain()
&& shape.object_size <= usize::try_from(INLINE_FAST_PATH_MAX_OBJECT_SIZE).expect("inline fast path limit fits usize")
&& fi.erasure.data_blocks != 0
&& fi.erasure.block_size != 0
}
enum SmallWritePath {
@@ -27,6 +27,8 @@ use std::collections::{BTreeMap, BTreeSet, HashSet};
use std::sync::Arc;
use uuid::Uuid;
mod operator;
const DISCOVERY_PAGE_SIZE: u32 = 128;
const MAX_DISCOVERED_TRANSACTIONS: usize = 1_000_000;
/// Inline cleanup attempts after a certified decision. Later attempts start
@@ -227,6 +229,18 @@ fn validate(inspection: &DeleteRecoveryInspection, object: &str, transaction: Uu
Ok(())
}
fn explicit_legacy_copy_safe(copy: &DeleteRecoveryInspection) -> bool {
copy.current_digest.is_none()
&& match copy.state {
DeleteRecoveryState::Absent => true,
DeleteRecoveryState::LegacyAmbiguous => copy.explicit_delete_safe,
DeleteRecoveryState::QuorumCommitted | DeleteRecoveryState::CommitFinished => {
copy.journal.as_ref().is_some_and(|journal| journal.target_digest.is_none())
}
DeleteRecoveryState::NewPrepared => false,
}
}
async fn inspect(
disks: &[Option<DiskStore>],
bucket: &str,
@@ -263,6 +277,15 @@ async fn discover_bounded(
) -> Result<Vec<DeleteTransactionId>> {
let admitted = budget.admit(crate::disk::DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES)?;
let page = disk.discover_delete_transactions_bounded(bucket, object, admitted).await?;
bounded_discovery_ids(page, object, admitted, budget)
}
fn bounded_discovery_ids(
page: crate::disk::BoundedDeleteRecoveryDiscovery,
object: &str,
admitted: u32,
budget: &crate::object_api::DeleteRecoveryWorkBudget,
) -> Result<Vec<DeleteTransactionId>> {
if page.visited > admitted
|| page.transactions.len() > 256
|| page
@@ -515,7 +538,7 @@ impl SetDisks {
let owner = context
.namespace_owner
.take()
.unwrap_or_else(|| self.ctx.begin_namespace_commit());
.unwrap_or_else(|| self.ctx.begin_namespace_commit_for_bucket(bucket));
context.namespace_owner = Some(Arc::new((owner, Arc::clone(&transaction_guard))));
let can_commit = context.can_commit;
let can_try_local_fast_path = disks.len() == self.set_drive_count
@@ -791,7 +814,7 @@ impl SetDisks {
let namespace_owner = context
.namespace_owner
.clone()
.unwrap_or_else(|| self.ctx.begin_namespace_commit());
.unwrap_or_else(|| self.ctx.begin_namespace_commit_for_bucket(bucket));
context.namespace_owner = Some(Arc::new((namespace_owner, guard.clone())));
}
let can_commit = context.can_commit;
@@ -821,7 +844,7 @@ impl SetDisks {
let owner = context
.namespace_owner
.take()
.unwrap_or_else(|| self.ctx.begin_namespace_commit());
.unwrap_or_else(|| self.ctx.begin_namespace_commit_for_bucket(bucket));
context.namespace_owner = Some(Arc::new((owner, Arc::clone(&transaction_guard))));
}
let DeleteRecoveryContext {
@@ -983,18 +1006,7 @@ impl SetDisks {
// A fresh authorized removal may discard safe legacy residue beside
// certified absence receipts. Every participant must have no live metadata;
// the receipts remain behind the key-wide barrier until all groups settle.
let explicit_legacy_removal = explicit_delete
&& all_states.values().flatten().all(|copy| {
copy.current_digest.is_none()
&& match copy.state {
DeleteRecoveryState::Absent => true,
DeleteRecoveryState::LegacyAmbiguous => copy.explicit_delete_safe,
DeleteRecoveryState::QuorumCommitted | DeleteRecoveryState::CommitFinished => {
copy.journal.as_ref().is_some_and(|journal| journal.target_digest.is_none())
}
DeleteRecoveryState::NewPrepared => false,
}
});
let explicit_legacy_removal = explicit_delete && all_states.values().flatten().all(explicit_legacy_copy_safe);
// This is a fresh authorized delete of an absent unversioned key,
// rather than evidence about the outcome of a historical transaction.
let fresh_absence_decision = explicit_delete
@@ -1025,7 +1037,7 @@ impl SetDisks {
let namespace_owner = if dry_run {
None
} else {
Some(namespace_owner.unwrap_or_else(|| self.ctx.begin_namespace_commit()))
Some(namespace_owner.unwrap_or_else(|| self.ctx.begin_namespace_commit_for_bucket(bucket)))
};
if !dry_run && metadata_may_change {
self.invalidate_get_object_metadata_cache(bucket, object).await;
@@ -3102,13 +3114,9 @@ mod tests {
);
}
#[tokio::test]
async fn delete_recovery_set_legacy_requires_explicit_removal() {
let bucket = "recover-legacy-remove";
let (dirs, store, disks) = fixture(bucket).await;
put(&store, bucket, &vec![42; 4096], false).await;
async fn interrupt_legacy_delete(disks: &[DiskStore], bucket: &str) -> Uuid {
let transaction = Uuid::new_v4();
for disk in &disks {
for disk in disks {
disk.delete_version(
bucket,
"object",
@@ -3125,6 +3133,229 @@ mod tests {
.await
.expect("create undecided historical delete shape");
}
transaction
}
#[tokio::test]
async fn delete_recovery_operator_discard_requires_complete_unchanged_evidence() {
let bucket = "recover-operator-proof";
let (dirs, store, disks) = fixture(bucket).await;
put(&store, bucket, &vec![42; 4096], false).await;
let transaction = interrupt_legacy_delete(&disks, bucket).await;
let root = dirs[0].path().join(bucket).join("object");
let backup = root.join(transaction.to_string()).join("xl.meta.bkp");
let original = std::fs::read(&backup).expect("historical bytes");
let preview = store
.inspect_delete_residue(bucket, "object", 0, 0)
.await
.expect("inspect without writes");
assert!(preview.complete, "{preview:?}");
assert_eq!(preview.transactions.len(), 1);
assert_eq!(preview.participants.len(), 4);
assert_eq!(preview.transactions[0].copies.len(), 4);
let proof = preview.fingerprint.expect("eligible legacy layout");
assert_eq!(proof.len(), 64);
let sibling = root.join(Uuid::new_v4().to_string());
std::fs::create_dir(&sibling).expect("introduce a later sibling");
std::fs::write(sibling.join("xl.meta.bkp"), &original).expect("seed sibling evidence");
assert!(matches!(
store.discard_delete_residue(bucket, "object", 0, 0, &proof).await,
Err(StorageError::PreconditionFailed)
));
assert_eq!(std::fs::read(&backup).expect("first sibling retained"), original);
assert_eq!(std::fs::read(sibling.join("xl.meta.bkp")).expect("new sibling retained"), original);
std::fs::remove_dir_all(sibling).expect("restore fixture to previewed evidence");
let copy = disks[0]
.inspect_delete_transaction(bucket, "object", transaction)
.await
.expect("disk evidence");
let wrong_disk = disks[1].get_disk_id().await.expect("other disk").expect("formatted");
for transaction in [None, Some(transaction)] {
assert!(matches!(
disks[0]
.read_delete_residue_on_disk(bucket, "object", wrong_disk, transaction, 4096)
.await,
Err(DiskError::OutdatedXLMeta)
));
}
assert!(matches!(
disks[0]
.settle_delete_transaction(
bucket,
"object",
transaction,
copy.fingerprint,
DeleteRecoveryAction::DiscardLegacyOnDisk { disk_id: wrong_disk }
)
.await,
Err(DiskError::OutdatedXLMeta)
));
assert_eq!(std::fs::read(&backup).expect("wrong disk retained all evidence"), original);
// An actual missing slot invalidates the same fingerprint; the preview reports
// the incomplete per-drive picture even though the key is visible on a quorum.
let set = &store.pools[0].disk_set[0];
let offline = set.disks.write().await[3].take();
let incomplete = store
.inspect_delete_residue(bucket, "object", 0, 0)
.await
.expect("partial evidence");
assert!(!incomplete.complete && incomplete.fingerprint.is_none());
assert!(matches!(
store.discard_delete_residue(bucket, "object", 0, 0, &proof).await,
Err(StorageError::PreconditionFailed)
));
set.disks.write().await[3] = offline;
// A live copy appearing after the preview must not become a fresh ordinary DELETE.
let current = root.join("xl.meta");
std::fs::write(&current, &original).expect("publish a live generation");
let live = store
.inspect_delete_residue(bucket, "object", 0, 0)
.await
.expect("inspect current object");
assert!(live.fingerprint.is_none());
assert!(matches!(
store.discard_delete_residue(bucket, "object", 0, 0, &proof).await,
Err(StorageError::PreconditionFailed)
));
assert_eq!(std::fs::read(&current).expect("live object untouched"), original);
std::fs::remove_file(current).expect("restore absent fixture");
let proof = store
.inspect_delete_residue(bucket, "object", 0, 0)
.await
.expect("fresh preview")
.fingerprint
.expect("eligible");
let discarded = store
.discard_delete_residue(bucket, "object", 0, 0, &proof)
.await
.expect("authorized exact discard");
assert_eq!(discarded.reclaimed, 1);
assert!(discarded.fingerprint.is_none());
assert!(dirs.iter().all(|dir| {
!dir.path()
.join(bucket)
.join("object")
.join(transaction.to_string())
.join("xl.meta.bkp")
.exists()
}));
assert!(matches!(
store.discard_delete_residue(bucket, "object", 0, 0, &proof).await,
Err(StorageError::PreconditionFailed)
));
}
#[tokio::test]
async fn delete_recovery_operator_rejects_versioning_and_object_lock_after_preview() {
use crate::bucket::metadata::{BUCKET_VERSIONING_CONFIG, OBJECT_LOCK_CONFIG};
for (bucket, config, xml) in [
(
"recover-operator-enabled",
BUCKET_VERSIONING_CONFIG,
"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>",
),
(
"recover-operator-suspended",
BUCKET_VERSIONING_CONFIG,
"<VersioningConfiguration><Status>Suspended</Status></VersioningConfiguration>",
),
(
"recover-operator-lock",
OBJECT_LOCK_CONFIG,
"<ObjectLockConfiguration><ObjectLockEnabled>Enabled</ObjectLockEnabled></ObjectLockConfiguration>",
),
] {
let (dirs, store, disks) = fixture(bucket).await;
put(&store, bucket, &vec![21; 4096], false).await;
let transaction = interrupt_legacy_delete(&disks, bucket).await;
let backup = dirs[0]
.path()
.join(bucket)
.join("object")
.join(transaction.to_string())
.join("xl.meta.bkp");
let original = std::fs::read(&backup).expect("historical bytes");
let pre = store
.inspect_delete_residue(bucket, "object", 0, 0)
.await
.expect("first inspection");
let proof = pre.fingerprint.expect("eligible before bucket config change");
crate::bucket::metadata_sys::update_in(&store.ctx, bucket, config, xml.as_bytes().to_vec())
.await
.expect("publish bucket protection");
let blocked = store
.inspect_delete_residue(bucket, "object", 0, 0)
.await
.expect("inspect protected residue");
assert_eq!(blocked.reason, "bucket_protected");
assert!(blocked.fingerprint.is_none());
assert!(matches!(
store.discard_delete_residue(bucket, "object", 0, 0, &proof).await,
Err(StorageError::PreconditionFailed)
));
assert_eq!(std::fs::read(backup).expect("protected evidence retained"), original);
}
}
#[tokio::test]
async fn delete_recovery_operator_preview_does_not_survive_bucket_recreation() {
let bucket = "recover-operator-recreate";
let (dirs, store, disks) = fixture(bucket).await;
put(&store, bucket, b"stable bytes", false).await;
interrupt_legacy_delete(&disks, bucket).await;
let old = store
.inspect_delete_residue(bucket, "object", 0, 0)
.await
.expect("old generation");
let old_proof = old.fingerprint.expect("historical shape eligible");
// Preserve every original byte and drive slot while changing only bucket identity.
for dir in &dirs {
std::fs::rename(dir.path().join(bucket).join("object"), dir.path().join("saved-residue"))
.expect("hold original evidence");
}
store
.delete_bucket(bucket, &DeleteBucketOptions::default())
.await
.expect("delete empty original bucket");
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("recreate bucket");
for dir in &dirs {
std::fs::rename(dir.path().join("saved-residue"), dir.path().join(bucket).join("object"))
.expect("restore original evidence");
}
let new = store
.inspect_delete_residue(bucket, "object", 0, 0)
.await
.expect("new generation");
assert_ne!(old.incarnation, new.incarnation);
assert_eq!(old.transactions, new.transactions);
assert_ne!(new.fingerprint.as_deref(), Some(old_proof.as_str()));
assert!(matches!(
store.discard_delete_residue(bucket, "object", 0, 0, &old_proof).await,
Err(StorageError::PreconditionFailed)
));
assert_eq!(
store
.inspect_delete_residue(bucket, "object", 0, 0)
.await
.expect("unchanged evidence"),
new
);
}
#[tokio::test]
async fn delete_recovery_set_legacy_requires_explicit_removal() {
let bucket = "recover-legacy-remove";
let (dirs, store, disks) = fixture(bucket).await;
put(&store, bucket, &vec![42; 4096], false).await;
let transaction = interrupt_legacy_delete(&disks, bucket).await;
let report = store
.recover_delete_transactions(bucket, &HealOpts::default())
.await
@@ -0,0 +1,275 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Read-only exact-key evidence and conditional legacy-only disposal for administrators.
use super::*;
use crate::disk::{DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES, DELETE_RECOVERY_MAX_INSPECTION_ENTRIES, DeleteRecoveryRpcReply};
use crate::error::Result as StorageResult;
use crate::object_api::DeleteRecoveryWorkBudget;
use rustfs_madmin::heal_commands::{
DeleteResidueCopy, DeleteResidueInspection, DeleteResidueParticipant, DeleteResidueTransaction,
};
use sha2::{Digest, Sha256};
fn status(error: &DiskError) -> &'static str {
if unsupported_recovery(error) {
"unsupported"
} else {
match error {
DiskError::DeleteRecoveryWorkLimit => "work_limit",
DiskError::FileCorrupt => "corrupt",
_ if unavailable(error) => "unavailable",
_ => "failed",
}
}
}
fn hex_digest(bytes: &[u8]) -> String {
hex_simd::encode_to_string(bytes, hex_simd::AsciiCase::Lower)
}
fn evidence_copy(slot: usize, inspection: &DeleteRecoveryInspection) -> DeleteResidueCopy {
let state = match inspection.state {
DeleteRecoveryState::Absent => "absent",
DeleteRecoveryState::NewPrepared => "prepared",
DeleteRecoveryState::QuorumCommitted => "committed",
DeleteRecoveryState::CommitFinished => "finished",
DeleteRecoveryState::LegacyAmbiguous => "legacy_ambiguous",
};
DeleteResidueCopy {
slot,
state: state.to_owned(),
current_metadata_present: Some(inspection.current_digest.is_some()),
backup_present: Some(inspection.backup_digest.is_some()),
journal_present: Some(inspection.journal.is_some()),
fingerprint: Some(hex_digest(&inspection.fingerprint)),
}
}
impl SetDisks {
/// Retains the original exact-key and transaction guards through the discard write tail.
pub(crate) async fn operator_delete_residue(
&self,
object: &str,
mut report: DeleteResidueInspection,
expected: Option<&str>,
bucket_allows_discard: bool,
expected_disks: &[Uuid],
can_commit: &(dyn Fn() -> bool + Sync),
) -> StorageResult<DeleteResidueInspection> {
let bucket = &report.bucket;
// Lock order matches normal recovery: caller's lifecycle and movement fences,
// raw object write lock, then its set-local transaction lock.
let object_guard = Arc::new(self.acquire_write_lock_diag("inspect_delete_residue", bucket, object).await?);
let transaction_guard = self.acquire_delete_transaction_lock(bucket, object).await?;
let fence = || can_commit() && !object_guard.is_lock_lost() && !transaction_guard.is_lock_lost();
let disks = self.disk_inventory().await;
let budget = DeleteRecoveryWorkBudget::for_disks(self.set_drive_count);
let pages = join_all(disks.iter().map(|disk| async {
let disk = disk.as_ref().ok_or(DiskError::DiskNotFound)?;
let id = disk.probe_delete_residue_disk(bucket).await?;
let admitted = budget.admit(DELETE_RECOVERY_MAX_DISCOVERY_ENTRIES)?;
let page = match disk.read_delete_residue_on_disk(bucket, object, id, None, admitted).await {
Ok(DeleteRecoveryRpcReply::DiscoveredBounded(page)) => bounded_discovery_ids(page, object, admitted, &budget),
Ok(_) => Err(DiskError::FileCorrupt),
Err(error) => Err(error),
};
Ok::<_, DiskError>((id, page))
}))
.await;
let mut ids = BTreeSet::new();
let mut disk_ids = HashSet::new();
report.complete = disks.len() == self.set_drive_count && !disks.is_empty() && expected_disks.len() == disks.len();
for (slot, page) in pages.into_iter().enumerate() {
let (disk_id, discovery) = match page {
Ok((id, Ok(page))) if expected_disks.get(slot) == Some(&id) && disk_ids.insert(id) => {
ids.extend(page.into_iter().map(|item| item.transaction));
(Some(id.to_string()), "complete")
}
Ok((id, Ok(_))) => (Some(id.to_string()), "disk_identity_mismatch"),
Ok((id, Err(error))) => (Some(id.to_string()), status(&error)),
Err(error) => (None, status(&error)),
};
report.complete &= discovery == "complete";
report.participants.push(DeleteResidueParticipant {
slot,
disk_id,
discovery: discovery.to_owned(),
});
}
let mut groups = BTreeMap::new();
let mut legacy = false;
let mut valid_generation = true;
// A bounded exact-key discovery admits at most 256 records per disk.
// A severely divergent set cannot multiply that bound into an unbounded response.
let truncated = ids.len() > 256;
report.complete &= !truncated;
for id in ids.into_iter().take(256) {
let observations = join_all(disks.iter().enumerate().map(|(slot, disk)| {
let budget = &budget;
async move {
let disk = disk.as_ref().ok_or(DiskError::DiskNotFound)?;
let physical_id = *expected_disks.get(slot).ok_or(DiskError::InconsistentDisk)?;
let admitted = budget.admit(DELETE_RECOVERY_MAX_INSPECTION_ENTRIES)?;
let DeleteRecoveryRpcReply::InspectedBounded(reply) = disk
.read_delete_residue_on_disk(bucket, object, physical_id, Some(id), admitted)
.await?
else {
return Err(DiskError::FileCorrupt);
};
if reply.visited > admitted {
return Err(DiskError::FileCorrupt);
}
budget.charge(reply.visited);
validate(&reply.inspection, object, id)?;
Ok(reply.inspection)
}
}))
.await;
let mut copies = Vec::with_capacity(observations.len());
let mut states = Vec::with_capacity(observations.len());
for (slot, observation) in observations.into_iter().enumerate() {
match observation {
Ok(mut state) => {
legacy |= state.state == DeleteRecoveryState::LegacyAmbiguous;
valid_generation &= state.journal.as_ref().is_none_or(|journal| {
journal
.bucket_incarnation
.is_some_and(|incarnation| incarnation.to_string() == report.incarnation)
});
copies.push(evidence_copy(slot, &state));
// Only the causal edges, classification and fingerprints survive
// this iteration. Never retain full version histories or request
// payloads across the bounded list of sibling transactions.
state.current_versions = Vec::new();
if let Some(journal) = &mut state.journal {
journal.requests = Vec::new();
journal.original_versions = Vec::new();
journal.data_dirs = Vec::new();
journal.deleted_versions = Vec::new();
}
states.push(state);
}
Err(error) => {
report.complete = false;
copies.push(DeleteResidueCopy {
slot,
state: status(&error).to_owned(),
current_metadata_present: None,
backup_present: None,
journal_present: None,
fingerprint: None,
});
}
}
}
report.transactions.push(DeleteResidueTransaction {
id: id.to_string(),
copies,
});
groups.insert(id, states);
}
let reason = if truncated {
"work_limit"
} else if !report.complete {
"incomplete"
} else if !legacy {
"no_legacy_residue"
} else if !bucket_allows_discard {
"bucket_protected"
} else if !valid_generation || causal_transaction_order(&groups).is_err() {
"conflicting_history"
} else if groups.values().flatten().any(|copy| copy.current_digest.is_some()) {
"current_object_present"
} else if !groups.values().flatten().all(explicit_legacy_copy_safe) {
"unsafe_layout"
} else {
"discard_eligible"
};
report.reason = reason.to_owned();
if !fence() {
return Err(StorageError::PreconditionFailed);
}
if reason == "discard_eligible" {
// Fixed-order structs and sorted transaction UUIDs cover the exact action,
// logical/physical identities, all siblings and every disk fingerprint.
let serialized = serde_json::to_vec(&("rustfs:discard-legacy:v1", &report)).map_err(StorageError::other)?;
report.fingerprint = Some(hex_digest(&Sha256::digest(serialized)));
}
let Some(expected) = expected else { return Ok(report) };
if report.fingerprint.as_deref() != Some(expected) {
return Err(StorageError::PreconditionFailed);
}
let owner: Arc<dyn Send + Sync> = Arc::new((
self.ctx.begin_namespace_commit_for_bucket(bucket),
object_guard.clone(),
transaction_guard.clone(),
));
let work = RecoveryWorkScope {
namespace_owner: Some(&owner),
budget: Some(&budget),
};
for copies in groups
.values()
.filter(|copies| copies.iter().any(|copy| copy.state == DeleteRecoveryState::LegacyAmbiguous))
{
// Compare both the physical disk ID and its original evidence on the owning node.
// A late reply or a partial failure requires a new preview of the remaining evidence.
let results = join_all(
copies
.iter()
.enumerate()
.filter(|(_, copy)| copy.state == DeleteRecoveryState::LegacyAmbiguous)
.map(|(slot, copy)| {
let fence = &fence;
let disks = &disks;
async move {
let disk = disks[slot].as_ref().ok_or(DiskError::DiskNotFound)?;
settle(
disk,
bucket,
copy,
DeleteRecoveryAction::DiscardLegacyOnDisk {
disk_id: expected_disks[slot],
},
fence,
work,
)
.await
}
}),
)
.await;
for result in results {
match result {
Err(DiskError::DiskOngoingReq | DiskError::OutdatedXLMeta) => return Err(StorageError::PreconditionFailed),
Ok(next) if next.state == DeleteRecoveryState::Absent && next.current_digest.is_none() => {}
Ok(_) => return Err(StorageError::PreconditionFailed),
Err(error) => return Err(error.into()),
}
}
if !fence() {
return Err(StorageError::PreconditionFailed);
}
report.reclaimed += 1;
}
report.reason = "discarded".to_owned();
report.fingerprint = None;
Ok(report)
}
}
+16 -71
View File
@@ -1285,7 +1285,7 @@ impl SetDisks {
result.after.drives.push(HealDriveInfo {
uuid: "".to_string(),
endpoint: self.set_endpoints[index].to_string(),
state: drive_state.to_string(),
state: drive_state,
});
}
@@ -2060,7 +2060,7 @@ impl SetDisks {
_write_lock_guard.clone(),
_read_repair_write_lock_guard.clone(),
bucket_heal_scope.clone(),
self.ctx.begin_namespace_commit(),
self.ctx.begin_namespace_commit_for_bucket(bucket),
))
});
match disk
@@ -3102,7 +3102,7 @@ impl SetDisks {
result.after.drives.push(HealDriveInfo {
uuid: "".to_string(),
endpoint,
state: drive_state.to_string(),
state: drive_state,
});
}
@@ -3147,7 +3147,7 @@ impl SetDisks {
_ => DriveState::Corrupt.to_string(),
},
};
result.after.drives[index].state = drive_state.to_string();
result.after.drives[index].state = drive_state;
}
}
@@ -3761,7 +3761,7 @@ impl SetDisks {
namespace_owner: (!opts.dry_run).then(|| {
Arc::new((
scope.clone(),
self.ctx.begin_namespace_commit(),
self.ctx.begin_namespace_commit_for_bucket(bucket),
_write_lock_guard.clone(),
transaction_guard.clone(),
)) as Arc<dyn Send + Sync>
@@ -7079,71 +7079,16 @@ mod heal_result_report_tests {
#[tokio::test]
#[serial_test::serial]
async fn heal_object_with_queued_disk_renewal() {
use crate::layout::endpoints::SetupType;
use crate::runtime::instance::InstanceContext;
use crate::set_disk::core::io_primitives::disk_call_counters;
use std::collections::HashMap;
use std::future::Future;
use std::task::Poll;
use std::time::Duration;
use tokio::io::AsyncReadExt;
// renew_disk still registers local disks on the ambient context. Match
// the default serial group used by its other setup/registry fixtures,
// and restore only this temporary endpoint, including on a failed join.
struct RenewDiskTestState {
ctx: Arc<InstanceContext>,
was_dist_erasure: bool,
map: Arc<RwLock<HashMap<String, Option<DiskStore>>>>,
endpoint: String,
previous_disk: Option<Option<DiskStore>>,
}
impl Drop for RenewDiskTestState {
fn drop(&mut self) {
let ctx = self.ctx.clone();
let was_dist_erasure = self.was_dist_erasure;
let map = self.map.clone();
let endpoint = self.endpoint.clone();
let previous_disk = self.previous_disk.take();
let handle = tokio::runtime::Handle::current();
std::thread::spawn(move || {
handle.block_on(async move {
let mut map = map.write().await;
match previous_disk {
Some(disk) => {
map.insert(endpoint, disk);
}
None => {
map.remove(&endpoint);
}
}
drop(map);
if was_dist_erasure {
ctx.update_erasure_type(SetupType::DistErasure).await;
}
});
})
.join()
.expect("renew fixture state restoration should finish");
}
}
let (_temp_dirs, disks, set) = hermetic_set_disks_isolated(4).await;
let endpoint = set.set_endpoints[0].clone();
let ctx = crate::runtime::global::current_ctx();
let map = ctx.local_disk_map();
let restore = RenewDiskTestState {
ctx: ctx.clone(),
was_dist_erasure: ctx.is_dist_erasure().await,
map: map.clone(),
endpoint: endpoint.to_string(),
previous_disk: map.read().await.get(&endpoint.to_string()).cloned(),
};
// Only distributed erasure needs an override to avoid the ambient slot array.
if restore.was_dist_erasure {
ctx.update_erasure_type(SetupType::Erasure).await;
}
crate::runtime::sources::record_local_disks(&set.ctx, disks.clone()).await;
let map = set.ctx.local_disk_map();
let bucket = "heal-disk-renewal-bucket";
let object = "heal-disk-renewal-object";
@@ -7209,14 +7154,6 @@ mod heal_result_report_tests {
)
.await
.expect("real renewal must queue its topology writer behind the read gate");
let registered = map
.read()
.await
.get(&endpoint.to_string())
.cloned()
.flatten()
.expect("renewal must register the connected disk before its inventory write");
assert!(!Arc::ptr_eq(&registered, &disks[0]), "renewal must construct a new disk handle");
tokio::time::timeout(Duration::from_secs(5), async {
while calls.total(disk_call_counters::KIND_READ_VERSION) < 4 {
tokio::task::yield_now().await;
@@ -7240,6 +7177,14 @@ mod heal_result_report_tests {
let renewed = set.get_disks_internal().await[0]
.clone()
.expect("the renewed slot must remain online");
let registered = map
.read()
.await
.get(&endpoint.to_string())
.cloned()
.flatten()
.expect("renewed RPC route");
assert!(!Arc::ptr_eq(&registered, &disks[0]), "renewal must replace the original handle");
assert!(Arc::ptr_eq(&renewed, &registered), "the set must publish the newly connected handle");
assert_eq!(renewed.endpoint(), endpoint);
let format = load_format_erasure(&renewed, false)
@@ -10994,7 +10939,7 @@ mod heal_result_report_tests {
set.heal_object(bucket, object, "", &heal_opts),
);
let put_info = put_res.expect("overwrite must succeed under lock serialization");
last_etag = put_info.etag.clone().unwrap_or_default();
last_etag = put_info.etag.unwrap_or_default();
// Heal outcome is unconstrained (may hit the tail race and report a
// retryable error); the invariant is checked on the end state.
drop(heal_res);
+167 -76
View File
@@ -31,7 +31,6 @@ use futures::{StreamExt, stream, stream::FuturesUnordered};
use rand::prelude::SliceRandom;
use rustfs_heal_contracts::heal_channel::{create_auto_replacement_disks_request, send_heal_request};
use rustfs_utils::path::object_name_disk_identity;
#[cfg(test)]
use uuid::Uuid;
// Bound batching when another reconnect is still pending.
@@ -446,7 +445,7 @@ impl SetDisks {
// Reconnect drives concurrently without holding the inventory lock.
let mut reconnects = stream::iter(disks.into_iter().enumerate())
.map(|(i, opdisk)| async move {
if let Some(disk) = opdisk {
if let Some(disk) = &opdisk {
if !active_monitoring
&& disk.get_disk_location().set_idx.is_some()
&& let disk::Disk::Local(local) = disk.as_ref()
@@ -465,7 +464,9 @@ impl SetDisks {
}
let endpoint = self.set_endpoints.get(i)?;
self.reconnect_disk(endpoint).await.then(|| endpoint.to_string())
self.reconnect_disk(endpoint, opdisk.as_ref())
.await
.then(|| endpoint.to_string())
})
.buffer_unordered(MAX_ERASURE_SET_DRIVE_COUNT);
let mut submissions = FuturesUnordered::new();
@@ -515,26 +516,22 @@ impl SetDisks {
}
pub async fn renew_disk(&self, ep: &Endpoint) {
if self.reconnect_disk(ep).await {
let Some(disk_idx) = self.set_endpoints.iter().position(|endpoint| endpoint == ep) else {
return;
};
let previous = self.disks.read().await[disk_idx].clone();
if self.reconnect_disk(ep, previous.as_ref()).await {
self.submit_replacement_heal(vec![ep.to_string()]).await;
}
}
/// Return whether this reconnect attached a verified unformatted target.
async fn reconnect_disk(&self, ep: &Endpoint) -> bool {
async fn reconnect_disk(&self, ep: &Endpoint, previous: Option<&DiskStore>) -> bool {
debug!("renew_disk: start {:?}", ep);
let previous_disk = {
let disks = self.disks.read().await;
disks
.iter()
.filter_map(|disk| disk.as_ref())
.find(|disk| disk.endpoint() == *ep)
.cloned()
};
let previous_health = previous_disk
.as_ref()
.and_then(|disk| disk.local_health_tracker_epoch_for_reconnect());
if previous.is_some_and(|disk| disk.endpoint() != *ep) {
return false;
}
let previous_health = previous.and_then(|disk| disk.local_health_tracker_epoch_for_reconnect());
let (new_disk, fm) = match Self::connect_endpoint(ep, previous_health).await {
Ok(res) => res,
@@ -544,7 +541,7 @@ impl SetDisks {
return false;
}
let attached = match self.attach_unformatted_replacement_disk(ep).await {
let attached = match self.attach_unformatted_replacement_disk(ep, previous).await {
Ok(attached) => attached,
Err(err) => {
warn!(endpoint = %ep, error = ?err, "renew_disk: unformatted replacement probe failed");
@@ -580,21 +577,18 @@ impl SetDisks {
return false;
}
let _ = new_disk.set_disk_id(Some(fm.erasure.this)).await;
if !new_disk.is_local() {
if let disk::Disk::Local(local) = new_disk.as_ref() {
local.set_disk_id_state(Some(fm.erasure.this)).await;
self.publish_local_reconnect(disk_idx, previous, new_disk, Some(fm.erasure.this))
.await;
} else {
let _ = new_disk.set_disk_id(Some(fm.erasure.this)).await;
let displaced = {
let mut disks = self.disks.write().await;
let slot = &mut disks[disk_idx];
let unchanged = match (slot.as_ref(), previous_disk.as_ref()) {
(None, None) => true,
(Some(current), Some(previous)) => Arc::ptr_eq(current, previous),
_ => false,
};
// A GET probe or another reconnect may have filled this slot while
// format I/O was pending. Only the successful publisher owns the
// displaced monitor; a failed or superseded probe must retain it.
if unchanged {
// format I/O was pending. Only the winner owns the displaced monitor.
if Self::same_disk_handle(slot.as_ref(), previous) {
new_disk.enable_health_check();
slot.replace(new_disk)
} else {
@@ -604,36 +598,79 @@ impl SetDisks {
if let Some(disk) = displaced {
let _ = disk.close().await;
}
}
false
}
fn same_disk_handle(current: Option<&DiskStore>, expected: Option<&DiskStore>) -> bool {
match (current, expected) {
(None, None) => true,
(Some(current), Some(expected)) => Arc::ptr_eq(current, expected),
_ => false,
}
}
/// Commit a freshly probed attachment to its set and all local RPC routes together.
async fn publish_local_reconnect(
&self,
disk_idx: usize,
previous: Option<&DiskStore>,
candidate: DiskStore,
format_id: Option<Uuid>,
) -> bool {
let disk::Disk::Local(local) = candidate.as_ref() else { return false };
let path = candidate.endpoint().to_string();
let dist = self.ctx.is_dist_erasure().await;
// A busy set must not block RPC lookups across the instance. Lock the set
// first, then follow the registry order used by startup and RPC registration:
// distributed drive table -> path map -> ID map.
let mut disks = self.disks.write().await;
let slot = &mut disks[disk_idx];
if !Self::same_disk_handle(slot.as_ref(), previous) {
return false;
}
new_disk.enable_health_check();
if new_disk.is_local() {
let local_disk_map = runtime_sources::local_disk_map_handle();
let mut global_local_disk_map = local_disk_map.write().await;
let path = new_disk.endpoint().to_string();
global_local_disk_map.insert(path, Some(new_disk.clone()));
if runtime_sources::setup_is_dist_erasure().await {
let local_disk_set_drives = runtime_sources::local_disk_set_drives_handle();
let mut local_set_drives = local_disk_set_drives.write().await;
local_set_drives[self.pool_index][set_idx][disk_idx] = Some(new_disk.clone());
}
let table = self.ctx.local_disk_set_drives();
let mut table = if dist { Some(table.write().await) } else { None };
let route_slot = if let Some(table) = table.as_mut() {
let Some(slot) = table
.get_mut(self.pool_index)
.and_then(|pool| pool.get_mut(self.set_index))
.and_then(|set| set.get_mut(disk_idx))
else {
return false;
};
Some(slot)
} else {
None
};
let routes = self.ctx.local_disk_map();
let mut routes = routes.write().await;
if let Some(id) = format_id {
self.ctx.local_disk_id_map().write().await.insert(id, path.clone());
}
debug!("renew_disk: update {:?}", fm.erasure.this);
let mut disk_lock = self.disks.write().await;
disk_lock[disk_idx] = Some(new_disk);
false
// No await after publication begins: cancellation cannot expose half an attachment.
local.get_disk().usage_watch_binding.bind(&self.ctx.usage_watches);
candidate.enable_health_check();
if let Some(disk::Disk::Local(displaced)) = slot.as_deref() {
displaced.retire_handle();
}
routes.insert(path, Some(candidate.clone()));
if let Some(route_slot) = route_slot {
*route_slot = Some(candidate.clone());
}
*slot = Some(candidate);
true
}
/// Attach a replacement target only after proving that the exact local slot
/// is present and unformatted. A health-checked reconnect may reject a
/// blank target before it reaches the format-heal path; that target still
/// has to be visible in this set for the formatter to claim it safely.
async fn attach_unformatted_replacement_disk(&self, ep: &Endpoint) -> disk::error::Result<bool> {
async fn attach_unformatted_replacement_disk(
&self,
ep: &Endpoint,
previous: Option<&DiskStore>,
) -> disk::error::Result<bool> {
if !ep.is_local
|| usize::try_from(ep.pool_idx).ok() != Some(self.pool_index)
|| usize::try_from(ep.set_idx).ok() != Some(self.set_index)
@@ -659,28 +696,7 @@ impl SetDisks {
Err(err) => return Err(err),
}
{
let mut disks = self.disks.write().await;
if disks[disk_idx].as_ref().is_some_and(|existing| existing.endpoint() != *ep) {
warn!(endpoint = %ep, disk_idx, "renew_disk rejected unformatted replacement for an occupied foreign slot");
return Ok(false);
}
disks[disk_idx] = Some(replacement.clone());
}
let local_disk_map = runtime_sources::local_disk_map_handle();
local_disk_map
.write()
.await
.insert(replacement.endpoint().to_string(), Some(replacement.clone()));
if runtime_sources::setup_is_dist_erasure().await {
let local_disk_set_drives = runtime_sources::local_disk_set_drives_handle();
let mut local_set_drives = local_disk_set_drives.write().await;
local_set_drives[self.pool_index][self.set_index][disk_idx] = Some(replacement);
}
Ok(true)
Ok(self.publish_local_reconnect(disk_idx, previous, replacement, None).await)
}
pub(in crate::set_disk) fn find_disk_index(&self, fm: &FormatV3) -> Result<(usize, usize)> {
@@ -690,9 +706,10 @@ impl SetDisks {
return Err(Error::other("DriveID: offline"));
}
for i in 0..self.format.erasure.sets.len() {
for j in 0..self.format.erasure.sets[0].len() {
if fm.erasure.this == self.format.erasure.sets[i][j] {
let set_drive_count = self.format.erasure.sets[0].len();
for (i, set) in self.format.erasure.sets.iter().enumerate() {
for (j, drive) in set.iter().enumerate().take(set_drive_count) {
if fm.erasure.this == *drive {
return Ok((i, j));
}
}
@@ -1247,6 +1264,9 @@ mod tests {
.await
.expect("renewed disk_info should record a drive API metric");
renewed_disk.force_runtime_state_for_test(disk::health_state::RuntimeDriveHealthState::Offline);
assert!(
matches!(renewed_disk.as_ref(), disk::Disk::Local(local) if matches!(local.check_scanner_handle(), Err(DiskError::FaultyDisk)))
);
set_disks.renew_disk(&endpoints[0]).await;
@@ -1254,6 +1274,13 @@ mod tests {
let renewed_again = disks[0]
.as_ref()
.expect("second renew_disk should keep the recovered disk attached");
let (disk::Disk::Local(retired), disk::Disk::Local(current)) = (renewed_disk.as_ref(), renewed_again.as_ref()) else {
panic!("local reconnect fixture must retain a local slot");
};
assert!(matches!(retired.check_scanner_handle(), Err(DiskError::DiskNotFound)));
current
.check_scanner_handle()
.expect("reconnected scanner handle must be usable");
assert_eq!(
renewed_again
.metrics_snapshot()
@@ -1265,6 +1292,70 @@ mod tests {
drop(temp_dirs);
}
#[tokio::test]
async fn stale_local_reconnect_cannot_retire_the_published_rpc_source() {
use crate::layout::endpoints::SetupType;
use crate::runtime::instance::InstanceContext;
let format = FormatV3::new(1, 1);
let id = format.erasure.sets[0][0];
let (directory, endpoint, original) = make_formatted_local_disk(0, &format).await;
let path = endpoint.to_string();
let ctx = Arc::new(InstanceContext::new());
ctx.update_erasure_type(SetupType::DistErasure).await;
*ctx.local_disk_set_drives().write().await = vec![vec![vec![Some(original.clone())]]];
ctx.local_disk_map()
.write()
.await
.insert(path.clone(), Some(original.clone()));
let set = SetDisks::new_with_instance_ctx(
"local-reconnect-race".to_owned(),
Arc::new(RwLock::new(vec![Some(original.clone())])),
1,
0,
0,
0,
vec![endpoint.clone()],
format,
Vec::new(),
ctx.clone(),
)
.await;
// Two reconnects saw the same set slot. Finish one before allowing the
// other probe to publish, first with its format and then as a blank target.
let delayed = set.get_disks_internal().await[0].clone();
set.renew_disk(&endpoint).await;
let winner = set.get_disks_internal().await[0].clone().expect("published handle");
assert!(
original
.local_retirement_token()
.expect("original local handle")
.is_cancelled()
);
assert!(!Arc::ptr_eq(&original, &winner));
for blank in [false, true] {
if blank {
tokio::fs::remove_file(directory.path().join(disk::RUSTFS_META_BUCKET).join(disk::FORMAT_CONFIG_FILE))
.await
.expect("turn delayed source into a verified blank replacement");
}
assert!(
!set.reconnect_disk(&endpoint, delayed.as_ref()).await,
"superseded blank cannot schedule a heal"
);
let active = set.get_disks_internal().await[0].clone().expect("set slot");
let route = ctx.local_disk_map().read().await[&path].clone().expect("RPC route");
let topology = ctx.local_disk_set_drives().read().await[0][0][0]
.clone()
.expect("distributed drive slot");
assert!(Arc::ptr_eq(&active, &winner) && Arc::ptr_eq(&route, &winner) && Arc::ptr_eq(&topology, &winner));
assert_eq!(ctx.local_disk_id_map().read().await.get(&id), Some(&path));
assert!(!winner.local_retirement_token().expect("current local handle").is_cancelled());
}
winner.close().await.expect("stop the winning monitor");
}
#[tokio::test]
async fn reconnect_sweep_collects_all_blank_targets_and_excludes_original_disks() {
use rustfs_heal_contracts::heal_channel::{HealAdmissionResult, HealChannelCommand, init_heal_channel};
@@ -1462,7 +1553,7 @@ mod tests {
assert!(
set_disks
.attach_unformatted_replacement_disk(&replacement_endpoint)
.attach_unformatted_replacement_disk(&replacement_endpoint, None)
.await
.expect("a blank local replacement should be admitted")
);
+2 -2
View File
@@ -4688,7 +4688,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
.with_multipart_commit_owner(multipart_commit_owner)
.with_namespace_commit_guard(
(!crate::bucket::utils::is_meta_bucketname(&commit_bucket))
.then(|| commit_set.ctx.begin_namespace_commit()),
.then(|| commit_set.ctx.begin_namespace_commit_for_bucket(&commit_bucket)),
),
)
.await;
@@ -11546,7 +11546,7 @@ mod tests {
break;
}
key_marker = page.next_key_marker.clone();
upload_id_marker = page.next_upload_id_marker.clone();
upload_id_marker = page.next_upload_id_marker;
assert!(key_marker.is_some(), "a truncated page must carry a next key marker");
assert!(upload_id_marker.is_some(), "a truncated upload page must carry a next upload marker");
}
+31 -33
View File
@@ -3918,7 +3918,7 @@ impl SetDisks {
explicit_delete: true,
transaction_guard: transaction_guard.cloned(),
namespace_owner: Some(Arc::new((
self.ctx.begin_namespace_commit(),
self.ctx.begin_namespace_commit_for_bucket(bucket),
lock_guard.cloned(),
opts.object_lock_config_snapshot.clone(),
opts.scanner_publication_commit_scope.clone(),
@@ -5661,7 +5661,8 @@ impl SetDisks {
.with_publication_scope(commit_scanner_publication_scope.clone())
.with_rollback_receipt(commit_rollback_receipt.clone())
.with_namespace_commit_guard(
(!is_meta_bucketname(&commit_bucket)).then(|| commit_set.ctx.begin_namespace_commit()),
(!is_meta_bucketname(&commit_bucket))
.then(|| commit_set.ctx.begin_namespace_commit_for_bucket(&commit_bucket)),
),
)
.await;
@@ -9063,7 +9064,7 @@ impl SetDisks {
let disks = self.disk_inventory().await;
let namespace_owner = (!is_meta_bucketname(bucket)).then(|| {
Arc::new((
self.ctx.begin_namespace_commit(),
self.ctx.begin_namespace_commit_for_bucket(bucket),
commit_context.guard.clone(),
commit_context.transaction_guard.clone(),
commit_context
@@ -10469,7 +10470,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
let rollback_dir = Uuid::new_v4();
let namespace_owner = (!is_meta_bucketname(bucket)).then(|| {
Arc::new((
self.ctx.begin_namespace_commit(),
self.ctx.begin_namespace_commit_for_bucket(bucket),
Arc::clone(&transaction_guards),
opts.delete_mutation_owner.clone(),
opts.object_lock_config_snapshot.clone(),
@@ -10973,7 +10974,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
work_budget: opts.delete_recovery_work_budget.clone(),
explicit_delete,
namespace_owner: Some(Arc::new((
self.ctx.begin_namespace_commit(),
self.ctx.begin_namespace_commit_for_bucket(bucket),
opts.delete_mutation_owner.clone(),
opts.object_lock_config_snapshot.clone(),
))),
@@ -11329,7 +11330,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
work_budget: opts.delete_recovery_work_budget.clone(),
transaction_guard: transaction_guard.clone(),
namespace_owner: Some(Arc::new((
self.ctx.begin_namespace_commit(),
self.ctx.begin_namespace_commit_for_bucket(bucket),
_lock_guard.clone(),
opts.object_lock_config_snapshot.clone(),
opts.scanner_publication_commit_scope.clone(),
@@ -28723,13 +28724,15 @@ mod delete_recovery_inventory_tests {
.await
.expect("seed actual version");
let endpoint = set.set_endpoints[0].clone();
let ctx = crate::runtime::global::current_ctx();
let was_dist = ctx.is_dist_erasure().await;
if was_dist {
ctx.update_erasure_type(crate::layout::endpoints::SetupType::Erasure).await;
}
let map = ctx.local_disk_map();
let previous = map.read().await.get(&endpoint.to_string()).cloned();
let map = set.ctx.local_disk_map();
let original = map
.read()
.await
.get(&endpoint.to_string())
.cloned()
.flatten()
.expect("original RPC route");
assert!(Arc::ptr_eq(&original, &disks[0]));
let barrier = DeleteCleanupBarrier::install(bucket);
let delete_store = store.clone();
@@ -28766,14 +28769,14 @@ mod delete_recovery_inventory_tests {
)
.await
.expect("queue real renewal writer");
let connected = map
.read()
.await
let routed = map
.try_read()
.expect("waiting for one set must leave RPC lookups available")
.get(&endpoint.to_string())
.cloned()
.flatten()
.expect("renewal registered new handle");
assert!(!Arc::ptr_eq(&connected, &disks[0]));
.expect("original route is still published");
assert!(Arc::ptr_eq(&routed, &original));
drop(extra_gate);
barrier.release();
let outcome = tokio::time::timeout(Duration::from_secs(3), async { tokio::join!(renewal.as_mut(), &mut deletion) }).await;
@@ -28784,25 +28787,20 @@ mod delete_recovery_inventory_tests {
if outcome.is_err() {
let _ = deletion.await;
}
{
let mut map = map.write().await;
match previous {
Some(old) => {
map.insert(endpoint.to_string(), old);
}
None => {
map.remove(&endpoint.to_string());
}
}
}
if was_dist {
ctx.update_erasure_type(crate::layout::endpoints::SetupType::DistErasure)
.await;
}
let (_, result) =
outcome.expect("batch DELETE and real queued disk renewal must finish without nested inventory-read deadlock");
let (_, errors) = result.expect("delete task must finish");
assert!(errors.iter().all(Option::is_none), "DELETE result must remain successful: {errors:?}");
let connected = set.get_disks_internal().await[0].clone().expect("renewed set slot");
let registered = map
.read()
.await
.get(&endpoint.to_string())
.cloned()
.flatten()
.expect("renewed RPC route");
assert!(Arc::ptr_eq(&registered, &connected), "DELETE and RPC must see the same attachment");
assert!(!Arc::ptr_eq(&registered, &original), "renewal must replace the original handle");
}
}
@@ -25,14 +25,15 @@ pub(crate) mod heal {
pub(crate) mod internode {
pub(crate) use rustfs_storage_api::{
NS_SCANNER_BODY_SHA256_QUERY, NS_SCANNER_CAPABILITY_CHALLENGE_QUERY, NS_SCANNER_CHECKPOINT_COVERAGE_QUERY,
NS_SCANNER_CYCLE_QUERY, NS_SCANNER_LEADER_EPOCH_QUERY, NS_SCANNER_PROTOCOL_VERSION, NS_SCANNER_PROTOCOL_VERSION_QUERY,
NS_SCANNER_COMPLETE_OBSERVATION_QUERY, NS_SCANNER_CYCLE_QUERY, NS_SCANNER_DURABLE_PROGRESS_QUERY,
NS_SCANNER_LEADER_EPOCH_QUERY, NS_SCANNER_PROTOCOL_VERSION, NS_SCANNER_PROTOCOL_VERSION_QUERY,
NS_SCANNER_REQUEST_ID_QUERY, NS_SCANNER_SERVER_EPOCH_QUERY, NS_SCANNER_SESSION_ID_QUERY,
NS_SCANNER_SESSION_SEQUENCE_QUERY, NS_SCANNER_TIER_REGISTRY_GENERATION_QUERY, NsScannerCapabilityResponse,
PUT_FILE_AUTH_QUERY, PUT_FILE_AUTH_TRAILER_DIGEST_LEN, PUT_FILE_AUTH_TRAILER_LEN, PUT_FILE_AUTH_TRAILER_MAC_LEN,
PUT_FILE_AUTH_TRAILER_MAGIC, PUT_FILE_AUTH_V1, PUT_FILE_CAPABILITY_CHALLENGE_QUERY, PUT_FILE_CAPABILITY_QUERY,
PUT_FILE_CAPABILITY_VERSION, PUT_FILE_NONCE_QUERY, PUT_FILE_SERVER_EPOCH_QUERY, PutFileCapabilityResponse,
SCANNER_ACTIVITY_LEGACY_PROTOCOL_VERSION, SCANNER_ACTIVITY_PREVIOUS_PROTOCOL_VERSION, SCANNER_ACTIVITY_PROTOCOL_VERSION,
SCANNER_ACTIVITY_V6_PROTOCOL_VERSION, SCANNER_DIRTY_USAGE_SNAPSHOT_MAX_ENTRIES,
NS_SCANNER_SESSION_SEQUENCE_QUERY, NS_SCANNER_TIER_REGISTRY_GENERATION_QUERY, NsScannerCapabilities,
NsScannerCapabilityResponse, PUT_FILE_AUTH_QUERY, PUT_FILE_AUTH_TRAILER_DIGEST_LEN, PUT_FILE_AUTH_TRAILER_LEN,
PUT_FILE_AUTH_TRAILER_MAC_LEN, PUT_FILE_AUTH_TRAILER_MAGIC, PUT_FILE_AUTH_V1, PUT_FILE_CAPABILITY_CHALLENGE_QUERY,
PUT_FILE_CAPABILITY_QUERY, PUT_FILE_CAPABILITY_VERSION, PUT_FILE_NONCE_QUERY, PUT_FILE_SERVER_EPOCH_QUERY,
PutFileCapabilityResponse, SCANNER_ACTIVITY_LEGACY_PROTOCOL_VERSION, SCANNER_ACTIVITY_PREVIOUS_PROTOCOL_VERSION,
SCANNER_ACTIVITY_PROTOCOL_VERSION, SCANNER_ACTIVITY_V6_PROTOCOL_VERSION, SCANNER_DIRTY_USAGE_SNAPSHOT_MAX_ENTRIES,
SCANNER_DIRTY_USAGE_SNAPSHOT_PROTOCOL_VERSION, SCANNER_DIRTY_USAGE_SNAPSHOT_RPC_MAX_MESSAGE_SIZE,
WALK_DIR_BODY_SHA256_QUERY, WALK_DIR_STREAM_COMPLETION_QUERY, WALK_DIR_STREAM_COMPLETION_V1,
};
@@ -0,0 +1,507 @@
// Copyright 2026 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Read-only paged observation of physical delete evidence. Sources can report
//! the same transaction; only an exact-key inspection can classify the set.
use super::ECStore;
use crate::bucket::utils::{check_bucket_and_object_names, check_valid_bucket_name_strict, is_meta_bucketname};
use crate::disk::error::DiskError;
use crate::disk::{DeleteRecoveryCursor, DeleteRecoveryPage, DeleteTransactionId, Disk, DiskStore, MAX_DELETE_RESIDUE_PAGE_SIZE};
use crate::error::{Result, StorageError};
use crate::set_disk::SetDisks;
use rustfs_utils::path::GLOBAL_DIR_SUFFIX;
use serde::Serialize;
use std::collections::HashMap;
use std::sync::{Arc, Mutex, Weak};
use std::time::Duration;
use tokio::sync::{OwnedSemaphorePermit, Semaphore};
use tokio::time::Instant;
use uuid::Uuid;
const MAX_SESSIONS: usize = 8;
const CURSOR_TTL: Duration = Duration::from_secs(300);
/// Coordinates of the single configured drive examined for this page.
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, Serialize)]
pub struct DeleteResidueSource {
pub pool: usize,
pub set: usize,
pub drive: usize,
}
impl DeleteResidueSource {
fn erasure_set(self, store: &ECStore) -> Result<&SetDisks> {
store
.pools
.get(self.pool)
.and_then(|pool| pool.disk_set.get(self.set))
.filter(|set| self.drive < set.set_drive_count)
.map(Arc::as_ref)
.ok_or(StorageError::ErasureReadQuorum)
}
fn next(self, store: &ECStore) -> Result<Option<Self>> {
let set = self.erasure_set(store)?;
if self.drive + 1 < set.set_drive_count {
return Ok(Some(Self {
drive: self.drive + 1,
..self
}));
}
if self.set + 1 < store.pools[self.pool].disk_set.len() {
return Ok(Some(Self {
pool: self.pool,
set: self.set + 1,
drive: 0,
}));
}
Ok((self.pool + 1 < store.pools.len()).then_some(Self {
pool: self.pool + 1,
..Self::default()
}))
}
}
#[derive(Clone, Copy, Debug, PartialEq, Eq, Serialize)]
#[serde(rename_all = "camelCase")]
pub enum DeleteResidueSourceStatus {
Scanned,
Unavailable,
Unreadable,
}
/// One physical discovery page. A candidate may appear on other drives or
/// pages. Completion describes traversal coverage, never the outcome of a delete.
#[derive(Debug, Serialize)]
#[serde(rename_all = "camelCase")]
pub struct DeleteResidueInventoryPage {
pub bucket: String,
pub incarnation: Uuid,
pub source: DeleteResidueSource,
pub source_status: DeleteResidueSourceStatus,
pub candidates: Vec<DeleteTransactionId>,
pub next_cursor: Option<Uuid>,
/// Cumulative count of sources skipped because their contents could not be read.
pub incomplete_sources: usize,
pub complete: bool,
}
struct PhysicalContinuation {
disk: Weak<Disk>,
id: Uuid,
cursor: DeleteRecoveryCursor,
}
struct InventorySession {
store: Uuid,
bucket: String,
incarnation: Uuid,
source: DeleteResidueSource,
physical: Option<PhysicalContinuation>,
incomplete_sources: usize,
expires: Instant,
_permit: OwnedSemaphorePermit,
}
/// Tokens are single-use, node-local capabilities. The permit follows the
/// session during I/O so concurrent first-page requests also share the bound.
pub(crate) struct DeleteResidueCursors {
sessions: Mutex<HashMap<Uuid, InventorySession>>,
capacity: Arc<Semaphore>,
}
impl Default for DeleteResidueCursors {
fn default() -> Self {
Self {
sessions: Mutex::default(),
capacity: Arc::new(Semaphore::new(MAX_SESSIONS)),
}
}
}
impl DeleteResidueCursors {
fn take(&self, token: Option<Uuid>, store: Uuid, bucket: &str) -> Result<Option<InventorySession>> {
let now = Instant::now();
let mut sessions = self.sessions.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
sessions.retain(|_, session| session.expires > now);
let Some(token) = token else { return Ok(None) };
if !sessions.get(&token).is_some_and(|s| s.store == store && s.bucket == bucket) {
return Err(StorageError::PreconditionFailed);
}
Ok(sessions.remove(&token))
}
fn save(&self, mut session: InventorySession) -> Uuid {
let token = Uuid::new_v4();
session.expires = Instant::now() + CURSOR_TTL;
self.sessions
.lock()
.unwrap_or_else(std::sync::PoisonError::into_inner)
.insert(token, session);
token
}
}
async fn source_disk(set: &SetDisks, drive: usize) -> Result<Option<DiskStore>> {
set.disks
.read()
.await
.get(drive)
.cloned()
.ok_or(StorageError::ErasureReadQuorum)
}
async fn observe_source(
set: &SetDisks,
session: &mut InventorySession,
expected_id: Option<Uuid>,
limit: u32,
) -> Result<(DeleteResidueSourceStatus, Vec<DeleteTransactionId>)> {
let continuation = session.physical.take();
let Some(disk) = source_disk(set, session.source.drive).await? else {
if continuation.is_some() {
return Err(StorageError::PreconditionFailed);
}
return Ok((DeleteResidueSourceStatus::Unavailable, Vec::new()));
};
let id = expected_id.filter(|id| !id.is_nil());
if let Some(prior) = &continuation
&& (Some(prior.id) != id || !prior.disk.ptr_eq(&Arc::downgrade(&disk)))
{
return Err(StorageError::PreconditionFailed);
}
let Some(id) = id else {
return Ok((DeleteResidueSourceStatus::Unreadable, Vec::new()));
};
let resuming = continuation.is_some();
let page = disk
.list_delete_residue_on_disk(&session.bucket, id, continuation.map(|state| state.cursor), limit)
.await;
let DeleteRecoveryPage { mut transactions, next } = match page {
Ok(page) => page,
Err(DiskError::FileNotFound | DiskError::InvalidPath | DiskError::OutdatedXLMeta | DiskError::InconsistentDisk)
if resuming =>
{
return Err(StorageError::PreconditionFailed);
}
Err(_) => return Ok((DeleteResidueSourceStatus::Unreadable, Vec::new())),
};
if transactions.len() > usize::try_from(limit).map_err(|_| StorageError::FileCorrupt)?
|| next.as_ref().is_some_and(|cursor| cursor.token.is_nil())
{
return Err(StorageError::FileCorrupt);
}
for candidate in &mut transactions {
// A literal suffix can precede the final on-disk directory marker.
// Remove exactly one so re-encoding the listed key selects the same physical path.
if let Some(prefix) = candidate.object.strip_suffix(GLOBAL_DIR_SUFFIX) {
candidate.object = format!("{prefix}/");
}
if candidate.transaction.is_nil()
|| candidate.object.len() > 1024
|| check_bucket_and_object_names(&session.bucket, &candidate.object).is_err()
{
return Err(StorageError::FileCorrupt);
}
}
if !source_disk(set, session.source.drive)
.await?
.is_some_and(|current| Arc::ptr_eq(&current, &disk))
{
return Err(StorageError::PreconditionFailed);
}
session.physical = next.map(|cursor| PhysicalContinuation {
disk: Arc::downgrade(&disk),
id,
cursor,
});
Ok((DeleteResidueSourceStatus::Scanned, transactions))
}
impl ECStore {
/// Lists candidates from one bounded raw disk page of a user bucket. Pass
/// the single-use continuation to the same node until it returns no token.
/// Later disk mutations can change the observations between pages.
///
/// # Errors
/// Returns a precondition error for a stale continuation or changed source,
/// a throttling error when the cursor limit is reached, or a storage error
/// when the bucket identity or peer response cannot be validated. A skipped
/// disk is reported explicitly and prevents a complete terminal result.
pub async fn list_delete_residue_candidates(
self: &Arc<Self>,
bucket: &str,
cursor: Option<Uuid>,
limit: u32,
) -> Result<DeleteResidueInventoryPage> {
if is_meta_bucketname(bucket) || check_valid_bucket_name_strict(bucket).is_err() {
return Err(StorageError::BucketNameInvalid(bucket.to_owned()));
}
if !(1..=MAX_DELETE_RESIDUE_PAGE_SIZE).contains(&limit) {
return Err(StorageError::InvalidArgument(
bucket.to_owned(),
String::new(),
"limit must be 1..=64".to_owned(),
));
}
let cursors = &self.ctx.delete_residue_cursors;
let mut session = match cursors.take(cursor, self.id, bucket)? {
Some(session) => session,
None => {
let permit = Arc::clone(&cursors.capacity)
.try_acquire_owned()
.map_err(|_| StorageError::SlowDown)?;
InventorySession {
store: self.id,
bucket: bucket.to_owned(),
incarnation: self.bucket_incarnation_id_from_disk(bucket).await?,
source: DeleteResidueSource::default(),
physical: None,
incomplete_sources: 0,
expires: Instant::now(),
_permit: permit,
}
}
};
let fence = self.acquire_bucket_incarnation_fence(bucket, session.incarnation).await?;
let incarnation = session.incarnation;
let source = session.source;
let set = source.erasure_set(self)?;
let expected_id = self.pools[source.pool]
.format
.erasure
.sets
.get(source.set)
.and_then(|ids| ids.get(source.drive))
.copied();
let (source_status, candidates) = observe_source(set, &mut session, expected_id, limit).await?;
if fence.is_lock_lost() {
return Err(StorageError::PreconditionFailed);
}
if source_status != DeleteResidueSourceStatus::Scanned {
session.incomplete_sources += 1;
}
let incomplete_sources = session.incomplete_sources;
let next_source = if session.physical.is_some() {
Some(source)
} else {
source.next(self)?
};
let next_cursor = next_source.map(|next| {
session.source = next;
cursors.save(session)
});
Ok(DeleteResidueInventoryPage {
bucket: bucket.to_owned(),
incarnation,
source,
source_status,
candidates,
next_cursor,
incomplete_sources,
complete: next_cursor.is_none() && incomplete_sources == 0,
})
}
}
#[cfg(test)]
mod tests {
use super::*;
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::disk::DiskAPI;
use crate::storage_api_contracts::bucket::{BucketOperations, MakeBucketOptions};
async fn fixture(bucket: &str) -> (Vec<tempfile::TempDir>, Arc<ECStore>) {
let (dirs, store) = isolated_store_over_temp_disks().await;
store.background_cancel_token().expect("fixture background token").cancel();
tokio::task::yield_now().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create discovery bucket");
(dirs, store)
}
fn seed_residue(disk: &DiskStore, bucket: &str, key: &str, txn: Uuid) -> std::path::PathBuf {
let data = disk.path().join(bucket).join(key).join(Uuid::new_v4().to_string());
std::fs::create_dir_all(&data).expect("create historical data directory");
let marker = data.join(format!("reserve-delete-data.{txn}"));
std::fs::write(&marker, []).expect("seed undiscoverable-by-S3 reservation");
marker
}
#[tokio::test]
async fn delete_residue_inventory_pages_cover_configured_sources_and_preserve_hidden_evidence() {
let bucket = "residue-inventory";
let (_dirs, store) = fixture(bucket).await;
let disks = store.pools[0].disk_set[0].disk_inventory().await;
let transaction = Uuid::new_v4();
let mut markers = Vec::new();
for disk in disks.iter().flatten() {
markers.push(seed_residue(disk, bucket, "hidden", transaction));
}
let mut cursor = None;
let mut observed = std::collections::HashSet::new();
for _ in 0..(disks.len() * 4) {
let page = store
.list_delete_residue_candidates(bucket, cursor, 1)
.await
.expect("read disk discovery page");
assert!(page.candidates.len() <= 1);
assert_eq!(page.source_status, DeleteResidueSourceStatus::Scanned);
for item in page.candidates {
assert_eq!((item.object.as_str(), item.transaction), ("hidden", transaction));
observed.insert(page.source.drive);
}
cursor = page.next_cursor;
if cursor.is_none() {
assert!(page.complete);
break;
}
assert!(!page.complete);
}
assert!(cursor.is_none(), "traversal must reach a terminal page");
assert_eq!(observed.len(), disks.len(), "scan every participant independently");
for marker in markers {
assert!(std::fs::read(marker).expect("retain legacy marker bytes").is_empty());
}
}
#[tokio::test]
async fn delete_residue_inventory_wrong_disk_never_certifies_complete() {
let bucket = "residue-wrong-disk";
let (_dirs, store) = fixture(bucket).await;
let set = &store.pools[0].disk_set[0];
let disks = set.disk_inventory().await;
let other = disks[1].as_ref().expect("second fixture drive");
seed_residue(other, bucket, "only-on-second-drive", Uuid::new_v4());
set.disks.write().await[0] = Some(other.clone());
let page = store
.list_delete_residue_candidates(bucket, None, 4)
.await
.expect("report unverified slot");
assert_eq!(page.source.drive, 0);
assert_eq!(page.source_status, DeleteResidueSourceStatus::Unreadable);
assert_eq!(page.incomplete_sources, 1);
assert!(!page.complete);
assert!(page.candidates.is_empty(), "never attribute the other slot's evidence to this one");
}
#[tokio::test]
async fn delete_residue_inventory_preserves_literal_suffix_in_directory_key() {
use rustfs_utils::path::encode_dir_object;
let bucket = "residue-dir-suffix";
let (_dirs, store) = fixture(bucket).await;
let disk = store.pools[0].disk_set[0].disk_inventory().await[0]
.clone()
.expect("first drive");
let key = format!("object{GLOBAL_DIR_SUFFIX}/");
let physical = encode_dir_object(&key);
seed_residue(&disk, bucket, &physical, Uuid::new_v4());
let page = store
.list_delete_residue_candidates(bucket, None, 4)
.await
.expect("list directory key");
assert_eq!(page.candidates.len(), 1);
assert_eq!(page.candidates[0].object, key);
assert_eq!(encode_dir_object(&page.candidates[0].object), physical);
}
#[tokio::test]
async fn delete_residue_inventory_missing_source_never_certifies_empty() {
let bucket = "residue-offline";
let (_dirs, store) = fixture(bucket).await;
let set = &store.pools[0].disk_set[0];
let failed = set.set_drive_count - 1;
set.disks.write().await[failed] = None;
let mut cursor = None;
for index in 0..set.set_drive_count {
let page = store
.list_delete_residue_candidates(bucket, cursor, 1)
.await
.expect("page with an offline source");
assert_eq!(page.source.drive, index);
assert!(page.candidates.is_empty());
assert!(!page.complete);
if index == failed {
assert_eq!(page.source_status, DeleteResidueSourceStatus::Unavailable);
assert_eq!(page.incomplete_sources, 1);
}
cursor = page.next_cursor;
}
assert!(cursor.is_none(), "failed source is explicitly accounted for at end of walk");
}
#[tokio::test]
async fn delete_residue_inventory_rejects_replaced_source_during_page() {
let bucket = "residue-source-change";
let (_dirs, store) = fixture(bucket).await;
let set = &store.pools[0].disk_set[0];
let disks = set.disk_inventory().await;
let first = disks[0].as_ref().expect("first fixture participant");
seed_residue(first, bucket, "first", Uuid::new_v4());
seed_residue(first, bucket, "second", Uuid::new_v4());
let page = store
.list_delete_residue_candidates(bucket, None, 1)
.await
.expect("open cursor on first drive");
assert_eq!(page.candidates.len(), 1);
let cursor = page.next_cursor.expect("additional physical pages");
assert!(matches!(
store
.list_delete_residue_candidates("different-bucket", Some(cursor), 1)
.await,
Err(StorageError::PreconditionFailed)
));
set.disks.write().await[0] = None;
assert!(matches!(
store.list_delete_residue_candidates(bucket, Some(cursor), 1).await,
Err(StorageError::PreconditionFailed)
));
assert!(matches!(
store.list_delete_residue_candidates(bucket, Some(cursor), 1).await,
Err(StorageError::PreconditionFailed)
));
}
#[tokio::test(start_paused = true)]
async fn delete_residue_inventory_expired_cursor_releases_capacity() {
let cursors = DeleteResidueCursors::default();
let store = Uuid::new_v4();
let mut token = Uuid::nil();
for _ in 0..MAX_SESSIONS {
token = cursors.save(InventorySession {
store,
bucket: "scope".to_owned(),
incarnation: Uuid::new_v4(),
source: DeleteResidueSource::default(),
physical: None,
incomplete_sources: 0,
expires: Instant::now(),
_permit: cursors.capacity.clone().try_acquire_owned().expect("bounded capacity"),
});
}
assert!(cursors.capacity.clone().try_acquire_owned().is_err());
assert!(matches!(
cursors.take(Some(token), Uuid::new_v4(), "scope"),
Err(StorageError::PreconditionFailed)
));
tokio::time::advance(CURSOR_TTL).await;
assert!(matches!(cursors.take(Some(token), store, "scope"), Err(StorageError::PreconditionFailed)));
assert_eq!(cursors.capacity.available_permits(), MAX_SESSIONS);
}
}
@@ -19,6 +19,7 @@
use super::ECStore;
use crate::bucket::utils::is_meta_bucketname;
use crate::disk::{DeleteRecoveryCursor, DiskAPI, DiskStore};
use crate::error::StorageError;
use futures::{StreamExt, future::BoxFuture, stream::FuturesUnordered};
use rustfs_heal_contracts::heal_channel::HealOpts;
use std::collections::{HashMap, HashSet, VecDeque};
@@ -41,6 +42,8 @@ const EVENT_DELETE_RECOVERY_STATUS: &str = "delete_recovery_status";
const KEYS_PER_PASS: usize = 16;
const RECOVERY_CONCURRENCY: usize = 4;
const INTERVAL: Duration = Duration::from_secs(1);
const INITIAL_RETRY_DELAY: Duration = Duration::from_secs(5);
const MAX_RETRY_DELAY: Duration = Duration::from_secs(5 * 60);
const INVENTORY_INTERVAL: Duration = Duration::from_secs(30);
const DISCOVERY_BUDGET: Duration = Duration::from_secs(5);
const ANTI_ENTROPY_INTERVAL: Duration = Duration::from_secs(60 * 60);
@@ -60,6 +63,7 @@ pub(crate) struct DeleteRecoveryTicket {
key: Arc<DeleteRecoveryKey>,
generation: Uuid,
admission: Option<Arc<AdmissionGuard>>,
retry: Option<RetrySchedule>,
}
#[derive(Debug)]
@@ -82,10 +86,40 @@ impl Drop for AdmissionGuard {
}
}
#[derive(Clone, Copy, Debug)]
struct RetrySchedule {
ready_at: Instant,
delay: Duration,
}
impl RetrySchedule {
fn after_failure(previous: Option<Self>, now: Instant) -> Self {
let delay = previous.map_or(INITIAL_RETRY_DELAY, |retry| (retry.delay * 2).clamp(INITIAL_RETRY_DELAY, MAX_RETRY_DELAY));
Self {
ready_at: now + delay,
delay,
}
}
}
#[derive(Debug)]
struct PendingHint {
generation: Uuid,
queue_generation: Uuid,
kind: PendingHintKind,
}
#[derive(Clone, Copy, Debug)]
enum PendingHintKind {
Foreground,
Discovered,
Retry(RetrySchedule),
}
#[derive(Debug)]
struct HintState {
pending: HashMap<Arc<DeleteRecoveryKey>, (Uuid, Uuid, bool)>,
order: VecDeque<DeleteRecoveryTicket>,
pending: HashMap<Arc<DeleteRecoveryKey>, PendingHint>,
order: VecDeque<(Arc<DeleteRecoveryKey>, Uuid)>,
rescan: Option<Uuid>,
started: HashSet<Uuid>,
#[cfg(test)]
@@ -93,6 +127,18 @@ struct HintState {
}
impl HintState {
fn enqueue(&mut self, key: Arc<DeleteRecoveryKey>, generation: Uuid, kind: PendingHintKind) {
self.pending.insert(
key.clone(),
PendingHint {
generation,
queue_generation: generation,
kind,
},
);
self.order.push_back((key, generation));
}
fn compact_order(&mut self, capacity: usize) {
// Twice the pending capacity guarantees at least capacity completed
// entries between compactions, even with a full retry backlog. This
@@ -106,7 +152,7 @@ impl HintState {
self.compaction_visits += self.order.len();
}
let Self { pending, order, .. } = self;
order.retain(|ticket| pending.get(&ticket.key).is_some_and(|entry| entry.1 == ticket.generation));
order.retain(|(key, generation)| pending.get(key).is_some_and(|entry| entry.queue_generation == *generation));
}
}
@@ -142,32 +188,28 @@ impl DeleteRecoveryHints {
let key = Arc::new(key);
let mut state = self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
let generation = Uuid::new_v4();
if let Some((current, _, _)) = state.pending.get_mut(&key) {
*current = generation;
if let Some(entry) = state.pending.get_mut(&key) {
entry.generation = generation;
entry.kind = PendingHintKind::Foreground;
} else {
if state.pending.len() >= self.capacity {
// Durable evidence remains the authority. Rotate the oldest
// hint so permanently blocked keys cannot exclude later raw
// discoveries forever; the next scan revisits evicted keys.
while let Some(oldest) = state.order.pop_front() {
while let Some((oldest, generation)) = state.order.pop_front() {
if state
.pending
.get(&oldest.key)
.is_some_and(|entry| entry.1 == oldest.generation)
.get(&oldest)
.is_some_and(|entry| entry.queue_generation == generation)
{
state.pending.remove(&oldest.key);
state.pending.remove(&oldest);
break;
}
}
state.rescan = Some(Uuid::new_v4());
}
state.compact_order(self.capacity);
state.pending.insert(key.clone(), (generation, generation, false));
state.order.push_back(DeleteRecoveryTicket {
key: key.clone(),
generation,
admission: None,
});
state.enqueue(key.clone(), generation, PendingHintKind::Foreground);
}
Some(DeleteRecoveryTicket {
admission: Some(Arc::new(AdmissionGuard {
@@ -177,6 +219,7 @@ impl DeleteRecoveryHints {
})),
key,
generation,
retry: None,
})
}
@@ -188,28 +231,21 @@ impl DeleteRecoveryHints {
if state.pending.len() >= self.capacity {
// Only previously attempted work may yield its slot. Unattempted
// keys apply backpressure so a fast scan cannot overwrite them.
let retried = state.order.iter().position(|ticket| {
let retried = state.order.iter().position(|(queued, generation)| {
state
.pending
.get(&ticket.key)
.is_some_and(|entry| entry.1 == ticket.generation && entry.2)
.get(queued)
.is_some_and(|entry| entry.queue_generation == *generation && matches!(entry.kind, PendingHintKind::Retry(_)))
});
let Some(index) = retried else { return false };
if let Some(oldest) = state.order.remove(index) {
state.pending.remove(&oldest.key);
if let Some((oldest, _)) = state.order.remove(index) {
state.pending.remove(&oldest);
}
// This durable key has already been attempted. The hourly sweep
// revisits it; immediate rescans would loop on permanent failures.
}
state.compact_order(self.capacity);
let key = Arc::new(key);
let generation = Uuid::new_v4();
state.pending.insert(key.clone(), (generation, generation, false));
state.order.push_back(DeleteRecoveryTicket {
key,
generation,
admission: None,
});
state.enqueue(Arc::new(key), Uuid::new_v4(), PendingHintKind::Discovered);
true
}
@@ -221,33 +257,36 @@ impl DeleteRecoveryHints {
if state
.pending
.get(&ticket.key)
.is_some_and(|entry| entry.0 == ticket.generation)
.is_some_and(|entry| entry.generation == ticket.generation)
{
state.pending.remove(&ticket.key);
}
}
fn take_available(&self, active: &HashSet<Arc<DeleteRecoveryKey>>) -> Option<DeleteRecoveryTicket> {
fn take_available(&self, active: &HashSet<Arc<DeleteRecoveryKey>>, now: Instant) -> Option<DeleteRecoveryTicket> {
let mut state = self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
let queued = state.order.len();
for _ in 0..queued {
let ticket = state.order.pop_front()?;
if let Some(&(generation, slot, _)) = state.pending.get(&ticket.key)
&& slot == ticket.generation
{
if active.contains(&ticket.key) {
// Preserve a newer hint while this key still owns a slot.
// Other keys can proceed without waiting on its lock.
state.order.push_back(ticket);
continue;
}
state.pending.remove(&ticket.key);
return Some(DeleteRecoveryTicket {
key: ticket.key,
generation,
admission: None,
});
let (key, slot) = state.order.pop_front()?;
let Some(entry) = state.pending.get(&key).filter(|entry| entry.queue_generation == slot) else {
continue;
};
let retry = match entry.kind {
PendingHintKind::Retry(retry) => Some(retry),
PendingHintKind::Discovered | PendingHintKind::Foreground => None,
};
if retry.is_some_and(|retry| retry.ready_at > now) || active.contains(&key) {
state.order.push_back((key, slot));
continue;
}
let ticket = DeleteRecoveryTicket {
generation: entry.generation,
admission: None,
retry,
key,
};
state.pending.remove(&ticket.key);
return Some(ticket);
}
None
}
@@ -255,24 +294,38 @@ impl DeleteRecoveryHints {
#[cfg(test)]
fn take(&self, limit: usize) -> Vec<DeleteRecoveryTicket> {
let active = HashSet::new();
std::iter::from_fn(|| self.take_available(&active)).take(limit).collect()
std::iter::from_fn(|| self.take_available(&active, Instant::now()))
.take(limit)
.collect()
}
fn retry(&self, ticket: DeleteRecoveryTicket) {
fn retry(&self, ticket: DeleteRecoveryTicket, retry: RetrySchedule) {
let mut state = self.state.lock().unwrap_or_else(std::sync::PoisonError::into_inner);
if state.pending.contains_key(&ticket.key) {
if let Some(entry) = state.pending.get_mut(&ticket.key) {
// Discovery can see the same journal while replay is in flight.
// Keep its queue slot, but only new foreground work resets backoff.
if matches!(entry.kind, PendingHintKind::Discovered) {
entry.kind = PendingHintKind::Retry(retry);
}
return;
}
if state.pending.len() >= self.capacity {
// Attempted work yields to new admissions and the periodic durable sweep.
return;
}
state.compact_order(self.capacity);
if state.pending.len() >= self.capacity {
// Previously attempted work yields to admitted keys until the
// periodic sweep. Foreground admission retains its own rescan guard.
return;
}
state
.pending
.insert(ticket.key.clone(), (ticket.generation, ticket.generation, true));
state.order.push_back(ticket);
state.enqueue(ticket.key, ticket.generation, PendingHintKind::Retry(retry));
}
#[cfg(test)]
fn retry_now(&self, ticket: DeleteRecoveryTicket) {
self.retry(
ticket,
RetrySchedule {
ready_at: Instant::now(),
delay: Duration::ZERO,
},
);
}
#[cfg(test)]
@@ -642,7 +695,12 @@ impl<'a> RecoveryWork<'a> {
&& Instant::now() < self.deadline
&& !self.cancel.is_cancelled()
{
let Some(ticket) = self.store.ctx.delete_recovery_hints().take_available(&self.active) else {
let Some(ticket) = self
.store
.ctx
.delete_recovery_hints()
.take_available(&self.active, Instant::now())
else {
break;
};
self.active.insert(ticket.key.clone());
@@ -688,15 +746,32 @@ async fn recover_hint(
) => Some(result),
};
let mut failures = RecoveryFailures::default();
if !matches!(outcome, Some(Ok(_))) {
match outcome {
Some(Err(crate::error::StorageError::ErasureReadQuorum)) => failures.pending = 1,
Some(Err(crate::error::StorageError::SlowDown)) => failures.blocked = 1,
Some(Err(crate::error::StorageError::FileCorrupt)) => failures.corrupt = 1,
Some(Err(_)) => failures.failed = 1,
_ => {}
let now = Instant::now();
let retry = match outcome {
Some(Ok(_)) => None,
Some(Err(StorageError::ErasureReadQuorum)) => {
failures.pending = 1;
// Check returning participants on the normal maintenance cadence.
Some(RetrySchedule {
ready_at: now + INTERVAL,
delay: INTERVAL,
})
}
store.ctx.delete_recovery_hints().retry(ticket);
Some(Err(error)) => {
match error {
StorageError::SlowDown => failures.blocked = 1,
StorageError::FileCorrupt => failures.corrupt = 1,
_ => failures.failed = 1,
}
Some(RetrySchedule::after_failure(ticket.retry, now))
}
None => Some(ticket.retry.unwrap_or(RetrySchedule {
ready_at: now,
delay: Duration::ZERO,
})),
};
if let Some(retry) = retry {
store.ctx.delete_recovery_hints().retry(ticket, retry);
}
(key, failures)
}
@@ -826,7 +901,7 @@ mod tests {
assert_eq!(taken[0].generation, newer.generation);
let latest = hints.register(key("a")).expect("hint during recovery");
hints.ack(&taken[0]);
hints.retry(taken.into_iter().next().expect("selected ticket"));
hints.retry_now(taken.into_iter().next().expect("selected ticket"));
let pending = hints.take(2);
assert_eq!(pending[0].key.object, "b");
assert_eq!(pending[1].generation, latest.generation, "old recovery cannot replace a newer hint");
@@ -865,7 +940,7 @@ mod tests {
assert!(hints.state.lock().expect("hint state").order.len() <= 2);
let mut tickets = hints.take(2);
assert_eq!(tickets.len(), 2);
hints.retry(tickets.remove(0));
hints.retry_now(tickets.remove(0));
assert_eq!(hints.take(2).len(), 1, "failed work survives an empty queue");
}
@@ -877,18 +952,48 @@ mod tests {
let active = HashSet::from([admitted.key.clone()]);
let newer = hints.register(key("active")).expect("new foreground generation");
hints.register(key("later")).expect("independent key");
let later = hints.take_available(&active).expect("skip the active key");
let later = hints.take_available(&active, Instant::now()).expect("skip the active key");
assert_eq!(later.key.object, "later");
assert!(hints.take_available(&active).is_none(), "a newer generation cannot occupy a second slot");
assert!(
hints.take_available(&active, Instant::now()).is_none(),
"a newer generation cannot occupy a second slot"
);
hints.ack(&original);
hints.retry(admitted);
hints.retry_now(admitted);
assert!(hints.is_pending(&key("active")), "old success or cancellation cannot clear newer work");
let next = hints
.take_available(&HashSet::new())
.take_available(&HashSet::new(), Instant::now())
.expect("active key becomes eligible after completion");
assert_eq!(next.generation, newer.generation, "retry cannot overwrite the foreground generation");
}
#[test]
fn delete_recovery_backoff_survives_rediscovery_and_resets_for_foreground_work() {
let hints = DeleteRecoveryHints::with_capacity(2);
let active = HashSet::new();
let key = key("blocked");
let mut now = Instant::now();
assert!(hints.register_discovered(key.clone()));
let mut ticket = hints.take_available(&active, now).expect("discovery is immediately eligible");
for seconds in [5, 10, 20, 40, 80, 160, 300, 300] {
assert!(hints.register_discovered(key.clone()), "discovery arrives during replay");
let retry = RetrySchedule::after_failure(ticket.retry, now);
hints.retry(ticket, retry);
assert!(hints.register_discovered(key.clone()), "discovery also sees the queued retry");
now += Duration::from_secs(seconds);
assert!(hints.take_available(&active, now - Duration::from_nanos(1)).is_none());
ticket = hints.take_available(&active, now).expect("admit at the retry deadline");
}
let retry = RetrySchedule::after_failure(ticket.retry, now);
hints.retry(ticket, retry);
let foreground = hints.register(key).expect("foreground request during backoff");
let ticket = hints
.take_available(&active, now)
.expect("admit new foreground work promptly");
assert_eq!(ticket.generation, foreground.generation);
assert!(ticket.retry.is_none());
}
#[test]
fn delete_recovery_hint_compaction_stays_amortized_near_capacity() {
let capacity = HINT_CAPACITY;
@@ -925,7 +1030,7 @@ mod tests {
hints.register(key("blocked-a")).expect("first blocked key");
hints.register(key("blocked-b")).expect("second blocked key");
for ticket in hints.take(2) {
hints.retry(ticket);
hints.retry_now(ticket);
}
hints
.register(key("healthy-later"))
@@ -1118,7 +1223,8 @@ mod tests {
publish_one(&disks[0], bucket, transaction).await;
let set = &store.pools[0].disk_set[0];
let offline = set.disks.write().await[3].take();
pass.run(&store, &CancellationToken::new()).await;
let cancel = CancellationToken::new();
assert_eq!(recover_hints(&store, &cancel).await.pending, 1, "the participant is required");
assert_eq!(
store
.ctx
@@ -1131,7 +1237,10 @@ mod tests {
1
);
set.disks.write().await[3] = offline;
pass.run(&store, &CancellationToken::new()).await;
tokio::time::pause();
tokio::time::advance(INTERVAL).await;
tokio::time::resume();
pass.run(&store, &cancel).await;
assert!(
store.ctx.delete_recovery_hints().take(1).is_empty(),
"returning participant releases retained work"
@@ -1184,7 +1293,7 @@ mod tests {
assert!(hints.register_discovered(key(&format!("blocked-{index}"))));
}
for ticket in hints.take(KEYS_PER_PASS) {
hints.retry(ticket);
hints.retry_now(ticket);
}
for index in 0..KEYS_PER_PASS {
assert!(
@@ -1194,7 +1303,7 @@ mod tests {
}
let in_flight = hints.take(1).pop().expect("one admitted recovery attempt");
assert!(hints.register_discovered(key("arrived-during-recovery")));
hints.retry(in_flight);
hints.retry_now(in_flight);
let mut pass = RecoveryPass {
next_inventory: Some(Instant::now() + ANTI_ENTROPY_INTERVAL),
scan_generation: Some(scanning),
@@ -1221,6 +1330,65 @@ mod tests {
assert_eq!(hints.rescan(), Some(foreground), "an older scan cannot clear a foreground obligation");
}
#[tokio::test]
async fn delete_recovery_blocked_legacy_key_backs_off_without_delaying_other_keys() {
let bucket = "runtime-legacy-backoff";
let (dirs, store, disks) = fixture(bucket).await;
let transaction = Uuid::new_v4();
for disk in &disks {
disk.delete_version(
bucket,
"object",
FileInfo {
name: "object".to_owned(),
..Default::default()
},
false,
DeleteOptions {
old_data_dir: Some(transaction),
..Default::default()
},
)
.await
.expect("leave an undecided historical delete");
}
let backups: Vec<_> = dirs
.iter()
.map(|dir| {
dir.path()
.join(bucket)
.join("object")
.join(transaction.to_string())
.join("xl.meta.bkp")
})
.collect();
let before: Vec<_> = backups
.iter()
.map(|path| std::fs::read(path).expect("read original evidence"))
.collect();
register(&store, bucket).await;
let cancel = CancellationToken::new();
assert_eq!(recover_hints(&store, &cancel).await.blocked, 1);
let hints = store.ctx.delete_recovery_hints();
let healthy = DeleteRecoveryKey {
object: "absent-key".to_owned(),
bucket: bucket.to_owned(),
incarnation: Some(store.bucket_incarnation_id_from_disk(bucket).await.expect("bucket identity")),
..key("absent-key")
};
assert!(hints.register_discovered(healthy.clone()));
assert_eq!(
recover_hints(&store, &cancel).await.blocked,
0,
"do not re-inspect a blocked key on the next pass"
);
assert!(!hints.is_pending(&healthy), "the independent key still settles promptly");
assert_eq!(hints.pending_len(), 1, "the blocked key remains eligible for a later retry");
for (path, bytes) in backups.iter().zip(before) {
assert_eq!(std::fs::read(path).expect("retained historical evidence"), bytes);
}
}
#[tokio::test]
async fn delete_recovery_admission_stays_within_each_tick_budget() {
let bucket = "runtime-admission-budget";
@@ -1413,7 +1581,10 @@ mod tests {
.find(|(key, _)| key.object == format!("waiting-{index}"))
.expect("unattempted key remains queued")
.1;
assert!(!queued.2, "budget exhaustion must not grant discovery permission to evict untouched work");
assert!(
matches!(queued.kind, PendingHintKind::Foreground),
"budget exhaustion must not grant discovery permission to evict untouched work"
);
}
}
@@ -1434,7 +1605,7 @@ mod tests {
for ticket in hints.take(KEYS_PER_PASS) {
let index: usize = ticket.key.object.parse().expect("numeric test key");
if index % 1_024 < KEYS_PER_PASS {
hints.retry(ticket);
hints.retry_now(ticket);
} else {
recovered.insert(index);
}
+186 -1
View File
@@ -29,6 +29,7 @@ const LOG_SUBSYSTEM_HEAL: &str = "heal";
const EVENT_HEAL_ABANDONED_PARTS: &str = "heal_abandoned_parts";
const EVENT_HEAL_FORMAT_COMPLETED: &str = "heal_format_completed";
const EVENT_HEAL_OBJECT_STARTED: &str = "heal_object_started";
const EVENT_HEAL_DELETE_RECOVERY_BLOCKED: &str = "heal_delete_recovery_blocked";
/// An explicit bucket-heal admission owns one lifecycle generation through its write tail.
pub(crate) struct BucketHealScope {
@@ -314,6 +315,104 @@ impl ECStore {
.map_err(|error| Error::other_with_context("bucket heal owner task failed", error))?
}
/// Inspect every participant and sibling transaction for one user key in its selected set.
/// A partial inspection never produces a fingerprint authorizing a subsequent discard.
pub async fn inspect_delete_residue(
self: &Arc<Self>,
bucket: &str,
object: &str,
pool: usize,
set: usize,
) -> Result<rustfs_madmin::heal_commands::DeleteResidueInspection> {
self.admin_delete_residue(bucket, object, pool, set, None).await
}
/// Discard only eligible historical evidence under a fresh exact-key inspection.
/// A stale or ineligible preview returns PreconditionFailed. If a drive fails after
/// others have discarded their copies, inspect again before retrying.
pub async fn discard_delete_residue(
self: &Arc<Self>,
bucket: &str,
object: &str,
pool: usize,
set: usize,
expected_fingerprint: &str,
) -> Result<rustfs_madmin::heal_commands::DeleteResidueInspection> {
if expected_fingerprint.len() != 64
|| !expected_fingerprint
.bytes()
.all(|b| b.is_ascii_digit() || (b'a'..=b'f').contains(&b))
{
return Err(StorageError::PreconditionFailed);
}
self.admin_delete_residue(bucket, object, pool, set, Some(expected_fingerprint.to_owned()))
.await
}
async fn admin_delete_residue(
self: &Arc<Self>,
bucket: &str,
object: &str,
pool_index: usize,
set_index: usize,
expected: Option<String>,
) -> Result<rustfs_madmin::heal_commands::DeleteResidueInspection> {
use rustfs_madmin::heal_commands::DeleteResidueInspection;
check_valid_bucket_name_strict(bucket)?;
check_object_args(bucket, object)?;
let incarnation = self.bucket_incarnation_id_from_disk(bucket).await?;
let object = object.to_owned();
self.run_bucket_heal_at_incarnation(bucket, incarnation, &HealOpts::default(), move |store, bucket, _| async move {
let scope = bucket_heal_scope(&bucket).ok_or(StorageError::PreconditionFailed)?;
let gate = store.ctx.data_movement_operation_gate();
let _movement_guard = gate.read().await;
let pool = store
.pools
.get(pool_index)
.ok_or_else(|| invalid_heal_pool_index(pool_index, store.pools.len()))?;
let physical_key = encode_dir_object(&object);
let set = pool
.disk_set
.get(set_index)
.filter(|set| Arc::ptr_eq(set, &pool.get_disks_by_key(&physical_key)))
.ok_or_else(|| {
StorageError::InvalidArgument(
"delete residue".to_owned(),
"set".to_owned(),
"selected set does not own the key".to_owned(),
)
})?;
if store.is_suspended(pool_index).await || store.is_pool_rebalancing(pool_index).await {
return Err(StorageError::SlowDown);
}
let (config_guard, allows_discard) =
metadata_sys::legacy_discard_config_guard_in(&store.ctx, &bucket, incarnation).await?;
let report = DeleteResidueInspection {
bucket,
object,
pool: pool_index,
set: set_index,
incarnation: incarnation.to_string(),
complete: false,
reason: String::new(),
participants: Vec::new(),
transactions: Vec::new(),
fingerprint: None,
reclaimed: 0,
};
set.operator_delete_residue(
&physical_key,
report,
expected.as_deref(),
allows_discard,
pool.format.erasure.sets.get(set_index).map(Vec::as_slice).unwrap_or_default(),
&|| scope.check().is_ok() && !config_guard.is_lock_lost(),
)
.await
})
.await
}
/// Recover hidden delete journals through the lifecycle-owned heal boundary.
pub async fn recover_delete_transactions(self: &Arc<Self>, bucket: &str, opts: &HealOpts) -> Result<HealResultItem> {
Ok(self
@@ -385,7 +484,7 @@ impl ECStore {
dry_run: opts.dry_run,
explicit_delete,
namespace_owner: (!opts.dry_run)
.then(|| Arc::new((scope.clone(), store.ctx.begin_namespace_commit())) as Arc<dyn Send + Sync>),
.then(|| Arc::new((scope.clone(), store.ctx.begin_namespace_commit_for_bucket(&bucket))) as Arc<dyn Send + Sync>),
..crate::set_disk::DeleteRecoveryContext::new(Some(scope.incarnation), &can_commit)
};
let mut summary = crate::set_disk::DeleteRecoverySummary::default();
@@ -409,6 +508,24 @@ impl ECStore {
}
}
}
if let Some(object) = &object
&& !opts.dry_run
&& summary.blocked > 0
{
tracing::debug!(
event = EVENT_HEAL_DELETE_RECOVERY_BLOCKED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_HEAL,
state = "insufficient_evidence",
bucket = ?bucket,
object = ?decode_dir_object(object),
pool = opts.pool,
set = opts.set,
legacy_ambiguous = summary.legacy_ambiguous,
other_blocked = summary.blocked - summary.legacy_ambiguous,
"Exact-key delete recovery is blocked"
);
}
if object.is_some() && summary.pending > 0 {
return Err(StorageError::ErasureReadQuorum);
}
@@ -1702,6 +1819,74 @@ mod tests {
.expect("minimal pool should build")
}
#[tokio::test]
#[serial_test::serial]
async fn delete_recovery_operator_discard_is_scoped_to_the_selected_pool() {
let (root, store, shutdown) = multi_pool_heal_store().await;
let bucket = "operator-cross-pool";
let key = "object";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
let options = ObjectOptions {
write_completion: crate::object_api::WriteCompletion::TailDrained,
..Default::default()
};
for (pool, contents) in store.pools.iter().zip([b"old bytes", b"new bytes"]) {
pool.put_object(bucket, key, &mut PutObjReader::from_vec(contents.to_vec()), &options)
.await
.expect("put distinct pool copy");
}
let transaction = Uuid::new_v4();
for disk in store.pools[0].disk_set[0].disk_inventory().await.into_iter().flatten() {
disk.delete_version(
bucket,
key,
rustfs_filemeta::FileInfo {
name: key.to_owned(),
..Default::default()
},
false,
DeleteOptions {
old_data_dir: Some(transaction),
..Default::default()
},
)
.await
.expect("create legacy source pool residue");
}
let other_pool_metadata = (0..4)
.map(|slot| root.path().join(format!("pool1-disk{slot}/{bucket}/{key}/xl.meta")))
.map(|path| {
let bytes = std::fs::read(&path).expect("other pool's live copy");
(path, bytes)
})
.collect::<Vec<_>>();
let proof = store
.inspect_delete_residue(bucket, key, 0, 0)
.await
.expect("inspect source pool")
.fingerprint
.expect("source residue");
assert!(matches!(
store.discard_delete_residue(bucket, key, 1, 0, &proof).await,
Err(StorageError::PreconditionFailed)
));
assert_eq!(
store
.discard_delete_residue(bucket, key, 0, 0, &proof)
.await
.expect("scoped cleanup")
.reclaimed,
1
);
for (path, contents) in other_pool_metadata {
assert_eq!(std::fs::read(path).expect("other pool remains live"), contents);
}
shutdown.cancel();
}
#[tokio::test]
#[serial_test::serial]
async fn admin_bucket_recovery_recreates_missing_directories_at_current_incarnation() {
+34 -35
View File
@@ -17932,10 +17932,10 @@ mod tests {
.expect("shared-tuple recursive prefix delete should commit");
assert_eq!(tier_delete_journal_count(store.clone()).await, 1);
for pool_idx in 0..2 {
assert!(store.pools.len() >= 2);
for (pool_idx, pool) in store.pools.iter().enumerate().take(2) {
assert!(
store.pools[pool_idx]
.get_disks_by_key(object)
pool.get_disks_by_key(object)
.load_file_info_versions_exact(bucket, object)
.await
.expect("shared-tuple post-delete metadata should decode")
@@ -19768,8 +19768,9 @@ mod tests {
}
wait_for_expiry_workers_idle(&store).await;
let mut free_version_ids = Vec::new();
for pool_idx in 0..2 {
let retained = store.pools[pool_idx]
assert!(store.pools.len() >= 2);
for (pool_idx, pool) in store.pools.iter().enumerate().take(2) {
let retained = pool
.get_disks_by_key(object)
.load_file_info_versions_exact(&bucket, object)
.await
@@ -19805,10 +19806,10 @@ mod tests {
);
wait_for_expiry_workers_idle(&store).await;
assert_eq!(backend.remove_count().await, 1, "shared remote tuple should be deleted exactly once");
for pool_idx in 0..2 {
assert!(store.pools.len() >= 2);
for (pool_idx, pool) in store.pools.iter().enumerate().take(2) {
assert!(
store.pools[pool_idx]
.get_disks_by_key(object)
pool.get_disks_by_key(object)
.load_file_info_versions_exact(&bucket, object)
.await
.expect("post-delete metadata should decode")
@@ -19858,9 +19859,10 @@ mod tests {
.expect("multi-pool source bucket should be created");
let mut remote_tuples = Vec::new();
for pool_idx in 0..2 {
assert!(store.pools.len() >= 2);
for (pool_idx, pool) in store.pools.iter().enumerate().take(2) {
let mut reader = PutObjReader::from_vec(vec![b'a' + pool_idx as u8; 1024 * 1024]);
let source = store.pools[pool_idx]
let source = pool
.put_object(
bucket,
object,
@@ -19872,31 +19874,27 @@ mod tests {
)
.await
.expect("pool-local source should be written");
store.pools[pool_idx]
.transition_object(
bucket,
object,
&ObjectOptions {
transition: TransitionOptions {
status: TRANSITION_PENDING.to_string(),
tier: tier_name.to_string(),
etag: source.etag.clone().expect("pool-local source should have an etag"),
..Default::default()
},
mod_time: source.mod_time,
pool.transition_object(
bucket,
object,
&ObjectOptions {
transition: TransitionOptions {
status: TRANSITION_PENDING.to_string(),
tier: tier_name.to_string(),
etag: source.etag.clone().expect("pool-local source should have an etag"),
..Default::default()
},
)
.await
.expect("pool-local source should transition");
let transitioned = store.pools[pool_idx]
mod_time: source.mod_time,
..Default::default()
},
)
.await
.expect("pool-local source should transition");
let transitioned = pool
.get_object_info(bucket, object, &ObjectOptions::default())
.await
.expect("pool-local transitioned source should be readable");
remote_tuples.push((
transitioned.transitioned_object.name.clone(),
transitioned.transitioned_object.version_id.clone(),
));
remote_tuples.push((transitioned.transitioned_object.name.clone(), transitioned.transitioned_object.version_id));
}
assert_ne!(remote_tuples[0], remote_tuples[1], "fixture must use distinct remote cleanup tuples");
assert_eq!(backend.object_count().await, 2);
@@ -19906,8 +19904,9 @@ mod tests {
.delete_object_with_tier_delete_journal(bucket, object, ObjectOptions::default())
.await
.expect("store-level delete should remove both pool copies");
for pool_idx in 0..2 {
let retained = store.pools[pool_idx]
assert!(store.pools.len() >= 2);
for (pool_idx, pool) in store.pools.iter().enumerate().take(2) {
let retained = pool
.get_disks_by_key(object)
.load_file_info_versions_exact(bucket, object)
.await
@@ -19946,10 +19945,10 @@ mod tests {
removed.sort();
remote_tuples.sort();
assert_eq!(removed, remote_tuples, "recovery must use each pool's exact remote tuple");
for pool_idx in 0..2 {
assert!(store.pools.len() >= 2);
for (pool_idx, pool) in store.pools.iter().enumerate().take(2) {
assert!(
store.pools[pool_idx]
.get_disks_by_key(object)
pool.get_disks_by_key(object)
.load_file_info_versions_exact(bucket, object)
.await
.expect("post-recovery pool metadata should decode")
+10 -10
View File
@@ -8504,7 +8504,7 @@ mod test {
let mut results = Vec::new();
while let Some(entry) = out_rx.recv().await {
results.push(entry.name.clone());
results.push(entry.name);
}
handle.await.unwrap().unwrap();
@@ -8533,7 +8533,7 @@ mod test {
let mut results = Vec::new();
while let Some(entry) = out_rx.recv().await {
results.push(entry.name.clone());
results.push(entry.name);
}
handle.await.unwrap().unwrap();
@@ -9177,7 +9177,7 @@ mod test {
let mut results = Vec::new();
while let Some(entry) = out_rx.recv().await {
results.push(entry.name.clone());
results.push(entry.name);
}
handle.await.unwrap().unwrap();
@@ -9265,7 +9265,7 @@ mod test {
let mut results = Vec::new();
while let Some(entry) = out_rx.recv().await {
results.push(entry.name.clone());
results.push(entry.name);
}
handle.await.unwrap().unwrap();
@@ -9295,7 +9295,7 @@ mod test {
let mut results = Vec::new();
while let Some(entry) = out_rx.recv().await {
results.push(entry.name.clone());
results.push(entry.name);
}
handle.await.unwrap().unwrap();
@@ -9367,7 +9367,7 @@ mod test {
let mut results = Vec::new();
while let Some(entry) = out_rx.recv().await {
results.push(entry.name.clone());
results.push(entry.name);
}
handle.await.unwrap().unwrap();
@@ -9401,7 +9401,7 @@ mod test {
let mut results = Vec::new();
while let Some(entry) = out_rx.recv().await {
results.push(entry.name.clone());
results.push(entry.name);
}
handle.await.unwrap().unwrap();
@@ -9491,7 +9491,7 @@ mod test {
let mut results = Vec::new();
while let Some(entry) = out_rx.recv().await {
assert!(entry.is_object(), "the directory-marker object must shadow the same-named prefix dir");
results.push(entry.name.clone());
results.push(entry.name);
}
handle.await.unwrap().unwrap();
@@ -9514,7 +9514,7 @@ mod test {
let mut results = Vec::new();
while let Some(entry) = out_rx.recv().await {
results.push(entry.name.clone());
results.push(entry.name);
}
handle.await.unwrap().unwrap();
@@ -9538,7 +9538,7 @@ mod test {
let mut results = Vec::new();
while let Some(entry) = out_rx.recv().await {
results.push(entry.name.clone());
results.push(entry.name);
}
handle.await.unwrap().unwrap();
+123
View File
@@ -427,6 +427,7 @@ mod bucket;
mod bucket_fence;
pub(crate) use bucket::await_bucket_namespace_operation;
pub use bucket_fence::BucketIncarnationFenceGuard;
pub(crate) mod delete_recovery_inventory;
pub(crate) mod delete_recovery_runtime;
mod heal;
pub use heal::{HealObjectAbsenceProof, HealObjectStorageResult};
@@ -451,6 +452,8 @@ pub use object::{
mod peer;
mod rebalance;
#[cfg(test)]
mod usage_watch_tests;
#[cfg(test)]
pub(crate) use rebalance::PoolSpaceForTest;
pub(crate) mod utils;
@@ -829,6 +832,13 @@ impl ECStore {
runtime_sources::bucket_metadata_sys()
}
/// Recover this instance's weak metadata owner without accepting a
/// process-global store belonging to another scanner or test fixture.
pub async fn scanner_usage_watch_store_handle(&self) -> Option<Arc<Self>> {
let owner = crate::bucket::metadata_sys::object_store_if_initialized_in(&self.ctx).await?;
std::ptr::eq(self, owner.as_ref()).then_some(owner)
}
/// Get the global endpoints
pub fn endpoints(&self) -> EndpointServerPools {
runtime_sources::endpoint_pools().unwrap_or_else(|| Vec::new().into())
@@ -906,6 +916,119 @@ impl ECStore {
list_objects::scanner_namespace_mutation_generation().saturating_add(self.ctx.namespace_commit_generation())
}
/// Query named buckets against their current storage-owned incarnation.
/// Unknown identities, expired tokens and resource pressure remain explicit
/// Unknown entries; absence from a dirty list never creates a clean proof.
pub async fn scanner_usage_watch(
&self,
requests: &[crate::runtime::usage_watch::ScannerUsageWatchRequest],
) -> Vec<crate::runtime::usage_watch::ScannerUsageWatchObservation> {
use crate::runtime::usage_watch::ScannerUsageWatchObservation;
let mut observations = Vec::with_capacity(requests.len().min(32));
if requests.len() > 32 {
return observations;
}
let Some(owner) = self.scanner_usage_watch_owner_id() else {
return requests
.iter()
.map(|request| ScannerUsageWatchObservation {
bucket: request.bucket.clone(),
known: None,
})
.collect();
};
let local_map = self.ctx.local_disk_map();
let local_disks = local_map.read().await;
let bindings_complete = !local_disks.is_empty()
&& local_disks.values().all(|disk| {
disk.as_ref().is_some_and(|disk| match disk.as_ref() {
crate::disk::Disk::Local(wrapper) => wrapper.get_disk().usage_watch_binding.matches(&self.ctx.usage_watches),
crate::disk::Disk::Remote(_) => false,
})
});
drop(local_disks);
for request in requests {
if request.release {
// Retirement needs the exact saved token, not current bucket
// metadata. Deleted buckets must not retain scarce watch slots.
self.ctx.usage_watches.observe(owner, request);
observations.push(ScannerUsageWatchObservation {
bucket: request.bucket.clone(),
known: None,
});
continue;
}
let current = self.bucket_incarnation_id_from_disk(&request.bucket).await;
let known = match current {
Ok(incarnation) if bindings_complete && incarnation == request.bucket_incarnation && !incarnation.is_nil() => {
self.ctx.usage_watches.observe(owner, request)
}
_ => None,
};
observations.push(ScannerUsageWatchObservation {
bucket: request.bucket.clone(),
known,
});
}
observations
}
/// Invalidate session proofs after replay or an unsupported producer mode
/// transition; this does not change durable dirty ACK responsibility.
pub fn invalidate_scanner_usage_watches(&self) {
self.ctx.usage_watches.invalidate();
}
pub fn scanner_usage_watch_session_id(&self) -> Uuid {
self.ctx.usage_watches.session_id()
}
/// Producer identity for this node's bucket mutation watches. Its session changes on restart.
pub fn scanner_usage_watch_owner_id(&self) -> Option<Uuid> {
self.scanner_usage_watch_owner_for_route(None)
}
/// Resolve an authenticated notification peer against this instance's storage topology.
pub fn scanner_usage_watch_peer_owner_id(&self, grid_host: &str) -> Option<Uuid> {
self.scanner_usage_watch_owner_for_route(Some(grid_host))
}
fn scanner_usage_watch_owner_for_route(&self, grid_host: Option<&str>) -> Option<Uuid> {
let slot = &self.ctx.usage_watches.owners;
let owners = if let Some(owners) = slot.get() {
owners
} else if let Some(endpoints) = self.instance_endpoints() {
// InstanceContext installs this topology once. Publish only a successfully validated table.
let owners = crate::runtime::usage_watch::UsageWatchOwners::from_topology(self.id, &endpoints)?;
slot.get_or_init(|| owners)
} else {
// Direct file-backed stores can open before an instance endpoint registry exists.
let all_local_paths = !self.pools.is_empty()
&& self.pools.iter().all(|pool| {
let endpoints = pool.endpoints.endpoints.as_ref();
!endpoints.is_empty() && endpoints.iter().all(|ep| ep.is_local && ep.get_type() == EndpointType::Path)
});
return (grid_host.is_none() && !self.id.is_nil() && all_local_paths).then_some(self.id);
};
if owners.deployment != self.id {
return None;
}
match grid_host {
Some(host) => owners.peers.get(host).copied(),
None => Some(owners.local),
}
}
/// Revoke an exact local token without awaiting storage or peer I/O. A
/// failed keepalive must not let a later successful query revive its proof.
pub fn scanner_release_usage_watch(&self, request: &crate::runtime::usage_watch::ScannerUsageWatchRequest) {
if request.release
&& let Some(owner) = self.scanner_usage_watch_owner_id()
{
self.ctx.usage_watches.observe(owner, request);
}
}
pub async fn scanner_data_movement_active(&self) -> bool {
let (decommission, rebalance) = tokio::join!(self.is_decommission_running(), self.is_rebalance_started());
decommission || rebalance
+24 -9
View File
@@ -88,7 +88,7 @@ impl ECStore {
/// Resolve a quota lease against this request's captured storage instance.
pub async fn quota_mutation_fence_disk(&self, disk_ref: &str) -> DiskResult<DiskStore> {
let (disk, id) = local_disk_candidate(&self.ctx, disk_ref).await?;
let _owner = admit_local_disk(&self.ctx, &disk, id, false).await?;
let _owner = admit_local_disk(&self.ctx, &disk, id, false, None).await?;
Ok(disk)
}
@@ -106,7 +106,7 @@ impl ECStore {
let path = target.1.to_owned();
self.run_bucket_heal_at_incarnation(target.0, expected, &Default::default(), move |store, volume, _| async move {
let (disk, id) = local_disk_candidate(&store.ctx, &disk_ref).await?;
let _owner = admit_local_disk(&store.ctx, &disk, id, true).await?;
let _owner = admit_local_disk(&store.ctx, &disk, id, true, Some(&volume)).await?;
disk.write_metadata("", &volume, &path, fi).await.map_err(Into::into)
})
.await
@@ -127,7 +127,7 @@ impl ECStore {
let path = target.1.to_owned();
self.run_bucket_heal_at_incarnation(target.0, expected, &Default::default(), move |store, volume, _| async move {
let (disk, id) = local_disk_candidate(&store.ctx, &disk_ref).await?;
let owner = admit_local_disk(&store.ctx, &disk, id, true).await?;
let owner = admit_local_disk(&store.ctx, &disk, id, true, Some(&volume)).await?;
let scope = super::bucket_heal_scope(&volume).ok_or_else(|| StorageError::other("missing bucket heal scope"))?;
scope.check()?;
disk.delete_with_namespace_owner(&volume, &path, opts, Some(Arc::new((scope, owner))))
@@ -159,7 +159,7 @@ impl ECStore {
.map_err(|error| DiskError::other(error.to_string()))?,
);
let (disk, id) = local_disk_candidate(&self.ctx, disk_ref).await?;
let owner = admit_local_disk(&self.ctx, &disk, id, true).await?;
let owner = admit_local_disk(&self.ctx, &disk, id, true, Some(target.0)).await?;
if fence.is_lock_lost() {
return Err(DiskError::other("bucket heal incarnation fence was lost"));
}
@@ -283,7 +283,7 @@ impl ECStore {
) -> DiskResult<crate::disk::multipart_source::MultipartSourceResponse> {
request.validate()?;
let (disk, id) = local_disk_candidate(&self.ctx, disk_ref).await?;
let owner = admit_local_disk(&self.ctx, &disk, id, request.operation.is_mutation()).await?;
let owner = admit_local_disk(&self.ctx, &disk, id, request.operation.is_mutation(), None).await?;
let Disk::Local(wrapper) = disk.as_ref() else {
return Err(DiskError::DiskNotFound);
};
@@ -356,6 +356,7 @@ async fn admit_local_disk(
disk: &DiskStore,
disk_id: Option<Uuid>,
mutates_namespace: bool,
bucket: Option<&str>,
) -> DiskResult<Option<Arc<NamespaceCommitGuard>>> {
if !matches!(disk.as_ref(), Disk::Local(_)) {
return Err(DiskError::DiskNotFound);
@@ -378,7 +379,10 @@ async fn admit_local_disk(
}
// Admission linearizes under the registry read: replacement/quarantine
// before this point rejects; later changes do not revoke physical I/O.
Ok(mutates_namespace.then(|| ctx.begin_namespace_commit()))
Ok(mutates_namespace.then(|| match bucket {
Some(bucket) => ctx.begin_namespace_commit_for_bucket(bucket),
None => ctx.begin_namespace_commit(),
}))
}
async fn rename_local_data_with_ctx(
@@ -398,7 +402,18 @@ async fn rename_local_data_with_ctx(
}
let mutates_namespace = !is_meta_bucketname(source.0) || !is_meta_bucketname(destination.0);
let admit_started = rustfs_io_metrics::put_stage_timer();
let owner = admit_local_disk(ctx, &disk, disk_id, mutates_namespace).await?;
let owner = admit_local_disk(
ctx,
&disk,
disk_id,
mutates_namespace,
Some(if !is_meta_bucketname(destination.0) {
destination.0
} else {
source.0
}),
)
.await?;
if source.0 == crate::disk::RUSTFS_META_MULTIPART_BUCKET {
let Disk::Local(wrapper) = disk.as_ref() else {
return Err(DiskError::DiskNotFound);
@@ -444,7 +459,7 @@ async fn undo_local_write_with_ctx(
return Err(DiskError::other("target undo requires undo_write"));
}
let (disk, disk_id) = local_disk_candidate(ctx, disk_ref).await?;
let owner = admit_local_disk(ctx, &disk, disk_id, !is_meta_bucketname(volume)).await?;
let owner = admit_local_disk(ctx, &disk, disk_id, !is_meta_bucketname(volume), Some(volume)).await?;
let physical_owner = owner.as_ref().map(|owner| owner.clone() as Arc<dyn Send + Sync>);
let result = disk
.undo_write_with_namespace_owner(volume, path, fi, opts, physical_owner)
@@ -874,7 +889,7 @@ mod tests {
}
drop(entries);
assert!(
matches!(admit_local_disk(&ctx, &disk, None, true).await, Err(DiskError::DiskNotFound)),
matches!(admit_local_disk(&ctx, &disk, None, true, None).await, Err(DiskError::DiskNotFound)),
"{state}"
);
assert!(!ctx.namespace_commits_pending());
+13 -13
View File
@@ -1722,20 +1722,20 @@ mod tests {
.await
.expect("bucket should be created in both pools");
for pool_index in 0..2 {
assert!(store.pools.len() >= 2);
for (pool_index, pool) in store.pools.iter().enumerate().take(2) {
let mut reader = PutObjReader::from_vec(format!("pool-{pool_index}-history").into_bytes());
store.pools[pool_index]
.put_object(
&bucket,
object,
&mut reader,
&ObjectOptions {
versioned: true,
..Default::default()
},
)
.await
.expect("historical version should be stored");
pool.put_object(
&bucket,
object,
&mut reader,
&ObjectOptions {
versioned: true,
..Default::default()
},
)
.await
.expect("historical version should be stored");
}
let marker = store.pools[0]
.delete_object(
@@ -0,0 +1,216 @@
// Copyright 2026 RustFS Team
// Licensed under the Apache License, Version 2.0.
use super::*;
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::runtime::usage_watch::{ScannerUsageWatchKnown, ScannerUsageWatchRequest};
use crate::storage_api_contracts::object::{ObjectIO as _, ObjectOperations as _};
async fn known(store: &ECStore, request: &ScannerUsageWatchRequest) -> ScannerUsageWatchKnown {
tokio::time::timeout(Duration::from_secs(10), async {
loop {
let response = store.scanner_usage_watch(std::slice::from_ref(request)).await;
assert_eq!(response.len(), 1);
assert_eq!(response[0].bucket, request.bucket);
let proof = response[0].known.clone().expect("storage-backed watch proof");
if proof.inflight == 0 {
return proof;
}
// API quorum completion can precede a remaining physical tail.
tokio::time::sleep(Duration::from_millis(1)).await;
}
})
.await
.expect("all counted physical tails settle")
}
fn watch_node_topology(local_nodes: &[usize]) -> EndpointServerPools {
let endpoints = (0..4)
.map(|slot| {
let node = slot % 2;
let url = format!("http://node-{node}:9000/data{slot}");
let mut endpoint = Endpoint::try_from(url.as_str()).expect("node endpoint");
endpoint.is_local = local_nodes.contains(&node);
endpoint.set_pool_index(0);
endpoint.set_set_index(0);
endpoint.set_disk_index(slot);
endpoint
})
.collect::<Vec<_>>();
vec![crate::layout::endpoints::PoolEndpoints {
legacy: false,
set_count: 1,
drives_per_set: endpoints.len(),
endpoints: endpoints.into(),
cmd_line: String::new(),
platform: String::new(),
}]
.into()
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn shared_deployment_issues_distinct_usage_watch_owners_through_both_node_views() {
// Open the same formatted storage with two independent producer registries.
// Each sees the same deployment and bucket through its own node's routing view.
let (_dirs, first) = isolated_store_over_temp_disks().await;
let disks = EndpointServerPools::from(vec![first.pools[0].endpoints.clone()]);
let second_ctx = Arc::new(InstanceContext::new());
init_local_disks_with_instance_ctx(&second_ctx, disks.clone())
.await
.expect("second disk registry");
let second = ECStore::new_with_instance_ctx(
"127.0.0.1:0".parse().expect("local test address"),
disks,
CancellationToken::new(),
second_ctx,
)
.await
.expect("second storage instance");
assert_eq!(first.id, second.id, "both producers read the same deployment from disk");
assert_eq!(
first.scanner_usage_watch_owner_id(),
Some(first.id),
"a direct file-backed store keeps its owner"
);
assert_eq!(first.scanner_usage_watch_peer_owner_id("http://node-1:9000"), None);
first.ctx.set_endpoints(watch_node_topology(&[0]));
second.ctx.set_endpoints(watch_node_topology(&[1]));
init_bucket_metadata_sys(first.clone(), Vec::new()).await;
init_bucket_metadata_sys(second.clone(), Vec::new()).await;
let bucket = "shared-watch-deployment";
first
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create shared bucket");
let incarnation = first
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("first incarnation");
assert_eq!(
second
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("second incarnation"),
incarnation
);
let mut request = ScannerUsageWatchRequest {
bucket: bucket.into(),
bucket_incarnation: incarnation,
watch_id: None,
session_id: None,
release: false,
};
let a = known(&first, &request).await;
let b = known(&second, &request).await;
assert_ne!(a.owner_id, b.owner_id, "watch owners must distinguish nodes in one deployment");
assert_eq!(first.scanner_usage_watch_owner_id(), Some(a.owner_id));
assert_eq!(second.scanner_usage_watch_owner_id(), Some(b.owner_id));
assert_eq!(first.scanner_usage_watch_peer_owner_id("http://node-1:9000"), Some(b.owner_id));
assert_eq!(second.scanner_usage_watch_peer_owner_id("http://node-0:9000"), Some(a.owner_id));
assert_eq!(first.scanner_usage_watch_peer_owner_id("http://node-0:9000"), None);
assert_eq!(first.scanner_usage_watch_peer_owner_id("http://unknown:9000"), None);
assert_ne!(a.session_id, b.session_id);
request.watch_id = Some(a.watch_id);
request.session_id = Some(a.session_id);
request.release = true;
second.scanner_release_usage_watch(&request);
assert!(
known(
&second,
&ScannerUsageWatchRequest {
watch_id: Some(b.watch_id),
session_id: Some(b.session_id),
release: false,
..request.clone()
}
)
.await
.unchanged_since(&b)
);
first.scanner_release_usage_watch(&request);
request.release = false;
assert!(
first.scanner_usage_watch(&[request]).await[0].known.is_none(),
"released token cannot be revived"
);
}
#[test]
fn usage_watch_routes_require_one_unambiguous_local_node() {
use crate::runtime::usage_watch::UsageWatchOwners;
let deployment = Uuid::new_v4();
for local_nodes in [&[][..], &[0, 1][..]] {
assert!(UsageWatchOwners::from_topology(deployment, &watch_node_topology(local_nodes)).is_none());
}
assert!(UsageWatchOwners::from_topology(Uuid::nil(), &watch_node_topology(&[0])).is_none());
let mut incomplete = watch_node_topology(&[0]);
incomplete.0[0].drives_per_set += 1;
assert!(UsageWatchOwners::from_topology(deployment, &incomplete).is_none());
}
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
async fn usage_watch_api_tracks_real_put_delete_and_bucket_recreation() {
let (_dirs, store) = isolated_store_over_temp_disks().await;
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
let bucket = "watch-real-store";
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create watched bucket");
let incarnation = store
.bucket_incarnation_id_from_disk(bucket)
.await
.expect("bucket incarnation");
let mut request = ScannerUsageWatchRequest {
bucket: bucket.into(),
bucket_incarnation: incarnation,
watch_id: None,
release: false,
session_id: None,
};
let first = known(&store, &request).await;
assert!(first.is_well_formed());
assert_eq!(first.inflight, 0);
request.watch_id = Some(first.watch_id);
assert!(known(&store, &request).await.unchanged_since(&first));
let reader =
rustfs_rio::HashReader::from_stream(std::io::Cursor::new(b"payload"), 7, 7, None, None, false).expect("PUT body reader");
store
.put_object(bucket, "object", &mut PutObjReader::new(reader), &ObjectOptions::default())
.await
.expect("real PUT");
let written = known(&store, &request).await;
assert_eq!(written.watch_id, first.watch_id);
assert!(written.revision > first.revision);
assert_eq!(written.inflight, 0);
store
.delete_object(bucket, "object", ObjectOptions::default())
.await
.expect("real object delete");
let deleted = known(&store, &request).await;
assert_eq!(deleted.watch_id, first.watch_id);
assert!(deleted.revision > written.revision);
assert_eq!(deleted.inflight, 0);
store
.delete_bucket(bucket, &DeleteBucketOptions::default())
.await
.expect("delete empty bucket");
assert_eq!(store.scanner_usage_watch(std::slice::from_ref(&request)).await[0].known, None);
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("recreate watched bucket");
let recreated_incarnation = store.bucket_incarnation_id_from_disk(bucket).await.expect("new incarnation");
assert_ne!(recreated_incarnation, incarnation);
assert_eq!(store.scanner_usage_watch(std::slice::from_ref(&request)).await[0].known, None);
request.bucket_incarnation = recreated_incarnation;
assert_eq!(store.scanner_usage_watch(std::slice::from_ref(&request)).await[0].known, None);
request.watch_id = None;
let recreated = known(&store, &request).await;
assert_ne!(recreated.watch_id, first.watch_id);
assert!(!recreated.unchanged_since(&first));
assert_eq!(recreated.inflight, 0);
}
@@ -0,0 +1,35 @@
// Copyright 2026 RustFS Team
// SPDX-License-Identifier: Apache-2.0
mod storage_api;
use storage_api::local_drive_retirement::{Disk, DiskAPI, DiskOption, Endpoint, new_disk};
#[tokio::test]
async fn local_wrapper_retirement_is_shared_and_independent_of_monitoring() {
let dir = tempfile::tempdir().expect("retirement fixture directory");
let endpoint = Endpoint::try_from(dir.path().to_str().expect("UTF-8 fixture path")).expect("local endpoint");
let disk = new_disk(
&endpoint,
&DiskOption {
cleanup: false,
health_check: false,
},
)
.await
.expect("local disk");
let Disk::Local(wrapper) = disk.as_ref() else { panic!("fixture must use a local disk") };
let cloned = wrapper.clone();
let observer = disk.local_retirement_token().expect("local retirement observer");
let cancelled_observer = disk.local_retirement_token().expect("another observer");
cancelled_observer.cancel();
assert!(!observer.is_cancelled(), "observers cannot retire the wrapper");
wrapper.stop_monitoring().await;
assert!(!observer.is_cancelled(), "monitoring policy is not retirement");
cloned.close().await.expect("close shared wrapper");
assert!(observer.is_cancelled(), "clones share retirement");
assert!(
disk.local_retirement_token().expect("late observer").is_cancelled(),
"late observers see retirement"
);
disk.close().await.expect("retirement is idempotent");
}
+5
View File
@@ -94,3 +94,8 @@ pub(crate) mod pinned_erasure_fixtures {
storage::init_local_disks_with_instance_ctx,
};
}
pub(crate) mod local_drive_retirement {
pub(crate) use super::{DiskAPI, DiskOption, Endpoint, new_disk};
pub(crate) use rustfs_ecstore::api::disk::Disk;
}
+1 -1
View File
@@ -905,7 +905,7 @@ mod tests {
assert_indexed_matches_scalar(&original, selected);
}
if inline_mode > 0 {
let mut legacy_null = original.clone();
let mut legacy_null = original;
let payload = legacy_null
.data
.find(NULL_VERSION_ID)
+1 -5
View File
@@ -113,11 +113,7 @@ impl FileMetaVersionHeader {
}
// Check erasure coding parameters
if self.has_ec() && (self.ec_n == 0 || self.ec_m == 0 || self.ec_m < self.ec_n) {
return false;
}
true
!(self.has_ec() && (self.ec_n == 0 || self.ec_m == 0 || self.ec_m < self.ec_n))
}
}
+4 -7
View File
@@ -4223,9 +4223,10 @@ mod tests {
o_hdr.ec_m = o_m;
o_hdr.ec_n = o_n;
let actual = FileMeta::headers_match_strict(&s_hdr, &o_hdr);
let expected = {
let mut s_cmp = s_hdr.clone();
let mut o_cmp = o_hdr.clone();
let mut s_cmp = s_hdr;
let mut o_cmp = o_hdr;
if s_cmp.has_ec() != o_cmp.has_ec() {
s_cmp.ec_m = 0;
s_cmp.ec_n = 0;
@@ -4234,11 +4235,7 @@ mod tests {
}
s_cmp == o_cmp
};
assert_eq!(
FileMeta::headers_match_strict(&s_hdr, &o_hdr),
expected,
"mismatch for s=({s_m},{s_n}) vs o=({o_m},{o_n})"
);
assert_eq!(actual, expected, "mismatch for s=({s_m},{s_n}) vs o=({o_m},{o_n})");
}
}
}
+2 -2
View File
@@ -570,7 +570,7 @@ mod tests {
tonic::Status::internal("operation was canceled"),
] {
assert!(
!Error::Storage(EcstoreError::from(status).clone()).is_recoverable_heal(),
!Error::Storage(EcstoreError::from(status)).is_recoverable_heal(),
"cancellation text alone must not change application error classification"
);
}
@@ -718,7 +718,7 @@ mod tests {
for error in [
Error::Disk(DiskError::from(status.clone())),
Error::Storage(EcstoreError::from(status.clone())),
Error::Io(std::io::Error::from(DiskError::from(status.clone()))),
Error::Io(std::io::Error::from(DiskError::from(status))),
] {
assert!(!error.is_recoverable_heal(), "application RPC errors must stay terminal: {error:?}");
}
+1 -1
View File
@@ -543,7 +543,7 @@ impl HealManager {
.filter_map(|target| local_endpoints.get(target).cloned())
.collect::<Vec<_>>();
if targets.len() != state.replacement_targets.len() {
deferred_recovery_sets.insert(state.set_disk_id.clone());
deferred_recovery_sets.insert(state.set_disk_id);
continue;
}
// A pending handoff may resolve the predecessor to
+28 -12
View File
@@ -257,8 +257,6 @@ impl HealManager {
let notify_clone = notify.clone();
let root_recovery_clone = root_recovery.clone();
let manager_cancel_token = cancel_token.clone();
let task_type_label_for_spawn = task_type_label.clone();
let task_set_label_for_spawn = task_set_label.clone();
let config_for_spawn = config.clone();
// start heal task
@@ -270,8 +268,8 @@ impl HealManager {
subsystem = LOG_SUBSYSTEM_MANAGER,
task_id,
priority = ?task_priority,
heal_type = %task_type_label_for_spawn,
set = %task_set_label_for_spawn,
heal_type = %task_type_label,
set = %task_set_label,
state = "task_started",
"Heal scheduler task started"
);
@@ -286,8 +284,8 @@ impl HealManager {
component = LOG_COMPONENT_HEAL,
subsystem = LOG_SUBSYSTEM_MANAGER,
task_id,
heal_type = %task_type_label_for_spawn,
set = %task_set_label_for_spawn,
heal_type = %task_type_label,
set = %task_set_label,
state = "task_completed",
"Heal scheduler task completed"
);
@@ -300,8 +298,8 @@ impl HealManager {
subsystem = LOG_SUBSYSTEM_MANAGER,
state = "verification_unavailable",
task_id,
heal_type = %task_type_label_for_spawn,
set = %task_set_label_for_spawn,
heal_type = %task_type_label,
set = %task_set_label,
detail = %e,
"Heal attempt completed without independent verification"
);
@@ -314,8 +312,8 @@ impl HealManager {
component = LOG_COMPONENT_HEAL,
subsystem = LOG_SUBSYSTEM_MANAGER,
task_id,
heal_type = %task_type_label_for_spawn,
set = %task_set_label_for_spawn,
heal_type = %task_type_label,
set = %task_set_label,
state = "task_retrying",
retry_attempt = task.retry_attempts.saturating_add(1),
error = %e,
@@ -328,8 +326,8 @@ impl HealManager {
subsystem = LOG_SUBSYSTEM_MANAGER,
state = "task_failed",
task_id,
heal_type = %task_type_label_for_spawn,
set = %task_set_label_for_spawn,
heal_type = %task_type_label,
set = %task_set_label,
error = %e,
"Heal scheduler task failed"
});
@@ -494,11 +492,29 @@ impl HealManager {
}
publish_verified_mrf_repair_events(&notice_targets, &completed_status_for_verified_events);
publish_unverified_legacy_mrf_events(&notice_targets, &completed_status_for_verified_events, &result);
// Single-object MRF attempts retain one storage result.
// Use the original result even if the report was size
// limited, without certifying independent integrity.
let unverified_repair_bytes = if completed_status == HealTaskStatus::Completed
&& matches!(&result, Err(Error::LegacyMrfUnverified { .. }))
{
task.result_items
.read()
.await
.iter()
.find(|(_, item)| item.drives_healed().is_some_and(|healed| healed > 0))
.map(|(_, item)| u64::try_from(item.object_size).unwrap_or(u64::MAX))
} else {
None
};
// update statistics
let mut stats = statistics_clone.write().await;
match completed_status {
HealTaskStatus::Completed if matches!(&result, Err(Error::LegacyMrfUnverified { .. })) => {
stats.update_unverified_task_completion();
if let Some(bytes) = unverified_repair_bytes {
stats.add_unverified_repair(bytes);
}
}
HealTaskStatus::Completed => {
stats.update_task_completion(true);
+52 -4
View File
@@ -1106,7 +1106,10 @@ impl HealStorageAPI for MockStorage {
if bucket.starts_with("legacy-ambiguity-scheduler-") {
return Err(Error::Storage(EcstoreError::DeleteRecoveryLegacyAmbiguous));
}
if bucket == "legacy-held-pending-scheduler" {
if matches!(
bucket,
"legacy-held-pending-scheduler" | "legacy-held-repaired-scheduler" | "legacy-held-repaired-oversized-scheduler"
) {
let drives = vec![rustfs_madmin::heal_commands::HealDriveInfo {
state: "ok".to_owned(),
..Default::default()
@@ -1114,8 +1117,25 @@ impl HealStorageAPI for MockStorage {
return Ok((
HealResultItem {
legacy_identity_unverified: true,
detail: rustfs_heal_contracts::heal_channel::LEGACY_OBJECT_IDENTITY_UNVERIFIED_DETAIL.to_owned(),
before: rustfs_madmin::heal_commands::Infos { drives: drives.clone() },
resolved_version_id: Some([0; 16]),
disk_count: 1,
object_size: 8192,
detail: if bucket == "legacy-held-repaired-oversized-scheduler" {
"x".repeat(MAX_COMPLETED_HEAL_RESULT_BYTES)
} else {
rustfs_heal_contracts::heal_channel::LEGACY_OBJECT_IDENTITY_UNVERIFIED_DETAIL.to_owned()
},
before: rustfs_madmin::heal_commands::Infos {
drives: vec![rustfs_madmin::heal_commands::HealDriveInfo {
state: if bucket.starts_with("legacy-held-repaired-") {
"missing"
} else {
"ok"
}
.to_owned(),
..Default::default()
}],
},
after: rustfs_madmin::heal_commands::Infos { drives },
..Default::default()
},
@@ -2171,11 +2191,32 @@ fn only_typed_mrf_result_can_pause_unverified_legacy() {
#[tokio::test(flavor = "current_thread")]
async fn mrf_legacy_completed_attempt_retains_responsibility_without_failed_statistics() {
mrf_legacy_completed_attempt_scenario(false, false).await;
}
#[tokio::test(flavor = "current_thread")]
async fn mrf_legacy_repaired_attempt_retains_responsibility_and_physical_statistics() {
mrf_legacy_completed_attempt_scenario(true, false).await;
}
#[tokio::test(flavor = "current_thread")]
async fn mrf_legacy_repair_statistics_survive_completed_result_truncation() {
mrf_legacy_completed_attempt_scenario(true, true).await;
}
async fn mrf_legacy_completed_attempt_scenario(repaired: bool, oversized_result: bool) {
use crate::heal::failure_logging::test_support::FailureLogCapture;
let capture = FailureLogCapture::default();
let _subscriber = tracing::subscriber::set_default(capture.clone());
let (target, _) = unverified_legacy_fixture("legacy-held-pending-scheduler");
let bucket = if oversized_result {
"legacy-held-repaired-oversized-scheduler"
} else if repaired {
"legacy-held-repaired-scheduler"
} else {
"legacy-held-pending-scheduler"
};
let (target, _) = unverified_legacy_fixture(bucket);
let bucket = target.bucket.clone();
let anchor = target.durable_anchor.clone().expect("durable MRF anchor");
let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None);
@@ -2202,8 +2243,15 @@ async fn mrf_legacy_completed_attempt_retains_responsibility_without_failed_stat
assert_eq!(stats.failed_tasks, 0);
assert_eq!(stats.successful_tasks, 0);
assert_eq!(stats.total_objects_healed, 0);
assert_eq!(stats.total_bytes_healed, 0);
assert_eq!(stats.total_unverified_objects_repaired, u64::from(repaired));
assert_eq!(stats.total_unverified_bytes_repaired, if repaired { 8192 } else { 0 });
let report = manager.get_task_report(&receipt.task_id).await.expect("retained task report");
assert_eq!(report.status, HealTaskStatus::Completed);
if oversized_result {
assert!(report.result_items_truncated);
assert!(report.result_items.is_empty());
}
let outcome = report.outcome.expect("retained unknown outcome");
assert_eq!(outcome.counters.unknown, 1);
assert_eq!(outcome.counters.failed, 0);
+48 -7
View File
@@ -50,6 +50,7 @@ use uuid::Uuid;
use crate::heal::task::{HealOptions, HealPriority, HealRequest, HealType};
pub mod management;
mod partial_write;
pub use partial_write::{MrfLegacyResponsibility, MrfOperatorAcceptance};
use partial_write::{PartialWrites, ResponsibilityCheckpoint};
@@ -64,6 +65,8 @@ static GLOBAL_MRF_LIFECYCLE_SENDER: std::sync::OnceLock<mpsc::Sender<MrfLifecycl
pub enum MrfLifecycleControlError {
#[error("MRF lifecycle controller is unavailable")]
Unavailable,
#[error("MRF read query budget is exhausted; retry later")]
Busy,
#[error("the MRF checkpoint is unreadable; retain its files and restore a compatible reader")]
CheckpointUnreadable,
#[error("the requested responsibility changed or does not exist")]
@@ -94,7 +97,7 @@ pub struct MrfLegacyRiskAcceptanceRequest {
pub request_id: Uuid,
}
#[derive(Clone, Debug, Serialize)]
#[derive(Clone, Debug, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct MrfLegacyResponsibilitySnapshot {
pub contract_version: u32,
@@ -119,14 +122,19 @@ pub async fn list_legacy_responsibilities(
"cursor must be non-nil and limit must be between 1 and 256".to_string(),
));
}
let permit = management::read_permit()?;
let sender = GLOBAL_MRF_LIFECYCLE_SENDER
.get()
.ok_or(MrfLifecycleControlError::Unavailable)?;
let (response, receiver) = tokio::sync::oneshot::channel();
sender
.send(MrfLifecycleCommand::List { after, limit, response })
.await
.map_err(|_| MrfLifecycleControlError::Unavailable)?;
.try_send(MrfLifecycleCommand::List {
after,
limit,
response,
_permit: permit,
})
.map_err(management::queue_error)?;
receiver.await.map_err(|_| MrfLifecycleControlError::Unavailable)?
}
@@ -181,7 +189,13 @@ pub async fn accept_unverified_legacy_risk(request: MrfLegacyRiskAcceptanceReque
}
enum MrfLifecycleCommand {
Get {
id: Uuid,
_permit: tokio::sync::OwnedSemaphorePermit,
response: tokio::sync::oneshot::Sender<Result<management::MrfResponsibilityObservation, MrfLifecycleControlError>>,
},
List {
_permit: tokio::sync::OwnedSemaphorePermit,
after: Option<Uuid>,
limit: usize,
response: tokio::sync::oneshot::Sender<Result<MrfLegacyResponsibilitySnapshot, MrfLifecycleControlError>>,
@@ -1325,8 +1339,31 @@ impl MrfRuntime {
async fn handle_lifecycle_command(&mut self, manager: &HealManager, command: MrfLifecycleCommand) {
match command {
MrfLifecycleCommand::List { after, limit, response } => {
let _ = response.send(self.list_unverified_legacy(after, limit));
MrfLifecycleCommand::List {
after,
limit,
response,
_permit,
} => {
if !response.is_closed() {
let _ = response.send(self.list_unverified_legacy(after, limit));
}
return;
}
MrfLifecycleCommand::Get { id, response, _permit } => {
if !response.is_closed() {
let result = if self.replay_readable {
Ok(management::MrfResponsibilityObservation {
contract_version: 1,
observed_at_ms: management::now_ms(),
responsibility: self.partial_writes.responsibility(id),
})
} else {
Err(MrfLifecycleControlError::CheckpointUnreadable)
};
let _ = response.send(result);
}
return;
}
MrfLifecycleCommand::Recheck {
responsibility_id,
@@ -1826,6 +1863,7 @@ impl MrfRuntime {
gauge!("rustfs_heal_mrf_legacy_generation_unknown").set(metric_f64(lifecycle.legacy_generation_unknown_count));
gauge!("rustfs_heal_mrf_bucket_incarnation_changed").set(metric_f64(lifecycle.bucket_incarnation_changed_count));
gauge!("rustfs_heal_mrf_operator_accepted_unverified").set(metric_f64(lifecycle.operator_accepted_count));
management::publish_summary(self, &lifecycle);
let now_ms = std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|duration| u64::try_from(duration.as_millis()).unwrap_or(u64::MAX))
@@ -2199,7 +2237,7 @@ async fn replay_into(
// runtime dispatches it after publishing the successor, avoiding
// a duplicate task in the startup-replay and steady-state paths.
if let Ok(Some(anchor)) = manager.durable_mrf_repair_anchor(&intent).await {
durable_replay_anchors.push(anchor.clone());
durable_replay_anchors.push(anchor);
}
let restored = intent_digest(&intent).and_then(|digest| take_lifecycle_record(&mut lifecycle, digest));
partial_writes.push((intent, restored));
@@ -2499,6 +2537,9 @@ async fn finalize_mrf_consumer(
lifecycle_receiver.close();
while let Some(command) = lifecycle_receiver.recv().await {
match command {
MrfLifecycleCommand::Get { response, .. } => {
let _ = response.send(Err(MrfLifecycleControlError::Unavailable));
}
MrfLifecycleCommand::List { response, .. } => {
let _ = response.send(Err(MrfLifecycleControlError::Unavailable));
}
@@ -0,0 +1,315 @@
// Copyright 2026 RustFS Team
// Licensed under the Apache License, Version 2.0.
//! Bounded runtime observations. These values never authorize lifecycle changes or prove retirement.
use super::{GLOBAL_MRF_LIFECYCLE_SENDER, MrfLegacyResponsibility, MrfLifecycleCommand, MrfLifecycleControlError, MrfRuntime};
use serde::{Deserialize, Serialize};
use std::sync::{Arc, LazyLock, RwLock};
use tokio::sync::{OwnedSemaphorePermit, Semaphore, mpsc, oneshot};
use uuid::Uuid;
const READ_CAPACITY: usize = 8;
static READ_BUDGET: LazyLock<Arc<Semaphore>> = LazyLock::new(|| Arc::new(Semaphore::new(READ_CAPACITY)));
static SUMMARY: RwLock<Option<MrfLifecycleSummary>> = RwLock::new(None);
#[derive(Clone, Debug, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct MrfLifecycleSummary {
pub contract_version: u32,
pub sampled_at_ms: u64,
pub checkpoint_readable: bool,
pub checkpoint_owner: Option<Uuid>,
pub checkpoint_sequence: Option<u64>,
pub active_responsibility_count: usize,
pub held_count: usize,
pub legacy_generation_unknown_count: usize,
pub bucket_incarnation_changed_count: usize,
pub operator_accepted_count: usize,
pub best_effort_queue_depth: usize,
pub responsibility_bytes: usize,
}
#[derive(Clone, Debug, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct MrfResponsibilityObservation {
pub contract_version: u32,
pub observed_at_ms: u64,
/// Absence means not observed in this runtime, never verified or discharged.
pub responsibility: Option<MrfLegacyResponsibility>,
}
/// Returns the last published sample without scanning or queueing work.
///
/// # Errors
/// Returns unavailable until the consumer publishes a sample, or after it stops.
pub fn lifecycle_summary() -> Result<MrfLifecycleSummary, MrfLifecycleControlError> {
if GLOBAL_MRF_LIFECYCLE_SENDER.get().is_none_or(mpsc::Sender::is_closed) {
return Err(MrfLifecycleControlError::Unavailable);
}
SUMMARY
.read()
.map_err(|_| MrfLifecycleControlError::Unavailable)?
.clone()
.ok_or(MrfLifecycleControlError::Unavailable)
}
/// Observes one current responsibility, including Active entries excluded from legacy lists.
/// Cancellation skips work that has not started; observations are not durable success receipts.
///
/// # Errors
/// Rejects nil IDs, an exhausted read budget, or an unavailable/unreadable controller.
pub async fn responsibility(id: Uuid) -> Result<MrfResponsibilityObservation, MrfLifecycleControlError> {
if id.is_nil() {
return Err(MrfLifecycleControlError::InvalidAction("responsibility ID must be non-nil".into()));
}
let permit = read_permit()?;
let sender = GLOBAL_MRF_LIFECYCLE_SENDER
.get()
.ok_or(MrfLifecycleControlError::Unavailable)?;
let (response, receiver) = oneshot::channel();
sender
.try_send(MrfLifecycleCommand::Get {
id,
response,
_permit: permit,
})
.map_err(queue_error)?;
receiver.await.map_err(|_| MrfLifecycleControlError::Unavailable)?
}
pub(super) fn read_permit() -> Result<OwnedSemaphorePermit, MrfLifecycleControlError> {
READ_BUDGET
.clone()
.try_acquire_owned()
.map_err(|_| MrfLifecycleControlError::Busy)
}
pub(super) fn queue_error<T>(error: mpsc::error::TrySendError<T>) -> MrfLifecycleControlError {
match error {
mpsc::error::TrySendError::Full(_) => MrfLifecycleControlError::Busy,
mpsc::error::TrySendError::Closed(_) => MrfLifecycleControlError::Unavailable,
}
}
pub(super) fn now_ms() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map_or(0, |time| u64::try_from(time.as_millis()).unwrap_or(u64::MAX))
}
pub(super) fn publish_summary(runtime: &MrfRuntime, counts: &super::partial_write::PartialWriteLifecycleSummary) {
let unverified = counts.held_count
+ counts.legacy_generation_unknown_count
+ counts.bucket_incarnation_changed_count
+ counts.operator_accepted_count;
let summary = MrfLifecycleSummary {
contract_version: 1,
sampled_at_ms: now_ms(),
checkpoint_readable: runtime.replay_readable,
checkpoint_owner: runtime.runtime_checkpoint.map(|(owner, _)| owner),
checkpoint_sequence: runtime.runtime_checkpoint.map(|(_, sequence)| sequence),
active_responsibility_count: runtime.partial_writes.depth().saturating_sub(unverified),
held_count: counts.held_count,
legacy_generation_unknown_count: counts.legacy_generation_unknown_count,
bucket_incarnation_changed_count: counts.bucket_incarnation_changed_count,
operator_accepted_count: counts.operator_accepted_count,
best_effort_queue_depth: runtime.queue.depth(),
responsibility_bytes: runtime.partial_writes.bytes(),
};
if let Ok(mut published) = SUMMARY.write() {
*published = Some(summary);
}
}
#[cfg(test)]
mod tests {
use super::super::{MRF_LIFECYCLE_FORMAT_VERSION, MrfConsumerConfig, MrfQueue, PartialWrites, intent_digest};
use super::*;
use crate::heal::manager::HealManager;
use crate::heal::storage::ECStoreHealStorage;
use rustfs_common::mrf_channel::{MrfIntent, MrfKind, MrfScope};
#[test]
fn cancelled_read_holds_budget_until_dequeued() {
let budget = Arc::new(Semaphore::new(1));
let (response, receiver) = oneshot::channel();
let command = MrfLifecycleCommand::Get {
id: Uuid::new_v4(),
response,
_permit: budget.clone().try_acquire_owned().unwrap(),
};
drop(receiver);
assert!(budget.clone().try_acquire_owned().is_err());
drop(command);
assert!(budget.try_acquire_owned().is_ok());
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn management_queries_preserve_active_records_and_unreadable_checkpoint() {
let env = rustfs_test_utils::TestECStoreEnv::builder()
.prefix("mrf_management_queries")
.build()
.await;
let manager =
HealManager::new_without_root_recovery_for_test(Arc::new(ECStoreHealStorage::new(env.ecstore.clone())), None);
let mut runtime = MrfRuntime {
queue: MrfQueue::new(4, 8192),
partial_writes: PartialWrites::default(),
config: MrfConsumerConfig::default(),
checkpoint_owner: Uuid::new_v4(),
next_checkpoint_sequence: 2,
lifecycle_format_version: MRF_LIFECYCLE_FORMAT_VERSION,
new_since_flush: 0,
dirty: false,
journal_on_disk: true,
retain_replay_journal: true,
replay_readable: true,
durable_replay_anchors: Vec::new(),
replay_cleanup: None,
runtime_checkpoint: Some((Uuid::new_v4(), 1)),
backoff_until: None,
};
for (key, source) in [("active", Some(Uuid::new_v4())), ("legacy", None)] {
runtime
.admit_partial_write(
MrfIntent {
bucket: Arc::from("management"),
object: Arc::from(key),
version_id: None,
kind: MrfKind::PartialWrite,
delete_marker_purge: None,
scope: Some(MrfScope {
pool_index: 0,
set_index: 0,
}),
lease: None,
enqueued_at_ms: 1,
attempts: 0,
},
source,
)
.expect("retain responsibility fixture");
}
let active = runtime
.partial_writes
.checkpoint_records(intent_digest)
.into_iter()
.find(|row| row.source_bucket_incarnation_id.is_some())
.expect("source-bound responsibility")
.responsibility_id;
let before = runtime.snapshot();
let (sender, mut receiver) = mpsc::channel(16);
assert!(GLOBAL_MRF_LIFECYCLE_SENDER.set(sender).is_ok(), "query test requires an isolated process");
assert!(matches!(lifecycle_summary(), Err(MrfLifecycleControlError::Unavailable)));
publish_summary(&runtime, &runtime.partial_writes.lifecycle_summary());
let mut sample = lifecycle_summary().expect("published summary");
assert_eq!(sample.active_responsibility_count, 1);
assert_eq!(sample.legacy_generation_unknown_count, 1);
assert!(sample.checkpoint_readable);
sample.sampled_at_ms = 0;
*SUMMARY.write().expect("summary fixture") = Some(sample.clone());
let permits: Vec<_> = (0..READ_CAPACITY).map(|_| read_permit().expect("read slot")).collect();
assert!(matches!(responsibility(active).await, Err(MrfLifecycleControlError::Busy)));
assert!(matches!(
super::super::list_legacy_responsibilities(None, 1).await,
Err(MrfLifecycleControlError::Busy)
));
drop(permits);
let (observed, ()) = tokio::join!(responsibility(active), async {
let command = receiver.recv().await.expect("queued Get");
runtime.handle_lifecycle_command(&manager, command).await;
});
let row = observed
.expect("Get succeeds")
.responsibility
.expect("Active responsibility remains observable");
assert_eq!(row.responsibility_id, active);
assert_eq!(row.status, "active");
let (listed, ()) = tokio::join!(super::super::list_legacy_responsibilities(None, 2), async {
let command = receiver.recv().await.expect("queued List");
runtime.handle_lifecycle_command(&manager, command).await;
});
let listed = listed.expect("legacy list succeeds");
assert_eq!(listed.responsibilities.len(), 1);
assert_ne!(listed.responsibilities[0].responsibility_id, active);
assert_eq!(runtime.snapshot(), before, "observations must preserve the retained payloads");
assert_eq!(
lifecycle_summary().expect("cached summary").sampled_at_ms,
sample.sampled_at_ms,
"read commands must not republish summary metrics"
);
for list in [false, true] {
let cancelled = tokio::spawn(async move {
if list {
super::super::list_legacy_responsibilities(None, 1).await.map(|_| ())
} else {
responsibility(active).await.map(|_| ())
}
});
let command = tokio::time::timeout(std::time::Duration::from_secs(5), receiver.recv())
.await
.expect("query is enqueued promptly")
.expect("queued cancellable query");
cancelled.abort();
assert!(cancelled.await.expect_err("query task cancelled").is_cancelled());
assert_eq!(
READ_BUDGET.available_permits(),
READ_CAPACITY - 1,
"queued work retains its slot after client cancellation"
);
runtime.handle_lifecycle_command(&manager, command).await;
assert_eq!(READ_BUDGET.available_permits(), READ_CAPACITY);
assert_eq!(runtime.snapshot(), before);
}
runtime.replay_readable = false;
publish_summary(&runtime, &runtime.partial_writes.lifecycle_summary());
assert!(
!lifecycle_summary()
.expect("unreadable summary remains observable")
.checkpoint_readable
);
let (observed, ()) = tokio::join!(responsibility(active), async {
let command = receiver.recv().await.expect("Get against unreadable checkpoint");
runtime.handle_lifecycle_command(&manager, command).await;
});
assert!(matches!(observed, Err(MrfLifecycleControlError::CheckpointUnreadable)));
let (listed, ()) = tokio::join!(super::super::list_legacy_responsibilities(None, 1), async {
let command = receiver.recv().await.expect("List against unreadable checkpoint");
runtime.handle_lifecycle_command(&manager, command).await;
});
assert!(matches!(listed, Err(MrfLifecycleControlError::CheckpointUnreadable)));
assert_eq!(runtime.snapshot(), before);
receiver.close();
assert!(matches!(responsibility(active).await, Err(MrfLifecycleControlError::Unavailable)));
assert!(
matches!(lifecycle_summary(), Err(MrfLifecycleControlError::Unavailable)),
"a stale sample cannot hide shutdown"
);
assert_eq!(READ_BUDGET.available_permits(), READ_CAPACITY);
}
#[tokio::test]
async fn management_query_bounds_and_queue_errors_are_explicit() {
assert!(matches!(
responsibility(Uuid::nil()).await,
Err(MrfLifecycleControlError::InvalidAction(_))
));
let (sender, receiver) = mpsc::channel(1);
sender.try_send(()).expect("fill queue");
assert!(matches!(
queue_error(sender.try_send(()).expect_err("full queue")),
MrfLifecycleControlError::Busy
));
drop(receiver);
assert!(matches!(
queue_error(sender.try_send(()).expect_err("closed queue")),
MrfLifecycleControlError::Unavailable
));
assert_eq!(READ_BUDGET.available_permits(), READ_CAPACITY);
}
}
+131 -53
View File
@@ -776,29 +776,9 @@ impl PartialWrites {
) -> (Vec<MrfLegacyResponsibility>, Option<Uuid>) {
let mut selected = BTreeMap::new();
for entry in self.entries.values() {
let (status, bucket_incarnation_id, held_since_ms) = match &entry.state {
ResponsibilityState::Active {} if entry.generation_unobserved() => ("legacy_generation_unobserved", None, None),
ResponsibilityState::Active {} => continue,
ResponsibilityState::SourceAndCurrentGenerationUnknown { since_ms, .. } => {
("source_and_current_generation_unknown", None, Some(*since_ms))
}
ResponsibilityState::HeldUnverifiedLegacy {
bucket_incarnation_id,
since_ms,
} => ("held_unverified_legacy", Some(*bucket_incarnation_id), Some(*since_ms)),
ResponsibilityState::LegacyGenerationUnknown {
observed_bucket_incarnation_id,
detected_at_ms,
} => ("legacy_generation_unknown", Some(*observed_bucket_incarnation_id), Some(*detected_at_ms)),
ResponsibilityState::BucketIncarnationChanged {
observed_bucket_incarnation_id,
detected_at_ms,
..
} => ("bucket_incarnation_changed", Some(*observed_bucket_incarnation_id), Some(*detected_at_ms)),
ResponsibilityState::OperatorAcceptedUnverified {
bucket_incarnation_id, ..
} => ("operator_accepted_unverified", Some(*bucket_incarnation_id), None),
};
if !entry.is_parked() {
continue;
}
if after.is_some_and(|after| entry.responsibility_id <= after) {
continue;
}
@@ -809,33 +789,7 @@ impl PartialWrites {
{
continue;
}
let row = MrfLegacyResponsibility {
responsibility_id: entry.responsibility_id,
bucket: entry.intent.bucket.to_string(),
object: entry.intent.object.to_string(),
source_bucket_incarnation_id: entry.source_bucket_incarnation_id,
version_id: entry
.intent
.version_id
.and_then(|bytes| Uuid::from_slice(&bytes).ok())
.filter(|version| !version.is_nil())
.map(|version| version.to_string()),
pool_index: entry.intent.scope.map(|scope| scope.pool_index),
set_index: entry.intent.scope.map(|scope| scope.set_index),
bucket_incarnation_id,
enqueued_at_ms: entry.intent.enqueued_at_ms,
status,
held_since_ms,
accepted: entry.last_operator_acceptance.clone(),
last_identity_probe_result: entry.last_identity_probe_result,
last_identity_probe_at_ms: entry.last_identity_probe_at_ms,
next_identity_probe_at_ms: entry.generation_unobserved().then(|| {
unix_now_ms().saturating_add(
u64::try_from(entry.next_attempt.saturating_duration_since(Instant::now()).as_millis())
.unwrap_or(u64::MAX),
)
}),
};
let row = Self::responsibility_row(entry);
selected.insert(entry.responsibility_id, row);
if selected.len() > limit + 1 {
selected.pop_last();
@@ -852,6 +806,65 @@ impl PartialWrites {
(selected.into_values().collect(), next_cursor)
}
pub(super) fn responsibility(&self, id: Uuid) -> Option<MrfLegacyResponsibility> {
self.entries
.values()
.find(|entry| entry.responsibility_id == id)
.map(Self::responsibility_row)
}
fn responsibility_row(entry: &Responsibility) -> MrfLegacyResponsibility {
let (status, bucket_incarnation_id, held_since_ms) = match &entry.state {
ResponsibilityState::Active {} if entry.generation_unobserved() => ("legacy_generation_unobserved", None, None),
ResponsibilityState::Active {} => ("active", None, None),
ResponsibilityState::SourceAndCurrentGenerationUnknown { since_ms, .. } => {
("source_and_current_generation_unknown", None, Some(*since_ms))
}
ResponsibilityState::HeldUnverifiedLegacy {
bucket_incarnation_id,
since_ms,
} => ("held_unverified_legacy", Some(*bucket_incarnation_id), Some(*since_ms)),
ResponsibilityState::LegacyGenerationUnknown {
observed_bucket_incarnation_id,
detected_at_ms,
} => ("legacy_generation_unknown", Some(*observed_bucket_incarnation_id), Some(*detected_at_ms)),
ResponsibilityState::BucketIncarnationChanged {
observed_bucket_incarnation_id,
detected_at_ms,
..
} => ("bucket_incarnation_changed", Some(*observed_bucket_incarnation_id), Some(*detected_at_ms)),
ResponsibilityState::OperatorAcceptedUnverified {
bucket_incarnation_id, ..
} => ("operator_accepted_unverified", Some(*bucket_incarnation_id), None),
};
MrfLegacyResponsibility {
responsibility_id: entry.responsibility_id,
bucket: entry.intent.bucket.to_string(),
object: entry.intent.object.to_string(),
source_bucket_incarnation_id: entry.source_bucket_incarnation_id,
version_id: entry
.intent
.version_id
.and_then(|bytes| Uuid::from_slice(&bytes).ok())
.filter(|version| !version.is_nil())
.map(|version| version.to_string()),
pool_index: entry.intent.scope.map(|scope| scope.pool_index),
set_index: entry.intent.scope.map(|scope| scope.set_index),
bucket_incarnation_id,
enqueued_at_ms: entry.intent.enqueued_at_ms,
status: status.to_string(),
held_since_ms,
accepted: entry.last_operator_acceptance.clone(),
last_identity_probe_result: entry.last_identity_probe_result.map(str::to_string),
last_identity_probe_at_ms: entry.last_identity_probe_at_ms,
next_identity_probe_at_ms: entry.generation_unobserved().then(|| {
unix_now_ms().saturating_add(
u64::try_from(entry.next_attempt.saturating_duration_since(Instant::now()).as_millis()).unwrap_or(u64::MAX),
)
}),
}
}
pub(super) fn replace_observed_incarnation(
&mut self,
responsibility_id: Uuid,
@@ -1154,7 +1167,7 @@ fn validate_operator_audit_fields(actor: &str, reason: &str, reference: &str, re
Ok(())
}
#[derive(Clone, Debug, Serialize)]
#[derive(Clone, Debug, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
pub struct MrfLegacyResponsibility {
pub responsibility_id: Uuid,
@@ -1166,11 +1179,11 @@ pub struct MrfLegacyResponsibility {
pub set_index: Option<u32>,
pub bucket_incarnation_id: Option<Uuid>,
pub enqueued_at_ms: u64,
pub status: &'static str,
pub status: String,
pub held_since_ms: Option<u64>,
pub accepted: Option<MrfOperatorAcceptance>,
#[serde(skip_serializing_if = "Option::is_none")]
pub last_identity_probe_result: Option<&'static str>,
pub last_identity_probe_result: Option<String>,
#[serde(skip_serializing_if = "Option::is_none")]
pub last_identity_probe_at_ms: Option<u64>,
/// Process-local schedule; replay restarts the conservative probe delay.
@@ -1230,6 +1243,71 @@ mod tests {
intent
}
#[test]
fn management_query_scale_baseline_and_active_lookup() {
for budget in [8 * 1024 * 1024, 128 * 1024 * 1024] {
let mut writes = PartialWrites::default();
let source = Uuid::new_v4();
let started = std::time::Instant::now();
for index in 0..100_000 {
let item = MrfIntent {
bucket: Arc::from("management-scale"),
object: Arc::from(format!("key-{index:06}")),
version_id: None,
kind: MrfKind::PartialWrite,
scope: Some(MrfScope {
pool_index: 0,
set_index: 0,
}),
lease: None,
enqueued_at_ms: 1,
attempts: 0,
delete_marker_purge: None,
};
if writes
.admit_with_source_incarnation(item, if index % 10 == 0 { Some(source) } else { None }, 100_000, budget)
.is_err()
{
break;
}
}
let admission_ms = started.elapsed().as_millis();
assert!(writes.depth() > 1000);
let active = writes
.entries
.values()
.find(|entry| !entry.is_parked())
.unwrap()
.responsibility_id;
assert_eq!(writes.responsibility(active).unwrap().status, "active");
assert!(writes.responsibility(Uuid::new_v4()).is_none());
let mut elapsed = Vec::new();
let mut after = None;
for _ in 0..32 {
let start = std::time::Instant::now();
let (rows, next) = writes.list_unverified_legacy(after, 256);
let _ = writes.lifecycle_summary();
assert!(writes.responsibility(active).is_some());
elapsed.push(start.elapsed().as_micros());
assert!(rows.len() <= 256);
assert!(
rows.iter()
.all(|row| row.responsibility_id != active && after.is_none_or(|id| row.responsibility_id > id))
);
after = next;
}
elapsed.sort_unstable();
eprintln!(
"MRF query baseline: entries={} bytes={} admission_ms={} page_summary_id_p50_us={} p99_us={}",
writes.depth(),
writes.bytes(),
admission_ms,
elapsed[16],
elapsed[31]
);
}
}
fn risk_acceptance(
responsibility_id: Uuid,
expected_bucket_incarnation_id: Uuid,

Some files were not shown because too many files have changed in this diff Show More