Compare commits

...

38 Commits

Author SHA1 Message Date
overtrue ecce96f51b Run scoped scanner Linux diagnostics 2026-10-28 06:14:13 +08:00
overtrue 03eef1ea79 Stabilize scanner lease timeout retry test 2026-10-28 05:42:21 +08:00
overtrue dbb9d66234 Retry local scanner lease channel timeouts 2026-10-28 05:41:39 +08:00
overtrue 9738480423 Make scanner peer deadline test deterministic 2026-10-28 05:40:54 +08:00
overtrue 980476acf0 Route scanner lease diagnostics through scan boundary 2026-10-28 05:15:50 +08:00
overtrue 019b61cb23 chore(scanner): merge main into lease diagnostics 2026-10-28 04:59:03 +08:00
Chris An 2b56666123 fix(lock): retain state during acquisition (#9014) 2026-10-28 04:57:46 +08:00
overtrue 9133869232 chore(scanner): merge main into lease diagnostics 2026-10-28 04:56:54 +08:00
overtrue 9aed6f57be fix(scanner): add publication lease diagnostics 2026-10-28 04:56:38 +08:00
Chris An a24d305665 fix(scanner): summarize bucket root collection (#9013) 2026-10-28 04:05:44 +08:00
Martin Claesson c47c8f9f74 fix(sftp): allow bucket-scoped root listings (#8998)
* fix(sftp): allow bucket-scoped root listings

Listing the SFTP root required s3:ListAllMyBuckets. Without it a
bucket-scoped user got PermissionDenied on `ls /`; with it they saw every
bucket on the instance. S3 ListBuckets and the WebDAV root (#6298)
already fall back to a per-bucket filtered listing.

When the global ListAllMyBuckets check is denied, list buckets through
the session-aware ListBuckets path instead, so IAM filters the result to
buckets the principal may ListBucket or GetBucketLocation. Principals
with ListAllMyBuckets keep the unfiltered listing, and IAM outages and
timeouts still fail as before.

list_buckets_for_session and its request builder were WebDAV-only; they
are now also compiled with the sftp feature.

* fix(protocols): re-check the session key for bucket-scoped listings

list_buckets_for_session built its request from the identity captured at
login. Through the SFTP root-listing fallback, a service account disabled
or expired after login could keep listing bucket names, unfiltered when
its parent may list every bucket.

Validate the session key with check_key_valid_with_context, as
authorize_protocol_request does for every other protocol operation, build
the request from the fresh credentials and is_owner, map an invalid key
to AccessDenied, and attach the server context.

The SFTP fallback now passes the session's request headers and transport
security, like every other SFTP authorization. DummyBackend fails closed
when no session listing is queued, matching the trait default, and gains
a stall_authorization hook. New tests cover a service account disabled
mid-session, IAM outage and timeout on the global check (no fallback),
non-AccessDenied backend errors, truncation of fallback output and the
fail-closed default.

* test(protocols): compare error code without referencing s3s

The session-listing test compared against s3s::S3ErrorCode, which made
policy_test.rs a new s3s-referencing file and tripped the s3s footprint
ratchet (210 files against a baseline of 209). Compare the code's string
form instead.

* fix(protocols): refresh session bucket discovery without stale copies

Build bucket listing requests directly from freshly validated credentials
and server-scoped ownership instead of cloning and replacing login state.
Reuse the same authorization path for WebDAV capacity discovery.

Preserve typed access-denied checks through the protocol facade, and cover
request conditions, session-policy updates, revocation, listing bounds,
and fallback deadlines with regression tests.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-28 03:53:56 +08:00
Chris An 3a00e9bd42 fix(scanner): add context to set cache failures (#9010) 2026-10-28 03:53:45 +08:00
cxymds c0c60dd91d fix(admin): make pool diagnostics partial-outage safe (#8995) 2026-10-28 03:53:39 +08:00
Chris An b5c253f3ed docs(listing): describe degraded strict quorum limits (#8981) 2026-10-28 03:53:24 +08:00
Ramakrishna Chilaka 63d97f0877 perf(ecstore): coalesce rename_part and multipart completion I/O (#9001) 2026-10-28 03:50:06 +08:00
Chris An 4f04338ee1 test(connect): report drive probe timeout phases (#8977) 2026-10-28 03:27:49 +08:00
Chris An f96c94f950 fix(listing): treat key markers as literal keys (#8978)
* fix(listing): treat version markers as literal keys

* fix(listing): keep start-after and V1 markers literal

* docs(listing): state cursor compatibility removal condition

* fix(listing): resume protected table pages from raw keys

* test(listing): keep protected pagination coverage in app layer
2026-10-28 02:59:46 +08:00
Chris An 5b5d68a444 fix(decommission): defer copies that time out on a namespace lock (#9006)
A node killed mid-copy leaves its namespace lock on the peers until the
lock lease expires. When the restarted leader resumes the decommission
and reaches that key inside the lease, every in-place copy attempt times
out acquiring the lock, and the version was counted as a failed object,
so the pool decommission could no longer complete.

A lock acquisition timeout leaves the source in place, like a version
copy that lost drive quorum, so defer the entry to the end of the round
instead of counting it as failed. A replay that still times out pauses
the decommission through the existing durable-pause path, which is
relaunched automatically.

This is the intermittent failure of the distributed e2e case
scanner_heal_topology_decommission_kill_before_source_cleanup_resumes_without_loss.
2026-10-28 02:59:20 +08:00
Henry Guo 0fb650a408 fix(table-catalog): reload peers after table bucket toggle (#8994)
* fix(table-catalog): reload peers after table bucket toggle

* test(table-catalog): cover bucket toggles across cached peers
2026-10-28 01:57:55 +08:00
Hauser b9f01162b2 feat(tools): inspect BTF scalar member layouts (#9003)
Report selected integer offsets and widths with strict member and payload validation. Cross-check compiler fixtures with bpftool and independent C offsetof/sizeof results.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-28 01:37:31 +08:00
Chris An 5ab1d5191e fix(heal): pace and parallelize replacement recovery (#9005) 2026-10-28 01:10:44 +08:00
Chris An acebee4ea3 fix(ecstore): select the latest version at read quorum like MinIO (#8982)
* fix(ecstore): select the latest version at read quorum like MinIO

GET and HEAD escalated latest-version selection to write quorum when
enough metadata answered and skipped a newer version below write quorum
on degraded reads. LIST and heal already pick the newer side of a split
write at read quorum, so such keys listed but answered 503 until heal.

Select with MinIO's getObjectFileInfo rule: the most common mod time
that reaches the object's data blocks wins, ties go to the newer time,
and one content identity at that time must reach the same quorum. Two
identities tied at the selected time fail instead of resolving by hash
order, and the selected identity's version-stack shape follows disk
order.

* fix(ecstore): resolve selection ties by disk order and guard writes

Review of the read-quorum selection found three gaps:

- Rejecting a same-time half split made explicit-version reads of a
  partially applied tagging update fail and blocked the retry that
  repairs it. Resolve ties to the identity on the lowest-index disk,
  for reads and heal alike, instead of rejecting or using map order.
- Delete markers and empty versions can lower the common read quorum
  below the selected version's data blocks, so HEAD could report a
  version GET cannot decode. Require that many copies, as MinIO's
  getLatestFileInfo does.
- A selection can now return exactly read-quorum disks, and partial
  metadata writes are not rolled back. Refuse an update before writing
  when its target disks cannot reach write quorum.

* fix(ecstore): count identity copies in the data-blocks guard

The first-candidate shortcut counts byte-identical copies, so copies
that differ only in identity-neutral fields such as replication status
were missing from the data-blocks check. When the shortcut's count
falls short, count every copy of the selected content identity before
refusing the version.

* fix(ecstore): keep commit-proof selection for system-bucket reads

Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an
ordinary read of pool.bin as proof that the record committed. Under the
read-quorum rule, a COMMIT that reached only half of a 2+2 set was served
as committed, and the pool_meta_native_physical_* integration tests
failed.

Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's
read-quorum rule, while is_meta_bucketname reads use the previous
algorithm unchanged as select_committed_fileinfo, so a split stays
unreadable until native recovery resolves it.
2026-10-28 00:47:50 +08:00
Chris An 268a5519f1 test(ecstore): stabilize metadata fanout lifecycle checks (#8999) 2026-10-28 00:07:05 +08:00
Chris An e452d837dd docs(scanner): clarify incomplete census diagnostics (#9002) 2026-10-27 23:33:28 +08:00
Chris An 1451f3189b fix(listing): leave delete recovery out of list requests (#8980)
* fix(listing): leave delete recovery out of list requests

* test(listing): expect residue preservation across walks
2026-10-27 23:04:22 +08:00
hector 59bc6d7c94 ci: build linux-aarch64 releases on Ubicloud ARM (#8996) 2026-10-27 21:18:05 +08:00
Ramakrishna Chilaka a56104206e perf(ecstore): reduce multipart source fencing directory and read I/O (#8986)
- Resolve existing directories below base on Linux in a single openat2(BENEATH | NO_SYMLINKS) syscall in open_existing_directory_below_base_std instead of opening and closing every ancestor component.
- Reuse the pinned source directory guard across sibling metadata reads and control probes in validate_source_mutation, load_source, OrdinaryMultipartDeleteCheck::validate, and prepare_multipart_commit_staging.
- Read normal multipart source heads (<= 64 KiB) in a single pass in load_source while preserving constant-memory streaming observation for large prepared histories.
- Record root_missing during load_source, reuse loaded.original in restore_original, and coalesce finish source directory fsync into persist_with_source_sync.
- Reuse the opened commit directory in prepare_rename_commit_guard and LinuxCommitDirectories::validate when source_parent == destination_parent.
2026-10-27 20:23:32 +08:00
Hauser be485e6cab test(scanner): add memory acceptance runner and budget probes (#8988)
* test(scanner): add memory acceptance runner and budget probes

Add owned S3 workload phases with process, endpoint, cgroup and binary
identity checks for scanner memory and scanner-disabled request bursts.
Keep scale, restart, heap-attribution and overall-acceptance limits explicit.

Characterize healthy and seeded coverage-gap cache retention through the
real folder walker and production cache codec without changing runtime code.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): bind acceptance capacity to effective storage

Validate the serving process's complete local disk inventory before load
writes and during runtime sampling. Resolve each disk inside the owned
storage root, reject nested mounts and filesystem mismatches, and bind
storage observations to phase identity for container and native runs.

Cover the unused-mount bypass, CLI-overridden environment, native paths,
symlink escapes, named volumes and stable multi-disk inventory ordering.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(scanner): fence status observations before workload writes

Close each status API window with a second storage-bound process sample
before Phase entry can authorize bucket creation or PUT requests.

Add a fail-before regression for a process change during the initial status
request, asserting no ownership or workload writes occur.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-27 16:45:37 +08:00
Hauser fbd94e129d test(e2e): validate late multipart part transaction packets (#8989)
Retain original prepare and commit-settlement RPCs beyond failed replacement and Abort cleanup, with and without bucket recreation. Require typed refusal or a physically verified idempotent result, unchanged fresh-upload bytes and complete object reads.

Extend scoped proxy observations without rewriting signed frames or trailers, and preserve registered test membership and existing budgets.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-27 13:55:21 +08:00
Hauser cf5a9e0ba3 docs: define the dirty-journal metadata source contract (#8984)
* docs: define the dirty-journal metadata source contract

Specify the single node-owned journal scope: verified bootstrap, physical fencing and tail completion, reader-first replay, exact new responsibility settlement and retained legacy unknowns. Explicitly distinguish the proposed contract from the existing asynchronous hint implementation.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* docs: require receiving peers to revalidate source claims

Keep user metadata, decoded records and authenticated transport distinct from storage-verified dirty-journal authority.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-27 13:54:19 +08:00
Hauser 254592717a feat(tools): add read-only BTF prototype inspection (#8985)
* feat(tools): add read-only BTF prototype inspection

Expose bounded declared prototype headers through a standard-library CLI, preserve explicit void and failure semantics, and wire local contracts plus an independent compiler/bpftool reference into CI.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* test(tools): retain native reference failure diagnostics

Clear checkout credentials and capture bounded native stderr alongside stdout, preserving exact exit status and owned child cleanup. Keep the first reference failure unresolved rather than guessing its missing diagnostics.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* ci(tools): select a concrete bpftool file parser

The hosted PATH wrapper can exist without tools for its Azure kernel. Select an executable ELF by availability/version before comparing the fixed fixture, installing generic tools only when needed. Data comparison errors remain fatal and never trigger tool switching.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-27 12:51:57 +08:00
Hauser aba9e2e9e6 test(e2e): cover partial EC:4 node recovery (#8983)
Verify a sixteen-drive set rejects reads and writes after two nodes stop,
then resumes complete S3 reads and new writes when only one node returns.
Keep the fourth node stopped throughout checks from all three live
coordinators and verify the original 12+4 shard geometry on disk.

Let the existing GET recovery probe retry the compatible SlowDownRead
response only with HTTP 503, retaining fatal status and byte checks.

Register the case in both platform selections and document the boundary
between process recovery and physical disk reattachment acceptance.

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-27 12:38:38 +08:00
Ramakrishna Chilaka 924700a24f docs(agents): add ecstore syscall and blocking-dispatch perf guardrails (#8987)
- Document rules in crates/ecstore/AGENTS.md and the adversarial performance lens against per-disk spawn_blocking hopping, parent-chain openat amplification, un-sized read_to_end buffers, and redundant state reloads under the same lock.

Co-authored-by: Copilot <223556219+Copilot@users.noreply.github.com>
Copilot-Session: d85aaf47-2a4b-4c2c-97a3-dbe8c7717e2a
2026-10-27 11:52:20 +08:00
Chris An b8eef694c7 fix(heal): group newly attached replacement disks (#8973)
fix(heal): batch replacement targets before freezing recovery scope
2026-10-27 11:35:15 +08:00
hector 26d55bb04e ci: remove never-green Performance A/B workflow (#8979) 2026-10-27 11:09:32 +08:00
Hauser c9f7e48a44 fix(observability): correlate physical namespace lock releases (#8974)
* fix(observability): correlate physical namespace lock releases

Map logical namespace guards to their actual granted peer lock IDs and local physical guard identities. Record physical completion only after successful manager release and registry removal, and record validated positive remote unlock responses without changing release behavior.

Exercise real distributed ownership, asynchronous cleanup, duplicate and disabled guards, stale-owner retirement, and explicit release followed by Drop. Document the required diagnostic filters and the remaining filesystem-tail boundaries.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* fix(ecstore): remove unused lock manager test import

The diagnostic regression uses inherent manager methods and does not need the LockManager trait import. Remove it to pass strict warning checks.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-27 10:10:17 +08:00
Hauser 11e1662c90 fix(scanner): reduce memory retention and redundant scan work (#8971)
* fix(scanner): compact completed subtrees during checkpoint sweeps

Release completed object entries while preserving resumable frontier nodes,
including heal rewalks and coverage gaps. Borrow child links during usage
aggregation and remove duplicate update-cache subtree copies.

Add real-disk restart and allocation-retention regressions for #8961.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(scanner): avoid unused progress trees and traversal copies

Bind progress caches to actual subscribers and keep durable scanner metadata
in one authoritative cache. Remove mirror-only heal synchronization, use one
tagged folder queue, and consume child ownership when deleting subtrees.

Preserve observer output, restart coverage and failed-object retry semantics.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

* perf(scanner): compact completed leaf entries in place

Keep completed-folder eligibility and frontier handling in one compaction
boundary. Borrow leaf statistics and update the cached sampling flag without
cloning, deleting, or reinserting the entry; aggregate only non-leaf nodes.

Pin allocation identity, complete metadata, failure guards and caller state
with a regression that fails under the previous compaction implementation.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-10-27 10:01:46 +08:00
Chris An 7ea3aa2a93 fix(listing): keep the V1 NextMarker escape and document optimal limits (#8976)
A cursor for a key that literally ends in a legacy-shaped tag is issued
wrapped in one extra `[rustfs_cache:v2,return:]`, because the listing layer
still strips one legacy tag from incoming markers. ListObjects (V1)
unwrapped it in `calculate_next_marker`; the client then sent the bare key
back as `marker`, the listing layer stripped the key's own tail, and the
next page restarted at the stem and returned the same key forever. Keep the
wrapper in V1 NextMarker and cover the V1 resume path (marker sent back as
StartAfter) on a real store.

Document that sampled list quorum modes can also omit committed objects
whose copies mostly sit outside the sample, and recommend `strict` for
backup, sync, replication and lifecycle tooling.
2026-10-27 08:16:34 +08:00
138 changed files with 12210 additions and 2404 deletions
@@ -6,7 +6,21 @@
- Route every new sync/flush through the durability-mode and bucket override
gates; mode `none` must not pay the new fsync.
- Keep blocking filesystem/CPU work off async runtime threads, but do not split
one small operation into many `spawn_blocking` round trips.
one small operation into many `spawn_blocking` round trips; coalesce `open`,
`read`/`write`, `fsync`, and `close` into one blocking closure per disk. When
coalescing steps across execution lanes, route the combined closure to the
dedicated fsync lane whenever any step performs file or directory sync so
stalled syncs cannot starve foreground metadata reads.
- Check for per-disk syscall amplification (`disks × parts/files`): pin a
directory once (`openat2` on Linux) and reuse the handle across sibling reads
or same-directory renames instead of re-walking ancestor components per file
or re-opening the same parent directory twice.
- When the file size is already known from `fstat` metadata, pre-size the
destination buffer (`len + 1`) before reading to EOF so small metadata reads
complete in a single `read()` + EOF check rather than exponential-growth
reallocations and syscalls.
- Avoid redundant disk re-reads or existence checks under the same exclusive
lock when the state or raw bytes were just loaded or persisted.
- Measure lock hold time across I/O and compare acquisition order for ABBA.
- Keep cleanup, extra stat/rename, and diagnostics out of the PUT commit critical
section when they need not be there.
+2 -2
View File
@@ -1,2 +1,2 @@
sha256-linux=14becc458b9eb24add510d94ee6cd04ba1fb19907d598effe1524e5242400d10
sha256-darwin=14becc458b9eb24add510d94ee6cd04ba1fb19907d598effe1524e5242400d10
sha256-linux=eb33067d90ee66c0a97e8f3658e46c152e47f897d2fbfdc6cb5ad93cfa55414b
sha256-darwin=eb33067d90ee66c0a97e8f3658e46c152e47f897d2fbfdc6cb5ad93cfa55414b
+1 -1
View File
@@ -229,7 +229,7 @@
{
"invariant": "list-objects-shared-corpus-orphan-cleanup-requires-authoritative-evidence",
"suite": "rustfs-ecstore",
"name": "store::list_objects::differential_tests::list_objects_shared_corpus_orphan_cleanup_requires_authoritative_evidence"
"name": "store::list_objects::differential_tests::list_objects_shared_corpus_orphan_listing_preserves_inputs"
},
{
"invariant": "list-objects-shared-corpus-consumer-drop-joins-real-disk-producer",
+1
View File
@@ -52,6 +52,7 @@ script-tests: ## Run shell script tests
./scripts/check_embedded_secrets.sh --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_btf_prototypes.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py
@@ -74,7 +74,6 @@ Notes:
- `.github/workflows/e2e-s3tests.yml` (weekly full s3-tests sweep)
- `.github/workflows/mint.yml` (weekly multi-SDK mint run)
- `.github/workflows/fuzz.yml` (nightly fuzz corpus jobs)
- `.github/workflows/performance-ab.yml` (nightly warp A/B gate)
- **Pending:** the ci-7 e2e nightly-full workflow does not exist yet
(backlog#1149 ci-7). When it lands, wire it up with the snippet above.
-1
View File
@@ -18,7 +18,6 @@
"never_ran_grace_until": "2026-09-08T00:00:00Z"
},
{ "workflow": ".github/workflows/nightly-gnu.yml", "max_age_hours": 36 },
{ "workflow": ".github/workflows/performance-ab.yml", "max_age_hours": 36 },
{
"workflow": ".github/workflows/runner-hygiene.yml",
"max_age_hours": 792,
+4 -4
View File
@@ -7,7 +7,7 @@
This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, which have custom labels and are not used here, and standard runners in private repositories.
Every other label in this repository is capacity RustFS pays for by the hour whether or not a job is running well:
Every other label in this repository is capacity RustFS pays for. ARC pods and lab machines accrue cost while the machine exists. A Ubicloud label is metered for the minutes a job holds its VM:
| Label | Paid by | What it is | Use it for |
|---|---|---|---|
@@ -15,7 +15,7 @@ Every other label in this repository is capacity RustFS pays for by the hour whe
| `windows-latest`, `macos-latest` | nobody | GitHub-hosted standard runners | Windows and macOS targets |
| `sm-standard-2` | RustFS | ARC pod, Ubuntu 24.04, minimal image: no `gh`, no C toolchain; Docker only on the `dind-` label | only a job that a hosted runner cannot run (see the placement rule) |
| `sm-standard-4` | RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds |
| `sm-standard-4-arm` | RustFS | ARC pod, aarch64; reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds |
| `ubicloud-standard-16-arm` | RustFS | Ubicloud VM, alias of `ubicloud-standard-16-arm-ubuntu-2404`: Ubuntu 24.04 arm64, 16 vCPU, 48 GB, 300 GB disk. Reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds (`linux-aarch64-musl`, `linux-aarch64-gnu`) |
| `dind-sm-standard-2` | RustFS | ARC pod with Docker-in-Docker | image builds that need `buildx` and a registry push |
| `smoke-testing`, `pf-testing` | RustFS | long-lived lab machines | the packaged functional chain and performance suites |
@@ -40,7 +40,7 @@ A hosted runner is also the only choice for a few things the pods cannot do: `mo
- Every `shared-key` has exactly one writer. `cache-warm.yml` owns `ci-dev`, `ci-feat-rio`, `ci-feat-proto` and `ci-uring`; every other `./.github/actions/setup` call states `cache-save-if` explicitly, and `scripts/security/check_cache_save_if.sh` fails the build when one does not.
- All readers of a key run on the same image as its writer. Keys that live on hosted runners say so in their name (`ci-uring` is warmed on `ubuntu-latest`; `ci-e2e-distributed-hosted`, `e2e-upgrade-server-hosted`, `e2e-upgrade-tests-hosted`).
- Moving a job between pools therefore means renaming its key. The new key is empty until a run from `main` (a scheduled run, or a manual dispatch of that workflow from `main`) saves it; until then that lane does a cold build.
- Moving a job between pools therefore means renaming its key. The new key is empty until a run from `main` (a scheduled run, or a manual dispatch of that workflow from `main`) saves it; until then that lane does a cold build. The `build.yml` aarch64 release legs append `-ubicloud-arm` for this reason. `cache-warm.yml` does not write those keys.
- `CARGO_BUILD_JOBS` and every other `CARGO*`, `CC*`, `CFLAGS*`, `CXX*`, `CMAKE*` and `RUST*` variable is hashed into the key. `ci.yml` sets `CARGO_BUILD_JOBS` at step level for that reason; promoting it to job level rotates the key for every lane that shares it.
## Where self-hosted minutes leak
@@ -62,7 +62,7 @@ gh api "repos/rustfs/rustfs/actions/runs/<run_id>/jobs?per_page=100" \
--jq '.jobs[] | "\(.conclusion)\t\(.labels | join(","))\t\(.started_at)\t\(.completed_at)\t\(.name)"'
```
Only rows whose label starts with `sm-`, `dind-`, `smoke-` or `pf-` cost anything. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` cost anything. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
## Checklist for a new or moved job
-5
View File
@@ -29,7 +29,6 @@ on:
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
- 'scripts/security/check_performance_ab_workflow.sh'
- 'scripts/security/check_preview_release_workflow.sh'
- 'scripts/security/check_tier_artifact_workflow.sh'
- 'scripts/security/check_workflow_pins.sh'
@@ -48,7 +47,6 @@ on:
- 'scripts/test_package_service_scripts.sh'
- 'scripts/test_package_versions.sh'
- 'scripts/test_docker_workflow.py'
- 'scripts/security/check_performance_ab_workflow.sh'
- 'scripts/security/check_preview_release_workflow.sh'
- 'scripts/security/check_tier_artifact_workflow.sh'
- 'scripts/security/check_workflow_pins.sh'
@@ -155,9 +153,6 @@ jobs:
- name: Check preview release workflow policy
run: ./scripts/security/check_preview_release_workflow.sh
- name: Check performance A/B workflow trust boundary
run: ./scripts/security/check_performance_ab_workflow.sh
- name: Check tier evidence workflow isolation
run: ./scripts/security/check_tier_artifact_workflow.sh
@@ -0,0 +1,75 @@
name: BTF prototype reference
on:
pull_request:
paths:
- "scripts/inspect_btf_prototypes.py"
- "scripts/test_inspect_btf_prototypes.py"
- "scripts/fixtures/btf_prototypes.c"
- ".github/workflows/btf-prototype-reference.yml"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: btf-prototype-reference-${{ github.ref }}
cancel-in-progress: true
jobs:
reference:
runs-on: ubuntu-latest
timeout-minutes: 10
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: "3.12"
- name: Check prototype and CLI contracts
run: python3 -B scripts/test_inspect_btf_prototypes.py
- name: Install only missing native reference tools
run: |
select_bpftool() {
local candidate
local candidates=()
if command -v bpftool >/dev/null; then
candidates+=("$(command -v bpftool)")
fi
for candidate in /usr/lib/linux-tools-*/bpftool /usr/lib/linux-tools/*/bpftool; do
if [[ -x "$candidate" ]]; then
candidates+=("$candidate")
fi
done
while IFS= read -r candidate; do
if [[ -x "$candidate" ]] && file -Lb "$candidate" | grep -q 'ELF' \
&& "$candidate" version >/dev/null 2>&1; then
printf '%s\n' "$candidate"
return 0
fi
done < <(printf '%s\n' "${candidates[@]}" | sort -Vr)
return 1
}
packages=()
command -v clang >/dev/null || packages+=(clang)
command -v llvm-objcopy >/dev/null || packages+=(llvm)
command -v gcc >/dev/null || packages+=(gcc)
select_bpftool >/dev/null || packages+=(linux-tools-generic)
if (( ${#packages[@]} )); then
sudo apt-get update
sudo apt-get install --no-install-recommends -y "${packages[@]}"
fi
# File parsing needs a working binary, not a host-kernel wrapper.
selected_bpftool="$(select_bpftool)"
"$selected_bpftool" version
printf 'BPFTOOL=%s\n' "$selected_bpftool" >> "$GITHUB_ENV"
- name: Compare declarations and layouts with independent references
run: |
clang -target bpf -O2 -g -c scripts/fixtures/btf_prototypes.c -o "$RUNNER_TEMP/prototypes.o"
llvm-objcopy --dump-section ".BTF=$RUNNER_TEMP/prototypes.btf" "$RUNNER_TEMP/prototypes.o"
gcc -std=c11 -Wall -Wextra -Werror -DRUSTFS_BTF_LAYOUT_REFERENCE \
scripts/fixtures/btf_prototypes.c -o "$RUNNER_TEMP/layout-reference"
"$RUNNER_TEMP/layout-reference" > "$RUNNER_TEMP/layout-reference.json"
python3 -B scripts/test_inspect_btf_prototypes.py --native-reference "$RUNNER_TEMP/prototypes.btf" \
--layout-reference "$RUNNER_TEMP/layout-reference.json"
+17 -7
View File
@@ -181,12 +181,18 @@ jobs:
selected="all"
fi
# linux-aarch64-* need a real aarch64 host (placement rule: hardware)
# so the packaged binary and console smoke execute the artifact.
# ubicloud-standard-16-arm is that host: Ubuntu 24.04 arm64, alias of
# ubicloud-standard-16-arm-ubuntu-2404 (16 vCPU, 48 GB). linux-x86_64-* stay on the
# ARC sm-standard-4 pool. GNU rows keep cross=true so cargo-zigbuild
# pins the glibc 2.31 floor on the same architecture; musl stays a
# native cargo build.
all='{"include":[
{"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
{"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
{"target_id":"linux-aarch64-musl","os":"ubicloud-standard-16-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
{"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
{"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
{"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-16-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
{"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""},
{"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":"","glibc":""}
]}'
@@ -244,7 +250,11 @@ jobs:
with:
rust-version: stable
target: ${{ matrix.target }}
cache-shared-key: build-${{ matrix.target }}${{ matrix.glibc && format('-glibc-{0}', matrix.glibc) || '' }}
# rust-cache keys include OS and architecture but not the runner image.
# The Ubicloud aarch64 legs therefore use their own suffix so they
# cannot restore a cache written on a different arm64 image.
# x86_64, macOS, and Windows keys stay unsuffixed.
cache-shared-key: build-${{ matrix.target }}${{ matrix.glibc && format('-glibc-{0}', matrix.glibc) || '' }}${{ matrix.os == 'ubicloud-standard-16-arm' && '-ubicloud-arm' || '' }}
# main only. A cache saved on refs/tags/X is scoped to that tag: no
# other tag, no main run and no PR can restore it, so every release
# cycle wrote up to 12 entries of 1-2GB (preview tag plus final tag,
@@ -764,9 +774,9 @@ jobs:
exit 0
fi
# The self-hosted Linux runners do not ship the aws CLI (GitHub-hosted
# images did). Install the official CLI bundle on demand so R2 uploads
# survive a fresh runner without relying on a distro package.
# ARC pods (the x86_64 legs) do not ship the aws CLI. Ubicloud arm64
# images follow the GitHub runner image and often do. Install the
# official CLI bundle when it is missing so R2 uploads work on either.
if ! command -v aws >/dev/null 2>&1; then
echo "aws CLI not found on runner; installing..."
if command -v apt-get >/dev/null 2>&1; then
+1 -1
View File
@@ -45,7 +45,7 @@ on:
# 07:00 UTC Sunday — staggered clear of the other Sunday crons: ci (00:00),
# build (01:00), e2e-s3tests (02:00), audit (03:00), nix-flake-update
# (05:00), mint (06:00), and the daily fuzz (02:00), minio-interop (03:17),
# e2e-replication-nightly (04:00) and performance-ab (06:00) lanes.
# and e2e-replication-nightly (04:00) lanes.
- cron: "43 7 * * 0"
concurrency:
@@ -38,8 +38,8 @@ name: e2e-nightly
on:
workflow_dispatch:
schedule:
# 04:00 UTC nightly — staggered clear of fuzz/e2e-s3tests (02:00),
# stale (01:30) and performance-ab (06:00).
# 04:00 UTC nightly — staggered clear of fuzz/e2e-s3tests (02:00)
# and stale (01:30).
- cron: "29 4 * * *"
# Only alert-on-failure needs more than read access; it declares its own
-384
View File
@@ -1,384 +0,0 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
# Warp A/B budget gate for the hotpath series (rustfs/backlog#935 HP-14, item 4).
#
# Two entry points, honestly scoped:
# * schedule (nightly, on main): post-merge detection — catches a regression
# within 24h of landing, not before merge.
# * workflow_dispatch: an explicitly selected trusted ref.
# The dispatch input can run the gate with --allow-regression so a deliberate
# correctness cost (e.g. the #4221 fsync durability fix) is recorded, not
# blocked (rustfs/backlog#935 correction 1).
name: Performance A/B
on:
schedule:
- cron: "31 6 * * *" # 06:31 UTC nightly, against main
workflow_dispatch:
inputs:
duration:
description: "warp duration per round"
required: false
default: "12s"
type: string
allow_regression:
description: "Pass the gate despite a FAIL (deliberate tradeoff)"
required: false
default: false
type: boolean
permissions:
actions: read
contents: read
env:
CARGO_TERM_COLOR: always
RUST_BACKTRACE: 1
jobs:
warp-ab:
if: github.repository == 'rustfs/rustfs'
name: Warp A/B budget gate
runs-on: sm-standard-2
# A normal nightly restores the last successful binary and builds only the
# candidate; daily access keeps that cache warm. A cache miss may build both
# and needs room for the A/B run plus artifact and cache publication.
timeout-minutes: 180
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
fetch-depth: 0 # baseline may be an earlier successful scheduled head
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: warp-ab-${{ hashFiles('**/Cargo.lock') }}
cache-save-if: ${{ github.ref == 'refs/heads/main' }}
- name: Install warp
run: |
set -euo pipefail
WARP_VERSION="1.0.0"
curl -fsSL "https://github.com/minio/warp/releases/download/v${WARP_VERSION}/warp_Linux_x86_64.tar.gz" \
| sudo tar -xz -C /usr/local/bin warp
warp --version
- name: Decide exemption
id: exempt
env:
INPUT_ALLOW_REGRESSION: ${{ github.event.inputs.allow_regression }}
run: |
allow="false"
if [[ "$INPUT_ALLOW_REGRESSION" == "true" ]]; then
allow="true"
fi
echo "allow_regression=$allow" >> "$GITHUB_OUTPUT"
# A failed regression run must keep comparing against the last known-good
# scheduled head. Otherwise the next nightly would absorb the regression
# into its baseline and turn green without a fix.
- name: Find last successful scheduled baseline
id: scheduled_baseline
if: github.event_name == 'schedule'
uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
with:
result-encoding: string
script: |
const { data } = await github.rest.actions.listWorkflowRuns({
owner: context.repo.owner,
repo: context.repo.repo,
workflow_id: "performance-ab.yml",
event: "schedule",
status: "success",
per_page: 1,
});
return data.workflow_runs[0]?.head_sha ?? "";
# Manual runs compare a selected ref with current main. Scheduled runs
# compare current main with the last successful scheduled head. With no
# history, the first run measures the candidate against itself and seeds
# that head only if the complete rig succeeds.
- name: Resolve baseline / candidate commits
id: commits
env:
SCHEDULED_BASELINE_SHA: ${{ steps.scheduled_baseline.outputs.result }}
run: |
set -euo pipefail
candidate_sha="$(git rev-parse HEAD)"
if [[ "${{ github.event_name }}" == "schedule" ]]; then
baseline_sha="${SCHEDULED_BASELINE_SHA:-$candidate_sha}"
else
baseline_sha="$(git rev-parse origin/main)"
fi
git cat-file -e "${baseline_sha}^{commit}"
if ! git merge-base --is-ancestor "$baseline_sha" "$candidate_sha"; then
echo "::error::baseline $baseline_sha is not an ancestor of candidate $candidate_sha; update the selected ref before comparing" >&2
exit 1
fi
echo "baseline_sha=$baseline_sha" >> "$GITHUB_OUTPUT"
echo "candidate_sha=$candidate_sha" >> "$GITHUB_OUTPUT"
echo "baseline commit: $baseline_sha"
echo "candidate commit: $candidate_sha"
# Exact-key restore of the candidate binary saved by its successful
# scheduled run. A miss leaves cache-hit unset and falls back to a source
# build of that known-good head.
- name: Restore cached baseline binary
id: baseline_cache
uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
with:
path: baseline-bin/rustfs
key: rustfs-baseline-${{ steps.commits.outputs.baseline_sha }}
# Self-heal: on a nightly/dispatch run where the candidate commit IS the
# baseline commit, a cache miss would make the rig build the same commit
# twice (~65min per side with the post-#4806 LTO profile — no job budget
# fits that). Build it once here, reuse it for both phases, and save it
# back to the cache so the next run hits.
- name: Build baseline on cache miss (same-commit self-heal)
id: selfheal
if: >-
steps.baseline_cache.outputs.cache-hit != 'true' &&
steps.commits.outputs.baseline_sha == steps.commits.outputs.candidate_sha
run: |
set -euo pipefail
cargo build --release --bin rustfs
mkdir -p baseline-bin
cp target/release/rustfs baseline-bin/rustfs
echo "built=true" >> "$GITHUB_OUTPUT"
- name: Build baseline on cache miss (different candidate)
id: baseline_build
if: >-
steps.baseline_cache.outputs.cache-hit != 'true' &&
steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha
run: |
set -euo pipefail
baseline_root="$RUNNER_TEMP/rustfs-baseline-${{ github.run_id }}"
baseline_target="$RUNNER_TEMP/rustfs-baseline-target-${{ github.run_id }}"
git worktree add --detach "$baseline_root" "${{ steps.commits.outputs.baseline_sha }}"
cargo build --release --manifest-path "$baseline_root/Cargo.toml" --bin rustfs --target-dir "$baseline_target"
mkdir -p baseline-bin
cp "$baseline_target/release/rustfs" baseline-bin/rustfs
git worktree remove --force "$baseline_root"
echo "built=true" >> "$GITHUB_OUTPUT"
- name: Build candidate binary
id: candidate_build
if: steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha
run: |
set -euo pipefail
cargo build --release --bin rustfs
mkdir -p candidate-bin
cp target/release/rustfs candidate-bin/rustfs
echo "built=true" >> "$GITHUB_OUTPUT"
- name: Save self-healed baseline to cache
if: steps.selfheal.outputs.built == 'true' || steps.baseline_build.outputs.built == 'true'
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
with:
path: baseline-bin/rustfs
key: rustfs-baseline-${{ steps.commits.outputs.baseline_sha }}
- name: Run warp A/B and gate
id: ab
env:
INPUT_DURATION: ${{ github.event.inputs.duration }}
run: |
set -euo pipefail
# The formal runner executes A1 baseline -> B1 candidate -> B2 candidate
# -> A2 baseline for each workload and drive-sync cell. It requires three
# rounds per leg to emit tail latency and error-rate evidence.
# --health-timeout 180 outlasts the server's own 120s startup-readiness
# budget, which the rig's previous 60s health poll undershot (the first
# two nightly failures). perf-6 recalibrates these once the noise study
# lands.
duration="${INPUT_DURATION:-12s}"
baseline_sha="${{ steps.commits.outputs.baseline_sha }}"
candidate_sha="${{ steps.commits.outputs.candidate_sha }}"
baseline_hit="${{ steps.baseline_cache.outputs.cache-hit }}"
selfheal_built="${{ steps.selfheal.outputs.built }}"
baseline_built="${{ steps.baseline_build.outputs.built }}"
candidate_built="${{ steps.candidate_build.outputs.built }}"
args=(--duration "$duration" --rounds 3 --cooldown 5 --health-timeout 180 --baseline-revision "$baseline_sha" --candidate-revision "$candidate_sha")
if [[ "$baseline_hit" == "true" || "$selfheal_built" == "true" || "$baseline_built" == "true" ]]; then
chmod +x baseline-bin/rustfs
base_bin="$PWD/baseline-bin/rustfs"
args+=(--baseline-bin "$base_bin")
if [[ "$baseline_hit" == "true" ]]; then
base_src="actions-cache (rustfs-baseline-$baseline_sha)"
elif [[ "$selfheal_built" == "true" ]]; then
base_src="source build (cache self-heal, saved as rustfs-baseline-$baseline_sha)"
else
base_src="isolated baseline source build (saved as rustfs-baseline-$baseline_sha)"
fi
if [[ "$candidate_sha" == "$baseline_sha" ]]; then
# No commits landed since the last successful baseline, so reuse
# the one binary for both phases and measure only rig drift.
args+=(--candidate-bin "$base_bin")
cand_src="same binary as baseline (same commit)"
elif [[ "$candidate_built" == "true" ]]; then
chmod +x candidate-bin/rustfs
args+=(--candidate-bin "$PWD/candidate-bin/rustfs")
cand_src="source build of the checked-out ref"
else
echo "::error::candidate binary was not built" >&2
exit 2
fi
else
echo "::error::baseline binary was not restored or built" >&2
exit 2
fi
echo "baseline binary: $base_src"
echo "candidate binary: $cand_src"
if [[ "${{ steps.exempt.outputs.allow_regression }}" == "true" ]]; then
args+=(--allow-regression --exemption-reason "workflow dispatch override")
fi
# Do not let a gate FAIL abort the job here; capture status and surface
# it after the step summary is written.
set +e
bash scripts/run_hotpath_warp_abba.sh "${args[@]}"
echo "status=$?" >> "$GITHUB_OUTPUT"
set -e
# Locate the newest run dir + gate.md for the summary/comment/artifact
# steps. On a startup failure there is no gate.md, but the run dir still
# holds server-logs/ for diagnosis.
# Run dirs are UTC-timestamp names (no special chars); ls is safe here.
# shellcheck disable=SC2012
run_dir="$(ls -td target/hotpath-abba/*/ 2>/dev/null | head -n1 || true)"
echo "run_dir=${run_dir%/}" >> "$GITHUB_OUTPUT"
# shellcheck disable=SC2012
gate_md="$(ls -t target/hotpath-abba/*/candidate_gate.md 2>/dev/null | head -n1 || true)"
echo "gate_md=$gate_md" >> "$GITHUB_OUTPUT"
- name: Upload A/B results
if: always()
uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
with:
name: hotpath-warp-ab-${{ github.run_number }}
# Includes per-cell median_summary.csv / baseline_compare.csv, both gates,
# and server-logs/ (rustfs.log + startup env per phase) so a failed run
# is diagnosable. Short retention: this is churny nightly debug data.
path: target/hotpath-abba/
if-no-files-found: warn
retention-days: 14
- name: Write gate summary
if: always()
run: |
set -euo pipefail
status="${{ steps.ab.outputs.status }}"
gate_md="${{ steps.ab.outputs.gate_md }}"
run_dir="${{ steps.ab.outputs.run_dir }}"
{
echo "## Hotpath warp A/B — run ${{ github.run_number }}"
echo
if [[ "$status" == "0" ]]; then
echo "Rig/gate exit: \`0\` (pass or warn)."
else
echo "Rig/gate exit: \`${status:-unknown}\` — **FAILED**."
fi
echo
if [[ -n "$gate_md" && -f "$gate_md" ]]; then
cat "$gate_md"
else
echo "No \`gate.md\` produced — the rig failed **before** the gate"
echo "(most likely server startup / health). Failing phase(s) below;"
echo "full logs in the \`hotpath-warp-ab-${{ github.run_number }}\` artifact."
if [[ -n "$run_dir" && -d "$run_dir/server-logs" ]]; then
for f in "$run_dir"/server-logs/*.log; do
[[ -f "$f" ]] || continue
echo
echo "<details><summary>$(basename "$f")</summary>"
echo
echo '```'
tail -n 30 "$f"
echo '```'
echo "</details>"
done
fi
fi
} >> "$GITHUB_STEP_SUMMARY"
- name: Stage successful candidate baseline
if: >-
steps.ab.outputs.status == '0' &&
steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha
run: |
set -euo pipefail
cp candidate-bin/rustfs baseline-bin/rustfs
- name: Cache successful candidate baseline
if: >-
steps.ab.outputs.status == '0' &&
steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha
uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
with:
path: baseline-bin/rustfs
key: rustfs-baseline-${{ steps.commits.outputs.candidate_sha }}
# Scheduled failure alerting is handled by the alert-on-failure job below
# (perf-2 consuming ci-8's schedule-failure-issue composite action).
- name: Enforce gate
if: always()
run: |
status="${{ steps.ab.outputs.status }}"
if [[ -z "$status" ]]; then
echo "::error::warp A/B setup failed before the rig ran. Check the first failed workflow step." >&2
exit 1
fi
if [[ "$status" != "0" ]]; then
echo "::error::warp A/B budget gate failed (exit $status). See the step summary / gate.md artifact." >&2
exit "$status"
fi
echo "warp A/B budget gate passed."
alert-on-failure:
name: Alert on scheduled failure
needs: [warp-ab]
# `always()` is required: without it this job is skipped when a needed
# job fails. Alerts only for scheduled (nightly) runs (backlog#1149
# ci-8); manual dispatch failures are already watched by a human.
# `cancelled` is included alongside `failure` on purpose: a job that hits
# timeout-minutes ends as `cancelled`, and the 2026-07-11..07-14 nightly
# timeouts went silent precisely because the guard was failure-only. The
# composite action already reports cancelled/timed-out jobs in the issue
# body.
if: >-
github.repository == 'rustfs/rustfs' && (
always() && github.event_name == 'schedule' &&
(contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
)
runs-on: ubuntu-latest
timeout-minutes: 10
permissions:
contents: read
issues: write
steps:
- uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Open or update failure-tracking issue
uses: ./.github/actions/schedule-failure-issue
with:
github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}
@@ -16,8 +16,8 @@
#
# Forces a job failure and then runs .github/actions/schedule-failure-issue
# through the exact `needs` + `always() && contains(needs.*.result,
# 'failure')` wiring used by the real consumers (e2e-s3tests, mint, fuzz,
# performance-ab). Dispatch it twice to verify both the issue-creation and
# 'failure')` wiring used by the real consumers (e2e-s3tests, mint, fuzz).
# Dispatch it twice to verify both the issue-creation and
# the dedupe-comment paths, then close the resulting
# "[scheduled-failure] Schedule Failure Alert Drill" issue in rustfs/backlog
# (the default target repository). Requires the BACKLOG_ISSUE_TOKEN secret
@@ -28,7 +28,6 @@ on:
- "mint"
- "minio-interop"
- "Nightly GNU Build"
- "Performance A/B"
- "Runner Hygiene"
types: [completed]
+43 -202
View File
@@ -1,224 +1,65 @@
# Copyright 2024 RustFS Team
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
name: Windows Filesystem Tests
name: Scanner Linux Hosted Preflight
on:
push:
branches: [ main ]
paths:
- "crates/ecstore/src/disk/**"
- "crates/ecstore/src/layout/**"
- "crates/ecstore/src/store/init_format.rs"
- "crates/ecstore/src/set_disk/ops/locking.rs"
- "crates/ecstore/src/object_api/write_commit_context.rs"
- "crates/ecstore/src/bucket/utils.rs"
- "crates/ecstore/Cargo.toml"
- "crates/utils/src/dunce.rs"
- "crates/utils/src/os/**"
- "crates/utils/Cargo.toml"
- "crates/utils/src/path.rs"
- "rustfs/build.rs"
- "rustfs/Cargo.toml"
- "rustfs/src/main.rs"
- "rustfs/src/server/runtime.rs"
- "rustfs/src/startup_*.rs"
- "Cargo.toml"
- "Cargo.lock"
- ".github/actions/setup/**"
- ".github/workflows/windows-filesystem.yml"
pull_request:
branches: [ main ]
paths:
- "crates/ecstore/src/disk/**"
- "crates/ecstore/src/layout/**"
- "crates/ecstore/src/store/init_format.rs"
- "crates/ecstore/src/set_disk/ops/locking.rs"
- "crates/ecstore/src/object_api/write_commit_context.rs"
- "crates/ecstore/src/bucket/utils.rs"
- "crates/ecstore/Cargo.toml"
- "crates/utils/src/dunce.rs"
- "crates/utils/src/os/**"
- "crates/utils/Cargo.toml"
- "crates/utils/src/path.rs"
- "rustfs/build.rs"
- "rustfs/Cargo.toml"
- "rustfs/src/main.rs"
- "rustfs/src/server/runtime.rs"
- "rustfs/src/startup_*.rs"
- "Cargo.toml"
- "Cargo.lock"
- ".github/actions/setup/**"
- ".github/workflows/windows-filesystem.yml"
workflow_dispatch:
permissions:
contents: read
concurrency:
group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
cancel-in-progress: true
group: scanner-linux-hosted-${{ github.ref }}
cancel-in-progress: false
env:
CARGO_TERM_COLOR: always
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
CARGO_TERM_COLOR: never
RUST_BACKTRACE: 1
FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: 'true'
NO_PROXY: 127.0.0.1,localhost
jobs:
rename-safety:
name: Rename Safety
runs-on: windows-latest
timeout-minutes: 60
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
with:
persist-credentials: false
- name: Setup Rust environment
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: build-x86_64-pc-windows-msvc
cache-save-if: 'false'
install-build-packaging-tools: 'false'
install-test-tools: 'false'
- name: Check production Windows dependencies
shell: pwsh
run: cargo check -p rustfs-ecstore --lib
- name: Test guarded rename publication
shell: pwsh
run: cargo test -p rustfs-ecstore --lib rename_all_ -- --nocapture
- name: Test multipart parent durability barriers
shell: pwsh
run: cargo test -p rustfs-ecstore --lib multipart_owned_record_parent_barriers_ -- --nocapture
- name: Test Windows path conversion
shell: pwsh
run: "cargo test -p rustfs-utils --no-default-features --features path --lib dunce:: -- --nocapture"
- name: Test Windows volume and disk-space probes
shell: pwsh
run: |
# Map a drive letter to the temp drive's administrative share so the
# remote-volume probe also covers mapped network drives.
$drive = (Split-Path -Qualifier ([System.IO.Path]::GetTempPath())).TrimEnd(':')
net use R: "\\localhost\$drive`$" /persistent:no
if ($LASTEXITCODE -eq 0) {
$env:RUSTFS_TEST_WINDOWS_MAPPED_DRIVE = 'R:'
} else {
Write-Output "::warning::could not map R: to \\localhost\$drive`$; mapped-drive test will skip"
}
cargo test -p rustfs-utils --features os --lib os:: -- --nocapture
$status = $LASTEXITCODE
if ($env:RUSTFS_TEST_WINDOWS_MAPPED_DRIVE) { net use R: /delete /y | Out-Null }
exit $status
- name: Test Windows handle guards
shell: pwsh
run: cargo test -p rustfs-ecstore --lib windows_ -- --nocapture
- name: Test endpoint path parsing
shell: pwsh
run: "cargo test -p rustfs-ecstore --lib layout::endpoint:: -- --nocapture"
- name: Test startup temporary-directory cleanup
shell: pwsh
run: cargo test -p rustfs-ecstore --lib cleanup_tmp_on_startup_ -- --nocapture
- name: Test object key Windows admission
shell: pwsh
run: "cargo test -p rustfs-ecstore --lib bucket::utils:: -- --nocapture"
- name: Test Windows file-name rules
shell: pwsh
run: "cargo test -p rustfs-utils --no-default-features --features path --lib dunce:: -- --nocapture"
- name: Test fresh format publication
shell: pwsh
run: cargo test -p rustfs-ecstore --lib fresh_format_load_initializes_all_disks -- --nocapture
# The native Windows main thread only gets the 1 MiB PE stack reserve, while
# Linux and macOS hand out 8 MiB. Startup runs the whole root future on the
# thread that calls `block_on`, so a debug server once overflowed before it
# listened on Windows only. Start a real server to keep that covered.
startup-smoke:
name: Startup Smoke
runs-on: windows-latest
scoped-tests:
runs-on: ubuntu-latest
timeout-minutes: 90
steps:
- name: Checkout repository
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
- name: Checkout candidate
uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a
with:
persist-credentials: false
- name: Setup Rust environment
fetch-depth: 2
- name: Setup Rust
uses: ./.github/actions/setup
with:
rust-version: stable
cache-shared-key: build-x86_64-pc-windows-msvc
cache-shared-key: ci-e2e-distributed-hosted
cache-save-if: 'false'
install-build-packaging-tools: 'false'
install-test-tools: 'false'
- name: Build debug server binary
shell: pwsh
run: cargo build --locked -p rustfs --bin rustfs
- name: Print version
shell: pwsh
- name: Check candidate and resources
shell: bash
run: |
& .\target\debug\rustfs.exe --version
if ($LASTEXITCODE -ne 0) { throw "rustfs --version exited with $LASTEXITCODE" }
- name: Start server and wait for readiness
shell: pwsh
set -euo pipefail
test "$(git rev-parse HEAD^)" = 03eef1ea79a782e15480a00ba87ed2e256a84f9b
test "$(git diff --name-only HEAD^ HEAD)" = .github/workflows/windows-filesystem.yml
df -h .
free -h
du -sh target "$HOME/.cargo" 2>/dev/null || true
- name: Run only the named diagnostic unit tests
shell: bash
env:
RUSTFS_ACCESS_KEY: rustfsadmin
RUSTFS_SECRET_KEY: rustfsadmin
CARGO_BUILD_JOBS: '1'
CARGO_INCREMENTAL: '0'
run: |
$volume = Join-Path $env:RUNNER_TEMP 'rustfs-startup-smoke'
$stdout = Join-Path $env:RUNNER_TEMP 'rustfs-startup-smoke.out.log'
$stderr = Join-Path $env:RUNNER_TEMP 'rustfs-startup-smoke.err.log'
New-Item -ItemType Directory -Force -Path $volume | Out-Null
$server = Start-Process -FilePath .\target\debug\rustfs.exe `
-ArgumentList @('--address', '127.0.0.1:9000', "`"$volume`"") `
-RedirectStandardOutput $stdout -RedirectStandardError $stderr `
-NoNewWindow -PassThru
# Cache the process handle now; otherwise ExitCode reads back empty.
$null = $server.Handle
$ready = $false
try {
for ($i = 0; $i -lt 180; $i++) {
if ($server.HasExited) { break }
try {
$response = Invoke-WebRequest -Uri 'http://127.0.0.1:9000/health/ready' -UseBasicParsing -NoProxy -TimeoutSec 2
if ($response.StatusCode -eq 200) { $ready = $true; break }
} catch {}
Start-Sleep -Seconds 1
}
} finally {
if (-not $server.HasExited) { Stop-Process -Id $server.Id -Force }
$server.WaitForExit()
Write-Host '--- server stdout (tail) ---'
Get-Content $stdout -Tail 100 -ErrorAction SilentlyContinue
Write-Host '--- server stderr (tail) ---'
Get-Content $stderr -Tail 100 -ErrorAction SilentlyContinue
}
if (-not $ready) {
throw ("rustfs did not report ready on /health/ready (exit code: {0})" -f $server.ExitCode)
set -euo pipefail
run_one() {
local package="$1"
local test_name="$2"
cargo test --locked -p "$package" --lib -- --exact "$test_name" --nocapture 2>&1 | tee scanner-proof-test.log
grep -F 'test result: ok. 1 passed; 0 failed;' scanner-proof-test.log
}
run_one rustfs-scanner scanner::tests::cycle_budget_publication::publication_lease_observations_keep_cycle_epoch_purpose_and_outcome
run_one rustfs-scanner scanner::tests::cycle_budget_publication::publication_lease_retry_logs_identify_interleaved_epochs
run_one rustfs-scanner scanner::tests::cycle_budget_publication::publication_lease_events_include_context_in_otlp_without_span_export
run_one rustfs-scanner scanner::tests::cycle_budget_publication::dropped_lease_wait_does_not_report_a_finished_outcome
run_one rustfs-scanner scanner::tests::cycle_budget_publication::publication_lease_retries_local_channel_deadline_and_stops_on_cancel
run_one rustfs-ecstore cluster::rpc::peer_rest_client::tests::scanner_publication_lease_validation_errors_are_typed_without_changing_display
run_one rustfs-ecstore cluster::rpc::peer_rest_client::tests::scanner_publication_lease_error_class_keeps_remote_reports_separate_from_local_errors
run_one rustfs-ecstore services::notification_sys::tests::scanner_lease_acquisition_error_identifies_peer_and_preserves_retry_suffix
run_one rustfs-ecstore services::notification_sys::tests::scanner_lease_failure_info_uses_the_actual_target_and_rejects_unbounded_peers
run_one rustfs-ecstore services::notification_sys::tests::scanner_lease_failure_info_accepts_real_grid_host_urls
run_one rustfs-ecstore services::notification_sys::tests::scanner_lease_failure_info_preserves_typed_connection_failures_without_logging_details
run_one rustfs-ecstore services::notification_sys::tests::scanner_lease_channel_deadline_preserves_public_error_and_diagnostic
Generated
+3
View File
@@ -10642,6 +10642,9 @@ dependencies = [
"hotpath",
"http 1.5.0",
"metrics",
"opentelemetry",
"opentelemetry-appender-tracing",
"opentelemetry_sdk",
"rand 0.10.3",
"rmp-serde",
"rustfs-common",
+14
View File
@@ -261,6 +261,20 @@ pub async fn cached_connection(addr: &str) -> Option<Channel> {
GLOBAL_CONN_MAP.read().await.get(addr).cloned()
}
/// Holds the connection cache write lock for an isolated test process.
#[cfg(any(test, feature = "test-util"))]
pub struct ConnectionCacheReadBlockerForTest {
_guard: tokio::sync::RwLockWriteGuard<'static, HashMap<String, Channel>>,
}
/// Block cache reads until the returned test guard is dropped.
#[cfg(any(test, feature = "test-util"))]
pub async fn block_connection_cache_reads_for_test() -> ConnectionCacheReadBlockerForTest {
ConnectionCacheReadBlockerForTest {
_guard: GLOBAL_CONN_MAP.write().await,
}
}
/// Cache a gRPC connection for the given address.
pub async fn cache_connection(addr: String, channel: Channel) {
GLOBAL_CONN_MAP.write().await.insert(addr, channel);
+132 -3
View File
@@ -13,18 +13,20 @@
// limitations under the License.
use super::harness::{
DistCluster, DistLayout, TestResult, assert_object_bytes, payload_for, put_object, retrying_get_equals, unique_bucket,
wait_for_ready, wait_until,
DistCluster, DistLayout, TestResult, assert_object_bytes, cluster_admin_ok, get_object_bytes, payload_for, put_object,
retrying_get_equals, retrying_put, s3_probe_client, s3_probe_error_code, unique_bucket, wait_for_ready, wait_until,
};
use crate::chaos::{census_object_version_on_disk, signed_admin_post};
use crate::common::{build_test_s3_config, init_logging};
use crate::fault_proxy::FaultMode;
use aws_sdk_s3::Client;
use aws_sdk_s3::error::SdkError;
use aws_sdk_s3::operation::{get_object::GetObjectError, put_object::PutObjectError};
use std::path::PathBuf;
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::{Barrier, mpsc};
use tokio::time::timeout;
use tokio::time::{Instant, timeout};
#[tokio::test]
async fn kill_and_restart_node_preserves_objects() -> TestResult {
@@ -44,6 +46,133 @@ async fn kill_and_restart_node_preserves_objects() -> TestResult {
Ok(())
}
#[tokio::test]
async fn ec4_partial_node_return_restores_read_write_quorum() -> TestResult {
init_logging();
let mut dist = DistCluster::start_with_env(
DistLayout::FourByFour,
&[
("RUSTFS_ERASURE_SET_DRIVE_COUNT", "16"),
("RUSTFS_STORAGE_CLASS_STANDARD", "EC:4"),
],
)
.await?;
let storage = cluster_admin_ok(&dist.cluster, http::Method::GET, "/rustfs/admin/v3/storageinfo", None).await?;
let storage: serde_json::Value = serde_json::from_str(&storage)?;
let storage: rustfs_madmin::StorageInfo = serde_json::from_value(storage["info"].clone())?;
assert_eq!(storage.disks.len(), 16);
assert_eq!(storage.backend.total_sets, vec![1]);
assert_eq!(storage.backend.drives_per_set, vec![16]);
assert_eq!(storage.backend.standard_sc_parity, Some(4));
let bucket = unique_bucket("partial-ec4");
dist.create_bucket(&bucket).await?;
let client = s3_probe_client(&dist.client(0)?);
let baseline = payload_for("partial-recovery/baseline", 8 * 1024 * 1024);
put_object(&client, &bucket, "baseline.bin", baseline.clone()).await?;
wait_until(
Duration::from_secs(30),
|| async {
for node in &dist.cluster.nodes {
for directory in &node.data_dirs {
let census = census_object_version_on_disk(std::path::Path::new(directory), &bucket, "baseline.bin", None)?;
if !census.is_complete() {
return Ok(false);
}
assert_eq!(census.data_blocks, Some(12), "the object must require twelve data shards");
assert_eq!(census.parity_blocks, Some(4), "the object must have four parity shards");
assert!(!census.expected_part_numbers.is_empty(), "the baseline must use physical shards");
}
}
Ok(true)
},
"the baseline is complete on all sixteen EC:4 drives",
)
.await?;
dist.cluster.stop_node(2)?;
dist.cluster.stop_node(3)?;
let read_error = timeout(Duration::from_secs(60), get_object_bytes(&client, &bucket, "baseline.bin"))
.await?
.expect_err("eight unavailable drives must prevent the baseline GET");
assert!(
matches!(
s3_probe_error_code(read_error.as_ref()),
Some("SlowDown" | "SlowDownRead" | "ServiceUnavailable")
),
"quorum loss must produce an availability error: {read_error:?}"
);
assert_eq!(
read_error
.downcast_ref::<SdkError<GetObjectError>>()
.and_then(SdkError::raw_response)
.map(|response| response.status().as_u16()),
Some(503),
"read quorum loss must remain a service availability failure"
);
let write_error = timeout(Duration::from_secs(60), put_object(&client, &bucket, "unavailable.bin", baseline.clone()))
.await?
.expect_err("eight unavailable drives must prevent a successful PUT");
assert!(
matches!(s3_probe_error_code(write_error.as_ref()), Some("SlowDown" | "ServiceUnavailable")),
"quorum loss must produce an availability error: {write_error:?}"
);
assert_eq!(
write_error
.downcast_ref::<SdkError<PutObjectError>>()
.and_then(SdkError::raw_response)
.map(|response| response.status().as_u16()),
Some(503),
"write quorum loss must remain a service availability failure"
);
// Recover only four drives. The remaining node must stay stopped while
// every live coordinator serves old data and acknowledges new writes.
dist.cluster.start_node_process(2).await?;
let recovery_deadline = Instant::now() + Duration::from_secs(90);
let mut recovered_objects = Vec::new();
for node in 0..3 {
assert!(dist.cluster.nodes[3].process.is_none(), "the fourth node must remain unavailable");
let client = s3_probe_client(&dist.client(node)?);
retrying_get_equals(
&client,
&bucket,
"baseline.bin",
&baseline,
recovery_deadline.saturating_duration_since(Instant::now()),
)
.await?;
let key = format!("partial-{node}.bin");
let body = payload_for(&key, 1024 * 1024);
retrying_put(
&client,
&bucket,
&key,
body.clone(),
recovery_deadline.saturating_duration_since(Instant::now()),
)
.await?;
recovered_objects.push((key, body));
}
for node in 0..3 {
assert!(dist.cluster.nodes[3].process.is_none(), "reads must complete before all drives return");
let client = s3_probe_client(&dist.client(node)?);
assert_object_bytes(&client, &bucket, "baseline.bin", &baseline).await?;
for (key, body) in &recovered_objects {
assert_object_bytes(&client, &bucket, key, body).await?;
}
}
dist.cluster.start_node(3).await?;
wait_for_ready(&dist.cluster).await?;
let client = s3_probe_client(&dist.client(3)?);
assert_object_bytes(&client, &bucket, "baseline.bin", &baseline).await?;
for (key, body) in &recovered_objects {
assert_object_bytes(&client, &bucket, key, body).await?;
}
Ok(())
}
#[tokio::test]
async fn full_cluster_restart_preserves_objects() -> TestResult {
init_logging();
+66 -5
View File
@@ -1181,11 +1181,16 @@ pub(crate) async fn retrying_get_equals(
)
.into()),
Err(error) => {
if matches!(s3_probe_error_code(error.as_ref()), Some("SlowDown" | "ServiceUnavailable")) {
Ok(false)
} else {
Err(error)
}
let retryable = match s3_probe_error_code(error.as_ref()) {
Some("SlowDown" | "ServiceUnavailable") => true,
// GET preserves this compatibility code for a lost data quorum.
Some("SlowDownRead") => error
.downcast_ref::<SdkError<GetObjectError>>()
.and_then(SdkError::raw_response)
.is_some_and(|response| response.status().as_u16() == 503),
_ => false,
};
if retryable { Ok(false) } else { Err(error) }
}
}
},
@@ -1324,6 +1329,62 @@ mod retry_tests {
Ok(())
}
#[tokio::test(start_paused = true)]
async fn retrying_get_allows_read_quorum_recovery() -> TestResult {
use aws_sdk_s3::config::Region;
use aws_smithy_http_client::test_util::{ReplayEvent, StaticReplayClient};
use aws_smithy_types::body::SdkBody;
let endpoint = "http://read-quorum.example.invalid";
let event = |status: u16, body: &'static str| {
ReplayEvent::new(
http::Request::builder()
.method(Method::GET)
.uri(format!("{endpoint}/retry-probe/key?x-id=GetObject"))
.body(SdkBody::empty())
.expect("build replay request"),
http::Response::builder()
.status(status)
.header("content-type", "application/xml")
.header("content-length", body.len())
.body(SdkBody::from(body))
.expect("build replay response"),
)
};
for status in [503, 403, 500] {
let transport = StaticReplayClient::new(vec![
event(status, "<Error><Code>SlowDownRead</Code></Error>"),
event(200, "expected bytes"),
]);
let client = Client::from_conf(
aws_sdk_s3::Config::builder()
.behavior_version_latest()
.credentials_provider(Credentials::new(FAKE_ACCESS_KEY, FAKE_SECRET_KEY, None, None, "read-quorum-test"))
.region(Region::new("us-east-1"))
.endpoint_url(endpoint)
.force_path_style(true)
.retry_config(RetryConfig::disabled())
.http_client(transport.clone())
.build(),
);
let result = retrying_get_equals(&client, "retry-probe", "key", b"expected bytes", Duration::from_secs(5)).await;
if status == 503 {
result?;
assert_eq!(
transport.actual_requests().count(),
2,
"the probe must retry the read-quorum error exactly once"
);
transport.assert_requests_match(&[]);
} else {
let error = result.expect_err("the compatibility code must not hide a non-503 failure");
assert_eq!(s3_probe_error_code(error.as_ref()), Some("SlowDownRead"));
assert_eq!(transport.actual_requests().count(), 1, "non-availability errors must fail immediately");
}
}
Ok(())
}
#[tokio::test]
async fn object_polls_distinguish_committed_and_replicated_keys() -> TestResult {
let target = FakeS3Target::start().await?;
+1
View File
@@ -40,5 +40,6 @@ mod scanner_heal_topology_fixture;
mod scanner_heal_topology_test;
mod scanner_heal_topology_tier_test;
mod site_replication_test;
mod table_catalog_test;
mod upgrade_test;
mod versioning_test;
@@ -1638,6 +1638,191 @@ async fn late_complete_packets_cannot_publish_after_abort_and_bucket_recreation(
"refused late packets must not recreate staging"
);
}
late_part_packets_preserve_retired_and_fresh_uploads().await?;
Ok(())
}
async fn late_part_packets_preserve_retired_and_fresh_uploads() -> TestResult {
for (method, settles) in [("PreparePartTransaction", false), ("SettlePartTransaction", true)] {
for recreate_bucket in [false, true] {
let (cluster, proxies, clients, bucket) = fixture(1, "EC:2").await?;
let key = "late-part-transaction";
let previous = payload_for("committed-before-late-part", 4096);
clients[0]
.put_object()
.bucket(&bucket)
.key(key)
.body(ByteStream::from(previous.clone()))
.send()
.await?;
let upload = clients[0].create_multipart_upload().bucket(&bucket).key(key).send().await?;
let upload_id = upload.upload_id().ok_or("missing original upload ID")?;
clients[0]
.upload_part()
.bucket(&bucket)
.key(key)
.upload_id(upload_id)
.part_number(1)
.body(ByteStream::from(vec![0x41; 9 * 1024 * 1024]))
.send()
.await?;
let path = upload_path_scope(&bucket, key, upload_id)?;
let hash = sha256_hex(format!("{bucket}/{key}").as_bytes());
let roots = cluster
.nodes
.iter()
.flat_map(|node| &node.data_dirs)
.map(|root| Path::new(root).join(RUSTFS_META_MULTIPART_BUCKET).join(&path))
.collect::<Vec<_>>();
assert_eq!(roots.len(), 4);
let original = multipart_sources(&cluster, Some(&hash))?;
assert_eq!(original.len(), 4);
assert!(
original.values().all(|source| source
.files
.keys()
.any(|path| path.file_name().and_then(|name| name.to_str()) == Some("part.1"))),
"all selected peers originally own the acknowledged part"
);
let scope = upload_scope(upload_id)?;
// Retain only the original attempt; recovery of the same upload
// must be able to issue fresh settlement requests through the gate.
let pauses = proxies[1..3]
.iter()
.map(|proxy| proxy.pause_late(&[method], 1, &scope))
.collect::<Result<Vec<_>, _>>()?;
let replacement = clients[0]
.upload_part()
.bucket(&bucket)
.key(key)
.upload_id(upload_id)
.part_number(1)
.body(ByteStream::from(vec![0x62; 8 * 1024 * 1024]));
let caller = tokio::spawn(async move { replacement.send().await });
wait_for_faults(&pauses, &[method]).await?;
let failure = tokio::time::timeout(DEADLINE, caller)
.await??
.expect_err("two retained peers cannot acknowledge part replacement");
assert_unavailable(&failure, "storage_write_quorum", None);
assert!(pauses.iter().all(|pause| pause.hits() == 1 && pause.responses() == 0));
let expected_actions = if settles { (1, 0) } else { (0, 0) };
assert!(
pauses.iter().all(|pause| pause.retained_settle_actions() == expected_actions),
"settlement capture must be a real Commit, not Rollback"
);
abort_after_fault(&clients[0], &bucket, key, upload_id, false).await?;
assert!(
roots.iter().all(|root| !root.exists()),
"ordinary Abort physically retires all original roots"
);
assert!(multipart_sources(&cluster, Some(&hash))?.is_empty());
for client in &clients {
assert_object_bytes_after_fault(client, &bucket, key, &previous).await?;
}
let protected = if recreate_bucket {
clients[0].delete_object().bucket(&bucket).key(key).send().await?;
clients[0].delete_bucket().bucket(&bucket).send().await?;
clients[1].create_bucket().bucket(&bucket).send().await?;
let fresh = payload_for("recreated-before-late-part", 4096);
clients[1]
.put_object()
.bucket(&bucket)
.key(key)
.body(ByteStream::from(fresh.clone()))
.send()
.await?;
fresh
} else {
previous
};
let fresh = clients[2].create_multipart_upload().bucket(&bucket).key(key).send().await?;
let fresh_id = fresh.upload_id().ok_or("missing fresh upload ID")?;
let fresh_body = payload_for("fresh-active-part", 4096);
let fresh_part = clients[2]
.upload_part()
.bucket(&bucket)
.key(key)
.upload_id(fresh_id)
.part_number(1)
.body(ByteStream::from(fresh_body.clone()))
.send()
.await?;
let before = multipart_sources(&cluster, Some(&hash))?;
assert_eq!(before.len(), 4, "the new upload exists on every original participant");
for pause in &pauses {
pause.release();
}
wait_until(
DEADLINE,
|| async {
Ok(pauses.iter().all(|pause| {
pause.responses() == pause.hits()
&& if settles {
pause.successful_mutation_replies() == pause.hits() && pause.refused_mutations() == 0
} else {
pause.refused_mutations() == pause.hits() && pause.successful_mutation_replies() == 0
}
}))
},
"every original part packet returns its typed authenticated terminal outcome",
)
.await?;
assert!(roots.iter().all(|root| !root.exists()), "late part packets cannot recreate retired roots");
assert!(
multipart_sources(&cluster, Some(&hash))? == before,
"late packets cannot change a fresh upload's shard or control bytes"
);
for client in &clients {
assert_object_bytes_after_fault(client, &bucket, key, &protected).await?;
let listed = client
.list_parts()
.bucket(&bucket)
.key(key)
.upload_id(fresh_id)
.send()
.await?;
assert_eq!(listed.parts().len(), 1);
assert_eq!(listed.parts()[0].e_tag(), fresh_part.e_tag());
let missing = client
.list_parts()
.bucket(&bucket)
.key(key)
.upload_id(upload_id)
.send()
.await
.expect_err("old upload remains retired");
assert_eq!(missing.raw_response().expect("old upload service response").status().as_u16(), 404);
assert_eq!(missing.as_service_error().and_then(|error| error.meta().code()), Some("NoSuchUpload"));
}
clients[2]
.complete_multipart_upload()
.bucket(&bucket)
.key(key)
.upload_id(fresh_id)
.multipart_upload(
CompletedMultipartUpload::builder()
.parts(
CompletedPart::builder()
.part_number(1)
.e_tag(fresh_part.e_tag().ok_or("missing fresh part ETag")?)
.build(),
)
.build(),
)
.send()
.await?;
for client in &clients {
assert_object_bytes_after_fault(client, &bucket, key, &fresh_body).await?;
}
assert!(multipart_sources(&cluster, Some(&hash))?.is_empty());
assert_no_uploads_after_fault(&clients, &bucket).await?;
eprintln!(
"MPU late part: method={method}, recreate_bucket={recreate_bucket}, retired_roots=4, delivered=2, refused={}, successful_replies={}, fresh_sources_unchanged=4, verified_readers=4",
pauses.iter().map(RpcPause::refused_mutations).sum::<usize>(),
pauses.iter().map(RpcPause::successful_mutation_replies).sum::<usize>()
);
}
}
Ok(())
}
@@ -0,0 +1,140 @@
// Copyright 2026 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use super::harness::{DistCluster, DistLayout, TestResult, get_object_bytes, put_object, unique_bucket};
use crate::common::{init_logging, signed_request};
use http::{Method, StatusCode};
use serde_json::{Value, json};
async fn catalog_request(
dist: &DistCluster,
node: usize,
method: Method,
path: &str,
body: Option<Value>,
expected: StatusCode,
) -> TestResult<Value> {
let response = signed_request(
method.clone(),
&format!("{}{path}", dist.cluster.nodes[node].url),
&dist.cluster.access_key,
&dist.cluster.secret_key,
body.map(|value| serde_json::to_vec(&value)).transpose()?,
Some("application/json"),
)
.await?;
let status = response.status();
let text = response.text().await?;
assert_eq!(status, expected, "node {node}: {method} {path}: {text}");
if text.is_empty() {
Ok(Value::Null)
} else {
Ok(serde_json::from_str(&text)?)
}
}
async fn assert_bucket_state(dist: &DistCluster, bucket: &str, enabled: bool) -> TestResult {
for node in 0..dist.cluster.nodes.len() {
let response =
catalog_request(dist, node, Method::GET, &format!("/iceberg/v1/buckets/{bucket}"), None, StatusCode::OK).await?;
assert_eq!(response["enabled"], enabled, "node {node}: {response}");
assert_eq!(response["table-bucket"], bucket);
let expected = if enabled { StatusCode::OK } else { StatusCode::BAD_REQUEST };
catalog_request(dist, node, Method::GET, &format!("/iceberg/v1/{bucket}/namespaces"), None, expected).await?;
assert_eq!(
get_object_bytes(&dist.client(node)?, bucket, "ordinary.txt").await?,
b"ordinary bucket object",
"node {node}: toggling table support changed an ordinary object"
);
}
Ok(())
}
async fn assert_table_state(dist: &DistCluster, table_path: &str, created: &Value, pointer: &Value) -> TestResult {
for node in 0..dist.cluster.nodes.len() {
let loaded = catalog_request(dist, node, Method::GET, table_path, None, StatusCode::OK).await?;
assert_eq!(loaded["metadata-location"], created["metadata-location"], "node {node}");
assert_eq!(loaded["metadata"], created["metadata"], "node {node}");
let actual_pointer =
catalog_request(dist, node, Method::GET, &format!("{table_path}/metadata-location"), None, StatusCode::OK).await?;
assert_eq!(&actual_pointer, pointer, "node {node}: catalog pointer changed");
}
Ok(())
}
#[tokio::test]
async fn table_bucket_toggles_reload_cached_metadata_on_every_peer() -> TestResult {
init_logging();
let dist = DistCluster::start_with_env(DistLayout::FourByFour, &[("RUSTFS_TABLE_CATALOG_BACKING", "object")]).await?;
let bucket = unique_bucket("table-peer-reload");
dist.create_bucket(&bucket).await?;
put_object(&dist.client(0)?, &bucket, "ordinary.txt", b"ordinary bucket object".to_vec()).await?;
let bucket_path = format!("/iceberg/v1/buckets/{bucket}");
// Warm each node's false cache before changing the marker. Do not poll:
// successful toggles must notify peers before their response is returned.
assert_bucket_state(&dist, &bucket, false).await?;
for (node, method, enabled) in [(0, Method::PUT, true), (1, Method::DELETE, false), (2, Method::PUT, true)] {
let response = catalog_request(&dist, node, method, &bucket_path, None, StatusCode::OK).await?;
assert_eq!(response["enabled"], enabled);
assert_bucket_state(&dist, &bucket, enabled).await?;
}
let namespaces = format!("/iceberg/v1/{bucket}/namespaces");
catalog_request(&dist, 3, Method::POST, &namespaces, Some(json!({"namespace": ["lab"]})), StatusCode::OK).await?;
let tables = format!("{namespaces}/lab/tables");
let created = catalog_request(
&dist,
0,
Method::POST,
&tables,
Some(json!({
"name": "events",
"schema": {
"type": "struct",
"fields": [{"id": 1, "name": "id", "required": false, "type": "long"}]
},
"stage-create": false
})),
StatusCode::OK,
)
.await?;
let uuid = created["metadata"]["table-uuid"]
.as_str()
.ok_or("create response omitted table UUID")?;
assert!(!uuid::Uuid::parse_str(uuid)?.is_nil());
assert!(
created["metadata-location"]
.as_str()
.is_some_and(|location| location.starts_with(&format!("s3://{bucket}/")))
);
let table_path = format!("{tables}/events");
let pointer =
catalog_request(&dist, 3, Method::GET, &format!("{table_path}/metadata-location"), None, StatusCode::OK).await?;
assert_eq!(pointer["metadata-location"], created["metadata-location"]);
assert!(pointer["version-token"].as_str().is_some_and(|token| !token.is_empty()));
assert!(pointer["generation"].as_u64().is_some());
assert_eq!(pointer["warehouse-location"], created["metadata"]["location"]);
assert_table_state(&dist, &table_path, &created, &pointer).await?;
// A rejected toggle must preserve both peer protections and catalog state.
catalog_request(&dist, 1, Method::DELETE, &bucket_path, None, StatusCode::CONFLICT).await?;
assert_bucket_state(&dist, &bucket, true).await?;
assert_table_state(&dist, &table_path, &created, &pointer).await?;
catalog_request(&dist, 2, Method::DELETE, &table_path, None, StatusCode::NO_CONTENT).await?;
catalog_request(&dist, 3, Method::DELETE, &format!("{namespaces}/lab"), None, StatusCode::NO_CONTENT).await?;
// The cluster fixture owns all remaining catalog objects and data roots.
Ok(())
}
+175 -11
View File
@@ -27,8 +27,8 @@ use hyper_util::{
};
use prost::Message as _;
use rustfs_protos::proto_gen::node_service::{
BatchGenerallyLockRequest, DeleteRequest, GenerallyLockRequest, ReadVersionRequest, RecoverMultipartSourceRequest,
RenameDataResponse,
BatchGenerallyLockRequest, DeleteRequest, GenerallyLockRequest, PreparePartTransactionResponse, ReadVersionRequest,
RecoverMultipartSourceRequest, RenameDataResponse, SettlePartTransactionRequest, SettlePartTransactionResponse,
};
use std::{
collections::HashSet,
@@ -56,6 +56,35 @@ enum Disposition {
Discard,
}
#[derive(Clone, Copy)]
enum MutationReply {
None,
Rename,
PreparePart,
SettlePart,
}
impl MutationReply {
fn outcome(self, bytes: &[u8]) -> io::Result<Option<(bool, bool)>> {
let result = match self {
Self::None => return Ok(None),
Self::Rename => {
let reply = RenameDataResponse::decode(unary_payload(bytes)?).map_err(io::Error::other)?;
(reply.success, reply.error.is_some())
}
Self::PreparePart => {
let reply = PreparePartTransactionResponse::decode(unary_payload(bytes)?).map_err(io::Error::other)?;
(reply.success, reply.error.is_some())
}
Self::SettlePart => {
let reply = SettlePartTransactionResponse::decode(unary_payload(bytes)?).map_err(io::Error::other)?;
(reply.success, reply.error.is_some())
}
};
Ok(Some(result))
}
}
struct PauseState {
methods: Vec<&'static str>,
scope: Option<Vec<u8>>,
@@ -64,6 +93,9 @@ struct PauseState {
limit: usize,
retain_after_disconnect: bool,
refused_mutations: AtomicUsize,
successful_mutation_replies: AtomicUsize,
settle_commits: AtomicUsize,
settle_rollbacks: AtomicUsize,
responses: AtomicUsize,
matched: AtomicUsize,
blocked: AtomicUsize,
@@ -122,6 +154,17 @@ impl RpcPause {
self.state.refused_mutations.load(Ordering::SeqCst)
}
pub(crate) fn successful_mutation_replies(&self) -> usize {
self.state.successful_mutation_replies.load(Ordering::SeqCst)
}
pub(crate) fn retained_settle_actions(&self) -> (usize, usize) {
(
self.state.settle_commits.load(Ordering::SeqCst),
self.state.settle_rollbacks.load(Ordering::SeqCst),
)
}
pub(crate) fn unsupported_source_inspection_routes(&self) -> HashSet<String> {
self.state
.unsupported_inspection_routes
@@ -394,6 +437,9 @@ impl RpcFaultProxy {
limit,
retain_after_disconnect,
refused_mutations: AtomicUsize::new(0),
successful_mutation_replies: AtomicUsize::new(0),
settle_commits: AtomicUsize::new(0),
settle_rollbacks: AtomicUsize::new(0),
responses: AtomicUsize::new(0),
matched: AtomicUsize::new(0),
blocked: AtomicUsize::new(0),
@@ -509,6 +555,7 @@ async fn forward_request(
if let Some(method) = method.filter(|name| pause.methods.contains(name)) {
let mut route = None;
let mut lock_owners = Vec::new();
let mut settle_action = None;
let matches = if let Some(scope) = pause.scope.as_deref() {
// Inspect bounded control routing while preserving the exact
// signed bytes and trailers forwarded to the backend.
@@ -522,12 +569,19 @@ async fn forward_request(
inspection = Some((pause.clone(), disk));
}
let mut matches = scope_matches(method, &bytes, scope)?;
if matches && method == "SettlePartTransaction" {
settle_action = Some(
SettlePartTransactionRequest::decode(unary_payload(&bytes)?)
.map_err(io::Error::other)?
.rollback,
);
}
if matches && (pause.record_lock_owners || pause.lock_owner.is_some()) {
lock_owners = scoped_lock_owners(method, &bytes, scope)?;
if let Some(owner) = &pause.lock_owner {
matches = lock_owners.iter().any(|candidate| candidate == owner);
if !matches && pause.record_lock_owners {
observed = Some((pause.clone(), None, std::mem::take(&mut lock_owners), false));
observed = Some((pause.clone(), None, std::mem::take(&mut lock_owners), MutationReply::None));
}
}
}
@@ -549,6 +603,14 @@ async fn forward_request(
true
};
if matches && pause.matched.fetch_add(1, Ordering::SeqCst) < pause.limit {
if let Some(rollback) = settle_action {
let counter = if rollback {
&pause.settle_rollbacks
} else {
&pause.settle_commits
};
counter.fetch_add(1, Ordering::SeqCst);
}
let mut released = pause.release.subscribe();
pause.blocked.fetch_add(1, Ordering::SeqCst);
pause.reached.notify_one();
@@ -568,10 +630,14 @@ async fn forward_request(
pause,
route,
lock_owners,
matches!(
method,
"RenameData" | "RenameMultipartData" | "RenameDataAtIncarnation" | "RenameDataWithQuotaFence"
),
match method {
"RenameData" | "RenameMultipartData" | "RenameDataAtIncarnation" | "RenameDataWithQuotaFence" => {
MutationReply::Rename
}
"PreparePartTransaction" => MutationReply::PreparePart,
"SettlePartTransaction" => MutationReply::SettlePart,
_ => MutationReply::None,
},
));
}
}
@@ -604,7 +670,7 @@ async fn forward_request(
.expect("inspection response routes")
.insert(disk);
}
if let Some((pause, route, lock_owners, rename)) = observed {
if let Some((pause, route, lock_owners, mutation_reply)) = observed {
pause.responses.fetch_add(1, Ordering::SeqCst);
if response.status().is_success() && response.headers().get("grpc-status").is_none_or(|status| status == "0") {
pause.forwarded.fetch_add(1, Ordering::SeqCst);
@@ -626,10 +692,14 @@ async fn forward_request(
.headers
.get("grpc-status")
.or_else(|| trailers.as_ref()?.get("grpc-status"));
if rename && parts.status.is_success() && status.is_none_or(|status| status == "0") {
let result = RenameDataResponse::decode(unary_payload(&bytes)?).map_err(io::Error::other)?;
if !result.success && result.error.is_some() {
if parts.status.is_success()
&& status.is_some_and(|status| status == "0")
&& let Some((success, has_error)) = mutation_reply.outcome(&bytes)?
{
if !success && has_error {
pause.refused_mutations.fetch_add(1, Ordering::SeqCst);
} else if success && !has_error {
pause.successful_mutation_replies.fetch_add(1, Ordering::SeqCst);
}
}
let body = Full::new(bytes)
@@ -676,6 +746,10 @@ mod tests {
}
async fn backend_with_status(status: Option<&'static str>) -> io::Result<Backend> {
backend_with_reply(status, None).await
}
async fn backend_with_reply(status: Option<&'static str>, reply: Option<Bytes>) -> io::Result<Backend> {
let listener = TcpListener::bind("127.0.0.1:0").await?;
let address = listener.local_addr()?;
let received = Arc::new(AtomicUsize::new(0));
@@ -687,9 +761,11 @@ mod tests {
accepted = listener.accept() => {
let Ok((socket, _)) = accepted else { break };
let received = received_task.clone();
let reply = reply.clone();
children.spawn(async move {
let service = service_fn(move |request: Request<Incoming>| {
let received = received.clone();
let reply = reply.clone();
async move {
received.fetch_add(1, Ordering::SeqCst);
let authority = request.headers().get(http::header::HOST).cloned().or_else(|| {
@@ -698,6 +774,7 @@ mod tests {
})
});
let body = request.into_body().collect().await?.to_bytes();
let body = reply.unwrap_or(body);
let mut trailers = http::HeaderMap::new();
trailers.insert("grpc-status", http::HeaderValue::from_static(status.unwrap_or("0")));
trailers.insert("x-proof", http::HeaderValue::from_static("unchanged-proof"));
@@ -1434,6 +1511,93 @@ mod tests {
.await
.expect("scoped discard deadline");
assert_late_packet_ownership().await;
assert_late_part_reply_refusal().await;
}
async fn assert_late_part_reply_refusal() {
use rustfs_protos::proto_gen::node_service::{Error, PreparePartTransactionRequest};
tokio::time::timeout(Duration::from_secs(10), async {
// method, success, error, gRPC status, rollback, expected successes/refusals
for (method, success, has_error, status, rollback, expected) in [
("PreparePartTransaction", false, true, None, false, (0, 1)),
("PreparePartTransaction", true, false, None, false, (1, 0)),
("PreparePartTransaction", false, false, None, false, (0, 0)),
("SettlePartTransaction", true, false, None, false, (1, 0)),
("SettlePartTransaction", false, true, None, true, (0, 1)),
("SettlePartTransaction", true, true, None, false, (0, 0)),
("SettlePartTransaction", false, true, Some("16"), false, (0, 0)),
("PreparePartTransaction", false, true, Some("16"), false, (0, 0)),
("RenameMultipartData", false, true, None, false, (0, 1)),
] {
let error = has_error.then(|| Error {
code: 1,
error_info: "retired source".to_owned(),
});
let reply = match method {
"PreparePartTransaction" => unary_frame(&PreparePartTransactionResponse { success, error }),
"SettlePartTransaction" => unary_frame(&SettlePartTransactionResponse { success, error }),
_ => unary_frame(&RenameDataResponse {
success,
error,
..Default::default()
}),
};
// An unknown field must survive typed inspection byte-for-byte.
let mut framed = reply.to_vec();
framed.extend_from_slice(&[0xfa, 0x03, 0x03, b'x', b'y', b'z']);
let payload_len = u32::try_from(framed.len() - 5).expect("small reply");
framed[1..5].copy_from_slice(&payload_len.to_be_bytes());
let reply = Bytes::from(framed);
let backend = backend_with_reply(status, Some(reply.clone()))
.await
.expect("typed reply backend");
let proxy = RpcFaultProxy::start_on("127.0.0.1:0".parse().expect("loopback"), backend.address)
.await
.expect("proxy");
let pause = proxy.pause_late(&[method], 1, "late-upload").expect("part gate");
let request = match method {
"PreparePartTransaction" => unary_frame(&PreparePartTransactionRequest {
disk: "disk".to_owned(),
dst_path: "late-upload/part.1".to_owned(),
..Default::default()
}),
"SettlePartTransaction" => unary_frame(&SettlePartTransactionRequest {
disk: "disk".to_owned(),
path: "late-upload/part.1".to_owned(),
rollback,
..Default::default()
}),
_ => Bytes::from_static(b"late-upload"),
};
let address = proxy.local_addr();
let caller = tokio::spawn(async move {
let client: ProxyClient = Client::builder(TokioExecutor::new()).http2_only(true).build_http();
let response = call_body(&client, address, method, request).await.expect("retained reply");
response.into_body().collect().await.expect("typed response frames")
});
pause.wait_for_hits(1).await.expect("actual retained request");
pause.release();
let response = caller.await.expect("caller task");
let trailers = response.trailers().expect("reply trailers");
assert_eq!(trailers["x-proof"], "unchanged-proof");
assert_eq!(trailers["grpc-status"], status.unwrap_or("0"));
assert_eq!(response.to_bytes(), reply);
assert_eq!(pause.responses(), 1);
assert_eq!(
(pause.successful_mutation_replies(), pause.refused_mutations()),
expected,
"typed {method} outcome"
);
let actions = if method == "SettlePartTransaction" {
if rollback { (0, 1) } else { (1, 0) }
} else {
(0, 0)
};
assert_eq!(pause.retained_settle_actions(), actions);
}
})
.await
.expect("part reply deadline");
}
async fn assert_late_packet_ownership() {
tokio::time::timeout(Duration::from_secs(10), async {
+3
View File
@@ -12,6 +12,9 @@ Applies to `crates/ecstore/`.
- Be careful with allocations and locking in hot paths.
- Keep network and disk operations async-friendly; avoid introducing unnecessary blocking.
- Coalesce related `open`, `read`/`write`, `fsync`, and `close` filesystem steps into a single per-disk blocking closure rather than chaining multiple async or blocking dispatches, and route any closure that performs file or directory sync to the disk's dedicated fsync lane so stalled syncs cannot starve foreground metadata reads.
- Pin a parent directory once (`openat2(BENEATH | NO_SYMLINKS)` on Linux) and reuse the pinned handle across sibling reads or same-directory renames instead of re-walking ancestor components per file.
- When the file size is already known from `fstat` metadata, pre-size the destination buffer (`len + 1`) before reading to EOF so small metadata reads complete in a single `read()` + EOF check rather than exponential-growth reallocations and syscalls.
- Benchmark-sensitive changes should include measurable rationale.
## Allocation Discipline in Hot Paths
+1
View File
@@ -245,6 +245,7 @@ windows-sys = { workspace = true, features = ["Win32_System_Ioctl"] }
[dev-dependencies]
aws-smithy-async.workspace = true
rustfs-common = { workspace = true, features = ["test-util"] }
rustfs-s3-ops.workspace = true
tokio = { workspace = true, features = ["rt-multi-thread", "macros", "test-util", "fs"] }
criterion = { workspace = true, features = ["html_reports"] }
+9 -9
View File
@@ -341,9 +341,9 @@ pub mod cache {
pub mod capacity {
pub use crate::core::pools::{
DecommissionUnresolvedEntry, PoolDecommissionInfo, PoolMeta, PoolStatus, get_total_usable_capacity,
get_total_usable_capacity_free, is_pool_activation_fleet_proof_error, path2_bucket_object,
path2_bucket_object_with_base_path,
DecommissionUnresolvedEntry, PoolAdminDiagnostic, PoolAvailability, PoolCapacityStatus, PoolDecommissionInfo,
PoolDiagnosticStatus, PoolMeta, PoolStatus, get_total_usable_capacity, get_total_usable_capacity_free,
is_pool_activation_fleet_proof_error, path2_bucket_object, path2_bucket_object_with_base_path,
};
pub use crate::store::utils::is_reserved_or_invalid_bucket;
}
@@ -527,15 +527,15 @@ pub mod notification {
pub use crate::services::notification_sys::{
ClusterTierDailyStats, CrossPoolFenceFleetProofToken, IlmRecoveryExportFleetProofToken,
LegacyTransitionStateReconcileFleetProofToken, MrfLifecycleWriterFleetProofToken, NotificationPeerErr, NotificationSys,
ScannerPublicationLeaseGrant, acquire_cross_pool_fence_fleet_proof, acquire_ilm_recovery_export_fleet_proof,
acquire_legacy_transition_state_reconcile_fleet_proof, acquire_mrf_lifecycle_writer_fleet_proof,
configured_fleet_topology_fingerprint, cross_pool_fence_fleet_proof_matches, get_global_notification_sys,
ilm_recovery_export_fleet_proof_matches, ilm_recovery_export_local_process_epoch,
ScannerPublicationLeaseFailureInfo, ScannerPublicationLeaseGrant, acquire_cross_pool_fence_fleet_proof,
acquire_ilm_recovery_export_fleet_proof, acquire_legacy_transition_state_reconcile_fleet_proof,
acquire_mrf_lifecycle_writer_fleet_proof, configured_fleet_topology_fingerprint, cross_pool_fence_fleet_proof_matches,
get_global_notification_sys, ilm_recovery_export_fleet_proof_matches, ilm_recovery_export_local_process_epoch,
ilm_recovery_export_member_epochs_sha256, ilm_recovery_export_topology_generation,
legacy_transition_state_reconcile_fleet_proof_matches, mrf_lifecycle_reader_local_process_epoch,
mrf_lifecycle_writer_fleet_proof_current, mrf_lifecycle_writer_fleet_proof_matches, new_global_notification_sys,
scanner_peer_transport_error_message_is_retryable, start_remote_version_state_fleet_probe,
with_mrf_lifecycle_writer_fleet_proof,
scanner_peer_transport_error_message_is_retryable, scanner_publication_lease_failure_info,
start_remote_version_state_fleet_probe, with_mrf_lifecycle_writer_fleet_proof,
};
#[cfg(any(test, feature = "test-util"))]
pub use crate::services::notification_sys::{
@@ -668,6 +668,136 @@ impl ScannerPublicationLease {
}
}
struct ScannerPublicationLeaseLocalError {
reason: &'static str,
detail: String,
}
impl std::fmt::Debug for ScannerPublicationLeaseLocalError {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter
.debug_struct("ScannerPublicationLeaseLocalError")
.field("reason", &self.reason)
.finish()
}
}
impl std::fmt::Display for ScannerPublicationLeaseLocalError {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter.write_str(&self.detail)
}
}
impl std::error::Error for ScannerPublicationLeaseLocalError {}
struct ScannerPublicationLeaseRemoteRejection(String);
impl std::fmt::Debug for ScannerPublicationLeaseRemoteRejection {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter.write_str("ScannerPublicationLeaseRemoteRejection")
}
}
impl std::fmt::Display for ScannerPublicationLeaseRemoteRejection {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter.write_str(&self.0)
}
}
impl std::error::Error for ScannerPublicationLeaseRemoteRejection {}
pub(crate) fn scanner_publication_lease_remote_rejection(detail: String) -> Error {
Error::other(ScannerPublicationLeaseRemoteRejection(detail))
}
fn scanner_publication_lease_local_error(reason: &'static str, detail: &'static str) -> Error {
Error::other(ScannerPublicationLeaseLocalError {
reason,
detail: detail.to_owned(),
})
}
fn scanner_publication_lease_local_io_error(reason: &'static str, error: std::io::Error) -> Error {
Error::Io(std::io::Error::new(
error.kind(),
ScannerPublicationLeaseLocalError {
reason,
detail: error.to_string(),
},
))
}
#[derive(Debug)]
pub(crate) struct ScannerPublicationLeaseErrorClass {
pub reason: Option<&'static str>,
pub reported_reason: Option<&'static str>,
}
pub(crate) fn scanner_publication_lease_error_class(error: &Error) -> ScannerPublicationLeaseErrorClass {
let mut result = ScannerPublicationLeaseErrorClass {
reason: None,
reported_reason: None,
};
match error {
Error::RemoteClientUnavailable(_) => result.reason = Some("client_unavailable"),
Error::RemoteNotInitialized => result.reason = Some("peer_not_initialized"),
Error::Timeout => result.reason = Some("timeout"),
Error::Io(error) => {
if let Some(local) = error
.get_ref()
.and_then(|source| source.downcast_ref::<ScannerPublicationLeaseLocalError>())
{
result.reason = Some(local.reason);
} else if let Some(remote) = error
.get_ref()
.and_then(|source| source.downcast_ref::<ScannerPublicationLeaseRemoteRejection>())
{
// These are the exact Display strings currently produced by
// the peer's lease acquisition path, not arbitrary suffixes.
result.reported_reason = match remote.0.as_str() {
"Io error: scanner publication lease capacity is exhausted" => Some("capacity_exhausted"),
"Io error: scanner publication lease generation is stale" => Some("generation_stale"),
"Io error: scanner publication lease is blocked by data movement" => Some("blocked_by_data_movement"),
_ => None,
};
} else if let Some(status) = embedded_tonic_status(error) {
result.reason = Some(match status.code() {
tonic::Code::Ok => "rpc_ok",
tonic::Code::Cancelled => "rpc_cancelled",
tonic::Code::Unknown => "rpc_unknown",
tonic::Code::InvalidArgument => "rpc_invalid_argument",
tonic::Code::DeadlineExceeded => "rpc_deadline_exceeded",
tonic::Code::NotFound => "rpc_not_found",
tonic::Code::AlreadyExists => "rpc_already_exists",
tonic::Code::PermissionDenied => "rpc_permission_denied",
tonic::Code::ResourceExhausted => "rpc_resource_exhausted",
tonic::Code::FailedPrecondition => "rpc_failed_precondition",
tonic::Code::Aborted => "rpc_aborted",
tonic::Code::OutOfRange => "rpc_out_of_range",
tonic::Code::Unimplemented => "rpc_unimplemented",
tonic::Code::Internal => "rpc_internal",
tonic::Code::Unavailable => "rpc_unavailable",
tonic::Code::DataLoss => "rpc_data_loss",
tonic::Code::Unauthenticated => "rpc_unauthenticated",
});
} else {
result.reason = match error.kind() {
std::io::ErrorKind::TimedOut => Some("io_timeout"),
std::io::ErrorKind::ConnectionRefused => Some("io_connection_refused"),
std::io::ErrorKind::ConnectionReset => Some("io_connection_reset"),
std::io::ErrorKind::ConnectionAborted => Some("io_connection_aborted"),
std::io::ErrorKind::BrokenPipe => Some("io_broken_pipe"),
std::io::ErrorKind::UnexpectedEof => Some("io_unexpected_eof"),
std::io::ErrorKind::NotConnected => Some("io_not_connected"),
_ => None,
};
}
}
_ => {}
}
result
}
fn validate_scanner_publication_lease_response_fields(
response: &ScannerPublicationLeaseResponse,
expected_session_id: &str,
@@ -675,7 +805,7 @@ fn validate_scanner_publication_lease_response_fields(
expected_observational_only: bool,
) -> Result<(Uuid, String)> {
if !response.success {
return Err(Error::other(
return Err(scanner_publication_lease_remote_rejection(
response
.error
.as_ref()
@@ -684,24 +814,39 @@ fn validate_scanner_publication_lease_response_fields(
));
}
if response.movement_generation != expected_generation {
return Err(Error::other("peer returned a different scanner publication lease generation"));
return Err(scanner_publication_lease_local_error(
"generation_mismatch",
"peer returned a different scanner publication lease generation",
));
}
if response.session_id != expected_session_id {
return Err(Error::other("peer returned a different scanner publication lease session"));
return Err(scanner_publication_lease_local_error(
"session_mismatch",
"peer returned a different scanner publication lease session",
));
}
if response.observational_only != expected_observational_only {
return Err(Error::other("peer returned a different scanner publication lease purpose"));
return Err(scanner_publication_lease_local_error(
"purpose_mismatch",
"peer returned a different scanner publication lease purpose",
));
}
let owner_id = Uuid::parse_str(&response.owner_id)
.ok()
.filter(|owner_id| !owner_id.is_nil())
.map(|owner_id| owner_id.to_string())
.ok_or_else(|| Error::other("peer returned an invalid scanner publication lease owner"))?;
.ok_or_else(|| {
scanner_publication_lease_local_error("invalid_owner", "peer returned an invalid scanner publication lease owner")
})?;
if response.lease_ttl_ms != crate::store::SCANNER_PUBLICATION_LEASE_TTL_MS {
return Err(Error::other("peer returned an unsupported scanner publication lease TTL"));
return Err(scanner_publication_lease_local_error(
"unsupported_ttl",
"peer returned an unsupported scanner publication lease TTL",
));
}
let token = Uuid::from_slice(response.token.as_ref())
.map_err(|_| Error::other("peer returned an invalid scanner publication lease token"))?;
let token = Uuid::from_slice(response.token.as_ref()).map_err(|_| {
scanner_publication_lease_local_error("invalid_token", "peer returned an invalid scanner publication lease token")
})?;
Ok((token, owner_id))
}
@@ -712,16 +857,27 @@ fn scanner_publication_lease_deadline(
) -> Result<std::time::Instant> {
let lease_window = Duration::from_millis(lease_ttl_ms)
.checked_sub(SCANNER_PUBLICATION_LEASE_SAFETY_MARGIN)
.ok_or_else(|| Error::other("scanner publication lease TTL is shorter than its safety margin"))?;
let elapsed = response_received
.checked_duration_since(request_started)
.ok_or_else(|| Error::other("scanner publication lease response clock moved backwards"))?;
.ok_or_else(|| {
scanner_publication_lease_local_error(
"ttl_below_safety_margin",
"scanner publication lease TTL is shorter than its safety margin",
)
})?;
let elapsed = response_received.checked_duration_since(request_started).ok_or_else(|| {
scanner_publication_lease_local_error(
"response_clock_backwards",
"scanner publication lease response clock moved backwards",
)
})?;
if elapsed >= lease_window {
return Err(Error::other("scanner publication lease response arrived after its safety window"));
return Err(scanner_publication_lease_local_error(
"response_after_safety_window",
"scanner publication lease response arrived after its safety window",
));
}
request_started
.checked_add(lease_window)
.ok_or_else(|| Error::other("scanner publication lease deadline overflowed"))
request_started.checked_add(lease_window).ok_or_else(|| {
scanner_publication_lease_local_error("deadline_overflow", "scanner publication lease deadline overflowed")
})
}
/// A peer's readiness view of one drive served by another host.
@@ -876,6 +1032,11 @@ fn tier_mutation_failed_response_error(version: u32, failure_class: i32, error_i
Error::other(message)
}
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(crate) enum ScannerLeaseConnectionFailure {
ChannelAcquisitionTimedOut,
}
impl PeerRestClient {
fn recovery_monitor_span(grid_host: &str) -> tracing::Span {
tracing::info_span!(
@@ -1003,19 +1164,37 @@ impl PeerRestClient {
}
pub async fn get_client(&self) -> Result<NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>> {
self.get_client_with_connection_failure().await.map_err(|(error, _)| error)
}
async fn get_client_with_connection_failure(
&self,
) -> std::result::Result<
NodeServiceClient<InterceptedService<AuthenticatedChannel, TonicInterceptor>>,
(Error, Option<ScannerLeaseConnectionFailure>),
> {
if self.offline.load(Ordering::Acquire) {
self.mark_offline_and_spawn_recovery();
return Err(Error::RemoteClientUnavailable(format!("peer {} is temporarily offline", self.grid_host)));
return Err((
Error::RemoteClientUnavailable(format!("peer {} is temporarily offline", self.grid_host)),
None,
));
}
node_service_time_out_client(&self.grid_host, TonicInterceptor::Signature(gen_tonic_signature_interceptor()))
.await
.map_err(|err| {
// Channel acquisition reports its deadline as a top-level
// typed I/O timeout. Do not infer kinds from dial error text.
let failure = err
.downcast_ref::<std::io::Error>()
.filter(|error| error.kind() == std::io::ErrorKind::TimedOut)
.map(|_| ScannerLeaseConnectionFailure::ChannelAcquisitionTimedOut);
let storage_err = Error::RemoteClientUnavailable(format!("can not get client, err: {err}"));
if Self::is_network_like_error(&storage_err) {
self.mark_offline_and_spawn_recovery();
}
storage_err
(storage_err, failure)
})
}
@@ -2508,54 +2687,91 @@ impl PeerRestClient {
expected_generation: u64,
observational_only: bool,
) -> Result<ScannerPublicationLease> {
let request_started = std::time::Instant::now();
self.finalize_result(
async {
let challenge = Uuid::new_v4();
let mut client = self
.get_client()
.await?
.max_decoding_message_size(SCANNER_ACTIVITY_MAX_MESSAGE_SIZE)
.max_encoding_message_size(SCANNER_ACTIVITY_MAX_MESSAGE_SIZE);
let mut request = Request::new(ScannerPublicationLeaseRequest {
challenge: challenge.as_bytes().to_vec().into(),
expected_movement_generation: expected_generation,
ttl_ms: crate::store::SCANNER_PUBLICATION_LEASE_TTL_MS,
expected_session_id: expected_session_id.to_string(),
token: Bytes::new(),
observational_only,
});
let canonical = rustfs_protos::canonical_scanner_publication_lease_request_body(request.get_ref())
.map_err(|_| Error::other("scanner publication lease request is too large to authenticate"))?;
set_tonic_canonical_body_digest(&mut request, &canonical)?;
let response = client.acquire_scanner_publication_lease(request).await?.into_inner();
let response_body =
rustfs_protos::canonical_scanner_publication_lease_response_body(challenge.as_bytes(), &response)
.map_err(|_| Error::other("scanner publication lease response is too large to authenticate"))?;
verify_tonic_rpc_response_proof(&response_body, &response.response_proof)
.map_err(|_| Error::other("peer returned an invalid scanner publication lease proof"))?;
let (token, owner_id) = validate_scanner_publication_lease_response_fields(
&response,
expected_session_id,
expected_generation,
observational_only,
)?;
Ok(ScannerPublicationLease {
token,
movement_generation: response.movement_generation,
observational_only,
owner_id,
session_id: response.session_id,
expires_at: scanner_publication_lease_deadline(
request_started,
std::time::Instant::now(),
response.lease_ttl_ms,
)?,
})
}
.await,
self.acquire_scanner_publication_lease_with_connection_failure(
expected_session_id,
expected_generation,
observational_only,
)
.await
.map_err(|(error, _)| error)
}
pub(crate) async fn acquire_scanner_publication_lease_with_connection_failure(
&self,
expected_session_id: &str,
expected_generation: u64,
observational_only: bool,
) -> std::result::Result<ScannerPublicationLease, (Error, Option<ScannerLeaseConnectionFailure>)> {
let request_started = std::time::Instant::now();
let mut connection_failure = None;
let result = self
.finalize_result(
async {
let challenge = Uuid::new_v4();
let mut client = self
.get_client_with_connection_failure()
.await
.map_err(|(error, failure)| {
connection_failure = failure;
error
})?
.max_decoding_message_size(SCANNER_ACTIVITY_MAX_MESSAGE_SIZE)
.max_encoding_message_size(SCANNER_ACTIVITY_MAX_MESSAGE_SIZE);
let mut request = Request::new(ScannerPublicationLeaseRequest {
challenge: challenge.as_bytes().to_vec().into(),
expected_movement_generation: expected_generation,
ttl_ms: crate::store::SCANNER_PUBLICATION_LEASE_TTL_MS,
expected_session_id: expected_session_id.to_string(),
token: Bytes::new(),
observational_only,
});
let canonical =
rustfs_protos::canonical_scanner_publication_lease_request_body(request.get_ref()).map_err(|_| {
scanner_publication_lease_local_error(
"request_too_large",
"scanner publication lease request is too large to authenticate",
)
})?;
set_tonic_canonical_body_digest(&mut request, &canonical)
.map_err(|error| scanner_publication_lease_local_io_error("request_digest_failed", error))?;
let response = client.acquire_scanner_publication_lease(request).await?.into_inner();
let response_body =
rustfs_protos::canonical_scanner_publication_lease_response_body(challenge.as_bytes(), &response)
.map_err(|_| {
scanner_publication_lease_local_error(
"response_too_large",
"scanner publication lease response is too large to authenticate",
)
})?;
verify_tonic_rpc_response_proof(&response_body, &response.response_proof).map_err(|_| {
scanner_publication_lease_local_error(
"invalid_proof",
"peer returned an invalid scanner publication lease proof",
)
})?;
let (token, owner_id) = validate_scanner_publication_lease_response_fields(
&response,
expected_session_id,
expected_generation,
observational_only,
)?;
Ok(ScannerPublicationLease {
token,
movement_generation: response.movement_generation,
observational_only,
owner_id,
session_id: response.session_id,
expires_at: scanner_publication_lease_deadline(
request_started,
std::time::Instant::now(),
response.lease_ttl_ms,
)?,
})
}
.await,
)
.await;
result.map_err(|error| (error, connection_failure))
}
/// Revalidate the exact token immediately before the coordinator commits
@@ -3146,6 +3362,162 @@ mod tests {
let error = scanner_publication_lease_deadline(started, started + Duration::from_secs(55), 60_000)
.expect_err("a response arriving at the safety boundary must fail closed");
assert!(error.to_string().contains("after its safety window"));
assert_eq!(scanner_publication_lease_error_class(&error).reason, Some("response_after_safety_window"));
assert!(
error
.to_string()
.ends_with("scanner publication lease response arrived after its safety window")
);
}
#[test]
fn scanner_publication_lease_validation_errors_are_typed_without_changing_display() {
let valid = ScannerPublicationLeaseResponse {
success: true,
token: Uuid::new_v4().as_bytes().to_vec().into(),
movement_generation: 7,
lease_ttl_ms: crate::store::SCANNER_PUBLICATION_LEASE_TTL_MS,
error: None,
response_proof: Bytes::new(),
owner_id: Uuid::new_v4().to_string(),
session_id: "session-a".to_owned(),
observational_only: false,
};
let cases = [
(
ScannerPublicationLeaseResponse {
movement_generation: 6,
..valid.clone()
},
"generation_mismatch",
"peer returned a different scanner publication lease generation",
),
(
ScannerPublicationLeaseResponse {
session_id: "other".to_owned(),
..valid.clone()
},
"session_mismatch",
"peer returned a different scanner publication lease session",
),
(
ScannerPublicationLeaseResponse {
observational_only: true,
..valid.clone()
},
"purpose_mismatch",
"peer returned a different scanner publication lease purpose",
),
(
ScannerPublicationLeaseResponse {
owner_id: Uuid::nil().to_string(),
..valid.clone()
},
"invalid_owner",
"peer returned an invalid scanner publication lease owner",
),
(
ScannerPublicationLeaseResponse {
lease_ttl_ms: 0,
..valid.clone()
},
"unsupported_ttl",
"peer returned an unsupported scanner publication lease TTL",
),
(
ScannerPublicationLeaseResponse {
token: Bytes::from_static(b"short"),
..valid.clone()
},
"invalid_token",
"peer returned an invalid scanner publication lease token",
),
];
for (response, reason, message) in cases {
let error = validate_scanner_publication_lease_response_fields(&response, "session-a", 7, false)
.expect_err("invalid lease response must be rejected");
let class = scanner_publication_lease_error_class(&error);
assert_eq!(class.reason, Some(reason));
assert_eq!(class.reported_reason, None);
assert_eq!(error.to_string(), format!("Io error: {message}"));
assert!(matches!(error, Error::Io(ref io) if io.kind() == std::io::ErrorKind::Other));
}
for (reason, message) in [
("request_too_large", "scanner publication lease request is too large to authenticate"),
("response_too_large", "scanner publication lease response is too large to authenticate"),
("invalid_proof", "peer returned an invalid scanner publication lease proof"),
("deadline_overflow", "scanner publication lease deadline overflowed"),
] {
let error = scanner_publication_lease_local_error(reason, message);
assert_eq!(scanner_publication_lease_error_class(&error).reason, Some(reason));
assert_eq!(error.to_string(), format!("Io error: {message}"));
assert!(matches!(error, Error::Io(ref io) if io.kind() == std::io::ErrorKind::Other));
}
let original = std::io::Error::new(std::io::ErrorKind::InvalidInput, "digest rejected");
let expected = format!("Io error: {original}");
let wrapped = scanner_publication_lease_local_io_error("request_digest_failed", original);
assert_eq!(scanner_publication_lease_error_class(&wrapped).reason, Some("request_digest_failed"));
assert_eq!(wrapped.to_string(), expected);
assert!(matches!(wrapped, Error::Io(ref io) if io.kind() == std::io::ErrorKind::InvalidInput));
}
#[test]
fn scanner_publication_lease_error_class_keeps_remote_reports_separate_from_local_errors() {
for (remote_message, reported) in [
("Io error: scanner publication lease capacity is exhausted", Some("capacity_exhausted")),
("Io error: scanner publication lease generation is stale", Some("generation_stale")),
(
"Io error: scanner publication lease is blocked by data movement",
Some("blocked_by_data_movement"),
),
("remote-secret Io error: scanner publication lease generation is stale", None),
("not Io error: scanner publication lease generation is stale", None),
("scanner publication lease generation is stale", None),
("Io error: scanner publication lease generation is stale ", None),
("Io error: scanner publication lease is blocked by pending namespace commit", None),
("Io error: scanner publication lease namespace changed during acquisition", None),
] {
let rejected = ScannerPublicationLeaseResponse {
success: false,
error: Some(rustfs_protos::proto_gen::node_service::Error {
code: 0,
error_info: remote_message.to_owned(),
}),
..Default::default()
};
let error = validate_scanner_publication_lease_response_fields(&rejected, "session", 0, false)
.expect_err("peer rejection must be returned");
let class = scanner_publication_lease_error_class(&error);
assert_eq!(class.reason, None);
assert_eq!(class.reported_reason, reported);
assert_eq!(error.to_string(), format!("Io error: {remote_message}"));
assert!(!format!("{error:?}").contains("remote-secret"));
}
let forged = Error::other("peer returned a different scanner publication lease generation");
let class = scanner_publication_lease_error_class(&forged);
assert_eq!(class.reason, None, "untyped error text must not be promoted to a local diagnosis");
assert_eq!(class.reported_reason, None, "untyped text must not be attributed to a peer response");
for (error, expected) in [
(Error::RemoteClientUnavailable("remote-secret".to_owned()), "client_unavailable"),
(Error::RemoteClientUnavailable("pretend timed out".to_owned()), "client_unavailable"),
(Error::Timeout, "timeout"),
(tonic::Status::unavailable("remote-secret").into(), "rpc_unavailable"),
(tonic::Status::deadline_exceeded("remote-secret").into(), "rpc_deadline_exceeded"),
(tonic::Status::permission_denied("remote-secret").into(), "rpc_permission_denied"),
(
Error::Io(std::io::Error::new(std::io::ErrorKind::TimedOut, "remote-secret")),
"io_timeout",
),
] {
let class = scanner_publication_lease_error_class(&error);
assert_eq!(class.reason, Some(expected));
assert_eq!(class.reported_reason, None);
assert!(!format!("{class:?}").contains("remote-secret"));
}
}
#[test]
@@ -41,6 +41,10 @@ use tonic::service::interceptor::InterceptedService;
use tonic::{Request, Response};
use tracing::{debug, info, warn};
const EVENT_NAMESPACE_LOCK_RELEASE: &str = "namespace_lock_release";
const LOG_COMPONENT_ECSTORE: &str = "ecstore";
const LOG_SUBSYSTEM_REMOTE_LOCK: &str = "remote_lock";
fn attach_lock_mutation_body_digest<T: rustfs_protos::CanonicalMutationBody>(request: &mut Request<T>) -> std::io::Result<()> {
set_tonic_rolling_mutation_body_digest(request)
}
@@ -233,6 +237,20 @@ pub struct RemoteClient {
impl RemoteClient {
const ONLINE_CHECK_RESOURCE: &'static str = "health-lock-online";
fn record_release_ack(&self, lock_id: &LockId) {
tracing::trace!(
target: "rustfs_ecstore::object_lock_diag",
event = EVENT_NAMESPACE_LOCK_RELEASE,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REMOTE_LOCK,
state = "acknowledged",
peer_kind = "remote",
lock_id = %lock_id,
peer = %self.addr,
"Remote namespace lock release acknowledged"
);
}
pub fn new(endpoint: String) -> Self {
Self { addr: endpoint }
}
@@ -1204,8 +1222,6 @@ impl LockClient for RemoteClient {
}
async fn release(&self, lock_id: &LockId) -> Result<bool> {
info!("remote release for {}", lock_id);
let unlock_request = Self::create_unlock_request(lock_id);
let request_string = serde_json::to_string(&unlock_request)
.map_err(|e| LockError::internal(format!("Failed to serialize request: {e}")))?;
@@ -1226,6 +1242,9 @@ impl LockClient for RemoteClient {
if let Some(error_info) = resp.error_info {
return Err(LockError::internal(error_info));
}
if resp.success {
self.record_release_ack(lock_id);
}
Ok(resp.success)
}
@@ -1261,7 +1280,13 @@ impl LockClient for RemoteClient {
Ok(lock_ids
.iter()
.enumerate()
.map(|(idx, _)| resp.results.get(idx).map(|result| result.success).unwrap_or(false))
.map(|(idx, lock_id)| {
let result = resp.results.get(idx);
if result.is_some_and(|result| result.success && result.error_info.is_none()) {
self.record_release_ack(lock_id);
}
result.map(|result| result.success).unwrap_or(false)
})
.collect())
}
+432 -16
View File
@@ -3720,6 +3720,46 @@ fn is_decommission_copy_quorum_deferral(err: &Error) -> bool {
data_movement::data_movement_stage_source_as::<DecommissionCopyQuorumFailure>(err).is_some()
}
/// The storage error of a version copy that kept timing out on a namespace
/// lock through its in-place retries, kept typed so the entry failure policy
/// defers it.
#[derive(Debug, thiserror::Error)]
#[error("{0}")]
struct DecommissionCopyLockTimeout(#[source] Error);
fn is_decommission_copy_lock_timeout_error(err: &Error) -> bool {
let err = data_movement::data_movement_stage_source(err).unwrap_or(err);
matches!(err, Error::Lock(rustfs_lock::LockError::Timeout { .. }))
}
/// A version copy that timed out acquiring a namespace lock leaves the source
/// in place; the next attempt copies the version again or finds an equivalent
/// committed target. The lock is held by another mutation of the key or, when
/// this node restarted mid-copy, by its previous incarnation until that lock's
/// lease expires on the peers. Either clears by waiting, so the entry is
/// deferred instead of being counted as a permanent per-object failure.
fn decommission_copy_lock_timeout_deferral(stage: &str, bucket: &str, object: &str, err: Error) -> Error {
data_movement::data_movement_context_error(
format!("decommission {stage} timed out on a namespace lock for {bucket}/{object}: {err}"),
DecommissionCopyLockTimeout(err),
)
}
fn is_decommission_copy_lock_timeout_deferral(err: &Error) -> bool {
data_movement::data_movement_stage_source_as::<DecommissionCopyLockTimeout>(err).is_some()
}
/// The deferral of a version copy whose in-place retries ended on `error`, if
/// that failure belongs to the pools at that moment rather than to the object.
fn take_decommission_copy_deferral(stage: &str, bucket: &str, object: &str, error: &mut Option<Error>) -> Option<Error> {
if let Some(err) = error.take_if(|err| is_decommission_copy_quorum_error(err)) {
return Some(decommission_copy_quorum_deferral(stage, bucket, object, err));
}
error
.take_if(|err| is_decommission_copy_lock_timeout_error(err))
.map(|err| decommission_copy_lock_timeout_deferral(stage, bucket, object, err))
}
fn resolve_decommission_entry_reload_result(result: Result<()>, bucket: &str, object_name: &str) -> Result<()> {
result.map_err(|err| Error::other(format!("decommission reload_pool_meta failed for {bucket}/{object_name}: {err}")))
}
@@ -5782,11 +5822,14 @@ enum DecommissionEntryOutcome {
/// entry is done and must not be counted as a failure or escalated. A quorum
/// failure of the source cleanup is deferred too: the copied source is still
/// in place and the next attempt cleans it up again. So is a version copy that
/// lost drive quorum through its in-place retries: the source still holds the
/// version and the next attempt copies it again. Every other failure stays
/// terminal.
/// lost drive quorum or timed out on a namespace lock through its in-place
/// retries: the source still holds the version and the next attempt copies it
/// again. Every other failure stays terminal.
fn resolve_decommission_entry_failure(err: Error) -> Result<DecommissionEntryOutcome> {
if is_decommission_source_cleanup_retryable_error(&err) || is_decommission_copy_quorum_deferral(&err) {
if is_decommission_source_cleanup_retryable_error(&err)
|| is_decommission_copy_quorum_deferral(&err)
|| is_decommission_copy_lock_timeout_deferral(&err)
{
return Ok(DecommissionEntryOutcome::Deferred(err));
}
match classify_decommission_capacity_failure(&err) {
@@ -11702,6 +11745,61 @@ pub struct PoolSpaceInfo {
pub used: usize,
}
/// Read-only observations used by the admin pool diagnostic surface. The HTTP
/// layer converts these stable values to strings without changing the pool
/// lifecycle status contract.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum PoolAvailability {
Online,
Degraded,
Offline,
Unknown,
}
impl PoolAvailability {
pub fn as_str(self) -> &'static str {
match self {
Self::Online => "online",
Self::Degraded => "degraded",
Self::Offline => "offline",
Self::Unknown => "unknown",
}
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum PoolCapacityStatus {
Live,
Cached,
Partial,
Unknown,
}
impl PoolCapacityStatus {
pub fn as_str(self) -> &'static str {
match self {
Self::Live => "live",
Self::Cached => "cached",
Self::Partial => "partial",
Self::Unknown => "unknown",
}
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct PoolDiagnosticStatus {
pub availability: PoolAvailability,
pub capacity_status: PoolCapacityStatus,
}
#[derive(Debug, Clone)]
pub struct PoolAdminDiagnostic {
pub id: usize,
pub cmd_line: String,
pub pool: Option<PoolStatus>,
pub diagnostic: PoolDiagnosticStatus,
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) struct DecommissionPoolCapacityInfo {
pool_index: usize,
@@ -11710,6 +11808,7 @@ pub(crate) struct DecommissionPoolCapacityInfo {
physical_free: usize,
physical_total: usize,
physical_used: usize,
diagnostic: PoolDiagnosticStatus,
}
impl DecommissionPoolCapacityInfo {
@@ -11732,6 +11831,10 @@ impl DecommissionPoolCapacityInfo {
physical_free,
physical_total,
physical_used,
diagnostic: PoolDiagnosticStatus {
availability: PoolAvailability::Unknown,
capacity_status: PoolCapacityStatus::Unknown,
},
}
}
}
@@ -11753,6 +11856,121 @@ where
Ok(capacities)
}
fn pool_diagnostic_status(
disks: &[rustfs_madmin::Disk],
pool_index: usize,
set_count: usize,
set_drive_count: usize,
read_quorum: usize,
) -> PoolDiagnosticStatus {
if set_count == 0 || set_drive_count == 0 || read_quorum == 0 || read_quorum > set_drive_count {
return PoolDiagnosticStatus {
availability: PoolAvailability::Unknown,
capacity_status: PoolCapacityStatus::Unknown,
};
}
let mut live_capacity = true;
let mut cached_capacity = false;
let mut partial_capacity = false;
let mut online_by_set = vec![0usize; set_count];
let mut observed_disks = 0usize;
let mut seen_slots = HashSet::new();
for disk in disks.iter().filter(|disk| disk.pool_index == pool_index as i32) {
observed_disks = observed_disks.saturating_add(1);
match disk.capacity_observation_source.as_deref() {
Some("live_probe") => {}
Some("snapshot") => {
live_capacity = false;
cached_capacity = true;
}
Some("missing") | None => {
live_capacity = false;
partial_capacity = true;
}
Some(_) => {
live_capacity = false;
partial_capacity = true;
}
}
let Some(set_index) = usize::try_from(disk.set_index).ok() else {
continue;
};
let Some(disk_index) = usize::try_from(disk.disk_index).ok() else {
continue;
};
if set_index >= set_count
|| disk_index >= set_drive_count
|| !seen_slots.insert((set_index, disk_index))
|| !disk_is_available_for_pool_diagnostic(disk)
{
continue;
}
online_by_set[set_index] = online_by_set[set_index].saturating_add(1);
}
let availability = if online_by_set.iter().any(|&online| online < read_quorum) {
PoolAvailability::Offline
} else if online_by_set.iter().all(|&online| online == set_drive_count) {
PoolAvailability::Online
} else {
PoolAvailability::Degraded
};
let capacity_status = if observed_disks == 0 {
PoolCapacityStatus::Unknown
} else if partial_capacity {
PoolCapacityStatus::Partial
} else if cached_capacity {
PoolCapacityStatus::Cached
} else if live_capacity {
PoolCapacityStatus::Live
} else {
PoolCapacityStatus::Unknown
};
PoolDiagnosticStatus {
availability,
capacity_status,
}
}
fn disk_is_available_for_pool_diagnostic(disk: &rustfs_madmin::Disk) -> bool {
let state_ok = disk.state.eq_ignore_ascii_case("ok") || disk.state.eq_ignore_ascii_case(rustfs_madmin::ITEM_ONLINE);
let runtime_ok = disk
.runtime_state
.as_deref()
.map(|state| state.eq_ignore_ascii_case(rustfs_madmin::ITEM_ONLINE))
.unwrap_or(state_ok);
state_ok && runtime_ok
}
fn pool_diagnostic_metadata_status(err: &Error) -> &'static str {
if let Some(failure) = err.pool_metadata_failure() {
return match failure.kind {
crate::error::PoolMetadataFailure::ReadUnavailable => "stale",
crate::error::PoolMetadataFailure::RecoveryRequired
| crate::error::PoolMetadataFailure::TransactionUnknown
| crate::error::PoolMetadataFailure::FenceLost => "recovery_required",
};
}
match err {
Error::Timeout
| Error::Lock(_)
| Error::DiskNotFound
| Error::FaultyDisk
| Error::FaultyRemoteDisk
| Error::RemoteClientUnavailable(_)
| Error::RemoteNotInitialized
| Error::ErasureReadQuorum
| Error::InsufficientReadQuorum(_, _)
| Error::NamespaceLockQuorumUnavailable { .. } => "stale",
_ => "recovery_required",
}
}
#[cfg(test)]
static DECOMMISSION_START_GATE_WAIT_OVERRIDES: std::sync::OnceLock<std::sync::Mutex<HashMap<uuid::Uuid, std::time::Duration>>> =
std::sync::OnceLock::new();
@@ -15125,6 +15343,70 @@ impl ECStore {
Ok(snapshot)
}
/// Admin-only diagnostic snapshot. A transient metadata read failure is
/// reported as stale without publishing the fallback or relaxing any
/// mutation/recovery path.
pub async fn admin_pool_diagnostic_snapshot(
&self,
target_pool_index: Option<usize>,
) -> (Vec<PoolAdminDiagnostic>, &'static str) {
let fallback = self.pool_meta.read().await.clone();
let (pool_meta, mut metadata_status) = match self.pool_status_meta_snapshot().await {
Ok(pool_meta) => (pool_meta, "fresh"),
Err(err) => (fallback, pool_diagnostic_metadata_status(&err)),
};
if metadata_status == "stale" && pool_meta.pools.len() != self.pools.len() {
metadata_status = "unknown";
}
let backend = rustfs_madmin::StorageInfo {
backend: StorageAdminApi::backend_info(self).await,
..Default::default()
};
let indices = target_pool_index.map_or_else(|| (0..self.pools.len()).collect(), |idx| vec![idx]);
let backend = &backend;
let samples = futures::future::join_all(indices.into_iter().map(|idx| async move {
(
idx,
self.get_decommission_pool_capacity_info_with_backend(idx, false, backend)
.await,
)
}))
.await;
let mut diagnostics = Vec::with_capacity(samples.len());
for (idx, sample) in samples {
let Some(sets) = self.pools.get(idx) else {
continue;
};
let (pool, diagnostic) = match sample {
Ok(capacity) => (
pool_meta
.pools
.get(idx)
.cloned()
.map(|pool| apply_decommission_status_space_info(pool, capacity.space)),
capacity.diagnostic,
),
Err(_) => (
pool_meta.pools.get(idx).cloned(),
PoolDiagnosticStatus {
availability: PoolAvailability::Unknown,
capacity_status: PoolCapacityStatus::Unknown,
},
),
};
diagnostics.push(PoolAdminDiagnostic {
id: idx,
cmd_line: sets.endpoints.cmd_line.clone(),
pool,
diagnostic,
});
}
(diagnostics, metadata_status)
}
/// Load and publish under the movement writer, so no movement read sees
/// the transition mid-operation and no transition interleaves the load.
async fn publish_pool_status_refresh(&self, _movement_guard: tokio::sync::OwnedRwLockWriteGuard<()>) -> Result<()> {
@@ -15201,6 +15483,7 @@ impl ECStore {
};
let (physical_total, physical_free, physical_used) =
decommission_physical_pool_capacity(&info.disks, idx, layout, space, owned_physical_used.as_ref());
let diagnostic = pool_diagnostic_status(&info.disks, idx, sets.set_count, sets.set_drive_count, layout.data);
Ok(DecommissionPoolCapacityInfo {
pool_index: idx,
@@ -15209,6 +15492,7 @@ impl ECStore {
physical_free,
physical_total,
physical_used,
diagnostic,
})
} else {
Err(invalid_decommission_pool_index_error(self.pools.len(), idx))
@@ -18150,13 +18434,11 @@ impl ECStore {
}
}
if failure && let Some(err) = error.take_if(|err| is_decommission_copy_quorum_error(err)) {
return Err(decommission_copy_quorum_deferral(
"delete_marker_copy",
bucket.as_str(),
version.name.as_str(),
err,
));
if failure
&& let Some(err) =
take_decommission_copy_deferral("delete_marker_copy", bucket.as_str(), version.name.as_str(), &mut error)
{
return Err(err);
}
if ignore {
@@ -18542,14 +18824,14 @@ impl ECStore {
if failure
&& !source_read_failed
&& let Some(err) = error.take_if(|err| is_decommission_copy_quorum_error(err))
{
return Err(decommission_copy_quorum_deferral(
&& let Some(err) = take_decommission_copy_deferral(
DECOMMISSION_STAGE_MIGRATE_OBJECT,
bucket.as_str(),
version.name.as_str(),
err,
));
&mut error,
)
{
return Err(err);
}
if ignore {
@@ -21803,6 +22085,85 @@ mod tests {
use crate::storage_api_contracts::multipart::MultipartOperations as _;
use serde::Serialize;
fn diagnostic_disk(set_index: i32, disk_index: i32, online: bool) -> rustfs_madmin::Disk {
rustfs_madmin::Disk {
pool_index: 0,
set_index,
disk_index,
state: if online { "ok" } else { "offline" }.to_string(),
runtime_state: Some(if online { "online" } else { "offline" }.to_string()),
capacity_observation_source: Some("live_probe".to_string()),
..Default::default()
}
}
#[test]
fn pool_diagnostic_status_tracks_read_quorum_and_capacity_source() {
let online = vec![
diagnostic_disk(0, 0, true),
diagnostic_disk(0, 1, true),
diagnostic_disk(1, 0, true),
diagnostic_disk(1, 1, true),
];
assert_eq!(
pool_diagnostic_status(&online, 0, 2, 2, 1),
PoolDiagnosticStatus {
availability: PoolAvailability::Online,
capacity_status: PoolCapacityStatus::Live,
}
);
let mut degraded = online.clone();
degraded[0] = diagnostic_disk(0, 0, false);
degraded[1].capacity_observation_source = Some("snapshot".to_string());
assert_eq!(
pool_diagnostic_status(&degraded, 0, 2, 2, 1),
PoolDiagnosticStatus {
availability: PoolAvailability::Degraded,
capacity_status: PoolCapacityStatus::Cached,
}
);
degraded[1] = diagnostic_disk(0, 1, false);
assert_eq!(pool_diagnostic_status(&degraded, 0, 2, 2, 1).availability, PoolAvailability::Offline);
assert_eq!(
pool_diagnostic_status(&[], 0, 2, 2, 1),
PoolDiagnosticStatus {
availability: PoolAvailability::Offline,
capacity_status: PoolCapacityStatus::Unknown,
}
);
assert_eq!(pool_diagnostic_status(&online, 0, 0, 2, 1).availability, PoolAvailability::Unknown);
}
#[tokio::test]
#[serial_test::serial]
async fn admin_pool_diagnostic_snapshot_lists_an_unavailable_nonfirst_pool() {
let (_temp_dirs, store, _other_store) = Box::pin(crate::services::rebalance::test_two_pool_stores(None)).await;
for set in &store.pools[1].disk_set {
let mut disks = set.disks.write().await;
let disk_count = disks.len();
*disks = vec![None; disk_count];
}
let (snapshot, metadata_status) =
tokio::time::timeout(std::time::Duration::from_secs(10), store.admin_pool_diagnostic_snapshot(None))
.await
.expect("pool diagnostics should remain bounded");
assert_eq!(metadata_status, "stale");
assert_eq!(snapshot.len(), 2, "configured topology must remain visible");
assert_eq!(snapshot[0].id, 0);
assert_eq!(snapshot[1].id, 1);
assert_eq!(snapshot[1].diagnostic.availability, PoolAvailability::Offline);
store
.pool_meta_save_gate
.lock()
.await
.ensure_write_safe("ordinary object write")
.expect("a failed diagnostic read must not latch the writer gate");
}
#[test]
fn decommission_set_local_usage_cache_classification_is_exact() {
for object in [
@@ -28296,6 +28657,61 @@ mod tests {
));
}
/// A restarted node's previous incarnation keeps its namespace locks on the
/// peers until their lease expires, so a resumed copy of a key that was in
/// flight at the crash times out acquiring the lock. That copy never
/// committed: it is deferred, never counted as a failed object.
#[test]
fn decommission_copy_lock_timeout_is_deferred_not_failed() {
let timeout = || {
Error::Lock(rustfs_lock::LockError::timeout(
"bucket-a/object-a@latest",
std::time::Duration::from_secs(5),
))
};
let prepare = data_movement::data_movement_stage_error_for_test(
"decommission_object",
"prepare_put_object",
"bucket-a",
"object-a",
timeout(),
);
for err in [timeout(), prepare] {
assert!(is_decommission_copy_lock_timeout_error(&err), "{err}");
assert!(!is_decommission_copy_quorum_error(&err), "{err}");
let mut error = Some(err);
let deferred = take_decommission_copy_deferral("migrate_object", "bucket-a", "object-a", &mut error)
.expect("a lock timeout copy must be deferred");
assert!(error.is_none());
assert!(is_decommission_copy_lock_timeout_deferral(&deferred));
assert!(!is_decommission_copy_quorum_deferral(&deferred));
let failure = data_movement::data_movement_stage_source_as::<DecommissionCopyLockTimeout>(&deferred)
.expect("the deferred error must keep its lock-timeout origin");
assert!(is_decommission_copy_lock_timeout_error(&failure.0));
assert!(matches!(
resolve_decommission_entry_failure(deferred),
Ok(DecommissionEntryOutcome::Deferred(_))
));
}
let mut quorum = Some(Error::ErasureWriteQuorum);
let deferred = take_decommission_copy_deferral("migrate_object", "bucket-a", "object-a", &mut quorum)
.expect("a lost drive quorum copy stays deferred");
assert!(is_decommission_copy_quorum_deferral(&deferred));
for other in [
Error::Lock(rustfs_lock::LockError::internal("lock backend failed")),
Error::other("Lock acquisition timeout for resource 'bucket-a/object-a@latest' after 5s"),
] {
let mut error = Some(other);
assert!(
take_decommission_copy_deferral("migrate_object", "bucket-a", "object-a", &mut error).is_none(),
"{error:?}"
);
assert!(error.is_some(), "a per-object failure keeps its error");
}
}
#[test]
fn decommission_target_capacity_error_accepts_wrapped_capacity_errors() {
let disk_full = Error::other(format!("decommission_object: put_object failed for bucket/object: {}", Error::DiskFull));
+87
View File
@@ -2480,6 +2480,93 @@ mod decommission_lock_order_tests {
});
}
/// A node killed mid-copy leaves its namespace lock on the peers until the
/// lock lease expires, so the resumed decommission times out acquiring the
/// lock of the key it was copying through every in-place retry. The entry
/// is deferred and copied once the lease is gone; it is never counted as a
/// failed object, which would fail the whole decommission.
#[test]
#[serial_test::serial]
fn decommission_defers_a_copy_that_timed_out_on_a_stale_namespace_lock() {
run_large_stack_async_test("decommission-copy-lock-timeout", || async {
let (_temp_dirs, store, _other_store) = test_three_pool_stores_with_isolated_node_contexts(None).await;
let bucket = test_bucket("copy-lock-timeout");
let object = "external/005.bin";
let payload = vec![0x3c; 2048];
store
.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("create the bucket");
put_source_object(&store, &bucket, object, payload.clone()).await;
// The stale lease outlives every in-place attempt of the first
// pass and has expired by the deferred replay.
let stale_lease_calls = crate::core::pools::DECOMMISSION_VERSION_COPY_ATTEMPTS;
let copy_calls = Arc::new(AtomicUsize::new(0));
let _fault = DecommissionTestFaultGuard::install_with_error(
Arc::new({
let copy_calls = Arc::clone(&copy_calls);
let bucket = bucket.clone();
move |stage, entry_bucket, entry_object, _, _| {
stage == "before_target_commit"
&& entry_bucket == bucket
&& entry_object == object
&& copy_calls.fetch_add(1, Ordering::AcqRel) < stale_lease_calls
}
}),
Some(Arc::new(|_, bucket, object| {
// The shape the target write reports when the lock domain
// still holds the previous incarnation's lock.
data_movement::data_movement_stage_error_for_test(
"decommission_object",
"prepare_put_object",
bucket,
object,
crate::error::Error::Lock(rustfs_lock::LockError::timeout(
format!("{bucket}/{object}@latest"),
Duration::from_secs(5),
)),
)
})),
);
start_transitioned_entry_decommission(&store, &bucket).await;
spawn_decommission_routine(&store, 1)
.await
.expect("the decommission routine task should join")
.expect("the decommission routine should return");
assert_eq!(
copy_calls.load(Ordering::Acquire),
stale_lease_calls + 1,
"the first pass exhausts its in-place retries and the replay copies the object once"
);
let local = store.pool_meta.read().await;
let info = local.pools[0]
.decommission
.as_ref()
.expect("the decommission state should remain present");
assert_eq!(
info.items_decommission_failed, 0,
"a lock held until its lease expires is not a permanent failure: {info:?}"
);
assert!(
info.complete && !info.failed && !info.canceled && info.capacity_blocked_reason.is_none(),
"the replayed entry must complete the decommission: {info:?}"
);
drop(local);
assert!(exact_versions(&store, 0, &bucket, object).await.is_empty(), "the source is cleaned up");
let mut reader = store
.get_object_reader(&bucket, object, None, HeaderMap::new(), &ObjectOptions::default())
.await
.expect("the migrated object stays readable");
let mut body = Vec::new();
reader.stream.read_to_end(&mut body).await.expect("read the migrated object");
assert_eq!(body, payload);
});
}
/// Seed `object` in pool 0 with three 2 KiB data versions and a newest
/// delete marker; returns the data versions oldest first.
async fn seed_versioned_entry_with_marker(
+16 -16
View File
@@ -28,7 +28,7 @@ use crate::{
error::{Error, classify_system_path_failure_reason},
object_api::{ObjectInfo, ObjectOptions, PutObjReader},
runtime::sources as runtime_sources,
store::{ECStore, list_objects::list_marker_key},
store::ECStore,
};
pub use local_snapshot::{LocalUsageSnapshot, read_snapshot as read_local_snapshot, snapshot_path};
use rustfs_data_usage::{
@@ -1966,8 +1966,8 @@ fn advance_version_listing_cursor(
let next_version_marker = next_version_marker
.filter(|next_version_marker| !next_version_marker.is_empty())
.ok_or_else(|| Error::other(format!("bucket {bucket} version listing was truncated without a version marker")))?;
let current_key_marker = marker.as_deref().map(list_marker_key);
let next_key_marker = list_marker_key(&next_marker);
let current_key_marker = marker.as_deref();
let next_key_marker = next_marker.as_str();
if current_key_marker == Some(next_key_marker) && version_marker.as_deref() == Some(next_version_marker.as_str()) {
return Err(Error::other(format!(
"bucket {bucket} version listing returned a repeated continuation marker"
@@ -5167,20 +5167,20 @@ mod tests {
);
assert!(ensure_truncated_version_page_has_entries("bucket-a", 0).is_err());
ensure_truncated_version_page_has_entries("bucket-a", 1).expect("a truncated page with an entry can advance");
}
let mut tagged_marker = Some("object-a[rustfs_cache:v2,id:old]".to_string());
let mut tagged_version_marker = Some("version-a".to_string());
assert!(
advance_version_listing_cursor(
"bucket-a",
&mut tagged_marker,
&mut tagged_version_marker,
Some("object-a[rustfs_cache:v2,id:new]".to_string()),
Some("version-a".to_string()),
)
.is_err(),
"different cache tags for the same logical marker must not count as progress"
);
#[test]
fn version_listing_cursor_compares_literal_cache_shaped_keys() {
let mut marker = Some("object-a".to_string());
let mut version_marker = Some("version-a".to_string());
advance_version_listing_cursor(
"bucket-a",
&mut marker,
&mut version_marker,
Some("object-a[rustfs_cache:v2,id:old]".to_string()),
Some("version-a".to_string()),
)
.expect("a distinct literal key advances the cursor");
}
#[test]
+249 -2
View File
@@ -19,6 +19,7 @@ use tracing::Span;
use uuid::Uuid;
const EVENT_OBJECT_NAMESPACE_LOCK: &str = "object_namespace_lock";
const EVENT_OBJECT_NAMESPACE_LOCK_PEER: &str = "object_namespace_lock_peer";
const LOG_COMPONENT_ECSTORE: &str = "ecstore";
const LOG_SUBSYSTEM_OBJECT_LOCK: &str = "object_lock";
@@ -56,6 +57,7 @@ struct AttemptContext {
requested_version_id: Option<String>,
namespace: String,
owner: String,
lock_id: Option<String>,
mode: &'static str,
timeout: Duration,
started_at: Instant,
@@ -85,6 +87,7 @@ impl ObjectLockAttempt {
requested_version_id: requested_version_id.map(str::to_owned),
namespace: lock.namespace().to_owned(),
owner: lock.owner().to_owned(),
lock_id: None,
mode,
timeout,
started_at: Instant::now(),
@@ -105,9 +108,29 @@ impl ObjectLockAttempt {
context.finished = true;
context.trace("disabled");
}
Ok(_) => {
Ok(guard) => {
context.lock_id = guard.lock_id().map(ToString::to_string);
context.acquired_at = Some(Instant::now());
context.trace("acquired");
if let NamespaceLockGuard::Standard(guard) = guard {
for (peer_index, peer_lock_id) in guard.granted_lock_ids().enumerate() {
context.parent.in_scope(|| {
tracing::trace!(
target: "rustfs_ecstore::object_lock_diag",
parent: &context.parent,
event = EVENT_OBJECT_NAMESPACE_LOCK_PEER,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_OBJECT_LOCK,
state = "granted",
lock_attempt_id = %context.id,
lock_id = context.lock_id.as_deref(),
peer_lock_id = %peer_lock_id,
peer_index,
"Object namespace peer owner granted"
);
});
}
}
}
Err(error) => {
context.finished = true;
@@ -171,6 +194,7 @@ impl AttemptContext {
mode = self.mode,
owner = %self.owner,
lock_attempt_id = %self.id,
lock_id = self.lock_id.as_deref(),
acquire_ms = duration_ms(acquired_at.duration_since(self.started_at)),
hold_ms = duration_ms(now.duration_since(acquired_at)),
timeout_ms = duration_ms(self.timeout),
@@ -235,15 +259,238 @@ pub(crate) mod tests {
}
pub(crate) fn rows(&self) -> Vec<serde_json::Value> {
self.all_rows()
.into_iter()
.filter(|row| row["fields"]["event"] == EVENT_OBJECT_NAMESPACE_LOCK)
.collect()
}
fn all_rows(&self) -> Vec<serde_json::Value> {
let bytes = self.0.lock().expect("lock diagnostic capture mutex");
let text = std::str::from_utf8(&bytes).expect("UTF-8 diagnostic events");
text.lines()
.map(|line| serde_json::from_str::<serde_json::Value>(line).expect("JSON diagnostic event"))
.filter(|row| row["fields"]["event"] == EVENT_OBJECT_NAMESPACE_LOCK)
.collect()
}
}
#[test]
#[serial_test::serial]
fn object_lock_diagnostics_join_guard_identity_to_physical_release() {
use rustfs_lock::{
DisabledLockManager, FastObjectLockManager, GlobalLockManager, LocalClient, LockClient, LockRequest, LockType,
NamespaceLock, ObjectKey,
};
temp_env::with_vars([(rustfs_config::ENV_OBJECT_LOCK_DIAG_ENABLE, Some("true"))], || {
let capture = CapturedLockEvents::default();
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("diagnostic release runtime");
tracing::subscriber::with_default(capture.subscriber(), || {
runtime.block_on(async {
let clients = (0..4)
.map(|_| Arc::new(LocalClient::with_manager(Arc::new(GlobalLockManager::new()))) as Arc<dyn LockClient>)
.collect::<Vec<_>>();
let lock = NamespaceLockWrapper::new(
NamespaceLock::with_clients_and_quorum("diagnostics".to_owned(), clients.clone(), 4),
ObjectKey::new("bucket", "object"),
"coordinator".to_owned(),
);
let request = tracing::info_span!("request", request_id = "original-multipart-request");
let mut attempt = request.in_scope(|| {
ObjectLockAttempt::start(
"abort_multipart_upload",
"bucket",
"object",
None,
&lock,
"write",
Duration::from_secs(5),
)
});
let result = lock.get_write_lock(Duration::from_secs(5)).await;
attempt.observe(&result);
let guard = result.expect("four real namespace owners");
let id = guard.lock_id().expect("distributed guard identity").clone();
let peer_ids = match &guard {
NamespaceLockGuard::Standard(guard) => {
assert_eq!(guard.granted_lock_ids().count(), 4);
guard.granted_lock_ids().cloned().collect::<std::collections::HashSet<_>>()
}
NamespaceLockGuard::Fast(_) => panic!("real distributed namespace required"),
};
let rows = capture.rows();
assert_eq!(rows.last().expect("acquired event")["fields"]["lock_id"], id.to_string());
let rows = capture.all_rows();
let mappings = rows
.iter()
.filter(|row| row["fields"]["event"] == EVENT_OBJECT_NAMESPACE_LOCK_PEER)
.collect::<Vec<_>>();
assert_eq!(mappings.len(), 4, "every granted peer must map to its logical guard");
assert!(mappings.iter().all(|row| {
row["fields"]["lock_id"] == id.to_string()
&& peer_ids
.iter()
.any(|peer_id| row["fields"]["peer_lock_id"] == peer_id.to_string())
}));
for client in &clients {
let mut owned = 0;
for peer_id in &peer_ids {
owned += usize::from(
client
.check_status(peer_id)
.await
.expect("actual peer owner status")
.is_some(),
);
}
assert_eq!(owned, 1, "each client must hold exactly one actual granted identity");
}
drop((guard, attempt));
assert!(
capture.all_rows().iter().all(|row| row["fields"]["state"] != "released"),
"enqueue is not release acknowledgement"
);
tokio::time::timeout(Duration::from_secs(5), async {
while capture
.all_rows()
.iter()
.filter(|row| {
row["fields"]["event"] == "namespace_lock_release" && row["fields"]["state"] == "released"
})
.count()
< 4
{
tokio::task::yield_now().await;
}
})
.await
.expect("all physical release callbacks finish");
for client in &clients {
for peer_id in &peer_ids {
assert!(
client
.check_status(peer_id)
.await
.expect("released peer owner status")
.is_none()
);
assert!(!client.release(peer_id).await.expect("duplicate release is absent"));
}
}
let disabled = LocalClient::with_manager(Arc::new(GlobalLockManager::Disabled(DisabledLockManager::new())));
let disabled_request =
LockRequest::new(ObjectKey::new("bucket", "disabled"), LockType::Exclusive, "disabled-owner".to_owned());
assert!(
disabled
.acquire_lock(&disabled_request)
.await
.expect("disabled acquire bypass")
.success
);
assert!(
disabled
.release(&disabled_request.lock_id)
.await
.expect("disabled release bypass")
);
let rows = capture.all_rows();
let releases = rows
.iter()
.filter(|row| row["fields"]["event"] == "namespace_lock_release" && row["fields"]["state"] == "released")
.collect::<Vec<_>>();
assert_eq!(
releases.len(),
4,
"duplicates and disabled guards cannot acknowledge another physical release"
);
let physical_mappings = rows
.iter()
.filter(|row| row["fields"]["event"] == "namespace_lock_release" && row["fields"]["state"] == "mapped")
.collect::<Vec<_>>();
assert_eq!(physical_mappings.len(), 4);
assert!(releases.iter().all(|row| physical_mappings.iter().any(|mapped| {
mapped["fields"]["physical_guard_id"] == row["fields"]["physical_guard_id"]
&& mapped["fields"]["process_id"] == row["fields"]["process_id"]
&& peer_ids
.iter()
.any(|peer_id| mapped["fields"]["lock_id"] == peer_id.to_string())
})));
let dropped = capture.rows().pop().expect("guard drop event");
assert_eq!(dropped["fields"]["lock_id"], id.to_string());
assert!(
dropped["spans"]
.as_array()
.expect("retained request span")
.iter()
.any(|span| span["request_id"] == "original-multipart-request")
);
let manager = Arc::new(FastObjectLockManager::new());
let stale_client = LocalClient::with_manager(Arc::new(GlobalLockManager::Enabled(manager.clone())));
let stale_request = LockRequest::new(
ObjectKey::new("bucket", "externally-released"),
LockType::Exclusive,
"stale-owner".to_owned(),
);
assert!(
stale_client
.acquire_lock(&stale_request)
.await
.expect("real stale owner")
.success
);
assert_eq!(manager.force_unlock(&stale_request.resource), 1);
let mut successor = manager
.acquire_write_lock(stale_request.resource.clone(), "successor-owner")
.await
.expect("real successor guard");
assert!(
stale_client
.release(&stale_request.lock_id)
.await
.expect("retire stale client record")
);
assert_eq!(
manager
.get_lock_info(&stale_request.resource)
.expect("successor remains locked")
.owner
.as_ref(),
"successor-owner"
);
assert_eq!(
capture
.all_rows()
.iter()
.filter(
|row| row["fields"]["event"] == "namespace_lock_release" && row["fields"]["state"] == "released"
)
.count(),
4,
"a stale guard's failed physical release must not produce an ACK"
);
let successor_id = successor.guard_id();
assert!(successor.release(), "the actual successor release succeeds");
assert!(!successor.release(), "a duplicate explicit release cannot complete again");
drop(successor);
assert_eq!(
capture
.all_rows()
.iter()
.filter(|row| row["fields"]["event"] == "namespace_lock_release"
&& row["fields"]["state"] == "released"
&& row["fields"]["physical_guard_id"] == successor_id)
.count(),
1,
"explicit release followed by Drop records one physical completion"
);
});
});
});
}
fn namespace() -> NamespaceLockWrapper {
let manager = rustfs_lock::GlobalLockManager::Enabled(Arc::new(rustfs_lock::fast_lock::FastObjectLockManager::new()));
NamespaceLockWrapper::new(
+281 -186
View File
@@ -7762,10 +7762,6 @@ impl LocalDisk {
// }
let multipart_check = self.multipart_delete_check(delete_path, true)?;
if let Some(check) = &multipart_check {
self.validate_multipart_delete_admission(check.clone(), namespace_owner.clone())
.await?;
}
let err = if let Some(check) = &multipart_check {
os::rename_all_with_precondition_and_parent_sync(
delete_path,
@@ -7842,13 +7838,15 @@ impl LocalDisk {
namespace_owner: Option<Arc<dyn Send + Sync>>,
) -> Result<()> {
multipart_staging::reject_native_control_write(&self.root, self.io_root(), volume, path)?;
let source_owner = self.acquire_multipart_delete_owner(volume, path, namespace_owner).await?;
let mut source_owner = self.acquire_multipart_delete_owner(volume, path, namespace_owner).await?;
let namespace_owner = source_owner.namespace_owner();
if volume == RUSTFS_META_MULTIPART_BUCKET
&& let Some(check) = self.multipart_delete_check(&self.io_get_object_path(volume, path)?, opt.recursive)?
&& (opt.orphan_data_dir || (opt.recursive && opt.immediate))
{
self.validate_multipart_delete_admission(check, namespace_owner.clone())
.await?;
source_owner.mark_validated();
}
if pool_meta_native::reject_raw_path(&self.root, volume, path).is_err() {
let owner = self.pool_meta_cleanup_owner(volume, path, &opt, namespace_owner).await?;
@@ -8066,7 +8064,10 @@ impl LocalDisk {
let file_path = self.io_get_object_path(volume, path)?;
check_path_length(file_path.to_string_lossy().as_ref())?;
let multipart_check = self.multipart_delete_check(&file_path, opt.recursive)?;
if let Some(check) = &multipart_check {
if !source_owner.is_validated()
&& (opt.orphan_data_dir || opt.old_data_dir.is_some() || !opt.recursive)
&& let Some(check) = &multipart_check
{
self.validate_multipart_delete_admission(check.clone(), namespace_owner.clone())
.await?;
}
@@ -8313,8 +8314,6 @@ impl LocalDisk {
// debug!("delete_file remove_dir done {:?}", &delete_path);
} else if let Err(err) = {
if let Some(check) = self.multipart_delete_check(delete_path, false)? {
self.validate_multipart_delete_admission(check.clone(), namespace_owner.clone())
.await?;
let path = delete_path.clone();
let lease = os::acquire_namespace_mutation_lease_with_owner(delete_path, namespace_owner.clone()).await;
os::run_blocking_namespace_operation(lease, move || {
@@ -8327,6 +8326,10 @@ impl LocalDisk {
}
} {
// debug!("remove_file err {:?} when {:?}", &err, &delete_path);
let err = match err.downcast::<DiskError>() {
Ok(disk_err) => return Err(disk_err),
Err(err) => err,
};
match err.kind() {
ErrorKind::NotFound => (),
_ => {
@@ -10009,9 +10012,7 @@ impl LocalDisk {
} {
meta.name.push_str(SLASH_SEPARATOR);
// Conservative listings verify physical prefixes. Never-versioned
// buckets use the bounded fast path, which only has to rule out
// the data dirs a deleted version leaves behind; an empty listing
// of such a prefix then reclaims committed residue.
// buckets first rule out the data dirs a deleted version leaves behind.
let listable = if opts.recursive || opts.incl_deleted {
true
} else if opts.skip_hidden_prefix_check {
@@ -10165,17 +10166,19 @@ impl LocalDisk {
/// Reads stay small on genuine prefixes: each directory is sampled with one
/// bounded batch, the sampled children are probed depth-first, and the
/// directory is only read in full once every sampled child proved to hold
/// nothing listable. A total read budget makes a residue tree larger than
/// the budget surface and be hidden one level down instead, so the cost on
/// a genuine prefix never exceeds its depth in small directory reads. A
/// directory that vanishes mid-probe holds nothing listable.
/// nothing listable. If the read budget is exhausted, verify the subtree
/// before exposing the prefix. A directory that vanishes mid-probe holds
/// nothing listable.
async fn directory_is_delete_residue(&self, bucket: &str, dir_name: &str, stall: Option<Duration>) -> Result<bool> {
let dir_name = dir_name.trim_end_matches(SLASH_SEPARATOR);
let mut reads = 0usize;
let mut pending = vec![ResidueProbeStep::Sample(dir_name.to_owned())];
while let Some(step) = pending.pop() {
if reads >= DELETE_RESIDUE_PROBE_READ_BUDGET {
return Ok(false);
return self
.directory_has_listing_entry(bucket, dir_name, false, stall)
.await
.map(|listable| !listable);
}
reads += 1;
@@ -12016,20 +12019,16 @@ impl DiskAPI for LocalDisk {
// Publish the immutable generation before preparing the part switch.
// Rollback retains the old generation; an unreferenced new index is
// reclaimed with the upload directory if preparation is interrupted.
os::rename_all_with_precondition(
os::rename_all_with_precondition_and_parent_sync(
&source,
&destination,
&dst_volume_dir,
&self.publication_root,
source_owner.clone(),
admission.as_ref().map(|check| check.precondition()),
durability.syncs_commit_metadata(),
)
.await?;
if durability.syncs_commit_metadata() {
os::fsync_dir_with_owner(destination.parent().ok_or(DiskError::FileCorrupt)?, source_owner.clone())
.await
.map_err(to_file_error)?;
}
}
let retained_owner = source_owner.clone();
tokio::task::spawn_blocking(move || {
@@ -12157,17 +12156,16 @@ impl DiskAPI for LocalDisk {
PartTransactionAction::Commit => (PART_TRANSACTION_NEW_META, PART_TRANSACTION_OLD_META),
PartTransactionAction::Rollback => (PART_TRANSACTION_OLD_META, PART_TRANSACTION_NEW_META),
};
if let (Ok(current), Ok(retained), Ok(obsolete)) = (
std::fs::read(&current_meta_path),
std::fs::read(transaction_path.join(retained_name)),
std::fs::read(transaction_path.join(obsolete_name)),
) && current == retained
if let Ok(obsolete) = std::fs::read(transaction_path.join(obsolete_name))
&& let Ok(obsolete) = ObjectPartInfo::unmarshal(&obsolete)
&& let Some(integrity) = obsolete.integrity
&& integrity.validate().is_ok()
&& usize::try_from(integrity.number).ok() == Some(obsolete.number)
&& current_data_path.file_name().and_then(|name| name.to_str())
== Some(format!("part.{}", obsolete.number).as_str())
&& let (Ok(current), Ok(retained)) =
(std::fs::read(&current_meta_path), std::fs::read(transaction_path.join(retained_name)))
&& current == retained
&& ObjectPartInfo::unmarshal(&retained)
.ok()
.and_then(|part| part.integrity)
@@ -12208,16 +12206,6 @@ impl DiskAPI for LocalDisk {
let source_owner = self.multipart_source_owner(dst_volume, dst_path, true, false, None).await?;
multipart_staging::reject_reserved_anchor(dst_path)?;
let admission = self.multipart_source_admission(dst_volume, dst_path, true, false)?;
if let Some(check) = admission.clone() {
let retained = source_owner.clone();
tokio::task::spawn_blocking(move || {
let _owner = retained;
check.validate()
})
.await
.map_err(DiskError::from)?
.map_err(to_file_error)?;
}
pool_meta_native::reject_raw_path(&self.root, src_volume, src_path)?;
pool_meta_native::reject_raw_path(&self.root, dst_volume, dst_path)?;
let src_volume_dir = self.io_get_bucket_path(src_volume)?;
@@ -12253,158 +12241,186 @@ impl DiskAPI for LocalDisk {
check_path_length(src_file_path.to_string_lossy().as_ref())?;
check_path_length(dst_file_path.to_string_lossy().as_ref())?;
if src_is_dir {
let meta_op = match lstat_std(&src_file_path).map_err(|e| to_file_error(e).into()) {
Ok(meta) => Some(meta),
Err(e) => {
return Err(e);
}
};
if let Some(meta) = meta_op
&& !meta.is_dir()
{
warn!(
event = EVENT_DISK_LOCAL_RENAME_REJECTED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
reason = "src_expected_dir_missing",
path = ?src_file_path,
"Disk local rename rejected"
);
return Err(DiskError::FileAccessDenied);
}
// Clear any stale destination before the directory rename. An absent
// destination is the normal case when renaming a directory to a new
// location, so tolerate NotFound instead of aborting the whole rename
// (MinIO's RenameFile ignores osIsNotExist here).
if let Err(e) = remove_std(&dst_file_path)
&& e.kind() != ErrorKind::NotFound
{
return Err(to_file_error(e).into());
}
} else {
let meta = lstat_std(&src_file_path).map_err(|e| -> DiskError { to_file_error(e).into() })?;
if meta.is_dir() {
warn!(
event = EVENT_DISK_LOCAL_RENAME_REJECTED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
reason = "src_unexpected_dir",
path = ?src_file_path,
"Disk local rename rejected"
);
return Err(DiskError::FileAccessDenied);
}
}
let transaction_publish_meta = if src_is_dir {
let dst_meta_path = if src_is_dir {
None
} else {
let transaction_path = self.io_get_object_path(dst_volume, &crate::disk::part_transaction_path(dst_path))?;
let transaction_meta_path = transaction_path.join(PART_TRANSACTION_NEW_META);
match fs::read(&transaction_meta_path).await {
Ok(expected_meta) => {
if expected_meta.as_slice() != meta.as_ref() {
return Err(DiskError::FileCorrupt);
}
let publish_meta_path = transaction_path.join(PART_TRANSACTION_PUBLISH_META);
let source_meta_path = transaction_meta_path.clone();
let publish_path = publish_meta_path.clone();
let retained_owner = source_owner.clone();
tokio::task::spawn_blocking(move || {
let _owner = retained_owner;
remove_file_if_exists(&publish_path)?;
std::fs::hard_link(source_meta_path, &publish_path)
})
.await
.map_err(DiskError::from)?
.map_err(to_file_error)?;
Some(publish_meta_path)
}
// Old peers know only RenamePart. The new coordinator never
// reaches rename unless prepare succeeded, so an absent
// transaction directory identifies the rolling-upgrade legacy
// path. A present directory without new.meta is corruption.
Err(err) if err.kind() == ErrorKind::NotFound => match fs::metadata(&transaction_path).await {
Ok(_) => return Err(DiskError::FileCorrupt),
Err(meta_err) if meta_err.kind() == ErrorKind::NotFound => None,
Err(meta_err) => return Err(to_file_error(meta_err).into()),
},
Err(err) => return Err(to_file_error(err).into()),
}
let path = self.io_get_object_path(dst_volume, &format!("{dst_path}.meta"))?;
check_path_length(path.to_string_lossy().as_ref())?;
Some(path)
};
let transaction_path = if src_is_dir {
None
} else {
Some(self.io_get_object_path(dst_volume, &crate::disk::part_transaction_path(dst_path))?)
};
// UploadPart is acknowledged once this rename lands, so the part data and
// its directory entry must be durable before we return. Relaxed keeps the
// part payload fdatasync but leaves the directory entry to the page cache.
let durability = effective_durability(dst_volume);
if durability.syncs_data_shards() && !src_is_dir {
let src = src_file_path.clone();
let retained_owner = source_owner.clone();
tokio::task::spawn_blocking(move || {
let _owner = retained_owner;
os::sync_file(&src)
let sync = (durability.syncs_data_shards() && !src_is_dir) || durability.syncs_commit_metadata();
let publication_root = self.publication_root.clone();
let src_file_for_cleanup = src_file_path.clone();
let src_volume_for_cleanup = src_volume_dir.clone();
let meta_for_legacy = meta.clone();
let lease = os::acquire_namespace_mutation_lease_with_owner(&dst_file_path, source_owner.clone()).await;
let published_transaction_meta =
os::run_blocking_namespace_operation_with_sync(lease, sync, move || -> std::io::Result<bool> {
if let Some(check) = &admission {
check.validate().map_err(to_file_error)?;
}
if src_is_dir {
let meta_op = lstat_std(&src_file_path).map_err(to_file_error)?;
if !meta_op.is_dir() {
warn!(
event = EVENT_DISK_LOCAL_RENAME_REJECTED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
reason = "src_expected_dir_missing",
path = ?src_file_path,
"Disk local rename rejected"
);
return Err(DiskError::FileAccessDenied.into());
}
// Clear any stale destination before the directory rename. An absent
// destination is the normal case when renaming a directory to a new
// location, so tolerate NotFound instead of aborting the whole rename
// (MinIO's RenameFile ignores osIsNotExist here).
if let Err(e) = remove_std(&dst_file_path)
&& e.kind() != ErrorKind::NotFound
{
return Err(to_file_error(e));
}
} else {
let meta_op = lstat_std(&src_file_path).map_err(to_file_error)?;
if meta_op.is_dir() {
warn!(
event = EVENT_DISK_LOCAL_RENAME_REJECTED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
reason = "src_unexpected_dir",
path = ?src_file_path,
"Disk local rename rejected"
);
return Err(DiskError::FileAccessDenied.into());
}
}
let transaction_publish_meta = if let Some(transaction_path) = transaction_path.as_ref() {
let transaction_meta_path = transaction_path.join(PART_TRANSACTION_NEW_META);
match std::fs::File::open(&transaction_meta_path) {
Ok(mut file) => {
let file_len = file.metadata().map_err(to_file_error)?.len();
if file_len != meta.len() as u64 {
return Err(DiskError::FileCorrupt.into());
}
let mut expected_meta = Vec::with_capacity(meta.len());
std::io::Read::read_to_end(&mut std::io::Read::take(&mut file, file_len), &mut expected_meta)
.map_err(to_file_error)?;
drop(file);
if expected_meta.as_slice() != meta.as_ref() {
return Err(DiskError::FileCorrupt.into());
}
let publish_meta_path = transaction_path.join(PART_TRANSACTION_PUBLISH_META);
remove_file_if_exists(&publish_meta_path).map_err(to_file_error)?;
std::fs::hard_link(&transaction_meta_path, &publish_meta_path).map_err(to_file_error)?;
Some(publish_meta_path)
}
// Old peers know only RenamePart. The new coordinator never
// reaches rename unless prepare succeeded, so an absent
// transaction directory identifies the rolling-upgrade legacy
// path. A present directory without new.meta is corruption.
Err(err) if err.kind() == ErrorKind::NotFound => match std::fs::metadata(transaction_path) {
Ok(_) => return Err(DiskError::FileCorrupt.into()),
Err(meta_err) if meta_err.kind() == ErrorKind::NotFound => None,
Err(meta_err) => return Err(to_file_error(meta_err)),
},
Err(err) => return Err(to_file_error(err)),
}
} else {
None
};
if durability.syncs_data_shards() && !src_is_dir {
os::sync_file(&src_file_path).map_err(to_file_error)?;
}
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
let precondition = admission.as_ref().map(|check| check.precondition());
#[cfg(not(all(any(test, feature = "test-util"), not(windows))))]
let precondition: Option<Arc<dyn Fn() -> std::io::Result<()> + Send + Sync>> = None;
os::rename_all_with_precondition_and_parent_sync_std(
&src_file_path,
&dst_file_path,
&dst_volume_dir,
&publication_root,
precondition.as_deref(),
durability.syncs_commit_metadata() && transaction_publish_meta.is_none(),
)
.map_err(to_file_error)?;
let dst_meta = lstat_std(&dst_file_path).map_err(to_file_error)?;
if src_is_dir != dst_meta.is_dir() {
warn!(
event = EVENT_DISK_LOCAL_RENAME_REJECTED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
reason = "dst_type_changed_after_rename",
path = ?dst_file_path,
"Disk local rename rejected"
);
return Err(DiskError::FileAccessDenied.into());
}
if let Some(transaction_publish_meta) = transaction_publish_meta {
let dst_meta_path = dst_meta_path.as_ref().ok_or(DiskError::InvalidPath)?;
os::rename_all_with_precondition_and_parent_sync_std(
&transaction_publish_meta,
dst_meta_path,
&dst_volume_dir,
&publication_root,
precondition.as_deref(),
durability.syncs_commit_metadata(),
)
.map_err(to_file_error)?;
for dir in src_file_path.ancestors().skip(1) {
if dir == src_volume_dir.as_path() || !dir.starts_with(&src_volume_dir) || is_root_path(dir) {
break;
}
match std::fs::remove_dir(dir) {
Ok(()) => {}
Err(err) if is_dir_not_empty_error(&err) || is_benign_object_rmdir_error(&err) => break,
Err(err) => {
return Err(Error::other(FileAccessDeniedWithContext {
path: dir.to_path_buf(),
source: err,
})
.into());
}
}
}
Ok(true)
} else {
Ok(false)
}
})
.await
.map_err(DiskError::from)?
.map_err(to_file_error)?;
}
.map_err(DiskError::from)?;
os::rename_all_with_precondition(
&src_file_path,
&dst_file_path,
&dst_volume_dir,
&self.publication_root,
source_owner.clone(),
admission.as_ref().map(|check| check.precondition()),
)
.await?;
if durability.syncs_commit_metadata()
&& let Some(parent) = dst_file_path.parent()
{
os::fsync_dir_with_owner(parent, source_owner.clone())
.await
.map_err(to_file_error)?;
}
let dst_meta = lstat_std(&dst_file_path).map_err(|e| -> DiskError { to_file_error(e).into() })?;
if src_is_dir != dst_meta.is_dir() {
warn!(
event = EVENT_DISK_LOCAL_RENAME_REJECTED,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_DISK_LOCAL,
reason = "dst_type_changed_after_rename",
path = ?dst_file_path,
"Disk local rename rejected"
);
return Err(DiskError::FileAccessDenied);
}
if let Some(transaction_publish_meta) = transaction_publish_meta {
let dst_meta_path = self.io_get_object_path(dst_volume, &format!("{dst_path}.meta"))?;
os::rename_all_with_precondition(
&transaction_publish_meta,
&dst_meta_path,
&dst_volume_dir,
&self.publication_root,
if !published_transaction_meta {
self.write_all_meta_with_namespace_owner(
dst_volume,
&format!("{dst_path}.meta"),
&meta_for_legacy,
true,
source_owner.clone(),
admission.as_ref().map(|check| check.precondition()),
)
.await?;
if durability.syncs_commit_metadata()
&& let Some(parent) = dst_meta_path.parent()
{
os::fsync_dir_with_owner(parent, source_owner.clone())
.await
.map_err(to_file_error)?;
}
} else {
self.write_all_meta_with_namespace_owner(dst_volume, &format!("{dst_path}.meta"), &meta, true, source_owner.clone())
.await?;
}
if src_is_dir {
@@ -12418,8 +12434,9 @@ impl DiskAPI for LocalDisk {
.await;
}
if let Some(parent) = src_file_path.parent() {
self.delete_file(&src_volume_dir, &parent.to_path_buf(), false, false).await?;
if !published_transaction_meta && let Some(parent) = src_file_for_cleanup.parent() {
self.delete_file(&src_volume_for_cleanup, &parent.to_path_buf(), false, false)
.await?;
}
Ok(())
@@ -12895,12 +12912,26 @@ impl DiskAPI for LocalDisk {
.map_err(|e| to_access_error(e, DiskError::VolumeAccessDenied))?;
}
let mut cached_owner: Option<(String, Option<Arc<dyn Send + Sync>>)> = None;
for path in paths.iter() {
let file_path = self.io_get_object_path(volume, path)?;
check_path_length(file_path.to_string_lossy().as_ref())?;
let owner = self.multipart_source_owner(volume, path, true, true, None).await?;
let owner = if volume == RUSTFS_META_MULTIPART_BUCKET {
let scope_key = multipart_staging::upload_scope_root(path, true).to_ascii_lowercase();
if let Some((cached_key, cached)) = &cached_owner
&& cached_key == &scope_key
{
cached.clone()
} else {
let acquired = self.multipart_source_owner(volume, path, true, true, None).await?;
cached_owner = Some((scope_key, acquired.clone()));
acquired
}
} else {
self.multipart_source_owner(volume, path, true, true, None).await?
};
self.move_to_trash_with_namespace_owner(&file_path, false, false, owner)
.await?;
@@ -25245,11 +25276,13 @@ mod test {
)
.await;
// A prefix with more residue directories than the probe budget stays
// visible rather than costing an unbounded walk.
// A prefix larger than the fast-probe budget still needs a complete
// read-only check before it can appear in a delimiter listing.
for hour in 0..(DELETE_RESIDUE_PROBE_READ_BUDGET + 1) {
write_residue(&bucket_dir.join(format!("bulk/2026/08/28/{hour:02}/a.parquet")), false).await;
write_residue(&bucket_dir.join(format!("bulk-live/2026/08/28/{hour:02}/a.parquet")), false).await;
}
write_object(&bucket_dir.join("bulk-live/live.parquet"), "bulk-live/live.parquet").await;
// Exactly at the budget: `edge/` with N object dirs costs one sampled
// read of `edge`, two reads per object dir (its own and its data dir),
// and one remainder read of `edge` once its 8-entry sample was full.
@@ -25330,10 +25363,8 @@ mod test {
vec!["logs/default/2026/08/28/23/".to_owned()]
);
assert_eq!(scan_names(&disk, bucket, "logs/").await, vec!["logs/default/".to_owned()]);
assert_eq!(scan_names(&disk, bucket, "bulk/2026/08/").await, vec!["bulk/2026/08/28/".to_owned()]);
// Listed directly, every object dir is probed on its own and hidden;
// the budget only bites where the whole tree hangs off one entry, so
// the root listing below shows `edge-show/` but not `edge-hide/`.
assert_eq!(scan_names(&disk, bucket, "bulk/2026/08/").await, Vec::<String>::new());
assert_eq!(scan_names(&disk, bucket, "bulk-live/").await, vec!["bulk-live/live.parquet".to_owned()]);
assert_eq!(scan_names(&disk, bucket, "edge-hide/").await, Vec::<String>::new());
assert_eq!(scan_names(&disk, bucket, "edge-show/").await, Vec::<String>::new());
assert_eq!(scan_names(&disk, bucket, "stray/").await, vec!["stray/2026/".to_owned()]);
@@ -25341,8 +25372,7 @@ mod test {
assert_eq!(
scan_names(&disk, bucket, "").await,
vec![
"bulk/".to_owned(),
"edge-show/".to_owned(),
"bulk-live/".to_owned(),
"logs/".to_owned(),
"odd/".to_owned(),
"stray/".to_owned(),
@@ -26504,6 +26534,71 @@ mod test {
}
}
#[tokio::test]
async fn rename_part_leaves_metadata_reads_on_foreground_lane() {
let dir = tempfile::tempdir().expect("disk root");
let endpoint = Endpoint::try_from(dir.path().to_string_lossy().as_ref()).expect("endpoint");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk");
ensure_test_volume(&disk, "tmp").await;
ensure_test_volume(&disk, "bucket").await;
disk.write_metadata(
"",
"bucket",
"unrelated/object",
test_file_info("unrelated/object", Uuid::new_v4(), Some(Uuid::new_v4()), None),
)
.await
.expect("write unrelated object metadata");
let payload = Bytes::from_static(b"part payload");
let meta = Bytes::from(
ObjectPartInfo {
number: 1,
size: payload.len(),
..Default::default()
}
.marshal_msg()
.expect("part metadata"),
);
disk.write_all("tmp", "upload/part.1", payload.clone())
.await
.expect("stage part");
disk.prepare_part_transaction("tmp", "upload/part.1", "bucket", "object/part.1", meta.clone())
.await
.expect("prepare part transaction");
let (started_tx, started_rx) = tokio::sync::oneshot::channel();
let (release_tx, release_rx) = tokio::sync::oneshot::channel();
let queue = Arc::clone(&disk.blocking_queue);
let blocker = tokio::spawn(async move {
queue
.run_fsync_cancellable(move |_| {
let _ = started_tx.send(());
let _ = release_rx.blocking_recv();
Ok(())
})
.await
});
started_rx.await.expect("fsync lane blocked");
let mut rename = std::pin::pin!(disk.rename_part("tmp", "upload/part.1", "bucket", "object/part.1", meta.clone()));
assert!(futures::FutureExt::now_or_never(&mut rename).is_none());
let read = tokio::time::timeout(
Duration::from_secs(5),
disk.read_version("", "bucket", "unrelated/object", "", &ReadOptions::default()),
)
.await
.expect("read_version on unrelated key must not be blocked by rename_part sync")
.expect("unrelated object metadata should be readable");
assert_eq!(read.name, "unrelated/object");
release_tx.send(()).expect("release fsync lane");
blocker.await.expect("blocker joined").expect("blocker completed");
rename.await.expect("rename_part completed");
}
#[tokio::test]
async fn metadata_read_job_retains_resource_unit_until_native_completion() {
use crate::disk::with_background_resource_unit;
@@ -6,6 +6,7 @@ use super::{
LocalDisk,
multipart_source_state::{
LoadedSource, SourceHead, SourceState, SourceStore, decode, encode, load_source, raw_sha, read_optional,
read_optional_in_pinned,
},
};
use crate::disk::{
@@ -127,20 +128,6 @@ pub(super) fn validate_retired_head(
}
impl SourceStore {
pub async fn root_missing(&self) -> Result<bool> {
let source = self.source.clone();
let base = self.base.clone();
let root = self.publication_root.clone();
os::run_blocking_namespace_operation(self.lease.clone(), move || {
match os::pin_multipart_source_directory(&source, &base, &root) {
Ok(_) => Ok(false),
Err(error) if error.kind() == ErrorKind::NotFound => Ok(true),
Err(error) => Err(error),
}
})
.await
.map_err(DiskError::from)
}
fn verify_abort_cohort(&self, certificate: &MultipartAbortCertificate) -> Result<()> {
certificate.verify_with_authority(&self.authority)?;
if certificate.disk_ids != self.disk_ids
@@ -265,7 +252,7 @@ impl SourceStore {
let certificate = self.read_abort_certificate(&loaded).await?;
Ok(MultipartSourceInspection {
disk_id: self.disk_id,
root_missing: self.root_missing().await?,
root_missing: loaded.root_missing,
abort_round: loaded
.state
.as_ref()
@@ -561,11 +548,18 @@ impl AbortDeletePermit {
pub(super) struct MultipartDeleteOwner {
owner: Option<Arc<dyn Send + Sync>>,
source_key: Option<PathBuf>,
admission_validated: bool,
}
impl MultipartDeleteOwner {
pub(super) fn namespace_owner(&self) -> Option<Arc<dyn Send + Sync>> {
self.owner.clone()
}
pub(super) fn mark_validated(&mut self) {
self.admission_validated = true;
}
pub(super) fn is_validated(&self) -> bool {
self.admission_validated
}
pub(super) fn chain(self, owner: Option<Arc<dyn Send + Sync>>) -> Self {
let owner = match (self.owner, owner) {
(None, owner) | (owner, None) => owner,
@@ -575,6 +569,7 @@ impl MultipartDeleteOwner {
Self {
owner,
source_key: self.source_key,
admission_validated: self.admission_validated,
}
}
}
@@ -595,7 +590,11 @@ impl LocalDisk {
) -> Result<MultipartDeleteOwner> {
let source_key = self.multipart_delete_owner_key(volume, path)?;
let owner = self.multipart_source_owner(volume, path, true, true, owner).await?;
Ok(MultipartDeleteOwner { owner, source_key })
Ok(MultipartDeleteOwner {
owner,
source_key,
admission_validated: false,
})
}
pub(super) fn validate_multipart_delete_owner(
&self,
@@ -620,12 +619,17 @@ pub(super) struct OrdinaryMultipartDeleteCheck {
}
impl OrdinaryMultipartDeleteCheck {
pub(super) fn validate(&self) -> Result<()> {
if let Some(raw) = read_optional(&self.source.join(STATE), &self.base, &self.root)? {
let pinned = match os::pin_multipart_source_directory(&self.source, &self.base, &self.root) {
Ok(pinned) => pinned,
Err(error) if error.kind() == ErrorKind::NotFound => return Ok(()),
Err(error) => return Err(error.into()),
};
if let Some(raw) = read_optional_in_pinned(&pinned, &self.source, STATE, &self.base, &self.root)? {
let state: SourceState = decode(&raw)?;
if state.abort.is_some() || state.phase == MultipartSourcePhase::Aborting {
return Err(DiskError::ConditionalDeleteConflict);
}
} else if os::multipart_source_has_native_controls(&self.source, &self.base, &self.root)? {
} else if os::multipart_source_has_native_controls_in_pinned(&self.source, &pinned)? {
return Err(DiskError::ConditionalDeleteConflict);
}
Ok(())
@@ -170,7 +170,7 @@ impl LocalDisk {
recovery: loaded.state.as_ref().and_then(|state| state.recovery.clone()),
negative_certificate,
ready_certificate,
root_missing: store.root_missing().await?,
root_missing: loaded.root_missing,
abort_round: loaded
.state
.as_ref()
@@ -552,25 +552,33 @@ impl LocalDisk {
let target = self.get_object_path(&publication.lineage.bucket, &publication.lineage.object)?;
let _target_owner = os::acquire_metadata_mutation_lease(&target, store.owner.clone()).await;
if require_target {
let raw = store
.read_target(publication)
.await?
.ok_or(DiskError::ConditionalDeleteConflict)?;
if !store.target_matches(publication, &raw)? {
return Err(DiskError::ConditionalDeleteConflict);
}
store.sync_target(publication).await?;
let expected_publication = publication.clone();
let physical_slot = store.physical_slot;
let disk_count = store.disk_ids.len();
store
.read_and_sync_target(publication, move |raw| {
if !super::multipart_source_state::SourceStore::target_matches_for_slot(
&expected_publication,
raw,
physical_slot,
disk_count,
)? {
return Err(DiskError::ConditionalDeleteConflict);
}
Ok(())
})
.await?;
}
if !matches!(
let snapshot = if !matches!(
state.phase,
MultipartSourcePhase::LocalTerminalSeal | MultipartSourcePhase::GlobalTerminalCertified
) {
state.capability_sha256 = None;
state.advance(MultipartSourcePhase::LocalTerminalSeal)?;
store.persist(&state).await?;
}
let current = store.load().await?;
let snapshot = current.snapshot.ok_or(DiskError::ConditionalDeleteConflict)?;
store.persist_with_source_sync(&state, false).await?
} else {
loaded.snapshot.ok_or(DiskError::ConditionalDeleteConflict)?
};
if snapshot.head_sha256.is_some()
|| snapshot.capability_sha256.is_some()
|| snapshot.publication.as_ref() != Some(publication)
@@ -106,6 +106,7 @@ impl SourceHead {
}
pub(super) struct LoadedSource {
pub root_missing: bool,
pub state: Option<SourceState>,
pub head: Option<SourceHead>,
pub original: Option<Bytes>,
@@ -224,11 +225,31 @@ pub(super) fn read_optional(path: &Path, base: &Path, root: &os::PublicationRoot
Err(error) => Err(error.into()),
}
}
pub(super) fn read_optional_in_pinned(
pinned: &os::ExistingBaseDirectoryGuard,
parent: &Path,
name: &str,
base: &Path,
root: &os::PublicationRoot,
) -> Result<Option<Bytes>> {
match os::read_multipart_source_metadata_in_pinned(pinned, parent, std::ffi::OsStr::new(name), base, root) {
Ok(bytes) => Ok(Some(Bytes::from(bytes))),
Err(error) if error.kind() == std::io::ErrorKind::NotFound => Ok(None),
Err(error) => Err(error.into()),
}
}
/// Canonical source mutations are admitted under SourceRW at the syscall,
/// not on the earlier async caller that may have timed out.
pub(super) fn validate_source_mutation(source: &Path, base: &Path, root: &os::PublicationRoot, initial: bool) -> Result<()> {
let state = read_optional(&source.join(STATE), base, root)?;
let head = read_optional(&source.join(STORAGE_FORMAT_FILE), base, root)?;
let pinned = match os::pin_multipart_source_directory(source, base, root) {
Ok(pinned) => pinned,
Err(error) if error.kind() == std::io::ErrorKind::NotFound => {
return if initial { Ok(()) } else { Err(DiskError::FileNotFound) };
}
Err(error) => return Err(error.into()),
};
let state = read_optional_in_pinned(&pinned, source, STATE, base, root)?;
let head = read_optional_in_pinned(&pinned, source, STORAGE_FORMAT_FILE, base, root)?;
let Some(head) = head else {
return if initial && state.is_none() {
Ok(())
@@ -314,6 +335,48 @@ fn target_backup_path(source: &Path, owner: Uuid) -> PathBuf {
source.join(".mpu-completions").join(owner.to_string()).join("target.xl.meta")
}
fn terminal_record_path(source: &Path, state: &SourceState) -> Option<PathBuf> {
matches!(
state.phase,
MultipartSourcePhase::LocalTerminalSeal | MultipartSourcePhase::GlobalTerminalCertified
)
.then(|| {
source
.join(".mpu-completions")
.join(state.backup_owner.to_string())
.join(format!("terminal-{}.msgpack", state.revision))
})
}
fn snapshot_for_state(
disk_id: Uuid,
state: &SourceState,
state_raw: &[u8],
head_sha256: Option<[u8; 32]>,
) -> Result<MultipartSourceSnapshot> {
let state_sha256 = raw_sha(state_raw);
let inventory = raw_sha(&encode(&(
Some(state_sha256),
head_sha256,
state.original_sha256,
state.negative_certificate_sha256,
state.ready_certificate_sha256,
))?);
Ok(MultipartSourceSnapshot {
disk_id,
physical_slot: state.physical_slot,
lineage: state.lineage.clone(),
generation: state.generation,
revision: state.revision,
phase: state.phase,
state_sha256,
head_sha256,
inventory_sha256: inventory,
capability_sha256: state.capability_sha256,
publication: state.publication.clone(),
})
}
impl SourceStore {
pub fn validate_certificate_topology(&self, certificate: &MultipartCertificate) -> Result<()> {
if certificate.disk_ids != self.disk_ids || certificate.disk_ids.get(self.physical_slot) != Some(&self.disk_id) {
@@ -348,31 +411,72 @@ impl SourceStore {
.map_err(DiskError::from)?
}
pub async fn persist(&self, state: &SourceState) -> Result<()> {
self.persist_with_source_sync(state, false).await.map(|_| ())
}
pub(super) async fn persist_with_source_sync(
&self,
state: &SourceState,
sync_source_before: bool,
) -> Result<MultipartSourceSnapshot> {
state.validate(self.disk_id, &self.path)?;
let raw = encode(state)?;
let snapshot = snapshot_for_state(self.disk_id, state, &raw, None)?;
let source = self.source.clone();
let base = self.base.clone();
let root = self.publication_root.clone();
let terminal = terminal_record_path(&source, state);
os::run_blocking_namespace_operation_with_sync(self.lease.clone(), true, move || {
Self::persist_state_std(&source, &base, &root, &raw, terminal.as_deref(), sync_source_before)
})
.await
.map_err(DiskError::from)?;
Ok(snapshot)
}
fn persist_state_std(
source: &Path,
base: &Path,
root: &os::PublicationRoot,
raw: &[u8],
terminal: Option<&Path>,
sync_source_before: bool,
) -> std::io::Result<()> {
#[cfg(unix)]
let mut pinned = os::pin_multipart_source_directory(source, base, root)?;
#[cfg(not(unix))]
let _pinned = os::pin_multipart_source_directory(source, base, root)?;
if sync_source_before {
#[cfg(unix)]
{
let opened = std::fs::File::from(pinned.pop().ok_or_else(|| IoError::other("empty source directory guard"))?);
os::fsync_directory_handle_std(source, Some(&opened))?;
}
#[cfg(not(unix))]
os::fsync_dir_std(source)?;
}
if let Some(path) = terminal {
os::replace_multipart_source_metadata(path, raw, base, root)?;
}
os::replace_multipart_source_metadata(&source.join(STATE), raw, base, root)
}
async fn write_owned_backup_and_persist(&self, owner: Uuid, target: bool, bytes: &[u8], state: &SourceState) -> Result<()> {
if owner.is_nil() {
return Err(DiskError::FileCorrupt);
}
state.validate(self.disk_id, &self.path)?;
let raw = encode(state)?;
let source = self.source.clone();
let base = self.base.clone();
let root = self.publication_root.clone();
let terminal = matches!(
state.phase,
MultipartSourcePhase::LocalTerminalSeal | MultipartSourcePhase::GlobalTerminalCertified
)
.then(|| {
source
.join(".mpu-completions")
.join(state.backup_owner.to_string())
.join(format!("terminal-{}.msgpack", state.revision))
});
os::run_blocking_namespace_operation(self.lease.clone(), move || {
let _pin = os::pin_multipart_source_directory(&source, &base, &root)?;
if let Some(path) = terminal {
os::replace_multipart_source_metadata(&path, &raw, &base, &root)?;
}
os::replace_multipart_source_metadata(&source.join(STATE), &raw, &base, &root)
let terminal = terminal_record_path(&source, state);
let bytes = bytes.to_vec();
os::run_blocking_namespace_operation_with_sync(self.lease.clone(), true, move || {
Self::write_owned_backup_std(&source, &base, &root, owner, target, &bytes)?;
Self::persist_state_std(&source, &base, &root, &raw, terminal.as_deref(), false)
})
.await
.map_err(DiskError::from)
}
#[cfg(test)]
pub async fn original(&self, state: &SourceState) -> Result<Bytes> {
let path = original_path(&self.source, state.backup_owner);
let base = self.base.clone();
@@ -421,54 +525,65 @@ impl SourceStore {
restore_from: None,
abort: None,
};
self.write_owned_backup(state.backup_owner, false, original).await?;
self.persist(&state).await?;
self.write_owned_backup_and_persist(state.backup_owner, false, original, &state)
.await?;
Ok(state)
}
#[cfg(test)]
pub async fn write_owned_backup(&self, owner: Uuid, target: bool, bytes: &[u8]) -> Result<()> {
if owner.is_nil() {
return Err(DiskError::FileCorrupt);
}
let path = if target {
target_backup_path(&self.source, owner)
} else {
original_path(&self.source, owner)
};
let source = self.source.clone();
let base = self.base.clone();
let root = self.publication_root.clone();
let bytes = bytes.to_vec();
os::run_blocking_namespace_operation(self.lease.clone(), move || {
let _pin = os::pin_multipart_source_directory(&source, &base, &root)?;
let parent = path
.parent()
.ok_or_else(|| std::io::Error::other("source backup has no parent"))?;
let _parents = os::mkdir_all_below_existing_base_std(parent, &base, &root)?;
let existed = if target {
match os::observe_multipart_source_metadata(&path, &base, &root, bytes.len() as u64) {
Ok((length, sha)) if length == bytes.len() as u64 && sha == raw_sha(&bytes) => true,
Ok(_) => {
return Err(IoError::new(std::io::ErrorKind::AlreadyExists, "immutable target backup changed"));
}
Err(error) if error.kind() == std::io::ErrorKind::NotFound => false,
Err(error) => return Err(error),
}
} else if let Some(previous) = read_optional(&path, &base, &root).map_err(IoError::other)? {
if previous.as_ref() != bytes.as_slice() {
return Err(IoError::new(std::io::ErrorKind::AlreadyExists, "immutable source backup changed"));
}
true
} else {
false
};
if !existed {
os::replace_multipart_source_metadata(&path, &bytes, &base, &root)?;
}
os::sync_multipart_owned_record_parents(&source, parent, existed, &base, &root)
Self::write_owned_backup_std(&source, &base, &root, owner, target, &bytes)
})
.await
.map_err(DiskError::from)
}
fn write_owned_backup_std(
source: &Path,
base: &Path,
root: &os::PublicationRoot,
owner: Uuid,
target: bool,
bytes: &[u8],
) -> std::io::Result<()> {
let path = if target {
target_backup_path(source, owner)
} else {
original_path(source, owner)
};
let _pin = os::pin_multipart_source_directory(source, base, root)?;
let parent = path
.parent()
.ok_or_else(|| std::io::Error::other("source backup has no parent"))?;
let _parents = os::mkdir_all_below_existing_base_std(parent, base, root)?;
let existed = if target {
match os::observe_multipart_source_metadata(&path, base, root, bytes.len() as u64) {
Ok((length, sha)) if length == bytes.len() as u64 && sha == raw_sha(bytes) => true,
Ok(_) => {
return Err(IoError::new(std::io::ErrorKind::AlreadyExists, "immutable target backup changed"));
}
Err(error) if error.kind() == std::io::ErrorKind::NotFound => false,
Err(error) => return Err(error),
}
} else if let Some(previous) = read_optional(&path, base, root).map_err(IoError::other)? {
if previous.as_ref() != bytes {
return Err(IoError::new(std::io::ErrorKind::AlreadyExists, "immutable source backup changed"));
}
true
} else {
false
};
if !existed {
os::replace_multipart_source_metadata(&path, bytes, base, root)?;
}
os::sync_multipart_owned_record_parents(source, parent, existed, base, root)
}
pub async fn replace_head(&self, bytes: &[u8]) -> Result<()> {
let path = self.source.join(STORAGE_FORMAT_FILE);
let base = self.base.clone();
@@ -490,7 +605,7 @@ impl SourceStore {
if encode(actual)? != encode(state)? {
return Err(DiskError::ConditionalDeleteConflict);
}
let raw = self.original(state).await?;
let raw = loaded.original.as_ref().ok_or(DiskError::FileCorrupt)?;
let allowed = match loaded.head.as_ref() {
None => {
state.phase == MultipartSourcePhase::Recovering
@@ -551,7 +666,7 @@ impl SourceStore {
if !allowed {
return Err(DiskError::ConditionalDeleteConflict);
}
let mut info = file_info(&raw, RUSTFS_META_MULTIPART_BUCKET, &self.path, state.lineage.original_version_id)?;
let mut info = file_info(raw, RUSTFS_META_MULTIPART_BUCKET, &self.path, state.lineage.original_version_id)?;
strip_native_controls(&mut info.metadata);
mark_native_source(&mut info);
set_source_status(
@@ -649,6 +764,14 @@ impl SourceStore {
Ok(metadata.map(|_| bytes))
}
pub fn target_matches(&self, publication: &PublicationIdentity, raw: &[u8]) -> Result<bool> {
Self::target_matches_for_slot(publication, raw, self.physical_slot, self.disk_ids.len())
}
pub(super) fn target_matches_for_slot(
publication: &PublicationIdentity,
raw: &[u8],
physical_slot: usize,
disk_count: usize,
) -> Result<bool> {
let info = file_info(raw, &publication.lineage.bucket, &publication.lineage.object, publication.version_id)?;
let slot = publication
.lineage
@@ -670,8 +793,8 @@ impl SourceStore {
&& info.erasure.parity_blocks == publication.lineage.geometry.parity_blocks
&& info.erasure.block_size == publication.lineage.geometry.block_size
&& info.erasure.distribution == publication.lineage.geometry.distribution
&& slot == self.physical_slot
&& publication.lineage.geometry.disks() == self.disk_ids.len())
&& slot == physical_slot
&& publication.lineage.geometry.disks() == disk_count)
}
pub async fn restore_exact_target(&self, state: &SourceState, owner: Option<Arc<dyn Send + Sync>>) -> Result<()> {
let publication = state.publication.as_ref().ok_or(DiskError::FileCorrupt)?;
@@ -772,13 +895,32 @@ pub(super) fn load_source(
physical_slot: usize,
disk_ids: &[Uuid],
) -> Result<LoadedSource> {
let state_raw = read_optional(&source.join(STATE), base, root)?;
let pinned = match os::pin_multipart_source_directory(source, base, root) {
Ok(pinned) => pinned,
Err(error) if error.kind() == std::io::ErrorKind::NotFound => {
return Ok(LoadedSource {
root_missing: true,
state: None,
head: None,
original: None,
info: None,
snapshot: None,
head_matches_state: false,
head_matches_activation: false,
});
}
Err(error) => return Err(error.into()),
};
let state_raw = read_optional_in_pinned(&pinned, source, STATE, base, root)?;
let state: Option<SourceState> = state_raw.as_deref().map(decode).transpose()?;
if state.is_none() && os::multipart_source_has_native_controls(source, base, root)? {
if state.is_none() && os::multipart_source_has_native_controls_in_pinned(source, &pinned)? {
return Err(DiskError::ConditionalDeleteConflict);
}
let head_path = source.join(STORAGE_FORMAT_FILE);
let head = if let Some(state) = state.as_ref() {
let head = if let Some(state) = state
.as_ref()
.filter(|state| state.prepared_length.is_some_and(|len| len > 64 * 1024))
{
let head_path = source.join(STORAGE_FORMAT_FILE);
let maximum = state
.prepared_length
.unwrap_or(0)
@@ -789,7 +931,8 @@ pub(super) fn load_source(
let raw = if is_proxy {
None
} else {
let raw = read_optional(&head_path, base, root)?.ok_or(DiskError::FileCorrupt)?;
let raw = read_optional_in_pinned(&pinned, source, STORAGE_FORMAT_FILE, base, root)?
.ok_or(DiskError::FileCorrupt)?;
if raw_sha(&raw) != sha256 {
return Err(DiskError::FileCorrupt);
}
@@ -801,10 +944,17 @@ pub(super) fn load_source(
Err(error) => return Err(error.into()),
}
} else {
read_optional(&head_path, base, root)?.map(|raw| SourceHead {
sha256: raw_sha(&raw),
length: raw.len() as u64,
raw: Some(raw),
read_optional_in_pinned(&pinned, source, STORAGE_FORMAT_FILE, base, root)?.map(|raw| {
let sha256 = raw_sha(&raw);
let length = raw.len() as u64;
let is_proxy = state
.as_ref()
.is_some_and(|state| Some(sha256) == state.prepared_sha256 && Some(length) == state.prepared_length);
SourceHead {
sha256,
length,
raw: (!is_proxy).then_some(raw),
}
})
};
let mut original = None;
@@ -870,26 +1020,12 @@ pub(super) fn load_source(
}
}
let head_sha = head.as_ref().map(|head| head.sha256);
let inventory = raw_sha(&encode(&(
state_raw.as_ref().map(|bytes| raw_sha(bytes)),
head_sha,
state.original_sha256,
state.negative_certificate_sha256,
state.ready_certificate_sha256,
))?);
snapshot = Some(MultipartSourceSnapshot {
snapshot = Some(snapshot_for_state(
disk_id,
physical_slot: state.physical_slot,
lineage: state.lineage.clone(),
generation: state.generation,
revision: state.revision,
phase: state.phase,
state_sha256: raw_sha(state_raw.as_ref().ok_or(DiskError::FileCorrupt)?),
head_sha256: head_sha,
inventory_sha256: inventory,
capability_sha256: state.capability_sha256,
publication: state.publication.clone(),
});
state,
state_raw.as_ref().ok_or(DiskError::FileCorrupt)?,
head_sha,
)?);
} else if let Some(observed) = &head {
let raw = observed.raw().ok_or(DiskError::FileCorrupt)?;
let current = get_file_info(
@@ -950,6 +1086,7 @@ pub(super) fn load_source(
head_matches_state = true;
}
Ok(LoadedSource {
root_missing: false,
state,
head,
original,
@@ -1195,11 +1332,7 @@ impl SourceStore {
directory: &Path,
lease: Arc<os::NamespaceMutationLease>,
) -> Result<()> {
let loaded = self.load().await?;
let current = loaded.state.as_ref().ok_or(DiskError::ConditionalDeleteConflict)?;
if encode(current)? != encode(state)?
|| !loaded.head_matches_state
|| state.lineage != publication.lineage
if state.lineage != publication.lineage
|| state.generation != publication.source_generation
|| state.disk_id != self.disk_id
|| state.physical_slot != self.physical_slot
@@ -1436,8 +1569,7 @@ impl LocalDisk {
};
// Copy the small source FI to an independent immutable inode. Its
// canonical inode is never truncated to build the destination proxy.
store.write_owned_backup(owner, false, original).await?;
store.persist(&state).await?;
store.write_owned_backup_and_persist(owner, false, original, &state).await?;
let prepared_path = store.source.join(format!(".mpu-prepared-{owner}"));
let context = NativeMultipartPublication {
store,
@@ -1480,14 +1612,19 @@ impl NativeMultipartPublication {
length: new_metadata.len() as u64,
};
if let Some(previous) = old_metadata {
self.store.write_owned_backup(state.backup_owner, true, previous).await?;
state.previous_target_sha256 = Some(raw_sha(previous));
state.previous_target_length = Some(previous.len() as u64);
}
state.prepared_sha256 = Some(validated.sha256);
state.prepared_length = Some(validated.length);
state.advance(MultipartSourcePhase::Prepared)?;
self.store.persist(&state).await?;
if let Some(previous) = old_metadata {
self.store
.write_owned_backup_and_persist(state.backup_owner, true, previous, &state)
.await?;
} else {
self.store.persist(&state).await?;
}
*self.state.lock() = state;
*self.validated_prepared.lock() = Some(validated);
Ok(())
@@ -1569,31 +1706,28 @@ impl NativeMultipartPublication {
if current.head.is_some() || encode(current_state)? != encode(&state)? {
return Err(DiskError::ConditionalDeleteConflict);
}
let target = self
.store
.read_target(&self.publication)
.await?
.ok_or(DiskError::ConditionalDeleteConflict)?;
{
let (expected_sha256, expected_length) = {
let guard = self.validated_prepared.lock();
let validated = guard.as_ref().ok_or(DiskError::ConditionalDeleteConflict)?;
if validated.publication != self.publication
|| state.prepared_sha256 != Some(validated.sha256)
|| state.prepared_length != Some(validated.length)
|| target.len() as u64 != validated.length
|| raw_sha(&target) != validated.sha256
{
return Err(DiskError::ConditionalDeleteConflict);
}
}
self.store.sync_target_with_lease(&self.publication, target_lease).await?;
let source = self.store.source.clone();
os::run_blocking_namespace_operation(self.store.lease.clone(), move || os::fsync_dir_std(&source)).await?;
(validated.sha256, validated.length)
};
self.store
.read_and_sync_target_with_lease(&self.publication, target_lease, move |target| {
if target.len() as u64 != expected_length || raw_sha(target) != expected_sha256 {
return Err(DiskError::ConditionalDeleteConflict);
}
Ok(())
})
.await?;
state.advance(MultipartSourcePhase::ConsumedUnsealed)?;
self.store.persist(&state).await?;
let snapshot = self.store.persist_with_source_sync(&state, true).await?;
*self.state.lock() = state;
let loaded = self.store.load().await?;
let snapshot = loaded.snapshot.ok_or(DiskError::FileCorrupt)?;
let proof = self
.store
.issue_proof(MultipartProofKind::Published, self.publication.attempt, None, snapshot)?;
@@ -1632,6 +1766,46 @@ impl SourceStore {
.await
.map_err(DiskError::from)
}
pub async fn read_and_sync_target(
&self,
publication: &PublicationIdentity,
validate: impl FnOnce(&[u8]) -> Result<()> + Send + 'static,
) -> Result<()> {
let io = self.target_path(publication)?;
let logical = self
.root
.join(io.strip_prefix(&self.base).map_err(|_| DiskError::InvalidPath)?);
let lease = os::acquire_namespace_mutation_lease_with_owner(&logical, Some(self.lease.clone())).await;
self.read_and_sync_target_with_lease(publication, lease, validate).await
}
pub async fn read_and_sync_target_with_lease(
&self,
publication: &PublicationIdentity,
lease: Arc<os::NamespaceMutationLease>,
validate: impl FnOnce(&[u8]) -> Result<()> + Send + 'static,
) -> Result<()> {
let directory = self.target_path(publication)?;
let base = self.base.clone();
let root = self.publication_root.clone();
os::run_blocking_namespace_operation_with_sync(lease, true, move || {
os::read_and_sync_multipart_target_metadata(
&directory,
std::ffi::OsStr::new(STORAGE_FORMAT_FILE),
&base,
&root,
|bytes| validate(bytes).map_err(IoError::other),
)
.map_err(|error| {
if error.kind() == std::io::ErrorKind::NotFound {
IoError::other(DiskError::ConditionalDeleteConflict)
} else {
error
}
})
})
.await
.map_err(DiskError::from)
}
async fn sync_source_absence(&self) -> Result<()> {
let source = self.source.clone();
let base = self.base.clone();
@@ -411,15 +411,23 @@ impl LocalDisk {
}
let _attempt = os::mkdir_all_below_existing_base_std(&private_object, &base, &publication_root)?;
if incoming.data.is_none() && !incoming.is_remote() {
let source_data_dir = source_object.join(source_dir.to_string());
let pinned_source_data = (!incoming.parts.is_empty())
.then(|| os::pin_multipart_source_directory(&source_data_dir, &base, &publication_root))
.transpose()?;
let mut names = Vec::with_capacity(incoming.parts.len().saturating_mul(3));
for part in &incoming.parts {
if cancelled() {
return Err(std::io::Error::new(std::io::ErrorKind::Interrupted, "multipart staging cancelled"));
}
let path = source_object
.join(source_dir.to_string())
.join(format!("part.{}.meta", part.number));
let bytes = os::read_multipart_source_metadata(&path, &base, &publication_root)?;
let meta_name = format!("part.{}.meta", part.number);
let bytes = os::read_multipart_source_metadata_in_pinned(
pinned_source_data.as_ref().expect("pinned source data directory"),
&source_data_dir,
std::ffi::OsStr::new(&meta_name),
&base,
&publication_root,
)?;
let actual = rustfs_filemeta::ObjectPartInfo::unmarshal(&bytes).map_err(std::io::Error::other)?;
let normalized = i64::try_from(actual.size)
.map_err(std::io::Error::other)?
@@ -441,13 +449,13 @@ impl LocalDisk {
// Preserve the native completed-MPU directory format. This
// immutable, validated generation record is distinct from
// the upload's mutable top-level xl.meta.
names.push(format!("part.{}.meta", part.number));
names.push(meta_name);
if let Some(integrity) = &part.integrity {
names.push(integrity.file_name());
}
}
os::hardlink_multipart_source_files(
&source_object.join(source_dir.to_string()),
&source_data_dir,
&private_object.join(final_dir.to_string()),
&names,
&base,
+236 -95
View File
@@ -3047,17 +3047,6 @@ pub(crate) async fn rename_all_with_lease(
/// A source admission must run before destination preparation can mkdir and
/// again before the consuming rename, inside the actual owned callback.
pub(crate) async fn rename_all_with_precondition(
source: &Path,
destination: &Path,
base: &Path,
publication_root: &PublicationRoot,
owner: Option<Arc<dyn Send + Sync>>,
precondition: Option<Arc<dyn Fn() -> io::Result<()> + Send + Sync>>,
) -> Result<()> {
rename_all_with_precondition_and_parent_sync(source, destination, base, publication_root, owner, precondition, false).await
}
pub(crate) async fn rename_all_with_precondition_and_parent_sync(
source: &Path,
destination: &Path,
@@ -3075,45 +3064,63 @@ pub(crate) async fn rename_all_with_precondition_and_parent_sync(
let base = base.to_path_buf();
let root = publication_root.clone();
let lease = acquire_namespace_mutation_lease_with_owner(&destination, owner).await;
run_blocking_namespace_operation(lease, move || {
if let Some(check) = precondition.as_ref() {
check()?;
}
let preparation = prepare_rename_with_retry(&source, &destination, &base, &root)?;
#[cfg(all(test, not(windows)))]
prepared_publication_test_hooks::run_rename_destination(&source, &destination);
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
{
prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, &source);
prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, &destination);
}
if let Some(check) = precondition.as_ref() {
check()?;
}
rename_prepared(&source, &destination, &preparation)?;
if sync_parent {
let parent = destination
.parent()
.ok_or_else(|| io::Error::other("source metadata parent is missing"))?;
#[cfg(unix)]
{
let pinned = preparation
.parent_guard
.as_ref()
.and_then(|guard| guard.last())
.ok_or_else(|| io::Error::other("source metadata parent was not pinned"))?;
let opened = std::fs::File::from(pinned.try_clone()?);
fsync_directory_handle_std(parent, Some(&opened))?;
}
#[cfg(not(unix))]
fsync_dir_std(parent)?;
}
Ok(())
run_blocking_namespace_operation_with_sync(lease, sync_parent, move || {
rename_all_with_precondition_and_parent_sync_std(
&source,
&destination,
&base,
&root,
precondition.as_deref(),
sync_parent,
)
})
.await
.map_err(|error| to_file_error(error).into())
}
pub(crate) fn rename_all_with_precondition_and_parent_sync_std(
source: &Path,
destination: &Path,
base: &Path,
publication_root: &PublicationRoot,
precondition: Option<&(dyn Fn() -> io::Result<()> + Send + Sync)>,
sync_parent: bool,
) -> io::Result<()> {
if let Some(check) = precondition {
check()?;
}
let preparation = prepare_rename_with_retry(source, destination, base, publication_root)?;
#[cfg(all(test, not(windows)))]
prepared_publication_test_hooks::run_rename_destination(source, destination);
#[cfg(all(any(test, feature = "test-util"), not(windows)))]
{
prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, source);
prepared_publication_test_hooks::run(prepared_publication_test_hooks::Stage::Rename, destination);
if let Some(check) = precondition {
check()?;
}
}
rename_prepared(source, destination, &preparation)?;
if sync_parent {
let parent = destination
.parent()
.ok_or_else(|| io::Error::other("source metadata parent is missing"))?;
#[cfg(unix)]
{
let pinned = preparation
.parent_guard
.as_ref()
.and_then(|guard| guard.last())
.ok_or_else(|| io::Error::other("source metadata parent was not pinned"))?;
let opened = std::fs::File::from(pinned.try_clone()?);
fsync_directory_handle_std(parent, Some(&opened))?;
}
#[cfg(not(unix))]
fsync_dir_std(parent)?;
}
Ok(())
}
/// Publish a conditional repair and sync its directory in one owned executor.
/// Cancellation cannot release its metadata/fleet/tier leases between rename
/// and fsync. The last authority check runs after destination preparation.
@@ -3186,26 +3193,32 @@ pub(crate) struct DurableFileReceipt {
#[cfg(unix)]
fn open_anchored_metadata(path: &Path, publication_root: &PublicationRoot) -> io::Result<std::fs::File> {
use rustix::fs::{Mode, OFlags, openat};
#[cfg(target_os = "linux")]
let logical_path = publication_root.path().join(publication_root.relative_path(path)?);
#[cfg(target_os = "linux")]
let path = logical_path.as_path();
let parents = open_existing_directory_below_base_std(
path.parent().ok_or_else(|| io::Error::other("metadata has no parent"))?,
publication_root.path(),
publication_root,
)?;
let directory = parents
.last()
.ok_or_else(|| io::Error::other("missing metadata directory guard"))?;
Ok(openat(
directory,
path.file_name().ok_or_else(|| io::Error::other("metadata has no name"))?,
OFlags::RDONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC,
Mode::empty(),
)?
.into())
{
let parent = path.parent().ok_or_else(|| io::Error::other("metadata has no parent"))?;
existing_base_directory_suffix(parent, publication_root.path())
.or_else(|_| existing_base_directory_suffix(parent, publication_root.io_path()))?;
publication_root.open_readonly(path)
}
#[cfg(not(target_os = "linux"))]
{
use rustix::fs::{Mode, OFlags, openat};
let parents = open_existing_directory_below_base_std(
path.parent().ok_or_else(|| io::Error::other("metadata has no parent"))?,
publication_root.path(),
publication_root,
)?;
let directory = parents
.last()
.ok_or_else(|| io::Error::other("missing metadata directory guard"))?;
Ok(openat(
directory,
path.file_name().ok_or_else(|| io::Error::other("metadata has no name"))?,
OFlags::RDONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC,
Mode::empty(),
)?
.into())
}
}
#[cfg(unix)]
@@ -3372,6 +3385,29 @@ fn open_existing_directory_below_base_std(
use rustix::fs::{Mode, OFlags, openat};
let relative = existing_base_directory_suffix(directory, base)?;
#[cfg(target_os = "linux")]
{
use rustix::fs::{ResolveFlags, openat2};
use std::os::fd::AsFd;
let flags = OFlags::RDONLY | OFlags::DIRECTORY | OFlags::NOFOLLOW | OFlags::CLOEXEC;
let owned_base;
let base_fd = if base == publication_root.io_path.as_path() || base == publication_root.path.as_path() {
publication_root.directory.as_fd()
} else {
owned_base = std::os::fd::OwnedFd::from(publication_root.open_directory(base)?);
owned_base.as_fd()
};
let target = if relative.as_os_str().is_empty() {
Path::new(".")
} else {
relative
};
match openat2(base_fd, target, flags, Mode::empty(), ResolveFlags::BENEATH | ResolveFlags::NO_SYMLINKS) {
Ok(leaf) => return Ok(vec![leaf]),
Err(rustix::io::Errno::NOSYS) => {}
Err(err) => return Err(err.into()),
}
}
#[cfg(target_os = "linux")]
let base = publication_root.open_directory(base)?.into();
#[cfg(not(target_os = "linux"))]
let base = {
@@ -3658,8 +3694,10 @@ pub(crate) fn read_destination_file_with_commit_guard(
};
match open_linux_relative(&parent.file, Path::new(name), OFlags::RDONLY, Mode::empty()) {
Ok(mut file) => {
let mut data = Vec::new();
std::io::Read::read_to_end(&mut file, &mut data)?;
let len = file.metadata()?.len();
let cap = usize::try_from(len).map_or(0, |len| len.saturating_add(1));
let mut data = Vec::with_capacity(cap);
std::io::Read::read_to_end(&mut std::io::Read::take(&mut file, len), &mut data)?;
Ok(Some(data))
}
Err(err) if err.kind() == io::ErrorKind::NotFound => Ok(None),
@@ -4925,12 +4963,16 @@ impl LinuxCommitDirectories {
use std::os::unix::fs::MetadataExt;
// A pinned directory may have been unlinked or replaced. Do not report a
// successful publication into a detached object tree.
for (path, directory) in [
(&self.source_path, &self.source),
(&self.destination_path, self.prepare_destination()?),
] {
let current = self.root.open_directory(path)?.metadata()?;
if (current.dev(), current.ino()) != (directory.device, directory.inode) {
let current = self.root.open_directory(&self.source_path)?.metadata()?;
if (current.dev(), current.ino()) != (self.source.device, self.source.inode) {
return Err(io::Error::new(io::ErrorKind::InvalidData, "commit directory identity changed"));
}
let destination = self.prepare_destination()?;
if self.source_path != self.destination_path
|| (self.source.device, self.source.inode) != (destination.device, destination.inode)
{
let current = self.root.open_directory(&self.destination_path)?.metadata()?;
if (current.dev(), current.ino()) != (destination.device, destination.inode) {
return Err(io::Error::new(io::ErrorKind::InvalidData, "commit directory identity changed"));
}
}
@@ -5223,12 +5265,20 @@ pub(crate) fn prepare_rename_commit_guard(
existing_base_directory_suffix(destination_parent, destination_base)?;
let source = LinuxCommitDirectory::new(publication_root.open_directory(source_parent)?)?;
let destination = std::sync::OnceLock::new();
match publication_root.open_directory(destination_parent) {
Ok(directory) => {
let _ = destination.set(LinuxCommitDirectory::new(directory)?);
if source_parent == destination_parent {
let _ = destination.set(LinuxCommitDirectory {
file: Arc::clone(&source.file),
device: source.device,
inode: source.inode,
});
} else {
match publication_root.open_directory(destination_parent) {
Ok(directory) => {
let _ = destination.set(LinuxCommitDirectory::new(directory)?);
}
Err(err) if err.kind() == io::ErrorKind::NotFound => {}
Err(err) => return Err(err),
}
Err(err) if err.kind() == io::ErrorKind::NotFound => {}
Err(err) => return Err(err),
}
Ok(RenameCommitGuard {
directories: Arc::new(LinuxCommitDirectories {
@@ -6088,12 +6138,20 @@ fn existing_base_directory_suffix<'a>(dir_path: &'a Path, base_dir: &Path) -> io
/// Missing state never downgrades surviving native control records to legacy.
/// Inspect only this pinned upload root; parts and proof trees are not traversed.
#[cfg(test)]
pub(crate) fn multipart_source_has_native_controls(source: &Path, base: &Path, root: &PublicationRoot) -> io::Result<bool> {
let _pin = match pin_multipart_source_directory(source, base, root) {
let pin = match pin_multipart_source_directory(source, base, root) {
Ok(pin) => pin,
Err(error) if error.kind() == io::ErrorKind::NotFound => return Ok(false),
Err(error) => return Err(error),
};
multipart_source_has_native_controls_in_pinned(source, &pin)
}
pub(crate) fn multipart_source_has_native_controls_in_pinned(
source: &Path,
_pin: &ExistingBaseDirectoryGuard,
) -> io::Result<bool> {
let mut entries = std::fs::read_dir(source)?;
for _ in 0..READ_DIR_PROBE_RAW_LIMIT {
let Some(entry) = entries.next() else {
@@ -6216,6 +6274,7 @@ pub(crate) fn replace_multipart_source_metadata(
publication_root: &PublicationRoot,
) -> io::Result<()> {
let parent = path.parent().ok_or_else(|| io::Error::other("MPU record has no parent"))?;
#[cfg(not(target_os = "linux"))]
let _pinned = pin_multipart_source_directory(parent, base, publication_root)?;
let guard = prepare_rename_commit_guard(parent, parent, base, publication_root)?;
let staged = parent.join(format!(".mpu-record-{}", uuid::Uuid::new_v4()));
@@ -6397,15 +6456,10 @@ pub(crate) fn sync_multipart_owned_record_parents(
if !directory.starts_with(source) {
return Err(io::Error::new(io::ErrorKind::InvalidInput, "native record barrier escaped source"));
}
let pinned = pin_multipart_source_directory(directory, base, publication_root)?;
let mut pinned = pin_multipart_source_directory(directory, base, publication_root)?;
#[cfg(unix)]
{
let opened = std::fs::File::from(
pinned
.last()
.ok_or_else(|| io::Error::other("empty native directory guard"))?
.try_clone()?,
);
let opened = std::fs::File::from(pinned.pop().ok_or_else(|| io::Error::other("empty native directory guard"))?);
fsync_directory_handle_std(directory, Some(&opened))?;
}
#[cfg(not(unix))]
@@ -6436,6 +6490,7 @@ pub(crate) fn install_multipart_prepared_head(
if original_path.parent() != Some(parent) || original_path == canonical_head {
return Err(io::Error::new(io::ErrorKind::InvalidInput, "MPU proxy must be a private sibling"));
}
#[cfg(not(target_os = "linux"))]
let _pinned = pin_multipart_source_directory(parent, base, publication_root)?;
let guard = prepare_rename_commit_guard(parent, parent, base, publication_root)?;
rename_prepared_source_std(&prepared, &original_path, canonical_head, base, publication_root, &guard)?;
@@ -6517,6 +6572,16 @@ pub(crate) fn read_multipart_source_metadata(
read_multipart_metadata(path, base, publication_root, Some(16 * 1024 * 1024))
}
pub(crate) fn read_multipart_source_metadata_in_pinned(
pinned: &ExistingBaseDirectoryGuard,
parent: &Path,
name: &std::ffi::OsStr,
base: &Path,
publication_root: &PublicationRoot,
) -> io::Result<Vec<u8>> {
read_multipart_metadata_in_pinned(pinned, parent, name, base, publication_root, Some(16 * 1024 * 1024))
}
/// Target FI histories have no source-control size limit. Keep the same
/// pinned-parent and no-follow checks when inspecting every referenced DD.
pub(crate) fn read_multipart_reclaim_target_metadata(
@@ -6558,8 +6623,10 @@ fn read_multipart_metadata_bytes(
Ok(bytes)
}
fn read_multipart_metadata(
path: &Path,
fn read_multipart_metadata_in_pinned(
pinned: &ExistingBaseDirectoryGuard,
parent: &Path,
name: &std::ffi::OsStr,
base: &Path,
publication_root: &PublicationRoot,
maximum: Option<u64>,
@@ -6567,16 +6634,10 @@ fn read_multipart_metadata(
#[cfg(unix)]
{
use rustix::fs::{Mode, OFlags, openat};
let parent = path
.parent()
.ok_or_else(|| io::Error::other("multipart metadata has no parent"))?;
let guards = open_existing_directory_below_base_std(parent, base, publication_root)?;
let directory = guards
let _ = (parent, base, publication_root);
let directory = pinned
.last()
.ok_or_else(|| io::Error::other("missing multipart source directory"))?;
let name = path
.file_name()
.ok_or_else(|| io::Error::other("multipart metadata has no name"))?;
let mut file = std::fs::File::from(openat(
directory,
name,
@@ -6589,6 +6650,86 @@ fn read_multipart_metadata(
}
read_multipart_metadata_bytes(&mut file, Some(metadata.len()), maximum)
}
#[cfg(not(unix))]
{
let _ = pinned;
read_multipart_metadata(&parent.join(name), base, publication_root, maximum)
}
}
pub(crate) fn read_and_sync_multipart_target_metadata(
directory: &Path,
name: &std::ffi::OsStr,
base: &Path,
publication_root: &PublicationRoot,
validate: impl FnOnce(&[u8]) -> io::Result<()>,
) -> io::Result<()> {
#[cfg(unix)]
{
use rustix::fs::{Mode, OFlags, openat};
let mut pinned = pin_multipart_source_directory(directory, base, publication_root)?;
let dir_fd = pinned
.last()
.ok_or_else(|| io::Error::other("missing multipart target directory"))?;
let mut file =
std::fs::File::from(openat(dir_fd, name, OFlags::RDONLY | OFlags::NOFOLLOW | OFlags::CLOEXEC, Mode::empty())?);
let metadata = file.metadata()?;
if !metadata.is_file() {
return Err(io::Error::new(io::ErrorKind::InvalidData, "multipart metadata is not a regular file"));
}
let bytes = read_multipart_metadata_bytes(&mut file, Some(metadata.len()), None)?;
validate(&bytes)?;
file.sync_data()?;
let opened = std::fs::File::from(pinned.pop().ok_or_else(|| io::Error::other("empty target directory guard"))?);
fsync_directory_handle_std(directory, Some(&opened))
}
#[cfg(not(unix))]
{
let _pinned = pin_multipart_source_directory(directory, base, publication_root)?;
let file = directory.join(name);
let bytes = read_multipart_metadata(&file, base, publication_root, None)?;
validate(&bytes)?;
sync_file(&file)?;
fsync_dir_std(directory)
}
}
fn read_multipart_metadata(
path: &Path,
base: &Path,
publication_root: &PublicationRoot,
maximum: Option<u64>,
) -> io::Result<Vec<u8>> {
#[cfg(target_os = "linux")]
{
let parent = path
.parent()
.ok_or_else(|| io::Error::other("multipart metadata has no parent"))?;
let name = path
.file_name()
.ok_or_else(|| io::Error::other("multipart metadata has no name"))?;
if name.is_empty() {
return Err(io::Error::new(io::ErrorKind::InvalidInput, "multipart metadata has no name"));
}
existing_base_directory_suffix(parent, base)?;
let mut file = publication_root.open_readonly(path)?;
let metadata = file.metadata()?;
if !metadata.is_file() {
return Err(io::Error::new(io::ErrorKind::InvalidData, "multipart metadata is not a regular file"));
}
read_multipart_metadata_bytes(&mut file, Some(metadata.len()), maximum)
}
#[cfg(all(unix, not(target_os = "linux")))]
{
let parent = path
.parent()
.ok_or_else(|| io::Error::other("multipart metadata has no parent"))?;
let name = path
.file_name()
.ok_or_else(|| io::Error::other("multipart metadata has no name"))?;
let guards = open_existing_directory_below_base_std(parent, base, publication_root)?;
read_multipart_metadata_in_pinned(&guards, parent, name, base, publication_root, maximum)
}
#[cfg(windows)]
{
use windows_sys::{
+364 -19
View File
@@ -13,6 +13,7 @@
// limitations under the License.
use crate::bucket::lifecycle::tier_last_day_stats::DailyAllTierStats;
use crate::cluster::rpc::peer_rest_client::{ScannerLeaseConnectionFailure, scanner_publication_lease_error_class};
use crate::cluster::rpc::{
PeerRestClient, ScannerDirtyUsageAcknowledgement, ScannerPeerActivity, ScannerPeerDirtyUsageSnapshot,
ScannerPublicationLease, TierConfigReloadOutcome,
@@ -158,16 +159,132 @@ struct PeerAdminCache {
last_server_success: Option<SystemTime>,
}
fn scanner_publication_lease_acquisition_error(host: &str, observational_only: bool, error: Error) -> Error {
enum ScannerPublicationLeaseFailure {
DuplicateTarget {
peer: String,
},
PeerUnavailable {
peer: String,
},
RequestFailed {
peer: String,
purpose: &'static str,
detail: String,
reason: Option<&'static str>,
reported_reason: Option<&'static str>,
},
}
impl std::fmt::Debug for ScannerPublicationLeaseFailure {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
formatter.write_str("ScannerPublicationLeaseFailure")
}
}
impl std::fmt::Display for ScannerPublicationLeaseFailure {
fn fmt(&self, formatter: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::DuplicateTarget { peer } => write!(formatter, "duplicate scanner publication lease target: {peer}"),
Self::PeerUnavailable { peer } => write!(formatter, "scanner publication lease peer {peer} is unavailable"),
Self::RequestFailed {
peer, purpose, detail, ..
} => write!(
formatter,
"scanner publication lease acquisition failed: peer={peer}, purpose={purpose}: {detail}"
),
}
}
}
impl std::error::Error for ScannerPublicationLeaseFailure {}
/// Bounded details from the target that actually failed lease acquisition.
#[derive(Clone, Copy, Debug)]
pub struct ScannerPublicationLeaseFailureInfo<'a> {
pub peer: Option<&'a str>,
pub kind: &'static str,
pub reason: Option<&'static str>,
pub reported_reason: Option<&'static str>,
}
fn scanner_publication_lease_target_peer(target: &str) -> Option<&str> {
if target.len() > 267 {
return None;
}
let (peer, parse_target) = if let Some(peer) = target.strip_prefix("http://").or_else(|| target.strip_prefix("https://")) {
(peer, std::borrow::Cow::Borrowed(target))
} else {
(target, std::borrow::Cow::Owned(format!("http://{target}")))
};
if peer.is_empty()
|| peer.len() > 259
|| peer.ends_with(':')
|| !peer
.bytes()
.all(|byte| byte.is_ascii_alphanumeric() || b".-_:[]".contains(&byte))
{
return None;
}
let parsed = url::Url::parse(&parse_target).ok()?;
if !matches!(parsed.scheme(), "http" | "https")
|| parsed.host().is_none()
|| !parsed.username().is_empty()
|| parsed.password().is_some()
|| parsed.path() != "/"
|| parsed.query().is_some()
|| parsed.fragment().is_some()
{
return None;
}
Some(peer)
}
/// Return structured lease failure context without parsing remote error text.
pub fn scanner_publication_lease_failure_info(error: &Error) -> Option<ScannerPublicationLeaseFailureInfo<'_>> {
let Error::Io(error) = error else { return None };
let failure = error.get_ref()?.downcast_ref::<ScannerPublicationLeaseFailure>()?;
let (peer, kind, reason, reported_reason) = match failure {
ScannerPublicationLeaseFailure::DuplicateTarget { peer } => (peer, "duplicate_target", None, None),
ScannerPublicationLeaseFailure::PeerUnavailable { peer } => (peer, "peer_unavailable", None, None),
ScannerPublicationLeaseFailure::RequestFailed {
peer,
reason,
reported_reason,
..
} => (peer, "peer_request_failed", *reason, *reported_reason),
};
let safe_peer = scanner_publication_lease_target_peer(peer);
Some(ScannerPublicationLeaseFailureInfo {
peer: safe_peer,
kind,
reason,
reported_reason,
})
}
fn scanner_publication_lease_acquisition_error(
host: &str,
observational_only: bool,
error: Error,
connection_failure: Option<ScannerLeaseConnectionFailure>,
) -> Error {
let purpose = if observational_only {
"observational"
} else {
"authoritative"
};
// Retry classification depends on the original error suffix.
Error::other(format!(
"scanner publication lease acquisition failed: peer={host}, purpose={purpose}: {error}"
))
let class = scanner_publication_lease_error_class(&error);
Error::other(ScannerPublicationLeaseFailure::RequestFailed {
peer: host.to_owned(),
purpose,
detail: error.to_string(),
reason: match connection_failure {
Some(ScannerLeaseConnectionFailure::ChannelAcquisitionTimedOut) => Some("channel_acquisition_timed_out"),
None => class.reason,
},
reported_reason: class.reported_reason,
})
}
#[derive(Default)]
@@ -3161,7 +3278,7 @@ impl NotificationSys {
targets.sort_by(|left, right| left.0.cmp(&right.0));
for pair in targets.windows(2) {
if pair[0].0 == pair[1].0 {
return Err(Error::other(format!("duplicate scanner publication lease target: {}", pair[0].0)));
return Err(Error::other(ScannerPublicationLeaseFailure::DuplicateTarget { peer: pair[0].0.clone() }));
}
}
@@ -3175,16 +3292,21 @@ impl NotificationSys {
.cloned()
else {
let _ = self.release_scanner_publication_leases(grants).await;
return Err(Error::other(format!("scanner publication lease peer {host} is unavailable")));
return Err(Error::other(ScannerPublicationLeaseFailure::PeerUnavailable { peer: host }));
};
match client
.acquire_scanner_publication_lease(&session_id, generation, observational_only)
.acquire_scanner_publication_lease_with_connection_failure(&session_id, generation, observational_only)
.await
{
Ok(lease) => grants.push(ScannerPublicationLeaseGrant { host, lease }),
Err(err) => {
Err((err, connection_failure)) => {
let _ = self.release_scanner_publication_leases(grants).await;
return Err(scanner_publication_lease_acquisition_error(&host, observational_only, err));
return Err(scanner_publication_lease_acquisition_error(
&host,
observational_only,
err,
connection_failure,
));
}
}
}
@@ -4185,20 +4307,243 @@ mod tests {
#[test]
fn scanner_lease_acquisition_error_identifies_peer_and_preserves_retry_suffix() {
use crate::cluster::rpc::peer_rest_client::scanner_publication_lease_remote_rejection;
for (observational_only, purpose) in [(false, "authoritative"), (true, "observational")] {
let reason = "scanner publication lease is blocked by pending namespace commit";
let error = scanner_publication_lease_acquisition_error(
"peer-a.example.invalid:9000",
observational_only,
Error::other(reason),
)
.to_string();
assert!(error.contains("peer-a.example.invalid:9000"), "missing peer: {error}");
assert!(error.contains(purpose), "missing purpose: {error}");
assert!(error.ends_with(reason), "retry classification must retain the original suffix");
let reason = "scanner publication lease capacity is exhausted";
let source = scanner_publication_lease_remote_rejection(format!("Io error: {reason}"));
let expected = format!(
"Io error: scanner publication lease acquisition failed: peer=peer-a.example.invalid:9000, purpose={purpose}: {source}"
);
let error =
scanner_publication_lease_acquisition_error("peer-a.example.invalid:9000", observational_only, source, None);
let info = scanner_publication_lease_failure_info(&error).expect("structured failure source");
assert_eq!(info.peer, Some("peer-a.example.invalid:9000"));
assert_eq!(info.kind, "peer_request_failed");
assert_eq!(info.reason, None);
assert_eq!(info.reported_reason, Some("capacity_exhausted"));
assert_eq!(error.to_string(), expected);
assert!(
error.to_string().ends_with(reason),
"retry classification must retain the original suffix"
);
let cloned = error.clone();
assert_eq!(cloned.to_string(), expected);
assert!(
scanner_publication_lease_failure_info(&cloned).is_none(),
"a cloned generic I/O error must not invent lost structured context"
);
}
}
#[test]
fn scanner_lease_failure_info_uses_the_actual_target_and_rejects_unbounded_peers() {
for (failure, expected_display, expected_kind) in [
(
ScannerPublicationLeaseFailure::DuplicateTarget {
peer: "peer-a:9000".to_owned(),
},
"Io error: duplicate scanner publication lease target: peer-a:9000",
"duplicate_target",
),
(
ScannerPublicationLeaseFailure::PeerUnavailable {
peer: "peer-a:9000".to_owned(),
},
"Io error: scanner publication lease peer peer-a:9000 is unavailable",
"peer_unavailable",
),
] {
let error = Error::other(failure);
let info = scanner_publication_lease_failure_info(&error).expect("structured failure source");
assert_eq!(info.peer, Some("peer-a:9000"));
assert_eq!(info.kind, expected_kind);
assert_eq!(info.reason, None);
assert_eq!(info.reported_reason, None);
assert_eq!(error.to_string(), expected_display);
}
let forged =
Error::other("scanner publication lease acquisition failed: peer=forged:9000, purpose=authoritative: rejected");
assert!(
scanner_publication_lease_failure_info(&forged).is_none(),
"raw error text must not establish peer identity"
);
for invalid in [
"peer-a:9000\nforged",
"user:secret@peer-a:9000",
"http://user:secret@peer-a:9000",
"http://@peer-a:9000",
"http://peer-a:65536",
"https://peer-a:bad",
"https://peer-a:",
"http://[2001:db8::1",
"http://[2001:db8::1]:bad",
"https://peer-a:9000/path",
"https://peer-a:9000?token=secret",
"https://peer-a:9000#secret",
"https://peer-a%2eexample:9000",
"ftp://peer-a:9000",
"",
] {
let error = scanner_publication_lease_acquisition_error(invalid, false, Error::other("remote-secret"), None);
let info = scanner_publication_lease_failure_info(&error).expect("structured failure source");
assert_eq!(info.peer, None, "invalid target must not reach logs");
assert_eq!(info.reported_reason, None, "unrecognized remote text must not become a reason");
assert!(!format!("{error:?}").contains("remote-secret"), "Debug must not expose remote error text");
}
let oversized = format!("https://{}:9000", "a".repeat(260));
let error = scanner_publication_lease_acquisition_error(&oversized, false, Error::other("remote-secret"), None);
assert_eq!(scanner_publication_lease_failure_info(&error).and_then(|info| info.peer), None);
}
#[test]
fn scanner_lease_failure_info_accepts_real_grid_host_urls() {
use crate::cluster::rpc::peer_rest_client::scanner_publication_lease_remote_rejection;
use crate::layout::endpoint::Endpoint;
for (endpoint, expected) in [
("http://peer-a.example:9000/export", "peer-a.example:9000"),
("https://peer-b.example:9443/export", "peer-b.example:9443"),
("http://[2001:db8::1]:9000/export", "[2001:db8::1]:9000"),
("https://peer-c.example/export", "peer-c.example"),
] {
let target = Endpoint::try_from(endpoint).expect("valid endpoint").grid_host();
let source = scanner_publication_lease_remote_rejection(
"Io error: scanner publication lease capacity is exhausted".to_owned(),
);
let expected_display =
format!("Io error: scanner publication lease acquisition failed: peer={target}, purpose=observational: {source}");
let error = scanner_publication_lease_acquisition_error(&target, true, source, None);
let info = scanner_publication_lease_failure_info(&error).expect("structured failure source");
assert_eq!(info.peer, Some(expected));
assert_eq!(info.reason, None);
assert_eq!(info.reported_reason, Some("capacity_exhausted"));
assert_eq!(error.to_string(), expected_display);
assert!(error.to_string().ends_with("scanner publication lease capacity is exhausted"));
}
}
#[test]
fn scanner_lease_failure_info_preserves_typed_connection_failures_without_logging_details() {
for (source, reason) in [
(Error::RemoteClientUnavailable("remote-secret".to_owned()), "client_unavailable"),
(Error::Timeout, "timeout"),
(tonic::Status::unavailable("remote-secret").into(), "rpc_unavailable"),
(tonic::Status::deadline_exceeded("remote-secret").into(), "rpc_deadline_exceeded"),
] {
let expected = format!(
"Io error: scanner publication lease acquisition failed: peer=https://peer-a:9000, purpose=authoritative: {source}"
);
let error = scanner_publication_lease_acquisition_error("https://peer-a:9000", false, source, None);
let info = scanner_publication_lease_failure_info(&error).expect("structured failure source");
assert_eq!(info.peer, Some("peer-a:9000"));
assert_eq!(info.kind, "peer_request_failed");
assert_eq!(info.reason, Some(reason));
assert_eq!(info.reported_reason, None);
assert_eq!(error.to_string(), expected);
assert!(!format!("{info:?}").contains("remote-secret"));
}
let spoofed = scanner_publication_lease_acquisition_error(
"https://peer-a:9000",
false,
Error::other("peer returned a different scanner publication lease generation"),
None,
);
let info = scanner_publication_lease_failure_info(&spoofed).expect("structured failure source");
assert_eq!(info.reason, None);
assert_eq!(info.reported_reason, None);
}
#[test]
fn scanner_lease_channel_deadline_preserves_public_error_and_diagnostic() {
const CHILD_MARKER: &str = "RUSTFS_TEST_SCANNER_LEASE_CHANNEL_DEADLINE_CHILD";
if std::env::var_os(CHILD_MARKER).is_none() {
// Isolate the zero deadline from concurrently running tests.
let module = module_path!()
.strip_prefix(concat!(env!("CARGO_CRATE_NAME"), "::"))
.unwrap_or(module_path!());
let name = format!("{module}::scanner_lease_channel_deadline_preserves_public_error_and_diagnostic");
let output = std::process::Command::new(std::env::current_exe().expect("test executable"))
.args(["--exact", &name, "--nocapture"])
.env(CHILD_MARKER, "1")
.env(rustfs_config::ENV_INTERNODE_CONNECT_TIMEOUT_SECS, "0")
.env("NO_PROXY", "127.0.0.1,localhost")
.env("no_proxy", "127.0.0.1,localhost")
.output()
.expect("deadline test subprocess");
assert!(
output.status.success() && String::from_utf8_lossy(&output.stdout).contains("1 passed"),
"deadline subprocess failed: {} {}",
String::from_utf8_lossy(&output.stdout),
String::from_utf8_lossy(&output.stderr)
);
return;
}
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("test runtime");
runtime.block_on(async {
// Keep the address unique to this isolated subprocess.
let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("local listener");
let address = listener.local_addr().expect("listener address");
let target = format!("https://{address}");
let make_client = || {
PeerRestClient::new(
XHost {
name: "127.0.0.1".to_owned(),
port: address.port(),
is_port_set: true,
},
target.clone(),
)
};
// Block the real channel cache read so the acquisition deadline
// wins independently of OS-specific connection or TLS errors.
let cache_read_blocker = rustfs_common::block_connection_cache_reads_for_test().await;
let public_error = match make_client().get_client().await {
Ok(_) => panic!("a stalled connection must exceed the channel deadline"),
Err(error) => error,
};
assert!(matches!(&public_error, Error::RemoteClientUnavailable(_)));
assert!(
public_error
.to_string()
.contains("internode channel acquisition deadline elapsed")
);
let mut notification = NotificationSys::new(EndpointServerPools::default()).await;
notification.peer_clients.push(Some(make_client()));
let lease_error = match notification
.acquire_scanner_publication_leases(vec![(target.clone(), "session".into(), 3)])
.await
{
Ok(_) => panic!("a stalled connection cannot grant a lease"),
Err(error) => error,
};
assert_eq!(
lease_error.to_string(),
format!(
"Io error: scanner publication lease acquisition failed: peer={target}, purpose=authoritative: {public_error}"
)
);
let info = scanner_publication_lease_failure_info(&lease_error).expect("structured lease failure");
let expected_peer = address.to_string();
assert_eq!(info.peer, Some(expected_peer.as_str()));
assert_eq!(info.reason, Some("channel_acquisition_timed_out"));
assert_eq!(info.reported_reason, None);
assert_eq!(
scanner_peer_transport_error_message_is_retryable(&lease_error.to_string()),
scanner_peer_transport_error_message_is_retryable(&public_error.to_string())
);
drop(cache_read_blocker);
drop(listener);
});
}
fn ring(total_size: u64) -> LastDayTierStats {
let mut stats = LastDayTierStats::default();
stats.add_stats(TierStats {
@@ -1,8 +1,8 @@
use super::meta::{
RebalanceMetaMergeOutcome, clone_first_arc, clone_rebalance_pool_stats, defer_bucket_in_rebalance_queue,
ensure_valid_rebalance_pool_index, invalid_rebalance_pool_index_error, is_rebalance_actively_running,
is_rebalance_conflicting_with_decommission, mark_rebalance_bucket_done, merge_rebalance_meta, percent_free_ratio,
rebalance_metadata_not_initialized_error, record_rebalance_cleanup_warning_in_meta,
RebalanceMetaMergeOutcome, RebalanceMetadataLoadError, clone_first_arc, clone_rebalance_pool_stats,
defer_bucket_in_rebalance_queue, ensure_valid_rebalance_pool_index, invalid_rebalance_pool_index_error,
is_rebalance_actively_running, is_rebalance_conflicting_with_decommission, mark_rebalance_bucket_done, merge_rebalance_meta,
percent_free_ratio, rebalance_metadata_not_initialized_error, record_rebalance_cleanup_warning_in_meta,
record_rebalance_stop_propagation_snapshot, resolve_next_rebalance_bucket, rollback_rebalance_start_meta_snapshot_for_id,
should_accept_rebalance_stats_update, should_pool_participate, stop_rebalance_meta_snapshot_for_id,
validate_init_rebalance_state,
@@ -853,6 +853,26 @@ impl ECStore {
Ok(snapshot)
}
/// Admin-only rebalance status snapshot. Diagnostic reads may use the last
/// published state when the persisted record is temporarily unavailable;
/// mutation paths continue to use the authoritative loaders above.
pub async fn rebalance_admin_diagnostic_snapshot(&self) -> (Option<RebalanceMeta>, &'static str) {
let fallback = self.rebalance_meta.read().await.clone();
match self.rebalance_status_meta_snapshot().await {
Ok(snapshot) => (snapshot, "fresh"),
Err(err) => {
let status = if error_chain_has_rebalance_metadata_load_error(&err) {
"recovery_required"
} else if fallback.is_some() {
"stale"
} else {
"unknown"
};
(fallback, status)
}
}
}
/// Persisted rebalance metadata, or `None` when no rebalance was recorded.
async fn load_rebalance_status_meta(&self) -> Result<Option<RebalanceMeta>> {
let pool = clone_first_arc(&self.pools, "refresh_rebalance_status_meta: no pools available")?;
@@ -860,7 +880,7 @@ impl ECStore {
match persisted.load(pool).await {
Ok(()) => Ok(Some(persisted)),
Err(Error::ConfigNotFound) => Ok(None),
Err(err) => Err(Error::other(format!("rebalance metadata refresh failed during pool status: {err}"))),
Err(err) => Err(Error::other_with_context("rebalance metadata refresh failed during pool status", err)),
}
}
@@ -1447,6 +1467,21 @@ impl ECStore {
}
}
fn error_chain_has_rebalance_metadata_load_error(err: &(dyn std::error::Error + 'static)) -> bool {
let mut current = Some(err);
while let Some(error) = current {
if error.downcast_ref::<RebalanceMetadataLoadError>().is_some() {
return true;
}
current = if let Some(io) = error.downcast_ref::<std::io::Error>() {
io.get_ref().map(|source| source as &(dyn std::error::Error + 'static))
} else {
error.source()
};
}
false
}
#[cfg(test)]
mod tests {
use super::*;
+17 -4
View File
@@ -154,7 +154,8 @@ impl RebalanceMeta {
ver => return Err(rebalance_meta_load_unknown_version_error(ver)),
}
let meta: Self = rmp_serde::from_read(Cursor::new(&data[4..]))?;
let meta: Self = rmp_serde::from_read(Cursor::new(&data[4..]))
.map_err(|source| Error::other(RebalanceMetadataLoadError::Decode(source)))?;
*self = meta;
self.last_refreshed_at = Some(OffsetDateTime::now_utc());
@@ -259,16 +260,28 @@ pub(super) fn first_rebalance_bucket(pool_stat: &RebalanceStats) -> Option<Strin
pool_stat.buckets.first().cloned()
}
#[derive(Debug, thiserror::Error)]
pub(super) enum RebalanceMetadataLoadError {
#[error("rebalance metadata load failed: metadata payload is too short")]
PayloadTooShort,
#[error("rebalance metadata load failed: unknown format {0}")]
UnknownFormat(u16),
#[error("rebalance metadata load failed: unknown version {0}")]
UnknownVersion(u16),
#[error("rebalance metadata load failed: payload decode failed: {0}")]
Decode(#[source] rmp_serde::decode::Error),
}
pub(super) fn rebalance_meta_load_no_data_error() -> Error {
Error::other("rebalance metadata load failed: metadata payload is too short")
Error::other(RebalanceMetadataLoadError::PayloadTooShort)
}
pub(super) fn rebalance_meta_load_unknown_format_error(fmt: u16) -> Error {
Error::other(format!("rebalance metadata load failed: unknown format {fmt}"))
Error::other(RebalanceMetadataLoadError::UnknownFormat(fmt))
}
pub(super) fn rebalance_meta_load_unknown_version_error(ver: u16) -> Error {
Error::other(format!("rebalance metadata load failed: unknown version {ver}"))
Error::other(RebalanceMetadataLoadError::UnknownVersion(ver))
}
pub fn encode_rebalance_stop_propagation_record(record: &RebalanceStopPropagationRecord) -> String {
@@ -5752,6 +5752,19 @@ impl SetDisks {
return Ok(());
}
// A metadata update that misses write quorum is not rolled back, so
// refuse it before writing when the target disks cannot reach quorum
// even if every write succeeds; otherwise the change a caller reports
// as failed would be served and spread by heal.
let write_quorum = fi.write_quorum(self.default_write_quorum());
let reachable = disks
.iter()
.map(|disk| disk.is_none().then_some(DiskError::DiskNotFound))
.collect::<Vec<_>>();
if let Some(err) = reduce_write_quorum_errs(&reachable, OBJECT_OP_IGNORED_ERRS, write_quorum) {
return Err(err);
}
self.invalidate_get_object_metadata_cache(bucket, object).await;
let mut futures = Vec::with_capacity(disks.len());
@@ -5781,7 +5794,7 @@ impl SetDisks {
}
}
if let Some(err) = reduce_write_quorum_errs(&errs, OBJECT_OP_IGNORED_ERRS, fi.write_quorum(self.default_write_quorum())) {
if let Some(err) = reduce_write_quorum_errs(&errs, OBJECT_OP_IGNORED_ERRS, write_quorum) {
return Err(err);
}
@@ -6338,40 +6351,6 @@ impl SetDisks {
Ok(purged)
}
/// Reclaim metadata-less directory trees discovered by a recursive empty
/// bucket listing. The initial directory read fails closed if any slot is
/// unavailable; per-prefix purges then reuse the full cross-disk scan.
pub(crate) async fn purge_orphan_dir_objects_in_bucket(&self, bucket: &str) -> bool {
let disks = self.get_disks_internal().await;
if disks.is_empty() || disks.iter().any(Option::is_none) {
return false;
}
let mut prefixes = HashSet::new();
for disk in disks.iter().flatten() {
let entries = match disk.list_dir("", bucket, "", 0).await {
Ok(entries) => entries,
Err(_) => return false,
};
prefixes.extend(
entries
.into_iter()
.filter(|entry| entry.ends_with(SLASH_SEPARATOR) && is_safe_orphan_dir_entry(entry)),
);
}
let mut prefixes = prefixes.into_iter().collect::<Vec<_>>();
prefixes.sort_unstable();
let mut purged = false;
for prefix in prefixes {
match self.purge_orphan_dir_object(bucket, &prefix).await {
Ok(prefix_purged) => purged |= prefix_purged,
Err(_) => return false,
}
}
purged
}
/// Orphan cleanup must not infer a complete scan from only the online
/// subset of a set's disks.
pub(crate) async fn orphan_purge_has_complete_disk_set(&self) -> bool {
@@ -9474,7 +9453,7 @@ mod tests {
"lifecycle-inline-data-get-fanout-bucket",
"lifecycle-inline-data-get-object",
GET_OBJECT_PATH_LEGACY_DUPLEX,
None,
false,
);
}
@@ -9485,7 +9464,7 @@ mod tests {
"lifecycle-bounded-inline-data-get-fanout-bucket",
"lifecycle-bounded-inline-data-get-object",
GET_OBJECT_PATH_LEGACY_DUPLEX,
Some("true"),
true,
);
}
@@ -9496,7 +9475,7 @@ mod tests {
RUSTFS_META_BUCKET,
"buckets/.usage-cache/lifecycle-inline-data-get-object",
GET_OBJECT_PATH_INTERNAL_META,
None,
false,
);
}
@@ -9597,7 +9576,7 @@ mod tests {
bucket: &'static str,
object_prefix: &str,
expected_path: &'static str,
bounded_fanout_env: Option<&'static str>,
bounded_fanout: bool,
) {
const DISKS: usize = 4;
let object = object_with_initial_data_shards(bucket, object_prefix, 2, 3);
@@ -9617,16 +9596,26 @@ mod tests {
[
("RUSTFS_GET_METADATA_EARLY_STOP_ENABLE", Some("true")),
("RUSTFS_GET_METADATA_DATA_READ_EARLY_STOP_ENABLE", Some("true")),
("RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT", bounded_fanout_env),
(
"RUSTFS_GET_METADATA_EARLY_STOP_BOUNDED_FANOUT",
Some(if bounded_fanout { "true" } else { "false" }),
),
],
async {
let barrier_disk = if bounded_fanout_env.is_some() {
bounded_spare_disk_index(bucket, &object, 2, 2)
} else {
3
};
let barrier_disk = bounded_spare_disk_index(bucket, &object, 2, 2);
let barrier =
rename_fanout_barrier::arm(&object, barrier_disk, rename_fanout_barrier::PHASE_READ_VERSION);
// Keep one initial bounded read pending so its hedge is scheduled;
// eager fanout needs all quorum reads parked until the spare starts.
let quorum_disks = if bounded_fanout {
vec![bounded_metadata_fanout_order(bucket, &object, DISKS, 2)[0]]
} else {
(0..DISKS).filter(|&index| index != barrier_disk).collect()
};
let quorum_barriers = quorum_disks
.into_iter()
.map(|index| rename_fanout_barrier::arm(&object, index, rename_fanout_barrier::PHASE_READ_VERSION))
.collect::<Vec<_>>();
let disks_for_read = disks.clone();
let object_for_read = object.clone();
let mut read = tokio::spawn(async move {
@@ -9648,6 +9637,9 @@ mod tests {
tokio::time::timeout(BARRIER_PAUSE_GUARD, barrier.wait_until_paused())
.await
.expect("metadata fanout should pause the spare metadata task");
for quorum_barrier in &quorum_barriers {
quorum_barrier.release();
}
let (parts_metadata, errs, diagnostics) = tokio::time::timeout(BARRIER_PAUSE_GUARD, &mut read)
.await
.expect("verified inline quorum should abort the paused eager fanout")
@@ -597,10 +597,8 @@ mod tests {
.iter()
.map(|slot| slot.file_info().expect("every disk must have arrived").clone())
.collect::<Vec<_>>();
let (_, selected, selection_quorum) =
SetDisks::select_valid_fileinfo(&vec![None; 4], &completed_metadata, &vec![None; 4], "", 2, 3)
.expect("three matching B disks must select the current identity");
assert_eq!(selection_quorum, 3);
let (_, selected) = SetDisks::select_valid_fileinfo(&vec![None; 4], &completed_metadata, &vec![None; 4], 2)
.expect("three matching B disks must select the current identity");
assert_eq!(selected.version_id, expected_current.version_id);
assert_eq!(selected.mod_time, expected_current.mod_time);
assert_eq!(selected.data_dir, expected_current.data_dir);
+108 -24
View File
@@ -665,14 +665,20 @@ impl SetDisks {
read_quorum
}
pub(super) fn select_valid_fileinfo(
/// The commit-proof rule kept for internal metadata in the system bucket.
/// Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an
/// ordinary read as proof that a record committed: when enough copies
/// answer to prove write quorum, the latest version must hold it, and a
/// degraded read never serves a newer version below write quorum. A split
/// write therefore stays unreadable until native recovery resolves it.
fn select_committed_fileinfo(
disks: &[Option<DiskStore>],
parts_metadata: &[FileInfo],
errs: &[Option<DiskError>],
version_id: &str,
read_quorum: usize,
write_quorum: usize,
) -> disk::error::Result<(Vec<Option<DiskStore>>, FileInfo, usize)> {
) -> disk::error::Result<(Vec<Option<DiskStore>>, FileInfo)> {
let selection_quorum =
Self::latest_fileinfo_selection_quorum(version_id, parts_metadata, errs, read_quorum, write_quorum);
let (usable_metadata, has_read_error) = Self::usable_fileinfo_count(parts_metadata, errs);
@@ -683,14 +689,71 @@ impl SetDisks {
&& usable_metadata >= write_quorum
&& selection_quorum == read_quorum
{
let (online_disks, fi) = Self::pick_degraded_latest_fileinfo(disks, parts_metadata, errs, read_quorum, write_quorum)?;
return Ok((online_disks, fi, read_quorum));
return Self::pick_degraded_latest_fileinfo(disks, parts_metadata, errs, read_quorum, write_quorum);
}
let (online_disks, mod_time, etag) = Self::list_online_disks(disks, parts_metadata, errs, selection_quorum);
let fi = Self::pick_valid_fileinfo(parts_metadata, mod_time, etag, selection_quorum)?;
Ok((online_disks, fi, selection_quorum))
Ok((online_disks, fi))
}
/// Select the version a read of `bucket` serves: S3 buckets use MinIO's
/// read-quorum rule (`select_valid_fileinfo`); the system bucket keeps the
/// commit-proof rule (`select_committed_fileinfo`).
#[allow(clippy::too_many_arguments)]
pub(super) fn select_fileinfo_for_read(
bucket: &str,
disks: &[Option<DiskStore>],
parts_metadata: &[FileInfo],
errs: &[Option<DiskError>],
version_id: &str,
read_quorum: usize,
write_quorum: usize,
) -> disk::error::Result<(Vec<Option<DiskStore>>, FileInfo)> {
if crate::bucket::utils::is_meta_bucketname(bucket) {
Self::select_committed_fileinfo(disks, parts_metadata, errs, version_id, read_quorum, write_quorum)
} else {
Self::select_valid_fileinfo(disks, parts_metadata, errs, read_quorum)
}
}
/// Select the version an S3 read serves with MinIO's rule (`getObjectFileInfo`
/// -> `listOnlineDisks` -> `pickValidFileInfo`): the modification time
/// shared by the most copies wins when it reaches `read_quorum`, a tie
/// goes to the newer time, and one content identity at that time must
/// itself reach `read_quorum`. Heal picks the latest version the same
/// way, so a read returns the version heal converges the set to; a write
/// that failed after reaching exactly `read_quorum` copies is readable
/// instead of failing until heal runs, as listing already shows it.
///
/// `read_quorum` follows the parity most copies agree on, which delete
/// markers and empty versions can lower below the selected version's own
/// data blocks; like MinIO's `getLatestFileInfo`, a version with local
/// shards also needs that many copies, or HEAD would report a version
/// GET cannot decode.
pub(super) fn select_valid_fileinfo(
disks: &[Option<DiskStore>],
parts_metadata: &[FileInfo],
errs: &[Option<DiskError>],
read_quorum: usize,
) -> disk::error::Result<(Vec<Option<DiskStore>>, FileInfo)> {
let (online_disks, mod_time, etag) = Self::list_online_disks(disks, parts_metadata, errs, read_quorum);
let (fi, copies) = Self::find_file_info_in_quorum_counted(parts_metadata, &mod_time, &etag, read_quorum)?;
if !fi.deleted && !fi.is_remote() && fi.size > 0 && copies < fi.erasure.data_blocks {
// The shortcut counts byte-identical copies only; count every copy
// of the content identity before refusing the version.
let identity = Self::file_info_quorum_hash(&fi);
let copies = parts_metadata
.iter()
.filter(|meta| file_info_is_valid_for_metadata(meta) && Self::file_info_quorum_hash(meta) == identity)
.count();
if copies < fi.erasure.data_blocks {
return Err(DiskError::ErasureReadQuorum);
}
}
Ok((online_disks, fi))
}
pub(super) fn pick_valid_fileinfo(
@@ -1025,6 +1088,23 @@ impl SetDisks {
etag: &Option<String>,
quorum: usize,
) -> disk::error::Result<FileInfo> {
Self::find_file_info_in_quorum_counted(metas, mod_time, etag, quorum).map(|(fi, _)| fi)
}
/// `find_file_info_in_quorum`, also returning how many copies carry the
/// selected content identity.
///
/// Two identities can share the top count only in a data = parity set
/// split exactly in half at one modification time, such as a metadata
/// update that failed after reaching half the drives. The identity held
/// by the lowest-index disk wins, so every read and heal picks the same
/// side and a retried metadata write can repair the split.
fn find_file_info_in_quorum_counted(
metas: &[FileInfo],
mod_time: &Option<OffsetDateTime>,
etag: &Option<String>,
quorum: usize,
) -> disk::error::Result<(FileInfo, usize)> {
if quorum < 1 {
warn!("find_file_info_in_quorum: quorum < 1");
return Err(DiskError::ErasureReadQuorum);
@@ -1053,7 +1133,7 @@ impl SetDisks {
if same_count >= quorum && same_count > total_valid - same_count {
let mut fi = first.clone();
fi.is_latest = fi.successor_mod_time.is_none();
return Ok(fi);
return Ok((fi, same_count));
}
}
@@ -1106,15 +1186,12 @@ impl SetDisks {
*count_map.entry(hash).or_insert(0) += 1;
}
let mut max_val = None;
let mut max_count = 0;
for (&val, &count) in &count_map {
if count > max_count {
max_val = Some(val);
max_count = count;
}
}
let max_count = count_map.values().copied().max().unwrap_or(0);
let max_val = meta_hashes
.iter()
.flatten()
.find(|hash| count_map.get(*hash) == Some(&max_count))
.copied();
if max_count < quorum {
warn!(
@@ -1155,17 +1232,24 @@ impl SetDisks {
if found {
let mut fi = found_fi.expect("operation should succeed");
for (val, &count) in &valid_obj_map {
if count >= quorum {
fi.successor_mod_time = val.successor_mod_time;
fi.num_versions = val.num_versions;
fi.is_latest = val.successor_mod_time.is_none();
break;
}
// Scan in disk order so that two version-stack shapes with equal
// counts resolve the same way on every read.
let quorum_props = metas
.iter()
.zip(meta_hashes.iter())
.filter(|(meta, hash)| hash.is_some() && **hash == max_val && file_info_is_valid_for_metadata(meta))
.map(|(meta, _)| ObjProps {
successor_mod_time: meta.successor_mod_time,
num_versions: meta.num_versions,
})
.find(|props| valid_obj_map.get(props).is_some_and(|&count| count >= quorum));
if let Some(props) = quorum_props {
fi.successor_mod_time = props.successor_mod_time;
fi.num_versions = props.num_versions;
fi.is_latest = props.successor_mod_time.is_none();
}
return Ok(fi);
return Ok((fi, max_count));
}
warn!("find_file_info_in_quorum: fileinfo not found");
+279 -110
View File
@@ -129,7 +129,7 @@ use rustfs_filemeta::{
};
use rustfs_heal_contracts::heal_channel::{
DriveState, HealAdmissionResult, HealChannelPriority, HealItemType, HealOpts, HealRequestSource, HealScanMode,
send_heal_replacement_disk, send_heal_request_with_admission,
send_heal_request_with_admission,
};
use rustfs_io_metrics::{
record_object_lock_diag_acquire_duration, record_object_lock_diag_enabled, record_object_lock_diag_hold_duration,
@@ -1790,7 +1790,7 @@ mod prepared_get_object_metadata_tests {
assert!(original.is_complete());
let (original_metadata, original_errors, _) = original.into_legacy();
assert!(original_errors.iter().all(Option::is_none));
let (_, generation_a, _) = SetDisks::select_valid_fileinfo(&disks, &original_metadata, &original_errors, "", 2, 3)
let (_, generation_a) = SetDisks::select_valid_fileinfo(&disks, &original_metadata, &original_errors, 2)
.expect("generation A must have real metadata quorum");
assert!(!generation_a.inline_data(), "fixture must use non-inline shard files");
let order = bounded_metadata_fanout_order(bucket, &object, 4, 2);
@@ -1851,8 +1851,8 @@ mod prepared_get_object_metadata_tests {
assert!(replacement.is_complete());
let (replacement_metadata, replacement_errors, _) = replacement.into_legacy();
assert!(replacement_errors.iter().all(Option::is_none));
let (online, generation_b, _) =
SetDisks::select_valid_fileinfo(&disks, &replacement_metadata, &replacement_errors, "", 2, 3)
let (online, generation_b) =
SetDisks::select_valid_fileinfo(&disks, &replacement_metadata, &replacement_errors, 2)
.expect("replacement must have healthy real metadata quorum");
assert_eq!(online.iter().flatten().count(), 4);
assert_ne!(generation_a.data_dir, generation_b.data_dir);
@@ -6829,7 +6829,7 @@ impl SetDisks {
let (metadata, errors) = Self::read_all_fileinfo(&disks, "", bucket, object, version_id, false, false, false).await?;
// Match delete_object_version_with_purge, independently of data parity or online inventory size.
let quorum = self.set_drive_count / 2 + 1;
let (_, marker, _) = Self::select_valid_fileinfo(&disks, &metadata, &errors, version_id, quorum, quorum)
let (_, marker) = Self::select_valid_fileinfo(&disks, &metadata, &errors, quorum)
.map_err(|error| to_object_err(error.into(), vec![bucket, object, version_id]))?;
if !marker.deleted || marker.tier_free_version() || marker.version_id.unwrap_or_else(Uuid::nil) != requested_version {
return Err(Error::DataMovementOverwriteErr(
@@ -10475,13 +10475,6 @@ mod tests {
assert!(!purged, "an incomplete disk scan must not claim the tree is an orphan");
assert!(prefix_dir.join("nested/leaf").exists(), "online disk contents must remain untouched");
let bucket_purged = set.purge_orphan_dir_objects_in_bucket("bucket").await;
assert!(!bucket_purged, "bucket-wide cleanup must fail closed with an unavailable disk slot");
assert!(
prefix_dir.join("nested/leaf").exists(),
"bucket-wide cleanup must preserve the online tree"
);
}
// Cross-disk safety: if any drive still holds object data under the prefix, refuse
@@ -11976,41 +11969,121 @@ mod tests {
}
#[test]
fn test_latest_fileinfo_selection_quorum_requires_write_quorum_when_full_metadata_is_available() {
fn test_latest_fileinfo_selection_breaks_same_time_tie_in_disk_order() {
let mod_time = OffsetDateTime::now_utc();
let data_dir = Uuid::new_v4();
let metas = vec![
quorum_test_fileinfo(mod_time, data_dir, "part-etag-a", 1),
quorum_test_fileinfo(mod_time, data_dir, "part-etag-a", 2),
quorum_test_fileinfo(mod_time, data_dir, "part-etag-b", 3),
quorum_test_fileinfo(mod_time, data_dir, "part-etag-b", 4),
];
let tagged = |index, tag: &str| {
let mut fi = quorum_test_fileinfo(mod_time, data_dir, "part-etag", index);
fi.metadata.insert("x-amz-tagging".to_string(), tag.to_string());
fi
};
let errs = vec![None, None, None, None];
let quorum = SetDisks::latest_fileinfo_selection_quorum("", &metas, &errs, 2, 3);
// A tagging write that failed after reaching half the drives.
for (metas, expected) in [
(vec![tagged(1, "a=1"), tagged(2, "a=1"), tagged(3, "a=2"), tagged(4, "a=2")], "a=1"),
(vec![tagged(1, "a=2"), tagged(2, "a=1"), tagged(3, "a=1"), tagged(4, "a=2")], "a=2"),
] {
// Each call hashes into fresh maps; a map-order choice would flip.
for _ in 0..32 {
let (_, selected) = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, 2)
.expect("a same-time half split serves the side on the lowest-index disk");
let healed =
SetDisks::find_file_info_in_quorum(&metas, &Some(mod_time), &None, 2).expect("heal resolves the same split");
assert_eq!(quorum, 3);
assert_eq!(selected.metadata.get("x-amz-tagging").map(String::as_str), Some(expected));
assert_eq!(healed.metadata.get("x-amz-tagging").map(String::as_str), Some(expected));
}
}
}
#[test]
fn test_latest_fileinfo_selection_quorum_preserves_read_quorum_for_version_or_degraded_reads() {
let mod_time = OffsetDateTime::now_utc();
let data_dir = Uuid::new_v4();
let metas = vec![
quorum_test_fileinfo(mod_time, data_dir, "part-etag-a", 1),
quorum_test_fileinfo(mod_time, data_dir, "part-etag-a", 2),
FileInfo::default(),
FileInfo::default(),
];
let degraded_errs = vec![None, None, Some(DiskError::DiskNotFound), Some(DiskError::DiskNotFound)];
let clean_errs = vec![None, None, None, None];
fn test_latest_fileinfo_selection_requires_the_versions_own_data_blocks() {
let old_mod_time = OffsetDateTime::now_utc();
let new_mod_time = old_mod_time + time::Duration::seconds(1);
let old = |index| quorum_test_fileinfo(old_mod_time, Uuid::from_u128(1), "part-etag-old", index);
let new = |index| {
let mut fi = quorum_test_fileinfo(new_mod_time, Uuid::from_u128(2), "part-etag-new", index);
fi.erasure.data_blocks = 3;
fi.erasure.parity_blocks = 1;
fi
};
let errs = vec![None, None, None, None];
assert_eq!(SetDisks::latest_fileinfo_selection_quorum("", &metas, &degraded_errs, 2, 3), 2);
assert_eq!(SetDisks::latest_fileinfo_selection_quorum("version-id", &metas, &clean_errs, 2, 3), 2);
// Read quorum two comes from the 2+2 copies; the newer 3+1 version
// wins the time tie but cannot be decoded from two shards.
let metas = vec![old(1), old(2), new(3), new(4)];
let result = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, 2);
assert!(matches!(result, Err(DiskError::ErasureReadQuorum)));
let metas = vec![old(1), new(2), new(3), new(4)];
let (_, selected) = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, 2)
.expect("three copies decode a 3+1 version");
assert_eq!(selected.mod_time, Some(new_mod_time));
// Replication-status drift does not change the content identity, so
// the drifted copy still counts toward the version's data blocks.
let replication_status_key = format!(
"{}{}",
rustfs_utils::http::RUSTFS_INTERNAL_PREFIX,
rustfs_utils::http::SUFFIX_REPLICATION_STATUS
);
let mut drifted = new(4);
drifted
.metadata
.insert(replication_status_key, "target-a=PENDING;".to_string());
let metas = vec![old(1), new(2), new(3), drifted];
let (_, selected) = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, 2)
.expect("a replication-status drift still leaves three copies of the version");
assert_eq!(selected.mod_time, Some(new_mod_time));
}
#[tokio::test]
async fn update_object_meta_below_write_quorum_writes_nothing() {
use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
use crate::storage_api_contracts::object::ObjectIO as _;
let (_dirs, set) = crate::ecstore_validation_blackbox::make_local_set_disks(4, 2).await;
let (bucket, object) = ("metadata-write-quorum", "object");
set.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("bucket should be created");
set.put_object(
bucket,
object,
&mut crate::object_api::PutObjReader::from_vec(b"body".to_vec()),
&ObjectOptions::default(),
)
.await
.expect("object should commit on every drive");
let disks = set.get_disks_internal().await;
let (metadata, _) = SetDisks::read_all_fileinfo(&disks, "", bucket, object, "", false, false, false)
.await
.expect("committed metadata should be readable");
let mut update = metadata[0].clone();
update.metadata.insert("x-amz-meta-probe".to_string(), "written".to_string());
// A read that selected one side of a 2+2 split returns two disks.
let mut targets = disks.clone();
targets[2] = None;
targets[3] = None;
let err = set
.update_object_meta(bucket, object, update, &targets)
.await
.expect_err("two of four disks cannot reach write quorum");
assert_eq!(err, DiskError::ErasureWriteQuorum);
let (after, _) = SetDisks::read_all_fileinfo(&disks, "", bucket, object, "", false, false, false)
.await
.expect("metadata should stay readable");
assert!(
after.iter().all(|fi| !fi.metadata.contains_key("x-amz-meta-probe")),
"a refused update must not reach any disk"
);
}
#[test]
fn test_latest_fileinfo_selection_quorum_keeps_read_quorum_for_partial_overwrite_with_read_error() {
fn test_latest_fileinfo_selection_keeps_older_majority_over_partial_overwrite() {
let old_mod_time = OffsetDateTime::now_utc();
let new_mod_time = old_mod_time + time::Duration::seconds(1);
let old_data_dir = Uuid::new_v4();
@@ -12023,26 +12096,15 @@ mod tests {
];
let errs = vec![None, None, None, Some(DiskError::DiskNotFound)];
let quorum = SetDisks::latest_fileinfo_selection_quorum("", &metas, &errs, 2, 3);
let (online_disks, mod_time, etag) = SetDisks::list_online_disks(&vec![None; metas.len()], &metas, &errs, quorum);
let fi = SetDisks::pick_valid_fileinfo(&metas, mod_time, etag, quorum)
.expect("old metadata should remain readable with read quorum");
assert_eq!(quorum, 2);
assert_eq!(online_disks.len(), metas.len());
assert_eq!(fi.data_dir, Some(old_data_dir));
assert_eq!(fi.parts[0].etag, "part-etag-old");
let (_, selected, selected_quorum) = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, "", 2, 3)
let (_, selected) = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, 2)
.expect("old metadata should remain selectable with read quorum");
assert_eq!(selected_quorum, 2);
assert_eq!(selected.data_dir, Some(old_data_dir));
assert_eq!(selected.parts[0].etag, "part-etag-old");
assert!(selected.is_latest);
}
#[test]
fn test_latest_fileinfo_selection_rejects_partial_latest_read_quorum_with_read_error() {
fn test_latest_fileinfo_selection_serves_newer_read_quorum_with_read_error() {
let old_mod_time = OffsetDateTime::now_utc();
let new_mod_time = old_mod_time + time::Duration::seconds(1);
let old_data_dir = Uuid::new_v4();
@@ -12055,9 +12117,120 @@ mod tests {
];
let errs = vec![None, None, None, Some(DiskError::DiskNotFound)];
let result = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, "", 2, 3);
// The new version may hold write quorum through the missing disk; the
// old one cannot, and heal converges the set to the new version.
let (_, selected) = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, 2)
.expect("the newer version reaches read quorum");
assert!(matches!(result, Err(DiskError::ErasureReadQuorum)));
assert_eq!(selected.data_dir, Some(new_data_dir));
assert_eq!(selected.mod_time, Some(new_mod_time));
}
#[test]
fn test_latest_fileinfo_selection_serves_newer_side_of_complete_split() {
let old_mod_time = OffsetDateTime::now_utc();
let new_mod_time = old_mod_time + time::Duration::seconds(1);
let old_data_dir = Uuid::new_v4();
let new_data_dir = Uuid::new_v4();
let old = |index| quorum_test_fileinfo(old_mod_time, old_data_dir, "part-etag-old", index);
let new = |index| quorum_test_fileinfo(new_mod_time, new_data_dir, "part-etag-new", index);
let errs = vec![None, None, None, None];
for metas in [vec![old(1), old(2), new(3), new(4)], vec![new(1), old(2), new(3), old(4)]] {
let (_, selected) = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, 2)
.expect("a 2+2 split with every disk answering serves the newer version");
assert_eq!(selected.data_dir, Some(new_data_dir));
assert_eq!(selected.parts[0].etag, "part-etag-new");
assert!(selected.is_latest);
}
}
#[test]
fn test_system_bucket_reads_keep_the_commit_proof_rule() {
let old_mod_time = OffsetDateTime::now_utc();
let new_mod_time = old_mod_time + time::Duration::seconds(1);
let old_data_dir = Uuid::new_v4();
let new_data_dir = Uuid::new_v4();
let old = |index| quorum_test_fileinfo(old_mod_time, old_data_dir, "part-etag-old", index);
let new = |index| quorum_test_fileinfo(new_mod_time, new_data_dir, "part-etag-new", index);
// A PREPARE/COMMIT split of pool metadata must stay unreadable so that
// native recovery, not an ordinary read, decides whether it committed.
for (metas, errs) in [
(vec![old(1), old(2), new(3), new(4)], vec![None, None, None, None]),
(
vec![new(1), new(2), old(3), FileInfo::default()],
vec![None, None, None, Some(DiskError::DiskNotFound)],
),
] {
let disks = vec![None; metas.len()];
let (_, served) = SetDisks::select_fileinfo_for_read("bucket", &disks, &metas, &errs, "", 2, 3)
.expect("an S3 read serves the newer side at read quorum");
assert_eq!(served.data_dir, Some(new_data_dir));
let internal = SetDisks::select_fileinfo_for_read(crate::disk::RUSTFS_META_BUCKET, &disks, &metas, &errs, "", 2, 3);
assert!(matches!(internal, Err(DiskError::ErasureReadQuorum)));
}
let committed = vec![new(1), new(2), new(3), old(4)];
let (_, internal) = SetDisks::select_fileinfo_for_read(
crate::disk::RUSTFS_META_BUCKET,
&vec![None; committed.len()],
&committed,
&[None, None, None, None],
"",
2,
3,
)
.expect("a write-quorum version stays readable in the system bucket");
assert_eq!(internal.data_dir, Some(new_data_dir));
}
#[test]
fn test_latest_fileinfo_selection_picks_version_stack_shape_in_disk_order() {
let mod_time = OffsetDateTime::now_utc();
let data_dir = Uuid::new_v4();
let successor = mod_time + time::Duration::seconds(1);
let shape = |index, successor_mod_time: Option<OffsetDateTime>, num_versions| {
let mut fi = quorum_test_fileinfo(mod_time, data_dir, "part-etag", index);
fi.successor_mod_time = successor_mod_time;
fi.is_latest = successor_mod_time.is_none();
fi.num_versions = num_versions;
fi
};
let errs = vec![None, None, None, None];
for (metas, expected) in [
(
vec![
shape(1, Some(successor), 2),
shape(2, None, 1),
shape(3, Some(successor), 2),
shape(4, None, 1),
],
Some(successor),
),
(
vec![
shape(1, None, 1),
shape(2, Some(successor), 2),
shape(3, None, 1),
shape(4, Some(successor), 2),
],
None,
),
] {
// Each call hashes into fresh maps; a map-order choice would flip.
for _ in 0..32 {
let (_, selected) = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, 2)
.expect("one content identity reaches read quorum");
assert_eq!(selected.successor_mod_time, expected);
assert_eq!(selected.is_latest, expected.is_none());
assert_eq!(selected.num_versions, if expected.is_none() { 1 } else { 2 });
}
}
}
#[test]
@@ -12075,11 +12248,9 @@ mod tests {
let metas = vec![first, second, FileInfo::default(), FileInfo::default()];
let errs = vec![None, None, Some(DiskError::DiskNotFound), Some(DiskError::DiskNotFound)];
let (_, mut selected, selected_quorum) =
SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, "", 2, 3)
.expect("read quorum should remain enough when no competing latest is visible");
let (_, mut selected) = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, 2)
.expect("read quorum should remain enough when no competing latest is visible");
assert_eq!(selected_quorum, 2);
assert_eq!(selected.data_dir, Some(data_dir));
assert_eq!(selected.parts[0].etag, "part-etag-old");
assert!(selected.parts[0].checksums.is_none());
@@ -12106,7 +12277,7 @@ mod tests {
let metas = vec![first, second, FileInfo::default(), FileInfo::default()];
let errs = vec![None, None, Some(DiskError::DiskNotFound), Some(DiskError::DiskNotFound)];
let (_, mut selected, _) = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, "", 2, 3)
let (_, mut selected) = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, 2)
.expect("winner selection should defer sidecar decoding");
let err = SetDisks::hydrate_selected_fileinfo_part_checksums(&mut selected)
.expect_err("a malformed degraded winner must fail closed when checksums are requested");
@@ -12173,10 +12344,9 @@ mod tests {
let metas = vec![latest_meta, stale_stack_meta, newer_stack_meta, FileInfo::default()];
let errs = vec![None, None, None, Some(DiskError::DiskNotFound)];
let (_, selected, selected_quorum) = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, "", 2, 3)
let (_, selected) = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, 2)
.expect("same object version should stay readable despite derived version stack drift");
assert_eq!(selected_quorum, 3);
assert_eq!(selected.data_dir, Some(data_dir));
assert_eq!(selected.parts[0].etag, "part-etag");
assert_eq!(selected.mod_time, Some(mod_time));
@@ -12269,81 +12439,77 @@ mod tests {
let metas = vec![meta_a, meta_b, meta_c, FileInfo::default()];
let errs = vec![None, None, None, Some(DiskError::DiskNotFound)];
let (_, selected, selected_quorum) = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, "", 2, 3)
let (_, selected) = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, 2)
.expect("replication status drift should not split readable object identity");
assert_eq!(selected_quorum, 3);
assert_eq!(selected.data_dir, Some(data_dir));
assert_eq!(selected.parts[0].etag, "part-etag");
}
#[test]
fn test_latest_fileinfo_selection_rejects_same_modtime_metadata_split_without_write_quorum() {
fn test_latest_fileinfo_selection_serves_same_time_metadata_majority() {
let mod_time = OffsetDateTime::now_utc();
let data_dir = Uuid::new_v4();
let mut old_meta_a = quorum_test_fileinfo(mod_time, data_dir, "part-etag", 1);
let mut old_meta_b = quorum_test_fileinfo(mod_time, data_dir, "part-etag", 2);
let mut partial_meta = quorum_test_fileinfo(mod_time, data_dir, "part-etag", 3);
old_meta_a.metadata.insert("x-amz-meta-color".to_string(), "blue".to_string());
old_meta_b.metadata.insert("x-amz-meta-color".to_string(), "blue".to_string());
partial_meta
.metadata
.insert("x-amz-meta-color".to_string(), "red".to_string());
let metas = vec![old_meta_a, old_meta_b, partial_meta, FileInfo::default()];
let colored = |index, color: &str| {
let mut fi = quorum_test_fileinfo(mod_time, data_dir, "part-etag", index);
fi.metadata.insert("x-amz-meta-color".to_string(), color.to_string());
fi
};
let errs = vec![None, None, None, Some(DiskError::DiskNotFound)];
let quorum = SetDisks::latest_fileinfo_selection_quorum("", &metas, &errs, 2, 3);
let result = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, "", 2, 3);
// With one disk missing, only the two-copy side can hold write quorum,
// whichever disk order the copies arrive in.
for (metas, expected) in [
(
vec![colored(1, "blue"), colored(2, "blue"), colored(3, "red"), FileInfo::default()],
"blue",
),
(vec![colored(1, "blue"), colored(2, "red"), colored(3, "red"), FileInfo::default()], "red"),
] {
let (_, selected) = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, 2)
.expect("the read-quorum side of a same-time metadata split is served");
assert_eq!(quorum, 2);
assert!(matches!(result, Err(DiskError::ErasureReadQuorum)));
assert_eq!(selected.metadata.get("x-amz-meta-color").map(String::as_str), Some(expected));
}
}
#[test]
fn test_latest_fileinfo_selection_rejects_same_modtime_partial_metadata_read_quorum() {
fn test_latest_fileinfo_selection_serves_same_time_transition_majority() {
let mod_time = OffsetDateTime::now_utc();
let data_dir = Uuid::new_v4();
let mut old_meta = quorum_test_fileinfo(mod_time, data_dir, "part-etag", 1);
let mut partial_meta_a = quorum_test_fileinfo(mod_time, data_dir, "part-etag", 2);
let mut partial_meta_b = quorum_test_fileinfo(mod_time, data_dir, "part-etag", 3);
old_meta.metadata.insert("x-amz-meta-color".to_string(), "blue".to_string());
partial_meta_a
.metadata
.insert("x-amz-meta-color".to_string(), "red".to_string());
partial_meta_b
.metadata
.insert("x-amz-meta-color".to_string(), "red".to_string());
let metas = vec![old_meta, partial_meta_a, partial_meta_b, FileInfo::default()];
let local = |index| quorum_test_fileinfo(mod_time, data_dir, "part-etag", index);
let remote = |index| {
let mut fi = quorum_test_fileinfo(mod_time, data_dir, "part-etag", index);
fi.transition_status = TRANSITION_COMPLETE.to_string();
fi.transition_tier = "WARM".to_string();
fi.transitioned_objname = "remote/object".to_string();
fi.transition_version_id = Some(Uuid::from_u128(7));
fi
};
let errs = vec![None, None, None, Some(DiskError::DiskNotFound)];
let result = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, "", 2, 3);
for (metas, expect_remote) in [
(vec![local(1), local(2), remote(3), FileInfo::default()], false),
(vec![local(1), remote(2), remote(3), FileInfo::default()], true),
] {
let (_, selected) = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, 2)
.expect("the read-quorum side of a same-time transition split is served");
assert!(matches!(result, Err(DiskError::ErasureReadQuorum)));
assert_eq!(selected.is_remote(), expect_remote);
}
for (tied, expect_remote) in [
(vec![local(1), local(2), remote(3), remote(4)], false),
(vec![remote(1), local(2), local(3), remote(4)], true),
] {
let (_, selected) = SetDisks::select_valid_fileinfo(&vec![None; tied.len()], &tied, &[None, None, None, None], 2)
.expect("a same-time half split serves the side on the lowest-index disk");
assert_eq!(selected.is_remote(), expect_remote);
}
}
#[test]
fn test_latest_fileinfo_selection_rejects_same_modtime_transition_split_without_write_quorum() {
let mod_time = OffsetDateTime::now_utc();
let data_dir = Uuid::new_v4();
let old_meta_a = quorum_test_fileinfo(mod_time, data_dir, "part-etag", 1);
let old_meta_b = quorum_test_fileinfo(mod_time, data_dir, "part-etag", 2);
let mut partial_meta = quorum_test_fileinfo(mod_time, data_dir, "part-etag", 3);
partial_meta.transition_status = TRANSITION_COMPLETE.to_string();
partial_meta.transition_tier = "WARM".to_string();
partial_meta.transitioned_objname = "remote/object".to_string();
partial_meta.transition_version_id = Some(Uuid::new_v4());
let metas = vec![old_meta_a, old_meta_b, partial_meta, FileInfo::default()];
let errs = vec![None, None, None, Some(DiskError::DiskNotFound)];
let quorum = SetDisks::latest_fileinfo_selection_quorum("", &metas, &errs, 2, 3);
let result = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, "", 2, 3);
assert_eq!(quorum, 2);
assert!(matches!(result, Err(DiskError::ErasureReadQuorum)));
}
#[test]
fn test_latest_fileinfo_selection_quorum_uses_write_quorum_for_degraded_committed_identity() {
fn test_latest_fileinfo_selection_serves_degraded_committed_identity() {
let mod_time = OffsetDateTime::now_utc();
let data_dir = Uuid::new_v4();
let metas = vec![
@@ -12354,7 +12520,10 @@ mod tests {
];
let errs = vec![None, None, None, Some(DiskError::DiskNotFound)];
assert_eq!(SetDisks::latest_fileinfo_selection_quorum("", &metas, &errs, 2, 3), 3);
let (_, selected) = SetDisks::select_valid_fileinfo(&vec![None; metas.len()], &metas, &errs, 2)
.expect("a committed identity stays readable with one disk missing");
assert_eq!(selected.parts[0].etag, "part-etag-a");
}
#[test]
+228 -30
View File
@@ -21,19 +21,22 @@
use super::super::{
Arc, DiskError, DiskInfo, DiskInfoOptions, DiskOption, DiskStore, Endpoint, Error, FormatV3, HealChannelPriority, LockResult,
NamespaceLock, NamespaceLockWrapper, ObjectKey, Result, SetDisks, StorageError, debug, disk, info, load_format_erasure,
send_heal_replacement_disk, warn,
NamespaceLock, NamespaceLockWrapper, ObjectKey, Result, SetDisks, StorageError, debug, disk, load_format_erasure, warn,
};
use crate::disk::health_state::DriveMembershipSnapshot;
use crate::disk::{DiskAPI, new_disk};
use crate::layout::disks_layout::MAX_ERASURE_SET_DRIVE_COUNT;
use crate::runtime::sources as runtime_sources;
use futures::{StreamExt, stream};
use futures::{StreamExt, stream, stream::FuturesUnordered};
use rand::prelude::SliceRandom;
use rustfs_heal_contracts::heal_channel::{create_auto_replacement_disks_request, send_heal_request};
use rustfs_utils::path::object_name_disk_identity;
#[cfg(test)]
use uuid::Uuid;
// Bound batching when another reconnect is still pending.
const REPLACEMENT_BATCH_WAIT: std::time::Duration = std::time::Duration::from_secs(1);
#[async_trait::async_trait]
impl crate::storage_api_contracts::namespace::NamespaceLocking for SetDisks {
type Error = Error;
@@ -329,33 +332,82 @@ impl SetDisks {
async fn connect_disks_with_active_monitoring(&self, active_monitoring: bool) {
let disks = self.get_disks_internal().await;
// A slow failed endpoint must not hold back other returning drives.
// Keep the fanout within one set and release the inventory lock before I/O.
stream::iter(disks.into_iter().enumerate())
.for_each_concurrent(MAX_ERASURE_SET_DRIVE_COUNT, |(i, opdisk)| async move {
// Reconnect drives concurrently without holding the inventory lock.
let mut reconnects = stream::iter(disks.into_iter().enumerate())
.map(|(i, opdisk)| async move {
if let Some(disk) = opdisk {
if !active_monitoring
&& disk.get_disk_location().set_idx.is_some()
&& let disk::Disk::Local(local) = disk.as_ref()
&& local.can_skip_background_probe().await
{
return;
return None;
}
if disk.is_online().await && disk.get_disk_location().set_idx.is_some() {
return;
return None;
}
let _ = disk.close().await;
}
if let Some(endpoint) = self.set_endpoints.get(i) {
self.renew_disk(endpoint).await;
}
let endpoint = self.set_endpoints.get(i)?;
self.reconnect_disk(endpoint).await.then(|| endpoint.to_string())
})
.await;
.buffer_unordered(MAX_ERASURE_SET_DRIVE_COUNT);
let mut submissions = FuturesUnordered::new();
let mut batch = Vec::new();
let mut deadline = None;
let mut reconnects_done = false;
loop {
if reconnects_done && submissions.is_empty() {
break;
}
tokio::select! {
result = reconnects.next(), if !reconnects_done => match result {
Some(Some(endpoint)) => {
batch.push(endpoint);
deadline.get_or_insert_with(|| tokio::time::Instant::now() + REPLACEMENT_BATCH_WAIT);
}
Some(None) => {}
None => {
reconnects_done = true;
if !batch.is_empty() {
submissions.push(self.submit_replacement_heal(std::mem::take(&mut batch)));
}
deadline = None;
}
},
_ = tokio::time::sleep_until(deadline.unwrap_or_else(tokio::time::Instant::now)), if deadline.is_some() => {
submissions.push(self.submit_replacement_heal(std::mem::take(&mut batch)));
deadline = None;
}
_ = submissions.next(), if !submissions.is_empty() => {}
}
}
}
async fn submit_replacement_heal(&self, replacements: Vec<String>) {
if replacements.is_empty() {
return;
}
// Queue admission deduplicates by set, so publish ready targets together.
let request = create_auto_replacement_disks_request(
self.pool_index,
self.set_index,
replacements,
Some(HealChannelPriority::Normal),
);
let _ = send_heal_request(request).await;
}
pub async fn renew_disk(&self, ep: &Endpoint) {
if self.reconnect_disk(ep).await {
self.submit_replacement_heal(vec![ep.to_string()]).await;
}
}
/// Return whether this reconnect attached a verified unformatted target.
async fn reconnect_disk(&self, ep: &Endpoint) -> bool {
debug!("renew_disk: start {:?}", ep);
let previous_disk = {
@@ -375,28 +427,17 @@ impl SetDisks {
Err(e) => {
warn!("renew_disk: connect_endpoint err {:?}", &e);
if !matches!(e, DiskError::UnformattedDisk | DiskError::Io(_)) {
return;
return false;
}
let attached = match self.attach_unformatted_replacement_disk(ep).await {
Ok(attached) => attached,
Err(err) => {
warn!(endpoint = %ep, error = ?err, "renew_disk: unformatted replacement probe failed");
return;
return false;
}
};
if !attached {
return;
}
info!("renew_disk attached unformatted replacement and will trigger heal_disk, {:?}", ep);
let (Ok(pool_index), Ok(set_index)) = (usize::try_from(ep.pool_idx), usize::try_from(ep.set_idx)) else {
warn!("renew_disk: replacement target has invalid pool or set index, {:?}", ep);
return;
};
let _ =
send_heal_replacement_disk(pool_index, set_index, ep.to_string(), Some(HealChannelPriority::Normal)).await;
return;
return attached;
}
};
@@ -404,7 +445,7 @@ impl SetDisks {
Ok(res) => res,
Err(e) => {
warn!("renew_disk: find_disk_index err {:?}", e);
return;
return false;
}
};
@@ -422,7 +463,7 @@ impl SetDisks {
expected_set_index = self.set_index,
"renew_disk rejected a drive whose endpoint and format do not identify the same topology slot"
);
return;
return false;
}
let _ = new_disk.set_disk_id(Some(fm.erasure.this)).await;
@@ -442,7 +483,7 @@ impl SetDisks {
new_disk.enable_health_check();
*slot = Some(new_disk);
}
return;
return false;
}
new_disk.enable_health_check();
@@ -464,6 +505,7 @@ impl SetDisks {
let mut disk_lock = self.disks.write().await;
disk_lock[disk_idx] = Some(new_disk);
false
}
/// Attach a replacement target only after proving that the exact local slot
@@ -1102,6 +1144,162 @@ mod tests {
drop(temp_dirs);
}
#[tokio::test]
async fn reconnect_sweep_collects_all_blank_targets_and_excludes_original_disks() {
use rustfs_heal_contracts::heal_channel::{HealAdmissionResult, HealChannelCommand, init_heal_channel};
const CHILD: &str = "RUSTFS_TEST_GROUPED_RECONNECT_CHILD";
if std::env::var_os(CHILD).is_none() {
// The heal sender is process-global; keep this receiver isolated.
let output = std::process::Command::new(std::env::current_exe().expect("test executable"))
.args([
"--exact",
"set_disk::ops::locking::tests::reconnect_sweep_collects_all_blank_targets_and_excludes_original_disks",
"--nocapture",
])
.env(CHILD, "1")
.output()
.expect("isolated reconnect test should run");
let stdout = String::from_utf8_lossy(&output.stdout);
let stderr = String::from_utf8_lossy(&output.stderr);
assert!(output.status.success(), "isolated reconnect test failed: {stdout} {stderr}");
assert!(stdout.contains("1 passed"), "child must execute the reconnect test: {stdout}");
return;
}
let mut heal_rx = init_heal_channel().expect("isolated heal receiver");
let disk_count = 4;
let format = FormatV3::new(1, disk_count);
let mut directories = Vec::new();
let mut endpoints = Vec::new();
let mut disks = Vec::new();
for disk_idx in 0..2 {
let (directory, endpoint, disk) = make_formatted_local_disk(disk_idx, &format).await;
directories.push(directory);
endpoints.push(endpoint);
disks.push(Some(disk));
}
for disk_idx in 2..disk_count {
let directory = tempfile::tempdir().expect("replacement directory");
let mut endpoint =
Endpoint::try_from(directory.path().to_str().expect("UTF-8 replacement path")).expect("replacement endpoint");
endpoint.set_pool_index(0);
endpoint.set_set_index(0);
endpoint.set_disk_index(disk_idx);
directories.push(directory);
endpoints.push(endpoint);
disks.push(None);
}
let mut ready_targets = endpoints[2..].iter().map(ToString::to_string).collect::<Vec<_>>();
ready_targets.sort_unstable();
// Disk 1 returns with its format and data; disks 2 and 3 are blank.
disks[1]
.take()
.expect("original disk handle")
.close()
.await
.expect("close original disk");
let retained_data = directories[1].path().join("retained-data");
tokio::fs::write(&retained_data, b"original data")
.await
.expect("write original data");
let set = Arc::new(
SetDisks::new(
"reconnect-sweep".to_string(),
Arc::new(RwLock::new(disks)),
disk_count,
2,
0,
0,
endpoints.clone(),
format,
Vec::new(),
)
.await,
);
// Block disk 0's format check, then make it a late blank replacement.
let slow = set.get_disks_internal().await[0].clone().expect("original disk");
assert!(slow.is_online().await, "initialize original disk identity");
let disk::Disk::Local(wrapper) = slow.as_ref() else {
panic!("fixture must use a local disk");
};
let slow_local = wrapper.get_disk();
let mut slow_format_guard = slow_local.format_info.write().await;
slow_format_guard.last_check = None;
std::fs::remove_file(
directories[0]
.path()
.join(disk::RUSTFS_META_BUCKET)
.join(disk::FORMAT_CONFIG_FILE),
)
.expect("make the blocked disk a blank replacement");
let sweep = tokio::spawn({
let set = set.clone();
async move { set.connect_disks_with_active_monitoring(true).await }
});
tokio::time::timeout(std::time::Duration::from_secs(10), async {
loop {
let attached = set.get_disks_internal().await;
if attached[1..].iter().all(Option::is_some) {
break;
}
tokio::task::yield_now().await;
}
})
.await
.expect("other drives must reconnect while the first drive is blocked");
let command = tokio::time::timeout(std::time::Duration::from_secs(10), heal_rx.recv())
.await
.expect("ready targets must be submitted while a drive is blocked")
.expect("heal receiver should remain open");
let HealChannelCommand::Start { request, response_tx } = command else {
panic!("expected a replacement heal request");
};
assert_eq!(request.heal_endpoints, ready_targets, "ready blank disks must share one request");
assert!(!sweep.is_finished(), "the blocked drive must still be pending");
drop(slow_format_guard);
// Do not acknowledge the first request yet: a delayed admission
// response must not stop the late drive reconnecting and submitting.
let late_command = tokio::time::timeout(std::time::Duration::from_secs(10), heal_rx.recv())
.await
.expect("late target must be submitted while the first response is pending")
.expect("heal receiver should remain open");
let HealChannelCommand::Start {
request: late_request,
response_tx: late_response,
} = late_command
else {
panic!("expected a late replacement heal request");
};
assert_eq!(late_request.heal_endpoints, [endpoints[0].to_string()]);
response_tx
.send(Ok(HealAdmissionResult::Accepted))
.expect("acknowledge ready targets");
late_response
.send(Ok(HealAdmissionResult::Accepted))
.expect("acknowledge late target");
sweep.await.expect("reconnect sweep should finish");
assert!(heal_rx.try_recv().is_err(), "no duplicate request should be sent");
let attached = set.get_disks_internal().await;
for (index, disk) in attached.iter().enumerate() {
let disk = disk.as_ref().expect("each verified endpoint should be attached");
let format = load_format_erasure(disk, false).await;
if index == 1 {
assert!(format.is_ok(), "original disk must retain its format");
} else {
assert_eq!(format.expect_err("replacement must remain unformatted"), DiskError::UnformattedDisk);
assert!(!disk.health_check_enabled_for_test(), "format must precede replacement health checks");
}
disk.close().await.expect("close test disk");
}
assert_eq!(tokio::fs::read(&retained_data).await.expect("read original data"), b"original data");
let local_map = runtime_sources::local_disk_map_handle();
let mut local_map = local_map.write().await;
for endpoint in endpoints {
local_map.remove(&endpoint.to_string());
}
}
#[tokio::test]
async fn renew_disk_attaches_only_a_verified_local_unformatted_replacement() {
let disk_count = 4;
+5 -4
View File
@@ -9320,8 +9320,8 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
};
let src_version_id = src_opts.version_id.as_deref().unwrap_or_default();
let (online_disks, mut fi, _) =
Self::select_valid_fileinfo(&disks, &metas, &errs, src_version_id, read_quorum, write_quorum)
let (online_disks, mut fi) =
Self::select_fileinfo_for_read(src_bucket, &disks, &metas, &errs, src_version_id, read_quorum, write_quorum)
.map_err(|e| to_object_err(e.into(), vec![src_bucket, src_object]))?;
if fi.deleted {
@@ -11182,8 +11182,9 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
.map_err(|_| to_object_err(DiskError::ErasureWriteQuorum.into(), vec![bucket, object]))?;
let version_id = opts.version_id.as_deref().unwrap_or_default();
let (online_disks, mut fi, _) = Self::select_valid_fileinfo(&disks, &metas, &errs, version_id, read_quorum, write_quorum)
.map_err(|e| to_object_err(e.into(), vec![bucket, object]))?;
let (online_disks, mut fi) =
Self::select_fileinfo_for_read(bucket, &disks, &metas, &errs, version_id, read_quorum, write_quorum)
.map_err(|e| to_object_err(e.into(), vec![bucket, object]))?;
if fi.deleted {
return Err(to_object_err(Error::MethodNotAllowed, vec![bucket, object]));
+14 -14
View File
@@ -966,8 +966,9 @@ impl SetDisks {
return Err(to_object_err(err.into(), vec![bucket, object, &vid]));
}
let (op_online_disks, mut fi, fileinfo_selection_quorum) =
match Self::select_valid_fileinfo(&disks, &parts_metadata, &errs, vid.as_str(), read_quorum, write_quorum) {
let (op_online_disks, mut fi) =
match Self::select_fileinfo_for_read(bucket, &disks, &parts_metadata, &errs, vid.as_str(), read_quorum, write_quorum)
{
Ok(selection) => selection,
Err(err) => {
let err = StorageError::from(err);
@@ -987,7 +988,7 @@ impl SetDisks {
rustfs_utils::http::remove_str(&mut metadata.metadata, rustfs_utils::http::SUFFIX_PART_CHECKSUMS);
}
}
metadata_fanout_diagnostics.record_quorum_candidate_latency(metadata_metrics_path, fileinfo_selection_quorum);
metadata_fanout_diagnostics.record_quorum_candidate_latency(metadata_metrics_path, read_quorum);
let needs_late_metadata_fanout = !metadata_fanout_complete
&& allow_early_stop
&& non_inline_data_read_early_stop_allowed(read_data, bucket, object)
@@ -1014,7 +1015,6 @@ impl SetDisks {
&& allow_early_stop
&& metadata_fanout_diagnostics.valid_responses() >= read_quorum
&& errs.iter().all(Option::is_none)
&& fileinfo_selection_quorum >= read_quorum
&& op_online_disks.iter().filter(|disk| disk.is_some()).count() >= read_quorum))
{
#[cfg(test)]
@@ -2465,8 +2465,8 @@ impl SetDisks {
return Err(to_object_err(err.into(), vec![bucket, object]));
}
let (mut online_disks, full_fi, _) =
SetDisks::select_valid_fileinfo(fallback_disks, &parts_metadata, &errs, "", read_quorum, write_quorum)?;
let (mut online_disks, full_fi) =
SetDisks::select_fileinfo_for_read(bucket, fallback_disks, &parts_metadata, &errs, "", read_quorum, write_quorum)?;
if !late_metadata_read_identity_matches(expected, &full_fi) {
return Err(to_object_err(DiskError::ErasureReadQuorum.into(), vec![bucket, object]));
}
@@ -4052,8 +4052,7 @@ mod metadata_cache_tests {
assert!(original.is_complete());
let (baseline, errors, _) = original.into_legacy();
assert!(errors.iter().all(Option::is_none), "original real shards must all be readable");
let (_, selected, _) =
SetDisks::select_valid_fileinfo(&disks, &baseline, &errors, "", 2, 3).expect("original metadata quorum");
let (_, selected) = SetDisks::select_valid_fileinfo(&disks, &baseline, &errors, 2).expect("original metadata quorum");
assert!(!selected.inline_data(), "late fixture must retain real non-inline shard files");
let expected = LateMetadataIdentity::from_file_info(&selected);
let (_, _, original_online) =
@@ -4086,7 +4085,7 @@ mod metadata_cache_tests {
.expect("read changed real disk metadata");
assert!(changed.is_complete(), "late fixture must reach complete metadata observation");
let (metadata, errors, _) = changed.into_legacy();
let (online, selected, _) = SetDisks::select_valid_fileinfo(&disks, &metadata, &errors, "", 2, 3)
let (online, selected) = SetDisks::select_valid_fileinfo(&disks, &metadata, &errors, 2)
.expect("changed metadata still reaches selection quorum");
assert_eq!(online.iter().flatten().count(), 4, "the late identity guard must decide this failure");
if changed_generation {
@@ -5135,7 +5134,7 @@ mod metadata_cache_tests {
// This seam prepares a real, independently validated D1/P1/P2 quorum.
// The hedge test separately exercises its production publication timing.
let errs: Vec<Option<DiskError>> = (0..4).map(|_| None).collect();
let (online_disks, fi, _) = SetDisks::select_valid_fileinfo(&disks, &metadata, &errs, "", 2, 3)
let (online_disks, fi) = SetDisks::select_valid_fileinfo(&disks, &metadata, &errs, 2)
.expect("three real members satisfy latest-generation selection");
assert_eq!(online_disks.iter().flatten().count(), 3);
assert_eq!(fi.metadata.get("etag"), committed.etag.as_ref());
@@ -6240,16 +6239,17 @@ mod tests {
let parts_metadata = vec![first.clone(), second, third, FileInfo::default()];
let errs = vec![None, None, None, None];
let (read_quorum, write_quorum) = SetDisks::object_quorum_from_meta(&parts_metadata, &errs, 2)
let (read_quorum, _) = SetDisks::object_quorum_from_meta(&parts_metadata, &errs, 2)
.expect("partial early-stop metadata should preserve read quorum");
let read_quorum = usize::try_from(read_quorum).expect("read quorum should be non-negative");
let write_quorum = usize::try_from(write_quorum).expect("write quorum should be non-negative");
let selection_quorum = SetDisks::latest_fileinfo_selection_quorum("", &parts_metadata, &errs, read_quorum, write_quorum);
let (_, latest) = SetDisks::select_valid_fileinfo(&vec![None; parts_metadata.len()], &parts_metadata, &errs, read_quorum)
.expect("partial early-stop metadata should select the candidate");
let selected = SetDisks::pick_valid_fileinfo(&parts_metadata, None, Some("etag-1".to_string()), read_quorum)
.expect("partial early-stop metadata should preserve selected FileInfo");
assert_eq!(read_quorum, 2);
assert_eq!(selection_quorum, 3);
assert_eq!(latest.name, first.name);
assert_eq!(latest.get_etag(), first.get_etag());
assert_eq!(selected.name, first.name);
assert_eq!(selected.get_etag(), first.get_etag());
}
+466 -226
View File
@@ -22,9 +22,7 @@ fn to_filemeta_err(err: Error) -> rustfs_filemeta::Error {
}
use crate::bucket::lifecycle::bucket_lifecycle_ops::free_version_remote_tuple_matches;
use crate::bucket::metadata_sys::{
get_versioning_config, has_authoritative_never_versioned_state, has_authoritative_never_versioned_state_in,
};
use crate::bucket::metadata_sys::{get_versioning_config, has_authoritative_never_versioned_state};
use crate::bucket::utils::{check_list_objs_args, has_bad_path_component};
use crate::bucket::versioning::VersioningApi;
use crate::cache_value::metacache_set::{
@@ -373,49 +371,6 @@ async fn can_skip_hidden_prefix_check(options: &ListPathOptions) -> bool {
.unwrap_or(false)
}
/// Whether an empty listing of `prefix` is the proof that lets the caller
/// reclaim delete residue under it. Any first page that scanned the whole
/// prefix without finding an object or a sub-prefix qualifies, with or without
/// a delimiter: that is the shape clients issue when they stat, browse, or
/// recursively remove a phantom folder. The purge itself re-verifies every
/// directory on every disk before deleting anything.
fn should_purge_empty_directory_listing(
prefix: &str,
marker: Option<&str>,
max_keys: i32,
incl_deleted: bool,
result: &ListObjectsInfo,
) -> bool {
// list_path rejects leading slashes, so an empty result cannot justify cleanup.
!prefix.is_empty()
&& !prefix.starts_with(SLASH_SEPARATOR)
&& prefix.ends_with(SLASH_SEPARATOR)
&& marker.is_none()
&& max_keys > 0
&& !incl_deleted
&& !result.is_truncated
&& result.objects.is_empty()
&& result.prefixes.is_empty()
}
fn should_purge_empty_recursive_bucket_listing(
prefix: &str,
delimiter: Option<&str>,
marker: Option<&str>,
max_keys: i32,
incl_deleted: bool,
result: &ListObjectsInfo,
) -> bool {
prefix.is_empty()
&& delimiter.is_none()
&& marker.is_none()
&& max_keys > 0
&& !incl_deleted
&& !result.is_truncated
&& result.objects.is_empty()
&& result.prefixes.is_empty()
}
const MARKER_TAG_VERSION: &str = "v2";
const LEGACY_MARKER_TAG_VERSIONS: &[&str] = &["v1", MARKER_TAG_VERSION];
const LIST_CACHE_MARKER_PREFIX: &str = "[rustfs_cache:";
@@ -552,18 +507,6 @@ impl ListContinuationV2 {
}
}
/// Returns the key part of a marker that may still carry the legacy cursor tag.
/// The data-usage scanner compares successive version-listing markers through
/// it so two differently tagged markers for the same key never count as progress.
// RUSTFS_COMPAT_TODO(listing-cursor-legacy-tag): markers tagged by 1.0.x servers. Remove after every supported
// direct-upgrade release issues plain key markers; callers can then compare markers directly.
pub(crate) fn list_marker_key(marker: &str) -> &str {
marker
.rfind(LIST_CACHE_MARKER_PREFIX)
.filter(|start_idx| marker[*start_idx..].ends_with(']'))
.map_or(marker, |start_idx| &marker[..start_idx])
}
fn normalize_list_quorum(value: &str) -> &'static str {
let value = value.trim();
if value.eq_ignore_ascii_case("disk") {
@@ -631,8 +574,7 @@ fn versioned_prefix_page_boundary_key<'a>(
/// old-readable escape until legacy tag parsing can be removed.
// RUSTFS_COMPAT_TODO(listing-cursor-legacy-tag): Remove after the minimum upgrade source stops issuing tagged cursors.
fn escape_list_marker(marker: String) -> String {
// Both the legacy request parser and scanner cursor comparison recognize
// cache-shaped suffixes. One old-readable outer tag protects the literal key.
// One old-readable outer tag protects a literal cache-shaped key.
if marker
.rfind(LIST_CACHE_MARKER_PREFIX)
.is_some_and(|start| marker[start..].contains(']'))
@@ -669,9 +611,9 @@ fn build_list_versions_next_marker(objects: &[ObjectInfo], prefixes: &[String])
.unwrap_or_else(|| "null".to_string()),
)
};
(Some(escape_list_marker(last.name.clone())), version_marker)
(Some(last.name.clone()), version_marker)
} else if let Some(last_prefix) = prefixes.last() {
(Some(escape_list_marker(last_prefix.clone())), None)
(Some(last_prefix.clone()), None)
} else {
(None, None)
}
@@ -734,7 +676,7 @@ fn list_objects_paginate(
&& let Some(last_scanned) = last_scanned_key
{
is_truncated = true;
next_marker = Some(escape_list_marker(last_scanned.to_owned()));
next_marker = Some(last_scanned.to_owned());
} else if should_truncate {
is_truncated = true;
if include_version_id {
@@ -802,7 +744,7 @@ where
pagination.last_scanned_key,
);
if is_truncated && let Some(raw_key) = prefix_at_page_boundary {
next_marker = Some(escape_list_marker(raw_key));
next_marker = Some(raw_key);
next_version_idmarker = None;
}
@@ -894,10 +836,8 @@ impl ListPathOptions {
/// Releases up to 1.0.x appended `[rustfs_cache:v2,id:...,src:walker,gen:live]`
/// to every internal continuation marker and exposed it through the V2
/// continuation token and the ListObjectVersions `NextKeyMarker`. Current
/// servers emit plain keys, but a client that paginates across the upgrade
/// still sends the tagged form back as `continuation-token`, `key-marker`
/// or `marker`, and the opt-in index provider still carries its `src`/`gen`
/// cursor fields in the same tag.
/// servers still accept the tagged form in V2 continuation tokens and
/// legacy object-list markers. Version-listing key markers are literal keys.
///
/// Newly issued cursors protect cache-shaped literal keys with one legacy
/// outer tag. Parse only at the listing entry point; parsing the resulting
@@ -953,7 +893,9 @@ impl ECStore {
) -> Result<ListObjectsV2Info> {
let marker = {
if continuation_token.is_none() {
start_after
// StartAfter is a literal object key, even if it resembles a
// legacy cursor. Protect it from the compatibility parser.
start_after.map(escape_list_marker)
} else {
continuation_token.clone()
}
@@ -1022,39 +964,15 @@ impl ECStore {
let get_objects =
ObjectInfo::from_meta_cache_entries_sorted_infos(&page.entries, bucket, prefix, delimiter.clone()).await;
let (objects, prefixes, is_truncated, next_marker, next_version_idmarker) =
let (objects, prefixes, is_truncated, next_marker, _) =
list_objects_paginate(get_objects, &delimiter, max_keys, disk_has_more, false, last_scanned_key.as_deref());
let _ = next_version_idmarker;
let result = ListObjectsInfo {
Ok(ListObjectsInfo {
is_truncated,
next_marker,
objects,
prefixes,
};
let purge_exact_prefix =
should_purge_empty_directory_listing(prefix, opts.marker.as_deref(), max_keys, incl_deleted, &result);
let purge_empty_bucket = should_purge_empty_recursive_bucket_listing(
prefix,
delimiter.as_deref(),
opts.marker.as_deref(),
max_keys,
incl_deleted,
&result,
);
if (purge_exact_prefix || purge_empty_bucket)
&& has_authoritative_never_versioned_state_in(&self.ctx, bucket)
.await
.unwrap_or(false)
{
if purge_exact_prefix {
self.purge_orphan_dir_object(bucket, prefix).await;
} else {
self.purge_orphan_dir_objects_in_bucket(bucket).await;
}
}
Ok(result)
})
}
pub async fn inner_list_object_versions(
@@ -1198,7 +1116,7 @@ impl ECStore {
};
let effective_max_keys = list_versions_scan_limit(max_keys, has_version_marker);
let mut opts = ListPathOptions {
let opts = ListPathOptions {
bucket: bucket.to_owned(),
caller: if include_version_purge {
ListObjectsGatherCaller::Lifecycle
@@ -1215,10 +1133,6 @@ impl ECStore {
include_marker: has_version_marker,
..Default::default()
};
// Strip a legacy cursor tag before any name comparison (notably
// `forward_past`), see backlog#1047.
opts.parse_marker();
let mut page = self
.list_path(&opts)
.await
@@ -2416,7 +2330,7 @@ impl SetDisks {
let has_version_marker = version_marker.is_some();
let version_marker = version_marker.map(parse_version_marker).transpose()?;
let effective_max_keys = list_versions_scan_limit(max_keys, has_version_marker);
let mut opts = ListPathOptions {
let opts = ListPathOptions {
bucket: bucket.to_owned(),
prefix: prefix.to_owned(),
limit: effective_max_keys,
@@ -2427,8 +2341,6 @@ impl SetDisks {
include_marker: has_version_marker,
..Default::default()
};
opts.parse_marker();
let mut page = self
.list_path_result(&opts)
.await
@@ -2900,13 +2812,12 @@ mod lifecycle_exact_tests;
#[cfg(test)]
mod test {
use super::{
ENV_API_LIST_OBJECTS_QUORUM, GatherResultsState, ListObjectsInfo, ListPathOptions, ListSourceMode, MAX_OBJECT_LIST,
RUSTFS_META_BUCKET, VersionMarker, gather_results, list_marker_key, list_merged_entry_channel,
list_metadata_resolution_params, list_objects_paginate, list_objects_quorum_from_env, max_keys_plus_one,
merge_entry_channels, normalize_list_quorum, observe_scanner_namespace_mutations, parse_version_marker,
recursive_walk_list_path_options, reset_scanner_namespace_mutation_generation_for_test,
scanner_namespace_mutation_generation, send_or_cancel, should_purge_empty_directory_listing, version_marker_for_entries,
walk_result_from_set_errors,
ENV_API_LIST_OBJECTS_QUORUM, GatherResultsState, ListPathOptions, ListSourceMode, MAX_OBJECT_LIST, RUSTFS_META_BUCKET,
VersionMarker, gather_results, list_merged_entry_channel, list_metadata_resolution_params, list_objects_paginate,
list_objects_quorum_from_env, max_keys_plus_one, merge_entry_channels, normalize_list_quorum,
observe_scanner_namespace_mutations, parse_version_marker, recursive_walk_list_path_options,
reset_scanner_namespace_mutation_generation_for_test, scanner_namespace_mutation_generation, send_or_cancel,
version_marker_for_entries, walk_result_from_set_errors,
};
use crate::bucket::replication::ReplicationState;
use crate::cache_value::metacache_set::ListPathRawOutcome;
@@ -4411,7 +4322,10 @@ mod test {
list_objects_paginate(vec![plain_object(key), plain_object("zz")], &None, 1, false, include_version_id, None);
assert!(truncated);
let marker = marker.expect("truncated page needs a cursor");
assert_eq!(list_marker_key(&marker), key, "scanner must compare the complete literal key");
if include_version_id {
assert_eq!(marker, key, "version markers must be literal keys");
continue;
}
let mut opts = ListPathOptions {
marker: Some(marker),
..Default::default()
@@ -4456,6 +4370,7 @@ mod test {
.expect("seed literal object");
}
let mut marker = None;
let mut v1_marker = None;
let mut key_marker = None;
let mut version_marker = None;
for (index, key) in keys.into_iter().enumerate() {
@@ -4468,6 +4383,16 @@ mod test {
assert_eq!(page.is_truncated, index + 1 < keys.len());
marker = page.next_continuation_token;
// V1 sends the literal NextMarker back as StartAfter.
let v1_page = store
.clone()
.inner_list_objects_v2(bucket, "", None, None, 1, false, v1_marker, false)
.await
.expect("list next object through the V1 marker path");
assert_eq!(v1_page.objects.iter().map(|object| object.name.as_str()).collect::<Vec<_>>(), [key]);
assert_eq!(v1_page.is_truncated, index + 1 < keys.len());
v1_marker = v1_page.is_truncated.then(|| key.to_owned());
let versions = store
.clone()
.inner_list_object_versions(bucket, "", key_marker, version_marker, None, 1)
@@ -4475,11 +4400,44 @@ mod test {
.expect("list next version");
assert_eq!(versions.objects.iter().map(|object| object.name.as_str()).collect::<Vec<_>>(), [key]);
assert_eq!(versions.is_truncated, index + 1 < keys.len());
if versions.is_truncated {
assert_eq!(versions.next_marker.as_deref(), Some(key), "NextKeyMarker must be the literal key");
}
key_marker = versions.next_marker;
version_marker = versions.next_version_idmarker;
}
assert!(marker.is_none());
assert!(v1_marker.is_none());
assert!(key_marker.is_none());
let literal = "b[rustfs_cache:v1,return:]";
let after_literal = store
.clone()
.inner_list_objects_v2(bucket, "", None, None, 1, false, Some(literal.to_owned()), false)
.await
.expect("start after a caller-supplied literal key");
assert_eq!(after_literal.objects[0].name, "c[rustfs_cache:v2,id:x]suffix");
let resumed = store
.clone()
.inner_list_object_versions(bucket, "", Some(literal.to_owned()), None, None, 1)
.await
.expect("resume from a caller-supplied literal key marker");
assert_eq!(resumed.objects[0].name, "c[rustfs_cache:v2,id:x]suffix");
let resumed_version = store
.clone()
.inner_list_object_versions(bucket, "", Some(literal.to_owned()), Some("null".to_owned()), None, 1)
.await
.expect("resume after the literal key's null version");
assert_eq!(resumed_version.objects[0].name, "c[rustfs_cache:v2,id:x]suffix");
let recursive_delete_page = store.pools[0].disk_set[0]
.clone()
.inner_list_object_versions_for_recursive_delete(bucket, "", Some(literal.to_owned()), None, 1)
.await
.expect("resume recursive-delete validation from the literal key");
assert_eq!(recursive_delete_page.objects[0].name, "c[rustfs_cache:v2,id:x]suffix");
}
#[tokio::test]
@@ -4645,6 +4603,32 @@ mod test {
assert!(next_version_idmarker.is_none());
}
#[test]
fn versioned_delimiter_boundaries_preserve_cache_shaped_raw_keys() {
let delimiter = Some("-".to_owned());
let raw_key = "data-0002[rustfs_cache:v2,return:]";
let (_, _, truncated, marker, version_marker) =
list_objects_paginate(vec![folded_prefix_object("data-")], &delimiter, 2, true, true, Some(raw_key));
assert!(truncated);
assert_eq!(marker.as_deref(), Some(raw_key));
assert!(version_marker.is_none());
let (_, _, truncated, marker, version_marker) = super::list_objects_paginate_versions(
vec![folded_prefix_object("data-")],
super::VersionedListPagination {
delimiter: &delimiter,
max_keys: 1,
disk_has_more: true,
marker: None,
last_scanned_key: Some(raw_key),
raw_keys: std::iter::once(raw_key),
},
);
assert!(truncated);
assert_eq!(marker.as_deref(), Some(raw_key));
assert!(version_marker.is_none());
}
// End-to-end simulation of versioned pagination with `max_keys=1`, where the
// next page resumes via a bare key marker (`forward_past` keeps `name > marker`
// only, matching `inner_list_object_versions_with_projection` without a version
@@ -4935,77 +4919,114 @@ mod test {
assert_eq!(scanner_namespace_mutation_generation(), 2);
}
#[test]
fn empty_directory_listing_purge_requires_complete_first_page_of_prefix() {
let empty = ListObjectsInfo::default();
assert!(should_purge_empty_directory_listing("ghost/", None, 1, false, &empty));
assert!(should_purge_empty_directory_listing("ghost/", None, 1000, false, &empty));
assert!(!should_purge_empty_directory_listing("ghost", None, 1, false, &empty));
assert!(!should_purge_empty_directory_listing("ghost/", Some("marker"), 1, false, &empty));
assert!(!should_purge_empty_directory_listing("ghost/", None, 0, false, &empty));
assert!(!should_purge_empty_directory_listing("ghost/", None, 1, true, &empty));
#[tokio::test]
async fn listing_preserves_valid_legacy_residue_until_explicit_delete() {
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::storage_api_contracts::bucket::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions};
let mut live = ListObjectsInfo::default();
live.objects.push(ObjectInfo::default());
assert!(!should_purge_empty_directory_listing("ghost/", None, 1, false, &live));
use crate::storage_api_contracts::object::{ObjectIO as _, ObjectOperations as _};
let mut prefixed = ListObjectsInfo::default();
prefixed.prefixes.push("ghost/child/".to_owned());
assert!(!should_purge_empty_directory_listing("ghost/", None, 1, false, &prefixed));
let (dirs, store) = isolated_store_over_temp_disks().await;
let bucket = "stage-e-exact-delete-legacy";
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("bucket should be created with authoritative metadata");
let transaction = uuid::Uuid::new_v4();
let mut backups = Vec::new();
for object in ["repro/obj-00001.bin", "repro/obj-00002.bin"] {
let mut reader = crate::object_api::PutObjReader::from_vec(vec![0x62; 1024 * 1024]);
store
.put_object(
bucket,
object,
&mut reader,
&crate::object_api::ObjectOptions {
write_completion: crate::object_api::WriteCompletion::TailDrained,
..Default::default()
},
)
.await
.expect("create genuine per-drive metadata and shards");
for dir in &dirs {
let object_dir = dir.path().join(bucket).join(object);
let mut entries = tokio::fs::read_dir(&object_dir).await.expect("read actual object root");
let mut marked = 0;
while let Some(entry) = entries.next_entry().await.expect("read object entry") {
if uuid::Uuid::parse_str(&entry.file_name().to_string_lossy()).is_ok() {
tokio::fs::write(
entry
.path()
.join(format!("{}{}", crate::disk::local::DELETE_DATA_DIR_MARKER_PREFIX, transaction)),
[],
)
.await
.expect("mark actual data dir as historical delete residue");
marked += 1;
}
}
assert_eq!(marked, 1, "the real shard must have exactly one data directory");
let backup = object_dir.join(transaction.to_string());
tokio::fs::create_dir_all(&backup).await.expect("create legacy rollback root");
tokio::fs::rename(
object_dir.join(crate::disk::STORAGE_FORMAT_FILE),
backup.join(crate::disk::STORAGE_FORMAT_FILE_BACKUP),
)
.await
.expect("retain the actual valid metadata as a historical backup");
let backup = backup.join(crate::disk::STORAGE_FORMAT_FILE_BACKUP);
let bytes = tokio::fs::read(&backup).await.expect("read the historical backup");
backups.push((backup, bytes));
}
}
let truncated = ListObjectsInfo {
is_truncated: true,
..Default::default()
};
assert!(!should_purge_empty_directory_listing("ghost/", None, 1, false, &truncated));
}
let delimited = store
.clone()
.list_objects_generic(bucket, "", None, Some("/".to_owned()), 1000, false)
.await
.expect("delimiter listing should succeed");
assert!(delimited.objects.is_empty());
assert!(delimited.prefixes.is_empty(), "residue must not appear as a common prefix");
#[test]
fn empty_directory_listing_does_not_purge_rejected_leading_slash_prefixes() {
let empty = ListObjectsInfo::default();
for prefix in ["/", "//", "/ghost/"] {
let result = store
.clone()
.list_objects_generic(bucket, "", None, None, 1000, false)
.await
.expect("recursive bucket listing should succeed");
assert!(result.objects.is_empty());
let exact_prefix = store
.clone()
.list_objects_generic(bucket, "repro/", None, None, 1000, false)
.await
.expect("recursive prefix listing should succeed");
assert!(exact_prefix.objects.is_empty());
for (path, bytes) in backups {
assert_eq!(tokio::fs::read(path).await.expect("LIST must preserve backup"), bytes);
}
for dir in &dirs {
assert!(
!should_purge_empty_directory_listing(prefix, None, 1000, false, &empty),
"rejected prefix {prefix:?} must not initiate orphan cleanup"
dir.path().join(bucket).join("repro").exists(),
"LIST must preserve the historical residue"
);
}
assert!(should_purge_empty_directory_listing("ghost/", None, 1000, false, &empty));
}
#[test]
fn recursive_bucket_orphan_purge_requires_an_empty_complete_root_scan() {
let empty = ListObjectsInfo::default();
assert!(super::should_purge_empty_recursive_bucket_listing("", None, None, 1, false, &empty));
assert!(!super::should_purge_empty_recursive_bucket_listing(
"ghost/", None, None, 1, false, &empty
));
assert!(!super::should_purge_empty_recursive_bucket_listing("", Some("/"), None, 1, false, &empty));
assert!(!super::should_purge_empty_recursive_bucket_listing(
"",
None,
Some("marker"),
1,
false,
&empty
));
assert!(!super::should_purge_empty_recursive_bucket_listing("", None, None, 0, false, &empty));
assert!(!super::should_purge_empty_recursive_bucket_listing("", None, None, 1, true, &empty));
let live = ListObjectsInfo {
objects: vec![ObjectInfo::default()],
..Default::default()
};
assert!(!super::should_purge_empty_recursive_bucket_listing("", None, None, 1, false, &live));
let incomplete = ListObjectsInfo {
is_truncated: true,
..Default::default()
};
assert!(!super::should_purge_empty_recursive_bucket_listing("", None, None, 1, false, &incomplete));
for object in ["repro/obj-00001.bin", "repro/obj-00002.bin"] {
store
.delete_object(bucket, object, crate::object_api::ObjectOptions::default())
.await
.expect("fresh exact-key DELETE should reclaim eligible legacy residue");
for dir in &dirs {
assert!(!dir.path().join(bucket).join(object).exists(), "exact DELETE must reclaim the root");
}
}
store
.delete_bucket(bucket, &DeleteBucketOptions::default())
.await
.expect("the bucket is empty once the residue is reclaimed");
}
#[tokio::test]
async fn empty_recursive_listing_purges_committed_delete_residue() {
async fn empty_recursive_listing_preserves_committed_delete_residue() {
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
@@ -5048,20 +5069,73 @@ mod test {
assert!(result.objects.is_empty());
assert!(result.prefixes.is_empty());
for dir in &dirs {
let residue = dir.path().join(bucket).join("ghost/nested/object").join(data_dir.to_string());
assert_eq!(tokio::fs::read(residue.join("part.1")).await.expect("LIST preserves part"), b"stale");
assert!(
!dir.path().join(bucket).join("ghost").exists(),
"the empty listing should reclaim its committed delete residue"
residue
.join(format!("{}{}", crate::disk::local::DELETE_DATA_DIR_MARKER_PREFIX, transaction))
.exists()
);
}
}
// #6898: what an abandoned DeleteObjects left on every disk (rollback
// backup plus marked data dir) no longer blocks DeleteBucket after an empty
// recursive listing of the bucket.
#[tokio::test]
async fn empty_recursive_listing_purges_stranded_delete_rollback_backups() {
async fn large_preserved_residue_does_not_create_a_common_prefix() {
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
let (dirs, store) = isolated_store_over_temp_disks().await;
let bucket = "large-listing-residue";
init_bucket_metadata_sys(store.clone(), Vec::new()).await;
store
.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("create bucket");
for object in 0..20 {
let data_dir = uuid::Uuid::new_v4();
let transaction = uuid::Uuid::new_v4();
for dir in &dirs {
let residue = dir
.path()
.join(bucket)
.join(format!("ghost/{object:02}"))
.join(data_dir.to_string());
tokio::fs::create_dir_all(&residue).await.expect("create residue");
tokio::fs::write(residue.join("part.1"), b"stale")
.await
.expect("write residue part");
tokio::fs::write(
residue.join(format!("{}{}", crate::disk::local::DELETE_DATA_DIR_MARKER_PREFIX, transaction)),
[],
)
.await
.expect("write residue marker");
}
}
let recursive = store
.clone()
.list_objects_generic(bucket, "", None, None, 1000, false)
.await
.expect("list recursively");
assert!(recursive.objects.is_empty());
let delimited = store
.list_objects_generic(bucket, "", None, Some("/".to_owned()), 1000, false)
.await
.expect("list with delimiter");
assert!(delimited.objects.is_empty());
assert!(delimited.prefixes.is_empty(), "residue must not produce a common prefix");
for dir in &dirs {
assert!(dir.path().join(bucket).join("ghost/00").exists(), "listing must not remove residue");
}
}
// A corrupt historical backup must survive LIST and a failed exact DELETE.
#[tokio::test]
async fn empty_recursive_listing_preserves_corrupt_delete_rollback_backups() {
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::storage_api_contracts::bucket::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions};
use crate::storage_api_contracts::object::ObjectOperations as _;
let (dirs, store) = isolated_store_over_temp_disks().await;
let bucket = "listing-purge-rollback-backup";
@@ -5105,21 +5179,38 @@ mod test {
.expect("recursive bucket listing should succeed");
assert!(result.objects.is_empty());
for dir in &dirs {
assert!(
!dir.path().join(bucket).join("repro").exists(),
"the empty listing should reclaim the stranded delete residue"
let backup = dir
.path()
.join(bucket)
.join("repro/obj-00001.bin")
.join(transaction.to_string())
.join(crate::disk::STORAGE_FORMAT_FILE_BACKUP);
assert_eq!(tokio::fs::read(backup).await.expect("LIST preserves backup"), b"rollback metadata");
}
let err = store
.delete_object(bucket, "repro/obj-00001.bin", crate::object_api::ObjectOptions::default())
.await
.expect_err("corrupt metadata must not be reclaimed");
assert!(matches!(err, StorageError::FileCorrupt));
assert!(store.delete_bucket(bucket, &DeleteBucketOptions::default()).await.is_err());
for dir in &dirs {
let backup = dir
.path()
.join(bucket)
.join("repro/obj-00001.bin")
.join(transaction.to_string())
.join(crate::disk::STORAGE_FORMAT_FILE_BACKUP);
assert_eq!(
tokio::fs::read(backup).await.expect("failed cleanup preserves backup"),
b"rollback metadata"
);
}
store
.delete_bucket(bucket, &DeleteBucketOptions::default())
.await
.expect("the bucket is empty once the residue is reclaimed");
}
#[tokio::test]
async fn empty_recursive_bucket_listing_purges_orphan_directory_prefixes() {
async fn empty_recursive_bucket_listing_preserves_empty_directory_prefixes() {
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
use crate::storage_api_contracts::bucket::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions};
let (dirs, store) = isolated_store_over_temp_disks().await;
let bucket = "recursive-bucket-orphan-purge";
@@ -5145,10 +5236,14 @@ mod test {
assert!(result.prefixes.is_empty(), "a delimiter-less listing has no CommonPrefixes");
for dir in &dirs {
assert!(
!dir.path().join(bucket).join("ghost").exists(),
"an empty recursive bucket scan should reclaim its metadata-less orphan prefix"
dir.path().join(bucket).join("ghost/nested/leaf").exists(),
"LIST must preserve empty directories"
);
}
store
.delete_bucket(bucket, &DeleteBucketOptions::default())
.await
.expect("bucket deletion removes empty directories");
let versioned_bucket = "recursive-bucket-versioned-orphan";
store
@@ -5300,7 +5395,7 @@ mod test {
}
#[tokio::test]
async fn empty_delimiter_listing_hides_and_purges_committed_delete_residue() {
async fn empty_delimiter_listing_hides_and_preserves_committed_delete_residue() {
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
@@ -5345,15 +5440,22 @@ mod test {
assert!(result.objects.is_empty());
assert!(result.prefixes.is_empty(), "delete residue must not surface as a prefix");
for dir in &dirs {
let residue = dir
.path()
.join(bucket)
.join("metrics/2026/object.parquet")
.join(data_dir.to_string());
assert_eq!(tokio::fs::read(residue.join("part.1")).await.expect("LIST preserves part"), b"stale");
assert!(
!dir.path().join(bucket).join("metrics").join("2026").exists(),
"the empty delimiter listing should reclaim the committed delete residue under it"
residue
.join(format!("{}{}", crate::disk::local::DELETE_DATA_DIR_MARKER_PREFIX, transaction))
.exists()
);
}
}
#[tokio::test]
async fn empty_delimiter_listing_of_residue_ancestor_hides_and_purges_whole_tree() {
async fn empty_delimiter_listing_of_residue_ancestor_hides_and_preserves_whole_tree() {
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
@@ -5386,9 +5488,7 @@ mod test {
.expect("committed delete marker should be written");
}
// Browsing an ancestor of the deleted key must not show the empty
// date folders, and the empty result reclaims the whole residue tree
// in one pass instead of one level per listing.
// Browsing an ancestor must not expose the deleted key's date folders.
let result = store
.clone()
.list_objects_generic(bucket, "metrics/", None, Some("/".to_owned()), 1000, false)
@@ -5397,15 +5497,22 @@ mod test {
assert!(result.objects.is_empty());
assert!(result.prefixes.is_empty(), "ancestors of delete residue must not surface as prefixes");
for dir in &dirs {
let residue = dir
.path()
.join(bucket)
.join("metrics/kubelet/2026/08/28/23/74992556388248657933757.parquet")
.join(data_dir.to_string());
assert_eq!(tokio::fs::read(residue.join("part.1")).await.expect("LIST preserves part"), b"stale");
assert!(
!dir.path().join(bucket).join("metrics").exists(),
"the empty delimiter listing should reclaim the committed delete residue tree under it"
residue
.join(format!("{}{}", crate::disk::local::DELETE_DATA_DIR_MARKER_PREFIX, transaction))
.exists()
);
}
}
#[tokio::test]
async fn empty_delimiter_listing_of_mixed_residue_ancestor_reclaims_only_committed_subtree() {
async fn empty_delimiter_listing_of_mixed_residue_ancestor_preserves_every_subtree() {
use crate::bucket::metadata_sys::{init_bucket_metadata_sys, test_support::isolated_store_over_temp_disks};
use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
@@ -5478,9 +5585,18 @@ mod test {
assert!(result.prefixes.is_empty(), "residue subtrees may not surface as prefixes");
for dir in &dirs {
let metrics = dir.path().join(bucket).join("metrics");
assert!(
!metrics.join("cpu").exists(),
"the committed subtree must be reclaimed even though a sibling subtree is blocked"
let committed = metrics.join("cpu/2026/08/28/22/a.parquet").join(committed_dir.to_string());
assert_eq!(
tokio::fs::read(committed.join("part.1"))
.await
.expect("LIST preserves committed part"),
b"stale"
);
assert_eq!(
tokio::fs::read(committed.join(crate::disk::STORAGE_FORMAT_FILE_BACKUP))
.await
.expect("LIST preserves backup"),
b"rollback metadata"
);
assert!(
metrics
@@ -5488,7 +5604,7 @@ mod test {
.join(unmarked_dir.to_string())
.join("part.1")
.exists(),
"residue without a committed marker must survive the purge"
"residue without a committed marker must survive the listing"
);
assert_eq!(
tokio::fs::read(
@@ -6405,6 +6521,147 @@ mod test {
}
}
#[tokio::test]
async fn latest_get_follows_read_quorum_on_torn_writes() {
use crate::storage_api_contracts::bucket::{BucketOperations as _, MakeBucketOptions};
use crate::storage_api_contracts::object::ObjectOperations as _;
// HEAD and GET serve the version with the most copies once it reaches
// the object's data blocks, and a tie goes to the newer version, as in
// MinIO's getObjectFileInfo and as heal converges the set. GET used to
// demand write quorum whenever enough metadata answered, so a 2+2 or
// 8+8 split, or a newer side that could still hold write quorum
// through an offline drive, listed but answered 503.
let opts = ObjectOptions {
suppress_read_repair: true,
..Default::default()
};
for (drives, parity) in [(4_usize, 2_usize), (16, 8), (16, 4)] {
let data = drives - parity;
let bucket = format!("torn-write-get-{drives}-{parity}");
let (dirs, set) = crate::ecstore_validation_blackbox::make_local_set_disks(drives, parity).await;
set.make_bucket(&bucket, &MakeBucketOptions::default())
.await
.expect("bucket should be created");
// Two committed generations of one key give every drive its own
// inline shard of each generation; the cases copy them per drive.
let mut generations = Vec::new();
for body in [b"old generation".as_slice(), b"new generation".as_slice()] {
let info = set
.put_object(
&bucket,
"template",
&mut PutObjReader::from_vec(body.to_vec()),
&ObjectOptions {
no_lock: true,
..Default::default()
},
)
.await
.expect("generation PUT should commit on every drive");
let mut metadata = Vec::with_capacity(drives);
for dir in &dirs {
metadata.push(
tokio::fs::read(dir.path().join(&bucket).join("template").join(STORAGE_FORMAT_FILE))
.await
.expect("generation metadata should carry the inline shard"),
);
}
generations.push((body, info.etag.expect("PUT should report an etag"), metadata));
}
let (new_body, new_etag, new) = generations.pop().expect("new generation");
let (old_body, old_etag, old) = generations.pop().expect("old generation");
// New copies occupy the first drives, so the offline drive always
// removes one new copy. Each phase reads its own keys so the GET
// metadata cache never answers for the other phase.
let mut cases = Vec::new();
for offline in [false, true] {
for copies in 1..drives {
for overwrite in [false, true] {
let name = format!(
"{}-{}-{copies:02}",
if offline { "offline" } else { "online" },
if overwrite { "overwrite" } else { "new" }
);
for slot in 0..drives {
let bytes = if slot < copies {
&new[slot]
} else if overwrite {
&old[slot]
} else {
continue;
};
let path = dirs[slot].path().join(&bucket).join(&name);
tokio::fs::create_dir_all(&path).await.expect("create object directory");
tokio::fs::write(path.join(STORAGE_FORMAT_FILE), bytes)
.await
.expect("write generation metadata");
}
cases.push((offline, name, copies, overwrite));
}
}
}
for offline in [false, true] {
let parked = if offline { set.disks.write().await[0].take() } else { None };
for (_, name, copies, overwrite) in cases.iter().filter(|case| case.0 == offline) {
let new_copies = copies - usize::from(offline);
let old_copies = if *overwrite { drives - copies } else { 0 };
let expected = if new_copies >= data && new_copies >= old_copies {
Some((&new_etag, new_body))
} else if old_copies >= data {
Some((&old_etag, old_body))
} else {
None
};
let context = format!("{drives} drives EC {data}+{parity}, {name}: {new_copies} new, {old_copies} old");
let head = set.get_object_info(&bucket, name, &opts).await;
let get = set.get_object_reader(&bucket, name, None, HeaderMap::new(), &opts).await;
match (expected, head, get) {
(Some((etag, body)), Ok(info), Ok(mut reader)) => {
assert_eq!(info.etag.as_ref(), Some(etag), "{context}: HEAD");
assert_eq!(reader.object_info.etag.as_ref(), Some(etag), "{context}: GET");
let mut streamed = Vec::new();
reader
.stream
.read_to_end(&mut streamed)
.await
.unwrap_or_else(|err| panic!("{context}: GET body: {err:?}"));
assert_eq!(streamed, body, "{context}: GET body");
}
(Some((etag, _)), head, get) => panic!(
"{context}: expected {etag}, HEAD {:?}, GET {:?}",
head.map(|info| info.etag),
get.map(|reader| reader.object_info.etag)
),
(None, head, get) => {
for (method, result) in [
("HEAD", head.map(|info| info.etag)),
("GET", get.map(|reader| reader.object_info.etag)),
] {
let err = result.expect_err(&format!("{context}: {method} expected 404 or unavailable"));
assert!(
matches!(
err,
StorageError::ObjectNotFound(..)
| StorageError::FileNotFound
| StorageError::ErasureReadQuorum
| StorageError::InsufficientReadQuorum(..)
),
"{context}: {method} expected 404 or unavailable, got {err:?}"
);
}
}
}
}
if offline {
set.disks.write().await[0] = parked;
}
}
}
}
#[tokio::test]
async fn latest_listing_and_get_agree_on_hermetic_set_states() {
#[derive(Clone, Copy)]
@@ -6477,6 +6734,10 @@ mod test {
("d-torn-two", [Some(Generation::New), Some(Generation::New), None, None]),
("e-torn-one", [Some(Generation::New), None, None, None]),
("f-stale-one", [None, None, None, Some(Generation::Old)]),
(
"fa-overwrite-two-of-three",
[Some(Generation::New), Some(Generation::New), Some(Generation::Old), None],
),
(
"g-overwrite-torn-one",
[
@@ -6521,6 +6782,7 @@ mod test {
("b-copied", &new_etag),
("c-split", &new_etag),
("d-torn-two", &new_etag),
("fa-overwrite-two-of-three", &new_etag),
("g-overwrite-torn-one", &old_etag),
]
.into_iter()
@@ -6547,12 +6809,6 @@ mod test {
"{name}: GET must serve the version LIST selected"
);
}
// Known divergence: when every drive answers, GET of the latest
// version demands write quorum (three of four), so a 2+2 split
// answers 503 although LIST shows the newer generation, as
// MinIO's listing does. F.0's dangling classification does not
// cover a split; a GET change is a separate decision.
(Some(_), Err(StorageError::InsufficientReadQuorum(..))) if name == "c-split" => {}
(Some(_), Err(err)) => assert!(
matches!(
err,
@@ -6755,22 +7011,6 @@ mod test {
assert_eq!(parsed.set_idx, Some(7));
}
#[test]
fn list_marker_key_strips_only_trailing_cache_tag() {
assert_eq!(
list_marker_key("photos/2026/image.jpg[rustfs_cache:v2,id:list-cache-id,src:walker,gen:live]"),
"photos/2026/image.jpg"
);
assert_eq!(
list_marker_key("photos/[rustfs_cache:v2]/image.jpg"),
"photos/[rustfs_cache:v2]/image.jpg"
);
assert_eq!(
list_marker_key("photos/image.jpg[rustfs_cache:v2,id:list-cache-id"),
"photos/image.jpg[rustfs_cache:v2,id:list-cache-id"
);
}
#[test]
fn list_path_marker_parser_return_tag_forces_cache_refresh() {
let mut parsed = ListPathOptions {
@@ -755,7 +755,7 @@ async fn list_objects_shared_corpus_versions_delete_markers_and_free_versions()
}
#[tokio::test]
async fn list_objects_shared_corpus_orphan_cleanup_requires_authoritative_evidence() {
async fn list_objects_shared_corpus_orphan_listing_preserves_inputs() {
for authoritative in [false, true] {
let fixture = Fixture::new(
&format!("list-corpus-orphan-{authoritative}"),
@@ -774,10 +774,9 @@ async fn list_objects_shared_corpus_orphan_cleanup_requires_authoritative_eviden
let result = fixture.page("ghost/", None, None, 1).await;
assert!(result.objects.is_empty() && result.prefixes.is_empty() && !result.is_truncated);
for dir in &fixture.dirs {
assert_eq!(
dir.path().join(&fixture.bucket).join("ghost").exists(),
!authoritative,
"unknown metadata must fail closed"
assert!(
dir.path().join(&fixture.bucket).join("ghost/nested/leaf").exists(),
"LIST must preserve orphan input"
);
assert!(
dir.path()
@@ -785,7 +784,7 @@ async fn list_objects_shared_corpus_orphan_cleanup_requires_authoritative_eviden
.join("legal/child")
.join(STORAGE_FORMAT_FILE)
.exists(),
"purge must preserve legal siblings"
"LIST must preserve legal siblings"
);
}
fixture.assert_walk("legal/", None, None, 1).await;
@@ -796,7 +795,7 @@ async fn list_objects_shared_corpus_orphan_cleanup_requires_authoritative_eviden
.join("legal/child")
.join(STORAGE_FORMAT_FILE)
.exists(),
"an orphan in the same subtree must not cause the legitimate child to be purged"
"an orphan in the same subtree must not affect a legitimate child"
);
}
if authoritative {
@@ -811,17 +810,15 @@ async fn list_objects_shared_corpus_orphan_cleanup_requires_authoritative_eviden
.bucket_metadata_sys()
.expect("authoritative fixture has a metadata system");
let guard = metadata_sys.write().await;
assert!(
timeout(Duration::from_millis(100), fixture.page("timed-out/", None, None, 1))
.await
.is_err(),
"blocked authoritative evidence must time out before cleanup can classify the prefix as empty"
);
let result = timeout(Duration::from_secs(2), fixture.page("timed-out/", None, None, 1))
.await
.expect("LIST must not wait for cleanup authorization");
assert!(result.objects.is_empty() && result.prefixes.is_empty() && !result.is_truncated);
drop(guard);
for dir in &fixture.dirs {
assert!(
dir.path().join(&fixture.bucket).join("timed-out").exists(),
"a timed-out evidence check must preserve its orphan input"
"LIST must preserve its orphan input while metadata is locked"
);
}
}
-21
View File
@@ -5139,27 +5139,6 @@ impl ECStore {
purged
}
pub(super) async fn purge_orphan_dir_objects_in_bucket(&self, bucket: &str) -> bool {
if is_meta_bucketname(bucket) {
return false;
}
for pool in self.pools.iter() {
for set in pool.disk_set.iter() {
if !set.orphan_purge_has_complete_disk_set().await {
return false;
}
}
}
let mut purged = false;
for pool in self.pools.iter() {
for set in pool.disk_set.iter() {
purged |= set.purge_orphan_dir_objects_in_bucket(bucket).await;
}
}
purged
}
/// Delete one object (or a prefix) on behalf of a caller that may be
/// dropped mid-way, such as an S3 request whose client disconnects.
///
+31 -7
View File
@@ -683,14 +683,17 @@ fn create_auto_heal_disk_request(set_disk_id: String, priority: Option<HealChann
}
}
fn create_auto_replacement_disk_request(
/// Build one automatic replacement request for the verified targets of a set.
pub fn create_auto_replacement_disks_request(
pool_index: usize,
set_index: usize,
replacement_endpoint: String,
mut replacement_endpoints: Vec<String>,
priority: Option<HealChannelPriority>,
) -> HealChannelRequest {
replacement_endpoints.sort_unstable();
replacement_endpoints.dedup();
let mut request = create_auto_heal_disk_request(format!("pool_{pool_index}_set_{set_index}"), priority);
request.heal_endpoints = vec![replacement_endpoint];
request.heal_endpoints = replacement_endpoints;
request.pool_index = Some(pool_index);
request.set_index = Some(set_index);
request
@@ -711,10 +714,10 @@ pub async fn send_heal_replacement_disk(
replacement_endpoint: String,
priority: Option<HealChannelPriority>,
) -> Result<(), String> {
send_heal_request(create_auto_replacement_disk_request(
send_heal_request(create_auto_replacement_disks_request(
pool_index,
set_index,
replacement_endpoint,
vec![replacement_endpoint],
priority,
))
.await
@@ -726,8 +729,12 @@ mod auto_heal_disk_request_tests {
#[test]
fn replacement_disk_request_carries_its_exact_endpoint() {
let request =
create_auto_replacement_disk_request(2, 3, "http://node2:9000/drive3".to_string(), Some(HealChannelPriority::Normal));
let request = create_auto_replacement_disks_request(
2,
3,
vec!["http://node2:9000/drive3".to_string()],
Some(HealChannelPriority::Normal),
);
assert_eq!(request.disk.as_deref(), Some("pool_2_set_3"));
assert_eq!(request.heal_endpoints, ["http://node2:9000/drive3"]);
@@ -736,6 +743,23 @@ mod auto_heal_disk_request_tests {
assert_eq!(request.source, HealRequestSource::AutoHeal);
}
#[test]
fn replacement_disk_request_keeps_every_distinct_target() {
let request = create_auto_replacement_disks_request(
2,
3,
vec!["drive-b".to_string(), "drive-a".to_string(), "drive-b".to_string()],
Some(HealChannelPriority::Normal),
);
assert_eq!(request.disk.as_deref(), Some("pool_2_set_3"));
assert_eq!(request.heal_endpoints, ["drive-a", "drive-b"]);
assert_eq!(request.pool_index, Some(2));
assert_eq!(request.set_index, Some(3));
assert_eq!(request.priority, HealChannelPriority::Normal);
assert_eq!(request.source, HealRequestSource::AutoHeal);
}
#[test]
fn legacy_auto_heal_disk_request_has_no_replacement_endpoint() {
let request = create_auto_heal_disk_request("pool_2_set_3".to_string(), None);
+2
View File
@@ -19,6 +19,7 @@ use thiserror::Error;
use super::heal::{DiskError, EcstoreError};
const HEAL_DANGLING_DELETE_GRACE_MESSAGE: &str = "dangling object deletion deferred by heal grace window";
pub(crate) const MRF_MISSING_PROOF_PREFIX: &str = "Missing exact storage proof for durable MRF repair ";
/// Custom error type for heal operations
/// This enum defines various error variants that can occur during
@@ -221,6 +222,7 @@ pub(crate) fn classify_heal_error(error: &Error) -> HealErrorClass {
Error::Storage(error) => classify_storage_error(error),
Error::Disk(error) => classify_disk_error(error),
Error::Io(error) => classify_io_error(error),
Error::TaskExecutionFailed { message } if message.starts_with(MRF_MISSING_PROOF_PREFIX) => HealErrorClass::Permanent,
Error::TaskExecutionFailed { message } | Error::Other(message) => {
classify_error_display(message).unwrap_or(HealErrorClass::Permanent)
}
+497 -92
View File
@@ -58,6 +58,8 @@ fn result_object_size_u64(result: &HealResultItem) -> u64 {
const NEW_VERSION_SKIP_GRACE_SECS: u64 = 60;
const NANOS_PER_SECOND: i128 = 1_000_000_000;
const DEFAULT_REPLACEMENT_PAGE_OBJECT_CONCURRENCY: usize = 2;
const REPLACEMENT_OBJECT_MAX_RETRIES: u32 = 2;
fn should_skip_new_version(mod_time_unix_nanos: Option<i128>, started_at_secs: u64) -> bool {
let Some(mod_time_unix_nanos) = mod_time_unix_nanos else {
@@ -177,7 +179,9 @@ mod mainline_pacing_tests {
.await
.expect("existing object owns capacity");
let provider = Arc::new(TestPressure::new(WorkloadClass::ForegroundRead, 0));
let pacer = Arc::new(MainlinePacer::new(provider.clone(), 80, 80, Duration::from_millis(250)).expect("pacer"));
let pacer = Arc::new(
MainlinePacer::new(provider.clone(), 80, 80, Duration::from_millis(250), HealRequestSource::Admin).expect("pacer"),
);
let cancel = CancellationToken::new();
let waiting = tokio::spawn({
let semaphore = semaphore.clone();
@@ -216,7 +220,7 @@ mod mainline_pacing_tests {
let occupied = semaphore.clone().acquire_owned().await.expect("queue both waiters");
let provider = Arc::new(TestPressure::new(WorkloadClass::ForegroundRead, 0));
let pause = Duration::from_millis(250);
let pacer = MainlinePacer::new(provider.clone(), 80, 80, pause).expect("pacer");
let pacer = MainlinePacer::new(provider.clone(), 80, 80, pause, HealRequestSource::Admin).expect("pacer");
let cancel = CancellationToken::new();
let mut first = Box::pin(acquire_page_permit(semaphore.clone(), Some(&pacer), &cancel));
let mut second = Box::pin(acquire_page_permit(semaphore.clone(), Some(&pacer), &cancel));
@@ -304,9 +308,21 @@ impl ErasureSetHealer {
}
}
fn effective_heal_page_object_concurrency_for_source(source: HealRequestSource, scan_mode: HealScanMode) -> usize {
fn effective_heal_page_object_concurrency_for_source(
source: HealRequestSource,
scan_mode: HealScanMode,
replacement: bool,
) -> usize {
if matches!(source, HealRequestSource::AutoHeal) {
1
if !replacement || matches!(scan_mode, HealScanMode::Deep) || !Self::page_parallel_enabled() {
1
} else {
rustfs_utils::get_env_usize(
rustfs_config::ENV_HEAL_PAGE_OBJECT_CONCURRENCY,
DEFAULT_REPLACEMENT_PAGE_OBJECT_CONCURRENCY,
)
.clamp(1, Semaphore::MAX_PERMITS)
}
} else {
Self::effective_heal_page_object_concurrency_for_scan_mode(scan_mode)
}
@@ -1467,7 +1483,16 @@ impl ErasureSetHealer {
// transient cache id that changes between identical pages.
let mut previous_page_last: Option<(String, Option<String>)> = None;
let page_concurrency_limit =
Self::effective_heal_page_object_concurrency_for_source(self.source, self.heal_opts.scan_mode);
if self.source == HealRequestSource::AutoHeal && self.replacement_task_id.is_some() && self.heal_opts.no_lock {
// Versions of the same object must not race without its namespace lock.
1
} else {
Self::effective_heal_page_object_concurrency_for_source(
self.source,
self.heal_opts.scan_mode,
self.replacement_task_id.is_some(),
)
};
let in_flight = Arc::new(AtomicUsize::new(0));
// Per-bucket sample caps for per-object warn! lines: a flapping rebuild
// disk can fail/skip hundreds of thousands of versions in one sweep, so
@@ -1689,6 +1714,7 @@ impl ErasureSetHealer {
let semaphore = semaphore.clone();
let target_endpoints = self.target_endpoints.clone();
let replacement_commit_evidence_required = self.replacement_task_id.is_some();
let retry_replacement = replacement_commit_evidence_required && self.source == HealRequestSource::AutoHeal;
let mainline_pacer = self.mainline_pacer.clone();
let admin_identity = self.admin_identity(bucket, &object_name, version_id.as_deref(), incarnation);
@@ -1698,107 +1724,125 @@ impl ErasureSetHealer {
.as_ref()
.map(|_| (dedup_key.clone(), object_name.clone(), version_id.clone()));
let work = async move {
let permit = acquire_page_permit(semaphore, mainline_pacer.as_deref(), &cancel_token).await;
let mut retries = 0;
let mut admin_identity = admin_identity;
let mut admin_task_id = admin_task_id;
loop {
let permit = acquire_page_permit(semaphore.clone(), mainline_pacer.as_deref(), &cancel_token).await;
let _permit = match permit {
Ok(permit) => permit,
Err(err) => return (dedup_key, object_name, version_id, (0, Err(err)), None),
};
let permit = match permit {
Ok(permit) => permit,
Err(err) => return (dedup_key, object_name, version_id, (0, Err(err)), None),
};
let _in_flight_guard = PageConcurrencyGuard::new(in_flight, set_label);
let in_flight_guard = PageConcurrencyGuard::new(in_flight.clone(), set_label.clone());
if let (Some(identity), Some(task_id)) = (admin_identity, admin_task_id) {
let (result, canonical) =
admin::heal_object(storage.as_ref(), &heal_opts, identity, &task_id, &cancel_token).await;
return (dedup_key, object_name, version_id, result, canonical);
}
if let (Some(identity), Some(task_id)) = (admin_identity.take(), admin_task_id.take()) {
let (result, canonical) =
admin::heal_object(storage.as_ref(), &heal_opts, identity, &task_id, &cancel_token).await;
return (dedup_key, object_name, version_id, result, canonical);
}
// Always go through heal_object. Genuine absence flows through
// heal_object -> FileVersionNotFound/FileNotFound ->
// classify_heal_object_error -> Absent, so gone versions are
// recorded as skipped-ok rather than failed. The delete-marker
// vs data path is chosen internally in ops/heal.rs.
let result = if cancel_token.is_cancelled() {
(0, Err(Error::TaskCancelled))
} else {
match storage
.heal_object(&bucket_name, &object_name, version_id.as_deref(), &heal_opts)
.await
{
Ok((result, None)) if target_outcomes_complete(&result, &target_endpoints) => {
let object_size = result_object_size_u64(&result);
if !replacement_commit_evidence_required {
(object_size, Ok(true))
} else {
match storage
.replacement_targets_have_version(
&bucket_name,
&object_name,
version_id.as_deref(),
&heal_opts,
&target_endpoints,
)
.await
{
Ok(true) => (object_size, Ok(true)),
Ok(false) => (
object_size,
Err(Error::transient_skip(format!(
"Skipped heal for {bucket_name}/{object_name} because replacement target readback did not confirm the committed version"
))),
),
Err(err) => (
object_size,
Err(Error::transient_skip(format!(
"Skipped heal for {bucket_name}/{object_name} because replacement target readback failed: {err}"
))),
),
// Always go through heal_object. Genuine absence flows through
// heal_object -> FileVersionNotFound/FileNotFound ->
// classify_heal_object_error -> Absent, so gone versions are
// recorded as skipped-ok rather than failed. The delete-marker
// vs data path is chosen internally in ops/heal.rs.
let result = if cancel_token.is_cancelled() {
(0, Err(Error::TaskCancelled))
} else {
match storage
.heal_object(&bucket_name, &object_name, version_id.as_deref(), &heal_opts)
.await
{
Ok((result, None)) if target_outcomes_complete(&result, &target_endpoints) => {
let object_size = result_object_size_u64(&result);
if !replacement_commit_evidence_required {
(object_size, Ok(true))
} else {
match storage
.replacement_targets_have_version(
&bucket_name,
&object_name,
version_id.as_deref(),
&heal_opts,
&target_endpoints,
)
.await
{
Ok(true) => (object_size, Ok(true)),
Ok(false) => (
object_size,
Err(Error::transient_skip(format!(
"Skipped heal for {bucket_name}/{object_name} because replacement target readback did not confirm the committed version"
))),
),
Err(err) => (
object_size,
Err(Error::transient_skip(format!(
"Skipped heal for {bucket_name}/{object_name} because replacement target readback failed: {err}"
))),
),
}
}
}
}
Ok((result, None)) if !target_endpoints.is_empty() => (
result_object_size_u64(&result),
Err(Error::transient_skip(format!(
"Skipped heal for {bucket_name}/{object_name} because a replacement target was not committed"
))),
),
Ok((result, None)) => (result_object_size_u64(&result), Ok(true)),
Ok((result, Some(err))) if is_missing_object_dir_heal_result(&object_name, &err) => {
(result_object_size_u64(&result), Ok(false))
}
Ok((result, Some(err))) => {
let object_size = result_object_size_u64(&result);
match Self::classify_heal_object_error(&err) {
HealObjectOutcome::Absent => (object_size, Ok(false)),
Ok((result, None)) if !target_endpoints.is_empty() => (
result_object_size_u64(&result),
Err(Error::transient_skip(format!(
"Skipped heal for {bucket_name}/{object_name} because a replacement target was not committed"
))),
),
Ok((result, None)) => (result_object_size_u64(&result), Ok(true)),
Ok((result, Some(err))) if is_missing_object_dir_heal_result(&object_name, &err) => {
(result_object_size_u64(&result), Ok(false))
}
Ok((result, Some(err))) => {
let object_size = result_object_size_u64(&result);
match Self::classify_heal_object_error(&err) {
HealObjectOutcome::Absent => (object_size, Ok(false)),
HealObjectOutcome::DanglingDeleteDeferred { retry_not_before } => {
(object_size, Err(Error::DanglingDeleteDeferred { retry_not_before }))
}
HealObjectOutcome::Transient => (
object_size,
Err(Error::transient_skip(format!(
"Skipped heal for {bucket_name}/{object_name} due to transient error: {err}"
))),
),
HealObjectOutcome::Failed => (object_size, Err(err)),
}
}
Err(err) => match Self::classify_heal_object_error(&err) {
HealObjectOutcome::Absent => (0, Ok(false)),
HealObjectOutcome::DanglingDeleteDeferred { retry_not_before } => {
(object_size, Err(Error::DanglingDeleteDeferred { retry_not_before }))
(0, Err(Error::DanglingDeleteDeferred { retry_not_before }))
}
HealObjectOutcome::Transient => (
object_size,
0,
Err(Error::transient_skip(format!(
"Skipped heal for {bucket_name}/{object_name} due to transient error: {err}"
))),
),
HealObjectOutcome::Failed => (object_size, Err(err)),
}
HealObjectOutcome::Failed => (0, Err(err)),
},
}
Err(err) => match Self::classify_heal_object_error(&err) {
HealObjectOutcome::Absent => (0, Ok(false)),
HealObjectOutcome::DanglingDeleteDeferred { retry_not_before } => {
(0, Err(Error::DanglingDeleteDeferred { retry_not_before }))
}
HealObjectOutcome::Transient => (
0,
Err(Error::transient_skip(format!(
"Skipped heal for {bucket_name}/{object_name} due to transient error: {err}"
))),
),
HealObjectOutcome::Failed => (0, Err(err)),
},
}
};
};
(dedup_key, object_name, version_id, result, None)
drop(in_flight_guard);
drop(permit);
if retry_replacement
&& retries < REPLACEMENT_OBJECT_MAX_RETRIES
&& matches!(&result.1, Err(Error::TransientSkip { .. }))
{
retries += 1;
tokio::select! {
_ = cancel_token.cancelled() => return (dedup_key, object_name, version_id, (0, Err(Error::TaskCancelled)), None),
_ = tokio::time::sleep(Duration::from_millis(250 * u64::from(retries))) => {}
}
continue;
}
return (dedup_key, object_name, version_id, result, None);
}
};
page_tasks.push(async move {
if let (Some(execution), Some((key, object, version))) = (execution, failure_identity) {
@@ -2129,6 +2173,7 @@ mod tests {
ErasureSetHealer::effective_heal_page_object_concurrency_for_source(
HealRequestSource::AutoHeal,
HealScanMode::Normal,
false,
),
1
);
@@ -2136,6 +2181,73 @@ mod tests {
);
}
#[test]
fn automatic_replacement_uses_a_conservative_default_and_respects_serial_controls() {
temp_env::with_vars(
[
(rustfs_config::ENV_HEAL_PAGE_PARALLEL_ENABLE, Some("true")),
(rustfs_config::ENV_HEAL_PAGE_OBJECT_CONCURRENCY, None),
],
|| {
assert_eq!(
ErasureSetHealer::effective_heal_page_object_concurrency_for_source(
HealRequestSource::AutoHeal,
HealScanMode::Normal,
true,
),
2
);
assert_eq!(
ErasureSetHealer::effective_heal_page_object_concurrency_for_source(
HealRequestSource::AutoHeal,
HealScanMode::Deep,
true,
),
1
);
},
);
temp_env::with_vars(
[
(rustfs_config::ENV_HEAL_PAGE_PARALLEL_ENABLE, Some("false")),
(rustfs_config::ENV_HEAL_PAGE_OBJECT_CONCURRENCY, Some("8")),
],
|| {
assert_eq!(
ErasureSetHealer::effective_heal_page_object_concurrency_for_source(
HealRequestSource::AutoHeal,
HealScanMode::Normal,
true,
),
1
);
},
);
}
#[test]
fn automatic_replacement_concurrency_fits_the_semaphore() {
let maximum = usize::MAX.to_string();
for (value, expected) in [("0", 1), (maximum.as_str(), tokio::sync::Semaphore::MAX_PERMITS)] {
temp_env::with_vars(
[
(rustfs_config::ENV_HEAL_PAGE_PARALLEL_ENABLE, Some("true")),
(rustfs_config::ENV_HEAL_PAGE_OBJECT_CONCURRENCY, Some(value)),
],
|| {
assert_eq!(
ErasureSetHealer::effective_heal_page_object_concurrency_for_source(
HealRequestSource::AutoHeal,
HealScanMode::Normal,
true,
),
expected
);
},
);
}
}
#[test]
fn non_auto_normal_scan_page_object_concurrency_uses_effective_limit() {
temp_env::with_vars(
@@ -2148,6 +2260,7 @@ mod tests {
ErasureSetHealer::effective_heal_page_object_concurrency_for_source(
HealRequestSource::Admin,
HealScanMode::Normal,
false,
),
11
);
@@ -2520,6 +2633,8 @@ mod resume_loop_tests {
lifecycle_expired: Mutex<HashSet<String>>,
/// every heal_object call recorded as (name, version_id)
heal_calls: Mutex<Vec<(String, Option<String>)>>,
heal_gate: Mutex<Option<Arc<tokio::sync::Semaphore>>>,
heal_started: tokio::sync::Notify,
list_include_lifecycle_object_info: Mutex<Vec<bool>>,
replacement_target_identity_sequences: Mutex<VecDeque<Vec<ReplacementTargetIdentity>>>,
replacement_targets_unready: AtomicBool,
@@ -2704,6 +2819,11 @@ mod resume_loop_tests {
.lock()
.unwrap()
.push((object.to_string(), version_id.map(str::to_string)));
self.heal_started.notify_one();
let heal_gate = self.heal_gate.lock().unwrap().clone();
if let Some(gate) = heal_gate {
gate.acquire().await.unwrap().forget();
}
let key = compose_key(object, version_id);
let outcome = {
let mut outcomes = self.outcomes.lock().unwrap();
@@ -3898,6 +4018,287 @@ mod resume_loop_tests {
assert_eq!(unique.len(), 50, "no version healed twice");
}
#[tokio::test]
async fn auto_replacement_runs_two_object_repairs_within_the_configured_limit() {
temp_env::async_with_vars(
[
(rustfs_config::ENV_HEAL_PAGE_PARALLEL_ENABLE, Some("true")),
(rustfs_config::ENV_HEAL_PAGE_OBJECT_CONCURRENCY, Some("2")),
],
async {
let mut env = make_env().await;
env.healer.source = HealRequestSource::AutoHeal;
env.healer = env
.healer
.with_replacement_targets(vec!["replacement-a".to_string()], Some(env.task_id.clone()));
let gate = Arc::new(tokio::sync::Semaphore::new(0));
*env.storage.heal_gate.lock().unwrap() = Some(gate.clone());
let objects = ["a", "b", "c"];
for name in objects {
env.storage
.set_result(name, None, replacement_target_ok_result("replacement-a", name));
}
env.storage.set_page(
None,
Page {
items: objects.into_iter().map(|name| item(name, None, false)).collect(),
next: None,
truncated: false,
},
);
let observe = async {
let two_started = tokio::time::timeout(std::time::Duration::from_secs(2), async {
loop {
let notified = env.storage.heal_started.notified();
if env.storage.calls().len() >= 2 {
break;
}
notified.await;
}
})
.await
.is_ok();
let started = env.storage.calls().len();
gate.add_permits(objects.len());
assert!(two_started, "automatic replacement should start two repairs before either finishes");
assert_eq!(started, 2, "configured concurrency must bound in-flight repairs");
};
let ((processed, successful, failed, skipped, result), ()) = tokio::join!(run(&env), observe);
result.expect("all replacement objects must be verified");
assert_eq!((processed, successful, failed, skipped), (3, 3, 0, 0));
assert_eq!(env.storage.calls().len(), 3);
},
)
.await;
}
#[tokio::test]
async fn auto_replacement_without_object_locks_stays_serial() {
temp_env::async_with_vars(
[
(rustfs_config::ENV_HEAL_PAGE_PARALLEL_ENABLE, Some("true")),
(rustfs_config::ENV_HEAL_PAGE_OBJECT_CONCURRENCY, Some("2")),
],
async {
let mut env = make_env().await;
env.healer.source = HealRequestSource::AutoHeal;
env.healer.heal_opts.no_lock = true;
env.healer = env
.healer
.with_replacement_targets(vec!["replacement-a".to_string()], Some(env.task_id.clone()));
let gate = Arc::new(tokio::sync::Semaphore::new(0));
*env.storage.heal_gate.lock().unwrap() = Some(gate.clone());
for version in ["v1", "v2"] {
env.storage
.set_result("object", Some(version), replacement_target_ok_result("replacement-a", "object"));
}
env.storage.set_page(
None,
Page {
items: vec![item("object", Some("v1"), false), item("object", Some("v2"), false)],
next: None,
truncated: false,
},
);
let observe = async {
let first_started = tokio::time::timeout(std::time::Duration::from_secs(5), async {
loop {
let notified = env.storage.heal_started.notified();
if !env.storage.calls().is_empty() {
break;
}
notified.await;
}
})
.await
.is_ok();
let two_started = tokio::time::timeout(std::time::Duration::from_secs(2), async {
loop {
let notified = env.storage.heal_started.notified();
if env.storage.calls().len() >= 2 {
break;
}
notified.await;
}
})
.await
.is_ok();
gate.add_permits(2);
assert!(first_started, "the first object repair must start");
assert!(!two_started, "unlocked object repairs must not overlap");
};
let ((processed, successful, failed, skipped, result), ()) = tokio::join!(run(&env), observe);
result.expect("the serial replacement must finish");
assert_eq!((processed, successful, failed, skipped), (2, 2, 0, 0));
},
)
.await;
}
#[tokio::test]
async fn auto_replacement_retries_a_transient_object_before_rewinding_the_scan() {
let mut env = make_env().await;
env.healer.source = HealRequestSource::AutoHeal;
env.healer = env
.healer
.with_replacement_targets(vec!["replacement-a".to_string()], Some(env.task_id.clone()));
env.storage.set_page(
None,
Page {
items: vec![item("object", Some("version"), false)],
next: None,
truncated: false,
},
);
env.storage
.set_outcome("object", Some("version"), HealOutcome::RpcCancelled(1));
env.storage
.set_result("object", Some("version"), replacement_target_ok_result("replacement-a", "object"));
let (processed, successful, failed, skipped, result) = run(&env).await;
result.expect("the transient repair should succeed during the current pass");
assert_eq!((processed, successful, failed, skipped), (1, 1, 0, 0));
assert_eq!(env.storage.calls().len(), 2);
}
#[tokio::test]
async fn auto_replacement_releases_capacity_during_a_bounded_retry() {
temp_env::async_with_vars(
[
(rustfs_config::ENV_HEAL_PAGE_PARALLEL_ENABLE, Some("true")),
(rustfs_config::ENV_HEAL_PAGE_OBJECT_CONCURRENCY, Some("1")),
],
async {
let mut env = make_env().await;
env.healer.source = HealRequestSource::AutoHeal;
env.healer = env
.healer
.with_replacement_targets(vec!["replacement-a".to_string()], Some(env.task_id.clone()));
env.storage.set_page(
None,
Page {
items: vec![item("retry", None, false), item("ready", None, false)],
next: None,
truncated: false,
},
);
env.storage.set_outcome("retry", None, HealOutcome::Transient);
env.storage
.set_result("ready", None, replacement_target_ok_result("replacement-a", "ready"));
let (processed, successful, failed, skipped, result) = run(&env).await;
result.expect("a failed object is retained as a transient skip");
assert_eq!((processed, successful, failed, skipped), (2, 1, 0, 1));
let calls = env.storage.calls();
let retries: Vec<_> = calls.iter().enumerate().filter(|(_, call)| call.0 == "retry").collect();
let ready = calls.iter().position(|call| call.0 == "ready").expect("ready object ran");
assert_eq!(retries.len(), 3, "only two retries are allowed");
assert!(ready < retries[1].0, "the backoff must release the only page permit");
},
)
.await;
}
#[tokio::test]
async fn auto_replacement_cancellation_interrupts_object_retry() {
let mut env = make_env().await;
env.healer.source = HealRequestSource::AutoHeal;
env.healer = env
.healer
.with_replacement_targets(vec!["replacement-a".to_string()], Some(env.task_id.clone()));
env.storage.set_page(
None,
Page {
items: vec![item("retry", None, false)],
next: None,
truncated: false,
},
);
env.storage.set_outcome("retry", None, HealOutcome::Transient);
let cancel = env.healer.cancel_token.clone();
let observe = async {
loop {
let notified = env.storage.heal_started.notified();
if !env.storage.calls().is_empty() {
break;
}
notified.await;
}
cancel.cancel();
};
let ((_, successful, _, _, result), ()) = tokio::join!(run(&env), observe);
assert!(matches!(result, Err(Error::TaskCancelled)));
assert_eq!(successful, 0);
assert_eq!(env.storage.calls().len(), 1);
assert!(env.checkpoint.get_checkpoint().await.processed_objects.is_empty());
}
#[tokio::test]
async fn auto_replacement_cancel_retains_workers_until_inflight_repairs_settle() {
temp_env::async_with_vars(
[
(rustfs_config::ENV_HEAL_PAGE_PARALLEL_ENABLE, Some("true")),
(rustfs_config::ENV_HEAL_PAGE_OBJECT_CONCURRENCY, Some("2")),
],
async {
let mut env = make_env().await;
env.healer.source = HealRequestSource::AutoHeal;
let execution = crate::heal::storage::ReplacementExecution::for_test(Vec::new(), Vec::new());
env.healer = env
.healer
.with_replacement_targets(vec!["replacement-a".to_string()], Some(env.task_id.clone()))
.with_replacement_execution(Some(execution.clone()));
let gate = Arc::new(tokio::sync::Semaphore::new(0));
*env.storage.heal_gate.lock().unwrap() = Some(gate.clone());
for name in ["a", "b", "c"] {
env.storage
.set_result(name, None, replacement_target_ok_result("replacement-a", name));
}
env.storage.set_page(
None,
Page {
items: ["a", "b", "c"].into_iter().map(|name| item(name, None, false)).collect(),
next: None,
truncated: false,
},
);
let cancel = env.healer.cancel_token.clone();
let observe = async {
tokio::time::timeout(std::time::Duration::from_secs(5), async {
loop {
let notified = env.storage.heal_started.notified();
if env.storage.calls().len() >= 2 {
break;
}
notified.await;
}
})
.await
.expect("two repairs must start");
cancel.cancel();
};
let ((_, successful, _, _, result), ()) = tokio::join!(run(&env), observe);
assert!(matches!(result, Err(Error::TaskCancelled)));
assert_eq!(successful, 0);
assert_eq!(env.storage.calls().len(), 2, "queued work must not start after cancellation");
assert!(env.checkpoint.get_checkpoint().await.processed_objects.is_empty());
assert!(Arc::strong_count(&execution) >= 4, "in-flight workers must retain execution ownership");
gate.add_permits(2);
tokio::time::timeout(std::time::Duration::from_secs(5), async {
while Arc::strong_count(&execution) > 2 {
tokio::task::yield_now().await;
}
})
.await
.expect("workers must release execution ownership after their storage work settles");
},
)
.await;
}
#[tokio::test]
async fn test_resume_after_version_deleted_midheal_no_skip() {
let env = make_env().await;
@@ -4312,7 +4713,11 @@ mod resume_loop_tests {
assert!(!state.completed, "readback errors must leave the replacement task incomplete");
assert_eq!(state.retry_count, 1, "readback errors must arm the bounded retry path");
assert!(env.checkpoint.get_checkpoint().await.processed_objects.is_empty());
assert_eq!(env.storage.calls(), vec![("object".to_string(), Some("dm-v1".to_string()))]);
assert_eq!(
env.storage.calls(),
vec![("object".to_string(), Some("dm-v1".to_string())); 3],
"unverified readback must exhaust only the bounded object retries"
);
}
#[tokio::test]
+84 -3
View File
@@ -825,6 +825,21 @@ fn admin_heal_options_compatible(left: &HealOptions, right: &HealOptions) -> boo
&& left.no_lock == right.no_lock
}
fn replacement_request_compatible(
request: &HealRequest,
owner_type: &HealType,
owner_options: &HealOptions,
owner_source: HealRequestSource,
) -> bool {
request.source == HealRequestSource::AutoHeal
&& owner_source == HealRequestSource::AutoHeal
&& matches!(request.heal_type, HealType::ErasureSet { .. })
&& request.heal_type == *owner_type
&& request.options.pool_index == owner_options.pool_index
&& request.options.set_index == owner_options.set_index
&& admin_heal_options_compatible(&request.options, owner_options)
}
fn admin_overlap_rejection(
request: &HealRequest,
owner_type: &HealType,
@@ -1276,7 +1291,9 @@ impl HealManager {
request: HealRequest,
config: &HealConfig,
context: &'static str,
replacement_targets_frozen: bool,
) -> HealAdmissionDecision {
let replacement_targets_frozen = replacement_targets_frozen || request.retry_attempts != 0;
let queue_len = queue.len();
publish_heal_queue_length(queue);
let queue_capacity = config.queue_size;
@@ -1284,6 +1301,25 @@ impl HealManager {
let best_effort_reserve = best_effort_queue_reserve(queue_capacity);
let best_effort_source_reserves = best_effort_source_reserves(queue_capacity);
// Scanner and retry admission bypass submit_heal_request's duplicate
// checks. Keep their target ownership checks ahead of queue capacity.
if !request.force_start
&& request.source == HealRequestSource::AutoHeal
&& !request.heal_endpoints.is_empty()
&& let Some(owner_id) = queue
.queued_request_id_for_dedup_key(&PriorityHealQueue::make_dedup_key(&request))
.map(ToOwned::to_owned)
{
let mut admission = Self::duplicate_admission_for_request(&request, config);
if admission == HealAdmissionResult::Merged
&& !queue.merge_replacement_targets(&owner_id, &request, true, replacement_targets_frozen)
{
admission = HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning);
}
Self::record_admission_metric(request.source, admission, context);
return HealAdmissionDecision::new(admission);
}
if queue_len >= queue_capacity && !request.force_start {
if Self::can_displace_queued_work(&request)
&& queue.can_displace_lower_priority_where(request.priority, Self::queued_request_can_be_displaced)
@@ -1443,7 +1479,7 @@ impl HealManager {
let priority = request.priority;
let source = request.source;
match queue.push(request) {
match queue.push_with_replacement_scope(request, replacement_targets_frozen) {
QueuePushOutcome::Accepted => {
publish_heal_queue_length(queue);
Self::record_admission_metric(source, HealAdmissionResult::Accepted, context);
@@ -2260,7 +2296,7 @@ impl HealManager {
// HS-06: under the minio_error overlap policy an exact duplicate
// admin start reports the typed AlreadyRunning rejection instead
// of the silent merge (MinIO's ErrHealAlreadyRunning).
let admission = if (request.source == HealRequestSource::Admin
let mut admission = if (request.source == HealRequestSource::Admin
&& config.overlap_policy == HealOverlapPolicy::MinioError)
|| (duplicate_state != "queued" && mrf_notice_target.as_ref().is_some_and(|target| target.kind.is_durable()))
{
@@ -2270,6 +2306,44 @@ impl HealManager {
} else {
Self::duplicate_admission_for_request(&request, &config)
};
if matches!(admission, HealAdmissionResult::Merged)
&& request.source == HealRequestSource::AutoHeal
&& !request.heal_endpoints.is_empty()
{
let (owner_recovered, incoming_recovered) = {
// Match queue -> recovery-anchor ordering in the scanner.
let anchors = self
.replacement_recovery_anchors
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner());
(anchors.contains_key(&merged_task_id), anchors.contains_key(&request.id))
};
let incoming_frozen = incoming_recovered || request.retry_attempts != 0;
let covered = if incoming_frozen && request.id != merged_task_id {
false
} else if duplicate_state == "queued" {
queue.merge_replacement_targets(&merged_task_id, &request, !owner_recovered, incoming_frozen)
} else {
let owner = active_heals
.get(&merged_task_id)
.map(|task| (&task.heal_type, &task.options, task.source, &task.heal_endpoints))
.or_else(|| {
retrying_heals.get(&merged_task_id).map(|retry| {
let owner = &retry.request;
(&owner.heal_type, &owner.options, owner.source, &owner.heal_endpoints)
})
});
owner.is_some_and(|(heal_type, options, source, endpoints)| {
replacement_request_compatible(&request, heal_type, options, source)
&& request.heal_endpoints.iter().all(|endpoint| endpoints.contains(endpoint))
})
};
if !covered {
// A frozen generation cannot certify newly arriving disks.
// Leave them discoverable by the scanner for the next pass.
admission = HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning);
}
}
if matches!(admission, HealAdmissionResult::Merged)
&& let Some(target) = mrf_notice_target
{
@@ -2426,7 +2500,14 @@ impl HealManager {
let request_id = request.id.clone();
let request_heal_type = request.heal_type.clone();
let request_source = request.source;
let admission_decision = Self::admit_request_to_queue(&mut queue, request, &config, "submit");
let recovered_replacement = request.source == HealRequestSource::AutoHeal
&& !request.heal_endpoints.is_empty()
&& self
.replacement_recovery_anchors
.lock()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.contains_key(&request.id);
let admission_decision = Self::admit_request_to_queue(&mut queue, request, &config, "submit", recovered_replacement);
let admission = admission_decision.result;
if durable_handoff_required && !admission.is_admitted() {
self.root_recovery
+3 -1
View File
@@ -682,6 +682,7 @@ impl HealManager {
HealPriority::Low,
);
let recovery_anchor = recovery.as_ref().map(|(_, _, _, anchor)| anchor.clone());
let recovered_replacement = recovery.is_some();
if let Some((task_id, _, _, _)) = recovery {
req.id = task_id;
}
@@ -691,7 +692,8 @@ impl HealManager {
let endpoint_count = req.heal_endpoints.len();
let config = config.read().await;
let mut queue = heal_queue.lock().await;
let admission_decision = Self::admit_request_to_queue(&mut queue, req, &config, "auto_scan");
let admission_decision =
Self::admit_request_to_queue(&mut queue, req, &config, "auto_scan", recovered_replacement);
let admission = admission_decision.result;
let should_notify =
matches!(admission, HealAdmissionResult::Accepted) && config.event_driven_scheduler_enable;
+52
View File
@@ -70,6 +70,8 @@ pub(super) struct PriorityQueueItem {
pub(super) sequence: u64,
pub(super) dedup_key: String,
pub(super) request: HealRequest,
/// Execution or durable recovery fixed these targets; grace waits can have zero retries.
pub(super) replacement_targets_frozen: bool,
}
impl Eq for PriorityQueueItem {}
@@ -316,6 +318,10 @@ impl PriorityHealQueue {
}
pub(super) fn push(&mut self, request: HealRequest) -> QueuePushOutcome {
self.push_with_replacement_scope(request, false)
}
pub(super) fn push_with_replacement_scope(&mut self, request: HealRequest, frozen: bool) -> QueuePushOutcome {
let key = Self::make_dedup_key(&request);
// Check for duplicates unless the caller explicitly forces admission.
@@ -340,6 +346,7 @@ impl PriorityHealQueue {
sequence: self.sequence,
dedup_key: key,
request,
replacement_targets_frozen: frozen,
});
if is_best_effort_source(source) {
self.best_effort_queued += 1;
@@ -680,6 +687,51 @@ impl PriorityHealQueue {
self.heap.iter().map(|item| &item.request)
}
pub(super) fn merge_replacement_targets(
&mut self,
owner_id: &str,
request: &HealRequest,
allow_expand: bool,
incoming_frozen: bool,
) -> bool {
let Some(item) = self.heap.iter().find(|item| item.request.id == owner_id) else {
return false;
};
let owner = &item.request;
let same_id = request.id == owner.id;
let eligible = replacement_request_compatible(request, &owner.heal_type, &owner.options, owner.source)
&& (!incoming_frozen || same_id);
let covered = eligible
&& request
.heal_endpoints
.iter()
.all(|endpoint| owner.heal_endpoints.contains(endpoint));
let mark_frozen = incoming_frozen && same_id && !item.replacement_targets_frozen;
let can_expand = eligible
&& allow_expand
&& !item.replacement_targets_frozen
&& !incoming_frozen
&& owner.retry_attempts == 0
&& request.retry_attempts == 0
&& !owner.heal_endpoints.is_empty();
if !mark_frozen && (covered || !can_expand) {
return covered;
}
// Neither targets nor their frozen flag affect heap ordering.
let mut items = std::mem::take(&mut self.heap).into_vec();
if let Some(item) = items.iter_mut().find(|item| item.request.id == owner_id) {
item.replacement_targets_frozen |= mark_frozen;
if !covered && can_expand {
item.request.heal_endpoints.extend(request.heal_endpoints.iter().cloned());
item.request.heal_endpoints.sort_unstable();
item.request.heal_endpoints.dedup();
}
}
self.heap = BinaryHeap::from(items);
covered || can_expand
}
#[cfg(test)]
pub(super) fn contains_request_id(&self, request_id: &str) -> bool {
self.heap.iter().any(|item| item.request.id == request_id)
+26 -14
View File
@@ -30,6 +30,30 @@ pub(super) fn retry_admission_delay(
Some(delay)
}
pub(super) fn running_mainline_pacer(
request: &HealRequest,
config: &HealConfig,
provider: &Option<WorkloadSnapshotProviderRef>,
) -> Option<Arc<crate::heal::pacing::MainlinePacer>> {
let pace_running = request.source == HealRequestSource::Admin
|| (request.source == HealRequestSource::AutoHeal
&& matches!(&request.heal_type, HealType::ErasureSet { .. })
&& !request.heal_endpoints.is_empty());
if !pace_running || !config.mainline_throttle_enable {
return None;
}
provider.as_ref().and_then(|provider| {
crate::heal::pacing::MainlinePacer::new(
provider.clone(),
config.mainline_read_utilization_high_percent,
config.mainline_write_utilization_high_percent,
config.mainline_max_sleep,
request.source,
)
.map(Arc::new)
})
}
impl HealManager {
/// Start scheduler
pub(super) async fn start_scheduler(&self) -> Result<()> {
@@ -202,19 +226,7 @@ impl HealManager {
.unwrap_or_else(|poisoned| poisoned.into_inner())
.get(&request.id)
.cloned();
let mainline_pacer = if request.source == HealRequestSource::Admin && config.mainline_throttle_enable {
workload_provider.as_ref().and_then(|provider| {
crate::heal::pacing::MainlinePacer::new(
provider.clone(),
config.mainline_read_utilization_high_percent,
config.mainline_write_utilization_high_percent,
config.mainline_max_sleep,
)
.map(Arc::new)
})
} else {
None
};
let mainline_pacer = running_mainline_pacer(&request, &config, workload_provider);
let task = Arc::new(
HealTask::from_replacement_recovery_request(request, storage.clone(), replacement_resume_endpoint)
.with_mainline_pacer(mainline_pacer)
@@ -582,7 +594,7 @@ impl HealManager {
return;
}
let admission_decision =
Self::admit_request_to_queue(&mut queue, retry_request.clone(), &retry_config, "retry");
Self::admit_request_to_queue(&mut queue, retry_request.clone(), &retry_config, "retry", true);
let admission = admission_decision.result;
let should_notify = matches!(admission, HealAdmissionResult::Accepted)
&& retry_config.event_driven_scheduler_enable;
+295
View File
@@ -2505,6 +2505,301 @@ fn test_heal_request_and_task_metric_labels_match() {
assert_eq!(task.metric_set_label(), "pool_0_set_1");
}
#[tokio::test]
async fn automatic_replacement_queued_targets_merge_without_changing_fifo_or_token() {
let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None);
let owner = automatic_replacement_request(&["replacement-a"]);
let owner_id = owner.id.clone();
let created_at = owner.created_at;
let first = manager
.submit_heal_request_with_receipt(owner)
.await
.expect("queue first target");
assert_eq!(first.result, HealAdmissionResult::Accepted);
let other = HealRequest::object("bucket".to_string(), "unrelated".to_string(), None);
let other_id = other.id.clone();
manager.submit_heal_request(other).await.expect("queue unrelated work");
let merged = manager
.submit_heal_request_with_receipt(automatic_replacement_request(&["replacement-b", "replacement-a"]))
.await
.expect("merge all fresh targets");
assert_eq!(merged.result, HealAdmissionResult::Merged);
assert_eq!(merged.task_id, owner_id);
let mut queue = manager.heal_queue.lock().await;
assert_eq!(queue.len(), 2);
let owner = queue.pop_next().expect("original task remains first");
assert_eq!(owner.id, owner_id);
assert_eq!(owner.created_at, created_at);
assert_eq!(owner.heal_endpoints, ["replacement-a", "replacement-b"]);
assert_eq!(queue.pop_next().expect("unrelated task remains queued").id, other_id);
}
fn automatic_replacement_request(targets: &[&str]) -> HealRequest {
let mut request = HealRequest::new(
HealType::ErasureSet {
buckets: Vec::new(),
set_disk_id: "pool_0_set_1".to_string(),
},
HealOptions {
pool_index: Some(0),
set_index: Some(1),
..Default::default()
},
HealPriority::Normal,
);
request.source = HealRequestSource::AutoHeal;
request.heal_endpoints = targets.iter().map(|target| (*target).to_string()).collect();
request
}
#[tokio::test]
async fn automatic_replacement_frozen_targets_reject_late_disks_but_cover_duplicates() {
for state in ["active", "retrying", "grace", "recovered", "recovery_anchor"] {
let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage);
let manager = HealManager::new_without_root_recovery_for_test(storage.clone(), None);
let mut owner = automatic_replacement_request(&["replacement-a"]);
let owner_id = owner.id.clone();
if state == "grace" {
let now = SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.expect("test clock")
.as_secs();
let task = HealTask::from_request(owner, storage.clone());
owner = retry_request_for_result_with_budget(
&task,
&Err(Error::DanglingDeleteDeferred {
retry_not_before: now + 60,
}),
)
.await
.expect("grace wait keeps the generation replayable")
.0;
assert_eq!(owner.retry_attempts, 0, "grace must not consume a failure retry");
}
match state {
"active" => {
manager
.active_heals
.lock()
.await
.insert(owner_id.clone(), Arc::new(HealTask::from_request(owner, storage.clone())));
}
"retrying" => {
manager.retrying_heals.lock().await.insert(
owner_id.clone(),
RetryingHeal {
request: owner,
error: "target unavailable".to_string(),
cancel_token: CancellationToken::new(),
},
);
}
"grace" | "recovered" => {
let config = manager.config.read().await.clone();
let mut queue = manager.heal_queue.lock().await;
let context = if state == "grace" { "retry" } else { "auto_scan" };
assert_eq!(
HealManager::admit_request_to_queue(&mut queue, owner, &config, context, true).result,
HealAdmissionResult::Accepted
);
assert!(
queue
.heap
.iter()
.any(|item| item.request.id == owner_id && item.replacement_targets_frozen),
"zero retries cannot mean the scope is fresh"
);
}
_ => {
manager.submit_heal_request(owner).await.expect("queue frozen generation");
if state == "recovery_anchor" {
manager
.replacement_recovery_anchors
.lock()
.expect("recovery anchors")
.insert(owner_id.clone(), "survivor".to_string());
}
}
}
let late = manager
.submit_heal_request(automatic_replacement_request(&["replacement-b"]))
.await
.expect("classify late target");
assert_eq!(late, HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning), "{state}");
let covered = manager
.submit_heal_request_with_receipt(automatic_replacement_request(&["replacement-a"]))
.await
.expect("covered target remains owned");
assert_eq!(covered.result, HealAdmissionResult::Merged, "{state}");
assert_eq!(covered.task_id, owner_id, "{state}");
let targets = match state {
"active" => manager.active_heals.lock().await[&owner_id].heal_endpoints.clone(),
"retrying" => manager.retrying_heals.lock().await[&owner_id].request.heal_endpoints.clone(),
_ => {
manager
.heal_queue
.lock()
.await
.pop_next()
.expect("retained generation")
.heal_endpoints
}
};
assert_eq!(targets, ["replacement-a"], "{state}: frozen scope must not grow");
}
}
#[tokio::test]
async fn automatic_replacement_merge_preserves_options_policy_and_recovery_identity() {
let manager = HealManager::new_without_root_recovery_for_test(Arc::new(MockStorage), None);
manager
.submit_heal_request(automatic_replacement_request(&["replacement-a"]))
.await
.expect("queue fresh generation");
for variant in [
"dry_run",
"bucket_scope",
"pool_scope",
"retried_incoming",
"recovery_anchor",
"merge_disabled",
] {
let mut incoming = automatic_replacement_request(&["replacement-b"]);
match variant {
"dry_run" => incoming.options.dry_run = true,
"bucket_scope" => {
if let HealType::ErasureSet { buckets, .. } = &mut incoming.heal_type {
buckets.push("other-bucket".to_string());
}
}
"pool_scope" => incoming.options.pool_index = Some(2),
"retried_incoming" => {
incoming.heal_endpoints = vec!["replacement-a".to_string()];
incoming.retry_attempts = 1;
}
"recovery_anchor" => {
incoming.heal_endpoints = vec!["replacement-a".to_string()];
manager
.replacement_recovery_anchors
.lock()
.expect("recovery anchors")
.insert(incoming.id.clone(), "survivor".to_string());
}
"merge_disabled" => {
manager.config.write().await.low_priority_merge_enable = false;
incoming.priority = HealPriority::Low;
}
_ => unreachable!("test variant"),
}
let result = manager
.submit_heal_request(incoming)
.await
.expect("classify incompatible generation");
let reason = if variant == "merge_disabled" {
HealAdmissionDropReason::PolicyDropped
} else {
HealAdmissionDropReason::AlreadyRunning
};
assert_eq!(result, HealAdmissionResult::Dropped(reason), "{variant}");
let queue = manager.heal_queue.lock().await;
assert_eq!(queue.len(), 1, "{variant}");
assert_eq!(
queue.requests().next().expect("original generation").heal_endpoints,
["replacement-a"],
"{variant}"
);
}
}
#[test]
fn automatic_replacement_direct_admission_checks_frozen_targets_before_capacity() {
let config = HealConfig {
queue_size: 1,
..HealConfig::default()
};
let mut queue = PriorityHealQueue::new();
let owner = automatic_replacement_request(&["replacement-a"]);
let owner_id = owner.id.clone();
assert_eq!(queue.push(owner), QueuePushOutcome::Accepted);
let admit = |queue: &mut PriorityHealQueue, request, frozen| {
HealManager::admit_request_to_queue(queue, request, &config, "auto_scan", frozen).result
};
assert_eq!(
admit(&mut queue, automatic_replacement_request(&["replacement-b"]), false),
HealAdmissionResult::Merged
);
let mut replay = automatic_replacement_request(&["replacement-a"]);
replay.id = owner_id.clone();
assert_eq!(admit(&mut queue, replay, true), HealAdmissionResult::Merged);
assert!(
queue
.heap
.iter()
.any(|item| item.request.id == owner_id && item.replacement_targets_frozen)
);
assert_eq!(
admit(&mut queue, automatic_replacement_request(&["replacement-c"]), false),
HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning)
);
assert_eq!(
admit(&mut queue, automatic_replacement_request(&["replacement-a"]), false),
HealAdmissionResult::Merged
);
assert_eq!(
admit(&mut queue, automatic_replacement_request(&["replacement-a"]), true),
HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning)
);
assert_eq!(queue.len(), 1);
let owner = queue.pop_next().expect("frozen owner retained");
assert_eq!(owner.id, owner_id);
assert_eq!(owner.heal_endpoints, ["replacement-a", "replacement-b"]);
let owner = automatic_replacement_request(&["replacement-a"]);
let owner_id = owner.id.clone();
assert_eq!(queue.push(owner), QueuePushOutcome::Accepted);
let mut conflicting_replay = automatic_replacement_request(&["replacement-b"]);
conflicting_replay.id = owner_id.clone();
assert_eq!(
admit(&mut queue, conflicting_replay, true),
HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning)
);
assert!(
queue
.heap
.iter()
.any(|item| item.request.id == owner_id && item.replacement_targets_frozen)
);
assert_eq!(
admit(&mut queue, automatic_replacement_request(&["replacement-c"]), false),
HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning)
);
let mut queue = PriorityHealQueue::new();
let owner = automatic_replacement_request(&["replacement-a"]);
let owner_id = owner.id.clone();
assert_eq!(queue.push(owner), QueuePushOutcome::Accepted);
let mut incompatible_replay = automatic_replacement_request(&["replacement-a"]);
incompatible_replay.id = owner_id.clone();
incompatible_replay.options.dry_run = true;
assert_eq!(
admit(&mut queue, incompatible_replay, true),
HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning)
);
assert!(
queue
.heap
.iter()
.any(|item| item.request.id == owner_id && item.replacement_targets_frozen)
);
assert_eq!(
admit(&mut queue, automatic_replacement_request(&["replacement-b"]), false),
HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning)
);
}
#[tokio::test]
async fn test_submit_heal_request_returns_merged_for_duplicate() {
let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage);
@@ -258,6 +258,16 @@ impl ScannerFixture {
assert_eq!(request.id, self.task_id, "scanner must replay the durable UUID");
assert_eq!(request.heal_endpoints, vec![self.local_targets[0].clone()]);
assert_eq!(request.source, HealRequestSource::AutoHeal);
assert!(
self.manager
.heal_queue
.lock()
.await
.heap
.iter()
.any(|item| item.request.id == request.id && item.replacement_targets_frozen),
"replayed durable targets cannot expand"
);
assert!(matches!(
request.heal_type,
HealType::ErasureSet { set_disk_id, buckets }
@@ -6,6 +6,41 @@ use crate::heal::storage::HealListItem;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
use tokio::sync::Semaphore;
#[tokio::test(start_paused = true)]
async fn automatic_replacement_uses_running_foreground_pacing() {
let provider = Arc::new(super::super::super::pacing::TestPressure::new(WorkloadClass::ForegroundRead, 100));
let provider: Option<WorkloadSnapshotProviderRef> = Some(provider);
let mut config = HealConfig {
mainline_throttle_enable: true,
mainline_read_utilization_high_percent: 80,
mainline_max_sleep: Duration::from_millis(250),
..HealConfig::default()
};
let mut request = HealRequest::new(
HealType::ErasureSet {
buckets: Vec::new(),
set_disk_id: "pool_0_set_0".to_owned(),
},
HealOptions::default(),
HealPriority::Low,
);
request.source = HealRequestSource::AutoHeal;
request.heal_endpoints = vec!["replacement-a".to_owned()];
let pacer = super::super::scheduler::running_mainline_pacer(&request, &config, &provider)
.expect("automatic replacement must pace under foreground pressure");
let started = tokio::time::Instant::now();
pacer.wait(&CancellationToken::new()).await.expect("bounded progress");
assert_eq!(started.elapsed(), Duration::from_millis(250));
assert!(super::super::scheduler::running_mainline_pacer(&request, &config, &None).is_none());
config.mainline_throttle_enable = false;
assert!(super::super::scheduler::running_mainline_pacer(&request, &config, &provider).is_none());
config.mainline_throttle_enable = true;
request.heal_endpoints.clear();
assert!(super::super::scheduler::running_mainline_pacer(&request, &config, &provider).is_none());
}
#[derive(Default)]
struct PressureProbe {
active: AtomicUsize,
+15 -7
View File
@@ -6,6 +6,7 @@ use rustfs_concurrency::{
WorkloadAdmissionSnapshotProvider,
workload::{ForegroundPressure, foreground_pressure},
};
use rustfs_heal_contracts::heal_channel::HealRequestSource;
use std::{sync::Arc, time::Duration};
use tokio::{sync::Mutex, time::Instant};
use tokio_util::sync::CancellationToken;
@@ -21,12 +22,13 @@ pub(crate) enum PacingDecision {
Wait(Option<ForegroundPressure>),
}
/// Cooperative pacing for one admin execution, not a storage admission permit.
/// Cooperative pacing for one heal execution, not a storage admission permit.
pub(crate) struct MainlinePacer {
provider: Arc<dyn WorkloadAdmissionSnapshotProvider + Send + Sync>,
read_high: usize,
write_high: usize,
pause: Duration,
source: HealRequestSource,
state: Mutex<PacingState>,
}
@@ -36,6 +38,7 @@ impl MainlinePacer {
read_high: usize,
write_high: usize,
pause: Duration,
source: HealRequestSource,
) -> Option<Self> {
if (read_high == 0 && write_high == 0) || pause.is_zero() {
return None;
@@ -45,6 +48,7 @@ impl MainlinePacer {
read_high: read_high.min(100),
write_high: write_high.min(100),
pause: pause.min(Duration::from_secs(1)),
source,
state: Mutex::new(PacingState::default()),
})
}
@@ -108,7 +112,7 @@ impl MainlinePacer {
}
metrics::counter!(
"rustfs_heal_mainline_throttle_total",
"source" => "admin",
"source" => self.source.as_str(),
"result" => "delayed",
"reason" => pressure.or(observed).map_or("recovery_window", |pressure| pressure.reason())
)
@@ -170,6 +174,7 @@ mod tests {
if class == WorkloadClass::ForegroundRead { 40 } else { 0 },
if class == WorkloadClass::ForegroundWrite { 40 } else { 0 },
pause,
HealRequestSource::Admin,
)
.expect("enabled pacer");
let cancel = CancellationToken::new();
@@ -192,9 +197,9 @@ mod tests {
#[tokio::test(start_paused = true)]
async fn running_mainline_huge_pause_is_capped_and_disabled_classes_do_not_sleep() {
let provider = Arc::new(TestPressure::new(WorkloadClass::ForegroundRead, 100));
assert!(MainlinePacer::new(provider.clone(), 0, 0, Duration::from_secs(1)).is_none());
assert!(MainlinePacer::new(provider.clone(), 80, 80, Duration::ZERO).is_none());
let pacer = MainlinePacer::new(provider, 80, 80, Duration::from_secs(3600)).expect("pacer");
assert!(MainlinePacer::new(provider.clone(), 0, 0, Duration::from_secs(1), HealRequestSource::Admin).is_none());
assert!(MainlinePacer::new(provider.clone(), 80, 80, Duration::ZERO, HealRequestSource::Admin).is_none());
let pacer = MainlinePacer::new(provider, 80, 80, Duration::from_secs(3600), HealRequestSource::Admin).expect("pacer");
let before = Instant::now();
pacer.wait(&CancellationToken::new()).await.expect("hard-capped pause");
assert_eq!(Instant::now() - before, Duration::from_secs(1));
@@ -203,7 +208,9 @@ mod tests {
#[tokio::test(start_paused = true)]
async fn running_mainline_waiters_cancel_and_task_latches_are_isolated() {
let provider = Arc::new(TestPressure::new(WorkloadClass::ForegroundRead, 100));
let paced = Arc::new(MainlinePacer::new(provider.clone(), 80, 80, Duration::from_secs(1)).expect("pacer"));
let paced = Arc::new(
MainlinePacer::new(provider.clone(), 80, 80, Duration::from_secs(1), HealRequestSource::Admin).expect("pacer"),
);
let cancel_first = CancellationToken::new();
let first = tokio::spawn({
let paced = paced.clone();
@@ -221,7 +228,8 @@ mod tests {
cancel_second.cancel();
assert!(matches!(second.await.expect("queued waiter"), Err(Error::TaskCancelled)));
provider.active.store(0, Ordering::SeqCst);
let other_task = MainlinePacer::new(provider.clone(), 80, 80, Duration::from_secs(1)).expect("independent task");
let other_task = MainlinePacer::new(provider.clone(), 80, 80, Duration::from_secs(1), HealRequestSource::Admin)
.expect("independent task");
let before = Instant::now();
other_task
.wait(&CancellationToken::new())
+8 -4
View File
@@ -489,9 +489,13 @@ impl HealTask {
))
.await?;
if let Some(error) = storage_result.error {
return Err(Error::TaskExecutionFailed {
message: format!("Failed to recreate missing object {bucket}/{object}: {error}"),
});
if error.is_dangling_delete_grace() {
// Preserve the existing MRF grace retry ownership.
return Err(Error::TaskExecutionFailed {
message: format!("Failed to recreate missing object {bucket}/{object}: {error}"),
});
}
return Err(error);
}
let object_size = storage_result.item.object_size as u64;
@@ -526,7 +530,7 @@ impl HealTask {
}
} else {
Error::TaskExecutionFailed {
message: format!("Missing exact storage proof for durable MRF repair {bucket}/{object}"),
message: format!("{}{bucket}/{object}", crate::error::MRF_MISSING_PROOF_PREFIX),
}
});
}
+38
View File
@@ -2912,6 +2912,44 @@ fn mrf_task_for_item(object: &str, item: HealResultItem) -> (HealTask, Arc<MockS
(HealTask::from_request(request, storage.clone()), storage)
}
#[tokio::test]
async fn durable_mrf_slowdown_remains_retryable_without_a_storage_receipt() {
let (task, storage) = mrf_task_for_item("object", HealResultItem::default());
*storage.heal_object_outcome.lock().unwrap() = Some(MockHealObjectOutcome::RetryableSlowDown);
let error = task
.execute()
.await
.expect_err("a failed repair cannot discharge the responsibility");
assert!(matches!(error, Error::Storage(EcstoreError::SlowDown)), "{error:?}");
assert!(error.is_recoverable_heal());
assert_eq!(task.get_outcome().await.counters.healed, 0);
}
#[tokio::test]
async fn durable_mrf_grace_remains_owned_by_the_responsibility() {
let (task, storage) = mrf_task_for_item("object", HealResultItem::default());
*storage.heal_object_outcome.lock().unwrap() = Some(MockHealObjectOutcome::DanglingGraceDeferred);
let error = task.execute().await.expect_err("grace must retain the responsibility");
assert!(error.is_dangling_delete_grace());
assert!(!error.is_recoverable_heal());
assert_eq!(task.get_outcome().await.counters.healed, 0);
}
#[tokio::test]
async fn durable_mrf_unverified_object_names_cannot_change_retry_classification() {
for object in ["ordinary", "timed out", "network error"] {
let (task, _) = mrf_task_for_item(object, HealResultItem::default());
let error = task
.execute()
.await
.expect_err("missing proof must retain the responsibility");
assert!(!error.is_recoverable_heal(), "object name must not change retry classification: {object}");
assert_eq!(task.get_outcome().await.counters.healed, 0);
}
}
#[tokio::test]
async fn mrf_legacy_object_names_cannot_change_retry_classification() {
for object in [
@@ -0,0 +1,29 @@
// Copyright 2026 RustFS Team
// Licensed under the Apache License, Version 2.0.
use rustfs_heal::{HealOptions, HealPriority, HealRequest, HealType};
use rustfs_heal_contracts::heal_channel::HealRequestSource;
use std::time::SystemTime;
#[test]
fn existing_heal_request_literal_remains_constructible() {
let now = SystemTime::now();
let request = HealRequest {
id: "request".to_string(),
heal_type: HealType::ErasureSet {
set_disk_id: "pool_0_set_0".to_string(),
buckets: Vec::new(),
},
bucket_incarnation_id: None,
expected_mrf_bucket_incarnation_id: None,
options: HealOptions::default(),
priority: HealPriority::Normal,
source: HealRequestSource::AutoHeal,
force_start: false,
retry_attempts: 0,
heal_endpoints: vec!["drive-a".to_string()],
created_at: now,
enqueued_at: now,
};
assert_eq!(request.heal_endpoints, ["drive-a"]);
}
+18
View File
@@ -26,6 +26,9 @@ use crate::{
/// Default shard count for guard storage (must be power of 2)
const DEFAULT_GUARD_SHARD_COUNT: usize = 64;
const EVENT_NAMESPACE_LOCK_RELEASE: &str = "namespace_lock_release";
const LOG_COMPONENT_LOCK: &str = "lock";
const LOG_SUBSYSTEM_NAMESPACE: &str = "namespace";
type GuardShard = Arc<RwLock<HashMap<LockId, LocalGuardEntry>>>;
type GuardStorage = Arc<Vec<GuardShard>>;
@@ -333,7 +336,22 @@ impl LockClient for LocalClient {
};
if let Some(guard) = guard {
// Guard automatically releases the lock when dropped
let physical_guard_id = (!guard.guard.is_disabled()).then(|| guard.guard.guard_id());
drop(guard.guard);
if let Some(physical_guard_id) = physical_guard_id {
tracing::trace!(
target: "rustfs_lock::namespace_owner",
event = EVENT_NAMESPACE_LOCK_RELEASE,
component = LOG_COMPONENT_LOCK,
subsystem = LOG_SUBSYSTEM_NAMESPACE,
state = "mapped",
peer_kind = "local",
lock_id = %lock_id,
process_id = std::process::id(),
physical_guard_id,
"Namespace peer lock mapped to physical guard"
);
}
Ok(true)
} else {
// Lock not found or already released
+5
View File
@@ -412,6 +412,11 @@ impl DistributedLockGuard {
&self.lock_id
}
/// Actual granted peer identities, which may differ from the logical guard ID.
pub fn granted_lock_ids(&self) -> impl Iterator<Item = &LockId> {
self.entries.iter().map(|entry| &entry.lock_id)
}
/// Whether the guard's heartbeat has observed a refresh-quorum loss.
pub fn is_lock_lost(&self) -> bool {
self.lock_lost.is_lost()
+23
View File
@@ -24,6 +24,9 @@ use std::sync::atomic::{AtomicU64, Ordering};
/// Global counter for guard IDs to prevent double-release
static GUARD_ID_COUNTER: AtomicU64 = AtomicU64::new(1);
const EVENT_NAMESPACE_LOCK_RELEASE: &str = "namespace_lock_release";
const LOG_COMPONENT_LOCK: &str = "lock";
const LOG_SUBSYSTEM_NAMESPACE: &str = "namespace";
/// RAII guard for fast object locks
///
@@ -109,6 +112,7 @@ impl FastLockGuard {
record_lock_held_release(self.mode);
// Unregister the guard after successful release
shard.unregister_guard(self.guard_id);
self.record_physical_release();
}
success
} else {
@@ -118,6 +122,22 @@ impl FastLockGuard {
}
}
fn record_physical_release(&self) {
tracing::trace!(
target: "rustfs_lock::namespace_owner",
event = EVENT_NAMESPACE_LOCK_RELEASE,
component = LOG_COMPONENT_LOCK,
subsystem = LOG_SUBSYSTEM_NAMESPACE,
state = "released",
peer_kind = "local",
process_id = std::process::id(),
physical_guard_id = self.guard_id,
resource = %self.key,
owner = %self.owner,
"Namespace physical guard release completed"
);
}
/// Check if the lock has been released
pub fn is_released(&self) -> bool {
self.released
@@ -168,6 +188,9 @@ impl Drop for FastLockGuard {
}
// Always unregister the guard to prevent leaks, regardless of release success
shard.unregister_guard(self.guard_id);
if success {
self.record_physical_release();
}
} else {
// If guard was already released or disabled, just unregister it
shard.unregister_guard(self.guard_id);
+235 -13
View File
@@ -39,7 +39,37 @@ struct GuardHolderKey {
owner: Arc<str>,
}
/// Lock shard to reduce global contention
#[cfg(test)]
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq)]
enum TestPausePoint {
FastExisting,
BeforeFastWrite,
FastWriteExisting,
SlowLookup,
}
#[cfg(test)]
struct TestPause {
entered: std::sync::mpsc::Sender<()>,
resume: std::sync::mpsc::Receiver<()>,
}
#[cfg(test)]
static TEST_PAUSES: std::sync::LazyLock<parking_lot::Mutex<HashMap<(TestPausePoint, ObjectKey), TestPause>>> =
std::sync::LazyLock::new(|| parking_lot::Mutex::new(HashMap::new()));
#[cfg(test)]
fn test_pause(point: TestPausePoint, key: &ObjectKey) {
let pause = TEST_PAUSES.lock().remove(&(point, key.clone()));
if let Some(pause) = pause {
pause.entered.send(()).expect("pause receiver");
pause.resume.recv_timeout(Duration::from_secs(5)).expect("resume signal");
}
}
/// Lock shard to reduce global contention.
///
/// When both are needed, lock `objects` before a state's owner metadata.
#[derive(Debug)]
pub struct LockShard {
/// Object lock states - using parking_lot for better performance
@@ -133,13 +163,16 @@ impl LockShard {
let objects = self.objects.read();
if let Some(state) = objects.get(&request.key) {
let state = state.clone();
drop(objects);
#[cfg(test)]
test_pause(TestPausePoint::FastExisting, &request.key);
// Try atomic acquisition
let success = match request.mode {
LockMode::Shared => state.try_acquire_shared_fast(&request.owner, request.lock_timeout),
LockMode::Exclusive => state.try_acquire_exclusive_fast(&request.owner, request.lock_timeout),
};
drop(objects);
if success {
return Some(state);
@@ -150,14 +183,21 @@ impl LockShard {
// If object doesn't exist and we're requesting exclusive lock,
// try to create and acquire atomically
if request.mode == LockMode::Exclusive {
#[cfg(test)]
test_pause(TestPausePoint::BeforeFastWrite, &request.key);
let mut objects = self.objects.write();
// Double-check after acquiring write lock
if let Some(state) = objects.get(&request.key) {
let state = state.clone();
drop(objects);
let objects = parking_lot::RwLockWriteGuard::downgrade(objects);
if state.try_acquire_exclusive_fast(&request.owner, request.lock_timeout) {
#[cfg(test)]
test_pause(TestPausePoint::FastWriteExisting, &request.key);
let acquired = state.try_acquire_exclusive_fast(&request.owner, request.lock_timeout);
drop(objects);
if acquired {
return Some(state);
}
} else {
@@ -193,10 +233,10 @@ impl LockShard {
const NOTIFY_WAIT_CAP: Duration = Duration::from_millis(50);
loop {
// Get or create object state
let state = {
// Keep the state in the map until its acquisition attempt completes.
let (state, success) = {
let mut objects = self.objects.write();
match objects.get(&request.key) {
let state = match objects.get(&request.key) {
Some(state) => state.clone(),
None => {
let state_box = self.object_pool.acquire();
@@ -204,13 +244,17 @@ impl LockShard {
objects.insert(request.key.clone(), state.clone());
state
}
}
};
};
let objects = parking_lot::RwLockWriteGuard::downgrade(objects);
// Try acquisition again
let success = match request.mode {
LockMode::Shared => state.try_acquire_shared_fast(&request.owner, request.lock_timeout),
LockMode::Exclusive => state.try_acquire_exclusive_fast(&request.owner, request.lock_timeout),
#[cfg(test)]
test_pause(TestPausePoint::SlowLookup, &request.key);
let success = match request.mode {
LockMode::Shared => state.try_acquire_shared_fast(&request.owner, request.lock_timeout),
LockMode::Exclusive => state.try_acquire_exclusive_fast(&request.owner, request.lock_timeout),
};
drop(objects);
(state, success)
};
if success {
@@ -905,6 +949,184 @@ impl LockShard {
mod tests {
use super::*;
use crate::fast_lock::types::{LockPriority, ObjectKey};
use crate::fast_lock::{FastLockGuard, FastObjectLockManager};
fn install_pause(point: TestPausePoint, key: &ObjectKey) -> (std::sync::mpsc::Receiver<()>, std::sync::mpsc::Sender<()>) {
let (entered_tx, entered_rx) = std::sync::mpsc::channel();
let (resume_tx, resume_rx) = std::sync::mpsc::channel();
let previous = TEST_PAUSES.lock().insert(
(point, key.clone()),
TestPause {
entered: entered_tx,
resume: resume_rx,
},
);
assert!(previous.is_none());
(entered_rx, resume_tx)
}
fn test_manager() -> Arc<FastObjectLockManager> {
// Construct outside a runtime so no background cleanup races the test.
Arc::new(
std::thread::spawn(|| {
FastObjectLockManager::with_config(crate::fast_lock::types::LockConfig {
shard_count: 4,
..Default::default()
})
})
.join()
.expect("manager construction"),
)
}
fn start_acquisition(
manager: &Arc<FastObjectLockManager>,
request: ObjectLockRequest,
) -> tokio::task::JoinHandle<Result<FastLockGuard, LockResult>> {
let manager = manager.clone();
tokio::spawn(async move { manager.acquire_lock(request).await })
}
async fn assert_cleanup_cannot_split_state(
manager: Arc<FastObjectLockManager>,
key: ObjectKey,
entered: std::sync::mpsc::Receiver<()>,
resume: std::sync::mpsc::Sender<()>,
pending: tokio::task::JoinHandle<Result<FastLockGuard, LockResult>>,
seed: Option<FastLockGuard>,
) {
entered.recv_timeout(Duration::from_secs(5)).expect("state lookup");
drop(seed);
// Idle timestamps have one-second precision, while cleanup compares milliseconds.
let initial_millis = SystemTime::now().duration_since(SystemTime::UNIX_EPOCH).unwrap().as_millis();
while SystemTime::now().duration_since(SystemTime::UNIX_EPOCH).unwrap().as_millis() == initial_millis {
std::thread::yield_now();
}
let can_write_map = manager.get_shard(&key).objects.try_write().is_some();
let (started_tx, started_rx) = std::sync::mpsc::channel();
let mut cleanup = {
let manager = manager.clone();
let key = key.clone();
tokio::task::spawn_blocking(move || {
started_tx.send(()).expect("cleanup start receiver");
manager.get_shard(&key).cleanup_expired_millis(0)
})
};
started_rx.recv_timeout(Duration::from_secs(5)).expect("cleanup started");
if can_write_map {
assert_eq!(
tokio::time::timeout(Duration::from_secs(5), &mut cleanup)
.await
.expect("cleanup completed")
.expect("cleanup task"),
1
);
let successor = manager
.acquire_write_lock(key.clone(), "second")
.await
.expect("successor should acquire after cleanup");
resume.send(()).expect("resume acquisition");
let result = tokio::time::timeout(Duration::from_secs(5), pending)
.await
.expect("acquisition completed")
.expect("acquisition task");
assert!(result.is_err(), "a removed state must not grant a conflicting lock");
drop(successor);
} else {
resume.send(()).expect("resume acquisition");
let first = tokio::time::timeout(Duration::from_secs(5), pending)
.await
.expect("acquisition completed")
.expect("acquisition task")
.expect("first acquisition");
assert_eq!(
tokio::time::timeout(Duration::from_secs(5), cleanup)
.await
.expect("cleanup completed")
.expect("cleanup task"),
0
);
let second = ObjectLockRequest::new_write(key.clone(), "second").with_acquire_timeout(Duration::from_millis(100));
assert!(
manager.acquire_lock(second).await.is_err(),
"conflicting locks must remain mutually exclusive"
);
drop(first);
let successor = manager
.acquire_write_lock(key, "second")
.await
.expect("successor after release");
drop(successor);
}
manager.shutdown().await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_cleanup_during_fast_existing_state_acquisition() {
let manager = test_manager();
let key = ObjectKey::new("bucket", "fast-existing-cleanup");
drop(manager.acquire_write_lock(key.clone(), "seed").await.expect("seed"));
let (entered, resume) = install_pause(TestPausePoint::FastExisting, &key);
let pending = start_acquisition(&manager, ObjectLockRequest::new_write(key.clone(), "first"));
assert_cleanup_cannot_split_state(manager, key, entered, resume, pending, None).await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_cleanup_during_fast_write_recheck_acquisition() {
let manager = test_manager();
let key = ObjectKey::new("bucket", "fast-recheck-cleanup");
let (first_entered, first_resume) = install_pause(TestPausePoint::BeforeFastWrite, &key);
let pending = start_acquisition(&manager, ObjectLockRequest::new_write(key.clone(), "first"));
first_entered
.recv_timeout(Duration::from_secs(5))
.expect("before write recheck");
drop(manager.acquire_write_lock(key.clone(), "seed").await.expect("seed"));
let (entered, resume) = install_pause(TestPausePoint::FastWriteExisting, &key);
first_resume.send(()).expect("resume before write recheck");
assert_cleanup_cannot_split_state(manager, key, entered, resume, pending, None).await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_cleanup_during_slow_existing_state_acquisition() {
let manager = test_manager();
let key = ObjectKey::new("bucket", "slow-existing-cleanup");
let seed = manager.acquire_write_lock(key.clone(), "seed").await.expect("seed");
let (entered, resume) = install_pause(TestPausePoint::SlowLookup, &key);
let pending = start_acquisition(&manager, ObjectLockRequest::new_write(key.clone(), "first"));
assert_cleanup_cannot_split_state(manager, key, entered, resume, pending, Some(seed)).await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
async fn test_cleanup_during_slow_new_state_acquisition() {
let manager = test_manager();
let key = ObjectKey::new("bucket", "slow-new-cleanup");
let (entered, resume) = install_pause(TestPausePoint::SlowLookup, &key);
let pending = start_acquisition(&manager, ObjectLockRequest::new_read(key.clone(), "first"));
assert_cleanup_cannot_split_state(manager, key, entered, resume, pending, None).await;
}
#[tokio::test]
async fn test_cleanup_keeps_new_fast_state_while_held() {
let manager = test_manager();
let key = ObjectKey::new("bucket", "fast-new-cleanup");
let first = manager
.acquire_write_lock(key.clone(), "first")
.await
.expect("first acquisition");
assert_eq!(manager.get_shard(&key).cleanup_expired_millis(0), 0);
let second = ObjectLockRequest::new_write(key.clone(), "second").with_acquire_timeout(Duration::from_millis(100));
assert!(manager.acquire_lock(second).await.is_err());
drop(first);
drop(
manager
.acquire_write_lock(key, "second")
.await
.expect("successor after release"),
);
manager.shutdown().await;
}
#[tokio::test]
async fn test_shard_fast_path() {
+1 -1
View File
@@ -111,7 +111,7 @@ pub trait StorageBackend: Send + Sync {
///
/// Backends that implement this must apply per-bucket authorization. The default denies the
/// request so existing backends cannot expose unfiltered bucket names.
#[cfg(feature = "webdav")]
#[cfg(any(feature = "webdav", feature = "sftp"))]
async fn list_buckets_for_session(
&self,
_session_context: &SessionContext,
+35 -6
View File
@@ -32,7 +32,7 @@
#[cfg(feature = "webdav")]
use crate::common::client::s3::SessionCapacityView;
use crate::common::client::s3::StorageBackend;
#[cfg(feature = "webdav")]
#[cfg(any(feature = "webdav", feature = "sftp"))]
use crate::common::session::SessionContext;
use async_trait::async_trait;
use bytes::Bytes;
@@ -254,6 +254,9 @@ impl Inner {
pub struct DummyBackend {
inner: Arc<Mutex<Inner>>,
deny_authorization: bool,
stall_authorization: bool,
#[cfg(any(feature = "webdav", feature = "sftp"))]
stall_session_listing: bool,
}
// Drivers whose trait bounds require `Debug` (for example FtpsDriver) cannot be
@@ -278,6 +281,9 @@ impl DummyBackend {
Self {
inner: Arc::new(Mutex::new(Inner::new())),
deny_authorization: false,
stall_authorization: false,
#[cfg(any(feature = "webdav", feature = "sftp"))]
stall_session_listing: false,
}
}
@@ -294,6 +300,19 @@ impl DummyBackend {
self
}
/// Make every authorize_operation await std::future::pending, so a
/// caller's authorization deadline elapses instead of an allow or deny.
pub fn stall_authorization(mut self) -> Self {
self.stall_authorization = true;
self
}
#[cfg(any(feature = "webdav", feature = "sftp"))]
pub fn stall_session_listing(mut self) -> Self {
self.stall_session_listing = true;
self
}
/// Opt in to consuming upload bodies for byte-for-byte protocol tests.
pub fn capture_upload_bodies(&self) {
self.inner.lock().expect("lock").capture_upload_bodies = true;
@@ -695,6 +714,9 @@ impl StorageBackend for DummyBackend {
if self.deny_authorization {
return Err(crate::common::gateway::AuthorizationError::AccessDenied);
}
if self.stall_authorization {
std::future::pending::<()>().await;
}
crate::common::gateway::authorize_operation(session, action, bucket, object).await
}
@@ -848,7 +870,7 @@ impl StorageBackend for DummyBackend {
}
}
#[cfg(feature = "webdav")]
#[cfg(any(feature = "webdav", feature = "sftp"))]
async fn list_buckets_for_session(
&self,
session_context: &SessionContext,
@@ -856,12 +878,19 @@ impl StorageBackend for DummyBackend {
secure_transport: bool,
) -> s3s::S3Result<ListBucketsOutput> {
let _ = session_context;
if self.stall_session_listing {
std::future::pending::<()>().await;
}
let mut inner = self.inner.lock().expect("lock");
inner.last_session_list_context = Some((request_headers.clone(), secure_transport));
inner
.session_list_buckets
.pop_front()
.unwrap_or_else(|| Ok(ListBucketsOutput::default()))
// Fail closed when nothing is queued, like the trait default for
// backends without session-aware listing.
inner.session_list_buckets.pop_front().unwrap_or_else(|| {
Err(s3s::S3Error::with_message(
s3s::S3ErrorCode::AccessDenied,
"Session-aware bucket listing is not supported",
))
})
}
/// Capacity reporting is opt-in; an empty queue models a backend without
+182 -7
View File
@@ -225,16 +225,32 @@ impl<S: StorageBackend + Send + Sync + 'static> SftpDriver<S> {
/// OPENDIR so a client without ListAllMyBuckets can still open the
/// root directory handle.
///
/// A principal without ListAllMyBuckets (a bucket-scoped user) gets
/// the session-aware ListBuckets instead, which IAM filters to the
/// buckets the principal may ListBucket or GetBucketLocation, the
/// same as S3 ListBuckets and the WebDAV root. It carries the
/// session's request headers and transport security, like every
/// other SFTP authorization.
///
/// ListBuckets is not batched in the S3 API. A single response
/// carries the full set. Truncate at ROOT_LISTING_MAX_ENTRIES so a
/// principal with many visible buckets produces a bounded Vec and
/// does not exceed the SSH channel window with a single response.
pub(super) async fn fetch_bucket_list(&self) -> Result<Vec<File>, SftpError> {
self.authorize(&S3Action::ListBuckets, "", None).await?;
let out = self
.run_backend("list_buckets", self.storage.list_buckets(self.credentials()))
.await?;
let out = if self.is_authorized(&S3Action::ListBuckets, "", None).await? {
self.run_backend("list_buckets", self.storage.list_buckets(self.credentials()))
.await?
} else {
self.run_backend(
"list_buckets",
self.storage.list_buckets_for_session(
&self.session_context,
&self.session_context.request_headers,
self.session_context.secure_transport,
),
)
.await?
};
let mut entries = Vec::new();
let mut truncated_at: Option<usize> = None;
@@ -444,15 +460,174 @@ impl<S: StorageBackend + Send + Sync + 'static> SftpDriver<S> {
#[cfg(test)]
mod tests {
use super::super::state::{DirCursor, HandleState, ListingContinuation};
use super::super::test_support::{TEST_PART_SIZE, build_driver, capture_tracing_at};
use super::super::test_support::{TEST_PART_SIZE, build_driver, build_driver_with_timeout, capture_tracing_at};
use crate::common::dummy_storage::{DummyBackend, DummyError};
use crate::common::gateway::with_test_auth_override;
use crate::common::gateway::{with_test_auth_override, with_test_iam_unavailable};
use russh_sftp::protocol::StatusCode;
use russh_sftp::server::Handler;
use std::sync::Arc;
use tokio::sync::Notify;
use tracing::Level;
fn buckets(names: &[&str]) -> s3s::dto::ListBucketsOutput {
s3s::dto::ListBucketsOutput {
buckets: Some(
names
.iter()
.map(|n| s3s::dto::Bucket {
name: Some((*n).to_string()),
..Default::default()
})
.collect(),
),
..Default::default()
}
}
fn names(files: &[russh_sftp::protocol::File]) -> Vec<&str> {
files.iter().map(|f| f.filename.as_str()).collect()
}
fn deny_list_all_my_buckets(action: &crate::common::gateway::S3Action, _: &str, _: Option<&str>) -> bool {
!matches!(action, crate::common::gateway::S3Action::ListBuckets)
}
#[tokio::test]
async fn root_listing_with_list_all_my_buckets_lists_every_bucket() {
let backend = Arc::new(DummyBackend::new());
backend.queue_list_buckets_ok(buckets(&["alpha", "beta"]));
backend.queue_session_list_buckets_ok(buckets(&["beta"]));
let driver = build_driver(backend.clone(), TEST_PART_SIZE);
let files = with_test_auth_override(|_, _, _| true, driver.fetch_bucket_list())
.await
.expect("global listing");
assert_eq!(names(&files), ["alpha", "beta"]);
assert!(backend.last_session_list_context().is_none(), "session listing must not run");
}
#[tokio::test]
async fn root_listing_without_list_all_my_buckets_lists_only_visible_buckets() {
let backend = Arc::new(DummyBackend::new());
backend.queue_list_buckets_ok(buckets(&["alpha", "beta"]));
backend.queue_session_list_buckets_ok(buckets(&["beta"]));
let mut driver = build_driver(backend.clone(), TEST_PART_SIZE);
driver
.session_context
.request_headers
.insert("user-agent", http::HeaderValue::from_static("sftp-listing-test"));
driver.session_context.secure_transport = false;
let files = with_test_auth_override(deny_list_all_my_buckets, driver.fetch_bucket_list())
.await
.expect("bucket-scoped listing");
assert_eq!(names(&files), ["beta"]);
let (headers, secure) = backend.last_session_list_context().expect("session listing ran");
assert_eq!(headers, driver.session_context.request_headers, "same headers as every other SFTP check");
assert_eq!(
secure, driver.session_context.secure_transport,
"same transport as every other SFTP check"
);
}
#[tokio::test]
async fn root_listing_without_any_visible_bucket_is_permission_denied() {
let backend = Arc::new(DummyBackend::new());
backend.queue_session_list_buckets_err(s3s::S3Error::new(s3s::S3ErrorCode::AccessDenied));
let driver = build_driver(backend.clone(), TEST_PART_SIZE);
let err = with_test_auth_override(deny_list_all_my_buckets, driver.fetch_bucket_list())
.await
.expect_err("no visible bucket");
assert_eq!(err.0, StatusCode::PermissionDenied);
assert!(backend.last_session_list_context().is_some(), "the session listing ran");
}
#[tokio::test]
async fn root_listing_fails_without_fallback_when_iam_is_unavailable() {
let backend = Arc::new(DummyBackend::new());
backend.queue_session_list_buckets_ok(buckets(&["beta"]));
let driver = build_driver(backend.clone(), TEST_PART_SIZE);
let err = with_test_iam_unavailable(driver.fetch_bucket_list())
.await
.expect_err("IAM outage");
assert_eq!(err.0, StatusCode::Failure);
assert!(backend.last_session_list_context().is_none(), "an IAM outage is not a policy deny");
}
#[tokio::test(start_paused = true)]
async fn root_listing_fails_without_fallback_when_the_global_check_times_out() {
let backend = Arc::new(DummyBackend::new().stall_authorization());
backend.queue_session_list_buckets_ok(buckets(&["beta"]));
let driver = build_driver_with_timeout(backend.clone(), TEST_PART_SIZE, 1);
let err = driver.fetch_bucket_list().await.expect_err("authorization deadline");
assert_eq!(err.0, StatusCode::Failure);
assert!(backend.last_session_list_context().is_none(), "a timeout is not a policy deny");
}
#[tokio::test(start_paused = true)]
async fn root_listing_fallback_respects_the_backend_deadline() {
let backend = Arc::new(DummyBackend::new().stall_session_listing());
let driver = build_driver_with_timeout(backend, TEST_PART_SIZE, 1);
let err = with_test_auth_override(deny_list_all_my_buckets, driver.fetch_bucket_list())
.await
.expect_err("session listing deadline");
assert_eq!(err.0, StatusCode::Failure);
}
#[tokio::test]
async fn root_listing_session_error_other_than_access_denied_is_failure() {
let backend = Arc::new(DummyBackend::new());
backend.queue_session_list_buckets_err(s3s::S3Error::with_message(
s3s::S3ErrorCode::InternalError,
"AccessDenied in the message is not an access denial",
));
let driver = build_driver(backend.clone(), TEST_PART_SIZE);
let err = with_test_auth_override(deny_list_all_my_buckets, driver.fetch_bucket_list())
.await
.expect_err("backend failure");
assert_eq!(err.0, StatusCode::Failure);
}
#[tokio::test]
async fn root_listing_fallback_is_truncated_at_the_root_listing_limit() {
for count in [super::ROOT_LISTING_MAX_ENTRIES, super::ROOT_LISTING_MAX_ENTRIES + 1] {
let backend = Arc::new(DummyBackend::new());
let many: Vec<String> = (0..count).map(|i| format!("bucket-{i:05}")).collect();
backend.queue_session_list_buckets_ok(buckets(&many.iter().map(String::as_str).collect::<Vec<_>>()));
let driver = build_driver(backend, TEST_PART_SIZE);
let files = with_test_auth_override(deny_list_all_my_buckets, driver.fetch_bucket_list())
.await
.expect("bucket-scoped listing");
assert_eq!(files.len(), super::ROOT_LISTING_MAX_ENTRIES, "bucket count {count}");
}
}
#[tokio::test]
async fn root_listing_fails_closed_when_the_backend_has_no_session_listing() {
let backend = Arc::new(DummyBackend::new());
let driver = build_driver(backend.clone(), TEST_PART_SIZE);
let err = with_test_auth_override(deny_list_all_my_buckets, driver.fetch_bucket_list())
.await
.expect_err("no session-aware listing");
assert_eq!(err.0, StatusCode::PermissionDenied);
}
#[tokio::test]
async fn opendir_and_readdir_authorize_the_same_listing_parameters() {
let backend = Arc::new(DummyBackend::new());
+22
View File
@@ -301,6 +301,28 @@ impl<S: StorageBackend + Send + Sync + 'static> SftpDriver<S> {
self.await_authorization(action, bucket, key, auth_fut).await
}
/// Like authorize, but a policy deny returns Ok(false) instead of
/// PermissionDenied, without the SFTP driver's access_denied warn,
/// for callers that fall back to a narrower operation when the
/// action is denied. The storage layer may still log the denial.
/// IamUnavailable and the deadline are handled exactly as in
/// authorize.
pub(super) async fn is_authorized(&self, action: &S3Action, bucket: &str, key: Option<&str>) -> Result<bool, SftpError> {
let auth_fut = self.storage.authorize_operation(&self.session_context, action, bucket, key);
let mut denied = false;
let deny_as_ok = async {
match auth_fut.await {
Err(AuthorizationError::AccessDenied) => {
denied = true;
Ok(())
}
other => other,
}
};
self.await_authorization(action, bucket, key, deny_as_ok).await?;
Ok(!denied)
}
pub(super) async fn authorize_list_objects(&self, input: &ListObjectsV2Input) -> Result<(), SftpError> {
let auth_fut = self.storage.authorize_list_objects(&self.session_context, input);
self.await_authorization(&S3Action::ListBucket, &input.bucket, input.prefix.as_deref(), auth_fut)
+4
View File
@@ -107,6 +107,10 @@ bytes.workspace = true
hex-simd.workspace = true
[dev-dependencies]
opentelemetry.workspace = true
opentelemetry-appender-tracing = { workspace = true, features = ["experimental_span_attributes", "experimental_metadata_attributes"] }
opentelemetry_sdk.workspace = true
rustfs-common = { workspace = true, features = ["test-util"] }
rustfs-heal.workspace = true
rustfs-test-utils.workspace = true
tracing-subscriber = { workspace = true, features = ["json", "env-filter", "time"] }
+47 -34
View File
@@ -1410,27 +1410,46 @@ impl DataUsageCache {
Some((flattened, visited.len()))
}
fn flatten_with_guard(&self, root: &DataUsageEntry, visited: &mut HashSet<String>, depth: usize) -> DataUsageEntry {
let mut root = root.clone();
fn flatten_with_guard<'a>(
&'a self,
root: &'a DataUsageEntry,
visited: &mut HashSet<&'a str>,
depth: usize,
) -> DataUsageEntry {
// A summary never owns child links. Cloning and clearing the set
// retains its allocation in every completed subtree and update.
let mut flat = DataUsageEntry {
children: HashSet::new(),
size: root.size,
objects: root.objects,
versions: root.versions,
delete_markers: root.delete_markers,
obj_sizes: root.obj_sizes.clone(),
obj_versions: root.obj_versions.clone(),
replication_stats: root.replication_stats.clone(),
compacted: root.compacted,
failed_objects: root.failed_objects,
all_tier_stats: root.all_tier_stats.clone(),
unknown_tier_stats: root.unknown_tier_stats.clone(),
tier_accounting_proof: root.tier_accounting_proof,
};
if depth >= MAX_DATA_USAGE_CACHE_DEPTH {
root.children.clear();
return root;
return flat;
}
for id in root.children.clone().iter() {
if !visited.insert(id.clone()) {
for id in &root.children {
if !visited.insert(id.as_str()) {
continue;
}
if let Some(e) = self.cache.get(id) {
let mut e = e.clone();
if !e.children.is_empty() {
e = self.flatten_with_guard(&e, visited, depth + 1);
if e.children.is_empty() {
flat.merge(e);
} else {
flat.merge(&self.flatten_with_guard(e, visited, depth + 1));
}
root.merge(&e);
}
}
root.children.clear();
root
flat
}
pub fn copy_with_children(&mut self, src: &DataUsageCache, hash: &DataUsageHash, parent: &Option<DataUsageHash>) {
@@ -1467,38 +1486,31 @@ impl DataUsageCache {
}
pub fn delete_recursive(&mut self, hash: &DataUsageHash) {
let mut visited = HashSet::new();
self.delete_recursive_guard(hash, &mut visited, 0);
self.delete_recursive_guard(&hash.0, 0);
}
fn delete_recursive_guard(&mut self, hash: &DataUsageHash, visited: &mut HashSet<String>, depth: usize) {
if !visited.insert(hash.key()) {
fn delete_recursive_guard(&mut self, path: &str, depth: usize) {
// Removal transfers child ownership and makes cyclic/shared references
// no-ops, without cloning paths or maintaining a second visited set.
let Some(children) = self.cache.remove(path).map(|entry| entry.children) else {
return;
}
let mut need_remove = Vec::new();
if let Some(v) = self.cache.get(&hash.string()) {
for child in v.children.iter() {
need_remove.push(child.clone());
}
}
self.cache.remove(&hash.string());
};
if depth >= MAX_DATA_USAGE_CACHE_DEPTH {
return;
}
for child in need_remove {
self.delete_recursive_guard(&DataUsageHash(child), visited, depth + 1);
for child in children {
self.delete_recursive_guard(&child, depth + 1);
}
}
pub fn size_recursive(&self, path: &str) -> Option<DataUsageEntry> {
match self.find(path) {
Some(root) => {
match self.cache.get_key_value(&hash_path(path).key()) {
Some((key, root)) => {
if root.children.is_empty() {
return Some(root.clone());
}
let mut visited = HashSet::new();
visited.insert(hash_path(path).key());
visited.insert(key.as_str());
let mut flat = self.flatten_with_guard(root, &mut visited, 0);
if flat.replication_stats.as_ref().is_some_and(|stats| stats.is_empty()) {
flat.replication_stats = None;
@@ -1553,7 +1565,7 @@ impl DataUsageCache {
}
pub fn reduce_children_of(&mut self, path: &DataUsageHash, limit: usize, compact_self: bool) {
let Some(e) = self.cache.get(&path.key()).cloned() else {
let Some(e) = self.cache.get(&path.0) else {
return;
};
@@ -1605,12 +1617,13 @@ impl DataUsageCache {
}
pub fn total_children_rec(&self, path: &str) -> usize {
let root = hash_path(path);
let mut visited = HashSet::new();
visited.insert(hash_path(path).key());
visited.insert(root.0.as_str());
self.total_children_rec_guard(path, &mut visited, 0)
}
fn total_children_rec_guard(&self, path: &str, visited: &mut HashSet<String>, depth: usize) -> usize {
fn total_children_rec_guard<'a>(&'a self, path: &str, visited: &mut HashSet<&'a str>, depth: usize) -> usize {
let Some(root) = self.find(path) else {
return 0;
};
@@ -1620,7 +1633,7 @@ impl DataUsageCache {
let mut n = 0;
for ch in root.children.iter() {
if visited.insert(ch.clone()) {
if visited.insert(ch.as_str()) {
n += 1 + self.total_children_rec_guard(ch, visited, depth + 1);
}
}
@@ -2428,6 +2428,69 @@ fn test_data_usage_cache_recursive_helpers_tolerate_cycles() {
assert!(!copied.cache.contains_key(&child_hash.key()));
}
#[test]
fn delete_recursive_preserves_depth_boundary_and_unrelated_entries() {
let mut cache = DataUsageCache::default();
for depth in 0..=MAX_DATA_USAGE_CACHE_DEPTH + 1 {
let mut entry = DataUsageEntry::default();
if depth <= MAX_DATA_USAGE_CACHE_DEPTH {
entry.children.insert(format!("node-{}", depth + 1));
}
cache.cache.insert(format!("node-{depth}"), entry);
}
cache.cache.insert("unrelated".to_string(), DataUsageEntry::default());
cache.delete_recursive(&DataUsageHash("node-0".to_string()));
assert_eq!(cache.cache.len(), 2, "delete only reachable entries within the depth limit");
assert!(cache.cache.contains_key(&format!("node-{}", MAX_DATA_USAGE_CACHE_DEPTH + 1)));
assert!(cache.cache.contains_key("unrelated"));
}
#[test]
fn scanner_summary_releases_wide_child_sets() {
let root = hash_path("bucket");
let mut cache = DataUsageCache::default();
cache.replace_hashed(
&root,
&None,
&DataUsageEntry {
size: 17,
objects: 1,
versions: 2,
delete_markers: 1,
failed_objects: 1,
all_tier_stats: Some(AllTierStats::new()),
..Default::default()
},
);
for index in 0..10_000 {
let child = hash_path(&format!("bucket/{index:05}"));
cache.replace_hashed(
&child,
&Some(root.clone()),
&DataUsageEntry {
size: 3,
objects: 1,
versions: 2,
..Default::default()
},
);
}
let summary = cache.size_recursive("bucket").expect("wide bucket summary");
assert_eq!((summary.size, summary.objects, summary.versions), (30_017, 10_001, 20_002));
assert_eq!((summary.delete_markers, summary.failed_objects), (1, 1));
assert_eq!(summary.children.capacity(), 0, "summaries must not retain the wide namespace allocation");
assert!(summary.all_tier_stats.is_some(), "preserve the root's optional statistics representation");
assert_eq!(cache.find("bucket").expect("unchanged root").children.len(), 10_000);
cache.cache.insert("unrelated".to_string(), DataUsageEntry::default());
cache.delete_recursive(&root);
assert_eq!(cache.cache.len(), 1, "wide subtree deletion must consume every child exactly once");
assert!(cache.cache.contains_key("unrelated"));
}
#[test]
fn test_data_usage_cache_flatten_does_not_count_root_twice_in_cycle() {
let root_hash = hash_path("bucket");
@@ -236,6 +236,7 @@ async fn profile_case(objects: usize, scenario: &str, samples: usize) {
let mut clone_ns = Vec::new();
let mut copy_ns = Vec::new();
let mut delete_ns = Vec::new();
let mut flatten_ns = Vec::new();
let mut encode_ns = Vec::new();
let mut save_ns = Vec::new();
@@ -257,6 +258,14 @@ async fn profile_case(objects: usize, scenario: &str, samples: usize) {
same_cache(&copied, &cache);
drop(copied);
let mut deleted = cache.clone();
let root = hash_path("bucket");
let started = WallInstant::now();
deleted.delete_recursive(black_box(&root));
delete_ns.push(elapsed_ns(started));
assert!(deleted.cache.is_empty(), "delete the root and every reachable entry");
drop(deleted);
let started = WallInstant::now();
let aggregate = black_box(cache.checked_flatten("bucket").expect("valid fixture tree"));
flatten_ns.push(elapsed_ns(started));
@@ -326,6 +335,7 @@ async fn profile_case(objects: usize, scenario: &str, samples: usize) {
"changed_entry_wire_bytes": changed_entry_wire_bytes, "save_body_bytes_per_sample": expected_wire.len() * 2,
"snapshot_complete": cache.info.snapshot_complete,
"clone": quantiles(clone_ns), "copy_with_children": quantiles(copy_ns), "checked_flatten": quantiles(flatten_ns),
"delete_recursive": quantiles(delete_ns),
"encode": quantiles(encode_ns), "save_inclusive": quantiles(save_ns), "memory_backend_ingest": quantiles(ingest_ns),
})
);
+66 -19
View File
@@ -2082,7 +2082,7 @@ where
None
} else if let Some(notification_system) = storeapi.scanner_notification_system() {
let lease_result = publication_plan
.await_leases(ctx, &cycle_budget, cycle_info.current, |observational_lease| {
.await_leases(ctx, &cycle_budget, cycle_info.current, leader_epoch, |observational_lease| {
let targets = remote_publication_lease_targets.clone();
let notification_system = Arc::clone(&notification_system);
async move {
@@ -3892,17 +3892,17 @@ impl ScannerCyclePublicationPlan {
self != Self::Authoritative
}
async fn await_leases<T, E, F, Fut>(
async fn await_leases<T, F, Fut>(
self,
parent: &CancellationToken,
budget: &ScannerCycleBudget,
cycle: u64,
leader_epoch: u64,
mut acquire: F,
) -> ScannerPublicationProofWait<T, E>
) -> ScannerPublicationProofWait<T, EcstoreError>
where
E: std::fmt::Display,
F: FnMut(bool) -> Fut,
Fut: Future<Output = Result<T, E>>,
Fut: Future<Output = Result<T, EcstoreError>>,
{
// Exhausting the walk budget must not cancel the movement proof needed
// to save partial progress. No usage update means namespace stability
@@ -3912,6 +3912,17 @@ impl ScannerCyclePublicationPlan {
} else {
budget.token()
};
let started_at = Instant::now();
let publication_kind = match self {
Self::ProgressOnly => "progress_only",
Self::Observational => "observational",
Self::Authoritative => "authoritative",
};
let lease_purpose = if self.uses_observational_lease() {
"observational"
} else {
"authoritative"
};
debug!(
target: "rustfs::scanner",
event = EVENT_SCANNER_CYCLE_STATE,
@@ -3919,30 +3930,66 @@ impl ScannerCyclePublicationPlan {
subsystem = LOG_SUBSYSTEM_RUNTIME,
state = "publication_proof_started",
cycle,
leader_epoch,
stage = "lease_acquire",
publication_kind = match self {
Self::ProgressOnly => "progress_only",
Self::Observational => "observational",
Self::Authoritative => "authoritative",
},
publication_kind,
lease_purpose,
"Scanner is acquiring peer publication proof"
);
let proof = await_scanner_publication_proof(
&proof_ctx,
cycle,
"lease_acquire",
ScannerPublicationProofContext {
cycle,
stage: "lease_acquire",
leader_epoch: Some(leader_epoch),
lease_purpose: Some(lease_purpose),
},
|| acquire(self.uses_observational_lease()),
|err| scanner_publication_lease_error_is_retryable(&err.to_string()),
|err| {
// The structured reason comes from local acquisition, not a peer report.
crate::storage_api::scan::scanner_publication_lease_failure_info(err)
.is_some_and(|info| info.reason == Some("channel_acquisition_timed_out"))
|| scanner_publication_lease_error_is_retryable(&err.to_string())
},
crate::storage_api::scan::scanner_publication_lease_failure_info,
);
if self == Self::ProgressOnly {
let (proof, timed_out) = if self == Self::ProgressOnly {
// Object/directory budgets can expire with no runtime deadline.
// Bound the control-plane drain independently of the walk budget.
tokio::time::timeout(SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT, proof)
.await
.unwrap_or(ScannerPublicationProofWait::Cancelled)
match tokio::time::timeout(SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT, proof).await {
Ok(proof) => (proof, false),
Err(_) => (ScannerPublicationProofWait::Cancelled, true),
}
} else {
proof.await
}
(proof.await, false)
};
let outcome = if timed_out {
"timed_out"
} else {
match &proof {
ScannerPublicationProofWait::Ready(_) => "acquired",
ScannerPublicationProofWait::Rejected(_) => "rejected",
ScannerPublicationProofWait::Cancelled => "cancelled",
}
};
debug!(
target: "rustfs::scanner",
event = EVENT_SCANNER_CYCLE_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_RUNTIME,
state = "publication_lease_wait_finished",
cycle,
leader_epoch,
stage = "lease_acquire",
publication_kind,
lease_purpose,
outcome,
parent_cancelled_at_return = parent.is_cancelled(),
proof_token_cancelled_at_return = proof_ctx.is_cancelled(),
elapsed = ?started_at.elapsed(),
"Scanner peer publication lease wait finished"
);
proof
}
}
+59 -10
View File
@@ -14,7 +14,7 @@
/// Cycle wake/backoff policy and scanner activity observation (probing, generations, topology digest).
use super::*;
use crate::storage_api::ScannerStorage;
use crate::storage_api::scan::SCANNER_ACTIVITY_V6_PROTOCOL_VERSION;
use crate::storage_api::scan::{SCANNER_ACTIVITY_V6_PROTOCOL_VERSION, ScannerPublicationLeaseFailureInfo};
use std::collections::HashSet;
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
@@ -239,7 +239,7 @@ pub(crate) fn scanner_publication_proof_retry_delay(consecutive_failures: u32) -
}
pub(crate) fn scanner_publication_activity_error_is_retryable(error: &str) -> bool {
crate::storage_api::scanner_peer_transport_error_message_is_retryable(error)
crate::storage_api::scan::scanner_peer_transport_error_message_is_retryable(error)
}
pub(crate) fn scanner_publication_lease_error_is_retryable(error: &str) -> bool {
@@ -256,12 +256,19 @@ pub(crate) enum ScannerPublicationProofWait<T, E> {
Cancelled,
}
pub(crate) struct ScannerPublicationProofContext {
pub cycle: u64,
pub stage: &'static str,
pub leader_epoch: Option<u64>,
pub lease_purpose: Option<&'static str>,
}
pub(crate) async fn await_scanner_publication_proof<T, E, F, Fut, Retryable>(
ctx: &CancellationToken,
cycle: u64,
stage: &'static str,
context: ScannerPublicationProofContext,
mut proof: F,
retryable: Retryable,
failure_info: for<'a> fn(&'a E) -> Option<ScannerPublicationLeaseFailureInfo<'a>>,
) -> ScannerPublicationProofWait<T, E>
where
F: FnMut() -> Fut,
@@ -269,6 +276,12 @@ where
E: std::fmt::Display,
Retryable: Fn(&E) -> bool,
{
let ScannerPublicationProofContext {
cycle,
stage,
leader_epoch,
lease_purpose,
} = context;
let started_at = Instant::now();
let mut consecutive_failures = 0u32;
@@ -288,6 +301,8 @@ where
state = "publication_proof_recovered",
cycle,
stage,
leader_epoch,
lease_purpose,
attempts = consecutive_failures.saturating_add(1),
pending_duration = ?started_at.elapsed(),
"Scanner publication proof recovered"
@@ -296,6 +311,11 @@ where
return ScannerPublicationProofWait::Ready(value);
}
Err(err) if !retryable(&err) => {
let failure = failure_info(&err);
let peer = failure.and_then(|info| info.peer);
let failure_kind = failure.map(|info| info.kind);
let reason = failure.and_then(|info| info.reason);
let reported_reason = failure.and_then(|info| info.reported_reason);
warn!(
target: "rustfs::scanner",
event = EVENT_SCANNER_CYCLE_STATE,
@@ -304,7 +324,12 @@ where
state = "publication_proof_rejected",
cycle,
stage,
error = %err,
leader_epoch,
lease_purpose,
peer,
failure_kind,
reason,
reported_reason,
"Scanner publication proof failed with a non-retryable cluster state"
);
return ScannerPublicationProofWait::Rejected(err);
@@ -312,6 +337,11 @@ where
Err(err) => {
consecutive_failures = consecutive_failures.saturating_add(1);
let retry_delay = scanner_publication_proof_retry_delay(consecutive_failures);
let failure = failure_info(&err);
let peer = failure.and_then(|info| info.peer);
let failure_kind = failure.map(|info| info.kind);
let reason = failure.and_then(|info| info.reason);
let reported_reason = failure.and_then(|info| info.reported_reason);
if consecutive_failures == 1 || consecutive_failures.is_multiple_of(20) {
warn!(
target: "rustfs::scanner",
@@ -321,10 +351,15 @@ where
state = "publication_proof_pending",
cycle,
stage,
leader_epoch,
lease_purpose,
peer,
failure_kind,
reason,
reported_reason,
attempt = consecutive_failures,
retry_delay = ?retry_delay,
pending_duration = ?started_at.elapsed(),
error = %err,
"Scanner is waiting for publication proof"
);
} else {
@@ -336,9 +371,14 @@ where
state = "publication_proof_retry",
cycle,
stage,
leader_epoch,
lease_purpose,
peer,
failure_kind,
reason,
reported_reason,
attempt = consecutive_failures,
retry_delay = ?retry_delay,
error = %err,
"Scanner publication activity proof retry scheduled"
);
}
@@ -363,9 +403,18 @@ where
F: FnMut() -> Fut,
Fut: Future<Output = Result<ScannerActivitySnapshot, String>>,
{
match await_scanner_publication_proof(ctx, cycle, stage, probe, |err: &String| {
scanner_publication_activity_error_is_retryable(err)
})
match await_scanner_publication_proof(
ctx,
ScannerPublicationProofContext {
cycle,
stage,
leader_epoch: None,
lease_purpose: None,
},
probe,
|err: &String| scanner_publication_activity_error_is_retryable(err),
|_| None,
)
.await
{
ScannerPublicationProofWait::Ready(snapshot) => Ok(snapshot),
+7 -2
View File
@@ -9204,8 +9204,12 @@ async fn publication_lease_retries_pending_namespace_commit_until_it_clears() {
let result = await_scanner_publication_proof(
&ctx,
17,
"lease_acquire",
ScannerPublicationProofContext {
cycle: 17,
stage: "lease_acquire",
leader_epoch: None,
lease_purpose: None,
},
move || {
let attempt = proof_attempts.fetch_add(1, Ordering::SeqCst);
async move {
@@ -9217,6 +9221,7 @@ async fn publication_lease_retries_pending_namespace_commit_until_it_clears() {
}
},
|error| scanner_publication_lease_error_is_retryable(error),
|_| None,
)
.await;
@@ -13,8 +13,629 @@
// limitations under the License.
use super::*;
use crate::storage_api::scan::ScannerPublicationLeaseFailureInfo;
use rustfs_utils::path::path_join_buf;
#[derive(Clone)]
struct LeaseLogWriter(Arc<std::sync::Mutex<Vec<u8>>>);
impl std::io::Write for LeaseLogWriter {
fn write(&mut self, bytes: &[u8]) -> std::io::Result<usize> {
self.0.lock().expect("lease log mutex").extend_from_slice(bytes);
Ok(bytes.len())
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
#[tokio::test(start_paused = true)]
async fn publication_lease_observations_keep_cycle_epoch_purpose_and_outcome() {
use tracing::instrument::WithSubscriber as _;
let output = Arc::new(std::sync::Mutex::new(Vec::new()));
let writer = LeaseLogWriter(Arc::clone(&output));
let subscriber = tracing_subscriber::fmt()
.json()
.with_max_level(tracing::Level::DEBUG)
.with_writer(move || writer.clone())
.with_ansi(false)
.without_time()
.finish();
let _interest_guard = tracing::Dispatch::new(tracing::subscriber::NoSubscriber::default());
async {
let ctx = CancellationToken::new();
let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default());
let ready = ScannerCyclePublicationPlan::ProgressOnly
.await_leases(&ctx, &budget, 501, 17, |_| async { Ok::<_, EcstoreError>(()) })
.await;
assert!(matches!(ready, ScannerPublicationProofWait::Ready(())));
let next_leader = ScannerCyclePublicationPlan::Observational
.await_leases(&ctx, &budget, 501, 22, |_| async { Ok::<_, EcstoreError>(()) })
.await;
assert!(matches!(next_leader, ScannerPublicationProofWait::Ready(())));
let rejected = ScannerCyclePublicationPlan::Observational
.await_leases(&ctx, &budget, 502, 18, |_| async {
Err::<(), _>(EcstoreError::other("protocol mismatch"))
})
.await;
assert!(matches!(rejected, ScannerPublicationProofWait::Rejected(_)));
ctx.cancel();
let cancelled = ScannerCyclePublicationPlan::Authoritative
.await_leases(&ctx, &budget, 503, 19, |_| async { Ok::<_, EcstoreError>(()) })
.await;
assert!(matches!(cancelled, ScannerPublicationProofWait::Cancelled));
let timeout_ctx = CancellationToken::new();
let timeout_budget = ScannerCycleBudget::new(&timeout_ctx, ScannerCycleBudgetConfig::default());
let timed_out = ScannerCyclePublicationPlan::ProgressOnly
.await_leases(&timeout_ctx, &timeout_budget, 504, 20, |_| async {
std::future::pending::<Result<(), EcstoreError>>().await
})
.await;
assert!(matches!(timed_out, ScannerPublicationProofWait::Cancelled));
let budget_ctx = CancellationToken::new();
let stopped_budget = ScannerCycleBudget::new(&budget_ctx, ScannerCycleBudgetConfig::default());
stopped_budget.cancel_for_runtime();
let budget_cancelled = ScannerCyclePublicationPlan::Observational
.await_leases(&budget_ctx, &stopped_budget, 505, 21, |_| async { Ok::<_, EcstoreError>(()) })
.await;
assert!(matches!(budget_cancelled, ScannerPublicationProofWait::Cancelled));
}
.with_subscriber(subscriber)
.await;
let bytes = output.lock().expect("lease log mutex").clone();
let logs = String::from_utf8(bytes).expect("lease logs are UTF-8");
let entries: Vec<serde_json::Value> = logs
.lines()
.map(|line| serde_json::from_str(line).expect("lease logs are JSON"))
.collect();
for (cycle, epoch, kind, purpose, outcome, parent_cancelled, proof_cancelled) in [
(501, 17, "progress_only", "observational", "acquired", false, false),
(502, 18, "observational", "observational", "rejected", false, false),
(503, 19, "authoritative", "authoritative", "cancelled", true, true),
(504, 20, "progress_only", "observational", "timed_out", false, false),
(505, 21, "observational", "observational", "cancelled", false, true),
(501, 22, "observational", "observational", "acquired", false, false),
] {
let started: Vec<_> = entries
.iter()
.filter_map(|entry| entry.get("fields"))
.filter(|fields| {
fields["state"] == "publication_proof_started" && fields["cycle"] == cycle && fields["leader_epoch"] == epoch
})
.collect();
assert_eq!(started.len(), 1, "missing or duplicated started event for cycle {cycle}: {logs}");
assert_eq!(started[0]["leader_epoch"], epoch);
assert_eq!(started[0]["publication_kind"], kind);
assert_eq!(started[0]["lease_purpose"], purpose);
let finished: Vec<_> = entries
.iter()
.filter_map(|entry| entry.get("fields"))
.filter(|fields| {
fields["state"] == "publication_lease_wait_finished"
&& fields["cycle"] == cycle
&& fields["leader_epoch"] == epoch
})
.collect();
assert_eq!(finished.len(), 1, "missing or duplicated finished event for cycle {cycle}: {logs}");
let fields = finished[0];
assert_eq!(fields["leader_epoch"], epoch);
assert_eq!(fields["publication_kind"], kind);
assert_eq!(fields["lease_purpose"], purpose);
assert_eq!(fields["outcome"], outcome);
assert_eq!(fields["parent_cancelled_at_return"], parent_cancelled);
assert_eq!(fields["proof_token_cancelled_at_return"], proof_cancelled);
}
}
#[tokio::test(start_paused = true)]
async fn publication_lease_retry_logs_identify_interleaved_epochs() {
use tracing::instrument::WithSubscriber as _;
for level in [tracing::Level::WARN, tracing::Level::INFO, tracing::Level::DEBUG] {
let output = Arc::new(std::sync::Mutex::new(Vec::new()));
let writer = LeaseLogWriter(Arc::clone(&output));
let subscriber = tracing_subscriber::fmt()
.json()
.flatten_event(true)
.with_env_filter(format!("off,rustfs::scanner={level}"))
.with_current_span(true)
.with_writer(move || writer.clone())
.with_ansi(false)
.without_time()
.finish();
let _interest_guard = tracing::Dispatch::new(tracing::subscriber::NoSubscriber::default());
exercise_interleaved_lease_waits().with_subscriber(subscriber).await;
let bytes = output.lock().expect("lease log mutex").clone();
let logs = String::from_utf8(bytes).expect("lease logs are UTF-8");
assert!(!logs.contains("remote-secret"), "untrusted error text must not enter logs");
assert!(!logs.contains("forged:9000"), "unstructured error text must not establish a peer");
let entries: Vec<serde_json::Value> = logs
.lines()
.map(|line| serde_json::from_str(line).expect("lease logs are JSON"))
.collect();
let find_lease_event = |state: &str| {
entries
.iter()
.position(|entry| entry["state"] == state && entry["stage"] == "lease_acquire")
.unwrap_or_else(|| panic!("missing {state} at {level}: {logs}"))
};
let pending = find_lease_event("publication_proof_pending");
let rejected = find_lease_event("publication_proof_rejected");
assert!(pending < rejected, "waits must interleave: {logs}");
for (index, epoch, purpose) in [(pending, 41, "observational"), (rejected, 42, "authoritative")] {
assert_eq!(entries[index]["cycle"], 601);
assert_eq!(entries[index]["leader_epoch"], epoch, "{logs}");
assert_eq!(entries[index]["lease_purpose"], purpose, "{logs}");
}
assert_eq!(entries[rejected]["peer"], "peer-a:9000", "{logs}");
assert_eq!(entries[rejected]["failure_kind"], "peer_unavailable", "{logs}");
let ipv6 = entries
.iter()
.find(|entry| entry["state"] == "publication_proof_rejected" && entry["cycle"] == 604)
.expect("IPv6 target failure should be logged");
assert_eq!(ipv6["peer"], "[2001:db8::1]:9443", "{logs}");
let secret_target = entries
.iter()
.find(|entry| entry["state"] == "publication_proof_rejected" && entry["cycle"] == 605)
.expect("credential-bearing target failure should be logged");
assert!(secret_target["peer"].is_null(), "credential-bearing target must not be logged");
assert!(!logs.contains("target-secret"), "target credentials must not enter logs");
let local = entries
.iter()
.find(|entry| entry["state"] == "publication_proof_rejected" && entry["cycle"] == 606)
.expect("local diagnostic should be logged");
assert_eq!(local["reason"], "invalid_proof");
assert!(local["reported_reason"].is_null());
let reported = entries
.iter()
.find(|entry| entry["state"] == "publication_proof_rejected" && entry["cycle"] == 607)
.expect("remote report should be logged");
assert!(reported["reason"].is_null());
assert_eq!(reported["reported_reason"], "generation_stale");
let timeout = entries
.iter()
.find(|entry| entry["state"] == "publication_proof_rejected" && entry["cycle"] == 608)
.expect("channel deadline should be logged");
assert_eq!(timeout["reason"], "channel_acquisition_timed_out");
assert!(timeout["reported_reason"].is_null());
assert!(!logs.contains("diagnostic-secret"), "error text must not enter logs");
if level != tracing::Level::WARN {
let recovered = find_lease_event("publication_proof_recovered");
assert!(rejected < recovered, "waits must interleave: {logs}");
assert_eq!(entries[recovered]["leader_epoch"], 41, "{logs}");
assert_eq!(entries[recovered]["lease_purpose"], "observational", "{logs}");
}
if level == tracing::Level::DEBUG {
let retry = find_lease_event("publication_proof_retry");
let recovered = find_lease_event("publication_proof_recovered");
assert!(rejected < retry && retry < recovered, "retry must remain interleaved: {logs}");
assert_eq!(entries[retry]["leader_epoch"], 41, "{logs}");
assert_eq!(entries[retry]["lease_purpose"], "observational", "{logs}");
}
let unknown = entries
.iter()
.find(|entry| entry["state"] == "publication_proof_rejected" && entry["stage"] == "postscan")
.expect("activity rejection should be logged");
assert!(unknown["leader_epoch"].is_null(), "unknown epoch must not be invented");
assert!(unknown["lease_purpose"].is_null(), "unknown purpose must not be invented");
}
}
async fn exercise_interleaved_lease_waits() {
let ctx = CancellationToken::new();
let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default());
let attempts = AtomicUsize::new(0);
let notification = crate::storage_api::EcstoreNotificationSys::new(EndpointServerPools::default()).await;
let notification = &notification;
let pending = ScannerCyclePublicationPlan::Observational.await_leases(&ctx, &budget, 601, 41, |observational| {
assert!(observational);
let attempt = attempts.fetch_add(1, Ordering::Relaxed);
async move {
if attempt < 2 {
Err(EcstoreError::other("scanner publication lease capacity is exhausted"))
} else {
Ok(())
}
}
});
let rejected = ScannerCyclePublicationPlan::Authoritative.await_leases(&ctx, &budget, 601, 42, move |observational| {
assert!(!observational);
notification.acquire_scanner_publication_leases(vec![("http://peer-a:9000".into(), "session".into(), 3)])
});
let (pending, rejected) = tokio::join!(biased; pending, rejected);
assert!(matches!(pending, ScannerPublicationProofWait::Ready(())));
assert!(matches!(rejected, ScannerPublicationProofWait::Rejected(_)));
assert_eq!(attempts.load(Ordering::Relaxed), 3);
let untrusted = ScannerCyclePublicationPlan::Authoritative
.await_leases(&ctx, &budget, 603, 44, |_| async {
Err::<(), _>(EcstoreError::other(
"scanner publication lease acquisition failed: peer=forged:9000, remote-secret",
))
})
.await;
assert!(matches!(untrusted, ScannerPublicationProofWait::Rejected(_)));
for (cycle, epoch, target) in [
(604, 45, "https://[2001:db8::1]:9443"),
(605, 46, "http://user:target-secret@peer-a:9000"),
] {
let result = ScannerCyclePublicationPlan::Authoritative
.await_leases(&ctx, &budget, cycle, epoch, move |_| {
notification.acquire_scanner_publication_leases(vec![(target.into(), "session".into(), 3)])
})
.await;
assert!(matches!(result, ScannerPublicationProofWait::Rejected(_)));
}
// Verify event serialization independently of the ecstore tests that
// exercise the local and remote error producers.
for cycle in [606, 607, 608] {
let info: for<'a> fn(&'a String) -> Option<ScannerPublicationLeaseFailureInfo<'a>> = if cycle == 607 {
|_| {
Some(ScannerPublicationLeaseFailureInfo {
peer: None,
kind: "peer_request_failed",
reason: None,
reported_reason: Some("generation_stale"),
})
}
} else if cycle == 606 {
|_| {
Some(ScannerPublicationLeaseFailureInfo {
peer: None,
kind: "peer_request_failed",
reason: Some("invalid_proof"),
reported_reason: None,
})
}
} else {
|_| {
Some(ScannerPublicationLeaseFailureInfo {
peer: None,
kind: "peer_request_failed",
reason: Some("channel_acquisition_timed_out"),
reported_reason: None,
})
}
};
let result = await_scanner_publication_proof(
&ctx,
ScannerPublicationProofContext {
cycle,
stage: "lease_acquire",
leader_epoch: Some(47),
lease_purpose: Some("authoritative"),
},
|| async { Err::<(), _>("diagnostic-secret".to_owned()) },
|_| false,
info,
)
.await;
assert!(matches!(result, ScannerPublicationProofWait::Rejected(_)));
}
let unknown =
await_scanner_publication_activity(&ctx, 601, "postscan", || async { Err("protocol mismatch".to_string()) }).await;
assert!(unknown.is_err());
}
#[derive(Clone, Debug, Default)]
struct CapturedOtelLogs(Arc<std::sync::Mutex<Vec<opentelemetry_sdk::logs::SdkLogRecord>>>);
impl opentelemetry_sdk::logs::LogProcessor for CapturedOtelLogs {
fn emit(&self, data: &mut opentelemetry_sdk::logs::SdkLogRecord, _: &opentelemetry::InstrumentationScope) {
self.0.lock().expect("OTLP log mutex").push(data.clone());
}
fn force_flush(&self) -> opentelemetry_sdk::error::OTelSdkResult {
Ok(())
}
fn shutdown_with_timeout(&self, _: std::time::Duration) -> opentelemetry_sdk::error::OTelSdkResult {
Ok(())
}
}
#[tokio::test(start_paused = true)]
async fn publication_lease_events_include_context_in_otlp_without_span_export() {
use opentelemetry::logs::AnyValue;
use opentelemetry_appender_tracing::layer::OpenTelemetryTracingBridge;
use tracing::{Instrument as _, instrument::WithSubscriber as _};
use tracing_subscriber::prelude::*;
for level in [tracing::Level::WARN, tracing::Level::INFO, tracing::Level::DEBUG] {
let captured = CapturedOtelLogs::default();
let provider = opentelemetry_sdk::logs::SdkLoggerProvider::builder()
.with_log_processor(captured.clone())
.build();
let subscriber = tracing_subscriber::registry()
.with(tracing_subscriber::EnvFilter::new(format!("off,rustfs::scanner={level}")))
.with(OpenTelemetryTracingBridge::new(&provider));
let _interest_guard = tracing::Dispatch::new(tracing::subscriber::NoSubscriber::default());
async {
// The bridge is configured like production: no tracing exporter or
// span attribute copying. A parent span must not supply these fields.
let unrelated = tracing::warn_span!(
target: "rustfs::scanner",
"unrelated_parent",
leader_epoch = 999,
lease_purpose = "unrelated"
);
exercise_interleaved_lease_waits().instrument(unrelated).await;
}
.with_subscriber(subscriber)
.await;
let records = captured.0.lock().expect("OTLP log mutex").clone();
let attribute = |record: &opentelemetry_sdk::logs::SdkLogRecord, name: &str| {
record
.attributes_iter()
.find(|(key, _)| key.as_str() == name)
.map(|(_, value)| value.clone())
};
let find = |state: &'static str, cycle: i64, stage: &'static str| {
records
.iter()
.find(|record| {
attribute(record, "state") == Some(AnyValue::from(state))
&& attribute(record, "cycle") == Some(AnyValue::Int(cycle))
&& attribute(record, "stage") == Some(AnyValue::from(stage))
})
.unwrap_or_else(|| panic!("missing {state} at {level}"))
};
let pending = find("publication_proof_pending", 601, "lease_acquire");
let rejected = find("publication_proof_rejected", 601, "lease_acquire");
for (record, epoch, purpose) in [(pending, 41, "observational"), (rejected, 42, "authoritative")] {
assert_eq!(attribute(record, "leader_epoch"), Some(AnyValue::Int(epoch)));
assert_eq!(attribute(record, "lease_purpose"), Some(AnyValue::from(purpose)));
}
assert_eq!(attribute(rejected, "peer"), Some(AnyValue::from("peer-a:9000")));
assert_eq!(attribute(rejected, "failure_kind"), Some(AnyValue::from("peer_unavailable")));
let ipv6 = find("publication_proof_rejected", 604, "lease_acquire");
assert_eq!(attribute(ipv6, "peer"), Some(AnyValue::from("[2001:db8::1]:9443")));
let secret_target = find("publication_proof_rejected", 605, "lease_acquire");
assert!(attribute(secret_target, "peer").is_none());
let local = find("publication_proof_rejected", 606, "lease_acquire");
assert_eq!(attribute(local, "reason"), Some(AnyValue::from("invalid_proof")));
assert!(attribute(local, "reported_reason").is_none());
let reported = find("publication_proof_rejected", 607, "lease_acquire");
assert!(attribute(reported, "reason").is_none());
assert_eq!(attribute(reported, "reported_reason"), Some(AnyValue::from("generation_stale")));
let timeout = find("publication_proof_rejected", 608, "lease_acquire");
assert_eq!(attribute(timeout, "reason"), Some(AnyValue::from("channel_acquisition_timed_out")));
assert!(attribute(timeout, "reported_reason").is_none());
if level != tracing::Level::WARN {
let recovered = find("publication_proof_recovered", 601, "lease_acquire");
assert_eq!(attribute(recovered, "leader_epoch"), Some(AnyValue::Int(41)));
assert_eq!(attribute(recovered, "lease_purpose"), Some(AnyValue::from("observational")));
}
if level == tracing::Level::DEBUG {
let retry = find("publication_proof_retry", 601, "lease_acquire");
assert_eq!(attribute(retry, "leader_epoch"), Some(AnyValue::Int(41)));
assert_eq!(attribute(retry, "lease_purpose"), Some(AnyValue::from("observational")));
}
let unknown = find("publication_proof_rejected", 601, "postscan");
assert!(attribute(unknown, "leader_epoch").is_none());
assert!(attribute(unknown, "lease_purpose").is_none());
let untrusted = find("publication_proof_rejected", 603, "lease_acquire");
assert!(attribute(untrusted, "peer").is_none());
assert!(!format!("{records:?}").contains("remote-secret"));
assert!(!format!("{records:?}").contains("forged:9000"));
assert!(!format!("{records:?}").contains("target-secret"));
assert!(!format!("{records:?}").contains("diagnostic-secret"));
}
}
#[tokio::test]
async fn dropped_lease_wait_does_not_report_a_finished_outcome() {
use tracing::instrument::WithSubscriber as _;
let output = Arc::new(std::sync::Mutex::new(Vec::new()));
let writer = LeaseLogWriter(Arc::clone(&output));
let subscriber = tracing_subscriber::fmt()
.json()
.with_max_level(tracing::Level::DEBUG)
.with_writer(move || writer.clone())
.with_ansi(false)
.without_time()
.finish();
let _interest_guard = tracing::Dispatch::new(tracing::subscriber::NoSubscriber::default());
async {
let ctx = CancellationToken::new();
let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default());
let entered = AtomicBool::new(false);
{
let proof = ScannerCyclePublicationPlan::Observational.await_leases(&ctx, &budget, 602, 43, |_| {
entered.store(true, Ordering::Relaxed);
std::future::pending::<Result<(), EcstoreError>>()
});
tokio::pin!(proof);
assert!(futures::poll!(&mut proof).is_pending());
assert!(entered.load(Ordering::Relaxed));
}
assert!(!ctx.is_cancelled());
assert!(!budget.token().is_cancelled());
}
.with_subscriber(subscriber)
.await;
let bytes = output.lock().expect("lease log mutex").clone();
let logs = String::from_utf8(bytes).expect("lease logs are UTF-8");
let entries: Vec<serde_json::Value> = logs
.lines()
.map(|line| serde_json::from_str(line).expect("lease logs are JSON"))
.collect();
let started = entries
.iter()
.filter(|entry| entry["fields"]["state"] == "publication_proof_started" && entry["fields"]["cycle"] == 602)
.count();
let finished = entries
.iter()
.filter(|entry| entry["fields"]["state"] == "publication_lease_wait_finished" && entry["fields"]["cycle"] == 602)
.count();
assert_eq!(started, 1, "the wait was polled: {logs}");
assert_eq!(finished, 0, "a dropped future has no observed outcome: {logs}");
}
#[test]
fn publication_lease_retries_local_channel_deadline_and_stops_on_cancel() {
const CHILD_MARKER: &str = "RUSTFS_TEST_SCANNER_LEASE_DIAL_RETRY_CHILD";
if std::env::var_os(CHILD_MARKER).is_none() {
let module = module_path!()
.strip_prefix(concat!(env!("CARGO_CRATE_NAME"), "::"))
.unwrap_or(module_path!());
let name = format!("{module}::publication_lease_retries_local_channel_deadline_and_stops_on_cancel");
let output = std::process::Command::new(std::env::current_exe().expect("test executable"))
.args(["--exact", &name, "--nocapture"])
.env(CHILD_MARKER, "1")
.env(rustfs_config::ENV_INTERNODE_CONNECT_TIMEOUT_SECS, "0")
.env("NO_PROXY", "127.0.0.1,localhost")
.env("no_proxy", "127.0.0.1,localhost")
.output()
.expect("channel deadline test subprocess");
assert!(
output.status.success() && String::from_utf8_lossy(&output.stdout).contains("1 passed"),
"channel deadline subprocess failed: {} {}",
String::from_utf8_lossy(&output.stdout),
String::from_utf8_lossy(&output.stderr)
);
return;
}
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.start_paused(true)
.build()
.expect("test runtime");
runtime.block_on(async {
use crate::storage_api::scan::EcstorePeerRestClient;
use rustfs_utils::net::XHost;
async fn channel_deadline_error(
notification: &crate::storage_api::EcstoreNotificationSys,
target: &str,
) -> EcstoreError {
let error = match notification
.acquire_scanner_publication_leases(vec![(target.to_owned(), "session".into(), 3)])
.await
{
Ok(_) => panic!("the stalled channel cannot grant a lease"),
Err(error) => error,
};
let info = crate::storage_api::scan::scanner_publication_lease_failure_info(&error)
.expect("local channel deadline must carry structured diagnostics");
assert_eq!(info.kind, "peer_request_failed");
assert_eq!(info.reason, Some("channel_acquisition_timed_out"));
assert_eq!(info.reported_reason, None);
assert!(!scanner_publication_lease_error_is_retryable(&error.to_string()));
error
}
// Keep the address unique to this isolated subprocess.
let listener = std::net::TcpListener::bind("127.0.0.1:0").expect("local listener");
let address = listener.local_addr().expect("listener address");
let target = format!("https://{address}");
let client = EcstorePeerRestClient::new(
XHost {
name: "127.0.0.1".to_owned(),
port: address.port(),
is_port_set: true,
},
target.clone(),
);
let mut notification = crate::storage_api::EcstoreNotificationSys::new(EndpointServerPools::default()).await;
notification.peer_clients.push(Some(client));
// Block the real channel cache read so the acquisition deadline wins
// independently of OS-specific connect and TLS error ordering.
let cache_read_blocker = rustfs_common::block_connection_cache_reads_for_test().await;
let ctx = CancellationToken::new();
let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default());
let attempts = AtomicUsize::new(0);
let started_at = Instant::now();
let result = ScannerCyclePublicationPlan::Authoritative
.await_leases(&ctx, &budget, 701, 51, |_| {
let attempt = attempts.fetch_add(1, Ordering::SeqCst);
let target = target.clone();
let notification = &notification;
async move {
if attempt > 0 {
return Ok(());
}
Err(channel_deadline_error(notification, &target).await)
}
})
.await;
assert!(matches!(result, ScannerPublicationProofWait::Ready(())));
assert_eq!(attempts.load(Ordering::SeqCst), 2);
assert!(started_at.elapsed() >= scanner_publication_proof_retry_delay(1));
for detail in [
"channel_acquisition_timed_out",
"scanner publication lease acquisition failed: peer=peer-a:9000, purpose=authoritative: Io error: internode channel acquisition deadline elapsed",
] {
let attempts = AtomicUsize::new(0);
let result = ScannerCyclePublicationPlan::Authoritative
.await_leases(&ctx, &budget, 702, 52, |_| {
attempts.fetch_add(1, Ordering::SeqCst);
async move { Err::<(), _>(EcstoreError::other(detail)) }
})
.await;
assert!(matches!(result, ScannerPublicationProofWait::Rejected(_)));
assert_eq!(attempts.load(Ordering::SeqCst), 1, "remote text must not create a local diagnosis");
}
let cancel_ctx = CancellationToken::new();
let cancel_budget = ScannerCycleBudget::new(&cancel_ctx, ScannerCycleBudgetConfig::default());
let attempts = AtomicUsize::new(0);
let started_at = Instant::now();
let result = ScannerCyclePublicationPlan::Authoritative
.await_leases(&cancel_ctx, &cancel_budget, 703, 53, |_| {
let attempt = attempts.fetch_add(1, Ordering::SeqCst);
let target = target.clone();
let notification = &notification;
let cancel_ctx = cancel_ctx.clone();
async move {
assert_eq!(attempt, 0, "cancellation must stop the scheduled retry");
let error = channel_deadline_error(notification, &target).await;
tokio::spawn(async move {
tokio::time::sleep(Duration::from_secs(1)).await;
cancel_ctx.cancel();
});
Err::<(), _>(error)
}
})
.await;
assert!(matches!(result, ScannerPublicationProofWait::Cancelled));
assert_eq!(attempts.load(Ordering::SeqCst), 1);
assert!(started_at.elapsed() < scanner_publication_proof_retry_delay(1));
let progress_ctx = CancellationToken::new();
let progress_budget = ScannerCycleBudget::new(&progress_ctx, ScannerCycleBudgetConfig::default());
let attempts = AtomicUsize::new(0);
let started_at = Instant::now();
let result = ScannerCyclePublicationPlan::ProgressOnly
.await_leases(&progress_ctx, &progress_budget, 704, 54, |observational| {
assert!(observational);
attempts.fetch_add(1, Ordering::SeqCst);
let target = target.clone();
let notification = &notification;
async move { Err::<(), _>(channel_deadline_error(notification, &target).await) }
})
.await;
assert!(matches!(result, ScannerPublicationProofWait::Cancelled));
assert_eq!(attempts.load(Ordering::SeqCst), 1);
assert!(started_at.elapsed() < scanner_publication_proof_retry_delay(1));
drop(cache_read_blocker);
drop(listener);
});
}
#[tokio::test]
#[serial]
async fn exhausted_budget_still_proves_peer_movement_without_namespace_authority() {
@@ -33,7 +654,7 @@ async fn exhausted_budget_still_proves_peer_movement_without_namespace_authority
let calls = AtomicUsize::new(0);
let proof = plan
.await_leases(&ctx, &budget, 352, |observational| {
.await_leases(&ctx, &budget, 352, 19, |observational| {
calls.fetch_add(1, Ordering::Relaxed);
store.acquire_scanner_publication_lease_with_purpose(generation, Duration::from_secs(60), observational)
})
@@ -57,7 +678,7 @@ async fn exhausted_budget_still_proves_peer_movement_without_namespace_authority
assert!(store.release_scanner_publication_lease(token).await);
let stale = plan
.await_leases(&ctx, &budget, 352, |observational| {
.await_leases(&ctx, &budget, 352, 19, |observational| {
store.acquire_scanner_publication_lease_with_purpose(generation + 1, Duration::from_secs(60), observational)
})
.await;
@@ -71,7 +692,7 @@ async fn exhausted_budget_still_proves_peer_movement_without_namespace_authority
assert!(store.scanner_data_movement_pause_status().await.paused);
let current_generation = store.scanner_data_movement_generation();
let moving = plan
.await_leases(&ctx, &budget, 352, |observational| {
.await_leases(&ctx, &budget, 352, 19, |observational| {
store.acquire_scanner_publication_lease_with_purpose(current_generation, Duration::from_secs(60), observational)
})
.await;
@@ -101,15 +722,21 @@ async fn emitted_updates_keep_budget_cancellation_and_publication_purpose() {
let ctx = CancellationToken::new();
let budget = ScannerCycleBudget::new(&ctx, ScannerCycleBudgetConfig::default());
let ready = plan
.await_leases(&ctx, &budget, 352, |observational| async move { Ok::<_, String>(observational) })
.await_leases(
&ctx,
&budget,
352,
19,
|observational| async move { Ok::<_, EcstoreError>(observational) },
)
.await;
assert!(matches!(ready, ScannerPublicationProofWait::Ready(observational) if observational == plan.is_observational()));
budget.cancel_for_runtime();
let calls = AtomicUsize::new(0);
let cancelled = plan
.await_leases(&ctx, &budget, 352, |_| {
.await_leases(&ctx, &budget, 352, 19, |_| {
calls.fetch_add(1, Ordering::Relaxed);
async { Ok::<(), String>(()) }
async { Ok::<(), EcstoreError>(()) }
})
.await;
assert!(matches!(cancelled, ScannerPublicationProofWait::Cancelled));
@@ -124,9 +751,9 @@ async fn progress_proof_honors_parent_cancellation() {
ctx.cancel();
let calls = AtomicUsize::new(0);
let proof = ScannerCyclePublicationPlan::ProgressOnly
.await_leases(&ctx, &budget, 352, |_| {
.await_leases(&ctx, &budget, 352, 19, |_| {
calls.fetch_add(1, Ordering::Relaxed);
async { Ok::<_, String>(()) }
async { Ok::<_, EcstoreError>(()) }
})
.await;
assert!(matches!(proof, ScannerPublicationProofWait::Cancelled));
@@ -149,13 +776,13 @@ async fn progress_proof_is_bounded_without_a_runtime_budget() {
let calls = AtomicUsize::new(0);
let started = Instant::now();
let proof = ScannerCyclePublicationPlan::ProgressOnly
.await_leases(&ctx, &budget, 352, |_| {
.await_leases(&ctx, &budget, 352, 19, |_| {
calls.fetch_add(1, Ordering::Relaxed);
async move {
if stalled {
std::future::pending::<()>().await;
}
Err::<(), _>("scanner publication lease capacity is exhausted")
Err::<(), _>(EcstoreError::other("scanner publication lease capacity is exhausted"))
}
})
.await;
+176 -163
View File
@@ -12,6 +12,7 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use std::borrow::Cow;
use std::collections::{HashMap, HashSet};
use std::fs::FileType;
use std::io::ErrorKind;
@@ -751,11 +752,36 @@ struct ScannerCheckpointAncestor {
entry: DataUsageEntry,
}
struct ScannerUsageUpdates {
sender: mpsc::Sender<DataUsageEntry>,
cache: DataUsageCache,
last_update: SystemTime,
}
impl ScannerUsageUpdates {
fn new(sender: mpsc::Sender<DataUsageEntry>, bucket: String) -> Self {
Self {
sender,
cache: DataUsageCache {
info: DataUsageCacheInfo {
name: bucket,
..Default::default()
},
..Default::default()
},
last_update: SystemTime::UNIX_EPOCH,
}
}
fn is_due(&self) -> bool {
self.last_update.elapsed().unwrap_or(Duration::ZERO) >= Duration::from_secs(60)
}
}
pub struct FolderScanner {
root: String,
old_cache: DataUsageCache,
new_cache: DataUsageCache,
update_cache: DataUsageCache,
data_usage_scanner_debug: bool,
heal_object_select: u32,
@@ -773,11 +799,10 @@ pub struct FolderScanner {
disks: Vec<Arc<Disk>>,
disks_quorum: usize,
updates: Option<mpsc::Sender<DataUsageEntry>>,
usage_updates: Option<ScannerUsageUpdates>,
checkpoint_tx: Option<mpsc::Sender<DataUsageCache>>,
checkpoint_ancestors: Vec<ScannerCheckpointAncestor>,
checkpoint_depth: usize,
last_update: SystemTime,
checkpoint_objects: u64,
object_metadata_visits: metrics::Counter,
last_checkpoint_objects: u64,
@@ -798,10 +823,6 @@ pub struct FolderScanner {
resume_failed_paths: Vec<String>,
coverage_gap: bool,
raw_enumeration_progress: Vec<RawEnumerationProgress>,
pending_heal_sync_deferred: bool,
pending_heal_batch_dirty: bool,
#[cfg(test)]
pending_heal_sync_count: usize,
pending_size_reconciliation_keys: HashSet<String>,
pending_size_reconciliation_scopes: HashSet<String>,
pending_size_reconciliation_truncated: bool,
@@ -1036,8 +1057,9 @@ impl FolderScanner {
fn reuse_clean_root_child(&mut self, child: &CachedFolder, child_hash: &DataUsageHash, into: &mut DataUsageEntry) {
self.new_cache.copy_with_children(&self.old_cache, child_hash, &child.parent);
self.update_cache
.copy_with_children(&self.old_cache, child_hash, &child.parent);
if let Some(updates) = &mut self.usage_updates {
updates.cache.copy_with_children(&self.old_cache, child_hash, &child.parent);
}
into.add_child(child_hash);
}
@@ -1145,8 +1167,8 @@ impl FolderScanner {
}
/// Apply the per-object size-resolution ledger updates in one place. The
/// scanner cache is the durable boundary; both working copies are updated
/// so an incremental publication cannot lose a debt or its resolution.
/// scanner cache is the durable boundary; progress updates contain only
/// usage totals and never own a copy of the reconciliation ledger.
fn apply_size_reconciliation(&mut self, summary: &SizeSummary) {
let now = Self::now_secs();
self.pending_size_reconciliation_keys
@@ -1159,26 +1181,25 @@ impl FolderScanner {
);
self.pending_size_reconciliation_truncated |= summary.size_reconciliation_truncated;
for info in [&mut self.new_cache.info, &mut self.update_cache.info] {
for incoming in &summary.size_reconciliation {
if let Some(existing) = info.size_reconciliation.get_mut(&incoming.key) {
existing.reason = incoming.reason.clone();
existing.physical_size = incoming.physical_size;
existing.generation = incoming.generation.clone();
existing.version_id = incoming.version_id.clone();
existing.attempts = existing.attempts.saturating_add(1);
continue;
}
if size_reconciliation_entry_bytes(incoming) > MAX_SIZE_RECONCILIATION_BYTES_PER_BUCKET {
continue;
}
let mut entry = incoming.clone();
entry.first_seen = now;
entry.attempts = 1;
info.size_reconciliation.insert(entry.key.clone(), entry);
let info = &mut self.new_cache.info;
for incoming in &summary.size_reconciliation {
if let Some(existing) = info.size_reconciliation.get_mut(&incoming.key) {
existing.reason = incoming.reason.clone();
existing.physical_size = incoming.physical_size;
existing.generation = incoming.generation.clone();
existing.version_id = incoming.version_id.clone();
existing.attempts = existing.attempts.saturating_add(1);
continue;
}
if size_reconciliation_entry_bytes(incoming) > MAX_SIZE_RECONCILIATION_BYTES_PER_BUCKET {
continue;
}
let mut entry = incoming.clone();
entry.first_seen = now;
entry.attempts = 1;
info.size_reconciliation.insert(entry.key.clone(), entry);
}
}
@@ -1188,27 +1209,24 @@ impl FolderScanner {
let scopes = std::mem::take(&mut self.pending_size_reconciliation_scopes);
let truncated = std::mem::replace(&mut self.pending_size_reconciliation_truncated, false);
for info in [&mut self.new_cache.info, &mut self.update_cache.info] {
if !truncated {
info.size_reconciliation.retain(|key, entry| {
!scopes.contains(&size_reconciliation_scope_key(&entry.bucket, &entry.object)) || current_keys.contains(key)
});
}
prune_size_reconciliation(info, now);
let info = &mut self.new_cache.info;
if !truncated {
info.size_reconciliation.retain(|key, entry| {
!scopes.contains(&size_reconciliation_scope_key(&entry.bucket, &entry.object)) || current_keys.contains(key)
});
}
prune_size_reconciliation(info, now);
}
fn record_scan_resume_hint(&mut self, folder: &str) {
self.new_cache.info.scan_resume_after = Some(folder.to_string());
self.update_cache.info.scan_resume_after = Some(folder.to_string());
let checkpoint = DataUsageScanCheckpoint::new(folder.to_string(), DataUsageScanCheckpointReason::Unknown);
global_metrics().record_scanner_checkpoint_set(
checkpoint.version,
checkpoint.resume_after.clone(),
checkpoint.reason.as_str(),
);
self.new_cache.info.scan_checkpoint = Some(checkpoint.clone());
self.update_cache.info.scan_checkpoint = Some(checkpoint);
self.new_cache.info.scan_checkpoint = Some(checkpoint);
}
fn record_completed_child(&mut self, folder: &str) {
@@ -1511,8 +1529,7 @@ impl FolderScanner {
self.record_scan_resume_hint_if_not_ancestor(&child_hash.key());
parent_entry.add_child(child_hash);
self.new_cache.replace_hashed(child_hash, parent, &child_entry);
self.update_cache.delete_recursive(child_hash);
self.update_cache.copy_with_children(&self.new_cache, child_hash, parent);
self.refresh_usage_subtree(child_hash, parent);
self.send_update().await;
}
}
@@ -1655,29 +1672,75 @@ impl FolderScanner {
self.data_usage_scanner_debug = debug;
}
/// Send update if enough time has passed
/// Should be called on a regular basis when the new_cache contains more recent total than previously.
/// May or may not send an update upstream.
fn should_send_update(&self) -> bool {
if self.updates.is_none() {
return false;
fn compact_completed_folder(&mut self, folder: &CachedFolder, hash: &DataUsageHash, entry: &DataUsageEntry) {
// Completed healthy subtrees can keep aggregate usage. Coverage gaps
// and failed-object debt still need child entries for precise resume
// and to avoid rereading healthy siblings on a retry.
let forward_sweep = self.old_cache.info.scan_progress.is_some();
if entry.compacted || self.new_cache.info.name == folder.name || (forward_sweep && self.coverage_gap) {
return;
}
let summary = if entry.children.is_empty() {
Cow::Borrowed(entry)
} else if let Some(flat) = self.new_cache.size_recursive(&hash.0) {
Cow::Owned(flat)
} else {
return;
};
if forward_sweep && summary.failed_objects != 0 {
return;
}
let should_compact = summary.objects < DATA_SCANNER_COMPACT_LEAST_OBJECT
|| entry.children.iter().all(|key| {
self.new_cache
.cache
.get(key)
.is_none_or(|child| child.children.is_empty() && child.objects <= 1)
});
if !should_compact {
return;
}
// Heal revisits also reach this path without a parent advancing
// coverage. Keep the endpoint addressable when its completed
// subtree is replaced by an aggregate.
if self.coverage_frontier.as_deref().is_some_and(|frontier| {
frontier
.strip_prefix(&folder.name)
.is_some_and(|suffix| suffix.starts_with('/'))
}) {
self.record_completed_child(&folder.name);
}
match summary {
Cow::Borrowed(_) => {
// The leaf is already cached; only its sampling mode changes.
if let Some(cached) = self.new_cache.cache.get_mut(&hash.0) {
cached.compacted = true;
}
}
Cow::Owned(mut flat) => {
flat.compacted = true;
self.new_cache.delete_recursive(hash);
self.new_cache.replace_hashed(hash, &folder.parent, &flat);
}
}
let elapsed = self.last_update.elapsed().unwrap_or(Duration::from_secs(0));
elapsed >= Duration::from_secs(60)
}
fn refresh_usage_subtree(&mut self, hash: &DataUsageHash, parent: &Option<DataUsageHash>) {
if let Some(updates) = &mut self.usage_updates {
updates.cache.delete_recursive(hash);
updates.cache.copy_with_children(&self.new_cache, hash, parent);
}
}
/// Send at most one aggregate per minute when a caller requested progress.
pub async fn send_update(&mut self) {
// Send at most an update every minute.
if !self.should_send_update() {
let Some(updates) = &mut self.usage_updates else { return };
if !updates.is_due() {
return;
}
if let Some(flat) = self.update_cache.size_recursive(&self.new_cache.info.name)
&& let Some(ref updates) = self.updates
{
// Try to send without blocking
if let Err(e) = updates.send(flat).await {
if let Some(flat) = updates.cache.size_recursive(&self.new_cache.info.name) {
if let Err(e) = updates.sender.send(flat).await {
error!(
target: "rustfs::scanner::folder",
event = EVENT_SCANNER_FOLDER_STATE,
@@ -1689,17 +1752,17 @@ impl FolderScanner {
"Scanner folder update send failed"
);
}
self.last_update = SystemTime::now();
updates.last_update = SystemTime::now();
}
}
async fn send_update_for_entry(&mut self, hash: &DataUsageHash, parent: &Option<DataUsageHash>, entry: &DataUsageEntry) {
if !self.should_send_update() {
return;
if let Some(updates) = &mut self.usage_updates
&& updates.is_due()
{
updates.cache.replace_hashed(hash, parent, entry);
self.send_update().await;
}
self.update_cache.replace_hashed(hash, parent, entry);
self.send_update().await;
}
async fn record_missing_erasure_metadata(
@@ -1838,8 +1901,7 @@ impl FolderScanner {
wait_for_scanner_io(&ctx, self.sleeper.sleep_folder()).await?;
}
let mut existing_folders: Vec<CachedFolder> = Vec::new();
let mut new_folders: Vec<CachedFolder> = Vec::new();
let mut queued_folders: Vec<QueuedFolder> = Vec::new();
let mut found_object_metadata = false;
let mut object_count: u64 = 0;
let yield_every_objects = scanner_yield_every_n_objects();
@@ -2147,12 +2209,21 @@ impl FolderScanner {
self.reuse_clean_root_child(&this, &h, into);
} else if exists {
abandoned_children.remove(&h.key());
existing_folders.push(this);
self.update_cache
.copy_with_children(&self.old_cache, &h, &Some(this_hash.clone()));
queued_folders.push(QueuedFolder {
folder: this,
source: FolderScanSource::Existing,
});
if let Some(updates) = &mut self.usage_updates {
updates
.cache
.copy_with_children(&self.old_cache, &h, &Some(this_hash.clone()));
}
} else {
abandoned_children.remove(&h.key());
new_folders.push(this);
queued_folders.push(QueuedFolder {
folder: this,
source: FolderScanSource::New,
});
}
continue;
}
@@ -2374,17 +2445,17 @@ impl FolderScanner {
// frontier. Keep the emergency compaction limit for large trees.
let should_compact = (self.new_cache.info.name != folder.name
&& self.old_cache.info.scan_progress.is_none()
&& existing_folders.len() + new_folders.len() >= DATA_SCANNER_COMPACT_AT_FOLDERS)
|| existing_folders.len() + new_folders.len() >= DATA_SCANNER_FORCE_COMPACT_AT_FOLDERS;
&& queued_folders.len() >= DATA_SCANNER_COMPACT_AT_FOLDERS)
|| queued_folders.len() >= DATA_SCANNER_FORCE_COMPACT_AT_FOLDERS;
let total_folders = existing_folders.len() + new_folders.len();
let total_folders = queued_folders.len();
self.alert_excessive_folders(&folder.name, total_folders);
if !into.compacted && should_compact {
into.compacted = true;
new_folders.append(&mut existing_folders);
existing_folders.clear();
for queued in &mut queued_folders {
queued.source = FolderScanSource::New;
}
if self.data_usage_scanner_debug {
debug!(
@@ -2393,20 +2464,13 @@ impl FolderScanner {
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_FOLDER,
folder = %folder.name,
entry_count = new_folders.len(),
entry_count = queued_folders.len(),
state = "preemptive_compaction",
"Scanner folder switched to compacted mode"
);
}
}
if !into.compacted {
for folder_item in &existing_folders {
let h = hash_path(&folder_item.name);
self.update_cache.copy_with_children(&self.old_cache, &h, &folder_item.parent);
}
}
let is_scan_root = folder.name == self.old_cache.info.name;
let scan_checkpoint = self.old_cache.info.scan_checkpoint.as_ref();
let checkpoint_resume_after = scan_checkpoint.and_then(|checkpoint| {
@@ -2430,15 +2494,6 @@ impl FolderScanner {
.and_then(|resume_after| folder_resume_match(&folder.name, resume_after))
.is_some_and(|resume_match| matches!(resume_match, FolderResumeMatch::Descendant));
let scan_resume_after = checkpoint_resume_after.or(self.old_cache.info.scan_resume_after.as_deref());
let mut queued_folders = Vec::with_capacity(new_folders.len() + existing_folders.len());
queued_folders.extend(new_folders.into_iter().map(|folder| QueuedFolder {
folder,
source: FolderScanSource::New,
}));
queued_folders.extend(existing_folders.into_iter().map(|folder| QueuedFolder {
folder,
source: FolderScanSource::Existing,
}));
let has_queued_folders = !queued_folders.is_empty();
let forward_sweep = self.old_cache.info.scan_progress.is_some();
let resume_order = if forward_sweep {
@@ -2479,19 +2534,21 @@ impl FolderScanner {
match queued_folder.source {
FolderScanSource::New => {
// Add new folders to the update tree so totals update for these.
if !into.compacted {
if !into.compacted
&& let Some(updates) = &mut self.usage_updates
{
let mut found_any = false;
let mut parent = this_hash.clone();
let update_cache_name_hash = hash_path(&self.update_cache.info.name);
let update_root_hash = hash_path(&updates.cache.info.name);
while parent != update_cache_name_hash {
while parent != update_root_hash {
let parent_key = parent.key();
let e = self.update_cache.find(&parent_key);
let e = updates.cache.find(&parent_key);
if e.is_none_or(|v| v.compacted) {
found_any = true;
break;
}
if let Some(next) = self.update_cache.search_parent(&parent) {
if let Some(next) = updates.cache.search_parent(&parent) {
parent = next;
} else {
found_any = true;
@@ -2500,7 +2557,8 @@ impl FolderScanner {
}
if !found_any {
// Add non-compacted empty entry.
self.update_cache
updates
.cache
.replace_hashed(&h, &Some(this_hash.clone()), &DataUsageEntry::default());
}
}
@@ -2574,13 +2632,17 @@ impl FolderScanner {
};
if outcome == FolderScanOutcome::Vanished {
self.new_cache.delete_recursive(&h);
self.update_cache.delete_recursive(&h);
if let Some(updates) = &mut self.usage_updates {
updates.cache.delete_recursive(&h);
}
into.children.remove(&h.key());
continue;
}
if outcome == FolderScanOutcome::ErasureData {
self.new_cache.delete_recursive(&h);
self.update_cache.delete_recursive(&h);
if let Some(updates) = &mut self.usage_updates {
updates.cache.delete_recursive(&h);
}
into.children.remove(&h.key());
// Earlier namespace siblings may have completed, but none
// can certify coverage of this unresolved parent object.
@@ -2600,23 +2662,13 @@ impl FolderScanner {
self.send_update_for_entry(&this_hash, &folder.parent, into).await;
tokio::task::yield_now().await;
} else {
tokio::task::yield_now().await;
into.add_child(&h);
self.record_completed_child(&folder_item.name);
tokio::task::yield_now().await;
self.maybe_send_checkpoint(&folder, &this_hash, into);
self.update_cache.delete_recursive(&h);
self.update_cache.copy_with_children(&self.new_cache, &h, &folder_item.parent);
self.refresh_usage_subtree(&h, &folder_item.parent);
self.send_update().await;
}
if queued_folder.source == FolderScanSource::New
&& !into.compacted
&& self.update_cache.find(&this_hash.key()).is_some_and(|v| !v.compacted)
{
self.update_cache.delete_recursive(&h);
self.update_cache
.copy_with_children(&self.new_cache, &h, &Some(this_hash.clone()));
}
}
// Scan for healing
@@ -3013,8 +3065,7 @@ impl FolderScanner {
into.add_child(&h);
// We scanned a folder, optionally send update.
self.update_cache.delete_recursive(&h);
self.update_cache.copy_with_children(&self.new_cache, &h, &folder_item.parent);
self.refresh_usage_subtree(&h, &folder_item.parent);
self.send_update().await;
self.maybe_send_checkpoint(&folder, &this_hash, into);
}
@@ -3028,36 +3079,7 @@ impl FolderScanner {
self.new_cache.replace_hashed(&this_hash, &folder.parent, into);
}
// Keep independently accounted children while the sweep cursor may
// reference them; the hard cardinality compaction below still applies.
if !into.compacted
&& self.old_cache.info.scan_progress.is_none()
&& self.new_cache.info.name != folder.name
&& let Some(mut flat) = self.new_cache.size_recursive(&this_hash.key())
{
flat.compacted = true;
let mut should_compact = false;
if flat.objects < DATA_SCANNER_COMPACT_LEAST_OBJECT {
should_compact = true;
} else {
// Compact if we only have objects as children...
should_compact = true;
for k in &into.children {
if let Some(v) = self.new_cache.cache.get(k)
&& (!v.children.is_empty() || v.objects > 1)
{
should_compact = false;
break;
}
}
}
if should_compact {
self.new_cache.delete_recursive(&this_hash);
self.new_cache.replace_hashed(&this_hash, &folder.parent, &flat);
}
}
self.compact_completed_folder(&folder, &this_hash, into);
// Compact if too many children...
if !into.compacted {
@@ -3070,12 +3092,13 @@ impl FolderScanner {
done_compact();
}
if self.update_cache.cache.contains_key(&this_hash.key()) && !was_compacted {
// Replace if existed before.
if let Some(flat) = self.new_cache.size_recursive(&this_hash.key()) {
self.update_cache.delete_recursive(&this_hash);
self.update_cache.replace_hashed(&this_hash, &folder.parent, &flat);
}
if !was_compacted
&& let Some(updates) = &mut self.usage_updates
&& updates.cache.cache.contains_key(&this_hash.key())
&& let Some(flat) = self.new_cache.size_recursive(&this_hash.key())
{
updates.cache.delete_recursive(&this_hash);
updates.cache.replace_hashed(&this_hash, &folder.parent, &flat);
}
self.finish_size_reconciliation_batch();
@@ -3173,10 +3196,7 @@ pub(crate) async fn scan_data_folder_scoped(
info: cache.info.clone(),
..Default::default()
};
let update_cache = DataUsageCache {
info: cache.info.clone(),
..Default::default()
};
let usage_updates = updates.map(|sender| ScannerUsageUpdates::new(sender, bucket_name.clone()));
let resume_failed_paths = if resume_frontier.is_some() {
failed_scan_paths(&cache)
} else {
@@ -3187,7 +3207,6 @@ pub(crate) async fn scan_data_folder_scoped(
root: base_path,
old_cache: cache,
new_cache,
update_cache,
data_usage_scanner_debug: false,
heal_object_select,
scan_mode,
@@ -3200,11 +3219,10 @@ pub(crate) async fn scan_data_folder_scoped(
sleeper,
disks,
disks_quorum,
updates,
usage_updates,
checkpoint_tx,
checkpoint_ancestors: Vec::new(),
checkpoint_depth: 0,
last_update: SystemTime::UNIX_EPOCH,
checkpoint_objects: 0,
object_metadata_visits,
last_checkpoint_objects: 0,
@@ -3220,11 +3238,7 @@ pub(crate) async fn scan_data_folder_scoped(
resume_frontier,
resume_failed_paths,
coverage_gap: false,
pending_heal_sync_deferred: false,
pending_heal_batch_dirty: false,
raw_enumeration_progress: Vec::new(),
#[cfg(test)]
pending_heal_sync_count: 0,
pending_size_reconciliation_keys: HashSet::new(),
pending_size_reconciliation_scopes: HashSet::new(),
pending_size_reconciliation_truncated: false,
@@ -3234,7 +3248,6 @@ pub(crate) async fn scan_data_folder_scoped(
let now = FolderScanner::now_secs();
prune_size_reconciliation(&mut scanner.new_cache.info, now);
prune_size_reconciliation(&mut scanner.update_cache.info, now);
// Check if context is cancelled
if ctx.is_cancelled() {
+26 -66
View File
@@ -18,28 +18,6 @@ use super::*;
const PENDING_HEAL_RETRY_BASE_SECS: u64 = 15 * 60;
const PENDING_HEAL_RETRY_CAP_SECS: u64 = 6 * 60 * 60;
pub(super) struct PendingHealSyncBatch<'a> {
pub(super) scanner: &'a mut FolderScanner,
}
impl<'a> PendingHealSyncBatch<'a> {
pub(super) fn new(scanner: &'a mut FolderScanner) -> Self {
scanner.pending_heal_sync_deferred = true;
scanner.pending_heal_batch_dirty = false;
Self { scanner }
}
}
impl Drop for PendingHealSyncBatch<'_> {
fn drop(&mut self) {
self.scanner.pending_heal_sync_deferred = false;
if self.scanner.pending_heal_batch_dirty {
self.scanner.pending_heal_batch_dirty = false;
self.scanner.sync_pending_heals();
}
}
}
pub(super) fn record_pending_heal_attempt(entry: &mut PendingScannerHeal, now: u64) {
entry.last_attempt = now;
entry.attempts = entry.attempts.saturating_add(1);
@@ -52,19 +30,6 @@ pub(super) fn observe_pending_heal_admission(entry: &mut PendingScannerHeal, res
}
impl FolderScanner {
pub(super) fn sync_pending_heals(&mut self) {
self.pending_heals_changed = true;
if self.pending_heal_sync_deferred {
self.pending_heal_batch_dirty = true;
return;
}
self.update_cache.info.pending_heals = self.new_cache.info.pending_heals.clone();
#[cfg(test)]
{
self.pending_heal_sync_count += 1;
}
}
pub(super) fn clear_pending_scanner_heal(
&mut self,
kind: PendingScannerHealKind,
@@ -78,7 +43,7 @@ impl FolderScanner {
.pending_heals
.retain(|entry| !pending_scanner_heal_matches(entry, kind, bucket, object, version_id));
if self.new_cache.info.pending_heals.len() != before {
self.sync_pending_heals();
self.pending_heals_changed = true;
}
}
@@ -100,7 +65,7 @@ impl FolderScanner {
.find(|entry| pending_scanner_heal_matches(entry, kind, bucket, object, version_id))
{
observe_pending_heal_admission(entry, result);
self.sync_pending_heals();
self.pending_heals_changed = true;
return;
}
@@ -116,9 +81,8 @@ impl FolderScanner {
last_admission_result: result.result_label().to_string(),
last_admission_reason: result.reason_label().to_string(),
});
if self.prune_pending_scanner_heal_capacity() == 0 {
self.sync_pending_heals();
}
self.pending_heals_changed = true;
self.prune_pending_scanner_heal_capacity();
}
/// Preserve the discovery reason when a candidate could not be admitted
@@ -140,7 +104,7 @@ impl FolderScanner {
.find(|entry| pending_scanner_heal_matches(entry, kind, bucket, object, version_id))
{
entry.last_admission_reason = reason.to_string();
self.sync_pending_heals();
self.pending_heals_changed = true;
}
}
@@ -169,22 +133,22 @@ impl FolderScanner {
state = "pending_heal_expired",
"Scanner pending heal ledger expired old entries"
);
self.sync_pending_heals();
self.pending_heals_changed = true;
}
self.prune_pending_scanner_heal_capacity();
}
fn prune_pending_scanner_heal_capacity(&mut self) -> usize {
fn prune_pending_scanner_heal_capacity(&mut self) {
let len = self.new_cache.info.pending_heals.len();
if len <= MAX_PENDING_SCANNER_HEALS_PER_BUCKET {
return 0;
return;
}
sort_pending_scanner_heals_for_retry(&mut self.new_cache.info.pending_heals);
let remove_count = len.saturating_sub(MAX_PENDING_SCANNER_HEALS_PER_BUCKET);
self.new_cache.info.pending_heals.drain(..remove_count);
self.sync_pending_heals();
self.pending_heals_changed = true;
counter!(
METRIC_SCANNER_PENDING_HEAL_PRUNE_TOTAL,
"bucket" => self.new_cache.info.name.clone()
@@ -201,7 +165,6 @@ impl FolderScanner {
state = "pending_heal_pruned",
"Scanner pending heal ledger pruned oldest entries"
);
remove_count
}
pub(super) fn update_pending_scanner_heal_after_admission(
@@ -233,20 +196,18 @@ impl FolderScanner {
.find(|entry| pending_scanner_heal_matches(entry, kind, bucket, object, version_id))
{
observe_pending_heal_admission(entry, result);
self.sync_pending_heals();
self.pending_heals_changed = true;
}
}
}
}
pub(super) async fn retry_pending_scanner_heals(&mut self) -> Result<(), ScannerError> {
let batch = PendingHealSyncBatch::new(self);
let scanner = &mut *batch.scanner;
if !scanner.should_heal().await {
if !self.should_heal().await {
return Ok(());
}
let bucket = scanner.new_cache.info.name.clone();
let bucket = self.new_cache.info.name.clone();
// Legacy notices cannot bind a verified disposition to the current
// incarnation, kind, set scope and responsibility generation.
let repaired = rustfs_common::mrf_channel::take_mrf_repaired_events_for(&bucket);
@@ -254,14 +215,14 @@ impl FolderScanner {
counter!("rustfs_scanner_unverified_repair_notices_total")
.increment(u64::try_from(repaired.len()).unwrap_or(u64::MAX));
}
scanner.prune_pending_scanner_heals();
for pending in pending_scanner_heal_retry_candidates(&scanner.new_cache.info.pending_heals, &bucket) {
if !scanner.should_heal().await {
self.prune_pending_scanner_heals();
for pending in pending_scanner_heal_retry_candidates(&self.new_cache.info.pending_heals, &bucket) {
if !self.should_heal().await {
break;
}
let Some(request) = build_pending_scanner_heal_request(&pending) else {
scanner.clear_pending_scanner_heal(pending.kind, &pending.bucket, None, pending.version_id.as_deref());
self.clear_pending_scanner_heal(pending.kind, &pending.bucket, None, pending.version_id.as_deref());
counter!(
METRIC_SCANNER_PENDING_HEAL_MALFORMED_TOTAL,
"bucket" => pending.bucket.clone(),
@@ -280,7 +241,7 @@ impl FolderScanner {
continue;
};
if let Some(entry) = scanner.new_cache.info.pending_heals.iter_mut().find(|entry| {
if let Some(entry) = self.new_cache.info.pending_heals.iter_mut().find(|entry| {
pending_scanner_heal_matches(
entry,
pending.kind,
@@ -290,17 +251,16 @@ impl FolderScanner {
)
}) {
record_pending_heal_attempt(entry, Self::now_secs());
scanner.sync_pending_heals();
self.pending_heals_changed = true;
}
scanner
.send_required_scanner_heal_request(
pending.kind,
pending.bucket.clone(),
pending.object.clone(),
pending.version_id.clone(),
request,
)
.await?;
self.send_required_scanner_heal_request(
pending.kind,
pending.bucket.clone(),
pending.object.clone(),
pending.version_id.clone(),
request,
)
.await?;
}
Ok(())
+12 -33
View File
@@ -31,6 +31,7 @@ pub(super) mod checkpoint_fixture;
pub(super) mod enumeration_restart;
mod incremental_enumeration;
mod metadata_visits;
mod usage_updates;
/// Reset the process-global alert cooldown map; test-only.
fn reset_alert_cooldowns() {
@@ -353,7 +354,6 @@ async fn build_test_scanner() -> (FolderScanner, std::path::PathBuf) {
root: temp_dir.to_string_lossy().to_string(),
old_cache: DataUsageCache::default(),
new_cache: DataUsageCache::default(),
update_cache: DataUsageCache::default(),
data_usage_scanner_debug: false,
heal_object_select: 0,
scan_mode: HealScanMode::Normal,
@@ -366,11 +366,10 @@ async fn build_test_scanner() -> (FolderScanner, std::path::PathBuf) {
sleeper: SCANNER_SLEEPER.clone(),
disks: Vec::new(),
disks_quorum: 0,
updates: None,
usage_updates: None,
checkpoint_tx: None,
checkpoint_ancestors: Vec::new(),
checkpoint_depth: 0,
last_update: SystemTime::UNIX_EPOCH,
checkpoint_objects: 0,
object_metadata_visits: metrics::Counter::noop(),
last_checkpoint_objects: 0,
@@ -391,9 +390,6 @@ async fn build_test_scanner() -> (FolderScanner, std::path::PathBuf) {
resume_failed_paths: Vec::new(),
coverage_gap: false,
raw_enumeration_progress: Vec::new(),
pending_heal_sync_deferred: false,
pending_heal_batch_dirty: false,
pending_heal_sync_count: 0,
pending_size_reconciliation_keys: HashSet::new(),
pending_size_reconciliation_scopes: HashSet::new(),
pending_size_reconciliation_truncated: false,
@@ -710,7 +706,6 @@ async fn malformed_size_reconciliation_replays_after_restart() {
scanner.finish_size_reconciliation_batch();
assert_eq!(scanner.new_cache.info.size_reconciliation.len(), 1);
assert_eq!(scanner.update_cache.info.size_reconciliation.len(), 1);
assert_eq!(scanner.new_cache.info.size_reconciliation[&entry.key].attempts, 2);
let encoded = rmp_serde::to_vec_named(&scanner.new_cache.info).expect("size ledger should encode");
@@ -724,7 +719,6 @@ async fn malformed_size_reconciliation_replays_after_restart() {
scanner.apply_size_reconciliation(&resolved);
scanner.finish_size_reconciliation_batch();
assert!(scanner.new_cache.info.size_reconciliation.is_empty());
assert!(scanner.update_cache.info.size_reconciliation.is_empty());
}
#[tokio::test]
@@ -1532,7 +1526,6 @@ async fn mrf_ownership_legacy_notices_preserve_pending_entries() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(u64::MAX, usize::MAX, &mut scanner, temp_dir);
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
scanner.heal_object_select = 1;
let version = Uuid::new_v4().to_string();
@@ -1679,7 +1672,6 @@ async fn test_pending_heal_prune_expires_stale_entries() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(u64::MAX, usize::MAX, &mut scanner, temp_dir);
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
let mut stale = pending_heal(PendingScannerHealKind::Object, "bucket", Some("stale"), None, 1, 1);
stale.first_seen = FolderScanner::now_secs().saturating_sub(MAX_PENDING_SCANNER_HEAL_AGE_SECS + 1);
@@ -1690,7 +1682,6 @@ async fn test_pending_heal_prune_expires_stale_entries() {
assert_eq!(scanner.new_cache.info.pending_heals.len(), 1);
assert_eq!(scanner.new_cache.info.pending_heals[0].object.as_deref(), Some("fresh"));
assert_eq!(scanner.update_cache.info.pending_heals, scanner.new_cache.info.pending_heals);
assert!(scanner.pending_heals_changed);
}
@@ -1699,7 +1690,6 @@ async fn test_pending_heal_update_keeps_stale_entry_until_retry_prune() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(u64::MAX, usize::MAX, &mut scanner, temp_dir);
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
let mut stale = pending_heal(PendingScannerHealKind::Object, "bucket", Some("object"), None, 1, 1);
stale.first_seen = FolderScanner::now_secs().saturating_sub(MAX_PENDING_SCANNER_HEAL_AGE_SECS + 1);
@@ -1717,7 +1707,6 @@ async fn test_pending_heal_update_keeps_stale_entry_until_retry_prune() {
assert_eq!(scanner.new_cache.info.pending_heals.len(), 1);
assert_eq!(scanner.new_cache.info.pending_heals[0].attempts, 1);
assert_eq!(scanner.new_cache.info.pending_heals[0].object.as_deref(), Some("object"));
assert_eq!(scanner.update_cache.info.pending_heals, scanner.new_cache.info.pending_heals);
}
#[tokio::test]
@@ -1725,7 +1714,6 @@ async fn test_pending_heal_queue_full_deduplicates_object_entry() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(u64::MAX, usize::MAX, &mut scanner, temp_dir);
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
scanner.update_pending_scanner_heal_after_admission(
PendingScannerHealKind::Object,
@@ -1751,7 +1739,6 @@ async fn test_pending_heal_queue_full_deduplicates_object_entry() {
assert_eq!(pending.attempts, 1);
assert_eq!(pending.last_admission_result, "dropped");
assert_eq!(pending.last_admission_reason, "queue_full");
assert_eq!(scanner.update_cache.info.pending_heals, scanner.new_cache.info.pending_heals);
assert!(scanner.pending_heals_changed);
}
@@ -1877,7 +1864,6 @@ async fn test_scan_folder_keeps_old_cache_bounded_by_loaded_entries() {
let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
scanner.is_erasure_mode = true;
for directory in 0..16 {
for object in 0..32 {
@@ -2297,7 +2283,6 @@ async fn test_scan_folder_skips_unreadable_child_directory() {
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
let folder = CachedFolder {
name: "bucket".to_string(),
@@ -2325,7 +2310,6 @@ async fn test_scan_folder_treats_child_removed_before_open_as_empty() {
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
let folder = CachedFolder {
name: "bucket/gone".to_string(),
@@ -2358,7 +2342,6 @@ async fn scan_folder_charges_each_enumerated_entry_to_the_disk_budget() {
scanner.sleeper = SCANNER_SLEEPER.for_disk(&temp_dir);
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
let folder = CachedFolder {
name: "bucket".to_string(),
@@ -2485,7 +2468,6 @@ async fn test_scan_folder_missing_bucket_root_still_reports_not_found() {
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
let folder = CachedFolder {
name: "bucket".to_string(),
@@ -2630,7 +2612,6 @@ async fn test_scan_folder_xl_meta_named_directory_uses_namespace_descent() {
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
scanner.is_erasure_mode = true;
let parent = CancellationToken::new();
@@ -2690,7 +2671,6 @@ async fn scan_corrupt_metadata_with_captured_logs(logs: &CapturedLogs) {
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
scanner.is_erasure_mode = true;
let parent = CancellationToken::new();
@@ -2841,7 +2821,6 @@ async fn test_scan_folder_missing_xl_meta_stops_erasure_data_dir_descent() {
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
scanner.is_erasure_mode = true;
scanner.sleeper = DynamicSleeper::new(rustfs_config::ScannerSpeed::Fastest).for_disk_with_budget_for_tests(
&temp_dir,
@@ -2934,8 +2913,9 @@ async fn test_scan_folder_uuid_namespace_part_name_directory_is_not_data_dir() {
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
scanner.is_erasure_mode = true;
let (sender, _receiver) = mpsc::channel(1);
scanner.usage_updates = Some(ScannerUsageUpdates::new(sender, "bucket".to_string()));
let namespace_hash = hash_path(&format!("bucket/{namespace_name}"));
scanner.old_cache.replace_hashed(
&namespace_hash,
@@ -2970,7 +2950,13 @@ async fn test_scan_folder_uuid_namespace_part_name_directory_is_not_data_dir() {
assert_eq!(budget.reason(), Some(crate::scanner_budget::ScannerCycleBudgetReason::Directories));
assert!(scanner.new_cache.info.failed_objects.is_empty());
assert!(
scanner.update_cache.find(&namespace_hash.key()).is_some(),
scanner
.usage_updates
.as_ref()
.expect("requested progress")
.cache
.find(&namespace_hash.key())
.is_some(),
"an existing UUID namespace must retain its cached subtree"
);
}
@@ -2991,7 +2977,6 @@ async fn test_scan_folder_non_erasure_metadata_keeps_namespace_descent() {
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
scanner.is_erasure_mode = false;
let parent = CancellationToken::new();
@@ -3032,7 +3017,6 @@ async fn scoped_root_scan_reuses_clean_top_level_entries_and_rescans_dirty_entri
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
scanner.old_cache.replace("bucket", "", DataUsageEntry::default());
scanner.old_cache.replace(
"bucket/clean",
@@ -3091,7 +3075,6 @@ async fn scan_hot_cold_segment_fixture(scoped: bool) -> (DataUsageEntry, Vec<Str
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
if scoped {
scanner.old_cache.replace("bucket", "", DataUsageEntry::default());
scanner.old_cache.replace(
@@ -3177,7 +3160,6 @@ async fn scoped_root_scan_preserves_erasure_health_walks() {
scanner.is_erasure_mode = true;
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
scanner.old_cache.replace("bucket", "", DataUsageEntry::default());
scanner.old_cache.replace(
"bucket/clean",
@@ -3221,11 +3203,9 @@ async fn test_scan_folder_compacted_parent_sends_partial_update() {
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
scanner.last_update = SystemTime::UNIX_EPOCH;
let (tx, mut rx) = mpsc::channel(1);
scanner.updates = Some(tx);
scanner.usage_updates = Some(ScannerUsageUpdates::new(tx, "bucket".to_string()));
let folder = CachedFolder {
name: "bucket".to_string(),
@@ -4226,7 +4206,6 @@ async fn test_scan_folder_ignores_symlinked_child_directory() {
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
let folder = CachedFolder {
name: "bucket".to_string(),
@@ -20,7 +20,9 @@ use crate::{DataUsageCacheSource, DataUsageScanPlanDigest};
use std::io::Cursor;
use tokio::io::AsyncReadExt;
mod completed_compaction;
mod deep_compacted;
mod memory_budget;
mod periodic_snapshot;
mod segment_observation;
@@ -783,7 +785,13 @@ async fn check_complete_sampling_resumption(resume_mode: HealScanMode) {
matches!(state, crate::scanner_io::DataUsageCacheScanState::Prepared { .. }),
"Normal complete is not Deep Current"
);
cache.prepare_bucket_checkpoint("bucket", 12, 7, SOURCE, PLAN, identity);
// The first sweep already compacts this prefix. Select its sampling
// cycle so the bounded scan exercises interruption rather than reuse.
let sampling_cycles = data_usage_update_dir_cycles();
let sampling_cycle = (12..12 + sampling_cycles)
.find(|cycle| hash_path("bucket/prefix").mod_(*cycle, sampling_cycles))
.expect("one prefix sampling cycle per rotation");
cache.prepare_bucket_checkpoint("bucket", u64::from(sampling_cycle), 7, SOURCE, PLAN, identity);
assert!(cache.info.scan_progress.is_none(), "complete unchanged baseline uses existing sampling");
let parent = CancellationToken::new();
let budget = ScannerCycleBudget::new_with_progress_tracking(
@@ -835,7 +843,7 @@ async fn check_complete_sampling_resumption(resume_mode: HealScanMode) {
.load_with_revisions(store.clone(), CACHE_NAME)
.await
.expect("reload partial each recovery round");
loaded.prepare_bucket_checkpoint("bucket", 12, 7, SOURCE, PLAN, resumed_identity);
loaded.prepare_bucket_checkpoint("bucket", u64::from(sampling_cycle), 7, SOURCE, PLAN, resumed_identity);
assert!(loaded.info.scan_progress.is_some(), "sampling partial must enter forward validation");
loaded.info.skip_healing = true;
let parent = CancellationToken::new();
@@ -0,0 +1,246 @@
// Copyright 2026 RustFS Team
// Licensed under the Apache License, Version 2.0.
use super::*;
#[tokio::test]
async fn completed_leaf_compaction_reuses_cached_storage() {
let (mut scanner, root) = build_test_scanner().await;
let _cleanup = TestGuard { temp_dir: Some(root) };
scanner.old_cache = bound_checkpoint().0;
let folder = CachedFolder {
name: "bucket/object".to_string(),
parent: Some(hash_path("bucket")),
object_heal_prob_div: 1,
};
let hash = hash_path(&folder.name);
for (coverage_gap, failed_objects, should_compact) in [(false, 0, true), (true, 0, false), (false, 1, false)] {
scanner.coverage_gap = coverage_gap;
scanner.new_cache = DataUsageCache {
info: scanner.old_cache.info.clone(),
..Default::default()
};
let mut target = String::with_capacity(4096);
target.push_str("arn:test:replica");
let allocation = target.as_ptr();
let capacity = target.capacity();
let mut leaf = DataUsageEntry {
size: 2048,
objects: 1,
versions: 3,
failed_objects,
replication_stats: Some(rustfs_data_usage::ReplicationAllStats {
targets: HashMap::from([(target, rustfs_data_usage::ReplicationTargetUsage::default())]),
..Default::default()
}),
all_tier_stats: Some(rustfs_data_usage::AllTierStats::new()),
..Default::default()
};
leaf.obj_sizes.add(2048);
leaf.obj_versions.add(3);
let accumulator = leaf.clone();
let mut expected = serde_json::to_value(&leaf).expect("leaf metadata");
expected["compacted"] = serde_json::json!(should_compact);
scanner.new_cache.replace(&folder.name, "bucket", leaf);
scanner.compact_completed_folder(&folder, &hash, &accumulator);
let cached = scanner.new_cache.find(&folder.name).expect("retained leaf");
assert_eq!(serde_json::to_value(cached).expect("compacted metadata"), expected);
let target = cached
.replication_stats
.as_ref()
.expect("retained replication statistics")
.targets
.keys()
.next()
.expect("retained replication target");
assert_eq!(target.capacity(), capacity, "leaf compaction must preserve owned allocations");
assert_eq!(target.as_ptr(), allocation);
assert!(!accumulator.compacted, "cache compaction must not change the caller's accounting mode");
}
}
async fn scan(
disk: &Arc<Disk>,
cache: DataUsageCache,
mode: HealScanMode,
max_objects: u64,
) -> (ScannerDiskScanOutcome, Arc<ScannerCycleBudget>) {
let budget = ScannerCycleBudget::new_with_progress_tracking(
&CancellationToken::new(),
ScannerCycleBudgetConfig {
max_objects: Some(max_objects),
..Default::default()
},
);
let outcome = disk
.clone()
.nsscanner_disk(budget.token(), budget.clone(), vec![disk.clone()], cache, None, scan_options(mode))
.await
.expect("scan checkpoint fixture through the real disk path");
(outcome, budget)
}
async fn assert_completed_compaction_resumes(mode: HealScanMode) {
let (scanner, root) = build_test_scanner().await;
let _cleanup = TestGuard {
temp_dir: Some(root.clone()),
};
for (prefix, size) in [("a", 2), ("b", 3), ("c", 5)] {
for object in 0..4 {
write_checkpoint_object(&root, &format!("nested/{prefix}/{object:04}"), &[(None, size)]).await;
}
}
let identity = crate::DataUsageScanIdentity {
scan_mode: mode,
tier_registry_generation: crate::runtime_tier_registry_for_cycle(11, 7).await.generation,
..bound_checkpoint().1
};
let mut cache = DataUsageCache::default();
cache.prepare_bucket_checkpoint("bucket", 11, 7, SOURCE, PLAN, identity);
cache.info.skip_healing = true;
let (outcome, budget) = scan(&scanner.local_disk, cache, mode, 6).await;
assert_eq!(budget.progress().0, 6);
assert_eq!(budget.reason(), Some(ScannerCycleBudgetReason::Objects));
let ScannerDiskScanOutcome::Partial(cache) = outcome else {
panic!("the object budget must interrupt the second prefix")
};
let completed = cache.find("bucket/nested/a").expect("completed prefix remains addressable");
assert!(completed.compacted, "a completed prefix must release leaves during the first sweep");
assert!(completed.children.is_empty());
assert_eq!((completed.objects, completed.size), (4, 8));
assert!(
cache.find("bucket/nested/a/0000").is_none(),
"completed leaves must not remain in the cache"
);
for prefix in ["bucket", "bucket/nested", "bucket/nested/b"] {
assert!(!cache.find(prefix).expect("unfinished ancestor retained").compacted);
}
assert!(
cache.find("bucket/nested/b/0000").is_some(),
"unfinished prefix retains its resume boundary"
);
let partial = cache
.checked_flatten_complete_scope("bucket")
.expect("partial graph has no orphan entries");
assert_eq!((partial.objects, partial.versions, partial.size), (6, 0, 14));
assert_eq!(cache.validated_scan_frontier(), Some("bucket/nested/b/0000"));
let store = FixtureStore::new();
let revisions = DataUsageCache::default()
.load_with_revisions(store.clone(), CACHE_NAME)
.await
.expect("initial fixture revisions");
cache
.save_with_revisions_for_epoch(store.clone(), CACHE_NAME, &revisions, 0)
.await
.expect("persist compacted partial cache through the production codec and CAS path");
let mut reloaded = store.strict_load().await;
assert_eq!(reloaded.info.scan_coverage_receipt, cache.info.scan_coverage_receipt);
assert_eq!(reloaded.validated_scan_frontier(), cache.validated_scan_frontier());
assert_eq!(
reloaded.prepare_bucket_checkpoint("bucket", 12, 8, SOURCE, PLAN, identity),
crate::DataUsageCachePrepareOutcome::Reused,
"a later cycle and leader must adopt compacted completed coverage"
);
let (outcome, budget) = scan(&scanner.local_disk, reloaded, mode, 13).await;
assert_eq!(
budget.progress().0,
7,
"only the interrupted boundary object and six unseen objects may be read on resume"
);
let ScannerDiskScanOutcome::Complete(completed) = outcome else {
panic!("resumed sweep must complete")
};
let totals = completed
.checked_flatten_complete_scope("bucket")
.expect("completed cache has no orphan entries");
assert_eq!((totals.objects, totals.versions, totals.size), (12, 0, 40));
assert!(completed.info.snapshot_complete);
assert!(completed.info.scan_progress.is_none());
assert!(!completed.has_scan_position());
assert!(completed.find("bucket/nested").expect("completed nested prefix").compacted);
assert_eq!(completed.cache.len(), 2, "only the bucket root and completed prefix aggregate remain");
}
#[tokio::test]
#[serial]
async fn completed_compaction_normal_resumes_without_revisiting_completed_prefix() {
assert_completed_compaction_resumes(HealScanMode::Normal).await;
}
#[tokio::test]
#[serial]
async fn completed_compaction_deep_resumes_without_revisiting_completed_prefix() {
assert_completed_compaction_resumes(HealScanMode::Deep).await;
}
#[tokio::test]
#[serial]
async fn completed_compaction_preserves_frontier_inside_coverage_gap() {
assert_compaction_frontier(true).await;
}
#[tokio::test]
#[serial]
async fn completed_compaction_promotes_frontier_without_parent_bookkeeping() {
assert_compaction_frontier(false).await;
}
async fn assert_compaction_frontier(coverage_gap: bool) {
let (mut scanner, root) = build_test_scanner().await;
let _cleanup = TestGuard {
temp_dir: Some(root.clone()),
};
for object in 0..3 {
write_checkpoint_object(&root, &format!("prefix/{object:04}"), &[(None, 2)]).await;
}
let identity = crate::DataUsageScanIdentity {
tier_registry_generation: crate::runtime_tier_registry_for_cycle(11, 7).await.generation,
..bound_checkpoint().1
};
scanner
.old_cache
.prepare_bucket_checkpoint("bucket", 11, 7, SOURCE, PLAN, identity);
scanner.new_cache.info = scanner.old_cache.info.clone();
scanner.is_erasure_mode = true;
scanner.skip_heal.store(true, Ordering::SeqCst);
scanner.sleeper = DynamicSleeper::new(rustfs_config::ScannerSpeed::Fastest);
// A failed child freezes the frontier. Without a gap, completing this
// subtree must preserve coverage even when the caller is a heal rewalk
// that does not advance the parent frontier after scan_folder returns.
scanner.coverage_gap = coverage_gap;
scanner.coverage_frontier = Some("bucket/prefix/0000".to_string());
let mut entry = DataUsageEntry::default();
scanner
.scan_folder(
CancellationToken::new(),
CachedFolder {
name: "bucket/prefix".to_string(),
parent: Some(hash_path("bucket")),
object_heal_prob_div: 1,
},
&mut entry,
)
.await
.expect("walk the ancestor of a frozen coverage frontier");
let expected_frontier = if coverage_gap { "bucket/prefix/0000" } else { "bucket/prefix" };
assert_eq!(scanner.coverage_frontier.as_deref(), Some(expected_frontier));
assert_eq!(
scanner.new_cache.find("bucket/prefix").expect("completed ancestor").compacted,
!coverage_gap
);
assert_eq!(scanner.new_cache.find("bucket/prefix/0000").is_some(), coverage_gap);
scanner
.new_cache
.seal_scan_frontier(scanner.coverage_frontier.as_deref())
.expect("seal the retained frontier");
assert_eq!(scanner.new_cache.validated_scan_frontier(), Some(expected_frontier));
let totals = scanner
.new_cache
.checked_flatten_complete_scope("bucket")
.expect("coverage-gap cache remains connected");
assert_eq!((totals.objects, totals.size), (3, 6));
}
@@ -72,18 +72,16 @@ impl CompactedFixture {
..bound_checkpoint().1
};
let mut cache = DataUsageCache::default();
// The first real scan builds coverage; the second same-plan scan takes
// the normal compaction path. No synthetic compacted cache is injected.
for cycle in [11, 12] {
cache.prepare_bucket_checkpoint("bucket", cycle, 7, SOURCE, PLAN, identity);
cache.info.skip_healing = true;
let (outcome, budget) = scan(&scanner.local_disk, cache, mode, OBJECTS + 1).await;
let ScannerDiskScanOutcome::Complete(completed) = outcome else {
panic!("fixture baseline must complete")
};
assert_eq!(budget.progress().0, OBJECTS, "baseline must read every metadata object");
cache = completed;
}
// Build and compact coverage in the first real sweep. No synthetic
// compacted cache is injected.
cache.prepare_bucket_checkpoint("bucket", 11, 7, SOURCE, PLAN, identity);
cache.info.skip_healing = true;
let (outcome, budget) = scan(&scanner.local_disk, cache, mode, OBJECTS + 1).await;
let ScannerDiskScanOutcome::Complete(completed) = outcome else {
panic!("fixture baseline must complete")
};
assert_eq!(budget.progress().0, OBJECTS, "baseline must read every metadata object");
cache = completed;
let store = FixtureStore::new();
cache = save_reload(&store, &cache).await;
assert!(cache.find(PREFIX).expect("baseline prefix").compacted);
@@ -263,9 +261,11 @@ async fn check_large_prefix_resumption(mode: HealScanMode) {
assert!(cache.info.scan_checkpoint.is_none());
assert!(cache.info.scan_coverage_receipt.is_none());
// Completing the sweep must restore compaction on the next same-plan scan.
// A completed prefix releases its children before the first sweep
// returns; a later same-plan scan keeps that compact representation.
let retained_entries = cache.cache.len();
assert!(!cache.find(PREFIX).expect("completed prefix").compacted);
assert!(cache.find(PREFIX).expect("completed prefix").compacted);
assert_eq!(retained_entries, 3, "only the bucket, compacted prefix and tail remain");
cache.prepare_bucket_checkpoint("bucket", 12 + round, 7 + round, SOURCE, PLAN, identity);
assert!(cache.info.scan_progress.is_none(), "a complete same-plan cache needs no forward sweep");
let (outcome, _) = scan(&scanner.local_disk, cache, mode, expected_reads + 1).await;
@@ -275,8 +275,8 @@ async fn check_large_prefix_resumption(mode: HealScanMode) {
let reloaded = save_reload(&store, &compacted).await;
assert!(reloaded.find(PREFIX).expect("compacted prefix").compacted);
assert!(
reloaded.cache.len() < retained_entries,
"completed sweeps must release child cache entries"
reloaded.cache.len() <= retained_entries,
"later sweeps must not retain the compacted prefix's children"
);
let total = reloaded.checked_flatten("bucket").expect("compacted bucket coverage");
assert_eq!((total.objects, total.size), (expected, expected));
@@ -0,0 +1,167 @@
// Copyright 2026 RustFS Team
// Licensed under the Apache License, Version 2.0.
use super::*;
const OBJECTS_PER_PREFIX: usize = 16;
const FROZEN_FRONTIER: &str = "bucket/prefix-0000/0000";
fn cache_shape(cache: &DataUsageCache, wire_bytes: usize) -> serde_json::Value {
let mut key_capacity_bytes = 0;
let mut child_links = 0;
let mut child_set_capacity = 0;
let mut child_key_capacity_bytes = 0;
for (key, entry) in &cache.cache {
key_capacity_bytes += key.capacity();
child_links += entry.children.len();
child_set_capacity += entry.children.capacity();
child_key_capacity_bytes += entry.children.iter().map(String::capacity).sum::<usize>();
}
// The canonical histograms own eleven and seven u64 counters. Map control
// bytes, allocator overhead, optional statistics and transient work are excluded.
let histogram_payload_lower_bound_bytes = cache.cache.len() * (11 + 7) * std::mem::size_of::<u64>();
let map_slot_size = std::mem::size_of::<(String, DataUsageEntry)>();
let retained_lower_bound_bytes = cache.cache.capacity() * map_slot_size
+ key_capacity_bytes
+ child_set_capacity * std::mem::size_of::<String>()
+ child_key_capacity_bytes
+ histogram_payload_lower_bound_bytes;
serde_json::json!({
"entries": cache.cache.len(),
"entry_capacity": cache.cache.capacity(),
"entry_size_bytes": std::mem::size_of::<DataUsageEntry>(),
"map_slot_size_bytes": map_slot_size,
"map_key_capacity_bytes": key_capacity_bytes,
"child_links": child_links,
"child_set_capacity": child_set_capacity,
"child_key_capacity_bytes": child_key_capacity_bytes,
"histogram_payload_lower_bound_bytes": histogram_payload_lower_bound_bytes,
"retained_lower_bound_bytes": retained_lower_bound_bytes,
"wire_bytes": wire_bytes,
})
}
async fn characterize_memory_budget(prefixes: usize, coverage_gap: bool) {
let (mut scanner, root) = build_test_scanner().await;
let _cleanup = TestGuard {
temp_dir: Some(root.clone()),
};
for prefix in 0..prefixes {
for object in 0..OBJECTS_PER_PREFIX {
write_checkpoint_object(&root, &format!("prefix-{prefix:04}/{object:04}"), &[(None, 1)]).await;
}
}
let identity = crate::DataUsageScanIdentity {
tier_registry_generation: crate::runtime_tier_registry_for_cycle(11, 7).await.generation,
..bound_checkpoint().1
};
scanner
.old_cache
.prepare_bucket_checkpoint("bucket", 11, 7, SOURCE, PLAN, identity);
scanner.new_cache.info = scanner.old_cache.info.clone();
scanner.is_erasure_mode = true;
scanner.skip_heal.store(true, Ordering::SeqCst);
scanner.sleeper = DynamicSleeper::new(rustfs_config::ScannerSpeed::Fastest);
// Isolate the already-established gap state from filesystem fault injection.
// Every object below is healthy; this control does not reproduce the fault
// or durable debt that caused a real scan's coverage gap.
scanner.coverage_gap = coverage_gap;
if coverage_gap {
scanner.coverage_frontier = Some(FROZEN_FRONTIER.to_string());
}
scanner
.scan_folder(
CancellationToken::new(),
CachedFolder {
name: "bucket".to_string(),
parent: None,
object_heal_prob_div: 1,
},
&mut DataUsageEntry::default(),
)
.await
.expect("walk the real wide-root metadata fixture");
let expected_objects = prefixes * OBJECTS_PER_PREFIX;
let totals = scanner
.new_cache
.checked_flatten_complete_scope("bucket")
.expect("the measured cache must be connected and contain no orphan entries");
assert_eq!((totals.objects, totals.size, totals.versions), (expected_objects, expected_objects, 0));
assert_eq!(totals.failed_objects, 0, "the gap control must not fabricate object failure debt");
assert!(
scanner.usage_updates.is_none(),
"production-style scans must not retain a progress mirror"
);
let bucket = scanner.new_cache.find("bucket").expect("retained root inventory");
assert!(!bucket.compacted, "these root fanouts preserve prefix statistics");
assert_eq!(bucket.children.len(), prefixes);
if coverage_gap {
assert_eq!(scanner.coverage_frontier.as_deref(), Some(FROZEN_FRONTIER));
assert!(
scanner.new_cache.find(FROZEN_FRONTIER).is_some(),
"the frozen endpoint must remain addressable"
);
} else {
for prefix in 0..prefixes {
let path = format!("bucket/prefix-{prefix:04}");
let entry = scanner.new_cache.find(&path).expect("completed root prefix");
assert!(entry.compacted);
assert!(entry.children.is_empty());
assert_eq!((entry.objects, entry.size), (OBJECTS_PER_PREFIX, OBJECTS_PER_PREFIX));
}
assert_eq!(scanner.new_cache.cache.len(), prefixes + 1, "healthy completed leaves must be released");
}
scanner
.new_cache
.seal_scan_frontier(scanner.coverage_frontier.as_deref())
.expect("seal the measured coverage endpoint");
assert!(scanner.new_cache.validated_scan_frontier().is_some());
let encoded = scanner
.new_cache
.marshal_msg()
.expect("encode the measured cache with the production codec");
let restored = DataUsageCache::unmarshal(&encoded).expect("reload the measured coverage graph");
assert_eq!(restored.validated_scan_frontier(), scanner.new_cache.validated_scan_frontier());
assert_eq!(restored.info.scan_coverage_receipt, scanner.new_cache.info.scan_coverage_receipt);
let restored_totals = restored
.checked_flatten_complete_scope("bucket")
.expect("reloaded accounting graph");
assert_eq!((restored_totals.objects, restored_totals.size), (expected_objects, expected_objects));
if coverage_gap {
assert_eq!(restored.validated_scan_frontier(), Some(FROZEN_FRONTIER));
}
println!(
"SCANNER_MEMORY_BUDGET {}",
serde_json::json!({
"schema": 1,
"measurement_point": "folder_walk_return_before_bucket_finalization",
"scenario": if coverage_gap { "seeded_coverage_gap" } else { "healthy_forward_sweep" },
"synthetic_gap_control": coverage_gap,
"prefixes": prefixes,
"objects_per_prefix": OBJECTS_PER_PREFIX,
"objects": expected_objects,
"object_bytes": expected_objects,
"validated_frontier": restored.validated_scan_frontier(),
"cache": cache_shape(&scanner.new_cache, encoded.len()),
})
);
}
#[tokio::test]
#[serial]
async fn memory_budget_characterizes_wide_root_and_seeded_coverage_gap() {
let prefixes: &[usize] = match std::env::var("RUSTFS_SCANNER_MEMORY_BUDGET_PROFILE") {
Err(std::env::VarError::NotPresent) => &[64],
Ok(value) if value == "1" => &[64, 128, 256],
_ => panic!("RUSTFS_SCANNER_MEMORY_BUDGET_PROFILE must be absent or 1"),
};
for &prefixes in prefixes {
assert!(prefixes * (OBJECTS_PER_PREFIX + 1) < DATA_SCANNER_COMPACT_AT_CHILDREN);
for coverage_gap in [false, true] {
characterize_memory_budget(prefixes, coverage_gap).await;
}
}
}
@@ -30,10 +30,6 @@ fn prepare_scanner(scanner: &mut FolderScanner, cache: DataUsageCache) {
info: cache.info.clone(),
..Default::default()
};
scanner.update_cache = DataUsageCache {
info: cache.info.clone(),
..Default::default()
};
scanner.old_cache = cache;
scanner.is_erasure_mode = true;
scanner.skip_heal.store(true, Ordering::SeqCst);
@@ -250,7 +250,6 @@ async fn walk_and_save(observe: bool) -> (Vec<String>, serde_json::Value) {
assert_ne!(before, after, "a changed key requires an observable successful fixture write");
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.update_cache.info.name = "bucket".to_string();
let paths = Arc::new(Mutex::new(Vec::<String>::new()));
let proposed_walked = Arc::new(Mutex::new(BTreeSet::<String>::new()));
scanner.update_current_path = Arc::new({
@@ -148,11 +148,7 @@ async fn erasure_candidate_finds_late_part_without_scanning_data_children() {
async fn candidate_scanner() -> (FolderScanner, std::path::PathBuf, CachedFolder) {
let (mut scanner, root) = build_test_scanner().await;
for info in [
&mut scanner.old_cache.info,
&mut scanner.new_cache.info,
&mut scanner.update_cache.info,
] {
for info in [&mut scanner.old_cache.info, &mut scanner.new_cache.info] {
info.name = "bucket".into();
}
scanner.is_erasure_mode = true;
@@ -354,8 +350,7 @@ fn enable_checkpoint(scanner: &mut FolderScanner) {
cache.prepare_bucket_checkpoint("bucket", 7, 3, crate::DataUsageCacheSource::new(1, 0), plan, identity);
cache.replace("bucket", "", DataUsageEntry::default());
scanner.old_cache = cache.clone();
scanner.new_cache = cache.clone();
scanner.update_cache = cache;
scanner.new_cache = cache;
}
#[tokio::test]
@@ -462,8 +457,7 @@ async fn erasure_candidate_cancelled_batch_keeps_partial_checkpoint_and_reclassi
let bytes = serde_json::to_vec(&snapshot).expect("serialize partial checkpoint");
let reloaded: DataUsageCache = serde_json::from_slice(&bytes).expect("reload partial checkpoint");
scanner.old_cache = reloaded.clone();
scanner.new_cache = reloaded.clone();
scanner.update_cache = reloaded;
scanner.new_cache = reloaded;
scanner.raw_enumeration_progress.clear();
let next_budget = ScannerCycleBudget::new_with_progress_tracking(&ctx, Default::default());
scanner.budget = next_budget.clone();
@@ -110,7 +110,7 @@ fn mrf_ownership_retry_due_boundaries_and_priority_are_bounded() {
}
#[tokio::test]
async fn mrf_ownership_full_hint_table_has_bounded_multicycle_work_and_sync() {
async fn mrf_ownership_full_hint_table_has_bounded_multicycle_work() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(u64::MAX, usize::MAX, &mut scanner, temp_dir);
let base = 1_700_000_000;
@@ -131,39 +131,23 @@ async fn mrf_ownership_full_hint_table_has_bounded_multicycle_work_and_sync() {
.enumerate()
.map(|(index, entry)| (entry.object.clone().expect("object identity"), index))
.collect();
scanner.sync_pending_heals();
let initial_syncs = scanner.pending_heal_sync_count;
let mut requests = 0usize;
let mut nonempty_batches = 0;
for minute in 0..24 * 60 {
let now = base + minute * 60;
let candidates = pending_scanner_heal_retry_candidates_at(&scanner.new_cache.info.pending_heals, "bucket", now);
assert!(candidates.len() <= MAX_PENDING_SCANNER_HEAL_RETRIES_PER_BUCKET);
if candidates.is_empty() {
continue;
for candidate in candidates {
assert_eq!(
build_pending_scanner_heal_request(&candidate)
.expect("retry request")
.priority,
HealChannelPriority::Low
);
let index = indices[candidate.object.as_ref().expect("object identity")];
record_pending_heal_attempt(&mut scanner.new_cache.info.pending_heals[index], now);
observe_pending_heal_admission(&mut scanner.new_cache.info.pending_heals[index], HealAdmissionResult::Accepted);
requests += 1;
}
nonempty_batches += 1;
let before = scanner.pending_heal_sync_count;
{
let batch = PendingHealSyncBatch::new(&mut scanner);
for candidate in candidates {
assert_eq!(
build_pending_scanner_heal_request(&candidate)
.expect("retry request")
.priority,
HealChannelPriority::Low
);
let index = indices[candidate.object.as_ref().expect("object identity")];
record_pending_heal_attempt(&mut batch.scanner.new_cache.info.pending_heals[index], now);
observe_pending_heal_admission(
&mut batch.scanner.new_cache.info.pending_heals[index],
HealAdmissionResult::Accepted,
);
batch.scanner.sync_pending_heals();
requests += 1;
}
}
assert_eq!(scanner.pending_heal_sync_count, before + 1, "one table clone per changed retry batch");
assert_eq!(scanner.new_cache.info.pending_heals.len(), MAX_PENDING_SCANNER_HEALS_PER_BUCKET);
}
assert!(requests >= MAX_PENDING_SCANNER_HEALS_PER_BUCKET, "every retained hint receives a retry");
@@ -172,12 +156,13 @@ async fn mrf_ownership_full_hint_table_has_bounded_multicycle_work_and_sync() {
"15min..6h backoff bounds repeated work within 24h"
);
assert!(scanner.new_cache.info.pending_heals.iter().all(|entry| entry.attempts >= 2));
assert_eq!(scanner.pending_heal_sync_count - initial_syncs, nonempty_batches);
assert_eq!(scanner.update_cache.info.pending_heals, scanner.new_cache.info.pending_heals);
let restored =
DataUsageCache::unmarshal(&scanner.new_cache.marshal_msg().expect("encode retry ledger")).expect("reload retry ledger");
assert_eq!(restored.info.pending_heals, scanner.new_cache.info.pending_heals);
}
#[tokio::test]
async fn mrf_ownership_cancelled_batch_restores_sync_without_per_item_clones() {
async fn mrf_ownership_cancelled_retry_retains_canonical_ledger() {
let (mut scanner, temp_dir) = build_test_scanner().await;
let _guard = TestGuard::new(u64::MAX, usize::MAX, &mut scanner, temp_dir);
scanner
@@ -185,34 +170,45 @@ async fn mrf_ownership_cancelled_batch_restores_sync_without_per_item_clones() {
.info
.pending_heals
.push(pending_heal(PendingScannerHealKind::Object, "bucket", Some("object"), None, 1, 1));
let before = scanner.pending_heal_sync_count;
let mut work = Box::pin(async {
let batch = PendingHealSyncBatch::new(&mut scanner);
record_pending_heal_attempt(&mut batch.scanner.new_cache.info.pending_heals[0], 100);
batch.scanner.sync_pending_heals();
batch.scanner.sync_pending_heals();
record_pending_heal_attempt(&mut scanner.new_cache.info.pending_heals[0], 100);
scanner.update_pending_scanner_heal_after_admission(
PendingScannerHealKind::Object,
"bucket",
Some("object"),
None,
HealScanMode::Normal,
HealAdmissionResult::Accepted,
);
std::future::pending::<()>().await;
});
assert!(futures::poll!(&mut work).is_pending());
drop(work);
assert!(!scanner.pending_heal_sync_deferred);
assert!(!scanner.pending_heal_batch_dirty);
assert_eq!(scanner.pending_heal_sync_count, before + 1);
assert!(scanner.pending_heals_changed, "cancellation must retain the dirty marker");
assert_eq!(scanner.new_cache.info.pending_heals[0].attempts, 2);
assert!(pending_scanner_heal_retry_candidates_at(&scanner.new_cache.info.pending_heals, "bucket", 101).is_empty());
assert_eq!(scanner.update_cache.info.pending_heals, scanner.new_cache.info.pending_heals);
let restored = DataUsageCache::unmarshal(&scanner.new_cache.marshal_msg().expect("encode cancelled retry"))
.expect("reload cancelled retry");
assert_eq!(restored.info.pending_heals, scanner.new_cache.info.pending_heals);
scanner.pending_heals_changed = false;
let result: std::result::Result<(), &'static str> = async {
let batch = PendingHealSyncBatch::new(&mut scanner);
record_pending_heal_attempt(&mut batch.scanner.new_cache.info.pending_heals[0], 200);
observe_pending_heal_admission(&mut batch.scanner.new_cache.info.pending_heals[0], HealAdmissionResult::Merged);
batch.scanner.sync_pending_heals();
Err("injected retry batch failure")
record_pending_heal_attempt(&mut scanner.new_cache.info.pending_heals[0], 200);
scanner.update_pending_scanner_heal_after_admission(
PendingScannerHealKind::Object,
"bucket",
Some("object"),
None,
HealScanMode::Normal,
HealAdmissionResult::Merged,
);
Err("injected retry failure")
}
.await;
assert!(result.is_err());
assert_eq!(scanner.pending_heal_sync_count, before + 2);
assert!(!scanner.pending_heal_sync_deferred);
assert_eq!(scanner.update_cache.info.pending_heals, scanner.new_cache.info.pending_heals);
assert!(scanner.pending_heals_changed, "an error must retain the dirty marker");
assert_eq!(scanner.new_cache.info.pending_heals[0].attempts, 3);
assert_eq!(scanner.new_cache.info.pending_heals[0].last_admission_result, "merged");
}
#[derive(Default)]
@@ -436,7 +432,6 @@ async fn mrf_ownership_manager_completion_preserves_scanner_pending() {
let _guard = TestGuard::new(u64::MAX, usize::MAX, &mut scanner, temp_dir);
let bucket = format!("mrf-ownership-{}", Uuid::new_v4());
scanner.new_cache.info.name = bucket.clone();
scanner.update_cache.info.name = bucket.clone();
scanner.heal_object_select = 1;
let storage = Arc::new(NoticeStorage {
bucket_incarnation_id: Uuid::new_v4(),
@@ -521,12 +516,12 @@ async fn mrf_ownership_manager_completion_preserves_scanner_pending() {
"the second object call remains blocked without a duplicate execution"
);
note_mrf_repaired(&bucket, object, Some(*version.as_bytes()));
let syncs_before_retry = scanner.pending_heal_sync_count;
scanner.pending_heals_changed = false;
scanner
.retry_pending_scanner_heals()
.await
.expect("real scanner ledger retry");
assert_eq!(scanner.pending_heal_sync_count, syncs_before_retry + 1, "the real retry batch syncs once");
assert!(scanner.pending_heals_changed, "the real retry marks its canonical ledger for persistence");
assert_eq!(
scanner.new_cache.info.pending_heals.len(),
index + 1,
@@ -0,0 +1,121 @@
// Copyright 2026 RustFS Team
// Licensed under the Apache License, Version 2.0.
use super::*;
#[tokio::test]
#[serial]
async fn usage_updates_preserve_totals_without_allocating_an_unrequested_cache() {
for observed in [false, true] {
let (mut scanner, root) = build_test_scanner().await;
let _guard = TestGuard {
temp_dir: Some(root.clone()),
};
scanner.old_cache.info.name = "bucket".to_string();
scanner.new_cache.info.name = "bucket".to_string();
scanner.sleeper = DynamicSleeper::new(rustfs_config::ScannerSpeed::Fastest);
scanner.skip_heal.store(true, Ordering::SeqCst);
for prefix in ["existing", "new"] {
for index in 0..8 {
let object = format!("{prefix}/{index:04}");
write_test_object_metadata_bytes(&root, "bucket", &object, &metadata_for_object_version("bucket", &object, None))
.await;
}
}
scanner.old_cache.replace("bucket", "", DataUsageEntry::default());
scanner
.old_cache
.replace("bucket/existing", "bucket", DataUsageEntry::default());
scanner.old_cache.replace(
"bucket/existing/0000",
"bucket/existing",
DataUsageEntry {
objects: 1,
size: 99,
..Default::default()
},
);
let (sender, mut receiver) = mpsc::channel(2);
if observed {
scanner.usage_updates = Some(ScannerUsageUpdates::new(sender, "bucket".to_string()));
}
scanner
.scan_folder(
CancellationToken::new(),
CachedFolder {
name: "bucket".to_string(),
parent: None,
object_heal_prob_div: 1,
},
&mut DataUsageEntry::default(),
)
.await
.expect("scan existing and new prefixes");
let totals = scanner
.new_cache
.checked_flatten_complete_scope("bucket")
.expect("complete accounting");
assert_eq!((totals.objects, totals.size), (16, 16));
assert_eq!(
scanner
.old_cache
.find("bucket/existing/0000")
.expect("retained baseline")
.size,
99
);
if observed {
while receiver.try_recv().is_ok() {}
let updates = scanner.usage_updates.as_mut().expect("requested progress owner");
let progress = updates.cache.checked_flatten("bucket").expect("progress cache is connected");
assert_eq!((progress.objects, progress.size), (16, 16));
updates.last_update = SystemTime::UNIX_EPOCH;
scanner.send_update().await;
let update = receiver.try_recv().expect("requested aggregate update");
assert_eq!((update.objects, update.size), (16, 16));
scanner.send_update().await;
assert!(receiver.try_recv().is_err(), "unchanged minute interval must not emit another aggregate");
} else {
assert!(scanner.usage_updates.is_none(), "unobserved scans must never allocate a progress cache");
}
}
}
#[tokio::test]
async fn usage_updates_do_not_mirror_durable_ledgers_or_checkpoint_metadata() {
let (mut scanner, root) = build_test_scanner().await;
let _guard = TestGuard { temp_dir: Some(root) };
scanner.new_cache.info.name = "bucket".to_string();
let (sender, _receiver) = mpsc::channel(1);
scanner.usage_updates = Some(ScannerUsageUpdates::new(sender, "bucket".to_string()));
scanner.record_pending_scanner_heal(
PendingScannerHealKind::Object,
"bucket",
Some("object"),
None,
HealScanMode::Normal,
HealAdmissionResult::Full,
);
scanner.record_scan_resume_hint("bucket/object");
let mut summary = SizeSummary::default();
summary.record_size_reconciliation(SizeReconciliationEntry {
key: "unresolved-size".to_string(),
bucket: "bucket".to_string(),
object: "object".to_string(),
reason: "invalid_declared_size".to_string(),
..Default::default()
});
scanner.apply_size_reconciliation(&summary);
scanner.finish_size_reconciliation_batch();
let restored = DataUsageCache::unmarshal(&scanner.new_cache.marshal_msg().expect("encode canonical state"))
.expect("reload canonical state");
assert_eq!(restored.info.pending_heals.len(), 1);
assert_eq!(restored.info.size_reconciliation.len(), 1);
assert_eq!(restored.info.scan_checkpoint, scanner.new_cache.info.scan_checkpoint);
let progress = &scanner.usage_updates.as_ref().expect("requested progress owner").cache;
assert_eq!(progress.info.name, "bucket");
assert!(progress.info.pending_heals.is_empty());
assert!(progress.info.size_reconciliation.is_empty());
assert!(progress.info.scan_checkpoint.is_none());
assert!(progress.info.scan_resume_after.is_none());
}
+105 -13
View File
@@ -1098,6 +1098,24 @@ pub(super) async fn send_cache_root_entry(
.map_err(|err| ScannerError::Other(format!("scanner cache root channel closed: {err}")))
}
fn log_missing_set_snapshot_metadata(store: &SetDisks, info: &DataUsageCacheInfo, reason: &'static str) {
warn!(
target: "rustfs::scanner::io",
event = EVENT_SCANNER_CACHE_PERSIST_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_IO,
state = "snapshot_metadata_missing",
reason,
store_pool = store.pool_index,
store_set = store.set_index,
snapshot_source_pool = info.source.map(|source| source.pool_index),
snapshot_source_set = info.source.map(|source| source.set_index),
snapshot_cycle = info.next_cycle,
snapshot_leader_epoch = info.leader_epoch,
"Scanner set snapshot is missing required metadata"
);
}
pub(super) async fn persist_and_publish_cache_snapshot(
store: Arc<SetDisks>,
updates: &mpsc::Sender<DataUsageCache>,
@@ -1106,9 +1124,20 @@ pub(super) async fn persist_and_publish_cache_snapshot(
cache_cycle_floor: &AtomicU64,
expected_publication_epoch: u64,
) -> Option<SystemTime> {
let source = cache_snapshot.info.source?;
let coverage_digest = cache_snapshot.info.scan_coverage_digest?;
let execution_digest = cache_snapshot.info.scan_execution_digest?;
let snapshot_cycle = cache_snapshot.info.next_cycle;
let snapshot_leader_epoch = cache_snapshot.info.leader_epoch;
let Some(source) = cache_snapshot.info.source else {
log_missing_set_snapshot_metadata(&store, &cache_snapshot.info, "source");
return None;
};
let Some(coverage_digest) = cache_snapshot.info.scan_coverage_digest else {
log_missing_set_snapshot_metadata(&store, &cache_snapshot.info, "coverage_digest");
return None;
};
let Some(execution_digest) = cache_snapshot.info.scan_execution_digest else {
log_missing_set_snapshot_metadata(&store, &cache_snapshot.info, "execution_digest");
return None;
};
let guard = match acquire_scanner_cache_locks(store.as_ref(), DATA_USAGE_CACHE_NAME, source).await {
Ok(guard) => guard,
Err(err) => {
@@ -1117,8 +1146,14 @@ pub(super) async fn persist_and_publish_cache_snapshot(
event = EVENT_SCANNER_CACHE_PERSIST_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_IO,
cache_name = DATA_USAGE_CACHE_NAME,
state = err.state(),
store_pool = store.pool_index,
store_set = store.set_index,
snapshot_source_pool = source.pool_index,
snapshot_source_set = source.set_index,
snapshot_cycle,
snapshot_leader_epoch,
cache_name = DATA_USAGE_CACHE_NAME,
error = %err,
"Scanner cache snapshot lock acquisition failed"
);
@@ -1135,15 +1170,24 @@ pub(super) async fn persist_and_publish_cache_snapshot(
event = EVENT_SCANNER_CACHE_PERSIST_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_IO,
cache_name = DATA_USAGE_CACHE_NAME,
state = "load_or_revision_lookup_failed",
store_pool = store.pool_index,
store_set = store.set_index,
snapshot_source_pool = source.pool_index,
snapshot_source_set = source.set_index,
snapshot_cycle,
snapshot_leader_epoch,
cache_name = DATA_USAGE_CACHE_NAME,
error = %err,
"Scanner cache snapshot load or revision lookup failed"
);
return None;
}
};
let scan_plan_digest = cache_snapshot.info.scan_plan_digest?;
let Some(scan_plan_digest) = cache_snapshot.info.scan_plan_digest else {
log_missing_set_snapshot_metadata(&store, &cache_snapshot.info, "plan_digest");
return None;
};
if persisted.info.next_cycle > cache_snapshot.info.next_cycle {
cache_cycle_floor.fetch_max(persisted.info.next_cycle, Ordering::AcqRel);
warn!(
@@ -1151,10 +1195,16 @@ pub(super) async fn persist_and_publish_cache_snapshot(
event = EVENT_SCANNER_CACHE_PERSIST_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_IO,
state = "stale_cycle_rejected",
store_pool = store.pool_index,
store_set = store.set_index,
snapshot_source_pool = source.pool_index,
snapshot_source_set = source.set_index,
snapshot_cycle,
snapshot_leader_epoch,
cache_name = DATA_USAGE_CACHE_NAME,
requested_cycle = cache_snapshot.info.next_cycle,
persisted_cycle = persisted.info.next_cycle,
state = "stale_cycle_rejected",
"Scanner rejected a set cache cycle regression"
);
return None;
@@ -1165,10 +1215,16 @@ pub(super) async fn persist_and_publish_cache_snapshot(
event = EVENT_SCANNER_CACHE_PERSIST_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_IO,
state = "stale_leader_rejected",
store_pool = store.pool_index,
store_set = store.set_index,
snapshot_source_pool = source.pool_index,
snapshot_source_set = source.set_index,
snapshot_cycle,
snapshot_leader_epoch,
cache_name = DATA_USAGE_CACHE_NAME,
requested_epoch = cache_snapshot.info.leader_epoch,
persisted_epoch = persisted.info.leader_epoch,
state = "stale_leader_rejected",
"Scanner rejected a set cache snapshot from an older leader epoch"
);
return None;
@@ -1199,6 +1255,12 @@ pub(super) async fn persist_and_publish_cache_snapshot(
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_IO,
state = "scan_baseline_revision_changed",
store_pool = store.pool_index,
store_set = store.set_index,
snapshot_source_pool = source.pool_index,
snapshot_source_set = source.set_index,
snapshot_cycle,
snapshot_leader_epoch,
cache_name = DATA_USAGE_CACHE_NAME,
"Scanner skipped set snapshot without an unchanged baseline revision"
);
@@ -1210,8 +1272,14 @@ pub(super) async fn persist_and_publish_cache_snapshot(
event = EVENT_SCANNER_CACHE_PERSIST_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_IO,
cache_name = DATA_USAGE_CACHE_NAME,
state = "lock_lost",
store_pool = store.pool_index,
store_set = store.set_index,
snapshot_source_pool = source.pool_index,
snapshot_source_set = source.set_index,
snapshot_cycle,
snapshot_leader_epoch,
cache_name = DATA_USAGE_CACHE_NAME,
"Scanner cache snapshot save skipped after lock loss"
);
return None;
@@ -1227,8 +1295,14 @@ pub(super) async fn persist_and_publish_cache_snapshot(
event = EVENT_SCANNER_CACHE_PERSIST_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_IO,
cache_name = DATA_USAGE_CACHE_NAME,
state = "save_failed",
store_pool = store.pool_index,
store_set = store.set_index,
snapshot_source_pool = source.pool_index,
snapshot_source_set = source.set_index,
snapshot_cycle,
snapshot_leader_epoch,
cache_name = DATA_USAGE_CACHE_NAME,
error = %e,
"Scanner cache snapshot persistence failed"
);
@@ -1243,8 +1317,14 @@ pub(super) async fn persist_and_publish_cache_snapshot(
event = EVENT_SCANNER_CACHE_PERSIST_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_IO,
cache_name = DATA_USAGE_CACHE_NAME,
state = "lock_lost_after_save",
store_pool = store.pool_index,
store_set = store.set_index,
snapshot_source_pool = source.pool_index,
snapshot_source_set = source.set_index,
snapshot_cycle,
snapshot_leader_epoch,
cache_name = DATA_USAGE_CACHE_NAME,
"Scanner cache snapshot publish skipped after lock loss"
);
return None;
@@ -1261,8 +1341,14 @@ pub(super) async fn persist_and_publish_cache_snapshot(
event = EVENT_SCANNER_CACHE_PERSIST_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_IO,
cache_name = DATA_USAGE_CACHE_NAME,
state = "publication_epoch_changed_before_publish",
store_pool = store.pool_index,
store_set = store.set_index,
snapshot_source_pool = source.pool_index,
snapshot_source_set = source.set_index,
snapshot_cycle,
snapshot_leader_epoch,
cache_name = DATA_USAGE_CACHE_NAME,
"Scanner cache root publish skipped after movement epoch change"
);
return None;
@@ -1276,8 +1362,14 @@ pub(super) async fn persist_and_publish_cache_snapshot(
event = EVENT_SCANNER_CACHE_PERSIST_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_IO,
cache_name = DATA_USAGE_CACHE_NAME,
state = "publish_failed",
store_pool = store.pool_index,
store_set = store.set_index,
snapshot_source_pool = source.pool_index,
snapshot_source_set = source.set_index,
snapshot_cycle,
snapshot_leader_epoch,
cache_name = DATA_USAGE_CACHE_NAME,
error = %e,
"Scanner cache snapshot publish failed"
);
+28 -1
View File
@@ -21,15 +21,41 @@ pub(super) async fn collect_completed_bucket_results(
) -> (DataUsageCache, HashSet<String>) {
let mut cancelled = false;
let mut completed = HashSet::new();
let mut received_roots = 0usize;
let mut duplicate_applied_roots = 0usize;
let mut rejected_roots = 0usize;
loop {
tokio::select! {
_ = ctx.cancelled(), if !cancelled => cancelled = true,
result = results.recv() => {
let Some(result) = result else {
debug!(
target: "rustfs::scanner::io",
event = EVENT_SCANNER_SET_STATE,
component = LOG_COMPONENT_SCANNER,
subsystem = LOG_SUBSYSTEM_IO,
state = "bucket_root_collector_drained",
collector_cache_pool = cache.info.source.map(|source| source.pool_index),
collector_cache_set = cache.info.source.map(|source| source.set_index),
collector_cache_cycle = cache.info.next_cycle,
collector_cache_leader_epoch = cache.info.leader_epoch,
received_roots,
distinct_applied_roots = completed.len(),
duplicate_applied_roots,
rejected_roots,
cancellation_observed = cancelled,
cancelled_at_drain = ctx.is_cancelled(),
"Scanner bucket root collector drained"
);
return (cache, completed);
};
received_roots = received_roots.saturating_add(1);
if let Some(bucket) = apply_bucket_result_to_cache(&mut cache, result, SystemTime::now()) {
completed.insert(bucket);
if !completed.insert(bucket) {
duplicate_applied_roots = duplicate_applied_roots.saturating_add(1);
}
} else {
rejected_roots = rejected_roots.saturating_add(1);
}
}
}
@@ -1807,6 +1833,7 @@ impl ScannerIOCache for SetDisks {
pool = source.pool_index,
set = source.set_index,
cycle = want_cycle,
leader_epoch,
completed_buckets = completed_count,
carried_buckets = carried_buckets.len(),
total_buckets = buckets.len(),
+141
View File
@@ -2211,6 +2211,147 @@ fn test_bucket_incarnations(buckets: &[&str]) -> HashMap<String, Uuid> {
.collect()
}
#[derive(Clone)]
struct SetCacheLogWriter(Arc<StdMutex<Vec<u8>>>);
impl std::io::Write for SetCacheLogWriter {
fn write(&mut self, bytes: &[u8]) -> std::io::Result<usize> {
self.0.lock().expect("set cache log buffer").extend_from_slice(bytes);
Ok(bytes.len())
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
async fn capture_set_cache_logs<F: std::future::Future>(future: F) -> (F::Output, Vec<serde_json::Value>) {
use tracing::instrument::WithSubscriber as _;
let buffer = Arc::new(StdMutex::new(Vec::new()));
let writer = Arc::clone(&buffer);
let subscriber = tracing_subscriber::fmt()
.json()
.without_time()
.with_ansi(false)
.with_max_level(tracing::Level::WARN)
.with_writer(move || SetCacheLogWriter(Arc::clone(&writer)))
.finish();
// Avoid a cached Never interest when other tests use an unsubscribed dispatcher.
let _interest_guard = tracing::Dispatch::new(tracing::subscriber::NoSubscriber::default());
let result = future.with_subscriber(subscriber).await;
let logs = buffer.lock().expect("set cache log buffer");
let events = std::str::from_utf8(&logs)
.expect("set cache logs should be UTF-8")
.lines()
.map(|line| serde_json::from_str(line).expect("set cache log should be JSON"))
.collect();
(result, events)
}
fn set_cache_event<'a>(events: &'a [serde_json::Value], state: &str) -> &'a serde_json::Value {
let matching = events
.iter()
.filter(|event| event["fields"]["state"] == state)
.collect::<Vec<_>>();
assert_eq!(matching.len(), 1, "expected one {state} event: {events:?}");
&matching[0]["fields"]
}
fn assert_set_cache_context(fields: &serde_json::Value, source: Option<(usize, usize)>) {
assert_eq!(fields["store_pool"], 0);
assert_eq!(fields["store_set"], 0);
assert_eq!(fields["snapshot_cycle"], 7);
assert_eq!(fields["snapshot_leader_epoch"], 11);
if let Some((pool, set)) = source {
assert_eq!(fields["snapshot_source_pool"], pool);
assert_eq!(fields["snapshot_source_set"], set);
} else {
assert!(fields.get("snapshot_source_pool").is_none());
assert!(fields.get("snapshot_source_set").is_none());
}
}
async fn set_cache_log_fixture() -> (tempfile::TempDir, Arc<ECStore>, DataUsageCache, u64) {
let (temp_dir, store) = setup_two_pool_scanner_store().await;
let set = Arc::clone(&store.pools[0].disk_set[0]);
let epoch = scanner_publication_epoch(Arc::clone(&set)).await.expect("idle set admission");
let mut cache = complete_set_usage_cache(&[("photos", 5)], DataUsageScanPlanDigest([1; 32]));
cache.info.source = Some(DataUsageCacheSource::new(0, 0));
cache.info.scan_execution_digest = Some(DataUsageScanPlanDigest([2; 32]));
cache.save(set, DATA_USAGE_CACHE_NAME).await.expect("seed set cache");
(temp_dir, store, cache, epoch)
}
#[tokio::test]
#[serial]
async fn set_cache_missing_metadata_logs_receiver_and_snapshot() {
let (_temp_dir, store, cache, epoch) = set_cache_log_fixture().await;
let set = Arc::clone(&store.pools[0].disk_set[0]);
let cycle_floor = AtomicU64::new(cache.info.next_cycle);
let (tx, _rx) = mpsc::channel(1);
for missing in ["source", "coverage_digest", "execution_digest", "plan_digest"] {
let mut candidate = cache.clone();
match missing {
"source" => candidate.info.source = None,
"coverage_digest" => candidate.info.scan_coverage_digest = None,
"execution_digest" => {
candidate.info.source = Some(DataUsageCacheSource::new(1, 2));
candidate.info.scan_execution_digest = None;
}
"plan_digest" => candidate.info.scan_plan_digest = None,
_ => unreachable!("fixture only uses the four checked metadata fields"),
}
let source = candidate.info.source.map(|source| (source.pool_index, source.set_index));
let (result, events) = capture_set_cache_logs(persist_and_publish_cache_snapshot(
Arc::clone(&set),
&tx,
candidate,
None,
&cycle_floor,
epoch,
))
.await;
assert!(result.is_none(), "missing {missing} should retain the existing rejection");
let fields = set_cache_event(&events, "snapshot_metadata_missing");
assert_eq!(fields["reason"], missing);
assert_set_cache_context(fields, source);
}
}
#[tokio::test]
#[serial]
async fn set_cache_revision_rejection_logs_receiver_and_snapshot() {
let (_temp_dir, store, mut cache, epoch) = set_cache_log_fixture().await;
let set = Arc::clone(&store.pools[0].disk_set[0]);
cache.info.scan_execution_digest = Some(DataUsageScanPlanDigest([3; 32]));
let cycle_floor = AtomicU64::new(cache.info.next_cycle);
let (tx, mut rx) = mpsc::channel(1);
let (result, events) =
capture_set_cache_logs(persist_and_publish_cache_snapshot(set, &tx, cache, None, &cycle_floor, epoch)).await;
assert!(result.is_none());
assert!(matches!(rx.try_recv(), Err(mpsc::error::TryRecvError::Empty)));
assert_set_cache_context(set_cache_event(&events, "scan_baseline_revision_changed"), Some((0, 0)));
}
#[tokio::test]
#[serial]
async fn set_cache_send_failure_logs_receiver_and_snapshot_without_changing_return() {
let (_temp_dir, store, cache, epoch) = set_cache_log_fixture().await;
let set = Arc::clone(&store.pools[0].disk_set[0]);
let cycle_floor = AtomicU64::new(cache.info.next_cycle);
let (tx, rx) = mpsc::channel(1);
drop(rx);
let (result, events) =
capture_set_cache_logs(persist_and_publish_cache_snapshot(set, &tx, cache, None, &cycle_floor, epoch)).await;
assert!(
result.is_some(),
"the existing return value must not be treated as receiver acknowledgement"
);
assert_set_cache_context(set_cache_event(&events, "publish_failed"), Some((0, 0)));
}
#[tokio::test]
#[serial]
async fn set_snapshot_reuse_requires_execution_identity_and_fences_stale_writers() {
@@ -354,3 +354,202 @@ async fn collector_worker_panic_waits_for_admitted_resume_and_preserves_checkpoi
async fn collector_parent_cancel_drains_accepted_root_and_preserves_checkpoint() {
run_collector_lifecycle(Stop::ParentCancellation).await;
}
#[derive(Clone)]
struct CollectorLogWriter(Arc<StdMutex<Vec<u8>>>);
impl std::io::Write for CollectorLogWriter {
fn write(&mut self, bytes: &[u8]) -> std::io::Result<usize> {
self.0.lock().expect("collector log buffer").extend_from_slice(bytes);
Ok(bytes.len())
}
fn flush(&mut self) -> std::io::Result<()> {
Ok(())
}
}
async fn capture_collector_logs<F: std::future::Future>(future: F) -> (F::Output, Vec<serde_json::Value>) {
use tracing::instrument::WithSubscriber as _;
let buffer = Arc::new(StdMutex::new(Vec::new()));
let writer = Arc::clone(&buffer);
let subscriber = tracing_subscriber::fmt()
.json()
.without_time()
.with_ansi(false)
.with_max_level(tracing::Level::DEBUG)
.with_writer(move || CollectorLogWriter(Arc::clone(&writer)))
.finish();
let _interest_guard = tracing::Dispatch::new(tracing::subscriber::NoSubscriber::default());
let result = future.with_subscriber(subscriber).await;
let logs = buffer.lock().expect("collector log buffer");
let events = std::str::from_utf8(&logs)
.expect("collector logs should be UTF-8")
.lines()
.map(|line| serde_json::from_str(line).expect("collector event should be JSON"))
.collect();
(result, events)
}
fn bucket_root(name: &str, generation: u64, size: usize) -> DataUsageEntryInfo {
DataUsageEntryInfo {
name: name.to_string(),
parent: DATA_USAGE_ROOT.to_string(),
entry: DataUsageEntry {
size,
..Default::default()
},
bucket_incarnation: None,
tier_registry_generation: Some(generation),
}
}
#[tokio::test]
#[serial]
async fn collector_reports_distinct_duplicate_and_rejected_roots_after_drain() {
for (cancel, recover_missing) in [(false, false), (false, true), (true, false), (true, true)] {
let ctx = CancellationToken::new();
let mut cache = DataUsageCache {
info: DataUsageCacheInfo {
name: DATA_USAGE_ROOT.to_string(),
source: Some(DataUsageCacheSource::new(2, 3)),
next_cycle: 17,
leader_epoch: 23,
tier_registry_generation: Some(7),
..Default::default()
},
..Default::default()
};
cache.replace(
"bucket-b",
DATA_USAGE_ROOT,
DataUsageEntry {
size: 19,
..Default::default()
},
);
let (tx, rx) = mpsc::channel(4);
if cancel {
ctx.cancel();
}
let run = async {
let collector = super::super::io_cache::collect_completed_bucket_results(ctx.clone(), cache, rx);
tokio::pin!(collector);
if cancel {
tokio::select! {
biased;
_ = &mut collector => panic!("collector cannot close while the sender is open"),
_ = tokio::task::yield_now() => {}
}
}
tx.send(bucket_root("bucket-a", 7, 1)).await.expect("first root send");
tx.send(bucket_root("bucket-a", 7, 2)).await.expect("duplicate root send");
tx.send(bucket_root("bucket-b", 8, 3)).await.expect("wrong-tier root send");
if recover_missing {
tx.send(bucket_root("bucket-b", 7, 4)).await.expect("current root send");
}
drop(tx);
collector.await
};
let ((cache, completed), events) = capture_collector_logs(run).await;
assert_eq!(completed.len(), if recover_missing { 2 } else { 1 });
assert_eq!(cache.find("bucket-a").expect("applied duplicate root").size, 2);
assert_eq!(
cache.find("bucket-b").expect("retained or recovered root").size,
if recover_missing { 4 } else { 19 }
);
assert_eq!(should_publish_completed_snapshot(completed.len(), 2, false, false), recover_missing);
assert!(!should_publish_completed_snapshot(completed.len(), 2, false, true));
let events = events
.iter()
.filter(|event| event["fields"]["state"] == "bucket_root_collector_drained")
.collect::<Vec<_>>();
assert_eq!(events.len(), 1, "each collector drain should report one bounded summary");
let fields = &events[0]["fields"];
assert_eq!(fields["collector_cache_pool"], 2);
assert_eq!(fields["collector_cache_set"], 3);
assert_eq!(fields["collector_cache_cycle"], 17);
assert_eq!(fields["collector_cache_leader_epoch"], 23);
assert_eq!(fields["received_roots"], if recover_missing { 4 } else { 3 });
assert_eq!(fields["distinct_applied_roots"], if recover_missing { 2 } else { 1 });
assert_eq!(fields["duplicate_applied_roots"], 1);
assert_eq!(fields["rejected_roots"], 1);
assert_eq!(fields["cancellation_observed"], cancel);
assert_eq!(fields["cancelled_at_drain"], cancel);
assert!(fields.get("bucket").is_none());
assert!(fields.get("object").is_none());
}
}
#[tokio::test]
#[serial]
async fn collector_reports_cancellation_when_close_and_cancel_are_both_ready() {
let ctx = CancellationToken::new();
ctx.cancel();
let (tx, rx) = mpsc::channel(1);
drop(tx);
let ((_, completed), events) = capture_collector_logs(super::super::io_cache::collect_completed_bucket_results(
ctx,
DataUsageCache::default(),
rx,
))
.await;
assert!(completed.is_empty());
let events = events
.iter()
.filter(|event| event["fields"]["state"] == "bucket_root_collector_drained")
.collect::<Vec<_>>();
assert_eq!(events.len(), 1);
let fields = &events[0]["fields"];
assert_eq!(fields["cancelled_at_drain"], true);
assert!(fields["cancellation_observed"].is_boolean(), "either ready branch may be selected first");
assert_eq!(fields["received_roots"], 0);
assert_eq!(fields["distinct_applied_roots"], 0);
}
#[tokio::test]
#[serial]
async fn incomplete_set_event_reports_requested_cycle_and_epoch() {
let ((), events) = capture_collector_logs(async {
let (_dir, store) = setup_local_scanner_store(1, 1).await;
clear_dirty_usage_buckets_for_tests();
create_bucket(&store, PARTIAL_BUCKET, 4).await;
let set = Arc::clone(&store.pools[0].disk_set[0]);
let parent = CancellationToken::new();
let budget = ScannerCycleBudget::new_with_progress_tracking(
&parent,
ScannerCycleBudgetConfig {
max_objects: Some(1),
..Default::default()
},
);
let (updates, mut results) = mpsc::channel(1);
set.clone()
.nsscanner_cache(
budget.token(),
Arc::clone(&budget),
scan_plan(&set, &[PARTIAL_BUCKET], 1).await,
updates,
1,
HealScanMode::Normal,
)
.await
.expect("budgeted fixture should return partial progress");
assert!(budget.budget_elapsed());
assert!(!results.recv().await.expect("incomplete set result").info.snapshot_complete);
clear_dirty_usage_buckets_for_tests();
})
.await;
let events = events
.iter()
.filter(|event| event["fields"]["state"] == "set_cache_publish_skipped")
.collect::<Vec<_>>();
assert_eq!(events.len(), 1);
let fields = &events[0]["fields"];
assert_eq!(fields["pool"], 0);
assert_eq!(fields["set"], 0);
assert_eq!(fields["cycle"], 1);
assert_eq!(fields["leader_epoch"], 11);
}
+6 -1
View File
@@ -109,7 +109,6 @@ pub(crate) use rustfs_ecstore::api::layout::{
SetupType as EcstoreSetupType,
};
pub(crate) use rustfs_ecstore::api::notification::NotificationSys as EcstoreNotificationSys;
pub(crate) use rustfs_ecstore::api::notification::scanner_peer_transport_error_message_is_retryable;
pub(crate) use rustfs_ecstore::api::object::{
SCANNER_PUBLICATION_LEASE_FENCE_METADATA_KEY, ScannerPublicationCommitScope, ScannerPublicationCommitState,
};
@@ -365,6 +364,12 @@ pub(crate) mod scan {
SCANNER_DIRTY_USAGE_SNAPSHOT_PROTOCOL_VERSION, SCANNER_DIRTY_USAGE_SNAPSHOT_RPC_MAX_MESSAGE_SIZE,
SCANNER_SCOPED_DIRTY_USAGE_ACK_MAX_ENTRIES,
};
pub(crate) use rustfs_ecstore::api::notification::{
ScannerPublicationLeaseFailureInfo, scanner_peer_transport_error_message_is_retryable,
scanner_publication_lease_failure_info,
};
#[cfg(test)]
pub(crate) use rustfs_ecstore::api::rpc::PeerRestClient as EcstorePeerRestClient;
}
pub(crate) mod scanner_io {
+1
View File
@@ -9,6 +9,7 @@ collection:
## Operations
- [Retained MRF responsibilities](operations/mrf-responsibilities.md) — unknown bucket identities, bounded probes, and unreadable checkpoint recovery.
- [Read-only BTF declaration inspection](operations/btf-prototype-inspection.md) — bounded prototypes and scalar member layouts, explicit void handling, and independent native references.
- [Internode HTTP streaming transport](operations/internode-http-transport.md) — protocol selection, pooled connection isolation, and distributed read validation.

Some files were not shown because too many files have changed in this diff Show More