Compare commits

..

41 Commits

Author SHA1 Message Date
houseme 1915a6fe37 fix(ecstore): tidy dst dir fsync group open
Co-Authored-By: heihutu <heihutu@gmail.com>
2026-08-19 01:33:43 +08:00
houseme fee6d5df50 feat(ecstore): add dst dir fsync group commit
Co-Authored-By: heihutu <heihutu@gmail.com>
2026-08-19 01:31:08 +08:00
hector c86a94a2dc fix(package): declare /etc/default/rustfs as a deb conffile (#6220)
Co-authored-by: houseme <housemecn@gmail.com>
2026-08-18 15:56:52 +00:00
houseme 905082893f fix(e2e): import serial test attribute (#6222) 2026-08-18 23:12:39 +08:00
houseme eed0ca3612 perf(ecstore): validate local IO paths with openat2 (#6221)
Use Linux openat2 with RESOLVE_BENEATH and RESOLVE_NO_SYMLINKS for LocalDisk I/O path validation while keeping the existing lstat walk as the public-path and unsupported-kernel fallback. Add focused regression coverage for traversal, symlink swaps, missing leaves, recreated parents, high-cardinality prefixes, final symlink leaves, and concurrent validation.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-18 22:40:43 +08:00
GatewayJ 91c97f3416 chore(deps): update s3s to upstream main (#6203)
* deps: update s3s to upstream main

* deps: refresh s3s upstream revision

* deps: pin s3s to latest upstream main

---------

Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-18 21:46:56 +08:00
唐小鸭 1d056d7605 test(e2e): pin bounded physical reads for compressed multipart range GETs (#6167)
Byte-exactness tests stay green if the compressed range seek regresses into
decoding from byte zero: the returned bytes are still correct and only the
read amplification explodes. Assert the cost side as well.

The observation reuses rustfs_io_get_object_shard_read_observed_bytes_total,
already emitted per shard read by the erasure layer, so no production code is
instrumented. The OTLP collector learns to accumulate a second counter, keyed
by its path/role/outcome labels rather than by data-point position, which is
not stable across exports.

Two failure modes the assertions guard against:

- With RUSTFS_OBS_METER_INTERVAL=1, treating one unchanged sample as settled
  measures a delta of zero, because the range read's counter has not been
  exported yet. Settling now requires several consecutive equal samples.
- An upper bound alone passes vacuously on a zero delta, so a lower bound
  turns "measured nothing" into a failure instead of a green run.

Refs rustfs/rustfs#5957, backlog#1848.
2026-08-18 21:46:16 +08:00
Zhengchao An 8315c23d49 test(kms): move the Vault KV2 doc guard into check_fips_wording.sh (#6215)
test(kms): move the Vault KV2 Transit-wrapping doc guard into check_fips_wording.sh

`test_vault_kv2_sources_do_not_claim_transit_wrapping` asserted that four
`include_str!`-pinned files never describe the Vault KV2 backend as wrapping key
material through Vault's Transit engine. The invariant is a documentation-claim
invariant with no behavioral twin by construction, and the test form was weak in
both directions: it saw only four files (the same prose in a fifth file passed
silently) and it stopped compiling — rather than reporting a violation — as soon
as one of them was renamed.

Move the four literals verbatim into `scripts/check_fips_wording.sh`, which
already guards the adjacent cryptographic over-claim class (unsupported FIPS
validation wording) and is anchored to the same policy document. The guard now
greps every file under `crates/kms` for the same four case-sensitive literals and
separately reports a moved pinned source instead of failing to build.

`check_fips_wording.sh` previously ran only in `make pre-commit` / `pre-pr`, so
wire it into the Quick Checks job of both CI workflows to keep the invariant's
failure visibility at least as strong as the deleted test's.
2026-08-18 21:46:00 +08:00
唐小鸭 1cf0f7af15 feat(replication): split oversized hot-path functions, proxy unreplicated reads, and fail SSE-C passthrough closed (#6170)
* refactor(replication): split four oversized hot-path functions into focused helpers

Pure-move decomposition of the four oversized functions flagged by the
replication compatibility review (P1-18), unblocking migration milestone
M2 which requires resyncer moves to stay mechanical:

- resync_bucket (522 lines -> 61-line step sequence): leader lock,
  target resolution, walk/collector/worker spawning, and dispatch loop
  extracted into focused helpers; pure decision helpers (DTO builders,
  HEAD-result classification) separated from IO orchestration.
- replicate_all (411 lines -> 113-line main body): initial target-info
  seeding, read/stat option builders, skip-path notes, target HEAD
  action resolution, and the multipart/single-put payload transport
  extracted as private free functions.
- start_mrf_processor (306 lines -> 46-line spawn body): recovery guard,
  ledger load, per-entry replay (delete/object/metadata), and retained
  entry resolution extracted; retry bookkeeping semantics preserved
  exactly (inner continue-paths push inside helpers, outer Missed push
  stays in the loop).
- apply_iam_item (255 lines -> match dispatch skeleton): one helper per
  IAM item type.

No behavior change: log texts, error paths, event emissions, and metric
counts are byte-identical; existing tests unchanged and green (238
ecstore replication/mrf/resync + 232 rustfs site-replication).

* feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets (#6172)

* feat(replication): proxy GET/HEAD/Tagging for unreplicated objects to replication targets

Implements the MinIO active-active read-proxy protocol (P1-5 of the
replication compatibility review): when a GET/HEAD/GetObjectTagging/
PutObjectTagging/DeleteObjectTagging request fails locally with
not-found and the bucket has replication targets, the request is proxied
to the targets in rule order, mirroring bucket-replication.go
proxyGetToReplicationTarget/proxyHeadToRepTarget/proxyTaggingToRepTarget.

Protocol surface:
- Anti-loop: inbound {x-rustfs-,x-minio-}source-proxy-request is parsed
  into ObjectOptions (proxy_request + proxy_header_set, matching MinIO
  ProxyRequest/ProxyHeaderSet); a request carrying the marker with ANY
  value is never re-proxied. Outbound client proxy calls send the marker
  as "true"; replication worker convergence HEADs send it as "false" so
  a peer's proxy layer cannot answer a convergence check by proxying
  back to the source (which would fake Completed without a PUT).
- Target selection: new replication_proxy.rs get_proxy_targets — empty
  when the marker is set, versioning is suspended, or no replication
  config; otherwise filter_target_arns -> TargetClient lookup, skipping
  targets with proxying disabled.
- TargetClient gains head_object_for_proxy/get_object (streaming) and
  the three tagging calls. Proxy calls never send the replication-check
  SSE-C exemption header; customer SSE-C keys are forwarded verbatim so
  the target performs real decryption. Conditional (If-*) headers are
  not forwarded (MinIO parity); Range and part_number are, with
  parts_count/tag_count/storage_class/expiration passed through.
- Metrics: proxy counters now count only real client proxy traffic,
  MinIO-aligned (one total per proxied request, one failed when no
  target served it). The previous misattributed counters — replication
  worker HEAD/PUT (#2672) and local tagging operations (#2682) — are
  removed; ReplProxyMetric now maps the tagging counters instead of
  dropping them.

e2e (fake_s3_target extended with tagging + header journaling): proxied
GET body + outbound header contract (marker present, no
replication-check, SSE-C passthrough), HEAD, anti-loop 404 with zero
outbound requests, GetObjectTagging, and metric mapping unit tests.

Rolling note: proxying only activates for buckets with replication
targets; requests carrying the marker keep pre-upgrade behavior.

Refs rustfs/backlog#1675 (P1-5)

* fix(replication): fail SSE-C passthrough closed on targets that drop transport headers (#6178)

SSE-C ciphertext passthrough replicates via X-Rustfs-Replication-* transport
headers. A MinIO/generic-S3 target silently discards them, storing bare
ciphertext with no decryption material — yet the PUT succeeded, so the object
reported COMPLETED with a silently unreadable replica (backlog#1675 N2).

Fail-closed design:
- SsecPassthroughCapability {Unknown, Supported, Unsupported} cached in
  BucketTargetSys per target ARN with a recording timestamp. Entries reset
  whenever the target is rebuilt, edited, or removed (arn_remotes_map
  lifecycle) and expire after SSEC_PASSTHROUGH_CAPABILITY_TTL (10 minutes):
  an expired verdict in either direction is re-earned through the audit, so
  an Unsupported target recovers automatically after an upgrade (at most one
  wasted PUT+HEAD audit per bad target per TTL window) and a Supported
  verdict cannot outlive a backend swapped behind the same endpoint.
- Replication worker (replicate_object and replicate_all): fresh Unsupported
  targets never receive the PUT — the attempt fails immediately into the
  normal MRF retry channel with a "run ?replication-check to re-probe" hint.
  Unknown or expired verdicts are audited: after the PUT the worker HEADs
  the replica back through the replication-check channel (source version id
  mapped through resolve_read_api_version_id, so null-version objects audit
  correctly) and requires SSE-C evidence (the echoed customer-algorithm
  header); missing evidence records Unsupported and fails the attempt.
  Convergence HEADs are audited the same way, so a broken ciphertext replica
  from an earlier attempt can never launder itself into COMPLETED via an
  ETag match. The gate/evidence policy is pure (replication_target_boundary,
  staleness folded in as an input) for the M2 worker migration.
- replication-check grows an SsecPassthrough probe phase: a probe PUT
  carrying the live transport-header shape, HEAD-back for evidence, and a
  machine-readable Code BucketRemoteSsecPassthroughUnsupported on failure.
  The probe verdict is synced into the runtime capability cache. Unlike
  VersionFidelity, a failed SsecPassthrough phase does NOT fail the target
  overall — it is a capability limit, not a broken replication contract,
  and a plaintext-only deployment against such a target must not turn red.
- fake_s3_target: default mode now models a RustFS target (stores the
  transport headers, echoes SSE-C evidence); the new
  drop_unlisted_replication_headers mode models MinIO. The journal records
  whether a request carried transport headers.

Receiver-echo verification: the replication-check HEAD exemption only skips
SSE-C key validation; the response has always built sse-customer-algorithm
from stored metadata (rustfs/src/app/object_usecase.rs), so no receiver
change was needed — pinned end to end by the replication-check e2e against
a real RustFS target.

Rolling-upgrade constraint: RustFS targets older than the replication-check
HEAD exemption (#5898) answer the audit HEAD without SSE-C evidence (or fail
it outright), so SSE-C replication to such targets reports FAILED. This is
deliberate — FAILED-and-retryable beats a silently undecryptable replica —
and self-heals: once the target is upgraded, the next TTL expiry (or a
manual ?replication-check re-probe) re-audits and records Supported.
Plaintext and managed-SSE replication are unaffected. The capability cache
is per-node; each node audits independently.

Known limitations:
- The audit judges evidence from the echoed customer-algorithm header only.
  A hypothetical target that preserves that one header while dropping other
  transport headers (partial-drop) would pass the audit; no known target
  behaves this way — observed targets drop the whole unknown-header family.
- A mixed-version target cluster can flap the verdict between audits routed
  to different target nodes until the rollout completes; the TTL bounds how
  long each stale verdict persists.

New e2e (backlog#1675 C1 + N2, red-first): fail-closed against a
header-dropping fake (FAILED + no second PUT via the capability cache,
journal-asserted; red run showed the old COMPLETED), replication-check
reports the SsecPassthrough phase Code while the target stays OK overall,
SSE-C heal convergence after a real target outage, and SSE-C
existing-object resync landing a REPLICA readable with the customer key.
TTL expiry in both directions is pinned at the cache and gate seams.

* refactor(replication): move resyncer pure decision logic into rustfs-replication (M2) (#6180)

* refactor(replication): move resyncer pure decision logic into rustfs-replication (M2)

Pure-move milestone M2 of the ECStore replication split (backlog#1675
P1-17): relocate the resyncer's IO-free decision helpers, with their unit
tests, into the crates they already belong to by type ownership. No
behavior change.

Moved into crates/replication:
- resync.rs: resync_status_duration
- delete.rs: resync_existing_delete_replication_info,
  replicate_delete_outcome, target_delete_version_id,
  delete_marker_purge_version_id, delete_marker_purge_mrf_entry
- object.rs: version_identity_drifted, is_replication_target_offline_error,
  SsecPassthroughCapability, SsecPassthroughGate, ssec_passthrough_gate,
  ssec_passthrough_evidence_present (param-demoted to the echoed
  customer-algorithm string; ECStore keeps the HeadObjectOutput adapter)
- filemeta.rs: NULL_VERSION_ID wire literal (crate-owned copy per the
  filemeta-independence contract)

ECStore rewiring (Rule #14: imports stay in *_boundary.rs):
- resync/object-decision/target boundaries re-export the moved symbols;
  resyncer call sites are unchanged
- bucket_target_sys keeps only the verdict cache + TTL and re-exports the
  capability enum so existing consumer paths keep compiling

Not moved (signatures carry ECStore or aws-sdk types):
verify_resync_head_result, resync_target_error_detail, the SdkError
classifiers, the replicate_all_* option/info builders, and the env-coupled
bounded_resync_max_jobs admission clamp. README milestone table updated.

* chore(replication): retire the datatypes.rs relay early

README sanctions retiring datatypes.rs ahead of M4. The module was a
pure relay (resync boundary -> datatypes -> mod.rs facade) with no
external consumer importing it directly, so the facade now re-exports
ResyncStatusType from replication_resync_boundary and the relay file is
deleted. Consumers stay behind the ECStore facade, keeping Migration
Rule #15 intact — the original retirement wording ("consumers import
through rustfs-replication directly") conflicted with that rule and is
corrected in the README.

* chore(arch): extend migration guards to the M2-moved decision contracts

The adversarial review of the M2 move found the per-symbol ratchet in
check_architecture_migration_rules.sh was not extended for the moved
symbols, leaving them free to be redefined in ECStore or imported past
their boundary without CI noticing:

- resync definition pin + boundary fences gain resync_status_duration;
- the object-decision boundary fences gain the five delete-family
  helpers (delete_marker_purge_mrf_entry, delete_marker_purge_version_id,
  replicate_delete_outcome, resync_existing_delete_replication_info,
  target_delete_version_id);
- the target-boundary fence gains the SSE-C gate family, the offline
  classifier, and version_identity_drifted;
- a new definition pin rejects ECStore redefinitions of the M2-moved
  fns/enums (ssec_passthrough_evidence_present deliberately excluded:
  ECStore keeps a thin HeadObjectOutput adapter under that name).

Mutation-verified: a probe fn ssec_passthrough_gate under
crates/ecstore/src/bucket/replication trips the new pin.

Also anchors the intentionally-duplicated NULL_VERSION_ID wire literal
from the filemeta side and tightens the M2 README note on
bounded_resync_max_jobs.
2026-08-18 21:45:38 +08:00
Zhengchao An 27d23b6135 test(io-metrics): assert record helper emissions; fix census heuristic (#6217)
* test(io-metrics): assert lib.rs record helper emissions

The 29 assertion-less record_* smoke tests in io-metrics/src/lib.rs called
their helpers and checked nothing; because METRICS_ENABLED defaults to
false they did not even reach the emission bodies. Replace them with six
DebuggingRecorder tests that enable the gate, pin every metric name the
helpers own, and pin the derived values, branch selection and label
mapping (rustfs/backlog#1836).

* test(tooling): anchor assertless-census delegation tokens to name segments
2026-08-18 21:21:34 +08:00
houseme 127b662f3f feat(app): add opt-in small GET body once path (#6216)
Use the merged s3s single-chunk StreamingBlob support for exact-length materialized GET bodies when RUSTFS_GET_SMALL_BODY_ONCE_ENABLE is enabled.

Keep the default path unchanged and fall back to the guarded MemoryTrackedBytesStream on length mismatch.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-18 11:53:06 +00:00
Zhengchao An 8a3c66e655 test(admin): replace a source-text guard with a behavior test (#6212) 2026-08-18 18:28:46 +08:00
Zhengchao An 9852e53b4c test(lifecycle,scanner): drop 47 no-op #[serial] markers (backlog#1846 T1) (#6213)
Second batch of the #[serial] sweep started in #6209. nextest is the
repository's authoritative runner and executes every test in its own
process, so serial_test's in-process mutex cannot serialize tests against
each other -- docs/testing/README.md documents this, and the mechanism
that actually serializes across the process boundary is a
.config/nextest.toml [test-groups] entry with max-threads = 1.

Unlike the first batch (e2e, process-isolated by construction), these are
in-crate unit tests that could genuinely share process state under the
`cargo test` fallback runner, where #[serial] IS still effective. Every
marker was therefore reviewed individually and removed only where the
test provably touches neither the process environment nor a process-global.

Removed (47, pure deletions, no test bodies touched):

  crates/lifecycle/src/core.rs   35
  crates/scanner/src/scanner.rs  12

The lifecycle removals are all validate_* / filter_rules_* /
has_active_rules_* / noncurrent_versions_expiration_limit_* tests that
build a local BucketLifecycleConfiguration and call a &self method
walking only that value. The scanner removals are pure duration
arithmetic (randomized_cycle_delay_for, initial_scanner_delay_for with an
explicit Some(secs), the bitrot-disabled early return of
scanner_clean_idle_max_interval) and background_heal_info_for_scan_complete
/ _for_scan_result field comparisons over locally built values.

Retained deliberately -- see the PR body for the full list and reasons:

  crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs  101 (all)
  crates/lifecycle/src/core.rs                                  44
  crates/scanner/src/scanner.rs                                 53

bucket_lifecycle_ops.rs keeps every marker: its test module caches a
process-wide `static STALE_MULTIPART_TEST_ENV: OnceLock<(Vec<PathBuf>,
Arc<ECStore>)>`, and its own reregister_env_local_disks helper documents
in-tree that sibling #[serial] tests reset and reshape the shared
local-disk registry for each other. That sharing is real, so the markers
stay.

No test was renamed, added, or deleted; no reserved migration-gate name
substring is affected; no .config/nextest.toml entry references any of
the 47 removed tests.
2026-08-18 18:28:13 +08:00
Zhengchao An a38743caf5 chore(zip): trim the unused extract and create surface (#6210)
rustfs-zip has one workspace consumer, and it uses only
CompressionFormat::{from_extension, extension, get_decoder} and
ArchiveLimits. Remove the tar/zip extract, zip create, and in-memory
compress helpers together with the types and dependencies that only
served them. Trimming public API is semver-major once the stable tag is
cut, so it costs least now.
2026-08-18 18:27:01 +08:00
houseme 382ae9529e feat(ecstore): instrument rename sync tail metrics (#6205)
Add default-off PUT stage attribution for the rename_data sync tail so strict durability probes can split queue wait, fdatasync, directory fsync, rename, per-disk wait, and quorum wait without changing commit ordering or S3-visible behavior.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-18 17:20:55 +08:00
Zhengchao An 68547ed7ea test(e2e): drop 187 no-op serial markers from the three densest e2e suites (#6209)
test(e2e): drop no-op serial markers from the three densest e2e suites

serial_test's #[serial] is an in-process mutex. cargo-nextest, this repo's
authoritative runner, executes every test in its own process, so the mutex is
never contended and the attribute is a documented no-op -- see the "Serial
execution & nextest profiles" section of docs/testing/README.md and the header
of .config/nextest.toml. Cross-process serialization is provided only by a
[test-groups] entry with max-threads = 1.

Remove 187 such markers (plus 3 now-unused imports) from the three
marker-densest modules of crates/e2e_test:

  multipart_auth_test.rs             85
  replication_extension_test.rs      68
  object_lock/object_lock_test.rs    34

None of these modules is covered by any [test-groups] entry, so the markers
were carrying no isolation for any lane. Every test in all three files builds
its own server via RustFSTestEnvironment::new(), which gives a UUID temp dir
and a uniquely allocated port -- the .config/nextest.toml comment on
replication_extension_test already states this explicitly ("parallel-safe by
construction"). No test mutates process env, binds a fixed port, or touches
process-global state, so nothing here needed temp_env or a test-group instead.

Pure deletion: 190 lines removed, 0 added, no test renamed, no behaviour
changed. serial_test stays in Cargo.toml -- 338 markers across 90 other files
in the crate still use it.

Refs: backlog#1846 (T1).
2026-08-18 17:05:30 +08:00
Zhengchao An f06a9c9cba fix(deps): record heal's bytes and crc-fast in the lockfile (#6211) 2026-08-18 17:04:55 +08:00
Zhengchao An bd296eff9e chore(io-core): drop eight zero-consumer modules (#6201) 2026-08-18 08:14:05 +00:00
houseme a5800033bd feat(heal): incremental status cursors and typed overlap policy (HS-06) (#6206)
* feat(heal): incremental heal status cursors and typed overlap policy (HS-06)

Incremental results: every retained result item now carries a monotonic
sequence number. The status query accepts a client cursor (sinceSeq on
the admin wire, Option<u64> internally) and returns only newer items,
plus nextSeq (the next cursor) and minSeq (the oldest retained
sequence). A cursor that fell behind the 1024-item retention window is
flagged through the existing truncated signal together with minSeq so
the client can restart from it. Sequencing survives task completion:
the completion archive stores the seq-stamped window. None keeps the
exact legacy full-snapshot behavior, so existing clients see no change.

Typed overlap handling for admin starts: RUSTFS_HEAL_OVERLAP_POLICY
(merge default | minio_error). Under minio_error, an admin start whose
path overlaps an active or queued task rejects with typed
already-running / overlapping-paths admission reasons (surfaced through
reason_label in the admin error body, sharing the existing
OperationAborted site because the s3s footprint ratchet forbids new
s3_error! sites); an exact duplicate start rejects with
already-running instead of silently merging. Scanner/autoheal/
read-repair sources never take the rejection path.

forceStart semantics now match MinIO for admin requests: an admin
forceStart first cancels the overlapping active admin task, then
admits the replacement.

Wire: the heal-control Query command grows an optional sinceSeq
(defaulted and skipped when absent, so older peers stay compatible);
the admin handler accepts the sinceSeq query parameter; the local
channel query gains the same cursor.

Tests: seq monotonicity and incremental slicing, window slide moving
minSeq with lagging-cursor flags, overlap matrix (same/containing/
contained/disjoint x policy x source), forceStart cancel-then-admit,
and the completion-archive window handoff.

Co-Authored-By: heihutu <heihutu@gmail.com>

* style: fmt after main merge

---------

Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
2026-08-18 16:09:30 +08:00
cxymds a4ea36b298 fix(ecstore): single-flight remote disk recovery (#6096)
* fix(ecstore): single-flight remote disk recovery

* test(ecstore): cover remote recovery review cases

* test(ecstore): exercise recovery through disk slot

* test(ecstore): match format reads exactly

* test(ecstore): cover recovery teardown races

* style(ecstore): format recovery race tests

* fix(ecstore): remove unused health snapshot helper

* fix(ecstore): group recovery monitor test state

* fix(protos): preserve production source in compatibility checks
2026-08-18 14:49:53 +08:00
cxymds 4f68f117ba fix(lock): reap expired local lease guards (#6094)
* fix(lock): reap expired local lease guards

* fix(lock): reject refresh after guard expiry

* test(lock): isolate expired refresh regression

* style(lock): format expired refresh assertion
2026-08-18 14:49:39 +08:00
cxymds 0f30a75fdb fix(ecstore): resume remote shard reads once (#6091)
* fix(ecstore): preserve CopyObject producer errors

* fix(ecstore): resume remote shard reads once

* fix(app): resume preserved relocation I/O errors

* fix(ecstore): reserve remote read recovery budget
2026-08-18 14:49:19 +08:00
Zhengchao An 355c8d2e22 fix(admin): classify missing kms config by error variant (#6196) 2026-08-18 05:40:10 +00:00
Zhengchao An 60eb139db9 refactor: import x-amz-checksum header names from the shared constants (#6193)
Co-authored-by: houseme <housemecn@gmail.com>
2026-08-18 05:27:38 +00:00
hector 9ef059c908 ci(package): auto-trigger DEB/RPM packaging on releases and upload to GitHub release assets (#6202) 2026-08-18 05:18:13 +00:00
Zhengchao An 51497cb533 fix(ecstore): give peer REST failures op and bucket context (#6200) 2026-08-18 04:57:44 +00:00
Zhengchao An c7a29ec0a7 chore(storage): drop dead backpressure and lock-optimizer wrappers (#6195)
Neither rustfs/src/storage/backpressure.rs nor rustfs/src/storage/lock_optimizer.rs had a production caller: their only non-self references were the pub mod lines in storage/mod.rs and a cfg(test) module, so the object transfer path never applied this backpressure and never took these lock shortcuts.

The six removed tests in concurrent_fix_test.rs duplicated tests that lived inside the deleted files; the shared primitives they shadowed keep their own coverage in rustfs-io-core.
2026-08-18 12:51:44 +08:00
Zhengchao An deb0edb7cc chore: adjudicate 26 bare dead_code allows across five crates (#6187)
Remove every bare `#[allow(dead_code)]` in io-core, object-capacity, targets, rio, and scanner. Each allow was stripped first and clippy was then asked which ones the compiler actually missed, so the verdicts rest on the diagnostic rather than on inspection.

23 were inert: they sat on `pub fn`s inside `pub mod`s, where `dead_code` does not apply, or on scanner integration-test helpers that the tests in the same file do call.

The remaining 3 are in rio's private `compress_index` module and the code behind them is deleted rather than annotated. `remove_index_headers` is dead and also wrong — after skipping the 4-byte chunk header it matches against `S2_INDEX_TRAILER` where `S2_INDEX_HEADER` sits, so it returns `None` for every well-formed index; rio-v2 carries the correct equivalent that is actually in use. `restore_index_headers` is its unreachable counterpart, likewise duplicated live in rio-v2. `Index::reset` is a private method with no caller.

Refs backlog#1823
2026-08-18 12:45:42 +08:00
houseme a08de9229b feat(heal): wire MRF intents with durable repair journal (HS-01) (#6189)
* feat(common): add MRF intent channel and Mrf request source (HS-01)

Introduce the producer-facing half of the mission repair feed: a global
bounded (8192) channel carrying lightweight MrfIntent values from IO
error paths, plus the RUSTFS_HEAL_MRF_ENABLE delivery kill-switch and
config constants for queue/journal sizing. Delivery is strictly
non-blocking (try_send, drop-on-full) so it can sit on decode-failure
and partial-write paths without adding latency. HealRequestSource grows
a 'mrf' variant so admission accounting can attribute replayed intents.

Part of backlog#1865 (option a: wire HealEvent-style intents with a
durable retry ledger).

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(heal): add MRF queue, durable journal, and intent consumer (HS-01)

Consumer half of the mission repair feed: a bounded pending queue
(100k intents / 8 MiB dual ceiling, drop-newest on overflow), a durable
journal at buckets/.heal/mrf/journal.bin holding the unaccepted pending
snapshot, and a consumer task that batches intents off the global
channel, translates them into prioritized heal requests (decode
failure -> Urgent ECDecode, metadata corruption -> High Metadata,
partial write -> Normal object heal), and retries full admissions with
a 5s backoff and a 3-attempt ceiling.

Durability: every journal record carries its own CRC32 and a
format/version header, so a torn tail truncates cleanly at replay; the
journal is deleted after a successful replay and when the pending set
drains (mirroring MinIO's post-replay list.bin unlink). Losing the last
500 ms flush window is acceptable: replayed duplicates merge via the
manager dedup key and read-repair remains the safety net.

Metrics: rustfs_heal_mrf_queue_depth/_queue_bytes, _dropped_total
{reason}, _replayed_total, _journal_bytes, _journal_fsync_total.
The consumer is wired at heal runtime bootstrap right after manager
start, honoring RUSTFS_HEAL_MRF_ENABLE (default on, rollback = off).

Tests: unit tests for the dual ceiling, record roundtrip, torn-tail
truncation, and the priority mapping; integration tests against a real
4-disk ECStore proving channel intents reach the manager queue as
Urgent/mrf-attributed requests and journal replay arms intents, drops
torn tails, and removes the file.

Part of backlog#1865 (option a).

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(ecstore,scanner): deliver MRF intents from error paths (HS-01)

Wire the three production delivery points, each a single non-blocking
try_send next to the existing in-memory heal paths, which stay as the
fast path:

- read.rs decode-error branch: DecodeFailure intent beside the existing
  read-repair submit, so an Urgent ECDecode request survives restarts
  even when the Low-priority read-repair request was dropped or lost.
- add_partial: PartialWrite intent, giving partial-write recovery a
  durable Normal-priority object heal across restarts.
- scanner_folder metadata-corruption classification: MetadataCorruption
  intent beside the existing High-priority scanner heal request.

All three are on error paths only: zero cost on healthy IO.

Part of backlog#1865 (option a).

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix: include mrf heal source counts

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix: keep node heal status wire compatibility

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-18 12:43:27 +08:00
Zhengchao An abffa5cf1b chore(storage): drop dead io-schedule metrics and helpers (#6199) 2026-08-18 04:35:36 +00:00
Zhengchao An b825c54850 refactor(admin): route kms management auth through shared gate (#6194) 2026-08-18 12:21:49 +08:00
houseme de9145e87a feat(storage): add default-off PUT admission gate (#6197)
Add an experimental fixed-count foreground PutObject admission gate for #1882 Phase 0 validation. The gate is default-off, returns SlowDown before body ingest when saturated, and keeps the admission permit with the spawned store commit owner until store PUT returns.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-18 04:15:19 +00:00
houseme 84bd76a3ce chore(deps): refresh cargo dependencies (#6198)
Update workspace Cargo dependency requirements and lockfile after cargo update/upgrade, including rumqttc-next 0.34.0 and MQTT API compatibility adjustments.

Verification:

- cargo update --verbose

- cargo upgrade --verbose

- cargo update -p rumqttc-next --precise 0.34.0 --verbose

- cargo tree --invert rumqttc-next --locked

- cargo metadata --locked --no-deps --format-version 1

- cargo fmt --all --check

- cargo check -p rustfs-targets --all-targets --locked

- cargo test -p rustfs-targets mqtt --locked

- make pre-pr

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-18 04:11:53 +00:00
houseme 9a2d06b370 test(heal): lock heal vs delete/overwrite race invariants (HS-12) (#6183)
* test(heal): add concurrency invariants for heal vs delete/overwrite races (HS-12)

Audit conclusion for backlog#1874: RustFS does not need a persistent
object-level healing marker (MinIO x-minio-healing) because every path
that can touch the same (bucket, object) commit surface serializes on
the same namespace write lock, and the heal lock guard spans the whole
rename commit including the HEAL_RENAME_INCOMPLETE partial path.

Lock the conclusion in with two race regression tests:

- heal_racing_version_delete_never_resurrects_the_deleted_version:
  shard damage is injected on the doomed version so a Deep heal has real
  reconstruction work while a versioned DELETE runs concurrently; the
  deleted version must stay deleted and the survivor intact.
- heal_racing_unversioned_overwrites_preserves_the_last_commit:
  unversioned overwrites (activating the post-commit tail that deletes
  the replaced data dir without the ns lock) race a Deep heal in a
  loop; the final current version must be exactly the last commit.

Also adds docs/operations/heal-concurrency-safety-notes-zh.md with the
full intersection matrix (17 intersections), lock-coverage argument,
and the residual-window classification (commit tail races are
fail-into-retry safe; bare prefix delete has zero production callers;
admin no_lock is an explicit operator opt-in).

Co-Authored-By: heihutu <heihutu@gmail.com>

* test: remove redundant heal etag clone

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-18 02:01:04 +00:00
Zhengchao An 00de43528c fix(ecstore): describe peer bucket RPC failures with no details (#6190)
heal_bucket, list_bucket, get_bucket_info and delete_bucket returned Error::other("") when a peer answered success=false without an error payload, so operators saw a bare "io error " after quorum reduction. Route all five bucket RPCs through peer_failure_without_details, which names the operation and bucket while staying identical across the peers of one operation so reduce_errs keeps grouping them into a single dominant error.
2026-08-18 01:55:29 +00:00
houseme 35a30cd614 feat(scanner): emit excess alerts as S3 notification events (HS-04) (#6176)
* feat(scanner): emit excess alerts as S3 notification events

The excess-versions / excess-version-size / excess-folders alerts were
metrics-and-logs only; consoles and external auditors had no way to hear
them (rustfs/backlog#1868, HS-04). MinIO emits s3:ObjectManyVersions /
s3:ObjectLargeVersions / s3:PrefixManyFolders for the same conditions —
RustFS carries those as EventName::Scanner* with s3:Scanner:* wire names
that already existed unpublished.

The three alert sites now also dispatch through the standard event
pipeline (send_event via the storage_api owner facade), carrying the
actual values and thresholds in req_params and UserAgent "Scanner".
Without a cooldown a single over-threshold object would re-emit on every
~60s scan cycle, so emissions are edge-held per (kind, bucket, object)
for 24h (RUSTFS_SCANNER_ALERT_COOLDOWN_SECS, 0 = every cycle), backed by
a process-global map with a 4096-key hard cap that clears rather than
grows. Metrics and structured logs stay level-triggered every cycle;
only the notification events are held back. A restart resets the
cooldown deliberately: one re-emission per still-hot key buys back
visibility after the restarts that accompany incident response.

Tests pin the edge-hold semantics (first fires, immediate re-check held,
independent keys, cooldown expiry re-fires, zero cooldown always emits,
hard bound) in one sequential test for the process-global map, and pin
the emitted wire names against EventName's canonical string forms so a
subscribed bucket notification can never silently stop matching.
docs/operations/scanner-excess-alerts.md documents the three events,
the metric-vs-event cadence difference, and the HS-15 threshold deltas
(alert_excess_folders 65538 vs MinIO 50000 is deliberate: Proxmox
Backup Server chunk layout compatibility).

Closes rustfs/backlog#1868.

Co-Authored-By: heihutu <heihutu@gmail.com>

* docs(operations): split scanner excess alerts into English and Chinese pages

The page shipped Chinese-only; keep it as scanner-excess-alerts_zh.md and
add a faithful English translation at the original path, cross-linked at
the top of both.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-18 08:46:32 +08:00
houseme 360bceafce feat(heal): add progress and trace observability (#6179)
* feat(heal): track erasure set progress baseline

Record erasure-set heal byte progress from per-object results and seed progress totals from complete usage-cache snapshots when available.

Keep usage-cache failures observational so heal execution continues without a baseline.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(heal): skip filtered erasure set versions

Skip erasure-set versions written after the durable heal start time, and queue lifecycle-expired versions for expiry before skipping them.

Track new-version and ILM-expired skips separately so progress can explain completed baseline work without treating these skips as retry-blocking failures.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(heal): wire abandoned data-dir cleanup check

Connect check_abandoned_parts through ECStore, pool, and set layers so heal can invoke the existing orphan data-dir reclaim path instead of returning NotImplemented.

Add dry-run support to the reclaim scan and cover dry-run plus scoped set behavior with regression tests.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): add heal scanner trace bus

Introduce an in-process broadcast trace bus with typed heal and scanner events, lazy event construction, and bounded lagged-subscriber behavior.

Cover zero-subscriber publishing, subscription delivery, drop accounting, and lagged receivers with focused common-crate tests.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): stream heal trace events from admin API

Wire the admin trace endpoint to the common trace bus for heal/scanner events, including kind, regex, and threshold filtering.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): emit heal trace events

Publish heal task lifecycle and abandoned-parts cleanup events through the common trace bus so the admin trace stream has live heal diagnostics.

Co-Authored-By: heihutu <heihutu@gmail.com>

* feat(obs): emit scanner trace events

Publish scanner folder, lifecycle action, and heal-candidate events through the common trace bus for live admin scanner diagnostics.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(heal): route data usage loader through storage api

Keep ECStore data-usage facade access behind the heal storage_api boundary so architecture migration guards can validate the heal progress path.

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(heal): avoid lifecycle snapshots on ordinary heal pages

Only request lifecycle object snapshots when the heal pass has lifecycle expiry context. This keeps ordinary listing and disk-walk pages from cloning FileInfo/ObjectInfo payloads while preserving the skip path that queues expired versions.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(heal): update bug-fix mocks for lifecycle snapshots

Carry the lifecycle snapshot opt-in argument through the remaining heal bug-fix test mocks so all-targets clippy covers the updated storage trait.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(rustfs): sync heal storage mock signature

Update the rustfs storage RPC test mock for the lifecycle snapshot opt-in argument and cover it with rustfs all-targets clippy.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(e2e): allocate smoke ports across nextest processes

Serialize E2E port selection with a small /tmp allocator so nextest workers do not reuse the same just-released ephemeral port before RustFS binds it.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-18 08:29:29 +08:00
Zhengchao An 7cb91a0190 chore(ecstore): adjudicate 32 bare dead_code allows (#6173)
Replace every bare `#[allow(dead_code)]` in ecstore with either a deletion or a per-item allow carrying a `reason`. Blanket allows at module, struct, and impl level silence the lint for future members too, so each is narrowed to the members that are actually dead.

Delete the dead cluster in `config/heal.rs` (`Config`, its three methods, `RUSTFS_BITROT_CYCLE_IN_MONTHS`, `parse_bitrot_config`) rather than annotate it: it has no callers and is unreachable outside the crate, and `parse_bitrot_config` would panic on its disabled path via `Duration::from_secs_f64(-1.0)`. `DEFAULT_KVS` stays, since the config registry uses it.

Correct two `reason` strings on `Checksum::new` and `PutObjReader::md5_current_hex_string`, which are methods but carried a field-only rationale.

Refs backlog#1823

Co-authored-by: houseme <housemecn@gmail.com>
2026-08-17 23:51:56 +00:00
hector beb6e1383e feat(helm): add TLSRoute passthrough support for gateway api (#6169)
Add an optional TLS passthrough listener to the Gateway API support. When gatewayApi.listeners.tls.enabled is true, the Gateway gets a TLS listener with tls.mode: Passthrough and a TLSRoute is rendered to the RustFS service so TLS terminates at the backend (end-to-end encryption).

Refs rustfs/rustfs#3862.
2026-08-18 01:21:15 +08:00
houseme 59b7d13095 feat(scanner): expose prefix-level bucket usage via admin API (HS-08) (#6171)
feat(scanner): expose prefix-level bucket usage via admin API

The scanner's per-bucket, per-set usage caches already hold a path-keyed
prefix tree, but dui() flattened it only to bucket names — consoles and
operators had no way to ask "what does this prefix hold" without an S3
listing sweep (rustfs/backlog#1872, MinIO loadPrefixUsageFromBackend
parity).

Add:

- data-usage: prefix_usage_in_cache — a shared aggregation over the
  entry map (arbitrary prefix, full counters, one-level sub-prefix
  breakdown with names recovered from the literal-path cache keys),
  hardened like the scanner's checked flatten: cycles, dangling child
  links, over-deep trees, and overflowing counters yield None rather
  than unbounded recursion or wrapped totals.
- ecstore: ECStore::all_set_disks — iterate every erasure set so a
  query can read each set's own cache copy; the hash-routed store path
  would always land on one set.
- scanner: bucket_prefix_usage — per-set loads (5s budget each, a slow
  set degrades to not-reporting instead of stalling the caller),
  merged across sets with partial/compacted/truncated flags, served
  from a bounded 30s cache (128 entries, hard-capped) that bucket
  writes invalidate through the dirty-usage hook.
- admin: GET /rustfs/admin/v3/usage/{bucket}?prefix=&max-entries=
  behind the same any-of gate as datausageinfo (DataUsageInfoAdminAction
  OR ListBucketAction), rejecting unknown query parameters and
  clamping max-entries to 1..=10000. Route registered in the policy
  table (deferred MultipleActions, matching datausageinfo) and the
  route matrix test.

Closes rustfs/backlog#1872.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-17 11:40:56 +00:00
唐小鸭 e0b87b0e7e fix(site-replication): admit only verifiable peer-edit fences (#6123) 2026-08-17 09:47:36 +00:00
182 changed files with 16733 additions and 10450 deletions
+2 -2
View File
@@ -66,8 +66,8 @@ s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
./scripts/check_s3s_footprint.sh
.PHONY: fips-wording-check
fips-wording-check: ## Check outward docs do not make unsupported FIPS claims
@echo "📣 Checking FIPS wording guard..."
fips-wording-check: ## Check docs and crates/kms do not over-claim crypto capabilities
@echo "📣 Checking cryptographic capability wording guard..."
./scripts/check_fips_wording.sh
.PHONY: log-analyzer-rules-check
+11
View File
@@ -87,6 +87,13 @@ test-group = 'ecstore-serial-flaky'
filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
test-group = 'ecstore-serial-flaky'
# Serialize the default-off dst-dir fsync group-commit tests. They use
# process-global test hooks/registry to deterministically freeze fsync batches;
# no retries, just one at a time under nextest too.
[[profile.default.overrides]]
filter = 'package(rustfs-ecstore) & test(dst_dir_fsync_group_commit)'
test-group = 'ecstore-serial-flaky'
# Serialize the 4-disk reliability / degraded-read e2e tests (see the
# e2e-reliability test-group note above). The matching ci-profile override is at
# the end of the file, after [profile.ci] is declared.
@@ -188,6 +195,10 @@ test-group = 'ecstore-serial-flaky'
filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
test-group = 'ecstore-serial-flaky'
[[profile.ci.overrides]]
filter = 'package(rustfs-ecstore) & test(dst_dir_fsync_group_commit)'
test-group = 'ecstore-serial-flaky'
# ---------------------------------------------------------------------------
# e2e-smoke profile — PR smoke subset of the e2e_test crate (backlog#1149 ci-4)
# ---------------------------------------------------------------------------
+3
View File
@@ -117,6 +117,9 @@ jobs:
- name: Check s3s footprint ratchet
run: ./scripts/check_s3s_footprint.sh
- name: Check cryptographic capability wording
run: ./scripts/check_fips_wording.sh
- name: Check no planning docs committed
run: ./scripts/check_no_planning_docs.sh
+3
View File
@@ -152,6 +152,9 @@ jobs:
- name: Check s3s footprint ratchet
run: ./scripts/check_s3s_footprint.sh
- name: Check cryptographic capability wording
run: ./scripts/check_fips_wording.sh
- name: Check no planning docs committed
run: ./scripts/check_no_planning_docs.sh
+90 -11
View File
@@ -15,28 +15,35 @@
# Package Workflow - Build DEB/RPM packages
#
# This workflow builds DEB and RPM packages from pre-built Linux binaries
# and uploads them to Cloudflare R2.
# and uploads them to Cloudflare R2 and the GitHub release.
#
# Trigger:
# - release published: automatically package when a GitHub release is published
# - workflow_dispatch: manual trigger with optional tag/run_id
# - workflow_run: automatically package after "Build and Release" completes
# for a release tag (the mac/windows/linux binaries are already uploaded
# to the GitHub release before packaging starts)
# - workflow_dispatch: manual fallback (backfill / re-run) with optional tag/run_id
#
# Flow:
# 1. Find the Build workflow run for the release tag
# 1. Resolve the triggering Build workflow run for the release tag
# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu) from build artifacts
# 3. Build DEB packages for amd64 and arm64
# 4. Build RPM packages for x86_64 and aarch64
# 5. Upload all packages to Cloudflare R2
# 5. Upload all packages to Cloudflare R2 and the GitHub release
name: Package DEB/RPM
permissions:
contents: read
# contents: write is required to upload packages to the GitHub release
contents: write
actions: read
on:
release:
types: [ published ]
# Follows the same pattern as docker.yml: run after the release build
# workflow completes, so packaging is triggered only by release tags
# (e.g. 1.0.0-rc.2, 1.0.0-rc.3), never by development builds.
workflow_run:
workflows: [ "Build and Release" ]
types: [ completed ]
workflow_dispatch:
inputs:
tag:
@@ -49,13 +56,26 @@ on:
type: string
concurrency:
group: ${{ github.workflow }}-${{ github.event.release.tag_name || github.event.inputs.tag || github.run_id }}
group: ${{ github.workflow }}-${{ github.event.workflow_run.head_branch || github.event.inputs.tag || github.run_id }}
cancel-in-progress: true
env:
HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
WORKFLOW_RUN_ID: ${{ github.event.workflow_run.id }}
jobs:
# Resolve which build run to use and extract version info
resolve:
name: Resolve Build
# Auto-trigger only from successful tag builds of "Build and Release".
# Tag pushes arrive as event == push with head_branch != main (a
# non-main push head_branch is the release tag name). Manual dispatch
# stays available as a fallback for backfills and re-runs.
if: >-
github.event_name == 'workflow_dispatch' ||
(github.event.workflow_run.conclusion == 'success' &&
github.event.workflow_run.event == 'push' &&
github.event.workflow_run.head_branch != 'main')
runs-on: ubuntu-latest
timeout-minutes: 10
outputs:
@@ -75,8 +95,8 @@ jobs:
set -euo pipefail
# Determine tag
if [[ "${{ github.event_name }}" == "release" ]]; then
TAG="${{ github.event.release.tag_name }}"
if [[ "${{ github.event_name }}" == "workflow_run" ]]; then
TAG="${HEAD_BRANCH}"
elif [[ -n "$INPUT_TAG" ]]; then
TAG="$INPUT_TAG"
else
@@ -93,6 +113,11 @@ jobs:
BUILD_RUN_ID="$INPUT_RUN_ID"
echo "Using explicit build run ID: $BUILD_RUN_ID"
elif [[ "${{ github.event_name }}" == "workflow_run" ]]; then
# Use the Build and Release run that triggered this workflow
BUILD_RUN_ID="${WORKFLOW_RUN_ID}"
echo "Using triggering workflow run: $BUILD_RUN_ID"
elif [[ -n "$TAG" ]]; then
# Find the build run that produced this tag
echo "Looking for build run for tag: $TAG"
@@ -265,6 +290,12 @@ jobs:
Homepage: https://rustfs.com
EOF
# Declare /etc/default/rustfs as a conffile so dpkg preserves user
# modifications on upgrade instead of silently overwriting them.
cat > "${PKG_DIR}/DEBIAN/conffiles" << 'CONFFILES'
/etc/default/rustfs
CONFFILES
cat > "${PKG_DIR}/DEBIAN/postinst" << 'POSTINST'
#!/bin/bash
set -e
@@ -456,6 +487,54 @@ jobs:
echo "✅ Latest packages updated"
fi
- name: Upload packages to GitHub Release
if: needs.resolve.outputs.tag != ''
env:
GH_TOKEN: ${{ github.token }}
shell: bash
run: |
set -euo pipefail
TAG="${{ needs.resolve.outputs.tag }}"
DEB_FILE="${{ steps.deb.outputs.deb_file }}"
RPM_FILE="${{ steps.rpm.outputs.rpm_file }}"
# Upload the packages, then refresh the release checksums so the new
# assets are covered, matching the binary release flow.
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
echo "📤 Uploading $(basename "$f") to GitHub release ${TAG}..."
gh release upload "$TAG" "$f" --clobber
fi
done
CHECKSUM_DIR="$(mktemp -d)"
gh release download "$TAG" -p 'SHA256SUMS' -p 'SHA512SUMS' \
-D "$CHECKSUM_DIR" --clobber 2>/dev/null || true
for spec in "SHA256SUMS:sha256sum" "SHA512SUMS:sha512sum"; do
asset="${spec%%:*}"
checksum_cmd="${spec##*:}"
checksum_file="${CHECKSUM_DIR}/${asset}"
touch "$checksum_file"
for f in "$DEB_FILE" "$RPM_FILE"; do
if [[ -n "$f" && -f "$f" ]]; then
base="$(basename "$f")"
# Remove any stale entry, then append the fresh digest
grep -Fv -- "$base" "$checksum_file" > "${checksum_file}.tmp" || true
mv "${checksum_file}.tmp" "$checksum_file"
(cd "$(dirname "$f")" && "$checksum_cmd" -- "$base") >> "$checksum_file"
fi
done
echo "📤 Updating ${asset} for release ${TAG}..."
gh release upload "$TAG" "$checksum_file" --clobber
done
echo "✅ GitHub release assets updated"
# Summary
summary:
name: Summary
+4 -4
View File
@@ -31,7 +31,7 @@ HTTP request
→ storage/ecfs (erasure coding, encryption, checksums)
→ ecstore (disk pool selection, data distribution)
→ rio (reader pipeline: encrypt → compress → hash → write)
→ io-core (zero-copy I/O, buffer pool, direct I/O)
→ io-core (buffer pool, storage profiling, admission control)
→ local disk / remote disk via RPC
```
@@ -55,7 +55,7 @@ rustfs/ # Workspace root (virtual manifest)
├── crates/ # library crates (authoritative list: Cargo.toml [workspace].members)
│ ├── ecstore/ # Erasure-coded storage engine
│ ├── rio/ # Reader I/O pipeline (encrypt, compress, hash)
│ ├── io-core/ # Zero-copy I/O, scheduling, buffer pool
│ ├── io-core/ # Buffer pool, storage profiling, admission control
│ ├── io-metrics/ # I/O metrics collection
│ ├── common/ # Shared runtime state, globals, data usage types
│ ├── config/ # Configuration types and parsing
@@ -302,7 +302,7 @@ The binary (`main.rs`) boots in this order:
│ │ │
┌─────▼──────┐ ┌──────▼──────┐ ┌──────▼──────┐
│ ecstore │ │ rio │ │ io-core │
│ (core) │ │ (readers) │ │ (zero-copy)
│ (core) │ │ (readers) │ │ (buffers)
└─────┬──────┘ └─────────────┘ └─────────────┘
┌─────┬──┼──┬─────┬──────┐
@@ -314,7 +314,7 @@ The binary (`main.rs`) boots in this order:
- **"Where does S3 PutObject go?"**
`server/` routes → `app/object_usecase` validates → `storage/ecfs` encodes →
`ecstore` distributes → `rio` encrypts/compresses → `io-core` writes
`ecstore` distributes → `rio` encrypts/compresses → `io-core` supplies buffers
- **"Where are bucket policies enforced?"**
`app/bucket_usecase` calls into `crates/policy/`
Generated
+110 -109
View File
@@ -964,9 +964,9 @@ dependencies = [
[[package]]
name = "aws-sdk-kms"
version = "1.114.0"
version = "1.115.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c0b7d906608ee41e7ddea9983577ba82200435644d567d63dc34e822e088b453"
checksum = "d5b034f8b7ceadb873d0bc607c30bb4b0be68e09a84c837174e7c2c6878ff882"
dependencies = [
"arc-swap",
"aws-credential-types",
@@ -990,9 +990,9 @@ dependencies = [
[[package]]
name = "aws-sdk-s3"
version = "1.141.0"
version = "1.142.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d9f9420d3a2467eed22ed3635ca653653162c386a0b0f65c78189f9bd3c1379e"
checksum = "f9e15a5c55e05f4b0b7e483160b3c85cccdf77cff02c95504f3e71d460855cd2"
dependencies = [
"arc-swap",
"aws-credential-types",
@@ -1027,9 +1027,9 @@ dependencies = [
[[package]]
name = "aws-sdk-sso"
version = "1.105.0"
version = "1.106.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6ffd0fbe7873cb548a7aa60f9573c268fff94155397fd4f14dc9f1ecaaab8516"
checksum = "2d0efcee834347b6705eca3eea2defd88242f43774f55d7326604222e3c86260"
dependencies = [
"arc-swap",
"aws-credential-types",
@@ -1053,9 +1053,9 @@ dependencies = [
[[package]]
name = "aws-sdk-ssooidc"
version = "1.107.0"
version = "1.108.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "175763eb222a46377df7aa257a3bca980ab3e96703fefc8f4d0b8da6ad2e254c"
checksum = "a59312a04cf19c962cfee32b64ecfee758f8786407ff6da5b30fff46ae96f201"
dependencies = [
"arc-swap",
"aws-credential-types",
@@ -1079,9 +1079,9 @@ dependencies = [
[[package]]
name = "aws-sdk-sts"
version = "1.110.0"
version = "1.111.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "dd8b14781dfbff48984017d57167b6ea0b6471c6920ec52b44a2677c7feb3c13"
checksum = "120e7eb63457a9e547f9986fe3b273f77c43679da4d04f46359fa881c5e19b6e"
dependencies = [
"arc-swap",
"aws-credential-types",
@@ -1598,7 +1598,7 @@ version = "0.10.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3078c7629b62d3f0439517fa394996acacc5cbc91c5a20d8c658e77abd503a71"
dependencies = [
"generic-array 0.14.9",
"generic-array 0.14.7",
]
[[package]]
@@ -1617,7 +1617,7 @@ version = "0.3.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a8894febbff9f758034a5b8e12d87918f56dfc64a8e1fe757d65e29041538d93"
dependencies = [
"generic-array 0.14.9",
"generic-array 0.14.7",
]
[[package]]
@@ -1858,9 +1858,9 @@ dependencies = [
[[package]]
name = "cc"
version = "1.4.2"
version = "1.4.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5d262e149917187838d5b42777c8253bcb64500067342904e7d429499a6f277e"
checksum = "509591b7bcd67f4ef775afad7662703b4935daaa6ec0e5605cfb1090b32a2b6d"
dependencies = [
"find-msvc-tools",
"jobserver",
@@ -1968,7 +1968,7 @@ version = "0.4.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "773f3b9af64447d2ce9850330c473515014aa235e6a783b02db81ff39e4a3dad"
dependencies = [
"crypto-common 0.1.6",
"crypto-common 0.1.7",
"inout 0.1.4",
]
@@ -2428,7 +2428,7 @@ version = "0.5.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0dc92fb57ca44df6db8059111ab3af99a63d5d0f8375d9972e319a379c6bab76"
dependencies = [
"generic-array 0.14.9",
"generic-array 0.14.7",
"rand_core 0.6.4",
"subtle",
"zeroize",
@@ -2453,11 +2453,11 @@ dependencies = [
[[package]]
name = "crypto-common"
version = "0.1.6"
version = "0.1.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1bfb12502f3fc46cca1bb51ac28df9d618d813cdc3d2f25b9fe775a34af26bb3"
checksum = "78c8292055d1c1df0cce5d180393dc8cce0abec0a7102adb6c7b1eef6016d60a"
dependencies = [
"generic-array 0.14.9",
"generic-array 0.14.7",
"typenum",
]
@@ -3664,7 +3664,7 @@ checksum = "9ed9a281f7bc9b7576e61468ba615a66a5c8cfdff42420a70aa82701a3b1e292"
dependencies = [
"block-buffer 0.10.4",
"const-oid 0.9.6",
"crypto-common 0.1.6",
"crypto-common 0.1.7",
"subtle",
]
@@ -3924,7 +3924,7 @@ dependencies = [
"crypto-bigint 0.5.5",
"digest 0.10.7",
"ff 0.13.1",
"generic-array 0.14.9",
"generic-array 0.14.7",
"group 0.13.0",
"hkdf 0.12.4",
"pem-rfc7468 0.7.0",
@@ -4148,9 +4148,9 @@ dependencies = [
[[package]]
name = "find-msvc-tools"
version = "0.1.10"
version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "26b73573e6edcd2af0cdf47bd6cb58f0b3839491263c314eaad1ccf24430e1de"
checksum = "d45db016d36b838f563236e9193d0ee6ce38f3f68b6c94e914b4929c96bbb890"
[[package]]
name = "findshlibs"
@@ -4369,9 +4369,9 @@ dependencies = [
[[package]]
name = "generic-array"
version = "0.14.9"
version = "0.14.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4bb6743198531e02858aeaea5398fcc883e71851fcbcb5a2f773e2fb6cb1edf2"
checksum = "85649ca51fd72272d7821adaf274ad91c288277713d9c18820d8499a7ff69e9a"
dependencies = [
"typenum",
"version_check",
@@ -4380,11 +4380,11 @@ dependencies = [
[[package]]
name = "generic-array"
version = "1.4.4"
version = "1.4.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ab4e5aa225bc56696909483320f0ff9b600f1a971b52e07a17d70f3d9b43254b"
checksum = "337d46834ee672ab3e48caca2cb0c78cc174fb12b3a68d0d88f99a0519a5e36e"
dependencies = [
"generic-array 0.14.9",
"generic-array 0.14.7",
"rustversion",
"typenum",
]
@@ -4726,9 +4726,9 @@ dependencies = [
[[package]]
name = "h2"
version = "0.4.15"
version = "0.4.16"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6cb093c84e8bd9b188d4c4a8cb6579fc016968d14c99882163cd3ff402a4f155"
checksum = "a9f37a958b41b3b19ee2707c06439c0e9e547e847223eb791ecb0cb821c65e27"
dependencies = [
"atomic-waker",
"bytes",
@@ -5028,9 +5028,9 @@ dependencies = [
[[package]]
name = "hotpath"
version = "0.23.2"
version = "0.23.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "62e810bedda5a467ef5c9b5c8a20763fefebc89b63ef36f7ee44a143085204a2"
checksum = "dce755d457a63bdd0c95e4c91511daad1b58b33209543b7f38027b676f387e5e"
dependencies = [
"arc-swap",
"async-channel",
@@ -5062,9 +5062,9 @@ dependencies = [
[[package]]
name = "hotpath-macros"
version = "0.23.2"
version = "0.23.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "01bdc59bfc1a9984bee2ff5da63b2f6fccbaa57cd9a4119d709524632bddf341"
checksum = "a903af89a8429cb07790c3818bc15270b394f80af1bc254e5ccf9c7de2961770"
dependencies = [
"proc-macro2",
"quote",
@@ -5073,15 +5073,15 @@ dependencies = [
[[package]]
name = "hotpath-macros-meta"
version = "0.23.2"
version = "0.23.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d9216e8a01abe1e1671c376dc8736fb1bf772d7a889538d25f9e1200120ced38"
checksum = "bcc0ab94ffbb2ee77f4a897df02b5a137a10cf24d69bda936e59aff4dd456e61"
[[package]]
name = "hotpath-meta"
version = "0.23.2"
version = "0.23.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "f22a9d20435fb79511b19dae37b3607224cd98f342a410702d84657cc38fc72f"
checksum = "053481f6cec8f775a3276c7f6e2f21123111d28261e4edc15ea7421c445964bb"
dependencies = [
"hotpath-macros-meta",
]
@@ -5280,9 +5280,9 @@ dependencies = [
[[package]]
name = "icu_collections"
version = "2.2.0"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2984d1cd16c883d7935b9e07e44071dca8d917fd52ecc02c04d5fa0b5a3f191c"
checksum = "fa68d21081c4a05d5a901a1c62add574c77048b6a1c67be3b50ce0b60d4ca513"
dependencies = [
"displaydoc",
"potential_utf",
@@ -5294,9 +5294,9 @@ dependencies = [
[[package]]
name = "icu_locale_core"
version = "2.2.0"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92219b62b3e2b4d88ac5119f8904c10f8f61bf7e95b640d25ba3075e6cac2c29"
checksum = "d56e28588da92eee5c3201a6eff33fabdd49b62269c8938d4ff050ce4d900deb"
dependencies = [
"displaydoc",
"litemap",
@@ -5307,9 +5307,9 @@ dependencies = [
[[package]]
name = "icu_normalizer"
version = "2.2.0"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c56e5ee99d6e3d33bd91c5d85458b6005a22140021cc324cea84dd0e72cff3b4"
checksum = "12f9cf5f235641ed274641dd81c3f28d870e276763d0797aeeab72317b1c646f"
dependencies = [
"icu_collections",
"icu_normalizer_data",
@@ -5321,16 +5321,17 @@ dependencies = [
[[package]]
name = "icu_normalizer_data"
version = "2.2.0"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "da3be0ae77ea334f4da67c12f149704f19f81d1adf7c51cf482943e84a2bad38"
checksum = "1563da1ed3e0b3bf3d74c9b85917ac9c56464d2f57242270c09c9e752f8021a0"
[[package]]
name = "icu_properties"
version = "2.2.0"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bee3b67d0ea5c2cca5003417989af8996f8604e34fb9ddf96208a033901e70de"
checksum = "7e7ca276ad3145661a65914e6daf131ca5120cd3dcee8f8f3214b8875184a148"
dependencies = [
"displaydoc",
"icu_collections",
"icu_locale_core",
"icu_properties_data",
@@ -5341,15 +5342,15 @@ dependencies = [
[[package]]
name = "icu_properties_data"
version = "2.2.0"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8e2bbb201e0c04f7b4b3e14382af113e17ba4f63e2c9d2ee626b720cbce54a14"
checksum = "e590f038c1464a96894fd6d10127e90a8be4509f56ff7ecef851b15cee0b7caa"
[[package]]
name = "icu_provider"
version = "2.2.0"
version = "2.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "139c4cf31c8b5f33d7e199446eff9c1e02decfc2f0eec2c8d71f65befa45b421"
checksum = "92a7ed671a6aad807a8651a2e1782a6598fda9ce5185dd8158549e95a91c6428"
dependencies = [
"displaydoc",
"icu_locale_core",
@@ -5417,7 +5418,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "879f10e63c20629ecabbb64a8010319738c66a5cd0c29b02d63d272b03751d01"
dependencies = [
"block-padding 0.3.3",
"generic-array 0.14.9",
"generic-array 0.14.7",
]
[[package]]
@@ -5968,9 +5969,9 @@ dependencies = [
[[package]]
name = "libredox"
version = "0.1.19"
version = "0.1.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2026a5056764a10b2bf5d56488cba40da507f5493a6a429340e2004d9ed085fa"
checksum = "28d0a00925a9f930d679b6789b721e3a7f9ed110f41b86d2497caa780c3a070a"
dependencies = [
"libc",
]
@@ -6033,9 +6034,9 @@ checksum = "32a66949e030da00e8c7d4434b251670a91556f4144941d37452769c25d58a53"
[[package]]
name = "litemap"
version = "0.8.2"
version = "0.8.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "92daf443525c4cce67b150400bc2316076100ce0b3686209eb8cf3c31612e6f0"
checksum = "47d9d19d1d6efa0109d2f65ff4c85cddd50bd572e5a00127ab10987290bcefae"
[[package]]
name = "local-ip-address"
@@ -6482,9 +6483,9 @@ dependencies = [
[[package]]
name = "mqttbytes-core-next"
version = "0.33.3"
version = "0.34.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3ff7ae19c74aba9e0ed6e4071cd52aa364e020076fa3cc6ef17e43662f756f3c"
checksum = "366b6ba2b4209ca4bc5ac731ccddf570d09831981eed07e5fbd63564cf0cf1aa"
dependencies = [
"bytes",
"thiserror 2.0.20",
@@ -6885,7 +6886,7 @@ version = "5.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "51e219e79014df21a225b1860a479e2dcd7cbd9130f4defd4bd0e191ea31d67d"
dependencies = [
"base64 0.21.7",
"base64 0.22.1",
"chrono",
"getrandom 0.2.17",
"http 1.5.0",
@@ -7344,9 +7345,9 @@ dependencies = [
[[package]]
name = "pageant"
version = "0.2.1"
version = "0.2.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "4f3a5ae18f65a85c67a77d18d42d3606c07948e3c17c1e5f74852b26589e88a5"
checksum = "3adadc44070da6f464b0918655a12f5792c156e088d8c4082d13e27d94c3e791"
dependencies = [
"base16ct 1.0.0",
"byteorder",
@@ -7728,9 +7729,9 @@ dependencies = [
[[package]]
name = "pkg-config"
version = "0.3.33"
version = "0.3.34"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "19f132c84eca552bf34cab8ec81f1c1dcc229b811638f9d283dceabe58c5569e"
checksum = "f6b464fbc74e149a392436b17d523f769e057cb6877f6a5c4618bc6f11800548"
[[package]]
name = "plotters"
@@ -7836,9 +7837,9 @@ dependencies = [
[[package]]
name = "potential_utf"
version = "0.1.5"
version = "0.1.6"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0103b1cef7ec0cf76490e969665504990193874ea05c85ff9bab8b911d0a0564"
checksum = "d83eb9bc6d8e5cf568e7a1101d60ee05e81ed50ea106026f3d18deeb046d7661"
dependencies = [
"zerovec",
]
@@ -8046,7 +8047,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf"
dependencies = [
"heck 0.5.0",
"itertools 0.10.5",
"itertools 0.14.0",
"log",
"multimap",
"once_cell",
@@ -8066,7 +8067,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042"
dependencies = [
"heck 0.5.0",
"itertools 0.10.5",
"itertools 0.14.0",
"log",
"multimap",
"petgraph 0.8.3",
@@ -8087,7 +8088,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d"
dependencies = [
"anyhow",
"itertools 0.10.5",
"itertools 0.14.0",
"proc-macro2",
"quote",
"syn 2.0.119",
@@ -8100,7 +8101,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf"
dependencies = [
"anyhow",
"itertools 0.10.5",
"itertools 0.14.0",
"proc-macro2",
"quote",
"syn 2.0.119",
@@ -8216,7 +8217,7 @@ dependencies = [
"reqwest",
"serde_json",
"smallvec",
"spin 0.12.2",
"spin 0.12.3",
"symbolic-demangle",
"tempfile",
"thiserror 2.0.20",
@@ -8285,9 +8286,9 @@ dependencies = [
[[package]]
name = "quinn-proto"
version = "0.11.16"
version = "0.11.17"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "2f4bfc015262b9df63c8845072ce59068853ff5872180c2ce2f13038b970e560"
checksum = "04759210543be93709136e28212294a659ef5001836ff4eab4d663e4529bba83"
dependencies = [
"aws-lc-rs",
"bytes",
@@ -8553,9 +8554,9 @@ dependencies = [
[[package]]
name = "redis"
version = "1.5.0"
version = "1.6.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3257df217f7eab0044627a268c9cc6cdb60c0c421c88f83ac41c4e31520b6b84"
checksum = "e37a4ca5c6ca42aa3e6df2fd32b987a65d32a4c2159a6f3fe0fd1df306a2658f"
dependencies = [
"arc-swap",
"arcstr",
@@ -8567,7 +8568,7 @@ dependencies = [
"futures-channel",
"futures-util",
"itoa",
"num-bigint 0.4.8",
"num-bigint 0.5.1",
"percent-encoding",
"pin-project-lite",
"rustls",
@@ -8868,9 +8869,9 @@ dependencies = [
[[package]]
name = "rumqttc-core-next"
version = "0.33.3"
version = "0.34.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "7d7d9205738dd41a2546e82d27a634d07d8b303dcf7558565ff70caf3ceb0f9c"
checksum = "249896ab27ed630590971738264baa8f722f18965d2e387c706c40a3c2a572cc"
dependencies = [
"async-tungstenite",
"futures-io",
@@ -8886,18 +8887,18 @@ dependencies = [
[[package]]
name = "rumqttc-next"
version = "0.33.3"
version = "0.34.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "ed1bad2180ff539da671da9a996152a921bc5316eb6d8a9cc3bd441653138b08"
checksum = "477c9bbfba8f3aecc7aad31c6de2eacb75822efaa18e7aeecb8d3d8e534fbf07"
dependencies = [
"rumqttc-v5-next",
]
[[package]]
name = "rumqttc-v5-next"
version = "0.33.3"
version = "0.34.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "229576cbedfa9089f90c17c9454e9429ac1e89cdd223bac5cb39d837593f79bc"
checksum = "3dfa6ddcc7a7dd5688f9bf78d8f81cb94f367bce56c055d8d94cf81ecb0518bf"
dependencies = [
"async-tungstenite",
"bytes",
@@ -8920,9 +8921,9 @@ dependencies = [
[[package]]
name = "russh"
version = "0.62.6"
version = "0.62.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b41043523e0edcbd4e31d00903e26f12994f63b21bae9904f7405c1ed92752a5"
checksum = "9decb68e4e44e1079700e54f17c8f23806ec53d7e0db73ab1c71d9dabc666812"
dependencies = [
"aes 0.9.2",
"aws-lc-rs",
@@ -8945,7 +8946,7 @@ dependencies = [
"enum_dispatch",
"flate2",
"futures",
"generic-array 1.4.4",
"generic-array 1.4.5",
"getrandom 0.4.3",
"ghash",
"hex-literal",
@@ -9280,6 +9281,7 @@ dependencies = [
"s3s",
"serde",
"serde_json",
"smallvec",
"tokio",
"tonic",
"tracing",
@@ -9463,6 +9465,7 @@ dependencies = [
"tokio-stream",
"tokio-util",
"tonic",
"tonic-prost",
"tower",
"tracing",
"tracing-core",
@@ -9490,7 +9493,7 @@ dependencies = [
"parking_lot",
"rayon",
"smallvec",
"spin 0.12.2",
"spin 0.12.3",
]
[[package]]
@@ -9536,6 +9539,8 @@ version = "1.0.0-rc.2"
dependencies = [
"async-trait",
"base64 0.23.1",
"bytes",
"crc-fast",
"futures",
"hotpath",
"http 1.5.0",
@@ -9608,7 +9613,6 @@ version = "1.0.0-rc.2"
dependencies = [
"bytes",
"hotpath",
"memmap2",
"rustfs-io-metrics",
"thiserror 2.0.20",
"tokio",
@@ -10251,6 +10255,7 @@ dependencies = [
"rustfs-ecstore",
"rustfs-filemeta",
"rustfs-lock",
"rustfs-s3-types",
"rustfs-storage-api",
"rustfs-utils",
"s3s",
@@ -10484,15 +10489,10 @@ dependencies = [
name = "rustfs-zip"
version = "1.0.0-rc.2"
dependencies = [
"astral-tokio-tar",
"async-compression",
"criterion",
"hotpath",
"tempfile",
"thiserror 2.0.20",
"tokio",
"tokio-stream",
"zip",
]
[[package]]
@@ -10678,7 +10678,7 @@ checksum = "9774ba4a74de5f7b1c1451ed6cd5285a32eddb5cccb8cc655a4e50009e06477f"
[[package]]
name = "s3s"
version = "0.14.1"
source = "git+https://github.com/rustfs/s3s.git?rev=d7028511a53f69d41ed3c69f36899f9b1aede647#d7028511a53f69d41ed3c69f36899f9b1aede647"
source = "git+https://github.com/rustfs/s3s.git?rev=d358a68783096df1db0c3e314127f2704603b29e#d358a68783096df1db0c3e314127f2704603b29e"
dependencies = [
"arc-swap",
"arrayvec",
@@ -10706,6 +10706,7 @@ dependencies = [
"numeric_cast",
"pin-project-lite",
"quick-xml",
"regex",
"serde",
"serde_json",
"serde_urlencoded",
@@ -10831,7 +10832,7 @@ checksum = "d3e97a565f76233a6003f9f5c54be1d9c5bdfa3eccfb189469f11ec4901c47dc"
dependencies = [
"base16ct 0.2.0",
"der 0.7.10",
"generic-array 0.14.9",
"generic-array 0.14.7",
"pkcs8 0.10.2",
"subtle",
"zeroize",
@@ -11395,9 +11396,9 @@ checksum = "023a211cb3138dbc438680b32560ad89f699977624c9f8dbb95a47d5b4c07dd3"
[[package]]
name = "spin"
version = "0.12.2"
version = "0.12.3"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8abadc99fd9c7bbb7d0ca2b31d72a067d0c0dcd7aad25ab8cac71ba91417694b"
checksum = "0134f9043ed38b087ac4f7d4af44c79e2c9e5094421fe3164f435ce585953b10"
dependencies = [
"lock_api",
]
@@ -11973,9 +11974,9 @@ dependencies = [
[[package]]
name = "tinystr"
version = "0.8.3"
version = "0.8.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "c8323304221c2a851516f22236c5722a72eaa19749016521d6dff0824447d96d"
checksum = "b1e27c91459209c2986af3dcf603a5a74a4368754ce37414f59acc971167f643"
dependencies = [
"displaydoc",
"zerovec",
@@ -12649,9 +12650,9 @@ checksum = "06abde3611657adf66d383f00b093d7faecc7fa57071cce2578660c9f1010821"
[[package]]
name = "uuid"
version = "1.24.0"
version = "1.24.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bf3923a6f5c4c6382e0b653c4117f48d631ea17f38ed86e2a828e6f7412f5239"
checksum = "2cefc03fd367c0c6d4305de1b312cf00248c4114f4a0418ce6a6af769e3b0bd9"
dependencies = [
"getrandom 0.4.3",
"js-sys",
@@ -13166,9 +13167,9 @@ dependencies = [
[[package]]
name = "writeable"
version = "0.6.3"
version = "0.6.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "1ffae5123b2d3fc086436f8834ae3ab053a283cfac8fe0a0b8eaae044768a4c4"
checksum = "3ad82d2a33cdc9674dc7465672f271e096168fcdbe0f799d9e6db8c5892679dc"
[[package]]
name = "x509-cert"
@@ -13348,9 +13349,9 @@ dependencies = [
[[package]]
name = "zerotrie"
version = "0.2.4"
version = "0.2.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "0f9152d31db0792fa83f70fb2f83148effb5c1f5b8c7686c3459e361d9bc20bf"
checksum = "4ea269c3bd32f0a32c321907a2ae912ba6f4649bb0fc764a15627e99a7095a3f"
dependencies = [
"displaydoc",
"yoke",
@@ -13359,9 +13360,9 @@ dependencies = [
[[package]]
name = "zerovec"
version = "0.11.6"
version = "0.11.7"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "90f911cbc359ab6af17377d242225f4d75119aec87ea711a880987b18cd7b239"
checksum = "94b5c6b5976d66c1d703c4fd17d3f5e43c8cedaacf604961b171adc7130896d8"
dependencies = [
"yoke",
"zerofrom",
@@ -13370,13 +13371,13 @@ dependencies = [
[[package]]
name = "zerovec-derive"
version = "0.11.3"
version = "0.11.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "625dc425cab0dca6dc3c3319506e6593dcb08a9f387ea3b284dbd52a92c40555"
checksum = "9f212a141d820099d57ffafb9569be9617a6f27d3dc881fbee8fb56642f917a9"
dependencies = [
"proc-macro2",
"quote",
"syn 2.0.119",
"syn 3.0.3",
]
[[package]]
+9 -9
View File
@@ -228,9 +228,9 @@ atoi = "3.1.0"
atomic_enum = "0.3.0"
aws-config = { version = "1.10.1" }
aws-credential-types = { version = "1.3.0" }
aws-sdk-kms = { default-features = false, version = "1.114.0" }
aws-sdk-s3 = { default-features = false, version = "1.141.0" }
aws-sdk-sts = { default-features = false, version = "1.110.0" }
aws-sdk-kms = { default-features = false, version = "1.115.0" }
aws-sdk-s3 = { default-features = false, version = "1.142.0" }
aws-sdk-sts = { default-features = false, version = "1.111.0" }
aws-smithy-http-client = { default-features = false, version = "1.3.0" }
aws-smithy-runtime-api = { version = "1.14.0" }
aws-smithy-types = { version = "1.6.2" }
@@ -284,13 +284,13 @@ rayon = "1.12.0"
reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "8.0.2" }
reed-solomon-simd = "3.1.0"
regex = { version = "1.13.1" }
rumqttc = { package = "rumqttc-next", version = "0.33.3" }
redis = { version = "1.5.0" }
rumqttc = { package = "rumqttc-next", version = "0.34.0" }
redis = { version = "1.6.0" }
rustify = { version = "0.7", default-features = false }
rustix = { version = "1.1.4" }
rust-embed = { version = "8.12.0" }
rustc-hash = { version = "2.1.3" }
s3s = { git = "https://github.com/rustfs/s3s.git", rev = "d7028511a53f69d41ed3c69f36899f9b1aede647" }
s3s = { git = "https://github.com/rustfs/s3s.git", rev = "d358a68783096df1db0c3e314127f2704603b29e" }
serial_test = "4.0.1"
shadow-rs = { default-features = false, version = "2.0.0" }
siphasher = "1.0.3"
@@ -313,7 +313,7 @@ tracing-subscriber = { version = "0.3.23" }
transform-stream = "0.3.1"
url = "2.5.8"
urlencoding = "2.1.3"
uuid = { version = "1.24.0" }
uuid = { version = "1.24.1" }
vaultrs = { version = "0.8.0" }
tar = "0.4.46"
walkdir = "2.5.0"
@@ -341,7 +341,7 @@ libunftp = { version = "0.23.0" }
unftp-core = "0.1.0"
suppaftp = { version = "10.0.1" }
rcgen = { version = "0.14.9", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
russh = { version = "0.62.6" }
russh = { version = "0.62.7" }
russh-sftp = "2.4.0"
# WebDAV
@@ -350,7 +350,7 @@ dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
mimalloc = { version = "0.1.52", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11" }
libmimalloc-sys = { version = "0.1.49", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11", features = ["extended"] }
hotpath = { version = "0.23.2", default-features = false }
hotpath = { version = "0.23.3", default-features = false }
# Snapshot testing for output format regression detection
insta = { version = "1.48" }
+1
View File
@@ -42,6 +42,7 @@ chrono = { workspace = true, features = ["serde"] }
jiff = { workspace = true, features = ["serde"] }
metrics = { workspace = true }
serde = { workspace = true, features = ["derive"] }
smallvec = { workspace = true }
rmp-serde = { workspace = true }
s3s = { workspace = true, features = ["minio"] }
tracing = { workspace = true }
+27
View File
@@ -224,6 +224,13 @@ pub struct HealOpts {
pub enum HealAdmissionDropReason {
QueueFull,
PolicyDropped,
/// HS-06: an admin heal start overlaps (same bucket with mutually
/// containing prefixes, or the same erasure set) an already running or
/// queued task. Only produced when RUSTFS_HEAL_OVERLAP_POLICY=minio_error.
AlreadyRunning,
/// HS-06: same as [`Self::AlreadyRunning`] but for paths that merely
/// contain (or are contained by) the active task's path.
OverlappingPaths,
}
impl HealAdmissionDropReason {
@@ -231,6 +238,8 @@ impl HealAdmissionDropReason {
match self {
Self::QueueFull => "queue_full",
Self::PolicyDropped => "policy_dropped",
Self::AlreadyRunning => "already_running",
Self::OverlappingPaths => "overlapping_paths",
}
}
}
@@ -287,6 +296,9 @@ pub enum HealRequestSource {
Scanner,
AutoHeal,
ReadRepair,
/// Mission Repair Feed: intents delivered by error paths and replayed
/// from the durable MRF journal.
Mrf,
}
impl HealRequestSource {
@@ -297,6 +309,7 @@ impl HealRequestSource {
Self::Scanner => "scanner",
Self::AutoHeal => "auto_heal",
Self::ReadRepair => "read_repair",
Self::Mrf => "mrf",
}
}
}
@@ -313,6 +326,9 @@ pub enum HealChannelCommand {
Query {
heal_path: String,
client_token: String,
/// Incremental result cursor (HS-06): only items with a sequence
/// greater than this are returned; `None` keeps the full snapshot.
since_seq: Option<u64>,
response_tx: oneshot::Sender<Result<HealChannelResponse, String>>,
},
/// Cancel heal task
@@ -518,10 +534,21 @@ async fn receive_heal_channel_response(
/// Send heal query request
pub async fn query_heal_status(heal_path: String, client_token: String) -> Result<HealChannelResponse, String> {
query_heal_status_since(heal_path, client_token, None).await
}
/// Incremental heal query (HS-06): pass the client's last seen sequence
/// number to receive only newer result items.
pub async fn query_heal_status_since(
heal_path: String,
client_token: String,
since_seq: Option<u64>,
) -> Result<HealChannelResponse, String> {
let (response_tx, response_rx) = oneshot::channel();
send_heal_command(HealChannelCommand::Query {
heal_path,
client_token,
since_seq,
response_tx,
})
.await?;
+2
View File
@@ -17,8 +17,10 @@ pub mod globals;
pub mod heal_channel;
pub mod last_minute;
pub mod metrics;
pub mod mrf_channel;
mod readiness;
pub mod table_catalog;
pub mod trace_bus;
pub use globals::*;
pub use readiness::{GlobalReadiness, SystemStage};
+203
View File
@@ -0,0 +1,203 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Mission Repair Feed (MRF) intent channel.
//!
//! Producers on error paths (read decode failure, scanner metadata
//! corruption, partial-write recovery) hand a lightweight [`MrfIntent`] to the
//! heal crate through a global bounded channel. Delivery is strictly
//! non-blocking: `try_send_mrf_intent` never awaits and drops the intent
//! (counting it) when the channel is full or uninitialized — losing one heal
//! hint is always preferred over stalling an IO path. Durable replay of
//! unconsumed intents is the consumer's job (see `rustfs-heal`
//! `heal::mrf_queue`), mirroring MinIO's `.heal/mrf/list.bin`.
use std::sync::{
Arc, OnceLock,
atomic::{AtomicBool, Ordering},
};
use tokio::sync::mpsc;
use uuid::Uuid;
/// Bounded capacity of the global MRF channel. Backpressure is resolved by
/// dropping (and counting) intents, never by blocking the producer.
const MRF_CHANNEL_CAPACITY: usize = 8192;
/// Why an intent was produced. Drives the heal priority mapping on the
/// consumer side (DecodeFailure -> Urgent, MetadataCorruption -> High,
/// PartialWrite -> Normal).
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
pub enum MrfKind {
/// Erasure decode failed while serving a read (read path).
DecodeFailure,
/// Scanner classified object metadata as corrupt.
MetadataCorruption,
/// A write left the object with fewer committed shards than the set size.
PartialWrite,
}
impl MrfKind {
pub const fn as_str(self) -> &'static str {
match self {
MrfKind::DecodeFailure => "decode-failure",
MrfKind::MetadataCorruption => "metadata-corruption",
MrfKind::PartialWrite => "partial-write",
}
}
}
/// One repair intent. Kept deliberately small so the in-memory queue and the
/// journal stay bounded; `bucket`/`object` are `Arc<str>` so re-arming an
/// intent never re-allocates the strings.
#[derive(Clone, Debug)]
pub struct MrfIntent {
pub bucket: Arc<str>,
pub object: Arc<str>,
/// Version the intent targets, as raw UUID bytes.
pub version_id: Option<[u8; 16]>,
pub kind: MrfKind,
pub enqueued_at_ms: u64,
/// Times this intent has already been offered to the heal manager.
/// Dropped by the consumer once it reaches `MRF_MAX_ATTEMPTS`.
pub attempts: u8,
}
/// Consumer-side retry ceiling before an intent is given up on.
pub const MRF_MAX_ATTEMPTS: u8 = 3;
impl MrfIntent {
/// Rough in-memory footprint used by the queue's byte budget.
pub fn estimated_bytes(&self) -> usize {
// Struct + strings + version bytes; buckets and objects are usually
// far below this bound, so rounding up keeps the budget conservative.
64 + self.bucket.len() + self.object.len()
}
}
static GLOBAL_MRF_SENDER: OnceLock<mpsc::Sender<MrfIntent>> = OnceLock::new();
/// Delivery kill-switch, set from `RUSTFS_HEAL_MRF_ENABLE`. Producers check
/// this before touching the channel so the disabled path stays allocation- and
/// sync-free.
static MRF_DELIVERY_ENABLED: AtomicBool = AtomicBool::new(true);
/// Override delivery (used at heal-runtime startup from configuration).
pub fn set_mrf_delivery_enabled(enabled: bool) {
MRF_DELIVERY_ENABLED.store(enabled, Ordering::Relaxed);
}
/// Whether producers currently deliver intents.
pub fn mrf_delivery_enabled() -> bool {
MRF_DELIVERY_ENABLED.load(Ordering::Relaxed)
}
/// Create the global MRF channel and return the consumer half. Fails if the
/// channel is already initialized (the heal runtime is a singleton).
pub fn init_mrf_channel() -> Result<mpsc::Receiver<MrfIntent>, &'static str> {
let (sender, receiver) = mpsc::channel(MRF_CHANNEL_CAPACITY);
GLOBAL_MRF_SENDER
.set(sender)
.map_err(|_| "MRF channel sender already initialized")?;
Ok(receiver)
}
/// Best-effort, non-blocking intent delivery from an error path.
///
/// Returns `true` when the intent was accepted into the channel. `false`
/// means the intent was dropped (feature disabled, channel not yet
/// initialized, or channel full) — callers must not retry or await; the
/// existing read-repair / scanner heal paths remain the safety net.
///
/// This runs on IO error paths, so it stays synchronous and cheap: one
/// bounded allocation for the two `Arc<str>` handles plus the channel slot.
pub fn try_send_mrf_intent(kind: MrfKind, bucket: &str, object: &str, version_id: Option<Uuid>) -> bool {
if !mrf_delivery_enabled() {
return false;
}
let Some(sender) = GLOBAL_MRF_SENDER.get() else {
return false;
};
let intent = MrfIntent {
bucket: Arc::from(bucket),
object: Arc::from(object),
version_id: version_id.map(|vid| *vid.as_bytes()),
kind,
enqueued_at_ms: unix_now_ms(),
attempts: 0,
};
sender.try_send(intent).is_ok()
}
fn unix_now_ms() -> u64 {
// Kept trivial: the timestamp is diagnostic metadata only; wall-clock
// failure would be a bug rather than something to handle here.
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0)
}
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn intents_estimate_is_conservative() {
let intent = MrfIntent {
bucket: Arc::from("bucket"),
object: Arc::from("object"),
version_id: Some([0u8; 16]),
kind: MrfKind::DecodeFailure,
enqueued_at_ms: 0,
attempts: 0,
};
assert!(intent.estimated_bytes() >= intent.bucket.len() + intent.object.len());
}
#[tokio::test]
async fn try_send_delivers_and_respects_capacity() {
let mut receiver = init_mrf_channel().expect("first initialization should succeed");
assert!(init_mrf_channel().is_err(), "double initialization must fail");
assert!(try_send_mrf_intent(MrfKind::DecodeFailure, "b", "o", Some(Uuid::nil())));
let intent = receiver.recv().await.expect("intent should arrive");
assert_eq!(intent.kind, MrfKind::DecodeFailure);
assert_eq!(intent.bucket.as_ref(), "b");
// Disable delivery: producers become no-ops.
set_mrf_delivery_enabled(false);
assert!(!try_send_mrf_intent(MrfKind::PartialWrite, "b", "o", None));
set_mrf_delivery_enabled(true);
// Fill the bounded channel past capacity: excess intents are dropped,
// never blocking.
let mut accepted = 0;
for _ in 0..(MRF_CHANNEL_CAPACITY + 64) {
if try_send_mrf_intent(MrfKind::PartialWrite, "b", "o", None) {
accepted += 1;
}
}
assert_eq!(accepted, MRF_CHANNEL_CAPACITY);
}
#[test]
fn try_send_without_channel_is_false() {
// This test may run after the tokio test above in the same process;
// the singleton semantics make a clean "uninitialized" case hard, so
// assert the flag-off behavior only.
set_mrf_delivery_enabled(false);
assert!(!try_send_mrf_intent(MrfKind::MetadataCorruption, "b", "o", None));
set_mrf_delivery_enabled(true);
}
}
+333
View File
@@ -0,0 +1,333 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use smallvec::SmallVec;
use std::{
sync::{
Arc, OnceLock,
atomic::{AtomicUsize, Ordering},
},
time::{Duration, SystemTime},
};
use tokio::sync::broadcast;
const DEFAULT_TRACE_BUS_CAPACITY: usize = 1024;
const TRACE_ATTR_INLINE_CAPACITY: usize = 8;
static GLOBAL_TRACE_BUS: OnceLock<TraceBus> = OnceLock::new();
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum TraceKind {
Heal,
Scanner,
}
impl TraceKind {
pub const fn as_str(self) -> &'static str {
match self {
Self::Heal => "heal",
Self::Scanner => "scanner",
}
}
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum TraceFunc {
HealTask,
HealBucket,
HealObject,
HealCheckAbandonedParts,
HealErasureSetPage,
ScannerFolder,
ScannerIlmAction,
ScannerHealCandidate,
Dropped,
}
impl TraceFunc {
pub const fn as_str(self) -> &'static str {
match self {
Self::HealTask => "heal.Task",
Self::HealBucket => "heal.Bucket",
Self::HealObject => "heal.Object",
Self::HealCheckAbandonedParts => "heal.CheckAbandonedParts",
Self::HealErasureSetPage => "heal.ErasureSetPage",
Self::ScannerFolder => "scanner.Folder",
Self::ScannerIlmAction => "scanner.IlmAction",
Self::ScannerHealCandidate => "scanner.HealCandidate",
Self::Dropped => "trace.Dropped",
}
}
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub enum TraceVal {
Bool(bool),
U64(u64),
I64(i64),
Str(Arc<str>),
}
impl From<bool> for TraceVal {
fn from(value: bool) -> Self {
Self::Bool(value)
}
}
impl From<u64> for TraceVal {
fn from(value: u64) -> Self {
Self::U64(value)
}
}
impl From<i64> for TraceVal {
fn from(value: i64) -> Self {
Self::I64(value)
}
}
impl From<&str> for TraceVal {
fn from(value: &str) -> Self {
Self::Str(Arc::from(value))
}
}
impl From<String> for TraceVal {
fn from(value: String) -> Self {
Self::Str(Arc::from(value))
}
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct TraceAttr {
pub key: &'static str,
pub value: TraceVal,
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct TraceEvent {
pub kind: TraceKind,
pub func: TraceFunc,
pub time: SystemTime,
pub bucket: Option<Arc<str>>,
pub object: Option<Arc<str>>,
pub duration: Duration,
pub bytes: u64,
pub attrs: SmallVec<[TraceAttr; TRACE_ATTR_INLINE_CAPACITY]>,
}
impl TraceEvent {
pub fn new(kind: TraceKind, func: TraceFunc) -> Self {
Self {
kind,
func,
time: SystemTime::now(),
bucket: None,
object: None,
duration: Duration::ZERO,
bytes: 0,
attrs: SmallVec::new(),
}
}
pub fn with_bucket(mut self, bucket: impl Into<Arc<str>>) -> Self {
self.bucket = Some(bucket.into());
self
}
pub fn with_object(mut self, object: impl Into<Arc<str>>) -> Self {
self.object = Some(object.into());
self
}
pub fn with_duration(mut self, duration: Duration) -> Self {
self.duration = duration;
self
}
pub fn with_bytes(mut self, bytes: u64) -> Self {
self.bytes = bytes;
self
}
pub fn with_attr(mut self, key: &'static str, value: impl Into<TraceVal>) -> Self {
self.attrs.push(TraceAttr {
key,
value: value.into(),
});
self
}
}
#[derive(Debug)]
pub struct TraceBus {
sender: broadcast::Sender<Arc<TraceEvent>>,
subscriber_count: Arc<AtomicUsize>,
}
impl TraceBus {
pub fn new(capacity: usize) -> Self {
let capacity = capacity.max(1);
let (sender, _receiver) = broadcast::channel(capacity);
Self {
sender,
subscriber_count: Arc::new(AtomicUsize::new(0)),
}
}
pub fn subscriber_count(&self) -> usize {
self.subscriber_count.load(Ordering::Acquire)
}
pub fn subscribe(&self) -> TraceSubscription {
let receiver = self.sender.subscribe();
self.subscriber_count.fetch_add(1, Ordering::AcqRel);
TraceSubscription {
receiver,
subscriber_count: Arc::clone(&self.subscriber_count),
}
}
pub fn emit(&self, build: impl FnOnce() -> TraceEvent) -> bool {
if self.subscriber_count() == 0 {
return false;
}
self.sender.send(Arc::new(build())).is_ok()
}
}
impl Default for TraceBus {
fn default() -> Self {
Self::new(DEFAULT_TRACE_BUS_CAPACITY)
}
}
#[derive(Debug)]
pub struct TraceSubscription {
receiver: broadcast::Receiver<Arc<TraceEvent>>,
subscriber_count: Arc<AtomicUsize>,
}
impl TraceSubscription {
pub async fn recv(&mut self) -> Result<Arc<TraceEvent>, broadcast::error::RecvError> {
self.receiver.recv().await
}
pub fn try_recv(&mut self) -> Result<Arc<TraceEvent>, broadcast::error::TryRecvError> {
self.receiver.try_recv()
}
}
impl Drop for TraceSubscription {
fn drop(&mut self) {
self.subscriber_count.fetch_sub(1, Ordering::AcqRel);
}
}
pub fn global_trace_bus() -> &'static TraceBus {
GLOBAL_TRACE_BUS.get_or_init(TraceBus::default)
}
pub fn subscribe_trace_events() -> TraceSubscription {
global_trace_bus().subscribe()
}
pub fn trace_emit(build: impl FnOnce() -> TraceEvent) -> bool {
global_trace_bus().emit(build)
}
pub fn trace_subscriber_count() -> usize {
global_trace_bus().subscriber_count()
}
#[cfg(test)]
mod tests {
use super::*;
use std::sync::atomic::AtomicUsize;
#[test]
fn trace_emit_skips_builder_without_subscribers() {
let bus = TraceBus::new(4);
let built = AtomicUsize::new(0);
let sent = bus.emit(|| {
built.fetch_add(1, Ordering::Relaxed);
TraceEvent::new(TraceKind::Heal, TraceFunc::HealTask)
});
assert!(!sent);
assert_eq!(built.load(Ordering::Relaxed), 0);
}
#[tokio::test]
async fn trace_subscriber_receives_event() {
let bus = TraceBus::new(4);
let mut subscription = bus.subscribe();
assert!(bus.emit(|| {
TraceEvent::new(TraceKind::Heal, TraceFunc::HealObject)
.with_bucket("bucket")
.with_object("object")
.with_duration(Duration::from_millis(7))
.with_bytes(11)
.with_attr("dry", true)
}));
let event = subscription
.recv()
.await
.expect("subscriber should receive emitted trace event");
assert_eq!(event.kind, TraceKind::Heal);
assert_eq!(event.func, TraceFunc::HealObject);
assert_eq!(event.bucket.as_deref(), Some("bucket"));
assert_eq!(event.object.as_deref(), Some("object"));
assert_eq!(event.duration, Duration::from_millis(7));
assert_eq!(event.bytes, 11);
assert_eq!(
event.attrs.as_slice(),
&[TraceAttr {
key: "dry",
value: TraceVal::Bool(true)
}]
);
}
#[test]
fn trace_subscription_drop_decrements_count() {
let bus = TraceBus::new(4);
let subscription = bus.subscribe();
assert_eq!(bus.subscriber_count(), 1);
drop(subscription);
assert_eq!(bus.subscriber_count(), 0);
}
#[tokio::test]
async fn lagged_subscriber_drops_events_without_blocking_publishers() {
let bus = TraceBus::new(2);
let mut subscription = bus.subscribe();
for index in 0_u64..4 {
assert!(bus.emit(|| { TraceEvent::new(TraceKind::Scanner, TraceFunc::ScannerFolder).with_attr("index", index) }));
}
let err = subscription
.recv()
.await
.expect_err("receiver should observe lag instead of blocking publishers");
assert!(matches!(err, broadcast::error::RecvError::Lagged(_)));
}
}
+2 -3
View File
@@ -14,9 +14,8 @@
//! Shared backpressure policy type.
//!
//! The runtime backpressure implementation (byte-watermark pipes and
//! monitors) lives in `rustfs/src/storage/backpressure.rs`; this module only
//! carries the watermark policy type that implementation shares.
//! This module only carries the watermark policy; the admission primitive it
//! projects into lives in `rustfs-io-core`.
use rustfs_io_core::BackpressureConfig as CoreBackpressureConfig;
+37
View File
@@ -177,3 +177,40 @@ pub const DEFAULT_HEAL_MAINLINE_WRITE_UTILIZATION_HIGH_PERCENT: usize = 80;
/// Default foreground pressure recheck delay for heal scheduler, in milliseconds.
pub const DEFAULT_HEAL_MAINLINE_MAX_SLEEP_MS: u64 = 250;
/// Environment variable that toggles the MRF (mission repair feed) intent
/// pipeline: error paths deliver repair intents to the heal runtime, and
/// unconsumed intents are replayed from the durable journal after a restart.
pub const ENV_HEAL_MRF_ENABLE: &str = "RUSTFS_HEAL_MRF_ENABLE";
/// Environment variable for the MRF in-memory queue capacity (intent count).
pub const ENV_HEAL_MRF_QUEUE_SIZE: &str = "RUSTFS_HEAL_MRF_QUEUE_SIZE";
/// Environment variable for the MRF journal byte budget. The journal is
/// compacted once its on-disk size crosses this bound.
pub const ENV_HEAL_MRF_JOURNAL_MAX_BYTES: &str = "RUSTFS_HEAL_MRF_JOURNAL_MAX_BYTES";
/// Environment variable for the MRF journal replay batch size (intents per
/// replay push round).
pub const ENV_HEAL_MRF_REPLAY_BATCH: &str = "RUSTFS_HEAL_MRF_REPLAY_BATCH";
/// Default behavior keeps the MRF intent pipeline enabled.
pub const DEFAULT_HEAL_MRF_ENABLE: bool = true;
/// Default MRF queue capacity (matches MinIO's 100k MRF list ceiling).
pub const DEFAULT_HEAL_MRF_QUEUE_SIZE: usize = 100_000;
/// Default MRF journal byte budget (8 MiB), mirroring the channel payload cap.
pub const DEFAULT_HEAL_MRF_JOURNAL_MAX_BYTES: usize = 8 * 1024 * 1024;
/// Default MRF replay batch size.
pub const DEFAULT_HEAL_MRF_REPLAY_BATCH: usize = 256;
/// Environment variable selecting how admin heal starts behave when the
/// requested path overlaps an already running or queued heal: `merge`
/// (default, keep today's dedup/merge semantics) or `minio_error` (return a
/// typed already-running / overlapping-paths rejection like madmin).
pub const ENV_HEAL_OVERLAP_POLICY: &str = "RUSTFS_HEAL_OVERLAP_POLICY";
/// Default overlap policy: merge duplicate/overlapping requests.
pub const DEFAULT_HEAL_OVERLAP_POLICY: &str = "merge";
+25
View File
@@ -234,6 +234,31 @@ pub const ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP: &str = "RUSTFS_OBJECT_DISK_WRITE_A
/// Default absolute per-object erasure write cap in seconds (`0` = disabled).
pub const DEFAULT_OBJECT_DISK_WRITE_ABSOLUTE_CAP: u64 = 0;
/// Enable foreground PutObject request admission.
///
/// This is an experimental, default-off foreground write backpressure gate for
/// strict commit tail investigations. When disabled, PUTs follow the legacy
/// path and only the existing request counters are updated.
pub const ENV_PUT_FOREGROUND_ADMISSION_ENABLE: &str = "RUSTFS_PUT_FOREGROUND_ADMISSION_ENABLE";
pub const DEFAULT_PUT_FOREGROUND_ADMISSION_ENABLE: bool = false;
/// Maximum foreground PutObject requests admitted concurrently per process.
///
/// The limit is used only when [`ENV_PUT_FOREGROUND_ADMISSION_ENABLE`] is true.
/// A value of `0` disables the gate even when the enable flag is present, so a
/// partially configured rollout cannot reject every PUT.
pub const ENV_PUT_FOREGROUND_ADMISSION_LIMIT: &str = "RUSTFS_PUT_FOREGROUND_ADMISSION_LIMIT";
pub const DEFAULT_PUT_FOREGROUND_ADMISSION_LIMIT: usize = 0;
/// Time in milliseconds a foreground PutObject waits for an admission permit.
///
/// Once this timeout expires the request fails before body ingest/storage
/// mutation with S3 `SlowDown`/503. `0` means fail fast when the limit is full.
pub const ENV_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: &str = "RUSTFS_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS";
pub const DEFAULT_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 0;
const _: () = assert!(!DEFAULT_PUT_FOREGROUND_ADMISSION_ENABLE);
/// Environment variable for minimum GetObject timeout in seconds.
///
/// When dynamic timeout calculation is enabled, this is the minimum timeout
+286
View File
@@ -870,6 +870,157 @@ pub struct DataUsageCacheInfo {
pub snapshot_complete: bool,
}
/// Prefix-level usage over a raw entry map — the shared core behind
/// [`DataUsageCache::prefix_usage`], usable by any cache-shaped reader (the
/// scanner's writer-side cache has the same map type).
///
/// Cache keys are cleaned literal paths (`bucket/pre/fix`), so sub-prefix
/// names come straight off the child keys — no reverse mapping exists or is
/// needed. A compacted prefix carries its aggregate but no children, which
/// the `compacted` flag reports so callers can say why the breakdown is
/// empty. `truncated` is set when the breakdown exceeded `max_entries` and
/// was cut (largest first).
pub fn prefix_usage_in_cache(
cache: &HashMap<String, DataUsageEntry>,
bucket: &str,
prefix: &str,
max_entries: usize,
) -> Option<PrefixUsageQuery> {
let prefix = prefix.trim_matches('/');
let root = if prefix.is_empty() {
bucket.to_string()
} else {
format!("{bucket}/{prefix}")
};
let entry = cache.get(&hash_path(&root).key())?.clone();
let usage = PrefixUsageSummary::from_entry(&flatten_entry(cache, &entry, 0)?);
let child_prefix = format!("{root}/");
let mut sub_prefixes: Vec<PrefixUsageEntry> = entry
.children
.iter()
.filter_map(|child_key| {
let child = cache.get(child_key)?;
let child_flat = flatten_entry(cache, child, 1)?;
// Child keys are literal `bucket/pre/name` paths; a trailing
// slash marks a directory object and is display-only here.
let name = child_key
.strip_prefix(child_prefix.as_str())
.unwrap_or(child_key.as_str())
.trim_end_matches('/')
.to_string();
Some(PrefixUsageEntry {
prefix: name,
usage: PrefixUsageSummary::from_entry(&child_flat),
})
})
.collect();
sub_prefixes.sort_by(|left, right| {
right
.usage
.size
.cmp(&left.usage.size)
.then_with(|| left.prefix.cmp(&right.prefix))
});
let truncated = sub_prefixes.len() > max_entries;
sub_prefixes.truncate(max_entries);
Some(PrefixUsageQuery {
usage,
compacted: entry.compacted,
truncated,
sub_prefixes,
})
}
/// Maximum subtree depth [`flatten_entry`] will walk before declaring the
/// cache corrupt — the same bound the scanner's checked flatten uses.
const PREFIX_USAGE_MAX_DEPTH: usize = 1024;
/// Flatten one entry's subtree into an aggregate: the free-function twin of
/// [`DataUsageCache::flatten`], carrying the scanner checked-flatten
/// hardening so a corrupt cache (cycles, over-deep trees, overflowing
/// counters) yields `None` instead of unbounded recursion or wrapped totals.
fn flatten_entry(cache: &HashMap<String, DataUsageEntry>, root: &DataUsageEntry, depth: usize) -> Option<DataUsageEntry> {
if depth > PREFIX_USAGE_MAX_DEPTH {
return None;
}
let mut flattened = DataUsageEntry::default();
if !flattened.checked_merge(root) {
return None;
}
flattened.compacted = root.compacted;
// The root itself is not pre-seeded: it is merged above, and a corrupt
// child edge pointing back at the root's own key is still terminated by
// the visited set on first encounter.
let mut visited: HashSet<&str> = HashSet::new();
let mut pending: Vec<(&String, usize)> = root.children.iter().map(|child| (child, depth + 1)).collect();
while let Some((key, child_depth)) = pending.pop() {
if child_depth > PREFIX_USAGE_MAX_DEPTH || !visited.insert(key.as_str()) {
return None;
}
let entry = cache.get(key)?;
if !flattened.checked_merge(entry) {
return None;
}
pending.extend(entry.children.iter().map(|child| (child, child_depth + 1)));
}
flattened.children.clear();
Some(flattened)
}
/// Flattened counters of one prefix subtree, as returned by
/// [`DataUsageCache::prefix_usage`].
#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, serde::Serialize)]
#[serde(rename_all = "camelCase")]
pub struct PrefixUsageSummary {
pub size: u64,
pub objects: u64,
pub versions: u64,
pub delete_markers: u64,
}
impl PrefixUsageSummary {
fn from_entry(entry: &DataUsageEntry) -> Self {
Self {
size: entry.size as u64,
objects: entry.objects as u64,
versions: entry.versions as u64,
delete_markers: entry.delete_markers as u64,
}
}
/// Add another set's counters into this one (entries are partitioned by
/// set, so per-set results sum).
pub fn merge(&mut self, other: &Self) {
self.size = self.size.saturating_add(other.size);
self.objects = self.objects.saturating_add(other.objects);
self.versions = self.versions.saturating_add(other.versions);
self.delete_markers = self.delete_markers.saturating_add(other.delete_markers);
}
}
/// One first-level sub-prefix row of a [`PrefixUsageQuery`].
#[derive(Clone, Debug, PartialEq, Eq, serde::Serialize)]
pub struct PrefixUsageEntry {
pub prefix: String,
pub usage: PrefixUsageSummary,
}
/// Result of [`DataUsageCache::prefix_usage`].
#[derive(Clone, Debug, Default, PartialEq, Eq, serde::Serialize)]
#[serde(rename_all = "camelCase")]
pub struct PrefixUsageQuery {
pub usage: PrefixUsageSummary,
/// The prefix entry was compacted by the scanner: its aggregate is valid
/// but no sub-prefix breakdown exists on disk.
pub compacted: bool,
/// The breakdown had more entries than `max_entries`; the largest remain.
pub truncated: bool,
pub sub_prefixes: Vec<PrefixUsageEntry>,
}
/// Read-only projection of a scanner-written `.usage-cache.bin` file.
///
/// The scanner-side `DataUsageCache` (`crates/scanner/src/data_usage_define.rs`)
@@ -997,6 +1148,21 @@ impl DataUsageCache {
}
}
/// Prefix-level usage for one bucket subtree, plus the one-level
/// breakdown below it (rustfs/backlog#1872, MinIO
/// `loadPrefixUsageFromBackend` parity and beyond: arbitrary prefixes and
/// full counters instead of first-level sizes only).
///
/// Cache keys are cleaned literal paths (`bucket/pre/fix`), so sub-prefix
/// names come straight off the child keys — no reverse mapping exists or
/// is needed. A compacted prefix carries its aggregate but no children,
/// which the `compacted` flag reports so callers can say why the
/// breakdown is empty. `truncated` is set when the breakdown exceeded
/// `max_entries` and was cut (largest first).
pub fn prefix_usage(&self, bucket: &str, prefix: &str, max_entries: usize) -> Option<PrefixUsageQuery> {
prefix_usage_in_cache(&self.cache, bucket, prefix, max_entries)
}
pub fn force_compact(&mut self, limit: usize) {
if self.cache.len() < limit {
return;
@@ -1898,6 +2064,126 @@ mod tests {
);
}
/// Build a cache shaped like `bucket/{a,b/{c,d}},bucket/loose` with
/// distinct counters so aggregation is observable.
fn prefix_usage_fixture_cache() -> DataUsageCache {
let mut cache = DataUsageCache::default();
let mut insert = |path: &str, parent: &str, size: usize, objects: usize, versions: usize, delete_markers: usize| {
cache.replace(
path,
parent,
DataUsageEntry {
size,
objects,
versions,
delete_markers,
..Default::default()
},
);
};
insert("bucket", "", 0, 0, 0, 0);
insert("bucket/a", "bucket", 100, 1, 1, 0);
insert("bucket/b", "bucket", 0, 0, 0, 0);
insert("bucket/b/c", "bucket/b", 200, 2, 2, 1);
insert("bucket/b/d", "bucket/b", 40, 1, 3, 0);
insert("bucket/loose", "bucket", 10, 1, 1, 1);
cache
}
#[test]
fn prefix_usage_aggregates_bucket_root_and_one_level_below() {
let cache = prefix_usage_fixture_cache();
let root = cache
.prefix_usage("bucket", "", 100)
.expect("root query must find the bucket entry");
assert_eq!(root.usage.size, 350, "root aggregate flattens the whole subtree");
assert_eq!(root.usage.objects, 5);
assert_eq!(root.usage.versions, 7);
assert_eq!(root.usage.delete_markers, 2);
assert!(!root.compacted);
assert!(!root.truncated);
// Breakdown is one level: b (240) before a (100) before loose (10),
// each flattened to its own subtree total.
let names: Vec<(&str, u64)> = root
.sub_prefixes
.iter()
.map(|entry| (entry.prefix.as_str(), entry.usage.size))
.collect();
assert_eq!(names, vec![("b", 240), ("a", 100), ("loose", 10)]);
}
#[test]
fn prefix_usage_drills_into_arbitrary_prefixes() {
let cache = prefix_usage_fixture_cache();
let b = cache.prefix_usage("bucket", "b", 100).expect("nested prefix must resolve");
assert_eq!(b.usage.size, 240);
assert_eq!(b.usage.versions, 5);
let names: Vec<&str> = b.sub_prefixes.iter().map(|entry| entry.prefix.as_str()).collect();
assert_eq!(names, vec!["c", "d"]);
// Prefix slashes are normalized away.
let slashed = cache.prefix_usage("bucket", "/b/", 100).expect("slash-insensitive lookup");
assert_eq!(slashed.usage.size, 240);
assert!(cache.prefix_usage("bucket", "absent", 100).is_none(), "unknown prefix must be a miss");
assert!(cache.prefix_usage("other", "", 100).is_none(), "unknown bucket must be a miss");
}
#[test]
fn prefix_usage_reports_and_respects_truncation() {
let cache = prefix_usage_fixture_cache();
let capped = cache.prefix_usage("bucket", "", 2).expect("root query");
assert!(capped.truncated, "three children capped to two must flag truncation");
let names: Vec<&str> = capped.sub_prefixes.iter().map(|entry| entry.prefix.as_str()).collect();
assert_eq!(names, vec!["b", "a"], "largest prefixes survive the cut");
}
#[test]
fn prefix_usage_marks_compacted_entries() {
let mut cache = DataUsageCache::default();
cache.replace(
"bucket",
"",
DataUsageEntry {
size: 999,
objects: 9,
compacted: true,
..Default::default()
},
);
let compacted = cache.prefix_usage("bucket", "", 100).expect("compacted root resolves");
assert!(compacted.compacted, "compaction must be visible to callers");
assert_eq!(compacted.usage.size, 999);
assert!(compacted.sub_prefixes.is_empty(), "a compacted entry carries no children");
}
#[test]
fn prefix_usage_rejects_cyclic_and_dangling_caches() {
// A self-referencing child (corrupt cache) must yield a miss for the
// whole query, not unbounded recursion.
let mut cache = prefix_usage_fixture_cache();
if let Some(entry) = cache.cache.get_mut("bucket/b") {
entry.children.insert("bucket/b".to_string());
}
assert!(cache.prefix_usage("bucket", "b", 100).is_none(), "a cyclic subtree must be rejected");
// The unaffected sibling still answers.
assert!(cache.prefix_usage("bucket", "a", 100).is_some());
// A child key with no entry (dangling link) is rejected rather than
// silently dropped: half a tree would under-report usage.
let mut dangling = prefix_usage_fixture_cache();
if let Some(entry) = dangling.cache.get_mut("bucket/b") {
entry.children.insert("bucket/b/ghost".to_string());
}
assert!(
dangling.prefix_usage("bucket", "b", 100).is_none(),
"a dangling child link must be rejected"
);
}
#[test]
fn hash_path_uses_portable_slash_semantics() {
for (input, expected) in [
+79 -4
View File
@@ -32,6 +32,7 @@ use rustfs_signer::sign_v4;
use s3s::Body;
use std::ffi::OsStr;
use std::fs as stdfs;
use std::io::ErrorKind;
use std::path::{Path, PathBuf};
use std::process::{Child, Command, Stdio};
use std::sync::Once;
@@ -51,6 +52,11 @@ pub(crate) const FAST_DATA_USAGE_SCANNER_ENV: &[(&str, &str)] =
&[("RUSTFS_SCANNER_CYCLE", "1"), ("RUSTFS_SCANNER_START_DELAY_SECS", "0")];
pub const TEST_BUCKET: &str = "e2e-test-bucket";
const RUSTFS_FULL_FEATURE: &str = "full";
const TEST_PORT_MIN: u16 = 20_000;
const TEST_PORT_RANGE: u16 = 40_000;
const TEST_PORT_COUNTER_PATH: &str = "/tmp/rustfs_e2e_next_port";
const TEST_PORT_LOCK_DIR: &str = "/tmp/rustfs_e2e_port_allocator.lock";
const TEST_PORT_LOCK_STALE_AFTER: Duration = Duration::from_secs(30);
fn capture_log_path(log_dir: &Path, temp_dir: &str) -> Option<PathBuf> {
let temp_name = Path::new(temp_dir).file_name()?.to_string_lossy();
@@ -67,6 +73,64 @@ fn configured_capture_log_path(temp_dir: &str) -> Option<String> {
capture_log_path(Path::new(&log_dir), temp_dir).map(|path| path.to_string_lossy().into_owned())
}
struct PortAllocatorGuard;
impl PortAllocatorGuard {
async fn acquire() -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
loop {
match stdfs::create_dir(TEST_PORT_LOCK_DIR) {
Ok(()) => return Ok(Self),
Err(err) if err.kind() == ErrorKind::AlreadyExists => {
remove_stale_port_allocator_lock();
sleep(Duration::from_millis(10)).await;
}
Err(err) => return Err(err.into()),
}
}
}
}
impl Drop for PortAllocatorGuard {
fn drop(&mut self) {
let _ = stdfs::remove_dir(TEST_PORT_LOCK_DIR);
}
}
fn advance_test_port(port: u16) -> u16 {
let offset = (port - TEST_PORT_MIN + 1) % TEST_PORT_RANGE;
TEST_PORT_MIN + offset
}
fn seeded_test_port() -> u16 {
let offset = (Uuid::new_v4().as_u128() % u128::from(TEST_PORT_RANGE)) as u16;
TEST_PORT_MIN + offset
}
fn read_next_test_port() -> u16 {
stdfs::read_to_string(TEST_PORT_COUNTER_PATH)
.ok()
.and_then(|value| value.trim().parse::<u16>().ok())
.filter(|port| (TEST_PORT_MIN..TEST_PORT_MIN + TEST_PORT_RANGE).contains(port))
.unwrap_or_else(seeded_test_port)
}
fn remove_stale_port_allocator_lock() {
let Ok(metadata) = stdfs::metadata(TEST_PORT_LOCK_DIR) else {
return;
};
let Ok(modified) = metadata.modified() else {
return;
};
if modified.elapsed().is_ok_and(|elapsed| elapsed > TEST_PORT_LOCK_STALE_AFTER) {
let _ = stdfs::remove_dir(TEST_PORT_LOCK_DIR);
}
}
fn write_next_test_port(port: u16) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
stdfs::write(TEST_PORT_COUNTER_PATH, port.to_string())?;
Ok(())
}
pub(crate) fn capture_command_logs(
command: &mut Command,
log_path: Option<&str>,
@@ -508,10 +572,21 @@ impl RustFSTestEnvironment {
/// Find an available port for the test
pub async fn find_available_port() -> Result<u16, Box<dyn std::error::Error + Send + Sync>> {
use std::net::TcpListener;
let listener = TcpListener::bind("127.0.0.1:0")?;
let port = listener.local_addr()?.port();
drop(listener);
Ok(port)
let _guard = PortAllocatorGuard::acquire().await?;
let mut next_port = read_next_test_port();
for _ in 0..TEST_PORT_RANGE {
let port = next_port;
next_port = advance_test_port(next_port);
write_next_test_port(next_port)?;
if let Ok(listener) = TcpListener::bind(("127.0.0.1", port)) {
drop(listener);
return Ok(port);
}
}
Err("no available E2E test port found".into())
}
/// Kill any existing RustFS processes
+2 -2
View File
@@ -4,8 +4,8 @@ This module is the shared failure-injection boundary for replication end-to-end
`FakeS3Target::start()` creates the listener. Add target buckets with `create_bucket`, point a RustFS remote target at `address()`, use `FAKE_ACCESS_KEY` / `FAKE_SECRET_KEY`, then enqueue per-operation faults with `inject`. Faults for one operation are consumed in FIFO order and do not consume faults queued for another operation. A fault is consumed only after `s3s` verifies the full request signature, so anonymous, other-access-key, and bad-signature traffic cannot disturb a script.
Supported data operations are HeadBucket, GetBucketVersioning, PUT/GET/HEAD/DELETE Object, and create/upload/complete/abort multipart upload. `create_bucket` models general-purpose buckets in S3's shared global namespace; account-regional namespace buckets and their `-an` names are intentionally out of scope. Buckets are versioned: PUT creates a version, DELETE without `versionId` creates a delete marker, and DELETE with `versionId` removes exactly that version. Internal source version IDs must be UUIDs and are stored canonically. Source mtime is honored only for source-replication PUT/DELETE requests; absent or invalid values use receipt time, matching RustFS, while multipart completion always uses receipt time. Replicated versions are ordered newest-first by source mtime so late older versions and delete markers do not become current. Equal mtimes prefer objects over delete markers, then canonical UUID order; RustFS's internal FileMeta signature tie-break is intentionally out of scope because it is not part of the target S3 protocol. Multipart part numbers follow S3's `1..=10000` range, and every completed part except the final part must be at least 5 MiB.
Supported data operations are HeadBucket, GetBucketVersioning, PUT/GET/HEAD/DELETE Object, Get/Put/Delete ObjectTagging (tags live per version; Put replaces the whole set, Delete clears it), and create/upload/complete/abort multipart upload. `create_bucket` models general-purpose buckets in S3's shared global namespace; account-regional namespace buckets and their `-an` names are intentionally out of scope. Buckets are versioned: PUT creates a version, DELETE without `versionId` creates a delete marker, and DELETE with `versionId` removes exactly that version. Internal source version IDs must be UUIDs and are stored canonically. Source mtime is honored only for source-replication PUT/DELETE requests; absent or invalid values use receipt time, matching RustFS, while multipart completion always uses receipt time. Replicated versions are ordered newest-first by source mtime so late older versions and delete markers do not become current. Equal mtimes prefer objects over delete markers, then canonical UUID order; RustFS's internal FileMeta signature tie-break is intentionally out of scope because it is not part of the target S3 protocol. Multipart part numbers follow S3's `1..=10000` range, and every completed part except the final part must be at least 5 MiB.
Fault actions cover HTTP 401/403/503 responses, pre-dispatch delay, connection abort when a logical request-body threshold is reached, streaming slow drain, and a deliberately wrong response ETag (including multipart-complete XML). `requests()` returns the ordered, credential-free request journal for assertions.
Fault actions cover HTTP 401/403/503 responses, pre-dispatch delay, connection abort when a logical request-body threshold is reached, streaming slow drain, and a deliberately wrong response ETag (including multipart-complete XML). `requests()` returns the ordered, credential-free request journal for assertions. Each record also journals a `ProxyHeaderSnapshot` — the read-proxy anti-loop marker (`x-{rustfs,minio}-source-proxy-request`), the replication-check exemption header, and the client SSE-C header family (algorithm and key-MD5 values; for the key itself only its presence) — so proxy tests can pin the exact wire contract.
The listener is loopback-only. It admits at most 64 active connections and two concurrently buffered request bodies; authenticated multipart-complete XML collection and assembly take both body permits. Keep-alive is disabled, request-header reads are bounded to 30 seconds, a parsed request is bounded to 65 seconds, and the complete connection lifetime is bounded to 100 seconds. It retains at most 256 buckets, 4,096 journal entries, 4,096 scripted faults, 4,096 object versions, 256 multipart uploads, and 10,000 multipart parts. Retained identifiers are capped at 1 KiB, user metadata at 2 KiB, and content type at 1 KiB. A PUT or uploaded part is capped at 64 MiB; a completed multipart object and all stored object/part data are capped at 128 MiB. Body drain, body-permit waits, delay, and slow-drain execution are bounded to 30 seconds; each slow-drain slice delay must be below that bound.
+284 -5
View File
@@ -30,10 +30,12 @@ use s3s::access::{S3Access, S3AccessContext};
use s3s::auth::SimpleAuth;
use s3s::dto::{
AbortMultipartUploadInput, AbortMultipartUploadOutput, CompleteMultipartUploadInput, CompleteMultipartUploadOutput,
CreateMultipartUploadInput, CreateMultipartUploadOutput, DeleteMarkerEntry, DeleteObjectInput, DeleteObjectOutput, ETag,
GetBucketVersioningInput, GetBucketVersioningOutput, GetObjectInput, GetObjectOutput, HeadBucketInput, HeadBucketOutput,
CreateMultipartUploadInput, CreateMultipartUploadOutput, DeleteMarkerEntry, DeleteObjectInput, DeleteObjectOutput,
DeleteObjectTaggingInput, DeleteObjectTaggingOutput, ETag, GetBucketVersioningInput, GetBucketVersioningOutput,
GetObjectInput, GetObjectOutput, GetObjectTaggingInput, GetObjectTaggingOutput, HeadBucketInput, HeadBucketOutput,
HeadObjectInput, HeadObjectOutput, ListObjectVersionsInput, ListObjectVersionsOutput, ObjectVersionId, PutObjectInput,
PutObjectOutput, StreamingBlob, Timestamp, TimestampFormat, UploadPartInput, UploadPartOutput,
PutObjectOutput, PutObjectTaggingInput, PutObjectTaggingOutput, StreamingBlob, Tag, TagSet, Timestamp, TimestampFormat,
UploadPartInput, UploadPartOutput,
};
use s3s::service::{S3Service, S3ServiceBuilder};
use s3s::validation::{AwsNameValidation, NameValidation};
@@ -88,6 +90,13 @@ const SOURCE_LEGALHOLD_TIMESTAMP_HEADERS: [&str; 2] = [
"x-rustfs-source-replication-legalhold-timestamp",
"x-minio-source-replication-legalhold-timestamp",
];
/// Wire prefix of the SSE-C passthrough replication transport headers
/// (`X-Rustfs-Replication-*`). In the default mode the fake stores them like a
/// RustFS target and echoes SSE-C evidence back on HEAD/GET; with
/// [`FakeS3Target::drop_unlisted_replication_headers`] it models MinIO /
/// generic S3, which silently discard unknown x-* headers.
const REPLICATION_SSE_TRANSPORT_PREFIX: &str = "x-rustfs-replication-";
const REPLICATION_SSEC_ALGORITHM_TRANSPORT_HEADER: &str = "x-rustfs-replication-ssec-algorithm";
const RESERVED_BUCKET_PREFIXES: [&str; 3] = ["xn--", "sthree-", "amzn-s3-demo-"];
const RESERVED_BUCKET_SUFFIXES: [&str; 6] = ["-s3alias", "--ol-s3", ".mrap", "--x-s3", "--table-s3", "-an"];
@@ -103,6 +112,9 @@ pub enum Operation {
GetObject,
HeadObject,
DeleteObject,
GetObjectTagging,
PutObjectTagging,
DeleteObjectTagging,
ListObjectVersions,
CreateMultipartUpload,
UploadPart,
@@ -149,6 +161,42 @@ impl ReplicationTimestampHeaders {
}
}
/// Read-proxy related headers observed on a request, journaled so proxy
/// tests can assert the exact wire contract: the anti-loop marker present,
/// the replication-check exemption absent, and the client SSE-C key family
/// forwarded verbatim. The SSE-C key value itself is never retained — only
/// its presence.
#[derive(Debug, Clone, Default, PartialEq, Eq)]
pub struct ProxyHeaderSnapshot {
pub source_proxy_request: Option<String>,
pub replication_check: Option<String>,
pub ssec_algorithm: Option<String>,
pub ssec_key_present: bool,
pub ssec_key_md5: Option<String>,
/// Whether the request carried any `X-Rustfs-Replication-*` SSE-C
/// passthrough transport header, so fail-closed tests can assert the
/// sender really shipped the material a dropping target discarded.
pub ssec_transport_present: bool,
}
impl ProxyHeaderSnapshot {
fn from_headers(headers: &HeaderMap) -> Self {
Self {
source_proxy_request: header_value(headers, &["x-rustfs-source-proxy-request", "x-minio-source-proxy-request"])
.map(bounded_journal_value),
replication_check: header_value(headers, &["x-rustfs-source-replication-check", "x-minio-source-replication-check"])
.map(bounded_journal_value),
ssec_algorithm: header_value(headers, &["x-amz-server-side-encryption-customer-algorithm"])
.map(bounded_journal_value),
ssec_key_present: headers.contains_key("x-amz-server-side-encryption-customer-key"),
ssec_key_md5: header_value(headers, &["x-amz-server-side-encryption-customer-key-md5"]).map(bounded_journal_value),
ssec_transport_present: headers
.keys()
.any(|name| name.as_str().starts_with(REPLICATION_SSE_TRANSPORT_PREFIX)),
}
}
}
/// Credential-free request metadata retained for deterministic assertions.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct RequestRecord {
@@ -163,6 +211,7 @@ pub struct RequestRecord {
pub content_length: Option<u64>,
pub consumed_bytes: Option<usize>,
pub replication_timestamps: ReplicationTimestampHeaders,
pub proxy_headers: ProxyHeaderSnapshot,
pub fault: Option<FaultAction>,
}
@@ -178,6 +227,10 @@ struct ControlState {
struct StoreState {
assign_own_version_ids: bool,
assign_own_multipart_version_ids: bool,
/// MinIO-like mode: silently discard non-whitelisted replication
/// transport headers instead of storing them (see
/// [`REPLICATION_SSE_TRANSPORT_PREFIX`]).
drop_unlisted_replication_headers: bool,
buckets: HashMap<String, BucketState>,
uploads: HashMap<String, MultipartState>,
total_bytes: usize,
@@ -199,6 +252,12 @@ struct ObjectVersion {
delete_marker: bool,
content_type: Option<String>,
metadata: Option<HashMap<String, String>>,
/// Object tags as ordered key/value pairs (PutObjectTagging replaces the
/// whole set, DeleteObjectTagging clears it).
tags: Vec<(String, String)>,
/// SSE-C passthrough transport headers stored with the version (RustFS
/// target behavior); empty when the drop mode discarded them.
replication_sse_headers: Vec<(String, String)>,
}
#[derive(Clone)]
@@ -208,6 +267,7 @@ struct MultipartState {
version_id: String,
content_type: Option<String>,
metadata: Option<HashMap<String, String>>,
replication_sse_headers: Vec<(String, String)>,
parts: BTreeMap<i32, MultipartPart>,
}
@@ -428,6 +488,15 @@ impl FakeS3Target {
/// Mint own version ids for the multipart path only — models a target
/// that adopts PutObject version ids but not CreateMultipartUpload ones.
/// MinIO-like mode: silently drop every `X-Rustfs-Replication-*` SSE-C
/// passthrough transport header instead of storing it. The default (off)
/// models a RustFS target, which preserves the headers and echoes SSE-C
/// evidence (`x-amz-server-side-encryption-customer-algorithm`) on
/// HEAD/GET of the replica.
pub fn drop_unlisted_replication_headers(&self, enabled: bool) {
lock(&self.backend.store).drop_unlisted_replication_headers = enabled;
}
pub fn assign_own_multipart_version_ids(&self, enabled: bool) {
lock(&self.backend.store).assign_own_multipart_version_ids = enabled;
}
@@ -569,6 +638,7 @@ impl S3Access for FaultAccess {
.and_then(|value| value.to_str().ok())
.and_then(|value| value.parse().ok());
let replication_timestamps = ReplicationTimestampHeaders::from_headers(context.headers());
let proxy_headers = ProxyHeaderSnapshot::from_headers(context.headers());
let fault = record_request(
&self.control,
operation,
@@ -576,6 +646,7 @@ impl S3Access for FaultAccess {
parsed,
content_length,
replication_timestamps,
proxy_headers,
);
if let Some(RequestFault {
action: FaultAction::Status(status),
@@ -615,6 +686,9 @@ fn operation_from_s3_name(name: &str) -> Operation {
"GetObject" => Operation::GetObject,
"HeadObject" => Operation::HeadObject,
"DeleteObject" => Operation::DeleteObject,
"GetObjectTagging" => Operation::GetObjectTagging,
"PutObjectTagging" => Operation::PutObjectTagging,
"DeleteObjectTagging" => Operation::DeleteObjectTagging,
"CreateMultipartUpload" => Operation::CreateMultipartUpload,
"UploadPart" => Operation::UploadPart,
"CompleteMultipartUpload" => Operation::CompleteMultipartUpload,
@@ -630,6 +704,7 @@ fn record_request(
parsed: ParsedRequest,
content_length: Option<u64>,
replication_timestamps: ReplicationTimestampHeaders,
proxy_headers: ProxyHeaderSnapshot,
) -> Option<RequestFault> {
let mut state = lock(control);
let action = parsed
@@ -655,6 +730,7 @@ fn record_request(
content_length,
consumed_bytes: None,
replication_timestamps,
proxy_headers,
fault: action.clone(),
});
action.map(|action| RequestFault { sequence, action })
@@ -721,6 +797,15 @@ fn parse_request(method: &Method, uri: &Uri) -> ParsedRequest {
(&Method::POST, true) if query.contains_key("uploads") => Operation::CreateMultipartUpload,
(&Method::POST, true) if upload_id.is_some() => Operation::CompleteMultipartUpload,
(&Method::DELETE, true) if upload_id.is_some() => Operation::AbortMultipartUpload,
(&Method::GET, true) if query.contains_key("tagging") && only_query_keys(&["tagging", "versionId"]) => {
Operation::GetObjectTagging
}
(&Method::PUT, true) if query.contains_key("tagging") && only_query_keys(&["tagging", "versionId"]) => {
Operation::PutObjectTagging
}
(&Method::DELETE, true) if query.contains_key("tagging") && only_query_keys(&["tagging", "versionId"]) => {
Operation::DeleteObjectTagging
}
// A replication PUT addresses the source version via `?versionId=`.
(&Method::PUT, true) if only_query_keys(&["versionId"]) => Operation::PutObject,
(&Method::GET, true) if only_query_keys(&["versionId"]) => Operation::GetObject,
@@ -788,6 +873,29 @@ fn new_version_id(headers: &HeaderMap, assign_own: bool) -> S3Result<String> {
Ok(version_id.to_string())
}
/// Capture the SSE-C passthrough transport headers a replication PUT carried.
/// Returns an empty set in the MinIO-like drop mode.
fn captured_replication_sse_headers(headers: &HeaderMap, drop_unlisted: bool) -> Vec<(String, String)> {
if drop_unlisted {
return Vec::new();
}
headers
.iter()
.filter(|(name, _)| name.as_str().starts_with(REPLICATION_SSE_TRANSPORT_PREFIX))
.filter_map(|(name, value)| Some((name.as_str().to_string(), value.to_str().ok()?.to_string())))
.collect()
}
/// SSE-C evidence a RustFS-like target echoes for a stored passthrough
/// replica: the customer algorithm restored from the transport headers.
fn stored_sse_customer_algorithm(version: &ObjectVersion) -> Option<String> {
version
.replication_sse_headers
.iter()
.find(|(name, _)| name == REPLICATION_SSEC_ALGORITHM_TRANSPORT_HEADER)
.map(|(_, value)| value.clone())
}
fn source_etag(headers: &HeaderMap) -> S3Result<Option<String>> {
header_value(headers, &SOURCE_ETAG_HEADERS)
.map(|value| validate_retained_identifier(value, "source ETag").map(|value| normalize_etag(&value)))
@@ -1135,6 +1243,33 @@ fn find_version(state: &StoreState, bucket: &str, key: &str, version_id: Option<
Ok(version.clone())
}
/// Replace (or clear, with an empty vec) the tag set of the addressed
/// version, returning its version id. Mirrors `find_version` addressing:
/// explicit version id or the latest version, delete markers rejected.
fn set_version_tags(
state: &mut StoreState,
bucket: &str,
key: &str,
version_id: Option<&str>,
tags: Vec<(String, String)>,
) -> S3Result<String> {
// Resolve first (immutable) so the error paths match find_version.
let resolved = find_version(state, bucket, key, version_id)?.version_id;
let versions = state
.buckets
.get_mut(bucket)
.expect("bucket existence checked by find_version")
.objects
.get_mut(key)
.expect("key existence checked by find_version");
let version = versions
.iter_mut()
.find(|version| version.version_id == resolved)
.expect("version existence checked by find_version");
version.tags = tags;
Ok(resolved)
}
#[async_trait]
impl S3 for FakeBackend {
async fn head_bucket(&self, req: S3Request<HeadBucketInput>) -> S3Result<S3Response<HeadBucketOutput>> {
@@ -1231,7 +1366,10 @@ impl S3 for FakeBackend {
let input = req.input;
let body = collect_stream(input.body, input.content_length, fault.as_ref(), &self.control).await?;
validate_stored_metadata(&input.content_type, &input.metadata)?;
let assign_own = lock(&self.store).assign_own_version_ids;
let (assign_own, drop_unlisted) = {
let state = lock(&self.store);
(state.assign_own_version_ids, state.drop_unlisted_replication_headers)
};
let version_id = new_version_id(&headers, assign_own)?;
let e_tag = match source_etag(&headers)? {
Some(value) => value,
@@ -1248,6 +1386,8 @@ impl S3 for FakeBackend {
delete_marker: false,
content_type: input.content_type,
metadata: input.metadata,
tags: Vec::new(),
replication_sse_headers: captured_replication_sse_headers(&headers, drop_unlisted),
};
upsert_version(&mut lock(&self.store), &input.bucket, input.key, version)?;
Ok(apply_response_fault(
@@ -1268,6 +1408,7 @@ impl S3 for FakeBackend {
let state = lock(&self.store);
find_version(&state, &input.bucket, &input.key, input.version_id.as_deref())?
};
let sse_customer_algorithm = stored_sse_customer_algorithm(&version);
Ok(apply_response_fault(
S3Response::new(GetObjectOutput {
body: Some(StreamingBlob::new(Body::from(version.body.clone()))),
@@ -1277,6 +1418,7 @@ impl S3 for FakeBackend {
e_tag: Some(ETag::Strong(version.e_tag)),
last_modified: Some(version.last_modified.clone()),
version_id: Some(version.version_id),
sse_customer_algorithm,
..Default::default()
}),
fault.as_ref(),
@@ -1291,6 +1433,7 @@ impl S3 for FakeBackend {
let state = lock(&self.store);
find_version(&state, &input.bucket, &input.key, input.version_id.as_deref())?
};
let sse_customer_algorithm = stored_sse_customer_algorithm(&version);
Ok(apply_response_fault(
S3Response::new(HeadObjectOutput {
content_length: Some(version.body.len() as i64),
@@ -1299,12 +1442,79 @@ impl S3 for FakeBackend {
e_tag: Some(ETag::Strong(version.e_tag)),
last_modified: Some(version.last_modified.clone()),
version_id: Some(version.version_id),
sse_customer_algorithm,
..Default::default()
}),
fault.as_ref(),
))
}
async fn get_object_tagging(&self, req: S3Request<GetObjectTaggingInput>) -> S3Result<S3Response<GetObjectTaggingOutput>> {
let fault = request_fault(&req);
apply_non_body_fault(fault.as_ref(), &self.control).await?;
let input = req.input;
let version = {
let state = lock(&self.store);
find_version(&state, &input.bucket, &input.key, input.version_id.as_deref())?
};
let tag_set: TagSet = version
.tags
.into_iter()
.map(|(key, value)| Tag {
key: Some(key),
value: Some(value),
})
.collect();
Ok(apply_response_fault(
S3Response::new(GetObjectTaggingOutput {
tag_set,
version_id: Some(ObjectVersionId::from(version.version_id)),
}),
fault.as_ref(),
))
}
async fn put_object_tagging(&self, req: S3Request<PutObjectTaggingInput>) -> S3Result<S3Response<PutObjectTaggingOutput>> {
let fault = request_fault(&req);
apply_non_body_fault(fault.as_ref(), &self.control).await?;
let input = req.input;
let tags = input
.tagging
.tag_set
.into_iter()
.map(|tag| (tag.key.unwrap_or_default(), tag.value.unwrap_or_default()))
.collect();
let version_id = {
let mut state = lock(&self.store);
set_version_tags(&mut state, &input.bucket, &input.key, input.version_id.as_deref(), tags)?
};
Ok(apply_response_fault(
S3Response::new(PutObjectTaggingOutput {
version_id: Some(ObjectVersionId::from(version_id)),
}),
fault.as_ref(),
))
}
async fn delete_object_tagging(
&self,
req: S3Request<DeleteObjectTaggingInput>,
) -> S3Result<S3Response<DeleteObjectTaggingOutput>> {
let fault = request_fault(&req);
apply_non_body_fault(fault.as_ref(), &self.control).await?;
let input = req.input;
let version_id = {
let mut state = lock(&self.store);
set_version_tags(&mut state, &input.bucket, &input.key, input.version_id.as_deref(), Vec::new())?
};
Ok(apply_response_fault(
S3Response::new(DeleteObjectTaggingOutput {
version_id: Some(ObjectVersionId::from(version_id)),
}),
fault.as_ref(),
))
}
async fn delete_object(&self, req: S3Request<DeleteObjectInput>) -> S3Result<S3Response<DeleteObjectOutput>> {
let fault = request_fault(&req);
apply_non_body_fault(fault.as_ref(), &self.control).await?;
@@ -1381,6 +1591,8 @@ impl S3 for FakeBackend {
delete_marker: true,
content_type: None,
metadata: None,
tags: Vec::new(),
replication_sse_headers: Vec::new(),
},
)?;
Ok(apply_response_fault(
@@ -1408,9 +1620,10 @@ impl S3 for FakeBackend {
ensure_upload_budget(&state)?;
validate_stored_metadata(&input.content_type, &input.metadata)?;
let upload_id = Uuid::new_v4().to_string();
// Read the flag before the mutable borrow of `state.uploads` below
// Read the flags before the mutable borrow of `state.uploads` below
// (and never re-lock the store: the mutex is not reentrant).
let mint_own = state.assign_own_version_ids || state.assign_own_multipart_version_ids;
let drop_unlisted = state.drop_unlisted_replication_headers;
let version_id = new_version_id(&headers, mint_own)?;
state.uploads.insert(
upload_id.clone(),
@@ -1420,6 +1633,7 @@ impl S3 for FakeBackend {
version_id,
content_type: input.content_type,
metadata: input.metadata,
replication_sse_headers: captured_replication_sse_headers(&headers, drop_unlisted),
parts: BTreeMap::new(),
},
);
@@ -1557,6 +1771,7 @@ impl S3 for FakeBackend {
version_id: upload.version_id.clone(),
content_type: upload.content_type.clone(),
metadata: upload.metadata.clone(),
replication_sse_headers: upload.replication_sse_headers.clone(),
parts: BTreeMap::new(),
},
selected,
@@ -1583,6 +1798,8 @@ impl S3 for FakeBackend {
delete_marker: false,
content_type: upload.content_type,
metadata: upload.metadata,
tags: Vec::new(),
replication_sse_headers: upload.replication_sse_headers,
};
let mut state = lock(&self.store);
let current = state
@@ -1787,6 +2004,65 @@ mod tests {
Ok(())
}
/// Default mode is RustFS-like: SSE-C passthrough transport headers are
/// stored and the customer algorithm is echoed on HEAD/GET. Drop mode is
/// MinIO-like: the headers are silently discarded, so no evidence comes
/// back — the exact difference the N2 fail-closed audit keys on. Both
/// modes journal that the sender shipped the transport headers.
#[tokio::test]
async fn ssec_passthrough_headers_echo_and_drop_modes() -> Result<(), BoxError> {
let target = FakeS3Target::start().await?;
target.create_bucket("target-bucket");
let client = client(&target);
let put_with_transport_headers = |key: &'static str| {
client
.put_object()
.bucket("target-bucket")
.key(key)
.body(ByteStream::from_static(b"ciphertext"))
.customize()
.map_request(move |mut request| {
let headers = request.headers_mut();
headers.insert("x-rustfs-replication-ssec-algorithm", "AES256");
headers.insert("x-rustfs-replication-ssec-key-md5", "AAAAAAAAAAAAAAAAAAAAAA==");
Ok::<_, std::convert::Infallible>(request)
})
.send()
};
put_with_transport_headers("kept").await?;
let head = client.head_object().bucket("target-bucket").key("kept").send().await?;
assert_eq!(head.sse_customer_algorithm(), Some("AES256"));
let get = client.get_object().bucket("target-bucket").key("kept").send().await?;
assert_eq!(get.sse_customer_algorithm(), Some("AES256"));
target.drop_unlisted_replication_headers(true);
put_with_transport_headers("dropped").await?;
let head = client.head_object().bucket("target-bucket").key("dropped").send().await?;
assert_eq!(head.sse_customer_algorithm(), None, "drop mode must discard SSE-C evidence");
let requests = target.requests();
for key in ["kept", "dropped"] {
let record = requests
.iter()
.find(|record| record.operation == Operation::PutObject && record.key.as_deref() == Some(key))
.expect("PUT must be journaled");
assert!(
record.proxy_headers.ssec_transport_present,
"the journal must prove the sender shipped the transport headers for {key}"
);
}
let plain_head = requests
.iter()
.find(|record| record.operation == Operation::HeadObject)
.expect("HEAD must be journaled");
assert!(!plain_head.proxy_headers.ssec_transport_present);
target.shutdown().await;
Ok(())
}
macro_rules! assert_sdk_error {
($error:expr, $status:expr, $code:expr) => {{
let error = &$error;
@@ -3052,6 +3328,7 @@ mod tests {
version_id: index.to_string(),
content_type: None,
metadata: None,
replication_sse_headers: Vec::new(),
parts: BTreeMap::new(),
},
);
@@ -3074,6 +3351,7 @@ mod tests {
},
Some(0),
ReplicationTimestampHeaders::default(),
ProxyHeaderSnapshot::default(),
);
}
let records = lock(&control).requests.clone();
@@ -3096,6 +3374,7 @@ mod tests {
},
None,
ReplicationTimestampHeaders::default(),
ProxyHeaderSnapshot::default(),
);
{
let bounded_records = lock(&bounded_control);
@@ -67,6 +67,9 @@ type MetricValues = Arc<Mutex<BTreeMap<String, MetricPointVersions>>>;
const KIB: usize = 1024;
const READER_PATH_COUNTER: &str = "rustfs_io_get_object_reader_path_by_size_total";
/// Physical bytes the erasure layer pulled from disk, emitted per shard read by
/// `crates/ecstore/src/erasure/coding/decode.rs`.
const SHARD_READ_BYTES_COUNTER: &str = "rustfs_io_get_object_shard_read_observed_bytes_total";
const MSGPACK_JSON_DECODE_COUNTER: &str = "rustfs_system_network_internode_msgpack_json_decode_total";
const MSGPACK_JSON_FALLBACK_COUNTER: &str = "rustfs_system_network_internode_msgpack_json_fallback_total";
const MSGPACK_JSON_DECODE_ERROR_COUNTER: &str = "rustfs_system_network_internode_msgpack_json_decode_error_total";
@@ -146,6 +149,7 @@ struct OtlpMetricCollector {
decode_values: MetricValues,
fallback_values: MetricValues,
decode_error_values: MetricValues,
shard_read_values: MetricValues,
task: JoinHandle<()>,
}
@@ -157,10 +161,12 @@ impl OtlpMetricCollector {
let decode_values = Arc::new(Mutex::new(BTreeMap::new()));
let fallback_values = Arc::new(Mutex::new(BTreeMap::new()));
let decode_error_values = Arc::new(Mutex::new(BTreeMap::new()));
let shard_read_values = Arc::new(Mutex::new(BTreeMap::new()));
let task_values = values.clone();
let task_decode_values = decode_values.clone();
let task_fallback_values = fallback_values.clone();
let task_decode_error_values = decode_error_values.clone();
let task_shard_read_values = shard_read_values.clone();
let task = tokio::spawn(async move {
loop {
let Ok((stream, _)) = listener.accept().await else {
@@ -170,6 +176,7 @@ impl OtlpMetricCollector {
let decode_values = task_decode_values.clone();
let fallback_values = task_fallback_values.clone();
let decode_error_values = task_decode_error_values.clone();
let shard_read_values = task_shard_read_values.clone();
tokio::spawn(async move {
let _ = hyper::server::conn::http1::Builder::new()
.serve_connection(
@@ -181,6 +188,7 @@ impl OtlpMetricCollector {
decode_values.clone(),
fallback_values.clone(),
decode_error_values.clone(),
shard_read_values.clone(),
)
}),
)
@@ -194,10 +202,48 @@ impl OtlpMetricCollector {
decode_values,
fallback_values,
decode_error_values,
shard_read_values,
task,
})
}
/// Total physical bytes read from disk across every shard-read label set.
async fn shard_read_bytes_total(&self) -> u64 {
self.shard_read_values
.lock()
.await
.values()
.map(|versions| versions.values().map(|(_, value)| *value).sum::<u64>())
.sum()
}
/// Waits until the shard-read counter stops advancing so a measurement window
/// is not polluted by exports still in flight.
///
/// Requires several consecutive equal samples spanning more than one export
/// interval (`RUSTFS_OBS_METER_INTERVAL=1`): a single unchanged sample only
/// proves the latest export has not landed yet, which silently reads as "no
/// disk reads happened" and makes any upper-bound assertion vacuous.
async fn wait_for_shard_read_bytes_to_settle(&self) -> TestResult<u64> {
const REQUIRED_STABLE_SAMPLES: usize = 5;
let mut last = self.shard_read_bytes_total().await;
let mut stable = 0;
for _ in 0..60 {
sleep(Duration::from_millis(500)).await;
let current = self.shard_read_bytes_total().await;
if current == last {
stable += 1;
if stable >= REQUIRED_STABLE_SAMPLES {
return Ok(current);
}
} else {
stable = 0;
last = current;
}
}
Err("timed out waiting for shard-read byte counter to settle".into())
}
async fn reader_path_total(&self, path: &str, object_class: &str, size_bucket: &str) -> u64 {
self.reader_path_values(path, object_class, size_bucket).await.values().sum()
}
@@ -321,6 +367,7 @@ async fn handle_metric_export(
decode_values: MetricValues,
fallback_values: MetricValues,
decode_error_values: MetricValues,
shard_read_values: MetricValues,
) -> Result<Response<Full<Bytes>>, Infallible> {
if request.uri().path() != "/v1/metrics" {
return Ok(response(StatusCode::NOT_FOUND));
@@ -354,7 +401,9 @@ async fn handle_metric_export(
let mut decode_values = decode_values.lock().await;
let mut fallback_values = fallback_values.lock().await;
let mut decode_error_values = decode_error_values.lock().await;
let mut shard_read_values = shard_read_values.lock().await;
record_reader_path_metrics(&export, &mut values);
record_shard_read_bytes_metrics(&export, &mut shard_read_values);
record_msgpack_decode_metrics(&export, &mut decode_values);
record_msgpack_fallback_metrics(&export, &mut fallback_values);
record_msgpack_decode_error_metrics(&export, &mut decode_error_values);
@@ -375,6 +424,50 @@ fn reader_path_metric_key(path: &str, object_class: &str, size_bucket: &str) ->
format!("{path}\u{1f}{object_class}\u{1f}{size_bucket}")
}
/// Accumulates `SHARD_READ_BYTES_COUNTER` across all label sets. Only the total
/// matters: it is the number of physical bytes the erasure layer actually pulled
/// from disk, which is what separates a bounded per-part read from a decode of
/// the whole object.
fn record_shard_read_bytes_metrics(export: &ExportMetricsServiceRequest, values: &mut BTreeMap<String, MetricPointVersions>) {
for resource_metrics in &export.resource_metrics {
for scope_metrics in &resource_metrics.scope_metrics {
for metric in &scope_metrics.metrics {
if metric.name != SHARD_READ_BYTES_COUNTER {
continue;
}
let Some(metric::Data::Sum(sum)) = &metric.data else {
continue;
};
for point in &sum.data_points {
let Some(number_data_point::Value::AsInt(value)) = point.value.as_ref() else {
continue;
};
let value = u64::try_from(*value).unwrap_or_default();
// Keyed by labels, not by position: point order within an export
// is not guaranteed stable, so an index key would alias distinct
// series across batches.
let key = format!(
"{}\u{1f}{}\u{1f}{}",
attribute_string(&point.attributes, "path").unwrap_or_default(),
attribute_string(&point.attributes, "role").unwrap_or_default(),
attribute_string(&point.attributes, "outcome").unwrap_or_default(),
);
values
.entry(key)
.or_default()
.entry(point.start_time_unix_nano)
.and_modify(|current| {
if point.time_unix_nano >= current.0 {
*current = (point.time_unix_nano, value);
}
})
.or_insert((point.time_unix_nano, value));
}
}
}
}
}
fn record_reader_path_metrics(export: &ExportMetricsServiceRequest, values: &mut BTreeMap<String, MetricPointVersions>) {
for resource_metrics in &export.resource_metrics {
for scope_metrics in &resource_metrics.scope_metrics {
@@ -1864,6 +1957,86 @@ async fn four_node_multipart_disk_compression_roundtrip() -> TestResult {
Ok(())
}
/// A tail range over a compressed multipart object must read only the physical
/// data it needs, not decode the object from byte zero.
///
/// The byte-exactness tests around this one stay green even if the seek path
/// regresses into decoding from the start of the object: the bytes returned are
/// still correct, only the read amplification explodes. This asserts the cost
/// side, using `SHARD_READ_BYTES_COUNTER` — already emitted per shard read by the
/// erasure layer, so no production code is instrumented for the test.
///
/// `get_compressed_offsets` skips whole preceding parts by their stored size and
/// then seeks inside the covering part via its compression index, so a bounded
/// read costs on the order of the covering part's block size against a ~5 MiB
/// object.
#[tokio::test]
#[serial]
async fn four_node_compressed_multipart_tail_range_reads_are_bounded() -> TestResult {
init_logging();
let collector = OtlpMetricCollector::start().await?;
let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
configure_reader_metric_cluster(&mut cluster, &collector);
cluster.set_env("RUSTFS_COMPRESSION_ENABLED", "true");
cluster.set_env("RUSTFS_COMPRESSION_MULTIPART_ENABLED", "true");
cluster.start().await?;
let bucket = "inline-multipart-compression-tail-range";
cluster.create_test_bucket(bucket).await?;
let client = cluster.create_s3_client(0)?;
let key = "multipart/tail-range.txt";
let (body, _second_part, etag) = put_two_part_multipart(&client, bucket, key).await?;
// Establish that the object really took the compressed read path; otherwise a
// small delta below would only prove compression never happened.
assert_reader_path(
&collector,
&client,
ReaderPathExpectation::for_class(ReaderObject::new(bucket, key, &body, etag.as_deref(), None), LEGACY_DUPLEX, COMPRESSED),
)
.await?;
let baseline = collector.wait_for_shard_read_bytes_to_settle().await?;
let tail_len = 4 * KIB;
let start = body.len() - tail_len;
let end = body.len() - 1;
let range = client
.get_object()
.bucket(bucket)
.key(key)
.range(format!("bytes={start}-{end}"))
.send()
.await?;
let tail = range.body.collect().await?.into_bytes();
assert_eq!(tail.as_ref(), &body[start..], "tail range returned wrong bytes");
let after = collector.wait_for_shard_read_bytes_to_settle().await?;
let read_bytes = after.saturating_sub(baseline);
// A zero delta means the window caught nothing — an unexported counter, or a
// read served without touching the erasure layer — which would make the upper
// bound vacuously true. Fail instead of passing blind.
assert!(
read_bytes > 0,
"no shard reads observed for the tail range; the budget assertion below would be vacuous"
);
// Part 1 alone is MPU_PART_1_SIZE, so a whole-object decode cannot come in
// under it. Half the logical size leaves generous headroom for erasure padding
// and unrelated background reads while still failing loudly on a full decode.
let budget = (body.len() / 2) as u64;
assert!(
read_bytes < budget,
"tail range read {read_bytes} physical bytes for a {tail_len}-byte range (budget {budget}, object {} bytes): \
the read is not bounded to the covering part",
body.len()
);
Ok(())
}
#[tokio::test]
#[serial]
async fn four_node_mixed_msgpack_compat_mode_preserves_fallback_controls() -> TestResult {
@@ -30,7 +30,6 @@ use md5::{Digest as Md5Digest, Md5};
use rustfs_signer::constants::UNSIGNED_PAYLOAD;
use rustfs_signer::sign_v4;
use s3s::Body;
use serial_test::serial;
use std::collections::HashMap;
use std::error::Error;
use std::io::Cursor;
@@ -356,7 +355,6 @@ async fn run_post_object_policy_case(
/// smuggles one extra field the policy never declared, and the upload must be
/// rejected with 403 AccessDenied naming the offending field.
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_fields_missing_from_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -484,7 +482,6 @@ async fn test_anonymous_post_object_rejects_fields_missing_from_policy_condition
/// sends a different one, and the upload must be rejected with 400
/// InvalidPolicyDocument naming the field.
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_exact_condition_policy_mismatches()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -689,7 +686,6 @@ async fn test_anonymous_post_object_rejects_exact_condition_policy_mismatches()
/// one of them with a different value, and the upload must be rejected with
/// 400 InvalidPolicyDocument naming the mismatched field.
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_object_lock_policy_mismatches() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
@@ -757,7 +753,6 @@ async fn test_anonymous_post_object_rejects_object_lock_policy_mismatches() -> R
/// exact values, the form sends a different parameter value, and the upload
/// must be rejected with 400 InvalidPolicyDocument naming the parameter.
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_sse_kms_policy_mismatches() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -839,7 +834,6 @@ async fn test_anonymous_post_object_rejects_sse_kms_policy_mismatches() -> Resul
/// NotImplemented (SSE-KMS POST uploads are not implemented), not with a
/// policy error.
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_sse_kms_params_outside_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -894,7 +888,6 @@ async fn test_anonymous_post_object_rejects_sse_kms_params_outside_policy_condit
}
#[tokio::test]
#[serial]
async fn test_anonymous_multipart_control_apis_require_auth() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -968,7 +961,6 @@ async fn test_anonymous_multipart_control_apis_require_auth() -> Result<(), Box<
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_requires_auth() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1002,7 +994,6 @@ async fn test_anonymous_post_object_requires_auth() -> Result<(), Box<dyn std::e
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_honors_success_action_status() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1066,7 +1057,6 @@ async fn test_anonymous_post_object_honors_success_action_status() -> Result<(),
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_honors_success_action_redirect() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1139,7 +1129,6 @@ async fn test_anonymous_post_object_honors_success_action_redirect() -> Result<(
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_defaults_to_no_content() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1185,7 +1174,6 @@ async fn test_anonymous_post_object_defaults_to_no_content() -> Result<(), Box<d
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_sse_kms() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1232,7 +1220,6 @@ async fn test_anonymous_post_object_rejects_sse_kms() -> Result<(), Box<dyn std:
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_sse_s3() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1290,7 +1277,6 @@ async fn test_anonymous_post_object_accepts_sse_s3() -> Result<(), Box<dyn std::
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_uses_bucket_default_sse_s3() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1363,7 +1349,6 @@ async fn test_anonymous_post_object_uses_bucket_default_sse_s3() -> Result<(), B
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_uses_bucket_default_sse_kms() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1437,7 +1422,6 @@ async fn test_anonymous_post_object_uses_bucket_default_sse_kms() -> Result<(),
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_sse_s3_policy_mismatch() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1488,7 +1472,6 @@ async fn test_anonymous_post_object_rejects_sse_s3_policy_mismatch() -> Result<(
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_sse_s3_missing_from_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1552,7 +1535,6 @@ async fn test_anonymous_post_object_accepts_sse_s3_missing_from_policy_condition
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_storage_class_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1606,7 +1588,6 @@ async fn test_anonymous_post_object_accepts_storage_class_exact_policy_match()
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_storage_class_missing_from_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1657,7 +1638,6 @@ async fn test_anonymous_post_object_rejects_storage_class_missing_from_policy_co
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_invalid_storage_class_value() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
@@ -1709,7 +1689,6 @@ async fn test_anonymous_post_object_rejects_invalid_storage_class_value() -> Res
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_checksum_algorithm_missing_from_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1765,7 +1744,6 @@ async fn test_anonymous_post_object_rejects_checksum_algorithm_missing_from_poli
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_checksum_algorithm_policy_mismatch()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1822,7 +1800,6 @@ async fn test_anonymous_post_object_rejects_checksum_algorithm_policy_mismatch()
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_checksum_auxiliary_fields_missing_from_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1886,7 +1863,6 @@ async fn test_anonymous_post_object_rejects_checksum_auxiliary_fields_missing_fr
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_allows_sse_c_fields_outside_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -1963,7 +1939,6 @@ async fn test_anonymous_post_object_allows_sse_c_fields_outside_policy_condition
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_sse_c_exact_policy_mismatch() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
@@ -2022,7 +1997,6 @@ async fn test_anonymous_post_object_rejects_sse_c_exact_policy_mismatch() -> Res
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_duplicate_key_form_values() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2072,7 +2046,6 @@ async fn test_anonymous_post_object_rejects_duplicate_key_form_values() -> Resul
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_invalid_success_action_status() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
@@ -2120,7 +2093,6 @@ async fn test_anonymous_post_object_rejects_invalid_success_action_status() -> R
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_invalid_success_action_redirect()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2168,7 +2140,6 @@ async fn test_anonymous_post_object_rejects_invalid_success_action_redirect()
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_form_fields_missing_from_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2223,7 +2194,6 @@ async fn test_anonymous_post_object_rejects_form_fields_missing_from_policy_cond
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_form_fields_covered_by_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2280,7 +2250,6 @@ async fn test_anonymous_post_object_accepts_form_fields_covered_by_policy_condit
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_starts_with_policy_mismatch() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
@@ -2335,7 +2304,6 @@ async fn test_anonymous_post_object_rejects_starts_with_policy_mismatch() -> Res
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_content_length_range_violation()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2388,7 +2356,6 @@ async fn test_anonymous_post_object_rejects_content_length_range_violation()
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_success_action_status_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2445,7 +2412,6 @@ async fn test_anonymous_post_object_accepts_success_action_status_exact_policy_m
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_success_action_redirect_policy_mismatch()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2502,7 +2468,6 @@ async fn test_anonymous_post_object_rejects_success_action_redirect_policy_misma
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_success_action_redirect_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2568,7 +2533,6 @@ async fn test_anonymous_post_object_accepts_success_action_redirect_exact_policy
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_success_action_redirect_missing_from_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2621,7 +2585,6 @@ async fn test_anonymous_post_object_rejects_success_action_redirect_missing_from
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_metadata_field_covered_by_starts_with()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2676,7 +2639,6 @@ async fn test_anonymous_post_object_accepts_metadata_field_covered_by_starts_wit
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_content_type_field_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2734,7 +2696,6 @@ async fn test_anonymous_post_object_accepts_content_type_field_exact_policy_matc
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_content_type_field_covered_by_starts_with()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2792,7 +2753,6 @@ async fn test_anonymous_post_object_accepts_content_type_field_covered_by_starts
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_content_disposition_field_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2850,7 +2810,6 @@ async fn test_anonymous_post_object_accepts_content_disposition_field_exact_poli
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_cache_control_field_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2908,7 +2867,6 @@ async fn test_anonymous_post_object_accepts_cache_control_field_exact_policy_mat
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_content_language_field_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -2966,7 +2924,6 @@ async fn test_anonymous_post_object_accepts_content_language_field_exact_policy_
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_content_encoding_field_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3024,7 +2981,6 @@ async fn test_anonymous_post_object_accepts_content_encoding_field_exact_policy_
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_website_redirect_location_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3082,7 +3038,6 @@ async fn test_anonymous_post_object_accepts_website_redirect_location_exact_poli
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_expires_field_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3140,7 +3095,6 @@ async fn test_anonymous_post_object_accepts_expires_field_exact_policy_match()
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_object_lock_retention_without_permission()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3196,7 +3150,6 @@ async fn test_anonymous_post_object_rejects_object_lock_retention_without_permis
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_object_lock_retention_missing_from_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3256,7 +3209,6 @@ async fn test_anonymous_post_object_rejects_object_lock_retention_missing_from_p
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_object_lock_legal_hold_without_permission()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3309,7 +3261,6 @@ async fn test_anonymous_post_object_rejects_object_lock_legal_hold_without_permi
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_object_lock_legal_hold_policy_mismatch()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3368,7 +3319,6 @@ async fn test_anonymous_post_object_rejects_object_lock_legal_hold_policy_mismat
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_object_lock_legal_hold_missing_from_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3426,7 +3376,6 @@ async fn test_anonymous_post_object_rejects_object_lock_legal_hold_missing_from_
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_tagging_field_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3492,7 +3441,6 @@ async fn test_anonymous_post_object_accepts_tagging_field_exact_policy_match()
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_accepts_metadata_field_exact_policy_match()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3551,7 +3499,6 @@ async fn test_anonymous_post_object_accepts_metadata_field_exact_policy_match()
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_allows_x_ignore_fields_outside_policy_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3604,7 +3551,6 @@ async fn test_anonymous_post_object_allows_x_ignore_fields_outside_policy_condit
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_sigv4_date_policy_mismatch() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3657,7 +3603,6 @@ async fn test_anonymous_post_object_rejects_sigv4_date_policy_mismatch() -> Resu
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_mismatched_bucket_form_field() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
@@ -3712,7 +3657,6 @@ async fn test_anonymous_post_object_rejects_mismatched_bucket_form_field() -> Re
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_multiple_bucket_values() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3764,7 +3708,6 @@ async fn test_anonymous_post_object_rejects_multiple_bucket_values() -> Result<(
}
#[tokio::test]
#[serial]
async fn test_anonymous_post_object_rejects_extra_content_disposition_field()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3820,7 +3763,6 @@ async fn test_anonymous_post_object_rejects_extra_content_disposition_field()
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_expands_tar_entries_with_prefix_headers()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3891,7 +3833,6 @@ async fn test_signed_put_object_extract_expands_tar_entries_with_prefix_headers(
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_preserves_request_metadata_on_extracted_objects()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -3956,7 +3897,6 @@ async fn test_signed_put_object_extract_preserves_request_metadata_on_extracted_
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_preserves_sse_s3_and_redirect() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4004,7 +3944,6 @@ async fn test_signed_put_object_extract_preserves_sse_s3_and_redirect() -> Resul
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_preserves_storage_class() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4047,7 +3986,6 @@ async fn test_signed_put_object_extract_preserves_storage_class() -> Result<(),
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_rejects_invalid_storage_class() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4083,7 +4021,6 @@ async fn test_signed_put_object_extract_rejects_invalid_storage_class() -> Resul
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_rejects_write_offset_bytes_header() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4137,7 +4074,6 @@ async fn test_signed_put_object_rejects_write_offset_bytes_header() -> Result<()
}
#[tokio::test]
#[serial]
async fn test_raw_signed_put_object_write_offset_bytes_returns_minio_compatible_error_body()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4176,7 +4112,6 @@ async fn test_raw_signed_put_object_write_offset_bytes_returns_minio_compatible_
}
#[tokio::test]
#[serial]
async fn test_anonymous_put_object_write_offset_bytes_returns_minio_compatible_error_body()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4235,7 +4170,6 @@ async fn test_anonymous_put_object_write_offset_bytes_returns_minio_compatible_e
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_uses_bucket_default_sse_s3() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4300,7 +4234,6 @@ async fn test_signed_put_object_extract_uses_bucket_default_sse_s3() -> Result<(
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_rejects_bucket_default_sse_kms() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4356,7 +4289,6 @@ async fn test_signed_put_object_extract_rejects_bucket_default_sse_kms() -> Resu
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_preserves_sse_c() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4421,7 +4353,6 @@ async fn test_signed_put_object_extract_preserves_sse_c() -> Result<(), Box<dyn
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_preserves_object_lock_legal_hold() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
@@ -4476,7 +4407,6 @@ async fn test_signed_put_object_extract_preserves_object_lock_legal_hold() -> Re
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_preserves_object_lock_retention() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
@@ -4536,7 +4466,6 @@ async fn test_signed_put_object_extract_preserves_object_lock_retention() -> Res
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_pax_retention_overrides_request_retention()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4600,7 +4529,6 @@ async fn test_signed_put_object_extract_pax_retention_overrides_request_retentio
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_returns_archive_etag() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4634,7 +4562,6 @@ async fn test_signed_put_object_extract_returns_archive_etag() -> Result<(), Box
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_preserves_entry_mtime() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4670,7 +4597,6 @@ async fn test_signed_put_object_extract_preserves_entry_mtime() -> Result<(), Bo
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_preserves_pax_metadata_and_version_id()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -4724,7 +4650,6 @@ async fn test_signed_put_object_extract_preserves_pax_metadata_and_version_id()
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_authorizes_each_pax_privilege_and_retention_conditions()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -5034,7 +4959,6 @@ async fn test_signed_put_object_extract_authorizes_each_pax_privilege_and_retent
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_accepts_compat_header() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -5076,7 +5000,6 @@ async fn test_signed_put_object_extract_accepts_compat_header() -> Result<(), Bo
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_preserves_directory_markers_by_default()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -5137,7 +5060,6 @@ async fn test_signed_put_object_extract_preserves_directory_markers_by_default()
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_expands_tar_gz_archive() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -5189,7 +5111,6 @@ async fn test_signed_put_object_extract_expands_tar_gz_archive() -> Result<(), B
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_expands_tgz_archive() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -5241,7 +5162,6 @@ async fn test_signed_put_object_extract_expands_tgz_archive() -> Result<(), Box<
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_expands_tbz2_archive() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -5293,7 +5213,6 @@ async fn test_signed_put_object_extract_expands_tbz2_archive() -> Result<(), Box
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_expands_txz_archive() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -5345,7 +5264,6 @@ async fn test_signed_put_object_extract_expands_txz_archive() -> Result<(), Box<
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_skips_invalid_entry_when_ignore_errors_enabled()
-> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -5419,7 +5337,6 @@ async fn test_signed_put_object_extract_skips_invalid_entry_when_ignore_errors_e
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_normalizes_prefix_header_value() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -5462,7 +5379,6 @@ async fn test_signed_put_object_extract_normalizes_prefix_header_value() -> Resu
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_expands_tzst_archive() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -5514,7 +5430,6 @@ async fn test_signed_put_object_extract_expands_tzst_archive() -> Result<(), Box
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_rejects_missing_archive_extension() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
{
init_logging();
@@ -5548,7 +5463,6 @@ async fn test_signed_put_object_extract_rejects_missing_archive_extension() -> R
}
#[tokio::test]
#[serial]
async fn test_signed_put_object_extract_rejects_invalid_tar_gz_payload() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
init_logging();
@@ -33,7 +33,6 @@ use aws_sdk_s3::types::{
ObjectLockMode, ObjectLockRetentionMode,
};
use chrono::{DateTime, Duration, Utc};
use serial_test::serial;
use tracing::info;
/// Initialize test logging
@@ -107,7 +106,6 @@ fn parse_s3_datetime(value: &aws_sdk_s3::primitives::DateTime) -> DateTime<Utc>
// ============================================================================
#[tokio::test]
#[serial]
async fn test_delete_object_blocked_by_compliance_retention() {
init_logging();
info!("🧪 Test: DeleteObject blocked by COMPLIANCE retention");
@@ -145,7 +143,6 @@ async fn test_delete_object_blocked_by_compliance_retention() {
}
#[tokio::test]
#[serial]
async fn test_delete_object_blocked_by_governance_without_bypass() {
init_logging();
info!("🧪 Test: DeleteObject blocked by GOVERNANCE retention without bypass");
@@ -175,7 +172,6 @@ async fn test_delete_object_blocked_by_governance_without_bypass() {
}
#[tokio::test]
#[serial]
async fn test_delete_object_allowed_by_governance_with_bypass() {
init_logging();
info!("🧪 Test: DeleteObject allowed by GOVERNANCE retention with bypass");
@@ -215,7 +211,6 @@ async fn test_delete_object_allowed_by_governance_with_bypass() {
}
#[tokio::test]
#[serial]
async fn test_delete_object_creates_delete_marker_for_retained_current_version() {
init_logging();
info!("🧪 Test: DeleteObject creates delete marker for retained current version");
@@ -266,7 +261,6 @@ async fn test_delete_object_creates_delete_marker_for_retained_current_version()
}
#[tokio::test]
#[serial]
async fn test_delete_object_blocked_by_legal_hold() {
init_logging();
info!("🧪 Test: DeleteObject blocked by Legal Hold");
@@ -299,7 +293,6 @@ async fn test_delete_object_blocked_by_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_delete_object_allowed_with_legal_hold_off() {
init_logging();
info!("🧪 Test: DeleteObject allowed with Legal Hold OFF");
@@ -335,7 +328,6 @@ async fn test_delete_object_allowed_with_legal_hold_off() {
}
#[tokio::test]
#[serial]
async fn test_delete_object_after_legal_hold_removed() {
init_logging();
info!("🧪 Test: DeleteObject succeeds after Legal Hold is removed");
@@ -369,7 +361,6 @@ async fn test_delete_object_after_legal_hold_removed() {
}
#[tokio::test]
#[serial]
async fn test_get_object_legal_hold_returns_updated_status() {
init_logging();
info!("🧪 Test: GetObjectLegalHold returns updated status");
@@ -425,7 +416,6 @@ async fn test_get_object_legal_hold_returns_updated_status() {
}
#[tokio::test]
#[serial]
async fn test_get_object_retention_returns_configured_values() {
init_logging();
info!("🧪 Test: GetObjectRetention returns configured values");
@@ -476,7 +466,6 @@ async fn test_get_object_retention_returns_configured_values() {
// creating a new current version. The lock protects the existing version
// from deletion; it never blocks new versions.
#[tokio::test]
#[serial]
async fn test_put_object_overwrite_creates_new_version_under_legal_hold() {
init_logging();
info!("🧪 Test: PutObject overwrite of a legal-hold version creates a new version");
@@ -561,7 +550,6 @@ async fn test_put_object_overwrite_creates_new_version_under_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_copy_object_applies_requested_legal_hold() {
init_logging();
info!("🧪 Test: CopyObject applies requested Legal Hold");
@@ -613,7 +601,6 @@ async fn test_copy_object_applies_requested_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_copy_object_does_not_inherit_source_legal_hold() {
init_logging();
info!("🧪 Test: CopyObject does not inherit source Legal Hold");
@@ -707,7 +694,6 @@ async fn test_copy_object_does_not_inherit_source_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_copy_object_overwrite_creates_new_version_under_legal_hold() {
init_logging();
info!("🧪 Test: CopyObject overwrite of a legal-hold destination creates a new version");
@@ -787,7 +773,6 @@ async fn test_copy_object_overwrite_creates_new_version_under_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_create_multipart_upload_applies_requested_legal_hold() {
init_logging();
info!("🧪 Test: CreateMultipartUpload applies requested Legal Hold");
@@ -853,7 +838,6 @@ async fn test_create_multipart_upload_applies_requested_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_create_multipart_upload_creates_new_version_under_compliance_retention() {
init_logging();
info!("🧪 Test: CreateMultipartUpload over a COMPLIANCE-retained key creates a new version");
@@ -933,7 +917,6 @@ async fn test_create_multipart_upload_creates_new_version_under_compliance_reten
}
#[tokio::test]
#[serial]
async fn test_delete_completed_multipart_object_blocked_by_legal_hold() {
init_logging();
info!("🧪 Test: Delete completed multipart object blocked by Legal Hold");
@@ -993,7 +976,6 @@ async fn test_delete_completed_multipart_object_blocked_by_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_delete_completed_multipart_object_blocked_by_retention() {
init_logging();
info!("🧪 Test: Delete completed multipart object blocked by retention");
@@ -1055,7 +1037,6 @@ async fn test_delete_completed_multipart_object_blocked_by_retention() {
}
#[tokio::test]
#[serial]
async fn test_complete_multipart_upload_creates_new_version_under_legal_hold() {
init_logging();
info!("🧪 Test: CompleteMultipartUpload creates a new version when the current version is under Legal Hold");
@@ -1135,7 +1116,6 @@ async fn test_complete_multipart_upload_creates_new_version_under_legal_hold() {
}
#[tokio::test]
#[serial]
async fn test_complete_multipart_upload_creates_new_version_under_compliance_retention() {
init_logging();
info!("🧪 Test: CompleteMultipartUpload creates a new version when the current version is under COMPLIANCE retention");
@@ -1209,7 +1189,6 @@ async fn test_complete_multipart_upload_creates_new_version_under_compliance_ret
}
#[tokio::test]
#[serial]
async fn test_write_paths_require_put_object_legal_hold_permission() {
init_logging();
info!("🧪 Test: write paths require PutObjectLegalHold permission");
@@ -1273,7 +1252,6 @@ async fn test_write_paths_require_put_object_legal_hold_permission() {
}
#[tokio::test]
#[serial]
async fn test_write_paths_require_put_object_retention_permission() {
init_logging();
info!("🧪 Test: write paths require PutObjectRetention permission");
@@ -1345,7 +1323,6 @@ async fn test_write_paths_require_put_object_retention_permission() {
// ============================================================================
#[tokio::test]
#[serial]
async fn test_delete_objects_mixed_locked_unlocked() {
init_logging();
info!("🧪 Test: DeleteObjects with mixed locked and unlocked objects");
@@ -1427,7 +1404,6 @@ async fn test_delete_objects_mixed_locked_unlocked() {
// ============================================================================
#[tokio::test]
#[serial]
async fn test_put_retention_compliance_cannot_shorten() {
init_logging();
info!("🧪 Test: PutObjectRetention cannot shorten COMPLIANCE retention");
@@ -1468,7 +1444,6 @@ async fn test_put_retention_compliance_cannot_shorten() {
}
#[tokio::test]
#[serial]
async fn test_put_retention_compliance_can_extend() {
init_logging();
info!("🧪 Test: PutObjectRetention can extend COMPLIANCE retention");
@@ -1509,7 +1484,6 @@ async fn test_put_retention_compliance_can_extend() {
}
#[tokio::test]
#[serial]
async fn test_put_retention_governance_extend_without_bypass() {
init_logging();
info!("🧪 Test: PutObjectRetention on GOVERNANCE can extend without bypass");
@@ -1553,7 +1527,6 @@ async fn test_put_retention_governance_extend_without_bypass() {
}
#[tokio::test]
#[serial]
async fn test_put_retention_governance_shorten_requires_bypass() {
init_logging();
info!("🧪 Test: PutObjectRetention on GOVERNANCE requires bypass to shorten");
@@ -1615,7 +1588,6 @@ async fn test_put_retention_governance_shorten_requires_bypass() {
// ============================================================================
#[tokio::test]
#[serial]
async fn test_default_retention_applied_to_new_objects() {
init_logging();
info!("🧪 Test: Default retention is applied to new objects");
@@ -1685,7 +1657,6 @@ async fn test_default_retention_applied_to_new_objects() {
}
#[tokio::test]
#[serial]
async fn test_delete_object_creates_delete_marker_for_default_retained_current_version() {
init_logging();
info!("🧪 Test: DeleteObject creates delete marker for default-retained current version");
@@ -1770,7 +1741,6 @@ async fn test_delete_object_creates_delete_marker_for_default_retained_current_v
}
#[tokio::test]
#[serial]
async fn test_put_copy_and_multipart_reject_incomplete_retention_headers() {
init_logging();
info!("🧪 Test: write paths reject incomplete Object Lock retention headers");
@@ -1869,7 +1839,6 @@ async fn test_put_copy_and_multipart_reject_incomplete_retention_headers() {
}
#[tokio::test]
#[serial]
async fn test_copy_object_retention_uses_destination_policy() {
init_logging();
info!("🧪 Test: CopyObject retention follows destination policy");
@@ -2051,7 +2020,6 @@ async fn test_copy_object_retention_uses_destination_policy() {
}
#[tokio::test]
#[serial]
async fn test_multipart_default_retention_fixed_at_create() {
init_logging();
info!("🧪 Test: multipart default retention is fixed at CreateMultipartUpload");
@@ -2122,7 +2090,6 @@ async fn test_multipart_default_retention_fixed_at_create() {
// ============================================================================
#[tokio::test]
#[serial]
async fn test_unretained_object_lock_object_delete_and_bucket_cleanup() {
init_logging();
info!("🧪 Test: Unretained Object Lock object delete and bucket cleanup (Issue #5339)");
@@ -2243,7 +2210,6 @@ async fn test_unretained_object_lock_object_delete_and_bucket_cleanup() {
}
#[tokio::test]
#[serial]
async fn test_versioning_auto_enabled_with_object_lock() {
init_logging();
info!("🧪 Test: Versioning is auto-enabled when Object Lock is configured");
@@ -2302,7 +2268,6 @@ async fn test_versioning_auto_enabled_with_object_lock() {
// ============================================================================
#[tokio::test]
#[serial]
async fn test_error_message_distinguishes_legal_hold_from_retention() {
init_logging();
info!("🧪 Test: Error messages distinguish Legal Hold from Retention");
File diff suppressed because it is too large Load Diff
+1
View File
@@ -273,6 +273,7 @@ proptest = "1"
rcgen.workspace = true
insta = { workspace = true, features = ["yaml", "json"] }
rustfs-crypto = { workspace = true }
tonic-prost = { workspace = true }
[build-dependencies]
shadow-rs = { workspace = true, default-features = false, features = ["build", "metadata"] }
+10 -8
View File
@@ -32,7 +32,7 @@ pub mod bucket {
pub mod bucket_target_sys {
pub use crate::bucket::bucket_target_sys::{
AdvancedPutOptions, BucketTargetError, BucketTargetSys, PutObjectOptions, RemoveObjectOptions, S3ClientError,
TargetClient, append_version_id_query,
SsecPassthroughCapability, TargetClient, append_version_id_query,
};
}
@@ -198,12 +198,13 @@ pub mod bucket {
ReplicationType, ResyncOpts, ResyncStatusType, RuntimeReplicationTargetBacklog, TargetReplicationResyncStatus,
VersionPurgeStatusType, XferStats, commit_force_delete_intent, complete_force_delete_intent,
delete_replication_state_from_config, delete_replication_version_id, get_global_replication_pool,
get_global_replication_stats, init_background_replication, invalid_replication_config_status_field,
persist_force_delete_intent, read_durable_mrf_backlog, replication_state_to_filemeta, replication_status_to_filemeta,
replication_statuses_map, replication_target_arns, resync_start_conflict_id, should_remove_replication_target,
should_schedule_delete_replication, should_use_existing_delete_replication_info,
should_use_existing_delete_replication_source, unsupported_replication_config_field,
validate_replication_config_structure, validate_replication_config_target_arns, version_purge_status_to_filemeta,
get_global_replication_stats, get_proxy_targets, init_background_replication,
invalid_replication_config_status_field, persist_force_delete_intent, read_durable_mrf_backlog,
replication_state_to_filemeta, replication_status_to_filemeta, replication_statuses_map, replication_target_arns,
resync_start_conflict_id, should_remove_replication_target, should_schedule_delete_replication,
should_use_existing_delete_replication_info, should_use_existing_delete_replication_source,
unsupported_replication_config_field, validate_replication_config_structure, validate_replication_config_target_arns,
version_purge_status_to_filemeta,
};
}
@@ -380,7 +381,7 @@ pub mod erasure {
pub mod event {
pub use crate::event::name::EventName;
pub use crate::services::event_notification::{EventArgs, register_event_dispatch_hook};
pub use crate::services::event_notification::{EventArgs, register_event_dispatch_hook, send_event};
}
pub mod global {
@@ -483,6 +484,7 @@ pub mod store_list {
}
pub mod storage {
pub use crate::core::pools::HealLifecycleExpiryContext;
pub use crate::store::HealWalkVersion;
pub use crate::store::{
ECStore, all_local_disk, all_local_disk_path, find_local_disk_by_ref, init_local_disks,
+309 -5
View File
@@ -27,10 +27,15 @@ use aws_sdk_s3::config::SharedHttpClient;
use aws_sdk_s3::error::ProvideErrorMetadata;
use aws_sdk_s3::error::SdkError;
use aws_sdk_s3::operation::complete_multipart_upload::CompleteMultipartUploadOutput;
use aws_sdk_s3::operation::delete_object_tagging::{DeleteObjectTaggingError, DeleteObjectTaggingOutput};
use aws_sdk_s3::operation::get_object::{GetObjectError, GetObjectOutput};
use aws_sdk_s3::operation::get_object_tagging::{GetObjectTaggingError, GetObjectTaggingOutput};
use aws_sdk_s3::operation::head_bucket::HeadBucketError;
use aws_sdk_s3::operation::head_object::HeadObjectError;
use aws_sdk_s3::operation::put_object_tagging::{PutObjectTaggingError, PutObjectTaggingOutput};
use aws_sdk_s3::operation::upload_part::UploadPartOutput;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::Tagging as SdkTagging;
use aws_sdk_s3::types::{
ChecksumMode, CompletedMultipartUpload, CompletedPart, ObjectLockLegalHoldStatus, ObjectLockRetentionMode,
};
@@ -57,8 +62,8 @@ use rustfs_utils::http::{
is_rustfs_header, is_standard_header, is_storageclass_header,
};
use rustfs_utils::http::{
SUFFIX_FORCE_DELETE, SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_ETAG, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_REPLICATION_CHECK,
SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP, SUFFIX_SOURCE_REPLICATION_REQUEST,
SUFFIX_FORCE_DELETE, SUFFIX_SOURCE_DELETEMARKER, SUFFIX_SOURCE_ETAG, SUFFIX_SOURCE_MTIME, SUFFIX_SOURCE_PROXY_REQUEST,
SUFFIX_SOURCE_REPLICATION_CHECK, SUFFIX_SOURCE_REPLICATION_LEGALHOLD_TIMESTAMP, SUFFIX_SOURCE_REPLICATION_REQUEST,
SUFFIX_SOURCE_REPLICATION_RETENTION_TIMESTAMP, SUFFIX_SOURCE_REPLICATION_TAGGING_TIMESTAMP, SUFFIX_SOURCE_VERSION_ID,
insert_header,
};
@@ -294,9 +299,41 @@ struct TargetClientBuildProbe {
release: Arc<tokio::sync::Semaphore>,
}
/// SSE-C passthrough capability verdicts (see the enum's own docs in
/// `rustfs-replication`) are cached here per target ARN: entries follow the
/// `arn_remotes_map` lifecycle (rebuilding or removing a target resets its
/// capability to `Unknown`) and additionally expire after
/// [`SSEC_PASSTHROUGH_CAPABILITY_TTL`], after which the next attempt
/// re-audits. Re-exported so existing `bucket_target_sys` consumers keep
/// their import path while the verdict vocabulary lives with the
/// replication decision logic.
pub use crate::bucket::replication::SsecPassthroughCapability;
/// How long an audited SSE-C passthrough verdict stays authoritative.
///
/// Trade-off: without a TTL a verdict is sticky for the process lifetime —
/// an `Unsupported` target that gets upgraded (or re-probed only via
/// replication-check) would keep failing SSE-C replication forever, and the
/// fail-open twin: a `Supported` verdict would outlive a backend swapped
/// behind the same endpoint/ARN. With the TTL, a bad target costs at most
/// one wasted PUT+HEAD audit per TTL window, and a changed backend is
/// re-discovered within the same window.
pub const SSEC_PASSTHROUGH_CAPABILITY_TTL: Duration = Duration::from_secs(10 * 60);
/// A recorded SSE-C passthrough verdict plus when it was recorded, so reads
/// can report staleness against [`SSEC_PASSTHROUGH_CAPABILITY_TTL`].
#[derive(Debug, Clone, Copy)]
struct SsecPassthroughRecord {
capability: SsecPassthroughCapability,
recorded_at: Instant,
}
#[derive(Debug, Default)]
pub struct BucketTargetSys {
pub arn_remotes_map: Arc<RwLock<HashMap<String, ArnTarget>>>,
/// SSE-C passthrough capability verdicts keyed by target ARN. See
/// [`SsecPassthroughCapability`]; reset alongside `arn_remotes_map`.
ssec_passthrough_map: Arc<RwLock<HashMap<String, SsecPassthroughRecord>>>,
pub targets_map: Arc<RwLock<HashMap<String, Vec<BucketTarget>>>>,
pub h_mutex: Arc<RwLock<HashMap<String, EpHealth>>>,
target_h_mutex: Arc<RwLock<HashMap<String, EpHealth>>>,
@@ -317,6 +354,7 @@ impl BucketTargetSys {
fn new() -> Self {
Self {
arn_remotes_map: Arc::new(RwLock::new(HashMap::new())),
ssec_passthrough_map: Arc::new(RwLock::new(HashMap::new())),
targets_map: Arc::new(RwLock::new(HashMap::new())),
h_mutex: Arc::new(RwLock::new(HashMap::new())),
target_h_mutex: Arc::new(RwLock::new(HashMap::new())),
@@ -580,19 +618,59 @@ impl BucketTargetSys {
let update_mutex = self.target_update_mutex(bucket).await;
let _update_guard = update_mutex.lock().await;
// Lock order: targets_map, then arn_remotes_map, then target_h_mutex.
// Lock order: targets_map, then arn_remotes_map, then target_h_mutex,
// then ssec_passthrough_map (always last; also taken standalone by the
// capability accessors).
let mut targets_map = self.targets_map.write().await;
let mut arn_remotes_map = self.arn_remotes_map.write().await;
let mut health_map = self.target_h_mutex.write().await;
if let Some(targets) = targets_map.remove(bucket) {
let mut ssec_map = self.ssec_passthrough_map.write().await;
for target in targets {
arn_remotes_map.remove(&target.arn);
health_map.remove(&target.arn);
ssec_map.remove(&target.arn);
}
}
}
/// Cached SSE-C passthrough capability for a target ARN, plus whether the
/// verdict is older than [`SSEC_PASSTHROUGH_CAPABILITY_TTL`]. `(Unknown,
/// false)` when no verdict has been recorded since the target was built.
/// Staleness is computed here so the gate policy stays a pure function.
pub async fn ssec_passthrough_capability(&self, arn: &str) -> (SsecPassthroughCapability, bool) {
match self.ssec_passthrough_map.read().await.get(arn) {
Some(record) => (record.capability, record.recorded_at.elapsed() >= SSEC_PASSTHROUGH_CAPABILITY_TTL),
None => (SsecPassthroughCapability::Unknown, false),
}
}
/// Record an audited SSE-C passthrough verdict for a target ARN. Written by
/// the replication worker's HEAD-back audit and by the replication-check
/// SsecPassthrough probe phase.
pub async fn record_ssec_passthrough_capability(&self, arn: &str, capability: SsecPassthroughCapability) {
self.ssec_passthrough_map.write().await.insert(
arn.to_string(),
SsecPassthroughRecord {
capability,
recorded_at: Instant::now(),
},
);
}
/// Test hook: age an existing verdict so TTL expiry is observable without
/// waiting out the real window.
#[cfg(test)]
pub(crate) async fn backdate_ssec_passthrough_capability(&self, arn: &str, age: Duration) {
let backdated = Instant::now()
.checked_sub(age)
.expect("system uptime must exceed the backdate age");
if let Some(record) = self.ssec_passthrough_map.write().await.get_mut(arn) {
record.recorded_at = backdated;
}
}
pub async fn set_target(
&self,
bucket: &str,
@@ -948,15 +1026,21 @@ impl BucketTargetSys {
}
}
// Lock order: targets_map, then arn_remotes_map, then target_h_mutex.
// Lock order: targets_map, then arn_remotes_map, then target_h_mutex,
// then ssec_passthrough_map (always last; also taken standalone by the
// capability accessors).
let mut targets_map = self.targets_map.write().await;
let mut arn_remotes_map = self.arn_remotes_map.write().await;
let mut health_map = self.target_h_mutex.write().await;
// Remove existing targets
if let Some(existing_targets) = targets_map.remove(bucket) {
let mut ssec_map = self.ssec_passthrough_map.write().await;
for target in existing_targets {
arn_remotes_map.remove(&target.arn);
health_map.remove(&target.arn);
// A rebuilt/edited target may point at a different service:
// the SSE-C passthrough verdict must be re-audited from Unknown.
ssec_map.remove(&target.arn);
self.update_bandwidth_limit(bucket, &target.arn, 0);
}
}
@@ -1446,6 +1530,43 @@ fn resolve_put_api_version_id(source_version_id: &str) -> Option<&str> {
}
}
/// Resolve the S3 `versionId` for a proxied read against a remote target.
/// RustFS represents the null version internally as the nil UUID while the S3
/// API addresses it as the literal "null" (same mapping as
/// [`resolve_put_api_version_id`]); empty means "no version requested".
pub(crate) fn resolve_read_api_version_id(version_id: Option<String>) -> Option<String> {
let version_id = version_id?;
let trimmed = version_id.trim();
if trimmed.is_empty() {
None
} else if Uuid::parse_str(trimmed).is_ok_and(|uuid| uuid.is_nil()) {
Some(rustfs_filemeta::NULL_VERSION_ID.to_string())
} else {
Some(trimmed.to_string())
}
}
/// Outbound header set for a proxied read: the caller-provided passthrough
/// headers (client SSE-C key family, conditional headers) plus the anti-loop
/// `source-proxy-request` marker in both the x-rustfs- and x-minio- prefixes
/// (a MinIO target only understands the latter). Never adds
/// `source-replication-check`: that exemption channel belongs exclusively to
/// the replication worker's HEAD.
fn proxy_outbound_headers(mut extra_headers: HeaderMap) -> HeaderMap {
insert_header(&mut extra_headers, SUFFIX_SOURCE_PROXY_REQUEST, "true");
extra_headers
}
/// Copy `headers` onto an SDK request inside `customize().map_request` (runs
/// before signing, so the headers join the SigV4 canonical request).
fn apply_extra_headers(mut req: HttpRequest, headers: &HeaderMap) -> Result<HttpRequest, std::convert::Infallible> {
for (k, v) in headers.iter() {
req.headers_mut()
.insert(k.as_str().to_string(), v.to_str().unwrap_or("").to_string());
}
Ok(req)
}
/// Append `versionId=<id>` to an already-built request URI. aws-sdk-s3's
/// `PutObjectInput` / `CreateMultipartUploadInput` expose no version id
/// member, so the query is spliced in via `map_request`, which runs at
@@ -1549,8 +1670,8 @@ impl Default for PutObjectOptions {
}
}
#[allow(dead_code)]
impl PutObjectOptions {
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
fn set_match_etag(&mut self, etag: &str) {
if etag == "*" {
self.custom_header
@@ -1561,6 +1682,7 @@ impl PutObjectOptions {
}
}
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
fn set_match_etag_except(&mut self, etag: &str) {
if etag == "*" {
self.custom_header
@@ -1696,6 +1818,7 @@ impl PutObjectOptions {
header
}
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
fn validate(&self, _c: Arc<TargetClient>) -> Result<(), std::io::Error> {
//if self.checksum.is_set() {
/*if !self.trailing_header_support {
@@ -1851,6 +1974,13 @@ impl TargetClient {
// worker cannot hold; otherwise SSE-C replicas never converge on HEAD.
let mut headers = HeaderMap::new();
insert_header(&mut headers, SUFFIX_SOURCE_REPLICATION_CHECK, "true");
// `source-proxy-request: false` (MinIO `ProxyHeaderSet` semantics):
// the header's mere presence tells the receiver to answer LOCALLY
// instead of proxying the miss back to us. Without it, a not-found on
// the target gets read-proxied back to this source, echoes the source
// object with an identical ETag, and the worker concludes the object
// already converged — so it never actually replicates it.
insert_header(&mut headers, SUFFIX_SOURCE_PROXY_REQUEST, "false");
match self
.client
.head_object()
@@ -1875,6 +2005,129 @@ impl TargetClient {
}
}
/// HEAD used by the read-proxy path (GET/HEAD of an object not yet
/// replicated locally, MinIO `proxyHeadToRepTarget`).
///
/// Deliberately different from [`TargetClient::head_object`]: it must NOT
/// send `source-replication-check` — that header is the replication
/// worker's SSE-C metadata exemption channel. A proxied client request
/// instead forwards the client's own SSE-C headers (`extra_headers`) so
/// the target performs the real SSE-C validation/decryption. The
/// `source-proxy-request` marker is always added so the target does not
/// proxy the request onward (anti-loop).
pub async fn head_object_for_proxy(
&self,
bucket: &str,
object: &str,
version_id: Option<String>,
range: Option<String>,
part_number: Option<i32>,
extra_headers: HeaderMap,
) -> Result<HeadObjectOutput, SdkError<HeadObjectError>> {
let headers = proxy_outbound_headers(extra_headers);
self.client
.head_object()
.bucket(bucket)
.key(object)
.set_version_id(resolve_read_api_version_id(version_id))
.set_range(range)
.set_part_number(part_number)
.customize()
.map_request(move |req| apply_extra_headers(req, &headers))
.send()
.await
}
/// GET used by the read-proxy path (MinIO `proxyGetToReplicationTarget`).
/// Returns the streaming SDK output; callers must forward the body without
/// buffering it. Same header contract as [`Self::head_object_for_proxy`]:
/// anti-loop marker on, replication-check never sent, client SSE-C /
/// conditional headers forwarded verbatim via `extra_headers`.
pub async fn get_object(
&self,
bucket: &str,
object: &str,
version_id: Option<String>,
range: Option<String>,
part_number: Option<i32>,
extra_headers: HeaderMap,
) -> Result<GetObjectOutput, SdkError<GetObjectError>> {
let headers = proxy_outbound_headers(extra_headers);
self.client
.get_object()
.bucket(bucket)
.key(object)
.set_version_id(resolve_read_api_version_id(version_id))
.set_range(range)
.set_part_number(part_number)
.customize()
.map_request(move |req| apply_extra_headers(req, &headers))
.send()
.await
}
/// GetObjectTagging for the tagging read-proxy path
/// (MinIO `proxyGetTaggingToRepTarget`). Anti-loop marker always added.
pub async fn get_object_tagging(
&self,
bucket: &str,
object: &str,
version_id: Option<String>,
) -> Result<GetObjectTaggingOutput, SdkError<GetObjectTaggingError>> {
let headers = proxy_outbound_headers(HeaderMap::new());
self.client
.get_object_tagging()
.bucket(bucket)
.key(object)
.set_version_id(resolve_read_api_version_id(version_id))
.customize()
.map_request(move |req| apply_extra_headers(req, &headers))
.send()
.await
}
/// PutObjectTagging for the tagging proxy path
/// (MinIO `proxyTaggingToRepTarget`). Anti-loop marker always added.
pub async fn put_object_tagging(
&self,
bucket: &str,
object: &str,
version_id: Option<String>,
tagging: SdkTagging,
) -> Result<PutObjectTaggingOutput, SdkError<PutObjectTaggingError>> {
let headers = proxy_outbound_headers(HeaderMap::new());
self.client
.put_object_tagging()
.bucket(bucket)
.key(object)
.set_version_id(resolve_read_api_version_id(version_id))
.tagging(tagging)
.customize()
.map_request(move |req| apply_extra_headers(req, &headers))
.send()
.await
}
/// DeleteObjectTagging for the tagging proxy path
/// (MinIO `proxyTaggingToRepTarget`). Anti-loop marker always added.
pub async fn delete_object_tagging(
&self,
bucket: &str,
object: &str,
version_id: Option<String>,
) -> Result<DeleteObjectTaggingOutput, SdkError<DeleteObjectTaggingError>> {
let headers = proxy_outbound_headers(HeaderMap::new());
self.client
.delete_object_tagging()
.bucket(bucket)
.key(object)
.set_version_id(resolve_read_api_version_id(version_id))
.customize()
.map_request(move |req| apply_extra_headers(req, &headers))
.send()
.await
}
/// On success returns the version id the target assigned (from
/// `x-amz-version-id`), letting callers audit the version-identity
/// contract — a target that adopts the source version echoes it back.
@@ -2504,6 +2757,57 @@ mod tests {
assert_eq!(health.last_online, Some(now));
}
/// N2 TTL contract, both flip directions: a recorded verdict is fresh
/// until [`SSEC_PASSTHROUGH_CAPABILITY_TTL`], then reads as expired; a
/// re-audit that records the OPPOSITE verdict replaces it as fresh. The
/// worker gate maps expired verdicts to ProceedWithAudit (pinned in
/// `replication_target_boundary`), so together this proves an Unsupported
/// target recovers to Supported through the audit once its verdict ages
/// out — and a stale Supported one is re-proven rather than trusted.
#[tokio::test]
async fn ssec_passthrough_capability_ttl_expires_and_reaudit_flips_verdict() {
let sys = BucketTargetSys::default();
let arn = "arn:rustfs:replication:us-east-1:bucket:ssec-ttl";
let expired_age = SSEC_PASSTHROUGH_CAPABILITY_TTL + Duration::from_secs(1);
assert_eq!(
sys.ssec_passthrough_capability(arn).await,
(SsecPassthroughCapability::Unknown, false),
"an unrecorded target must read Unknown and never expired"
);
sys.record_ssec_passthrough_capability(arn, SsecPassthroughCapability::Unsupported)
.await;
assert_eq!(
sys.ssec_passthrough_capability(arn).await,
(SsecPassthroughCapability::Unsupported, false)
);
sys.backdate_ssec_passthrough_capability(arn, expired_age).await;
assert_eq!(
sys.ssec_passthrough_capability(arn).await,
(SsecPassthroughCapability::Unsupported, true),
"an aged-out Unsupported verdict must read expired so the gate re-audits"
);
// The re-audit against an upgraded target records Supported afresh.
sys.record_ssec_passthrough_capability(arn, SsecPassthroughCapability::Supported)
.await;
assert_eq!(
sys.ssec_passthrough_capability(arn).await,
(SsecPassthroughCapability::Supported, false),
"a fresh Supported verdict replaces the expired Unsupported one"
);
// And the fail-open twin: Supported also ages out.
sys.backdate_ssec_passthrough_capability(arn, expired_age).await;
assert_eq!(
sys.ssec_passthrough_capability(arn).await,
(SsecPassthroughCapability::Supported, true),
"an aged-out Supported verdict must read expired so the gate re-proves it"
);
}
#[tokio::test]
async fn list_targets_applies_health_stats_by_arn_and_preserves_endpoint_port() {
let sys = BucketTargetSys::default();
@@ -456,16 +456,23 @@ impl<'a> LifecycleExpiryTrace<'a> {
}
}
#[allow(dead_code)]
impl ExpiryStats {
pub fn missed_tasks(&self) -> i64 {
self.missed_expiry_tasks.load(Ordering::SeqCst)
}
#[allow(
dead_code,
reason = "asserted by this file's tests; the lib target cannot see test-only consumers (backlog#1823)"
)]
fn missed_free_vers_tasks(&self) -> i64 {
self.missed_freevers_tasks.load(Ordering::SeqCst)
}
#[allow(
dead_code,
reason = "asserted by this file's tests; the lib target cannot see test-only consumers (backlog#1823)"
)]
fn missed_tier_journal_tasks(&self) -> i64 {
self.missed_tier_journal_tasks.load(Ordering::SeqCst)
}
@@ -1776,7 +1783,7 @@ impl TransitionState {
.await;
}
global_metrics().record_scanner_transition_failed(1);
if !is_err_version_not_found(&err) && !is_err_object_not_found(&err) && !is_network_or_host_down(&err.to_string(), false) && !err.to_string().contains("use of closed network connection") {
if !is_err_version_not_found(&err) && !is_err_object_not_found(&err) && !is_network_or_host_down(&err.to_string(), false) {
error!(
event = EVENT_LIFECYCLE_TIER_OPERATION_FAILED,
component = LOG_COMPONENT_ECSTORE,
+1 -1
View File
@@ -19,7 +19,7 @@ pub mod core;
pub mod evaluator;
pub mod manual_transition_job;
mod metadata_boundary;
pub(crate) use metadata_boundary::get_expiry_configs;
pub(crate) use metadata_boundary::{LifecycleExpiryConfigs, get_expiry_configs};
mod object_lock_boundary;
pub use self::core as lifecycle;
mod replication_sink;
@@ -80,7 +80,10 @@ impl LastDayTierStats {
}
}
#[allow(dead_code)]
#[allow(
dead_code,
reason = "asserted by this file's tests; the lib target cannot see test-only consumers (backlog#1823)"
)]
fn merge(&self, m: LastDayTierStats) -> LastDayTierStats {
let mut cl = self.clone();
let mut cm = m;
@@ -177,9 +177,10 @@ fn should_record_remote_delete_failure(err: &std::io::Error) -> bool {
}
#[derive(Default)]
#[allow(dead_code)]
struct ObjSweeper {
#[allow(dead_code, reason = "written but never read back (backlog#1823)")]
object: String,
#[allow(dead_code, reason = "written but never read back (backlog#1823)")]
bucket: String,
version_id: Option<Uuid>,
versioned: bool,
@@ -191,9 +192,9 @@ struct ObjSweeper {
remote_object: String,
}
#[allow(dead_code)]
impl ObjSweeper {
#[allow(clippy::new_ret_no_self)]
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
pub async fn new(bucket: &str, object: &str) -> Result<Self, std::io::Error> {
Ok(Self {
object: object.into(),
@@ -202,17 +203,20 @@ impl ObjSweeper {
})
}
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
pub fn with_version(&mut self, vid: Option<Uuid>) -> &Self {
self.version_id = vid.clone();
self
}
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
pub fn with_versioning(&mut self, versioned: bool, suspended: bool) -> &Self {
self.versioned = versioned;
self.suspended = suspended;
self
}
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
pub fn get_opts(&self) -> lifecycle::ObjectOpts {
let mut opts = ObjectOpts {
version_id: self.version_id.clone(),
@@ -226,6 +230,7 @@ impl ObjSweeper {
opts
}
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
pub fn set_transition_state(&mut self, info: TransitionedObject) {
self.transition_tier = info.tier;
self.transition_status = info.status;
@@ -266,6 +271,7 @@ impl ObjSweeper {
None
}
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
pub async fn sweep(&self, api: Arc<ECStore>) {
let Some(je) = self.should_remove_remote_object() else {
return;
-2
View File
@@ -312,9 +312,7 @@ mod tests {
}
#[derive(Deserialize)]
struct LegacyBucketQuota {
#[allow(dead_code)]
quota: Option<u64>,
#[allow(dead_code)]
quota_type: LegacyQuotaType,
}
let legacy = serde_json::from_slice::<LegacyBucketQuota>(&json)
@@ -11,9 +11,9 @@ paths.
| Module | Current role | Split blocker |
|---|---|---|
| `config.rs` | Replication config helpers, rule matching, and tag filtering. | Uses replication-local filemeta/tagging boundaries and S3 DTOs directly. |
| `datatypes.rs` | ECStore compatibility re-export for resync status enums. | Re-exports `rustfs-replication` contracts while downstream facade consumers migrate. |
| `replication_object_decision_boundary.rs` | Object replication option DTOs, resync target projection, delete replication decisions, and multipart planning helpers. | Keeps ECStore runtime modules from importing object decision contracts directly from `rustfs-replication`. |
| `replication_pool.rs` | Replication queue, worker pool, MRF persistence, bucket stats, and delete/object scheduling. | Depends on bucket target sys, bucket metadata sys, metadata paths, queue contracts through the queue boundary, file metadata replication contracts through local boundaries, config storage, storage contracts through the replication storage boundary, runtime sources, and notification state. |
| `replication_proxy.rs` | Proxy-target selection for GET/HEAD/Tagging reads of objects not yet replicated locally (MinIO `getProxyTargets` parity: anti-loop, version-suspended, and no-config empty branches). | Uses replication config lookup, rule matching, and target clients through local boundaries. |
| `replication_queue_boundary.rs` | Queue/admission DTOs, heal queue DTOs, worker sizing, and backpressure helpers. | Keeps ECStore runtime modules from importing queue/backpressure contracts directly from `rustfs-replication`. |
| `replication_resync_boundary.rs` | Resync DTOs, status classifiers, persisted resync/MRF codec wrappers, and ECStore error mapping. | Keeps ECStore runtime modules from importing resync contract helpers directly from `rustfs-replication`. |
| `replication_resyncer.rs` | Object replication, delete replication, resync execution, target calls, and multipart target upload paths. | Depends on target calls and target config types through the replication target boundary, metadata paths and metadata systems through the replication metadata boundary, file metadata replication contracts through the filemeta boundary, object decisions and multipart planning through the object decision boundary, resync contracts through the resync boundary, queue DTOs through the queue boundary, error contracts through the error boundary, versioning systems, storage contracts through the replication storage boundary, config-derived storage class labels through the config store, runtime sources, notification events and local event host selection through the event sink, bandwidth reader wrapping, and SetDisks lock timing. |
@@ -117,9 +117,12 @@ Target end state:
their file names — so batch-merging them beforehand is explicitly rejected:
it forces synchronized guard-script/mod/import churn with zero functional
gain;
- the only module that can retire early is `datatypes.rs`: delete it once its
facade consumers import the resync status enums through `rustfs-replication`
directly.
- `datatypes.rs` retired early (its sanctioned exception): it was a pure
relay (`boundary -> datatypes -> mod.rs`), so the facade now re-exports
`ResyncStatusType` from the resync boundary directly and the relay file is
deleted. Note the original retirement wording ("consumers import through
`rustfs-replication` directly") conflicted with Migration Rule #15
consumers stay behind the ECStore facade; only the relay hop dissolves.
## Milestones
@@ -127,9 +130,9 @@ Target end state:
|---|---|---|
| M0 | Record the completion criteria and end state (this section). | Done |
| M1 | Contract extraction: resync/queue/stats/object-decision/filemeta/storage wire contracts owned by `crates/replication`; ECStore imports concentrated in `*_boundary.rs`; event sink and runtime access behind local contracts. | Done — see Required Contracts |
| M2 | Move resyncer pure decision logic (no IO) into `crates/replication`. | Pending; sequence after splitting the oversized resyncer/pool functions (`resync_bucket`, `replicate_all`, `start_mrf_processor`) so moves stay mechanical |
| M2 | Move resyncer pure decision logic (no IO) into `crates/replication`. | Done — moved the pure decision helpers with their unit tests: `resync_status_duration` (resync), `resync_existing_delete_replication_info` / `replicate_delete_outcome` / `target_delete_version_id` / `delete_marker_purge_version_id` / `delete_marker_purge_mrf_entry` (delete), `version_identity_drifted` / `is_replication_target_offline_error` / the SSE-C passthrough gate family incl. `SsecPassthroughCapability` (object; `ssec_passthrough_evidence_present` was param-demoted to the echoed customer-algorithm string, ECStore keeps the `HeadObjectOutput` adapter). ECStore imports them through the resync/object-decision/target boundaries; `bucket_target_sys` keeps only the verdict cache + TTL and re-exports the capability enum. Not moved (signatures carry ECStore or aws-sdk types): `verify_resync_head_result`, `resync_target_error_detail`, the `SdkError` classifiers (`has_raw_status`, `is_version_id_format_mismatch`), the `replicate_all_*` option/info builders, and `bounded_resync_max_jobs` (itself a pure clamp, but it forms one local configuration unit with the env-reading `configured_resync_max_jobs` and its ECStore-local constants — moving the clamp alone has negative value). |
| M3 | Move the worker runtime (`replication_pool.rs`, the IO paths of `replication_resyncer.rs`, `replication_state.rs`) once the contract traits are stable. Highest-risk step of the whole plan; do it last. | Pending |
| M4 | Retire the boundary modules together with their guard-script entries; delete `datatypes.rs`. | Pending |
| M4 | Retire the boundary modules together with their guard-script entries. | Pending (`datatypes.rs` already retired early alongside M2) |
The original first code-bearing step (narrow `ReplicationEventSink` /
`ReplicationRuntime` contracts) has landed — `replication_event_sink.rs`
@@ -1,15 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
pub use super::replication_resync_boundary::ResyncStatusType;
+4 -2
View File
@@ -12,7 +12,6 @@
// See the License for the specific language governing permissions and
// limitations under the License.
pub mod datatypes;
mod replication_bandwidth_boundary;
mod replication_config_boundary;
mod replication_config_store;
@@ -29,6 +28,7 @@ mod replication_object_bridge;
mod replication_object_config;
mod replication_object_decision_boundary;
pub(crate) mod replication_pool;
mod replication_proxy;
mod replication_queue_boundary;
mod replication_resync_boundary;
mod replication_resyncer;
@@ -43,7 +43,6 @@ pub(crate) mod replication_timing;
mod replication_versioning_boundary;
mod runtime_boundary;
pub use datatypes::ResyncStatusType;
pub use replication_config_boundary::{
ObjectOpts, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_UNSUPPORTED_FIELDS, REMOTE_TARGET_WRITABLE_FIELDS,
REPLICATION_CAPABILITY_CONTRACT_VERSION, REPLICATION_READ_ONLY_HISTORICAL_FIELDS, REPLICATION_WRITABLE_FIELDS,
@@ -74,13 +73,16 @@ pub use replication_pool::{
get_global_replication_pool, get_global_replication_stats, init_background_replication, persist_force_delete_intent,
read_durable_mrf_backlog, resync_start_conflict_id,
};
pub use replication_proxy::get_proxy_targets;
pub use replication_queue_boundary::{
DeletedObjectReplicationInfo, ReplicationBatchAdmission, ReplicationHealQueueResult, ReplicationOperation,
ReplicationPriority, ReplicationQueueAdmission,
};
pub use replication_resync_boundary::ResyncStatusType;
pub use replication_resync_boundary::{BucketReplicationResyncStatus, ResyncOpts, TargetReplicationResyncStatus};
pub use replication_scanner_bridge::ReplicationScannerBridge;
pub use replication_state::{ReplicationStats, RuntimeReplicationTargetBacklog};
pub use replication_stats_boundary::{BucketReplicationStat, BucketReplicationStats, BucketStats, InQueueMetric, XferStats};
pub use replication_storage_boundary::{ReplicationObjectIO, ReplicationStorage};
pub use replication_target_boundary::SsecPassthroughCapability;
pub(crate) use replication_target_config_bridge::ReplicationTargetConfigBridge;
@@ -12,12 +12,11 @@
// See the License for the specific language governing permissions and
// limitations under the License.
pub(crate) use rustfs_filemeta::NULL_VERSION_ID;
pub use rustfs_replication::{MrfOpKind, MrfReplicateEntry};
pub(crate) use rustfs_replication::{
REPLICATE_EXISTING, REPLICATE_EXISTING_DELETE, REPLICATE_HEAL_DELETE, ReplicateTargetDecision, ReplicatedInfos,
ReplicatedTargetInfo, ReplicationAction, ReplicationWorkerOperation, ResyncDecision, get_replication_state,
parse_replicate_decision, replicate_decision_for_admitted_targets, target_reset_header, version_purge_statuses_map,
REPLICATE_EXISTING, REPLICATE_HEAL_DELETE, ReplicateTargetDecision, ReplicatedInfos, ReplicatedTargetInfo, ReplicationAction,
ReplicationWorkerOperation, ResyncDecision, get_replication_state, parse_replicate_decision,
replicate_decision_for_admitted_targets, target_reset_header, version_purge_statuses_map,
};
pub use rustfs_replication::{
REPLICATE_INCOMING_DELETE, ReplicateDecision, ReplicateObjectInfo, ReplicationState, ReplicationStatusType, ReplicationType,
@@ -18,9 +18,10 @@ pub use rustfs_replication::{
should_use_existing_delete_replication_source,
};
pub(crate) use rustfs_replication::{
ReplicationDeleteSource, ReplicationMultipartPartInput, ReplicationResyncTargetObject,
delete_replication_missing_source_decision, delete_replication_object_opts, heal_uses_delete_replication_path,
is_retryable_delete_replication_head_error, is_version_delete_replication, replication_etags_match,
replication_multipart_complete_actual_size, replication_multipart_part_plan, resync_target_for_object,
should_retry_delete_marker_purge,
ReplicationDeleteSource, ReplicationMultipartPartInput, ReplicationResyncTargetObject, delete_marker_purge_mrf_entry,
delete_marker_purge_version_id, delete_replication_missing_source_decision, delete_replication_object_opts,
heal_uses_delete_replication_path, is_retryable_delete_replication_head_error, is_version_delete_replication,
replicate_delete_outcome, replication_etags_match, replication_multipart_complete_actual_size,
replication_multipart_part_plan, resync_existing_delete_replication_info, resync_target_for_object,
should_retry_delete_marker_purge, target_delete_version_id,
};
@@ -667,6 +667,368 @@ async fn acknowledge_mrf_recovery<S: ReplicationStorage>(
Err(EcstoreError::PreconditionFailed)
}
/// Acquires the MRF recovery leader lock for the startup replay.
/// Returns `None` (after logging) when the lock cannot be created or another
/// node is already processing the backlog.
async fn acquire_mrf_recovery_guard<S: ReplicationStorage>(storage: &Arc<S>) -> Option<rustfs_lock::NamespaceLockGuard> {
let recovery_lock = match storage
.new_ns_lock(
ReplicationMetadataStore::rustfs_meta_bucket(),
ReplicationMetadataStore::MRF_REPLICATION_RECOVERY_LOCK,
)
.await
{
Ok(lock) => lock,
Err(error) => {
warn!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %error,
"Failed to create the MRF recovery leader lock"
);
return None;
}
};
match recovery_lock
.get_write_lock_quiet(ReplicationLockTiming::acquire_timeout())
.await
{
Ok(guard) => Some(guard),
Err(_) => {
debug!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
"Another node is already processing the MRF recovery backlog"
);
None
}
}
}
/// Reads and decodes the on-disk MRF recovery file.
/// Returns `None` when there is nothing to replay: missing file (publishes an
/// empty available summary), read failure, or corrupt data (quarantined).
async fn load_mrf_recovery_entries<S: ReplicationStorage>(storage: &Arc<S>) -> Option<Vec<MrfReplicateEntry>> {
let data = match ReplicationConfigStore::read(storage.clone(), ReplicationMetadataStore::MRF_REPLICATION_FILE).await {
Ok(d) => d,
Err(EcstoreError::ConfigNotFound) => {
set_durable_mrf_backlog_summary(DurableMrfBacklogSummary {
available: true,
buckets: Vec::new(),
});
return None;
}
Err(e) => {
warn!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %e,
"Failed to load MRF recovery file"
);
return None;
}
};
match decode_mrf_file(&data) {
Ok(v) => Some(v),
Err(e) => {
warn!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %e,
"Failed to decode MRF recovery file — preserving corrupt data"
);
quarantine_mrf_file(storage, &data).await;
None
}
}
}
/// Replays one MRF recovery entry by operation kind.
/// Returns `None` when the entry is skipped entirely (no admission outcome);
/// entries that must be retried later are pushed onto `retry_entries`.
async fn replay_mrf_entry<S: ReplicationStorage>(
entry: &MrfReplicateEntry,
storage: &Arc<S>,
retry_entries: &mut Vec<MrfReplicateEntry>,
) -> Option<ReplicationQueueAdmission> {
match entry.op {
MrfOpKind::Delete => replay_mrf_delete_entry(entry, storage, retry_entries).await,
MrfOpKind::Object | MrfOpKind::Heal | MrfOpKind::ExistingObject => {
replay_mrf_object_entry(entry, storage, retry_entries).await
}
MrfOpKind::Metadata => replay_mrf_metadata_entry(entry, storage, retry_entries).await,
}
}
/// Replays a delete-kind MRF entry: force-delete intents replay directly,
/// stale force-delete generations are skipped, and plain deletes are
/// reconstructed as heal deletes.
async fn replay_mrf_delete_entry<S: ReplicationStorage>(
entry: &MrfReplicateEntry,
storage: &Arc<S>,
retry_entries: &mut Vec<MrfReplicateEntry>,
) -> Option<ReplicationQueueAdmission> {
if should_replay_force_delete_intent(entry) {
let operation_id = entry.force_delete_id?;
let delete = force_delete_heal_replication_info(entry, operation_id);
if replicate_delete_with_outcome(delete, storage.clone()).await {
Some(ReplicationQueueAdmission::Queued)
} else {
Some(ReplicationQueueAdmission::Missed)
}
} else if entry.force_delete_id.is_some() {
Some(ReplicationQueueAdmission::Skipped)
} else {
replay_mrf_reconstructed_delete(entry, storage, retry_entries).await
}
}
/// Pure DTO construction: heal replication info for a replayed force-delete intent.
fn force_delete_heal_replication_info(entry: &MrfReplicateEntry, operation_id: uuid::Uuid) -> DeletedObjectReplicationInfo {
DeletedObjectReplicationInfo {
delete_object: ReplicationDeletedObject {
object_name: entry.object.clone(),
force_delete: true,
force_delete_id: Some(operation_id),
force_delete_target_arns: entry.target_arns.clone(),
force_delete_generation: entry.force_delete_generation,
..Default::default()
},
bucket: entry.bucket.clone(),
op_type: ReplicationType::Heal,
event_type: REPLICATE_HEAL_DELETE.to_string(),
..Default::default()
}
}
/// Reconstruct a heal delete and re-queue it. We do NOT call
/// get_object_info here because the delete-marker or version may
/// already be absent from the local store — that is expected.
async fn replay_mrf_reconstructed_delete<S: ReplicationStorage>(
entry: &MrfReplicateEntry,
storage: &Arc<S>,
retry_entries: &mut Vec<MrfReplicateEntry>,
) -> Option<ReplicationQueueAdmission> {
let versioned = ReplicationVersioningStore::prefix_enabled(&entry.bucket, &entry.object).await;
let oi = ObjectInfo {
bucket: entry.bucket.clone(),
name: entry.object.clone(),
version_id: entry.version_id,
delete_marker: entry.delete_marker,
..Default::default()
};
let dsc = resolve_mrf_delete_replicate_decision(entry, &oi, versioned, retry_entries).await?;
let dv = reconstructed_heal_delete_info(entry, &oi, &dsc);
if replicate_delete_with_outcome(dv, storage.clone()).await {
Some(ReplicationQueueAdmission::Queued)
} else {
Some(ReplicationQueueAdmission::Missed)
}
}
/// The MRF entry does not persist the replication decision and the
/// source object is gone, so re-derive the decision from the live
/// bucket config (mirroring get_heal_replicate_object_info) and set
/// it on the reconstructed delete. Without this the decision string
/// is empty and the delete replicates to zero targets — a silent
/// no-op that leaves replicas diverged (backlog#858 / #799 B9).
async fn resolve_mrf_delete_replicate_decision(
entry: &MrfReplicateEntry,
oi: &ObjectInfo,
versioned: bool,
retry_entries: &mut Vec<MrfReplicateEntry>,
) -> Option<ReplicateDecision> {
if entry.target_arns.is_empty() {
match ReplicationMetadataStore::optional_replication_config(&entry.bucket).await {
Ok(None) => None,
Err(_) => {
retry_entries.push(entry.clone());
None
}
Ok(Some(_)) => match check_replicate_delete_strict(
&entry.bucket,
&ObjectToDelete {
object_name: entry.object.clone(),
version_id: entry.version_id,
..Default::default()
},
oi,
&ObjectOptions {
versioned,
..Default::default()
},
None,
)
.await
{
Ok(dsc) => Some(dsc),
Err(_) => {
retry_entries.push(entry.clone());
None
}
},
}
} else {
Some(replicate_decision_for_admitted_targets(&entry.target_arns))
}
}
/// Pure DTO construction: reconstructed heal delete carrying the re-derived
/// replication decision.
fn reconstructed_heal_delete_info(
entry: &MrfReplicateEntry,
oi: &ObjectInfo,
dsc: &ReplicateDecision,
) -> DeletedObjectReplicationInfo {
let mut rstate = oi.replication_state();
rstate.replicate_decision_str = dsc.to_string();
let delete_marker_mtime = entry
.delete_marker_mtime
.and_then(|nanos| OffsetDateTime::from_unix_timestamp_nanos(i128::from(nanos)).ok());
DeletedObjectReplicationInfo {
delete_object: ReplicationDeletedObject {
object_name: entry.object.clone(),
version_id: entry.version_id,
delete_marker_version_id: entry.delete_marker_version_id,
delete_marker: entry.delete_marker,
delete_marker_mtime,
force_delete: entry.force_delete,
replication_state: Some(rstate),
..Default::default()
},
bucket: entry.bucket.clone(),
op_type: ReplicationType::Heal,
event_type: REPLICATE_HEAL_DELETE.to_string(),
..Default::default()
}
}
/// Replays an Object/Heal/ExistingObject MRF entry against the live source object.
async fn replay_mrf_object_entry<S: ReplicationStorage>(
entry: &MrfReplicateEntry,
storage: &Arc<S>,
retry_entries: &mut Vec<MrfReplicateEntry>,
) -> Option<ReplicationQueueAdmission> {
let opts = ObjectOptions {
version_id: entry.version_id.map(|u| u.to_string()),
..Default::default()
};
let oi = match storage.get_object_info(&entry.bucket, &entry.object, &opts).await {
Ok(oi) => oi,
Err(e) => {
debug!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
bucket = %entry.bucket,
object = %entry.object,
error = %e,
"MRF recovery: source object lookup failed"
);
if should_retry_mrf_source_lookup(&e) {
retry_entries.push(entry.clone());
}
return None;
}
};
if entry.target_arns.is_empty() {
// Legacy entries predate target admission persistence. They cannot
// be safely attributed, so retain the old live-config fallback.
Some(queue_replication_heal(&entry.bucket, oi, entry.retry_count.max(0) as u32).await)
} else {
let roi = admitted_mrf_replicate_object(oi, entry, entry.op.replication_type());
if replicate_object_with_outcome(roi, storage.clone()).await.1 {
Some(ReplicationQueueAdmission::Queued)
} else {
Some(ReplicationQueueAdmission::Missed)
}
}
}
/// Replays a metadata-kind MRF entry against the live source object.
async fn replay_mrf_metadata_entry<S: ReplicationStorage>(
entry: &MrfReplicateEntry,
storage: &Arc<S>,
retry_entries: &mut Vec<MrfReplicateEntry>,
) -> Option<ReplicationQueueAdmission> {
let opts = ObjectOptions {
version_id: entry.version_id.map(|u| u.to_string()),
..Default::default()
};
let oi = match storage.get_object_info(&entry.bucket, &entry.object, &opts).await {
Ok(oi) => oi,
Err(e) => {
debug!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
bucket = %entry.bucket,
object = %entry.object,
error = %e,
"MRF metadata recovery: source object lookup failed"
);
if should_retry_mrf_source_lookup(&e) {
retry_entries.push(entry.clone());
}
return None;
}
};
if entry.target_arns.is_empty() {
Some(queue_replication_metadata(&entry.bucket, oi, entry.retry_count.max(0) as u32).await)
} else {
let roi = admitted_mrf_replicate_object(oi, entry, ReplicationType::Metadata);
if replicate_object_with_outcome(roi, storage.clone()).await.1 {
Some(ReplicationQueueAdmission::Queued)
} else {
Some(ReplicationQueueAdmission::Missed)
}
}
}
/// Pure DTO construction: replicate-object info for an entry with persisted
/// admitted targets, carrying over the entry's retry count.
fn admitted_mrf_replicate_object(oi: ObjectInfo, entry: &MrfReplicateEntry, op_type: ReplicationType) -> ReplicateObjectInfo {
let dsc = replicate_decision_for_admitted_targets(&entry.target_arns);
let mut roi = replicate_object_info_from_object_info(oi, dsc, op_type);
roi.retry_count = entry.retry_count.max(0) as u32;
roi
}
/// Acknowledges the replayed MRF prefix and returns the retained backlog.
/// On acknowledgement failure the backlog is preserved for the next startup and
/// re-read (falling back to the replayed snapshot) so the published summary stays accurate.
async fn resolve_retained_mrf_entries<S: ReplicationStorage>(
storage: &Arc<S>,
recovery_guard: &rustfs_lock::NamespaceLockGuard,
entries: &[MrfReplicateEntry],
retry_entries: &[MrfReplicateEntry],
) -> Vec<MrfReplicateEntry> {
match acknowledge_mrf_recovery(storage.clone(), recovery_guard, entries, retry_entries).await {
Ok(retained) => retained,
Err(error) => {
warn!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %error,
"Failed to acknowledge the MRF recovery prefix; preserving it for the next startup"
);
match read_mrf_entries(storage.clone()).await {
Ok(current) => current,
Err(read_error) => {
warn!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %read_error,
"Failed to refresh the MRF backlog after acknowledgement failure"
);
entries.to_vec()
}
}
}
}
}
#[derive(Debug, thiserror::Error)]
#[error("replication resync {active_resync_id} is already active for {bucket}/{arn}")]
struct ResyncActiveConflictError {
@@ -1221,71 +1583,12 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
let storage = self.storage.clone();
let handle = tokio::spawn(async move {
let recovery_lock = match storage
.new_ns_lock(
ReplicationMetadataStore::rustfs_meta_bucket(),
ReplicationMetadataStore::MRF_REPLICATION_RECOVERY_LOCK,
)
.await
{
Ok(lock) => lock,
Err(error) => {
warn!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %error,
"Failed to create the MRF recovery leader lock"
);
return;
}
};
let recovery_guard = match recovery_lock
.get_write_lock_quiet(ReplicationLockTiming::acquire_timeout())
.await
{
Ok(guard) => guard,
Err(_) => {
debug!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
"Another node is already processing the MRF recovery backlog"
);
return;
}
let Some(recovery_guard) = acquire_mrf_recovery_guard(&storage).await else {
return;
};
let data = match ReplicationConfigStore::read(storage.clone(), ReplicationMetadataStore::MRF_REPLICATION_FILE).await {
Ok(d) => d,
Err(EcstoreError::ConfigNotFound) => {
set_durable_mrf_backlog_summary(DurableMrfBacklogSummary {
available: true,
buckets: Vec::new(),
});
return;
}
Err(e) => {
warn!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %e,
"Failed to load MRF recovery file"
);
return;
}
};
let entries = match decode_mrf_file(&data) {
Ok(v) => v,
Err(e) => {
warn!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %e,
"Failed to decode MRF recovery file — preserving corrupt data"
);
quarantine_mrf_file(&storage, &data).await;
return;
}
let Some(entries) = load_mrf_recovery_entries(&storage).await else {
return;
};
set_durable_mrf_backlog_snapshot(durable_mrf_backlog_summary_from_entries(&entries));
@@ -1294,187 +1597,8 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
let mut retry_entries = Vec::new();
for entry in entries.iter() {
let admission = match entry.op {
MrfOpKind::Delete => {
if should_replay_force_delete_intent(entry) {
let Some(operation_id) = entry.force_delete_id else {
continue;
};
let delete = DeletedObjectReplicationInfo {
delete_object: ReplicationDeletedObject {
object_name: entry.object.clone(),
force_delete: true,
force_delete_id: Some(operation_id),
force_delete_target_arns: entry.target_arns.clone(),
force_delete_generation: entry.force_delete_generation,
..Default::default()
},
bucket: entry.bucket.clone(),
op_type: ReplicationType::Heal,
event_type: REPLICATE_HEAL_DELETE.to_string(),
..Default::default()
};
if replicate_delete_with_outcome(delete, storage.clone()).await {
ReplicationQueueAdmission::Queued
} else {
ReplicationQueueAdmission::Missed
}
} else if entry.force_delete_id.is_some() {
ReplicationQueueAdmission::Skipped
} else {
// Reconstruct a heal delete and re-queue it. We do NOT call
// get_object_info here because the delete-marker or version may
// already be absent from the local store — that is expected.
//
// The MRF entry does not persist the replication decision and the
// source object is gone, so re-derive the decision from the live
// bucket config (mirroring get_heal_replicate_object_info) and set
// it on the reconstructed delete. Without this the decision string
// is empty and the delete replicates to zero targets — a silent
// no-op that leaves replicas diverged (backlog#858 / #799 B9).
let versioned = ReplicationVersioningStore::prefix_enabled(&entry.bucket, &entry.object).await;
let oi = ObjectInfo {
bucket: entry.bucket.clone(),
name: entry.object.clone(),
version_id: entry.version_id,
delete_marker: entry.delete_marker,
..Default::default()
};
let dsc = if entry.target_arns.is_empty() {
match ReplicationMetadataStore::optional_replication_config(&entry.bucket).await {
Ok(None) => continue,
Err(_) => {
retry_entries.push(entry.clone());
continue;
}
Ok(Some(_)) => match check_replicate_delete_strict(
&entry.bucket,
&ObjectToDelete {
object_name: entry.object.clone(),
version_id: entry.version_id,
..Default::default()
},
&oi,
&ObjectOptions {
versioned,
..Default::default()
},
None,
)
.await
{
Ok(dsc) => dsc,
Err(_) => {
retry_entries.push(entry.clone());
continue;
}
},
}
} else {
replicate_decision_for_admitted_targets(&entry.target_arns)
};
let mut rstate = oi.replication_state();
rstate.replicate_decision_str = dsc.to_string();
let delete_marker_mtime = entry
.delete_marker_mtime
.and_then(|nanos| OffsetDateTime::from_unix_timestamp_nanos(i128::from(nanos)).ok());
let dv = DeletedObjectReplicationInfo {
delete_object: ReplicationDeletedObject {
object_name: entry.object.clone(),
version_id: entry.version_id,
delete_marker_version_id: entry.delete_marker_version_id,
delete_marker: entry.delete_marker,
delete_marker_mtime,
force_delete: entry.force_delete,
replication_state: Some(rstate),
..Default::default()
},
bucket: entry.bucket.clone(),
op_type: ReplicationType::Heal,
event_type: REPLICATE_HEAL_DELETE.to_string(),
..Default::default()
};
if replicate_delete_with_outcome(dv, storage.clone()).await {
ReplicationQueueAdmission::Queued
} else {
ReplicationQueueAdmission::Missed
}
}
}
MrfOpKind::Object | MrfOpKind::Heal | MrfOpKind::ExistingObject => {
let opts = ObjectOptions {
version_id: entry.version_id.map(|u| u.to_string()),
..Default::default()
};
let oi = match storage.get_object_info(&entry.bucket, &entry.object, &opts).await {
Ok(oi) => oi,
Err(e) => {
debug!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
bucket = %entry.bucket,
object = %entry.object,
error = %e,
"MRF recovery: source object lookup failed"
);
if should_retry_mrf_source_lookup(&e) {
retry_entries.push(entry.clone());
}
continue;
}
};
if entry.target_arns.is_empty() {
// Legacy entries predate target admission persistence. They cannot
// be safely attributed, so retain the old live-config fallback.
queue_replication_heal(&entry.bucket, oi, entry.retry_count.max(0) as u32).await
} else {
let dsc = replicate_decision_for_admitted_targets(&entry.target_arns);
let mut roi = replicate_object_info_from_object_info(oi, dsc, entry.op.replication_type());
roi.retry_count = entry.retry_count.max(0) as u32;
if replicate_object_with_outcome(roi, storage.clone()).await.1 {
ReplicationQueueAdmission::Queued
} else {
ReplicationQueueAdmission::Missed
}
}
}
MrfOpKind::Metadata => {
let opts = ObjectOptions {
version_id: entry.version_id.map(|u| u.to_string()),
..Default::default()
};
let oi = match storage.get_object_info(&entry.bucket, &entry.object, &opts).await {
Ok(oi) => oi,
Err(e) => {
debug!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
bucket = %entry.bucket,
object = %entry.object,
error = %e,
"MRF metadata recovery: source object lookup failed"
);
if should_retry_mrf_source_lookup(&e) {
retry_entries.push(entry.clone());
}
continue;
}
};
if entry.target_arns.is_empty() {
queue_replication_metadata(&entry.bucket, oi, entry.retry_count.max(0) as u32).await
} else {
let dsc = replicate_decision_for_admitted_targets(&entry.target_arns);
let mut roi = replicate_object_info_from_object_info(oi, dsc, ReplicationType::Metadata);
roi.retry_count = entry.retry_count.max(0) as u32;
if replicate_object_with_outcome(roi, storage.clone()).await.1 {
ReplicationQueueAdmission::Queued
} else {
ReplicationQueueAdmission::Missed
}
}
}
let Some(admission) = replay_mrf_entry(entry, &storage, &mut retry_entries).await else {
continue;
};
if admission == ReplicationQueueAdmission::Missed {
@@ -1484,29 +1608,7 @@ impl<S: ReplicationStorage> ReplicationPool<S> {
}
}
let retained = match acknowledge_mrf_recovery(storage.clone(), &recovery_guard, &entries, &retry_entries).await {
Ok(retained) => retained,
Err(error) => {
warn!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %error,
"Failed to acknowledge the MRF recovery prefix; preserving it for the next startup"
);
match read_mrf_entries(storage.clone()).await {
Ok(current) => current,
Err(read_error) => {
warn!(
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REPLICATION,
error = %read_error,
"Failed to refresh the MRF backlog after acknowledgement failure"
);
entries.clone()
}
}
}
};
let retained = resolve_retained_mrf_entries(&storage, &recovery_guard, &entries, &retry_entries).await;
let retained_count = retained.len();
set_durable_mrf_backlog_snapshot(durable_mrf_backlog_summary_from_entries(&retained));
@@ -0,0 +1,150 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Proxy-target selection for reads of objects not yet replicated locally
//! (MinIO `getProxyTargets`, bucket-replication.go).
//!
//! During the active-active replication lag window a GET/HEAD/Tagging request
//! for an object the local site does not have yet may be served by proxying to
//! a replication target. This module only *selects* the candidate targets; the
//! request-path callers perform the remote calls and response translation.
use std::sync::Arc;
use tracing::debug;
use super::replication_config_boundary::{ObjectOpts, ReplicationConfigurationExt as _};
use super::replication_object_config::get_replication_config;
use super::replication_storage_boundary::ObjectOptions;
use super::replication_target_boundary::{ReplicationTargetStore, TargetClient};
/// Returns the replication-target clients eligible to serve a proxied read of
/// `bucket/object`, in rule order. Mirrors MinIO's `getProxyTargets`:
///
/// - the `source-proxy-request` header family was present at all
/// (`opts.proxy_request` / `opts.proxy_header_set`, MinIO `ProxyRequest` /
/// `ProxyHeaderSet`) -> empty. "true" is the anti-loop marker of an
/// already-proxied client read; "false" is what a peer's replication
/// worker sends on convergence HEADs so the receiver answers locally —
/// proxying that miss back would echo the source object and fake
/// convergence, permanently skipping replication;
/// - the bucket's versioning is suspended for the object -> empty;
/// - no replication configuration / no matching rule -> empty;
/// - otherwise every distinct target ARN whose rules match the object,
/// resolved through the bucket target system, skipping targets that opted
/// out of proxying (`disable_proxy`).
pub async fn get_proxy_targets(bucket: &str, object: &str, opts: &ObjectOptions) -> Vec<Arc<TargetClient>> {
if opts.proxy_request || opts.proxy_header_set {
return Vec::new();
}
if opts.version_suspended {
return Vec::new();
}
let cfg = match get_replication_config(bucket).await {
Ok(Some(cfg)) => cfg,
Ok(None) => return Vec::new(),
Err(err) => {
debug!(bucket, object, error = %err, "read proxy: failed to load replication config; not proxying");
return Vec::new();
}
};
let arns = cfg.filter_target_arns(&ObjectOpts {
name: object.to_string(),
..Default::default()
});
let mut targets = Vec::with_capacity(arns.len());
for arn in arns {
let Some(client) = ReplicationTargetStore::remote_target_client(bucket, &arn).await else {
debug!(bucket, object, arn, "read proxy: no client for replication target ARN");
continue;
};
if client.disable_proxy {
continue;
}
targets.push(client);
}
targets
}
#[cfg(test)]
mod tests {
use super::*;
fn opts() -> ObjectOptions {
ObjectOptions::default()
}
/// Anti-loop: a request that was already proxied by a peer must never be
/// proxied onward, regardless of replication configuration.
#[tokio::test]
async fn proxy_request_yields_no_targets() {
let targets = get_proxy_targets(
"bucket",
"object",
&ObjectOptions {
proxy_request: true,
..opts()
},
)
.await;
assert!(targets.is_empty());
}
/// MinIO `ProxyHeaderSet` parity: the header family being present at all
/// disables proxying, even with the value "false" — that is what a
/// peer's replication worker sends on convergence HEADs.
#[tokio::test]
async fn proxy_header_set_yields_no_targets() {
let targets = get_proxy_targets(
"bucket",
"object",
&ObjectOptions {
proxy_header_set: true,
proxy_request: false,
..opts()
},
)
.await;
assert!(targets.is_empty());
}
/// Suspended versioning disables proxying (MinIO parity): the local null
/// version is authoritative and a remote read could resurrect data.
#[tokio::test]
async fn version_suspended_yields_no_targets() {
let targets = get_proxy_targets(
"bucket",
"object",
&ObjectOptions {
version_suspended: true,
..opts()
},
)
.await;
assert!(targets.is_empty());
}
/// A bucket without replication configuration has nothing to proxy to.
/// (No metadata system is running in unit tests, so the config lookup
/// resolves to "no configuration" — the same empty-result contract.)
#[tokio::test]
async fn missing_replication_config_yields_no_targets() {
let targets = get_proxy_targets("bucket-without-replication", "object", &opts()).await;
assert!(targets.is_empty());
}
}
@@ -15,10 +15,15 @@
use super::replication_error_boundary::{Error, Result};
use super::replication_filemeta_boundary::MrfReplicateEntry;
/// Kept test-only: the runtime consumer was the worker HEAD's fake proxy
/// counting (removed in backlog#1675 P1-5); the resyncer tests still pin the
/// classifier's semantics for the real client read-proxy failure accounting.
#[cfg(test)]
pub(crate) use rustfs_replication::should_count_head_proxy_failure;
pub use rustfs_replication::{BucketReplicationResyncStatus, ResyncOpts, ResyncStatusType, TargetReplicationResyncStatus};
pub(crate) use rustfs_replication::{
is_version_id_mismatch, resync_state_accepts_update, sanitize_resync_error_detail, should_auto_resume_resync,
should_count_head_proxy_failure,
is_version_id_mismatch, resync_state_accepts_update, resync_status_duration, sanitize_resync_error_detail,
should_auto_resume_resync,
};
#[allow(
File diff suppressed because it is too large Load Diff
@@ -1161,6 +1161,31 @@ mod tests {
assert!(all.contains_key("proxy-only-bucket"));
}
/// Pins the read-proxy metric contract (backlog#1675 P1-5): the API
/// strings the GET/HEAD/Tagging proxy paths record map onto the
/// get/head/tagging totals, and only unexpected failures raise the
/// failed counters.
#[tokio::test]
async fn test_proxy_stats_map_read_proxy_apis_to_totals() {
let stats = ReplicationStats::new();
stats.inc_proxy("proxy-bucket", "GetObject", false).await;
stats.inc_proxy("proxy-bucket", "GetObject", true).await;
stats.inc_proxy("proxy-bucket", "HeadObject", false).await;
stats.inc_proxy("proxy-bucket", "GetObjectTagging", false).await;
stats.inc_proxy("proxy-bucket", "PutObjectTagging", false).await;
stats.inc_proxy("proxy-bucket", "DeleteObjectTagging", true).await;
let metric = stats.get_proxy_stats("proxy-bucket").await;
assert_eq!(metric.get_total, 2);
assert_eq!(metric.get_failed, 1);
assert_eq!(metric.head_total, 1);
assert_eq!(metric.head_failed, 0);
assert_eq!(metric.get_tag_total, 1);
assert_eq!(metric.put_tag_total, 1);
assert_eq!(metric.delete_tag_total, 1);
assert_eq!(metric.delete_tag_failed, 1);
}
#[tokio::test]
async fn test_calculate_bucket_replication_stats_merges_resync_metrics() {
let stats = ReplicationStats::new();
@@ -36,11 +36,15 @@ use time::OffsetDateTime;
use time::format_description::well_known::Rfc3339;
pub(crate) use crate::bucket::bucket_target_sys::{
AdvancedPutOptions, PutObjectOptions, PutObjectPartOptions, RemoveObjectOptions, TargetClient,
AdvancedPutOptions, PutObjectOptions, PutObjectPartOptions, RemoveObjectOptions, TargetClient, resolve_read_api_version_id,
};
#[cfg(test)]
pub(crate) use crate::bucket::target::BucketTarget;
pub(crate) use crate::bucket::target::BucketTargets;
pub use rustfs_replication::SsecPassthroughCapability;
pub(crate) use rustfs_replication::{
SsecPassthroughGate, is_replication_target_offline_error, ssec_passthrough_gate, version_identity_drifted,
};
use super::replication_config_store::ReplicationConfigStore;
use super::replication_error_boundary::{Error, Result};
@@ -65,6 +69,8 @@ static STANDARD_HEADERS: &[&str] = &[
];
const ERR_REPLICATION_ENCRYPTION_METADATA_UNSUPPORTED: &str = "replication source contains unsupported encryption metadata";
pub(crate) const ERR_REPLICATION_SSEC_PASSTHROUGH_UNSUPPORTED: &str = "replication target does not support SSE-C passthrough: the replica would lose its decryption material \
(run ?replication-check to re-probe)";
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum ReplicationSourceEncryption {
@@ -146,6 +152,13 @@ pub(crate) fn replication_object_is_ssec_encrypted(user_defined: &HashMap<String
rustfs_replication::is_ssec_encrypted(user_defined)
}
/// HeadObjectOutput adapter over the pure SSE-C passthrough evidence
/// judgment owned by `rustfs-replication`: extract the echoed
/// customer-algorithm header and let the crate-owned policy decide.
pub(crate) fn ssec_passthrough_evidence_present(head: &HeadObjectOutput) -> bool {
rustfs_replication::ssec_passthrough_evidence_present(head.sse_customer_algorithm.as_deref())
}
pub(crate) struct ReplicationTargetStore;
impl ReplicationTargetStore {
@@ -165,6 +178,17 @@ impl ReplicationTargetStore {
BucketTargetSys::get().mark_target_offline(target_client).await
}
/// Returns the cached verdict and whether it has outlived its TTL.
pub(crate) async fn ssec_passthrough_capability(arn: &str) -> (SsecPassthroughCapability, bool) {
BucketTargetSys::get().ssec_passthrough_capability(arn).await
}
pub(crate) async fn record_ssec_passthrough_capability(arn: &str, capability: SsecPassthroughCapability) {
BucketTargetSys::get()
.record_ssec_passthrough_capability(arn, capability)
.await
}
#[cfg(test)]
pub(crate) async fn register_test_target(target_client: &Arc<TargetClient>) {
BucketTargetSys::get().arn_remotes_map.write().await.insert(
@@ -898,6 +922,27 @@ mod tests {
}
}
/// Pins the HeadObjectOutput field extraction feeding the crate-owned
/// evidence judgment (the gate/evidence policy matrix itself is pinned in
/// `rustfs-replication`'s object tests).
#[test]
fn ssec_passthrough_evidence_requires_customer_algorithm_echo() {
let with_evidence = HeadObjectOutput::builder().sse_customer_algorithm("AES256").build();
assert!(ssec_passthrough_evidence_present(&with_evidence));
let empty_algorithm = HeadObjectOutput::builder().sse_customer_algorithm("").build();
assert!(
!ssec_passthrough_evidence_present(&empty_algorithm),
"an empty echo is not evidence of preserved SSE-C material"
);
let without_evidence = HeadObjectOutput::builder().e_tag("\"abc\"").content_length(8).build();
assert!(
!ssec_passthrough_evidence_present(&without_evidence),
"a plain HEAD response must classify the target as having dropped the material"
);
}
#[test]
fn replication_put_options_adds_ssec_checksum_metadata() {
let metadata = HashMap::from([(SSEC_ALGORITHM_HEADER.to_string(), "AES256".to_string())]);
+2 -2
View File
@@ -95,7 +95,6 @@ impl TransitionClient {
}
#[derive(Default)]
#[allow(dead_code)]
pub struct GetRequest {
pub buffer: Vec<u8>,
pub offset: i64,
@@ -107,11 +106,12 @@ pub struct GetRequest {
pub setting_object_info: bool,
}
#[allow(dead_code)]
pub struct GetResponse {
pub size: i64,
//pub error: error,
#[allow(dead_code, reason = "written but never read back (backlog#1823)")]
pub did_read: bool,
#[allow(dead_code, reason = "written but never read back (backlog#1823)")]
pub object_info: ObjectInfo,
}
+2 -2
View File
@@ -20,6 +20,7 @@
#![allow(clippy::all)]
use http::{HeaderMap, HeaderName, HeaderValue};
use rustfs_utils::http::headers::AMZ_CHECKSUM_MODE;
use std::collections::HashMap;
use time::OffsetDateTime;
use tracing::warn;
@@ -27,7 +28,6 @@ use tracing::warn;
use crate::client::api_error_response::err_invalid_argument;
#[derive(Default)]
#[allow(dead_code)]
pub struct AdvancedGetOptions {
pub replication_delete_marker: bool,
pub is_replication_ready_for_delete_marker: bool,
@@ -77,7 +77,7 @@ impl GetObjectOptions {
}
}
if self.checksum {
headers.insert(HeaderName::from_static("x-amz-checksum-mode"), HeaderValue::from_static("ENABLED"));
headers.insert(HeaderName::from_static(AMZ_CHECKSUM_MODE), HeaderValue::from_static("ENABLED"));
}
headers
}
-1
View File
@@ -360,7 +360,6 @@ impl TransitionClient {
}
#[derive(Default)]
#[allow(dead_code)]
pub struct ListObjectsOptions {
reverse_versions: bool,
with_versions: bool,
+3 -1
View File
@@ -137,8 +137,8 @@ impl Default for PutObjectOptions {
}
}
#[allow(dead_code)]
impl PutObjectOptions {
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
fn set_match_etag(&mut self, etag: &str) {
if etag == "*" {
self.custom_header.insert("If-Match", HeaderValue::from_static("*"));
@@ -149,6 +149,7 @@ impl PutObjectOptions {
}
}
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
fn set_match_etag_except(&mut self, etag: &str) {
if etag == "*" {
self.custom_header.insert("If-None-Match", HeaderValue::from_static("*"));
@@ -259,6 +260,7 @@ impl PutObjectOptions {
header
}
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
fn validate(&self, c: TransitionClient) -> Result<(), std::io::Error> {
//if self.checksum.is_set() {
/*if !self.trailing_header_support {
+2 -3
View File
@@ -55,7 +55,6 @@ pub struct RemoveBucketOptions {
const DELETE_RESPONSE_PREVIEW_LEN: usize = 1024;
#[derive(Debug)]
#[allow(dead_code)]
pub struct AdvancedRemoveOptions {
pub replication_delete_marker: bool,
pub replication_status: ReplicationStatus,
@@ -465,10 +464,10 @@ impl TransitionClient {
}
#[derive(Debug, Default)]
#[allow(dead_code)]
pub struct RemoveObjectError {
#[allow(dead_code, reason = "written but never read back (backlog#1823)")]
object_name: String,
#[allow(dead_code)]
#[allow(dead_code, reason = "written but never read back (backlog#1823)")]
version_id: String,
err: Option<std::io::Error>,
}
+3 -3
View File
@@ -372,8 +372,8 @@ pub struct Checksum {
computed: bool,
}
#[allow(dead_code)]
impl Checksum {
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
fn new(t: ChecksumMode, b: &[u8]) -> Checksum {
if t.is_set() && b.len() == t.raw_byte_len() {
return Checksum {
@@ -385,7 +385,7 @@ impl Checksum {
Checksum::default()
}
#[allow(dead_code)]
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
fn new_checksum_string(t: ChecksumMode, s: &str) -> Result<Checksum, std::io::Error> {
let b = match base64_decode(s.as_bytes()) {
Ok(b) => b,
@@ -412,7 +412,7 @@ impl Checksum {
base64_encode(&self.r)
}
#[allow(dead_code)]
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
fn raw(&self) -> Option<Vec<u8>> {
if !self.is_set() {
return None;
@@ -37,16 +37,17 @@ pub struct PutObjReader {
//pub sealMD5Fn: SealMD5CurrFn,
}
#[allow(dead_code)]
impl PutObjReader {
pub fn new(reader: HashReader) -> Self {
Self { reader }
}
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
fn md5_current_hex_string(&self) -> String {
self.reader.checksum().map(|v| v.encoded).unwrap_or_default()
}
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
fn with_encryption(&mut self, enc_reader: HashReader) -> Result<(), std::io::Error> {
self.reader = enc_reader;
+10 -6
View File
@@ -54,6 +54,10 @@ use rustfs_config::MAX_S3_CLIENT_RESPONSE_SIZE;
use rustfs_rio::HashReader;
use rustfs_utils::HashAlgorithm;
use rustfs_utils::{
http::headers::{
AMZ_CHECKSUM_CRC32, AMZ_CHECKSUM_CRC32C, AMZ_CHECKSUM_CRC64NVME, AMZ_CHECKSUM_MODE, AMZ_CHECKSUM_SHA1,
AMZ_CHECKSUM_SHA256,
},
net::get_endpoint_url,
retry::{DEFAULT_RETRY_CAP, DEFAULT_RETRY_UNIT, MAX_JITTER, MAX_RETRY, RetryTimer},
};
@@ -1383,12 +1387,12 @@ pub(crate) fn to_object_info_for_provider(
};
// Extract checksums
let checksum_crc32 = get_header("x-amz-checksum-crc32");
let checksum_crc32c = get_header("x-amz-checksum-crc32c");
let checksum_sha1 = get_header("x-amz-checksum-sha1");
let checksum_sha256 = get_header("x-amz-checksum-sha256");
let checksum_crc64nvme = get_header("x-amz-checksum-crc64nvme");
let checksum_mode = get_header("x-amz-checksum-mode");
let checksum_crc32 = get_header(AMZ_CHECKSUM_CRC32);
let checksum_crc32c = get_header(AMZ_CHECKSUM_CRC32C);
let checksum_sha1 = get_header(AMZ_CHECKSUM_SHA1);
let checksum_sha256 = get_header(AMZ_CHECKSUM_SHA256);
let checksum_crc64nvme = get_header(AMZ_CHECKSUM_CRC64NVME);
let checksum_mode = get_header(AMZ_CHECKSUM_MODE);
// Build and return the ObjectInfo struct
Ok(ObjectInfo {
@@ -233,11 +233,17 @@ pub struct NsScannerCapabilityRequest {
#[async_trait]
pub trait InternodeDataTransport: Send + Sync + std::fmt::Debug {
async fn open_read(&self, request: ReadStreamRequest) -> Result<FileReader>;
async fn open_read_fresh(&self, request: ReadStreamRequest) -> Result<FileReader> {
self.open_read(request).await
}
/// Opens an owned-chunk stream when this transport can retain receive-buffer
/// ownership. `None` preserves the established `open_read` fallback.
async fn open_read_chunks(&self, _request: ReadStreamRequest) -> Result<Option<ChunkReaderBox>> {
Ok(None)
}
async fn open_read_chunks_fresh(&self, request: ReadStreamRequest) -> Result<Option<ChunkReaderBox>> {
self.open_read_chunks(request).await
}
async fn open_write(&self, request: WriteStreamRequest) -> Result<FileWriter>;
async fn open_walk_dir(&self, request: WalkDirStreamRequest) -> Result<FileReader>;
async fn open_ns_scanner(&self, _request: NsScannerStreamRequest) -> Result<FileReader> {
@@ -269,6 +275,15 @@ impl InternodeDataTransport for TcpHttpInternodeDataTransport {
))
}
async fn open_read_fresh(&self, request: ReadStreamRequest) -> Result<FileReader> {
let url = build_read_file_stream_url(&request);
let mut headers = json_headers();
build_auth_headers(&url, &Method::GET, &mut headers)?;
Ok(Box::new(
HttpReader::new_fresh_connection_with_stall_timeout(url, Method::GET, headers, None, request.stall_timeout).await?,
))
}
async fn open_read_chunks(&self, request: ReadStreamRequest) -> Result<Option<ChunkReaderBox>> {
let url = build_read_file_stream_url(&request);
let mut headers = json_headers();
@@ -278,6 +293,16 @@ impl InternodeDataTransport for TcpHttpInternodeDataTransport {
)))
}
async fn open_read_chunks_fresh(&self, request: ReadStreamRequest) -> Result<Option<ChunkReaderBox>> {
let url = build_read_file_stream_url(&request);
let mut headers = json_headers();
build_auth_headers(&url, &Method::GET, &mut headers)?;
Ok(Some(Box::new(
HttpChunkReader::new_fresh_connection_with_stall_timeout(url, Method::GET, headers, None, request.stall_timeout)
.await?,
)))
}
async fn open_write(&self, request: WriteStreamRequest) -> Result<FileWriter> {
let server_epoch = self.put_file_auth_capability(&request.endpoint).await?;
let nonce = server_epoch.map(|_| Uuid::new_v4());
@@ -86,6 +86,25 @@ const PEER_REST_RECOVERY_MAX_BACKOFF: Duration = Duration::from_secs(30);
const SCANNER_ACTIVITY_MAX_MESSAGE_SIZE: usize = 1024;
const REPLICATION_STATS_MAX_MESSAGE_SIZE: usize = 8 * 1024 * 1024;
/// Error for a peer that reported `success = false` without an `error_info` payload.
///
/// Same shape as `peer_s3_client::peer_failure_without_details`, over `StorageError`
/// instead of `DiskError`. The message names the operation (and the bucket, where the
/// operation has one) and nothing else, for two reasons:
///
/// - `finalize_result` classifies failures by message substring, so any text matching
/// `message_has_network_needle` would take an answering peer offline and evict its
/// connection over a plain application-level rejection.
/// - Quorum aggregation (`reduce_errs`) buckets `Io` errors by kind plus rendered
/// message, so a per-peer detail such as the peer address would split one shared
/// failure into single-count buckets and downgrade the dominant error.
fn peer_failure_without_details(op: &str, bucket: Option<&str>) -> Error {
match bucket {
Some(bucket) => Error::other(format!("{op}({bucket}): peer returned failure without error details")),
None => Error::other(format!("{op}: peer returned failure without error details")),
}
}
fn decode_bucket_stats_response(response: GetBucketStatsDataResponse) -> Result<BucketStats> {
if !response.success {
return Err(Error::other(
@@ -696,7 +715,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("local_storage_info", None));
}
let data = response.storage_info;
@@ -719,7 +738,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("server_info", None));
}
let data = response.server_properties;
@@ -742,7 +761,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("get_cpus", None));
}
let data = response.cpus;
@@ -765,7 +784,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("get_net_info", None));
}
let data = response.net_info;
@@ -788,7 +807,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("get_partitions", None));
}
let data = response.partitions;
@@ -811,7 +830,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("get_os_info", None));
}
let data = response.os_info;
@@ -832,7 +851,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("get_se_linux_info", None));
}
let data = response.sys_services;
@@ -857,7 +876,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("get_sys_config", None));
}
let data = response.sys_config;
@@ -882,7 +901,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("get_sys_errors", None));
}
let data = response.sys_errors;
@@ -907,7 +926,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("get_mem_info", None));
}
let data = response.mem_info;
@@ -939,7 +958,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("get_metrics", None));
}
let data = response.realtime_metrics;
@@ -964,7 +983,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("get_live_events", None));
}
Ok(PeerLiveEventsBatch {
@@ -989,7 +1008,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("get_proc_info", None));
}
let data = response.proc_info;
@@ -1016,7 +1035,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("start_profiling", None));
}
Ok(())
}
@@ -1323,7 +1342,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("load_bucket_metadata", Some(bucket)));
}
Ok(())
}
@@ -1346,7 +1365,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("delete_bucket_metadata", Some(bucket)));
}
Ok(())
}
@@ -1369,7 +1388,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("delete_policy", None));
}
Ok(())
}
@@ -1392,7 +1411,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("load_policy", None));
}
Ok(())
}
@@ -1417,7 +1436,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("load_policy_mapping", None));
}
Ok(())
}
@@ -1440,7 +1459,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("delete_user", None));
}
Ok(())
}
@@ -1463,7 +1482,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("delete_service_account", None));
}
Ok(())
}
@@ -1487,7 +1506,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("load_user", None));
}
Ok(())
}
@@ -1510,7 +1529,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("load_service_account", None));
}
Ok(())
}
@@ -1533,7 +1552,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("load_group", None));
}
Ok(())
}
@@ -1554,7 +1573,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("reload_site_replication_config", None));
}
Ok(())
}
@@ -1597,7 +1616,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("signal_service", None));
}
validate_signal_service_protocol(sig, sub_sys, response.protocol_version)?;
Ok(response)
@@ -1667,7 +1686,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("reload_pool_meta", None));
}
Ok(())
@@ -1691,7 +1710,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("stop_rebalance", None));
}
Ok(())
@@ -1725,7 +1744,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("load_rebalance_meta", None));
}
Ok(())
@@ -1753,7 +1772,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("start_decommission", None));
}
Ok(())
@@ -1777,7 +1796,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("decommission_cancel", None));
}
Ok(())
@@ -1801,7 +1820,7 @@ impl PeerRestClient {
if let Some(msg) = response.error_info {
return Err(Error::other(msg));
}
return Err(Error::other(""));
return Err(peer_failure_without_details("clear_decommission", None));
}
Ok(())
@@ -1947,6 +1966,8 @@ fn tier_config_reload_status_outcome(status: tonic::Status) -> TierConfigReloadO
mod tests {
use super::*;
use crate::config::com::STORAGE_CLASS_SUB_SYS;
use crate::disk::error::DiskError;
use crate::disk::error_reduce::reduce_errs;
use crate::layout::{disks_layout::DisksLayout, endpoints::SetupType};
use rustfs_config::{ENV_KUBERNETES_SERVICE_HOST, ENV_LOCAL_ENDPOINT_HOST, ENV_STARTUP_TOPOLOGY_WAIT_MODE};
use serde_json::Value;
@@ -3098,4 +3119,115 @@ mod tests {
&& span.get("request_id").and_then(Value::as_str) == Some("req-peer-rest")
}));
}
/// Every operation name passed to `peer_failure_without_details` in this file.
const PEER_FAILURE_OPS: &[&str] = &[
"local_storage_info",
"server_info",
"get_cpus",
"get_net_info",
"get_partitions",
"get_os_info",
"get_se_linux_info",
"get_sys_config",
"get_sys_errors",
"get_mem_info",
"get_metrics",
"get_live_events",
"get_proc_info",
"start_profiling",
"load_bucket_metadata",
"delete_bucket_metadata",
"delete_policy",
"load_policy",
"load_policy_mapping",
"delete_user",
"delete_service_account",
"load_user",
"load_service_account",
"load_group",
"reload_site_replication_config",
"signal_service",
"reload_pool_meta",
"stop_rebalance",
"load_rebalance_meta",
"start_decommission",
"decommission_cancel",
"clear_decommission",
];
#[test]
fn peer_failure_without_details_names_operation_and_bucket() {
for op in PEER_FAILURE_OPS {
let message = peer_failure_without_details(op, None).to_string();
assert!(message.contains(op), "{op} message must name the operation: {message}");
}
for op in ["load_bucket_metadata", "delete_bucket_metadata"] {
let message = peer_failure_without_details(op, Some("ops-bucket")).to_string();
assert!(message.contains(op), "{op} message must name the operation: {message}");
assert!(message.contains("ops-bucket"), "{op} message must name the bucket: {message}");
}
}
#[test]
fn peer_failure_without_details_keeps_one_reduce_errs_bucket_per_operation() {
// reduce_errs groups Io errors by kind plus rendered message: peers failing the
// same operation must stay a single dominant error instead of one bucket per peer.
let per_peer_errs = (0..4)
.map(|_| Some(DiskError::from(peer_failure_without_details("load_bucket_metadata", Some("shared")))))
.collect::<Vec<_>>();
let (count, dominant) = reduce_errs(&per_peer_errs, &[]);
assert_eq!(count, 4, "one shared failure must not split into per-peer buckets");
assert_eq!(
dominant,
Some(DiskError::from(peer_failure_without_details("load_bucket_metadata", Some("shared"))))
);
assert_ne!(
peer_failure_without_details("load_bucket_metadata", Some("shared")).to_string(),
peer_failure_without_details("delete_bucket_metadata", Some("shared")).to_string()
);
assert_ne!(
peer_failure_without_details("load_bucket_metadata", Some("bucket-a")).to_string(),
peer_failure_without_details("load_bucket_metadata", Some("bucket-b")).to_string()
);
}
#[test]
fn peer_failure_without_details_never_reads_as_a_network_failure() {
// `finalize_result` marks the peer offline and evicts its connection whenever the
// message matches a network needle. A peer that answered `success = false` is alive,
// so no operation or bucket name may push this text over that classifier.
for op in PEER_FAILURE_OPS {
let err = peer_failure_without_details(op, None);
assert!(
!PeerRestClient::is_network_like_error(&err),
"{op} must not read as a transport failure: {err}"
);
let scoped = peer_failure_without_details(op, Some("bucket-name"));
assert!(
!PeerRestClient::is_network_like_error(&scoped),
"{op} must not read as a transport failure: {scoped}"
);
}
// The bucket name is caller-supplied. Every needle carries a space, which S3 bucket
// names cannot, and the name is closed by `)` before the literal text resumes, so no
// needle can straddle the boundary either.
for bucket in [
"timed-out",
"connection-reset",
"transport-error",
"broken-pipe",
"unavailable-logs",
] {
let err = peer_failure_without_details("load_bucket_metadata", Some(bucket));
assert!(
!PeerRestClient::is_network_like_error(&err),
"bucket {bucket} must not push the message over the network classifier: {err}"
);
}
}
}
@@ -214,6 +214,21 @@ fn pool_write_quorum(participant_count: usize) -> usize {
(participant_count / 2) + 1
}
/// Error for a peer that reported `success = false` without an error payload.
///
/// The message must stay identical across the peers of one operation: `reduce_errs`
/// buckets `Error::Io` by kind plus rendered message, so any per-peer detail (address,
/// timing) would split one shared failure into single-count buckets and downgrade a real
/// dominant error into `ErasureWriteQuorum`.
///
/// `peer_rest_client` carries the same helper over `StorageError` for the same response shape.
fn peer_failure_without_details(op: &str, bucket: Option<&str>) -> Error {
match bucket {
Some(bucket) => Error::other(format!("{op}({bucket}): peer returned failure without error details")),
None => Error::other(format!("{op}: peer returned failure without error details")),
}
}
fn reduce_pool_write_quorum_errs(per_pool_errs: &[Option<Error>]) -> Option<Error> {
if per_pool_errs.is_empty() {
return Some(Error::ErasureWriteQuorum);
@@ -1078,7 +1093,7 @@ impl PeerS3Client for RemotePeerS3Client {
return if let Some(err) = response.error {
Err(err.into())
} else {
Err(Error::other(""))
Err(peer_failure_without_details("heal_bucket", Some(bucket)))
};
}
@@ -1105,7 +1120,7 @@ impl PeerS3Client for RemotePeerS3Client {
return if let Some(err) = response.error {
Err(err.into())
} else {
Err(Error::other(""))
Err(peer_failure_without_details("list_bucket", None))
};
}
let bucket_infos = response
@@ -1136,9 +1151,7 @@ impl PeerS3Client for RemotePeerS3Client {
return if let Some(err) = response.error {
Err(err.into())
} else {
Err(Error::other(format!(
"make_bucket({bucket}): peer returned failure without error details"
)))
Err(peer_failure_without_details("make_bucket", Some(bucket)))
};
}
@@ -1162,7 +1175,7 @@ impl PeerS3Client for RemotePeerS3Client {
return if let Some(err) = response.error {
Err(err.into())
} else {
Err(Error::other(""))
Err(peer_failure_without_details("get_bucket_info", Some(bucket)))
};
}
let bucket_info = serde_json::from_str::<BucketInfo>(&response.bucket_info)?;
@@ -1190,7 +1203,7 @@ impl PeerS3Client for RemotePeerS3Client {
return if let Some(err) = response.error {
Err(err.into())
} else {
Err(Error::other(""))
Err(peer_failure_without_details("delete_bucket", Some(bucket)))
};
}
@@ -2314,4 +2327,37 @@ mod tests {
.collect::<Vec<_>>();
assert_eq!(calls, vec![1, 1, 0, 0, 0, 0, 0, 0]);
}
#[test]
fn peer_failure_without_details_names_operation_and_bucket() {
for op in ["heal_bucket", "make_bucket", "get_bucket_info", "delete_bucket"] {
let message = peer_failure_without_details(op, Some("ops-bucket")).to_string();
assert!(message.contains(op), "{op} message must name the operation: {message}");
assert!(message.contains("ops-bucket"), "{op} message must name the bucket: {message}");
}
let message = peer_failure_without_details("list_bucket", None).to_string();
assert!(message.contains("list_bucket"), "cluster-wide message must name the operation");
assert!(!message.trim().is_empty());
}
#[test]
fn peer_failure_without_details_keeps_one_reduce_errs_bucket_per_operation() {
// reduce_errs groups Io errors by kind plus rendered message: peers failing the
// same operation on the same bucket must still reach quorum as one dominant error.
let per_pool_errs = vec![
Some(peer_failure_without_details("delete_bucket", Some("shared"))),
Some(peer_failure_without_details("delete_bucket", Some("shared"))),
Some(peer_failure_without_details("delete_bucket", Some("shared"))),
];
assert_eq!(
reduce_pool_write_quorum_errs(&per_pool_errs),
Some(peer_failure_without_details("delete_bucket", Some("shared")))
);
assert_ne!(
peer_failure_without_details("delete_bucket", Some("shared")),
peer_failure_without_details("get_bucket_info", Some("shared"))
);
}
}
File diff suppressed because it is too large Load Diff
-3
View File
@@ -39,7 +39,6 @@ use rustfs_config::{
};
use std::sync::LazyLock;
#[allow(dead_code)]
#[allow(clippy::declare_interior_mutable_const)]
/// Default KVS for audit webhook settings.
pub static DEFAULT_AUDIT_WEBHOOK_KVS: LazyLock<KVS> = LazyLock::new(|| {
@@ -117,7 +116,6 @@ pub static DEFAULT_AUDIT_WEBHOOK_KVS: LazyLock<KVS> = LazyLock::new(|| {
])
});
#[allow(dead_code)]
#[allow(clippy::declare_interior_mutable_const)]
/// Default KVS for audit MQTT settings.
pub static DEFAULT_AUDIT_MQTT_KVS: LazyLock<KVS> = LazyLock::new(|| {
@@ -375,7 +373,6 @@ pub static DEFAULT_AUDIT_NATS_KVS: LazyLock<KVS> = LazyLock::new(|| {
])
});
#[allow(dead_code)]
pub static DEFAULT_AUDIT_PULSAR_KVS: LazyLock<KVS> = LazyLock::new(|| {
KVS(vec![
KV {
-59
View File
@@ -12,12 +12,9 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::error::{Error, Result};
use rustfs_config::server_config::{KV, KVS};
use rustfs_config::{DEFAULT_HEAL_BITROT_CYCLE_SECS, HEAL_BITROT_CYCLE};
use rustfs_utils::string::parse_bool;
use std::sync::LazyLock;
use std::time::Duration;
pub static DEFAULT_KVS: LazyLock<KVS> = LazyLock::new(|| {
KVS(vec![KV {
@@ -26,59 +23,3 @@ pub static DEFAULT_KVS: LazyLock<KVS> = LazyLock::new(|| {
hidden_if_empty: false,
}])
});
#[derive(Debug, Default)]
pub struct Config {
pub bitrot: String,
pub sleep: Duration,
pub io_count: usize,
pub drive_workers: usize,
pub cache: Duration,
}
impl Config {
pub fn bitrot_scan_cycle(&self) -> Duration {
self.cache
}
pub fn get_workers(&self) -> usize {
self.drive_workers
}
pub fn update(&mut self, nopts: &Config) {
self.bitrot = nopts.bitrot.clone();
self.io_count = nopts.io_count;
self.sleep = nopts.sleep;
self.drive_workers = nopts.drive_workers;
}
}
const RUSTFS_BITROT_CYCLE_IN_MONTHS: u64 = 1;
fn parse_bitrot_config(s: &str) -> Result<Duration> {
match parse_bool(s) {
Ok(enabled) => {
if enabled {
Ok(Duration::from_secs_f64(0.0))
} else {
Ok(Duration::from_secs_f64(-1.0))
}
}
Err(_) => {
if !s.ends_with("m") {
return Err(Error::other("unknown format"));
}
match s.trim_end_matches('m').parse::<u64>() {
Ok(months) => {
if months < RUSTFS_BITROT_CYCLE_IN_MONTHS {
return Err(Error::other(format!("minimum bitrot cycle is {RUSTFS_BITROT_CYCLE_IN_MONTHS} month(s)")));
}
Ok(Duration::from_secs(months * 30 * 24 * 60))
}
Err(err) => Err(Error::other(err)),
}
}
}
}
-1
View File
@@ -16,7 +16,6 @@
mod audit;
pub mod com;
#[allow(dead_code)]
pub mod heal;
mod notify;
mod oidc;
+100 -3
View File
@@ -16,6 +16,7 @@ use crate::bucket::replication::replication_state_from_filemeta;
use crate::bucket::versioning_sys::BucketVersioningSys;
use crate::bucket::{
lifecycle::{
LifecycleExpiryConfigs,
bucket_lifecycle_audit::LcEventSrc,
bucket_lifecycle_ops::{
LifecycleOps, apply_expiry_on_transitioned_object, apply_expiry_rule_in, eval_action_from_lifecycle,
@@ -1996,11 +1997,11 @@ impl PoolMeta {
Ok(false)
}
#[allow(dead_code)]
pub fn validate(&self, pools: Vec<Arc<Sets>>) -> Result<bool> {
struct PoolInfo {
position: usize,
completed: bool,
#[allow(dead_code, reason = "written but never read back (backlog#1823)")]
decom_started: bool,
}
@@ -2335,6 +2336,10 @@ fn lifecycle_action_removes_data_movement_version(action: IlmAction) -> bool {
)
}
fn lifecycle_action_skips_heal_version(action: IlmAction) -> bool {
action.delete()
}
fn resolve_data_movement_lifecycle_expiry_result(action: IlmAction, apply_actions: bool, applied: bool) -> Result<bool> {
if !apply_actions || applied {
return Ok(true);
@@ -2385,7 +2390,80 @@ pub(crate) async fn should_skip_lifecycle_for_data_movement(
}
}
pub struct HealLifecycleExpiryContext {
configs: LifecycleExpiryConfigs,
}
impl ECStore {
pub async fn load_heal_lifecycle_expiry_context(&self, bucket: &str) -> Result<Option<HealLifecycleExpiryContext>> {
if bucket == RUSTFS_META_BUCKET {
return Ok(None);
}
let configs = get_expiry_configs(self, bucket).await?;
if configs.lifecycle.is_none() {
return Ok(None);
}
Ok(Some(HealLifecycleExpiryContext { configs }))
}
pub async fn enqueue_heal_lifecycle_expiry(
self: &Arc<Self>,
context: &HealLifecycleExpiryContext,
bucket: &str,
object: &str,
version_id: Option<&str>,
object_info: Option<&crate::object_api::ObjectInfo>,
) -> Result<bool> {
let Some(lifecycle_config) = context.configs.lifecycle.as_ref() else {
return Ok(false);
};
let object_info = if let Some(object_info) = object_info {
if object_info.bucket != bucket || object_info.name != object {
return Ok(false);
}
let snapshot_version_id = object_info
.version_id
.filter(|version_id| !version_id.is_nil())
.map(|version_id| version_id.to_string());
if snapshot_version_id.as_deref() != version_id {
return Ok(false);
}
object_info.clone()
} else {
match self
.get_object_info(
bucket,
object,
&ObjectOptions {
version_id: version_id.map(str::to_string),
versioned: version_id.is_some(),
expected_bucket_incarnation_id: Some(context.configs.bucket_incarnation_id),
..Default::default()
},
)
.await
{
Ok(object_info) => object_info,
Err(err) if is_err_object_not_found(&err) || is_err_version_not_found(&err) => return Ok(false),
Err(err) => return Err(err),
}
};
let event = eval_action_from_lifecycle(lifecycle_config, context.configs.object_lock.as_deref(), &object_info).await;
if !lifecycle_action_skips_heal_version(event.action) {
return Ok(false);
}
if lifecycle_delete_all_versions_blocked_by_replication(self.clone(), bucket, &object_info.name, event.action).await? {
return Ok(false);
}
Ok(apply_expiry_rule_in(self.clone(), &event, &LcEventSrc::Scanner, &object_info).await)
}
async fn save_current_pool_meta(&self) -> Result<()> {
let _save_guard = self.pool_meta_save_gate.lock().await;
let snapshot = {
@@ -4287,6 +4365,19 @@ mod tests {
));
}
#[test]
fn lifecycle_action_skips_heal_version_for_every_delete_action() {
assert!(lifecycle_action_skips_heal_version(IlmAction::DeleteAction));
assert!(lifecycle_action_skips_heal_version(IlmAction::DeleteVersionAction));
assert!(lifecycle_action_skips_heal_version(IlmAction::DeleteRestoredAction));
assert!(lifecycle_action_skips_heal_version(IlmAction::DeleteRestoredVersionAction));
assert!(lifecycle_action_skips_heal_version(IlmAction::DeleteAllVersionsAction));
assert!(lifecycle_action_skips_heal_version(IlmAction::DelMarkerDeleteAllVersionsAction));
assert!(!lifecycle_action_skips_heal_version(IlmAction::TransitionAction));
assert!(!lifecycle_action_skips_heal_version(IlmAction::TransitionVersionAction));
assert!(!lifecycle_action_skips_heal_version(IlmAction::NoneAction));
}
#[test]
fn resolve_data_movement_lifecycle_expiry_result_allows_dry_run_skip() {
let skip = resolve_data_movement_lifecycle_expiry_result(IlmAction::DeleteVersionAction, false, false)
@@ -4958,13 +5049,19 @@ fn is_disk_online_state(state: &str) -> bool {
}
#[deprecated(since = "0.1.0", note = "Use fallback_total_capacity_dedup instead")]
#[allow(dead_code)]
#[allow(
dead_code,
reason = "superseded by the replacement named in the comment at pools.rs:5071 (backlog#1823)"
)]
fn fallback_total_capacity(disks: &[rustfs_madmin::Disk]) -> usize {
fallback_total_capacity_dedup(disks)
}
#[deprecated(since = "0.1.0", note = "Use fallback_free_capacity_dedup instead")]
#[allow(dead_code)]
#[allow(
dead_code,
reason = "superseded by the replacement named in the comment at pools.rs:5071 (backlog#1823)"
)]
fn fallback_free_capacity(disks: &[rustfs_madmin::Disk]) -> usize {
fallback_free_capacity_dedup(disks)
}
+20 -9
View File
@@ -1140,11 +1140,11 @@ impl crate::storage_api_contracts::heal::HealOperations for Sets {
Err(Error::DiskNotFound)
}
#[tracing::instrument(skip(self))]
async fn check_abandoned_parts(&self, _bucket: &str, _object: &str, _opts: &HealOpts) -> Result<()> {
// Multipart orphan reconciliation is intentionally retained above the pool/set layers
// until there is a concrete caller and a stable lower-level contract to implement.
Err(StorageError::NotImplemented)
#[tracing::instrument(level = "debug", skip(self, opts), fields(bucket = %bucket, object = %object, dry_run = opts.dry_run))]
async fn check_abandoned_parts(&self, bucket: &str, object: &str, opts: &HealOpts) -> Result<()> {
self.get_disks_for_heal_object(object, opts)?
.check_abandoned_parts(bucket, object, opts)
.await
}
}
@@ -1996,7 +1996,7 @@ mod tests {
}
#[tokio::test]
async fn sets_check_abandoned_parts_returns_typed_not_implemented_error() {
async fn sets_check_abandoned_parts_rejects_invalid_set_scope() {
let format = FormatV3::new(1, 1);
let sets = Sets {
id: format.id,
@@ -2021,10 +2021,21 @@ mod tests {
};
let err = sets
.check_abandoned_parts("bucket", "object", &HealOpts::default())
.check_abandoned_parts(
"bucket",
"object",
&HealOpts {
set: Some(1),
..Default::default()
},
)
.await
.expect_err("abandoned-parts ownership should stay above the pool/set storage layers");
assert!(matches!(err, StorageError::NotImplemented));
.expect_err("out-of-range abandoned-parts set scope must fail closed");
assert!(
matches!(err, StorageError::InvalidArgument(_, ref field, ref reason)
if field == "set" && reason.contains("invalid heal set index 1")),
"unexpected invalid set error: {err:?}"
);
}
// Builds a single-set `Sets` over `SET_DRIVE_COUNT` local temp-dir disks,
+104 -26
View File
@@ -418,6 +418,17 @@ pub struct DiskHealthTracker {
pub last_capacity_free: AtomicU64,
/// Last successful capacity probe timestamp
pub last_capacity_probe_unix_secs: AtomicI64,
/// Authoritative atomically published runtime/status pair.
state_snapshot: AtomicU64,
transition_lock: std::sync::Mutex<()>,
}
fn pack_health_state(runtime_state: RuntimeDriveHealthState, status: u32) -> u64 {
(u64::from(runtime_state as u32) << 32) | u64::from(status)
}
fn unpack_health_state(snapshot: u64) -> (RuntimeDriveHealthState, u32) {
(RuntimeDriveHealthState::from_u32((snapshot >> 32) as u32), snapshot as u32)
}
#[derive(Debug)]
@@ -739,6 +750,8 @@ impl DiskHealthTracker {
last_capacity_used: AtomicU64::new(0),
last_capacity_free: AtomicU64::new(0),
last_capacity_probe_unix_secs: AtomicI64::new(0),
state_snapshot: AtomicU64::new(pack_health_state(RuntimeDriveHealthState::Online, DISK_HEALTH_OK)),
transition_lock: std::sync::Mutex::new(()),
}
}
@@ -775,39 +788,52 @@ impl DiskHealthTracker {
/// Check if disk is faulty
pub fn is_faulty(&self) -> bool {
self.status.load(Ordering::Acquire) == DISK_HEALTH_FAULTY
unpack_health_state(self.state_snapshot.load(Ordering::Acquire)).1 == DISK_HEALTH_FAULTY
}
fn publish_state(&self, runtime_state: RuntimeDriveHealthState, status: u32) {
self.state_snapshot
.store(pack_health_state(runtime_state, status), Ordering::Release);
self.runtime_state.store(runtime_state as u32, Ordering::Release);
self.status.store(status, Ordering::Release);
}
/// Set disk as faulty
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
pub fn set_faulty(&self) {
self.status.store(DISK_HEALTH_FAULTY, Ordering::Release);
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
self.publish_state(RuntimeDriveHealthState::Offline, DISK_HEALTH_FAULTY);
}
/// Set disk as OK
pub fn set_ok(&self) {
self.status.store(DISK_HEALTH_OK, Ordering::Release);
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
self.publish_state(RuntimeDriveHealthState::Online, DISK_HEALTH_OK);
}
#[cfg(test)]
pub fn force_runtime_state_for_test(&self, state: RuntimeDriveHealthState) {
self.runtime_state.store(state as u32, Ordering::Release);
match state {
RuntimeDriveHealthState::Offline => self.set_faulty(),
RuntimeDriveHealthState::Online | RuntimeDriveHealthState::Suspect | RuntimeDriveHealthState::Returning => {
self.set_ok();
}
}
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let status = if state == RuntimeDriveHealthState::Offline {
DISK_HEALTH_FAULTY
} else {
DISK_HEALTH_OK
};
self.publish_state(state, status);
}
pub fn swap_ok_to_faulty(&self) -> bool {
self.status
.compare_exchange(DISK_HEALTH_OK, DISK_HEALTH_FAULTY, Ordering::AcqRel, Ordering::Relaxed)
.is_ok()
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let (_, status) = unpack_health_state(self.state_snapshot.load(Ordering::Acquire));
if status != DISK_HEALTH_OK {
return false;
}
self.publish_state(RuntimeDriveHealthState::Offline, DISK_HEALTH_FAULTY);
true
}
pub fn runtime_state(&self) -> RuntimeDriveHealthState {
RuntimeDriveHealthState::from_u32(self.runtime_state.load(Ordering::Acquire))
unpack_health_state(self.state_snapshot.load(Ordering::Acquire)).0
}
pub fn offline_duration(&self) -> Option<Duration> {
@@ -823,6 +849,7 @@ impl DiskHealthTracker {
}
pub fn mark_failure(&self, endpoint: &Endpoint, reason: &'static str) -> bool {
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let current = self.runtime_state();
let now = current_unix_secs();
let next = match current {
@@ -851,24 +878,19 @@ impl DiskHealthTracker {
};
let became_offline = next == RuntimeDriveHealthState::Offline && current != RuntimeDriveHealthState::Offline;
if next == RuntimeDriveHealthState::Offline {
self.status.store(DISK_HEALTH_FAULTY, Ordering::Release);
} else {
self.status.store(DISK_HEALTH_OK, Ordering::Release);
}
self.transition_state(endpoint, current, next, reason);
became_offline
}
#[allow(dead_code, reason = "asserted by this file's tests (backlog#1823)")]
pub fn mark_offline(&self, endpoint: &Endpoint, reason: &'static str) -> bool {
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let current = self.runtime_state();
if current == RuntimeDriveHealthState::Offline {
return false;
}
self.consecutive_successes.store(0, Ordering::Release);
self.status.store(DISK_HEALTH_FAULTY, Ordering::Release);
self.transition_state(endpoint, current, RuntimeDriveHealthState::Offline, reason);
true
}
@@ -882,11 +904,10 @@ impl DiskHealthTracker {
}
fn reset_for_store_init_retry_at(&self, endpoint: &Endpoint, now: Duration) {
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let now_nanos = unix_nanos(now);
let now_secs = unix_secs_i64(now);
self.status.store(DISK_HEALTH_OK, Ordering::Release);
self.runtime_state
.store(RuntimeDriveHealthState::Online as u32, Ordering::Release);
self.publish_state(RuntimeDriveHealthState::Online, DISK_HEALTH_OK);
self.consecutive_failures.store(0, Ordering::Release);
self.consecutive_successes.store(0, Ordering::Release);
self.offline_since_unix_secs.store(0, Ordering::Release);
@@ -898,6 +919,7 @@ impl DiskHealthTracker {
}
pub fn mark_recovery_success(&self, endpoint: &Endpoint, reason: &'static str) -> bool {
let _guard = self.transition_lock.lock().unwrap_or_else(|poisoned| poisoned.into_inner());
let current = self.runtime_state();
let next = match current {
RuntimeDriveHealthState::Online => RuntimeDriveHealthState::Online,
@@ -918,7 +940,6 @@ impl DiskHealthTracker {
let became_online = next == RuntimeDriveHealthState::Online;
if became_online {
self.status.store(DISK_HEALTH_OK, Ordering::Release);
self.consecutive_failures.store(0, Ordering::Release);
self.consecutive_successes.store(0, Ordering::Release);
}
@@ -948,7 +969,13 @@ impl DiskHealthTracker {
return;
}
self.runtime_state.store(next as u32, Ordering::Release);
let current_status = unpack_health_state(self.state_snapshot.load(Ordering::Acquire)).1;
let status = match next {
RuntimeDriveHealthState::Offline => DISK_HEALTH_FAULTY,
RuntimeDriveHealthState::Returning => current_status,
RuntimeDriveHealthState::Online | RuntimeDriveHealthState::Suspect => DISK_HEALTH_OK,
};
self.publish_state(next, status);
self.last_transition_unix_secs
.store(current_unix_secs() as i64, Ordering::Release);
@@ -1217,7 +1244,7 @@ impl LocalDiskWrapper {
return;
}
if health.status.load(Ordering::Relaxed) != DISK_HEALTH_OK {
if health.is_faulty() {
continue;
}
@@ -2909,6 +2936,57 @@ mod tests {
});
}
#[test]
#[serial_test::serial]
fn concurrent_failure_and_recovery_publish_one_health_snapshot() {
temp_env::with_var(rustfs_config::ENV_DRIVE_SUSPECT_FAILURE_THRESHOLD, Some("2"), || {
let endpoint = Endpoint::try_from("/tmp/concurrent-health-snapshot").expect("endpoint should parse");
let health = Arc::new(DiskHealthTracker::new());
let transition_guard = health
.transition_lock
.lock()
.expect("health transition lock should not be poisoned");
let start = Arc::new(std::sync::Barrier::new(3));
let (completed_tx, completed_rx) = std::sync::mpsc::channel();
let workers = (0..2)
.map(|_| {
let health = Arc::clone(&health);
let endpoint = endpoint.clone();
let start = Arc::clone(&start);
let completed_tx = completed_tx.clone();
std::thread::spawn(move || {
start.wait();
health.mark_failure(&endpoint, "concurrent_test");
completed_tx.send(()).expect("completion receiver should remain available");
})
})
.collect::<Vec<_>>();
start.wait();
assert!(
matches!(
completed_rx.recv_timeout(Duration::from_millis(250)),
Err(std::sync::mpsc::RecvTimeoutError::Timeout)
),
"concurrent transitions must wait for the serialization lock"
);
drop(transition_guard);
completed_rx
.recv_timeout(Duration::from_secs(1))
.expect("first failure transition should complete after lock release");
completed_rx
.recv_timeout(Duration::from_secs(1))
.expect("second failure transition should complete after lock release");
for worker in workers {
worker.join().expect("health transition worker should not panic");
}
assert_eq!(health.runtime_state(), RuntimeDriveHealthState::Offline);
assert!(health.is_faulty());
assert_eq!(health.consecutive_failures.load(Ordering::Acquire), 2);
});
}
#[test]
fn operation_success_recovers_suspect_drive_without_faulting() {
let endpoint = Endpoint::try_from("/tmp/runtime-state-suspect-success").expect("endpoint should parse");
+493 -25
View File
@@ -5618,11 +5618,37 @@ impl LocalDisk {
}
fn io_get_object_path(&self, bucket: &str, key: &str) -> Result<PathBuf> {
local_disk_object_path(self.io_root(), bucket, key)
self.local_disk_object_path(self.io_root(), bucket, key)
}
fn io_get_bucket_path(&self, bucket: &str) -> Result<PathBuf> {
local_disk_bucket_path(self.io_root(), bucket)
self.local_disk_bucket_path(self.io_root(), bucket)
}
fn local_disk_object_path(&self, root: &Path, bucket: &str, key: &str) -> Result<PathBuf> {
let (bucket_path, path) = build_local_disk_object_path(root, bucket, key);
#[cfg(target_os = "linux")]
{
check_local_disk_valid_object_path_at(root, &self.mount_lease, &bucket_path, &path)?;
}
#[cfg(not(target_os = "linux"))]
{
check_local_disk_valid_object_path(root, &bucket_path, &path)?;
}
Ok(path)
}
fn local_disk_bucket_path(&self, root: &Path, bucket: &str) -> Result<PathBuf> {
let bucket_path = build_local_disk_bucket_path(root, bucket);
#[cfg(target_os = "linux")]
{
check_local_disk_valid_path_at(root, &self.mount_lease, &bucket_path)?;
}
#[cfg(not(target_os = "linux"))]
{
check_local_disk_valid_path(root, &bucket_path)?;
}
Ok(bucket_path)
}
// Check if a path is valid
@@ -5631,7 +5657,14 @@ impl LocalDisk {
reason = "method wrapper over the live free function check_local_disk_valid_path; no caller in this port (backlog#1823)"
)]
fn check_valid_path<P: AsRef<Path>>(&self, path: P) -> Result<()> {
check_local_disk_valid_path(self.io_root(), path)
#[cfg(target_os = "linux")]
{
check_local_disk_valid_path_at(self.io_root(), &self.mount_lease, path)
}
#[cfg(not(target_os = "linux"))]
{
check_local_disk_valid_path(self.io_root(), path)
}
}
#[allow(
@@ -5639,7 +5672,14 @@ impl LocalDisk {
reason = "method wrapper over the live free function reject_local_disk_symlink_components; no caller in this port (backlog#1823)"
)]
fn reject_symlink_components(&self, path: &Path) -> Result<()> {
reject_local_disk_symlink_components(self.io_root(), path)
#[cfg(target_os = "linux")]
{
reject_local_disk_symlink_components_at(self.io_root(), &self.mount_lease, path)
}
#[cfg(not(target_os = "linux"))]
{
reject_local_disk_symlink_components(self.io_root(), path)
}
}
// Batch path generation with single lock acquisition
@@ -6562,7 +6602,7 @@ impl LocalDisk {
Ok(f)
}
#[allow(dead_code)]
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
fn get_metrics(&self) -> DiskMetrics {
DiskMetrics::default()
}
@@ -7323,29 +7363,54 @@ fn skip_access_checks(p: impl AsRef<str>) -> bool {
}
fn local_disk_object_path(root: &Path, bucket: &str, key: &str) -> Result<PathBuf> {
let (bucket_path, path) = build_local_disk_object_path(root, bucket, key);
check_local_disk_valid_object_path(root, &bucket_path, &path)?;
Ok(path)
}
fn build_local_disk_object_path(root: &Path, bucket: &str, key: &str) -> (PathBuf, PathBuf) {
let cache_key = if key.is_empty() {
bucket.to_string()
} else {
path_join_buf(&[bucket, key])
};
#[cfg(windows)]
let bucket_path = root.join(bucket.replace('/', "\\"));
#[cfg(not(windows))]
let bucket_path = root.join(bucket);
#[cfg(windows)]
let path = root.join(cache_key.replace('/', "\\"));
#[cfg(not(windows))]
let path = root.join(cache_key);
check_local_disk_valid_path(root, &path)?;
Ok(path)
(bucket_path, path)
}
fn local_disk_bucket_path(root: &Path, bucket: &str) -> Result<PathBuf> {
let bucket_path = build_local_disk_bucket_path(root, bucket);
check_local_disk_valid_path(root, &bucket_path)?;
Ok(bucket_path)
}
fn build_local_disk_bucket_path(root: &Path, bucket: &str) -> PathBuf {
#[cfg(windows)]
let bucket_path = root.join(bucket.replace('/', "\\"));
#[cfg(not(windows))]
let bucket_path = root.join(bucket);
check_local_disk_valid_path(root, &bucket_path)?;
Ok(bucket_path)
bucket_path
}
fn check_local_disk_valid_object_path(root: &Path, bucket_path: &Path, path: &Path) -> Result<()> {
let bucket_path = normalize_path_components(bucket_path);
let path = normalize_path_components(path);
if !bucket_path.starts_with(root) || !path.starts_with(&bucket_path) {
return Err(DiskError::InvalidPath);
}
reject_local_disk_symlink_components(root, &path)
}
fn check_local_disk_valid_path(root: &Path, path: impl AsRef<Path>) -> Result<()> {
@@ -7357,6 +7422,80 @@ fn check_local_disk_valid_path(root: &Path, path: impl AsRef<Path>) -> Result<()
reject_local_disk_symlink_components(root, &path)
}
#[cfg(target_os = "linux")]
fn check_local_disk_valid_object_path_at(root: &Path, root_fd: &std::fs::File, bucket_path: &Path, path: &Path) -> Result<()> {
let bucket_path = normalize_path_components(bucket_path);
let path = normalize_path_components(path);
if !bucket_path.starts_with(root) || !path.starts_with(&bucket_path) {
return Err(DiskError::InvalidPath);
}
reject_local_disk_symlink_components_at(root, root_fd, &path)
}
#[cfg(target_os = "linux")]
fn check_local_disk_valid_path_at(root: &Path, root_fd: &std::fs::File, path: impl AsRef<Path>) -> Result<()> {
let path = normalize_path_components(path);
if !path.starts_with(root) {
return Err(DiskError::InvalidPath);
}
reject_local_disk_symlink_components_at(root, root_fd, &path)
}
#[cfg(target_os = "linux")]
fn reject_local_disk_symlink_components_at(root: &Path, root_fd: &std::fs::File, path: &Path) -> Result<()> {
let relative = path.strip_prefix(root).map_err(|_| DiskError::InvalidPath)?;
match validate_existing_local_disk_prefix_at(root_fd, relative) {
Ok(()) => Ok(()),
Err(LocalDiskPathValidationAtError::Unsupported) => reject_local_disk_symlink_components(root, path),
Err(LocalDiskPathValidationAtError::InvalidPath) => Err(DiskError::InvalidPath),
Err(LocalDiskPathValidationAtError::Io(err)) => Err(to_file_error(err).into()),
}
}
#[cfg(target_os = "linux")]
enum LocalDiskPathValidationAtError {
Unsupported,
InvalidPath,
Io(std::io::Error),
}
#[cfg(target_os = "linux")]
fn validate_existing_local_disk_prefix_at(
root_fd: &std::fs::File,
relative: &Path,
) -> core::result::Result<(), LocalDiskPathValidationAtError> {
use rustix::fs::{Mode, OFlags, ResolveFlags, openat2};
use rustix::io::Errno;
if relative.as_os_str().is_empty() {
return Ok(());
}
let mut candidate = relative.to_path_buf();
loop {
match openat2(
root_fd,
&candidate,
OFlags::PATH | OFlags::CLOEXEC,
Mode::empty(),
ResolveFlags::BENEATH | ResolveFlags::NO_SYMLINKS,
) {
Ok(_) => return Ok(()),
Err(Errno::NOSYS) => return Err(LocalDiskPathValidationAtError::Unsupported),
Err(Errno::LOOP | Errno::XDEV) => return Err(LocalDiskPathValidationAtError::InvalidPath),
Err(Errno::NOENT) => {
let Some(parent) = candidate.parent().filter(|parent| !parent.as_os_str().is_empty()) else {
return Ok(());
};
candidate = parent.to_path_buf();
}
Err(err) => return Err(LocalDiskPathValidationAtError::Io(err.into())),
}
}
}
fn reject_local_disk_symlink_components(root: &Path, path: &Path) -> Result<()> {
let relative = path.strip_prefix(root).map_err(|_| DiskError::InvalidPath)?;
let mut current = root.to_path_buf();
@@ -9264,17 +9403,27 @@ impl DiskAPI for LocalDisk {
// accept that window (documented in docs/operations/durability-modes.md).
if durability.syncs_commit_metadata()
&& let Some(parent) = dst_file_path.parent()
&& let Err(err) = os::fsync_dir(parent).await
{
rollback_committed_rename_std(&dst_file_path, committed_new_data_path, rollback_data_dir)
.map_err(to_file_error)?;
// The commit rename changed the dst part inodes before this fsync
// failed and rolled them back; drop any fd cached during that
// window so readers re-open the restored inode (rustfs/backlog#1177).
for part_path in &invalidate_part_paths {
self.io_backend.invalidate_cached_fd(dst_volume, part_path).await;
let fsync_started = rustfs_io_metrics::put_stage_timer();
if let Err(err) = os::fsync_dst_dir_group_commit(parent).await {
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DST_DIR_FSYNC,
fsync_started,
);
rollback_committed_rename_std(&dst_file_path, committed_new_data_path, rollback_data_dir)
.map_err(to_file_error)?;
// The commit rename changed the dst part inodes before this fsync
// failed and rolled them back; drop any fd cached during that
// window so readers re-open the restored inode (rustfs/backlog#1177).
for part_path in &invalidate_part_paths {
self.io_backend.invalidate_cached_fd(dst_volume, part_path).await;
}
return Err(to_file_error(err).into());
}
return Err(to_file_error(err).into());
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DST_DIR_FSYNC,
fsync_started,
);
}
// First PUT of an object creates its directory (and any missing prefix
@@ -9293,7 +9442,12 @@ impl DiskAPI for LocalDisk {
if !dir.starts_with(&dst_volume_dir) {
break;
}
let fsync_started = rustfs_io_metrics::put_stage_timer();
if let Err(err) = os::fsync_dir(dir).await {
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_ANCESTOR_DIR_FSYNC,
fsync_started,
);
rollback_committed_rename_std(&dst_file_path, committed_new_data_path, rollback_data_dir)
.map_err(to_file_error)?;
// Same post-commit rollback window as above — drop cached
@@ -9304,6 +9458,10 @@ impl DiskAPI for LocalDisk {
}
return Err(to_file_error(err).into());
}
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_ANCESTOR_DIR_FSYNC,
fsync_started,
);
if dir == dst_volume_dir.as_path() {
break;
}
@@ -9532,10 +9690,21 @@ impl DiskAPI for LocalDisk {
}
if let Some(admission) = file_sync_admission.as_ref()
&& let Some(backup_parent) = backup_path.parent()
&& let Err(err) =
os::fsync_dir_with_namespace_file_sync_limit(backup_parent, mutation_lease.clone(), admission).await
{
return Err(DiskError::from(to_file_error(err)));
let fsync_started = rustfs_io_metrics::put_stage_timer();
if let Err(err) =
os::fsync_dir_with_namespace_file_sync_limit(backup_parent, mutation_lease.clone(), admission).await
{
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_BACKUP_DIR_FSYNC,
fsync_started,
);
return Err(DiskError::from(to_file_error(err)));
}
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_BACKUP_DIR_FSYNC,
fsync_started,
);
}
local_rollback_path = None;
}
@@ -9573,11 +9742,22 @@ impl DiskAPI for LocalDisk {
// Persist the commit rename's directory entry across power loss.
if let Some(admission) = file_sync_admission.as_ref()
&& let Some(dst_parent) = dst_file_path.parent()
&& let Err(err) =
os::fsync_dir_with_namespace_file_sync_limit(dst_parent, mutation_lease.clone(), admission).await
{
rollback_inline_metadata_commit_std(&dst_file_path, rollback_data_dir, local_rollback_path.as_deref())?;
return Err(err);
let fsync_started = rustfs_io_metrics::put_stage_timer();
if let Err(err) =
os::fsync_dir_with_namespace_file_sync_limit(dst_parent, mutation_lease.clone(), admission).await
{
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DST_DIR_FSYNC,
fsync_started,
);
rollback_inline_metadata_commit_std(&dst_file_path, rollback_data_dir, local_rollback_path.as_deref())?;
return Err(err);
}
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DST_DIR_FSYNC,
fsync_started,
);
}
// Same power-loss gap as the non-inline path (rustfs/backlog#922
@@ -9595,9 +9775,14 @@ impl DiskAPI for LocalDisk {
if !ancestor_dir.starts_with(&dst_volume_dir) {
break;
}
let fsync_started = rustfs_io_metrics::put_stage_timer();
if let Err(err) =
os::fsync_dir_with_namespace_file_sync_limit(ancestor_dir, mutation_lease.clone(), admission).await
{
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_ANCESTOR_DIR_FSYNC,
fsync_started,
);
rollback_inline_metadata_commit_std(
&dst_file_path,
rollback_data_dir,
@@ -9605,6 +9790,10 @@ impl DiskAPI for LocalDisk {
)?;
return Err(err);
}
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_ANCESTOR_DIR_FSYNC,
fsync_started,
);
if ancestor_dir == dst_volume_dir.as_path() {
break;
}
@@ -12833,6 +13022,76 @@ mod test {
);
}
#[tokio::test]
#[serial_test::serial(dst_dir_fsync_group_commit)]
async fn rename_data_non_inline_uses_dst_dir_fsync_group_commit_when_enabled() {
let _group_commit = os::set_dst_dir_fsync_group_commit_for_test(true);
let bucket = "grouped-dst-fsync-bucket";
let object = "dir/object";
let (disk, _dir) = commit_new_object(DurabilityMode::Strict, bucket, object).await;
let dst_meta_parent = disk
.get_object_path(bucket, &format!("{object}/{STORAGE_FORMAT_FILE}"))
.expect("dst meta path should resolve")
.parent()
.expect("dst meta should have a parent")
.to_path_buf();
assert_eq!(
os::fsync_dir_recorder::grouped_batch_sizes(&dst_meta_parent),
vec![1],
"enabled non-inline rename_data must route the dst parent fsync through the group commit coordinator"
);
}
#[tokio::test]
#[serial_test::serial(dst_dir_fsync_group_commit)]
async fn rename_data_non_inline_dst_dir_fsync_group_commit_failure_rolls_back_fresh_put() {
use tempfile::tempdir;
let _group_commit = os::set_dst_dir_fsync_group_commit_for_test(true);
let _mode = durability_mode_override::set(DurabilityMode::Strict);
let dir = tempdir().expect("temp dir should be created");
let endpoint = Endpoint::try_from(dir.path().to_str().expect("temp dir should be utf8")).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
let bucket = "grouped-dst-fsync-failure-bucket";
let object = "dir/object";
let tmp_object = "tmp-grouped-dst-fsync-failure";
let version_id = Uuid::parse_str("99999999-9999-9999-9999-999999999999").expect("version id should parse");
let new_data_dir = Uuid::parse_str("aaaaaaaa-aaaa-aaaa-aaaa-aaaaaaaaaaaa").expect("data dir should parse");
ensure_test_volume(&disk, bucket).await;
ensure_test_volume(&disk, RUSTFS_META_TMP_BUCKET).await;
let tmp_data_dir = dir
.path()
.join(RUSTFS_META_TMP_BUCKET)
.join(tmp_object)
.join(new_data_dir.to_string());
fs::create_dir_all(&tmp_data_dir)
.await
.expect("new tmp data dir should be created");
fs::write(tmp_data_dir.join("part.1"), b"new-data")
.await
.expect("new tmp data should be written");
let dst_meta_parent = dir.path().join(bucket).join(object);
os::fsync_dir_recorder::set_grouped_failure(&dst_meta_parent, io::ErrorKind::PermissionDenied);
let new_fi = test_file_info(object, version_id, Some(new_data_dir), None);
let err = disk
.rename_data(RUSTFS_META_TMP_BUCKET, tmp_object, new_fi, bucket, object)
.await
.expect_err("grouped dst dir fsync failure must fail the fresh PUT");
assert_eq!(err, DiskError::FileAccessDenied);
assert!(
!dst_meta_parent.join(STORAGE_FORMAT_FILE).exists(),
"fresh PUT rollback must remove the committed xl.meta after grouped dst dir fsync failure"
);
assert!(
!dst_meta_parent.join(new_data_dir.to_string()).exists(),
"fresh PUT rollback must remove the committed data dir after grouped dst dir fsync failure"
);
}
// Seed a first PUT of `object` (no prior version) through the non-inline
// rename_data path and return (disk, tempdir). The object dir and any prefix
// dirs are created during the commit.
@@ -18018,6 +18277,22 @@ mod test {
assert!(matches!(disk.get_bucket_path("escape-bucket"), Err(DiskError::InvalidPath)));
}
#[cfg(unix)]
#[tokio::test]
async fn get_bucket_path_for_io_rejects_symlink_escape() {
use std::os::unix::fs::symlink;
let root_dir = tempfile::tempdir().expect("temp dir should be created");
let outside_dir = tempfile::tempdir().expect("outside temp dir should be created");
let link_path = root_dir.path().join("escape-bucket");
symlink(outside_dir.path(), &link_path).expect("bucket symlink should be created");
let endpoint = Endpoint::try_from(root_dir.path().to_string_lossy().as_ref()).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
assert!(matches!(disk.get_bucket_path_for_io("escape-bucket"), Err(DiskError::InvalidPath)));
}
#[cfg(unix)]
#[tokio::test]
async fn test_get_object_path_rejects_symlink_component_escape() {
@@ -18037,6 +18312,199 @@ mod test {
assert!(matches!(disk.get_object_path("bucket", "escape/object.txt"), Err(DiskError::InvalidPath)));
}
#[cfg(unix)]
#[tokio::test]
async fn get_object_path_for_io_rejects_symlink_leaf() {
use std::os::unix::fs::symlink;
let root_dir = tempfile::tempdir().expect("temp dir should be created");
let outside_file = root_dir.path().join("outside-file");
fs::write(&outside_file, b"outside")
.await
.expect("outside file should be created");
let endpoint = Endpoint::try_from(root_dir.path().to_string_lossy().as_ref()).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
disk.make_volume("bucket").await.expect("bucket should be created");
symlink(&outside_file, root_dir.path().join("bucket/object")).expect("object symlink should be created");
assert!(matches!(disk.get_object_path_for_io("bucket", "object"), Err(DiskError::InvalidPath)));
}
#[tokio::test]
async fn get_object_path_rejects_key_traversal_out_of_bucket() {
let root_dir = tempfile::tempdir().expect("temp dir should be created");
let endpoint = Endpoint::try_from(root_dir.path().to_string_lossy().as_ref()).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
assert!(matches!(disk.get_object_path("bucket", "../outside"), Err(DiskError::InvalidPath)));
assert!(matches!(
disk.get_object_path("bucket", "prefix/../../outside"),
Err(DiskError::InvalidPath)
));
}
#[tokio::test]
async fn get_object_path_accepts_missing_leaf_under_existing_bucket() {
let root_dir = tempfile::tempdir().expect("temp dir should be created");
let endpoint = Endpoint::try_from(root_dir.path().to_string_lossy().as_ref()).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
disk.make_volume("bucket").await.expect("bucket should be created");
let object_path = disk
.get_object_path("bucket", "missing-object")
.expect("missing leaf under a valid bucket should resolve");
assert_eq!(object_path, disk.root.join("bucket/missing-object"));
}
#[tokio::test]
async fn get_object_path_for_io_rejects_key_traversal_out_of_bucket() {
let root_dir = tempfile::tempdir().expect("temp dir should be created");
let endpoint = Endpoint::try_from(root_dir.path().to_string_lossy().as_ref()).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
assert!(matches!(disk.get_object_path_for_io("bucket", "../outside"), Err(DiskError::InvalidPath)));
assert!(matches!(
disk.get_object_path_for_io("bucket", "prefix/../../outside"),
Err(DiskError::InvalidPath)
));
}
#[tokio::test]
async fn get_object_path_for_io_accepts_missing_leaf_under_existing_bucket() {
let root_dir = tempfile::tempdir().expect("temp dir should be created");
let endpoint = Endpoint::try_from(root_dir.path().to_string_lossy().as_ref()).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
disk.make_volume("bucket").await.expect("bucket should be created");
let object_path = disk
.get_object_path_for_io("bucket", "missing-object")
.expect("missing leaf under a valid I/O bucket should resolve");
assert!(object_path.ends_with("bucket/missing-object"));
}
#[cfg(unix)]
#[tokio::test]
async fn get_object_path_rejects_symlink_component_after_prior_valid_lookup() {
use std::os::unix::fs::symlink;
let root_dir = tempfile::tempdir().expect("temp dir should be created");
let outside_dir = tempfile::tempdir().expect("outside temp dir should be created");
let endpoint = Endpoint::try_from(root_dir.path().to_string_lossy().as_ref()).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
let prefix = root_dir.path().join("bucket/prefix");
fs::create_dir_all(&prefix).await.expect("prefix should be created");
disk.get_object_path("bucket", "prefix/object")
.expect("initial lookup should validate the real prefix");
fs::remove_dir(&prefix).await.expect("prefix should be removable");
symlink(outside_dir.path(), &prefix).expect("prefix should be replaced by a symlink");
assert!(matches!(disk.get_object_path("bucket", "prefix/object"), Err(DiskError::InvalidPath)));
}
#[cfg(unix)]
#[tokio::test]
async fn get_object_path_for_io_rejects_symlink_component_after_prior_valid_lookup() {
use std::os::unix::fs::symlink;
let root_dir = tempfile::tempdir().expect("temp dir should be created");
let outside_dir = tempfile::tempdir().expect("outside temp dir should be created");
let endpoint = Endpoint::try_from(root_dir.path().to_string_lossy().as_ref()).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
let prefix = root_dir.path().join("bucket/prefix");
fs::create_dir_all(&prefix).await.expect("prefix should be created");
disk.get_object_path_for_io("bucket", "prefix/object")
.expect("initial I/O lookup should validate the real prefix");
fs::remove_dir(&prefix).await.expect("prefix should be removable");
symlink(outside_dir.path(), &prefix).expect("prefix should be replaced by a symlink");
assert!(matches!(
disk.get_object_path_for_io("bucket", "prefix/object"),
Err(DiskError::InvalidPath)
));
}
#[tokio::test]
async fn get_object_path_accepts_parent_recreated_after_prior_valid_lookup() {
let root_dir = tempfile::tempdir().expect("temp dir should be created");
let endpoint = Endpoint::try_from(root_dir.path().to_string_lossy().as_ref()).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
let prefix = root_dir.path().join("bucket/prefix");
fs::create_dir_all(&prefix).await.expect("prefix should be created");
disk.get_object_path("bucket", "prefix/object")
.expect("initial lookup should validate the real prefix");
fs::remove_dir(&prefix).await.expect("prefix should be removable");
fs::create_dir(&prefix).await.expect("prefix should be recreated");
let object_path = disk
.get_object_path("bucket", "prefix/object")
.expect("recreated non-symlink parent should validate");
assert_eq!(object_path, disk.root.join("bucket/prefix/object"));
}
#[tokio::test]
async fn get_object_path_for_io_accepts_parent_recreated_after_prior_valid_lookup() {
let root_dir = tempfile::tempdir().expect("temp dir should be created");
let endpoint = Endpoint::try_from(root_dir.path().to_string_lossy().as_ref()).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
let prefix = root_dir.path().join("bucket/prefix");
fs::create_dir_all(&prefix).await.expect("prefix should be created");
disk.get_object_path_for_io("bucket", "prefix/object")
.expect("initial I/O lookup should validate the real prefix");
fs::remove_dir(&prefix).await.expect("prefix should be removable");
fs::create_dir(&prefix).await.expect("prefix should be recreated");
let object_path = disk
.get_object_path_for_io("bucket", "prefix/object")
.expect("recreated non-symlink parent should validate for I/O");
assert!(object_path.ends_with("bucket/prefix/object"));
}
#[tokio::test]
async fn get_object_path_handles_many_unique_missing_prefixes_without_state_growth() {
let root_dir = tempfile::tempdir().expect("temp dir should be created");
let endpoint = Endpoint::try_from(root_dir.path().to_string_lossy().as_ref()).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
disk.make_volume("bucket").await.expect("bucket should be created");
for index in 0..5000 {
let object_path = disk
.get_object_path("bucket", &format!("prefix-{index}/object"))
.expect("unique missing prefix should validate without shared state");
assert!(object_path.ends_with(format!("bucket/prefix-{index}/object")));
}
}
#[tokio::test]
async fn get_object_path_concurrent_validation_keeps_paths_under_bucket() {
let root_dir = tempfile::tempdir().expect("temp dir should be created");
let endpoint = Endpoint::try_from(root_dir.path().to_string_lossy().as_ref()).expect("endpoint should parse");
let disk = Arc::new(LocalDisk::new(&endpoint, false).await.expect("local disk should be created"));
disk.make_volume("bucket").await.expect("bucket should be created");
let barrier = Arc::new(tokio::sync::Barrier::new(32));
let mut tasks = Vec::with_capacity(32);
for index in 0..32 {
let disk = disk.clone();
let barrier = barrier.clone();
tasks.push(tokio::spawn(async move {
barrier.wait().await;
disk.get_object_path("bucket", &format!("object-{index}"))
.expect("concurrent validation should resolve object path")
}));
}
for task in tasks {
let object_path = task.await.expect("validation task should complete");
assert!(object_path.starts_with(disk.root.join("bucket")));
}
}
#[tokio::test]
async fn test_local_disk_file_operations() {
let test_dir = "./test_local_disk_file_ops";
+853 -9
View File
@@ -19,14 +19,14 @@ use futures::TryStreamExt;
use parking_lot::Mutex;
use rustfs_utils::path::SLASH_SEPARATOR;
use std::{
collections::HashMap,
collections::{HashMap, VecDeque},
io,
path::{Component, Path, PathBuf},
sync::{Arc, LazyLock, Weak},
};
use tokio::fs;
use tokio::sync::{
Mutex as AsyncMutex, OwnedMutexGuard, OwnedRwLockReadGuard, OwnedSemaphorePermit, RwLock, Semaphore, SemaphorePermit,
Mutex as AsyncMutex, OwnedMutexGuard, OwnedRwLockReadGuard, OwnedSemaphorePermit, RwLock, Semaphore, SemaphorePermit, oneshot,
};
use tracing::warn;
@@ -79,6 +79,7 @@ pub fn check_path_length(path_name: &str) -> Result<()> {
#[cfg(test)]
pub(crate) mod fsync_dir_recorder {
use std::collections::HashMap;
use std::io;
use std::path::{Path, PathBuf};
use std::sync::Mutex;
@@ -86,8 +87,17 @@ pub(crate) mod fsync_dir_recorder {
static RECORDED: Mutex<Vec<PathBuf>> = Mutex::new(Vec::new());
static LIMITED: Mutex<Vec<PathBuf>> = Mutex::new(Vec::new());
static GROUPED: Mutex<Vec<(PathBuf, usize)>> = Mutex::new(Vec::new());
static BEFORE_LIMITED: std::sync::LazyLock<Mutex<HashMap<PathBuf, Hook>>> =
std::sync::LazyLock::new(|| Mutex::new(HashMap::new()));
static BEFORE_GROUP_BATCH: std::sync::LazyLock<Mutex<HashMap<PathBuf, Hook>>> =
std::sync::LazyLock::new(|| Mutex::new(HashMap::new()));
static AFTER_GROUP_ENQUEUE: std::sync::LazyLock<Mutex<HashMap<PathBuf, Hook>>> =
std::sync::LazyLock::new(|| Mutex::new(HashMap::new()));
static BEFORE_GROUPED: std::sync::LazyLock<Mutex<HashMap<PathBuf, Hook>>> =
std::sync::LazyLock::new(|| Mutex::new(HashMap::new()));
static GROUPED_FAILURES: std::sync::LazyLock<Mutex<HashMap<PathBuf, io::ErrorKind>>> =
std::sync::LazyLock::new(|| Mutex::new(HashMap::new()));
fn record_path(paths: &Mutex<Vec<PathBuf>>, path: &Path, description: &str) {
let mut paths = paths.lock().expect(description);
@@ -106,6 +116,29 @@ pub(crate) mod fsync_dir_recorder {
.any(|recorded| recorded == path || canonical.as_ref().is_some_and(|canonical| recorded == canonical))
}
fn remove_path_keyed<T>(entries: &Mutex<HashMap<PathBuf, T>>, dir: &Path, description: &str) -> Option<T> {
let mut entries = entries.lock().expect(description);
if let Some(value) = entries.remove(dir) {
return Some(value);
}
let canonical = dir.canonicalize().ok();
let matching_key = entries
.keys()
.find(|registered| {
registered.as_path() == dir
|| canonical.as_ref().is_some_and(|canonical| *registered == canonical)
|| registered.canonicalize().ok().is_some_and(|registered_canonical| {
registered_canonical == dir || canonical.as_ref() == Some(&registered_canonical)
})
})
.cloned();
matching_key.and_then(|key| entries.remove(&key))
}
fn remove_hook(hooks: &Mutex<HashMap<PathBuf, Hook>>, dir: &Path, description: &str) -> Option<Hook> {
remove_path_keyed(hooks, dir, description)
}
pub(crate) fn record(dir: &Path) {
record_path(&RECORDED, dir, "fsync dir recorder");
}
@@ -116,7 +149,7 @@ pub(crate) mod fsync_dir_recorder {
pub(crate) fn record_limited(dir: &Path) {
record_path(&LIMITED, dir, "limited fsync dir recorder");
let hook = BEFORE_LIMITED.lock().expect("limited fsync hook poisoned").remove(dir);
let hook = remove_hook(&BEFORE_LIMITED, dir, "limited fsync hook poisoned");
if let Some(hook) = hook {
hook();
}
@@ -132,6 +165,78 @@ pub(crate) mod fsync_dir_recorder {
.expect("limited fsync hook poisoned")
.insert(dir.to_path_buf(), Box::new(hook));
}
pub(crate) fn record_grouped(dir: &Path, batch_len: usize) {
let mut grouped = GROUPED.lock().expect("grouped fsync dir recorder poisoned");
grouped.push((dir.to_path_buf(), batch_len));
if let Ok(canonical) = dir.canonicalize()
&& canonical != dir
{
grouped.push((canonical, batch_len));
}
drop(grouped);
let hook = remove_hook(&BEFORE_GROUPED, dir, "grouped fsync hook poisoned");
if let Some(hook) = hook {
hook();
}
}
pub(crate) fn run_before_group_batch(dir: &Path) {
let hook = remove_hook(&BEFORE_GROUP_BATCH, dir, "grouped fsync batch hook poisoned");
if let Some(hook) = hook {
hook();
}
}
pub(crate) fn set_before_group_batch(dir: &Path, hook: impl FnOnce() + Send + 'static) {
BEFORE_GROUP_BATCH
.lock()
.expect("grouped fsync batch hook poisoned")
.insert(dir.to_path_buf(), Box::new(hook));
}
pub(crate) fn run_after_group_enqueue(dir: &Path) {
let hook = remove_hook(&AFTER_GROUP_ENQUEUE, dir, "grouped fsync enqueue hook poisoned");
if let Some(hook) = hook {
hook();
}
}
pub(crate) fn set_after_group_enqueue(dir: &Path, hook: impl FnOnce() + Send + 'static) {
AFTER_GROUP_ENQUEUE
.lock()
.expect("grouped fsync enqueue hook poisoned")
.insert(dir.to_path_buf(), Box::new(hook));
}
pub(crate) fn grouped_batch_sizes(dir: &Path) -> Vec<usize> {
let grouped = GROUPED.lock().expect("grouped fsync dir recorder poisoned");
let canonical = dir.canonicalize().ok();
grouped
.iter()
.filter_map(|(recorded, batch_len)| {
(recorded == dir || canonical.as_ref().is_some_and(|canonical| recorded == canonical)).then_some(*batch_len)
})
.collect()
}
pub(crate) fn set_before_grouped(dir: &Path, hook: impl FnOnce() + Send + 'static) {
BEFORE_GROUPED
.lock()
.expect("grouped fsync hook poisoned")
.insert(dir.to_path_buf(), Box::new(hook));
}
pub(crate) fn set_grouped_failure(dir: &Path, kind: io::ErrorKind) {
GROUPED_FAILURES
.lock()
.expect("grouped fsync failure hook poisoned")
.insert(dir.to_path_buf(), kind);
}
pub(crate) fn take_grouped_failure(dir: &Path) -> Option<io::ErrorKind> {
remove_path_keyed(&GROUPED_FAILURES, dir, "grouped fsync failure hook poisoned")
}
}
#[cfg(all(test, windows))]
@@ -218,6 +323,374 @@ pub async fn fsync_dir(dir: impl AsRef<Path>) -> io::Result<()> {
}
}
const ENV_DST_DIR_FSYNC_GROUP_COMMIT_ENABLE: &str = "RUSTFS_EXPERIMENTAL_DST_DIR_FSYNC_GROUP_COMMIT_ENABLE";
const DEFAULT_DST_DIR_FSYNC_GROUP_COMMIT_ENABLE: bool = false;
#[cfg(not(test))]
const MAX_DST_DIR_FSYNC_GROUPS: usize = 1024;
#[cfg(test)]
const MAX_DST_DIR_FSYNC_GROUPS: usize = 4;
#[cfg(not(test))]
const MAX_DST_DIR_FSYNC_WAITERS: usize = 8192;
#[cfg(test)]
const MAX_DST_DIR_FSYNC_WAITERS: usize = 8;
static DST_DIR_FSYNC_GROUP_COMMIT_ENABLED: LazyLock<bool> = LazyLock::new(|| {
rustfs_utils::get_env_bool(ENV_DST_DIR_FSYNC_GROUP_COMMIT_ENABLE, DEFAULT_DST_DIR_FSYNC_GROUP_COMMIT_ENABLE)
});
#[cfg(test)]
mod dst_dir_fsync_group_commit_override {
use std::sync::{Mutex, MutexGuard, PoisonError, RwLock};
static OVERRIDE: RwLock<Option<bool>> = RwLock::new(None);
static SERIAL: Mutex<()> = Mutex::new(());
pub(crate) fn get() -> Option<bool> {
*OVERRIDE.read().unwrap_or_else(PoisonError::into_inner)
}
pub(crate) struct OverrideGuard {
_serial: MutexGuard<'static, ()>,
}
impl Drop for OverrideGuard {
fn drop(&mut self) {
*OVERRIDE.write().unwrap_or_else(PoisonError::into_inner) = None;
}
}
pub(crate) fn set(enabled: bool) -> OverrideGuard {
let serial = SERIAL.lock().unwrap_or_else(PoisonError::into_inner);
*OVERRIDE.write().unwrap_or_else(PoisonError::into_inner) = Some(enabled);
OverrideGuard { _serial: serial }
}
}
#[cfg(test)]
pub(crate) fn set_dst_dir_fsync_group_commit_for_test(enabled: bool) -> dst_dir_fsync_group_commit_override::OverrideGuard {
dst_dir_fsync_group_commit_override::set(enabled)
}
fn dst_dir_fsync_group_commit_enabled() -> bool {
#[cfg(test)]
if let Some(enabled) = dst_dir_fsync_group_commit_override::get() {
return enabled;
}
*DST_DIR_FSYNC_GROUP_COMMIT_ENABLED
}
#[derive(Clone, Eq, Hash, PartialEq)]
struct DstDirFsyncGroupKey {
canonical_path: PathBuf,
#[cfg(unix)]
dev: u64,
#[cfg(unix)]
ino: u64,
}
impl DstDirFsyncGroupKey {
fn from_metadata(canonical_path: PathBuf, metadata: std::fs::Metadata) -> io::Result<Self> {
if !metadata.is_dir() {
return Err(io::Error::new(io::ErrorKind::InvalidInput, "dst dir fsync group key must be a directory"));
}
#[cfg(unix)]
{
use std::os::unix::fs::MetadataExt;
Ok(Self {
canonical_path,
dev: metadata.dev(),
ino: metadata.ino(),
})
}
#[cfg(not(unix))]
{
Ok(Self { canonical_path })
}
}
}
struct OpenedDstDirFsyncGroup {
key: DstDirFsyncGroupKey,
#[cfg(any(test, not(unix)))]
dir: PathBuf,
#[cfg(unix)]
dir_file: Arc<std::fs::File>,
}
impl OpenedDstDirFsyncGroup {
fn open(dir: &Path) -> io::Result<Self> {
let canonical_path = dir.canonicalize()?;
#[cfg(unix)]
{
let file = std::fs::File::open(&canonical_path)?;
let key = DstDirFsyncGroupKey::from_metadata(canonical_path, file.metadata()?)?;
#[cfg(test)]
let dir = key.canonical_path.clone();
Ok(Self {
key,
#[cfg(test)]
dir,
dir_file: Arc::new(file),
})
}
#[cfg(not(unix))]
{
let metadata = std::fs::metadata(&canonical_path)?;
let key = DstDirFsyncGroupKey::from_metadata(canonical_path, metadata)?;
let dir = key.canonical_path.clone();
Ok(Self { key, dir })
}
}
}
struct DstDirFsyncWaiter {
result_tx: oneshot::Sender<SharedDstDirFsyncResult>,
}
#[derive(Clone)]
struct SharedDstDirFsyncError {
kind: io::ErrorKind,
message: Arc<str>,
}
impl SharedDstDirFsyncError {
fn from_error(err: io::Error) -> Self {
Self {
kind: err.kind(),
message: Arc::from(err.to_string()),
}
}
fn into_error(self) -> io::Error {
io::Error::new(self.kind, self.message.to_string())
}
}
type SharedDstDirFsyncResult = std::result::Result<(), SharedDstDirFsyncError>;
struct DstDirFsyncGroup {
key: DstDirFsyncGroupKey,
#[cfg(any(test, not(unix)))]
dir: PathBuf,
#[cfg(unix)]
dir_file: Arc<std::fs::File>,
inner: Mutex<DstDirFsyncGroupInner>,
}
#[derive(Default)]
struct DstDirFsyncGroupInner {
worker_running: bool,
pending: VecDeque<DstDirFsyncWaiter>,
}
#[derive(Default)]
struct DstDirFsyncGroupCommit {
inner: Mutex<DstDirFsyncGroupCommitInner>,
}
#[derive(Default)]
struct DstDirFsyncGroupCommitInner {
groups: HashMap<DstDirFsyncGroupKey, Arc<DstDirFsyncGroup>>,
total_waiters: usize,
}
static DST_DIR_FSYNC_GROUP_COMMIT: LazyLock<DstDirFsyncGroupCommit> = LazyLock::new(DstDirFsyncGroupCommit::default);
impl DstDirFsyncGroupCommit {
// Lock order: registry first, then per-group state. No path may hold a
// group lock while acquiring the registry lock.
fn enqueue_opened(
&self,
opened: OpenedDstDirFsyncGroup,
) -> io::Result<(oneshot::Receiver<SharedDstDirFsyncResult>, Option<Arc<DstDirFsyncGroup>>)> {
let (result_tx, result_rx) = oneshot::channel();
let mut registry = self.inner.lock();
if registry.total_waiters >= MAX_DST_DIR_FSYNC_WAITERS {
return Err(io::Error::new(
io::ErrorKind::WouldBlock,
"dst dir fsync group commit waiter limit reached",
));
}
let group = if let Some(group) = registry.groups.get(&opened.key) {
group.clone()
} else {
if registry.groups.len() >= MAX_DST_DIR_FSYNC_GROUPS {
return Err(io::Error::new(
io::ErrorKind::WouldBlock,
"dst dir fsync group commit active group limit reached",
));
}
let group = Arc::new(DstDirFsyncGroup {
key: opened.key.clone(),
#[cfg(any(test, not(unix)))]
dir: opened.dir,
#[cfg(unix)]
dir_file: opened.dir_file,
inner: Mutex::new(DstDirFsyncGroupInner::default()),
});
registry.groups.insert(opened.key, group.clone());
group
};
let mut group_state = group.inner.lock();
group_state.pending.push_back(DstDirFsyncWaiter { result_tx });
let start_worker = !group_state.worker_running;
if start_worker {
group_state.worker_running = true;
}
registry.total_waiters += 1;
drop(group_state);
drop(registry);
#[cfg(test)]
fsync_dir_recorder::run_after_group_enqueue(&group.dir);
Ok((result_rx, start_worker.then_some(group)))
}
fn complete_batch(&self, count: usize) {
let mut registry = self.inner.lock();
registry.total_waiters = registry.total_waiters.saturating_sub(count);
}
fn remove_idle_group(&self, group: &Arc<DstDirFsyncGroup>) {
let mut registry = self.inner.lock();
let group_state = group.inner.lock();
if !group_state.worker_running && group_state.pending.is_empty() {
registry.groups.remove(&group.key);
}
}
#[cfg(test)]
fn counts_for_test(&self) -> (usize, usize) {
let registry = self.inner.lock();
(registry.groups.len(), registry.total_waiters)
}
#[cfg(test)]
fn clear_for_test(&self) {
let mut registry = self.inner.lock();
registry.groups.clear();
registry.total_waiters = 0;
}
#[cfg(test)]
fn enqueue_for_test(
&self,
dir: &Path,
) -> io::Result<(oneshot::Receiver<SharedDstDirFsyncResult>, Option<Arc<DstDirFsyncGroup>>)> {
self.enqueue_opened(OpenedDstDirFsyncGroup::open(dir)?)
}
}
#[cfg(unix)]
async fn fsync_open_dst_dir_group(group: &DstDirFsyncGroup) -> io::Result<()> {
#[cfg(test)]
let dir = group.dir.clone();
let dir_file = group.dir_file.clone();
tokio::task::spawn_blocking(move || {
#[cfg(test)]
{
if let Some(kind) = fsync_dir_recorder::take_grouped_failure(&dir) {
return Err(io::Error::new(kind, "injected grouped dst dir fsync failure"));
}
fsync_dir_recorder::record(&dir);
}
dir_file.sync_all()
})
.await
.map_err(|err| io::Error::other(format!("blocking dst dir group fsync failed: {err}")))?
}
#[cfg(not(unix))]
async fn fsync_open_dst_dir_group(group: &DstDirFsyncGroup) -> io::Result<()> {
fsync_dir(&group.dir).await
}
async fn run_dst_dir_fsync_group_worker(group: Arc<DstDirFsyncGroup>) {
loop {
#[cfg(test)]
fsync_dir_recorder::run_before_group_batch(&group.dir);
tokio::task::yield_now().await;
let batch: Vec<DstDirFsyncWaiter> = {
let mut group_state = group.inner.lock();
group_state.pending.drain(..).collect()
};
if batch.is_empty() {
let mut group_state = group.inner.lock();
group_state.worker_running = false;
drop(group_state);
DST_DIR_FSYNC_GROUP_COMMIT.remove_idle_group(&group);
return;
}
#[cfg(test)]
fsync_dir_recorder::record_grouped(&group.dir, batch.len());
let result = fsync_open_dst_dir_group(&group)
.await
.map_err(SharedDstDirFsyncError::from_error);
let batch_len = batch.len();
DST_DIR_FSYNC_GROUP_COMMIT.complete_batch(batch_len);
let should_stop = {
let mut group_state = group.inner.lock();
if group_state.pending.is_empty() {
group_state.worker_running = false;
true
} else {
false
}
};
if should_stop {
DST_DIR_FSYNC_GROUP_COMMIT.remove_idle_group(&group);
}
for waiter in batch {
let _ = waiter.result_tx.send(result.clone());
}
if should_stop {
return;
}
}
}
async fn fsync_dst_dir_group_commit_with_enabled(dir: impl AsRef<Path>, enabled: bool) -> io::Result<()> {
if !enabled {
return fsync_dir(dir).await;
}
let dir = dir.as_ref().to_path_buf();
let opened = tokio::task::spawn_blocking(move || OpenedDstDirFsyncGroup::open(&dir))
.await
.map_err(|err| io::Error::other(format!("blocking dst dir group open failed: {err}")))??;
let (result_rx, worker) = DST_DIR_FSYNC_GROUP_COMMIT.enqueue_opened(opened)?;
if let Some(group) = worker {
tokio::spawn(run_dst_dir_fsync_group_worker(group));
}
match result_rx.await {
Ok(Ok(())) => Ok(()),
Ok(Err(err)) => Err(err.into_error()),
Err(_) => Err(io::Error::other("dst dir fsync group worker dropped the waiter")),
}
}
pub(crate) async fn fsync_dst_dir_group_commit(dir: impl AsRef<Path>) -> io::Result<()> {
fsync_dst_dir_group_commit_with_enabled(dir, dst_dir_fsync_group_commit_enabled()).await
}
#[cfg(test)]
pub(crate) async fn fsync_dst_dir_group_commit_for_test(dir: impl AsRef<Path>, enabled: bool) -> io::Result<()> {
fsync_dst_dir_group_commit_with_enabled(dir, enabled).await
}
#[cfg(test)]
pub(crate) fn dst_dir_fsync_group_commit_counts_for_test() -> (usize, usize) {
DST_DIR_FSYNC_GROUP_COMMIT.counts_for_test()
}
#[cfg(test)]
fn clear_dst_dir_fsync_group_commit_for_test() {
DST_DIR_FSYNC_GROUP_COMMIT.clear_for_test();
}
// Small object directories are cheaper to flush in one blocking task. Multipart
// directories fan out only once enough files can amortize per-task scheduling.
const PARALLEL_FILE_SYNC_THRESHOLD: usize = 16;
@@ -343,6 +816,7 @@ pub(crate) async fn acquire_rename_data_mutation_lease(
/// this order uniform prevents one slow disk from reserving global capacity
/// while it waits for its own concurrency slot.
async fn acquire_file_sync_permits(disk_permits: Arc<Semaphore>) -> io::Result<(OwnedSemaphorePermit, SemaphorePermit<'static>)> {
let wait_started = rustfs_io_metrics::put_stage_timer();
let disk_permit = disk_permits
.acquire_owned()
.await
@@ -351,6 +825,10 @@ async fn acquire_file_sync_permits(disk_permits: Arc<Semaphore>) -> io::Result<(
.acquire()
.await
.map_err(|_| io::Error::other("global file sync concurrency limiter closed"))?;
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_FILE_SYNC_PERMIT_WAIT,
wait_started,
);
Ok((disk_permit, global_permit))
}
@@ -551,9 +1029,19 @@ pub(crate) fn sync_file(path: &Path) -> io::Result<()> {
file.sync_data()
}
fn sync_file_with_put_stage_metric(path: &Path) -> io::Result<()> {
let sync_started = rustfs_io_metrics::put_stage_timer();
let result = sync_file(path);
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_FILE_FDATASYNC,
sync_started,
);
result
}
fn sync_files(paths: &[PathBuf]) -> io::Result<()> {
for path in paths {
sync_file(path)?;
sync_file_with_put_stage_metric(path)?;
}
Ok(())
}
@@ -599,7 +1087,13 @@ pub(crate) async fn sync_dir_files_with_limiter(dir: impl AsRef<Path>, disk_perm
let files = regular_files(&scan_dir)?;
if files.len() < PARALLEL_FILE_SYNC_THRESHOLD {
sync_files(&files)?;
fsync_dir_std(scan_dir)?;
let fsync_started = rustfs_io_metrics::put_stage_timer();
let result = fsync_dir_std(scan_dir);
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_SRC_DIR_FSYNC,
fsync_started,
);
result?;
return Ok(None);
}
Ok::<_, io::Error>(Some(files))
@@ -612,10 +1106,19 @@ pub(crate) async fn sync_dir_files_with_limiter(dir: impl AsRef<Path>, disk_perm
futures::stream::iter(files.into_iter().map(Ok::<_, io::Error>))
.try_for_each_concurrent(MAX_PARALLEL_FILE_SYNCS, |path| {
let disk_permits = disk_permits.clone();
async move { run_file_sync_blocking(disk_permits, move || sync_file(&path)).await }
async move { run_file_sync_blocking(disk_permits, move || sync_file_with_put_stage_metric(&path)).await }
})
.await?;
run_file_sync_blocking(disk_permits, move || fsync_dir_std(dir)).await
run_file_sync_blocking(disk_permits, move || {
let fsync_started = rustfs_io_metrics::put_stage_timer();
let result = fsync_dir_std(dir);
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_SRC_DIR_FSYNC,
fsync_started,
);
result
})
.await
}
/// Check if the given disk path is the root disk.
@@ -1174,10 +1677,15 @@ pub(crate) struct FileSyncAdmission {
}
pub(crate) async fn acquire_file_sync_admission(disk_permits: Arc<Semaphore>) -> io::Result<FileSyncAdmission> {
let wait_started = rustfs_io_metrics::put_stage_timer();
let disk_permit = disk_permits
.acquire_owned()
.await
.map_err(|_| io::Error::other("disk file sync concurrency limiter closed"))?;
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_FILE_SYNC_PERMIT_WAIT,
wait_started,
);
Ok(FileSyncAdmission {
disk_permit: Arc::new(disk_permit),
})
@@ -1200,10 +1708,15 @@ async fn run_blocking_namespace_file_sync_operation_with_global<T: Send + 'stati
global_permits: &Semaphore,
operation: impl FnOnce() -> io::Result<T> + Send + 'static,
) -> io::Result<T> {
let wait_started = rustfs_io_metrics::put_stage_timer();
let global_permit = global_permits
.acquire()
.await
.map_err(|_| io::Error::other("global file sync concurrency limiter closed"))?;
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_GLOBAL_FILE_SYNC_PERMIT_WAIT,
wait_started,
);
let disk_permit = admission.disk_permit.clone();
let result = tokio::task::spawn_blocking(move || {
let _lease = lease;
@@ -1420,7 +1933,13 @@ fn rename_into_existing_parent(
use rustix::fs::{Mode, OFlags, open, renameat};
let Some(parent_guard) = parent_guard else {
return super::fs::rename_std(src_file_path, dst_file_path);
let rename_started = rustfs_io_metrics::put_stage_timer();
let result = super::fs::rename_std(src_file_path, dst_file_path);
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_RENAME_SYSCALL,
rename_started,
);
return result;
};
let src_parent = src_file_path
.parent()
@@ -1441,7 +1960,13 @@ fn rename_into_existing_parent(
.last()
.ok_or_else(|| io::Error::other("rename destination parent guard is empty"))?;
renameat(&src_parent, src_name, dst_parent, dst_name).map_err(io::Error::from)
let rename_started = rustfs_io_metrics::put_stage_timer();
let result = renameat(&src_parent, src_name, dst_parent, dst_name).map_err(io::Error::from);
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_RENAME_SYSCALL,
rename_started,
);
result
}
#[cfg(windows)]
@@ -2890,6 +3415,7 @@ pub fn is_dir_not_empty_error(err: &io::Error) -> bool {
#[cfg(test)]
mod tests {
use super::*;
use crate::test_metrics::CapturingRecorder;
use std::sync::Mutex;
use std::time::Duration;
use tempfile::tempdir;
@@ -2910,6 +3436,42 @@ mod tests {
PublicationRoot::new(&common).expect("test publication root should open")
}
#[test]
#[serial_test::serial(file_sync_metrics)]
fn sync_file_with_put_stage_metric_records_fdatasync_only_when_enabled() {
let previous_gate = rustfs_io_metrics::put_stage_metrics_enabled();
rustfs_io_metrics::set_put_stage_metrics_enabled(false);
let dir = tempdir().expect("temp dir should be created");
let path = dir.path().join("part.1");
std::fs::write(&path, b"payload").expect("test file should be written");
let recorder = CapturingRecorder::default();
metrics::with_local_recorder(&recorder, || {
sync_file_with_put_stage_metric(&path).expect("disabled metric sync_file should succeed");
assert_eq!(
recorder.histogram_sample_count("rustfs_s3_put_object_stage_duration_ms"),
0,
"disabled PUT stage metrics must not emit fdatasync samples"
);
rustfs_io_metrics::set_put_stage_metrics_enabled(true);
sync_file_with_put_stage_metric(&path).expect("enabled metric sync_file should succeed");
rustfs_io_metrics::set_put_stage_metrics_enabled(false);
});
assert_eq!(
recorder
.histogram_values(
"rustfs_s3_put_object_stage_duration_ms",
&[("stage", rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_FILE_FDATASYNC)]
)
.len(),
1,
"enabled PUT stage metrics must emit one fdatasync sample"
);
rustfs_io_metrics::set_put_stage_metrics_enabled(previous_gate);
}
async fn rename_all(
src_file_path: impl AsRef<Path>,
dst_file_path: impl AsRef<Path>,
@@ -4661,6 +5223,288 @@ mod tests {
fsync_dir(temp_dir.path()).await.expect("fsync dir must succeed");
}
async fn wait_for_dst_dir_fsync_group_commit_idle() {
for _ in 0..100 {
if dst_dir_fsync_group_commit_counts_for_test() == (0, 0) {
return;
}
tokio::task::yield_now().await;
}
assert_eq!(
dst_dir_fsync_group_commit_counts_for_test(),
(0, 0),
"dst dir fsync group registry must release idle groups and waiters"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
#[serial_test::serial(dst_dir_fsync_group_commit)]
async fn dst_dir_fsync_group_commit_default_off_uses_direct_fsync() {
let temp_dir = tempdir().expect("create temp dir");
let dir = temp_dir.path().join("object");
std::fs::create_dir(&dir).expect("create object dir");
fsync_dst_dir_group_commit_for_test(&dir, false)
.await
.expect("direct dst dir fsync should succeed");
assert!(fsync_dir_recorder::was_fsynced(&dir), "default-off path must still fsync the dst dir");
assert!(
fsync_dir_recorder::grouped_batch_sizes(&dir).is_empty(),
"default-off path must not enter the group commit coordinator"
);
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
#[serial_test::serial(dst_dir_fsync_group_commit)]
async fn dst_dir_fsync_group_commit_batches_same_directory_waiters() {
use std::sync::mpsc;
let temp_dir = tempdir().expect("create temp dir");
let dir = temp_dir.path().join("object");
std::fs::create_dir(&dir).expect("create object dir");
let (batch_entered_tx, batch_entered_rx) = mpsc::channel();
let (release_batch_tx, release_batch_rx) = mpsc::channel();
fsync_dir_recorder::set_before_group_batch(&dir, move || {
batch_entered_tx.send(()).expect("signal first worker before freezing batch");
release_batch_rx.recv().expect("wait until second waiter is queued");
});
let first_dir = dir.clone();
let first = tokio::spawn(async move { fsync_dst_dir_group_commit_for_test(first_dir, true).await });
tokio::task::spawn_blocking(move || batch_entered_rx.recv_timeout(Duration::from_secs(30)))
.await
.expect("batch hook waiter should run")
.expect("first worker should reach the batch hook");
let (second_enqueued_tx, second_enqueued_rx) = mpsc::channel();
fsync_dir_recorder::set_after_group_enqueue(&dir, move || {
second_enqueued_tx.send(()).expect("signal second waiter enqueue");
});
let second_dir = dir.clone();
let second = tokio::spawn(async move { fsync_dst_dir_group_commit_for_test(second_dir, true).await });
tokio::task::spawn_blocking(move || second_enqueued_rx.recv_timeout(Duration::from_secs(30)))
.await
.expect("enqueue hook waiter should run")
.expect("second waiter should be enqueued");
assert_eq!(
dst_dir_fsync_group_commit_counts_for_test(),
(1, 2),
"second waiter must be queued before the first batch is released"
);
release_batch_tx.send(()).expect("release first batch");
let (first_result, second_result) = tokio::time::timeout(Duration::from_secs(30), async { tokio::join!(first, second) })
.await
.expect("same-directory fsync waiters should complete");
first_result
.expect("first waiter task should not panic")
.expect("first waiter should observe successful fsync");
second_result
.expect("second waiter task should not panic")
.expect("second waiter should observe successful fsync");
assert_eq!(
fsync_dir_recorder::grouped_batch_sizes(&dir),
vec![2],
"two waiters queued before the batch freezes must share exactly one dst dir fsync"
);
wait_for_dst_dir_fsync_group_commit_idle().await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
#[serial_test::serial(dst_dir_fsync_group_commit)]
async fn dst_dir_fsync_group_commit_late_join_waits_for_next_fsync() {
use std::sync::mpsc;
let temp_dir = tempdir().expect("create temp dir");
let dir = temp_dir.path().join("object");
std::fs::create_dir(&dir).expect("create object dir");
let (fsync_entered_tx, fsync_entered_rx) = mpsc::channel();
let (release_fsync_tx, release_fsync_rx) = mpsc::channel();
fsync_dir_recorder::set_before_grouped(&dir, move || {
fsync_entered_tx.send(()).expect("signal first frozen batch");
release_fsync_rx.recv().expect("wait until late waiter is queued");
});
let first_dir = dir.clone();
let first = tokio::spawn(async move { fsync_dst_dir_group_commit_for_test(first_dir, true).await });
tokio::task::spawn_blocking(move || fsync_entered_rx.recv_timeout(Duration::from_secs(30)))
.await
.expect("grouped fsync hook waiter should run")
.expect("first batch should reach fsync");
let second_dir = dir.clone();
let second = tokio::spawn(async move { fsync_dst_dir_group_commit_for_test(second_dir, true).await });
release_fsync_tx.send(()).expect("release first fsync");
let (first_result, second_result) = tokio::time::timeout(Duration::from_secs(30), async { tokio::join!(first, second) })
.await
.expect("late waiter should complete after a second fsync");
first_result
.expect("first waiter task should not panic")
.expect("first waiter should observe successful fsync");
second_result
.expect("second waiter task should not panic")
.expect("late waiter should observe successful fsync");
assert_eq!(
fsync_dir_recorder::grouped_batch_sizes(&dir),
vec![1, 1],
"a waiter queued after the first batch is frozen must not be covered by the earlier fsync"
);
wait_for_dst_dir_fsync_group_commit_idle().await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
#[serial_test::serial(dst_dir_fsync_group_commit)]
async fn dst_dir_fsync_group_commit_propagates_shared_fsync_failure() {
let temp_dir = tempdir().expect("create temp dir");
let dir = temp_dir.path().join("object");
std::fs::create_dir(&dir).expect("create object dir");
fsync_dir_recorder::set_grouped_failure(&dir, io::ErrorKind::Other);
let err = fsync_dst_dir_group_commit_for_test(&dir, true)
.await
.expect_err("shared dst dir fsync failure must be returned to the waiter");
assert_eq!(err.kind(), io::ErrorKind::Other);
wait_for_dst_dir_fsync_group_commit_idle().await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
#[serial_test::serial(dst_dir_fsync_group_commit)]
async fn dst_dir_fsync_group_commit_cancellation_releases_waiter_state() {
use std::sync::mpsc;
let temp_dir = tempdir().expect("create temp dir");
let dir = temp_dir.path().join("object");
std::fs::create_dir(&dir).expect("create object dir");
let (fsync_entered_tx, fsync_entered_rx) = mpsc::channel();
let (release_fsync_tx, release_fsync_rx) = mpsc::channel();
fsync_dir_recorder::set_before_grouped(&dir, move || {
fsync_entered_tx.send(()).expect("signal grouped fsync");
release_fsync_rx.recv().expect("wait for cancellation");
});
let cancelled_dir = dir.clone();
let cancelled = tokio::spawn(async move { fsync_dst_dir_group_commit_for_test(cancelled_dir, true).await });
tokio::task::spawn_blocking(move || fsync_entered_rx.recv_timeout(Duration::from_secs(30)))
.await
.expect("grouped fsync hook waiter should run")
.expect("first grouped fsync should start");
cancelled.abort();
assert!(
cancelled
.await
.expect_err("cancelled waiter task should abort")
.is_cancelled(),
"waiter cancellation must be observable"
);
release_fsync_tx.send(()).expect("release grouped fsync");
fsync_dst_dir_group_commit_for_test(&dir, true)
.await
.expect("a later waiter should not be blocked by cancelled waiter state");
wait_for_dst_dir_fsync_group_commit_idle().await;
}
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
#[serial_test::serial(dst_dir_fsync_group_commit)]
async fn dst_dir_fsync_group_commit_recreated_directory_gets_new_group() {
use std::sync::mpsc;
let temp_dir = tempdir().expect("create temp dir");
let dir = temp_dir.path().join("object");
std::fs::create_dir(&dir).expect("create object dir");
let (fsync_entered_tx, fsync_entered_rx) = mpsc::channel();
let (release_fsync_tx, release_fsync_rx) = mpsc::channel();
let dir_for_hook = dir.clone();
fsync_dir_recorder::set_before_grouped(&dir, move || {
std::fs::remove_dir(&dir_for_hook).expect("remove old object dir");
std::fs::create_dir(&dir_for_hook).expect("recreate object dir at the same path");
fsync_entered_tx.send(()).expect("signal grouped fsync");
release_fsync_rx.recv().expect("wait until recreated dir is enqueued");
});
let first_dir = dir.clone();
let first = tokio::spawn(async move { fsync_dst_dir_group_commit_for_test(first_dir, true).await });
tokio::task::spawn_blocking(move || fsync_entered_rx.recv_timeout(Duration::from_secs(30)))
.await
.expect("grouped fsync hook waiter should run")
.expect("first grouped fsync should start");
let (_result_rx, worker) = DST_DIR_FSYNC_GROUP_COMMIT
.enqueue_for_test(&dir)
.expect("recreated dir should enqueue separately");
assert!(worker.is_some(), "same path with a new inode must not join the stale in-flight group");
assert_eq!(
dst_dir_fsync_group_commit_counts_for_test().0,
2,
"old and recreated directory identities must be tracked as separate active groups"
);
release_fsync_tx.send(()).expect("release grouped fsync");
first
.await
.expect("first waiter task should not panic")
.expect("first stale directory fd should still fsync successfully");
clear_dst_dir_fsync_group_commit_for_test();
assert_eq!(
dst_dir_fsync_group_commit_counts_for_test(),
(0, 0),
"test registry cleanup must release the unstarted recreated-directory waiter"
);
}
#[test]
#[serial_test::serial(dst_dir_fsync_group_commit)]
fn dst_dir_fsync_group_commit_rejects_active_group_overflow() {
let temp_dir = tempdir().expect("create temp dir");
let mut receivers = Vec::new();
for index in 0..MAX_DST_DIR_FSYNC_GROUPS {
let dir = temp_dir.path().join(format!("object-{index}"));
std::fs::create_dir(&dir).expect("create object dir");
let (result_rx, _worker) = DST_DIR_FSYNC_GROUP_COMMIT
.enqueue_for_test(&dir)
.expect("group below cap should enqueue");
receivers.push(result_rx);
}
let overflow_dir = temp_dir.path().join("overflow");
std::fs::create_dir(&overflow_dir).expect("create overflow dir");
let err = match DST_DIR_FSYNC_GROUP_COMMIT.enqueue_for_test(&overflow_dir) {
Ok(_) => panic!("active group max+1 must fail closed"),
Err(err) => err,
};
assert_eq!(err.kind(), io::ErrorKind::WouldBlock);
clear_dst_dir_fsync_group_commit_for_test();
assert_eq!(dst_dir_fsync_group_commit_counts_for_test(), (0, 0));
drop(receivers);
}
#[test]
#[serial_test::serial(dst_dir_fsync_group_commit)]
fn dst_dir_fsync_group_commit_rejects_waiter_overflow() {
let temp_dir = tempdir().expect("create temp dir");
let dir = temp_dir.path().join("object");
std::fs::create_dir(&dir).expect("create object dir");
let mut receivers = Vec::new();
for _ in 0..MAX_DST_DIR_FSYNC_WAITERS {
let (result_rx, _worker) = DST_DIR_FSYNC_GROUP_COMMIT
.enqueue_for_test(&dir)
.expect("waiter below cap should enqueue");
receivers.push(result_rx);
}
let err = match DST_DIR_FSYNC_GROUP_COMMIT.enqueue_for_test(&dir) {
Ok(_) => panic!("waiter max+1 must fail closed"),
Err(err) => err,
};
assert_eq!(err.kind(), io::ErrorKind::WouldBlock);
clear_dst_dir_fsync_group_commit_for_test();
assert_eq!(dst_dir_fsync_group_commit_counts_for_test(), (0, 0));
drop(receivers);
}
#[tokio::test]
async fn file_sync_admission_is_reused_across_commit_barriers() {
let temp_dir = tempdir().expect("create temp dir");
+8
View File
@@ -1116,6 +1116,14 @@ mod tests {
assert!(encoder_source.is::<reed_solomon_erasure::Error>());
}
// The lifecycle transition worker relies on this arm alone to suppress the
// closed-connection noise (`bucket_lifecycle_ops.rs`); dropping it here would
// silently turn shutdown races back into `error!` log spam.
#[test]
fn is_network_or_host_down_covers_closed_network_connection() {
assert!(is_network_or_host_down("transition failed: use of closed network connection", false));
}
// Regression for #952 (ECA-11): an all-`DiskNotFound` slice (every drive in
// every set unreachable) must NOT be classified as "all not found",
// otherwise ListObjects silently returns an empty listing and masks a full
+14
View File
@@ -277,6 +277,20 @@ pub struct ObjectOptions {
/// fence avoids recursively acquiring the read lock behind a queued writer.
pub bucket_lifecycle_lock_fence: Option<NamespaceLockFence>,
pub replication_request: bool,
/// True when the inbound request carried the
/// `{x-rustfs-,x-minio-}source-proxy-request` header family with the
/// value "true": the request was already proxied by a replication peer,
/// so this server must not proxy a local miss onward (anti-loop,
/// MinIO-compatible). The header only disables proxying — it grants no
/// capability — so no authorization gate is required to honor it.
pub proxy_request: bool,
/// True when the `source-proxy-request` header family was present at
/// all, regardless of value (MinIO's `ProxyHeaderSet`). A replication
/// peer sends `source-proxy-request: false` on its worker convergence
/// HEADs precisely so the receiver answers locally instead of proxying
/// back — otherwise a proxied 404->200 echo makes the worker believe the
/// object already converged and it never replicates it.
pub proxy_header_set: bool,
/// Source-cluster LWW timestamps carried by an authorized replication
/// request; None when the source never modified the category. Only the
/// replication-authorized options builders may set these.
@@ -132,7 +132,6 @@ impl RebalanceStopPropagationRecord {
}
}
#[allow(dead_code)]
#[derive(Debug, Clone, Default)]
pub struct DiskStat {
pub total_space: u64,
@@ -16,8 +16,16 @@ use serde::{Deserialize, Serialize};
use std::{fmt::Display, io};
use tracing::info;
#[allow(
dead_code,
reason = "tier config wire version stamped by the parity constructors below (backlog#1823)"
)]
const C_TIER_CONFIG_VER: &str = "v1";
#[allow(
dead_code,
reason = "tier-name validation message reached only from the parity constructors below (backlog#1823)"
)]
const ERR_TIER_NAME_EMPTY: &str = "remote tier name empty";
const WASABI_US_EAST_ENDPOINT: &str = "https://s3.wasabisys.com";
const WASABI_ALTERNATIVE_ENDPOINTS: &[(&str, &str)] = &[
@@ -264,7 +272,6 @@ impl Clone for TierConfig {
}
}
#[allow(dead_code)]
impl TierConfig {
pub(crate) fn clone_with_credentials(&self) -> Self {
Self {
@@ -284,6 +291,7 @@ impl TierConfig {
}
}
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
fn endpoint(&self) -> String {
match self.tier_type {
TierType::S3 => self.s3.as_ref().map(|s| s.endpoint.clone()).unwrap_or_default(),
@@ -303,6 +311,7 @@ impl TierConfig {
}
}
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
fn bucket(&self) -> String {
match self.tier_type {
TierType::S3 => self.s3.as_ref().map(|s| s.bucket.clone()).unwrap_or_default(),
@@ -322,6 +331,7 @@ impl TierConfig {
}
}
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
fn prefix(&self) -> String {
match self.tier_type {
TierType::S3 => self.s3.as_ref().map(|s| s.prefix.clone()).unwrap_or_default(),
@@ -341,6 +351,7 @@ impl TierConfig {
}
}
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
fn region(&self) -> String {
match self.tier_type {
TierType::S3 => self.s3.as_ref().map(|s| s.region.clone()).unwrap_or_default(),
@@ -457,7 +468,7 @@ impl TierWasabi {
}
impl TierS3 {
#[allow(dead_code)]
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
fn create<F>(
name: &str,
access_key: &str,
@@ -528,7 +539,7 @@ pub struct TierMinIO {
}
impl TierMinIO {
#[allow(dead_code)]
#[allow(dead_code, reason = "MinIO-parity surface with no caller in this port (backlog#1823)")]
fn create<F>(
name: &str,
endpoint: &str,
@@ -14,7 +14,6 @@
use crate::services::tier::tier::TierConfigMgr;
#[allow(dead_code)]
impl TierConfigMgr {
pub fn msg_size(&self) -> usize {
100
@@ -3389,8 +3389,15 @@ impl SetDisks {
// A no-op immediately-ready future in production.
Self::rename_fanout_barrier(&dst_object, i, rename_fanout_barrier_phase::RENAME).await;
disk.rename_data_borrowed(&src_bucket, &src_object, file_info, &dst_bucket, &dst_object)
.await
let disk_wait_started = rustfs_io_metrics::put_stage_timer();
let result = disk
.rename_data_borrowed(&src_bucket, &src_object, file_info, &dst_bucket, &dst_object)
.await;
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_DISK_WAIT,
disk_wait_started,
);
result
})
.catch_unwind()
});
@@ -3403,7 +3410,13 @@ impl SetDisks {
let mut cleanup_data_dirs = vec![None; disk_count];
let mut old_current_sizes = vec![None; disk_count];
let (results, mut file_infos) = fanout.await.map_err(|_| DiskError::Unexpected)?;
let quorum_wait_started = rustfs_io_metrics::put_stage_timer();
let fanout_result = fanout.await;
rustfs_io_metrics::record_put_object_stage_duration_from(
rustfs_io_metrics::PUT_STAGE_SET_DISK_RENAME_QUORUM_WAIT,
quorum_wait_started,
);
let (results, mut file_infos) = fanout_result.map_err(|_| DiskError::Unexpected)?;
for (idx, result) in results.iter().enumerate() {
match result {
@@ -4860,6 +4873,14 @@ impl SetDisks {
/// is best-effort maintenance: individual delete failures are logged and
/// skipped rather than propagated.
pub(crate) async fn reclaim_orphan_data_dirs(&self, bucket: &str, object: &str) -> disk::error::Result<usize> {
self.reclaim_orphan_data_dirs_inner(bucket, object, false).await
}
pub(crate) async fn dry_run_reclaim_orphan_data_dirs(&self, bucket: &str, object: &str) -> disk::error::Result<usize> {
self.reclaim_orphan_data_dirs_inner(bucket, object, true).await
}
async fn reclaim_orphan_data_dirs_inner(&self, bucket: &str, object: &str, dry_run: bool) -> disk::error::Result<usize> {
let disks = self.get_disks_internal().await;
// Phase 1 (read-only): build the referenced-data-dir union and record the
@@ -4967,6 +4988,20 @@ impl SetDisks {
continue;
}
let stray = format!("{object}/{dir}");
if dry_run {
removed += 1;
debug!(
target: "rustfs_ecstore::set_disk",
event = "heal_abandoned_parts",
component = "ecstore",
subsystem = "heal",
state = "dry_run_matched",
result = "matched",
bucket, object, data_dir = %dir,
"Heal abandoned parts dry-run matched orphaned data directory"
);
continue;
}
match disk
.delete(
bucket,
+105 -4
View File
@@ -6998,6 +6998,100 @@ mod tests {
assert!(object_dir.join(STORAGE_FORMAT_FILE).exists(), "metadata must be preserved");
}
async fn recv_abandoned_parts_trace(
trace: &mut rustfs_common::trace_bus::TraceSubscription,
bucket: &str,
object: &str,
state: &str,
) -> rustfs_common::trace_bus::TraceEvent {
for _ in 0..32 {
let event = tokio::time::timeout(std::time::Duration::from_secs(1), trace.recv())
.await
.expect("abandoned-parts trace event should arrive")
.expect("trace bus should stay open");
if event.kind == rustfs_common::trace_bus::TraceKind::Heal
&& event.func == rustfs_common::trace_bus::TraceFunc::HealCheckAbandonedParts
&& event.bucket.as_deref() == Some(bucket)
&& event.object.as_deref() == Some(object)
&& trace_attr_string(&event, "state").as_deref() == Some(state)
{
return (*event).clone();
}
}
panic!("expected abandoned-parts trace state {state} for {bucket}/{object}");
}
fn trace_attr_string(event: &rustfs_common::trace_bus::TraceEvent, key: &str) -> Option<String> {
event.attrs.iter().find_map(|attr| {
if attr.key != key {
return None;
}
Some(match &attr.value {
rustfs_common::trace_bus::TraceVal::Bool(value) => value.to_string(),
rustfs_common::trace_bus::TraceVal::U64(value) => value.to_string(),
rustfs_common::trace_bus::TraceVal::I64(value) => value.to_string(),
rustfs_common::trace_bus::TraceVal::Str(value) => value.to_string(),
})
})
}
#[tokio::test]
async fn check_abandoned_parts_dry_run_counts_without_deleting() {
let mut trace = rustfs_common::trace_bus::subscribe_trace_events();
let (dir, disk) = make_single_local_disk().await;
let live = Uuid::new_v4();
let orphan = Uuid::new_v4();
let object_dir = dir.path().join("bucket").join("obj");
write_object_meta_with_data_dirs(&object_dir, "bucket", "obj", &[live]).await;
fs::create_dir_all(object_dir.join(live.to_string()))
.await
.expect("live data dir should be created");
fs::create_dir_all(object_dir.join(orphan.to_string()))
.await
.expect("orphan data dir should be created");
let set = make_set_disks_with(vec![Some(disk)]).await;
set.check_abandoned_parts(
"bucket",
"obj",
&HealOpts {
dry_run: true,
no_lock: true,
..Default::default()
},
)
.await
.expect("dry-run abandoned-parts check should succeed");
let dry_run_trace = recv_abandoned_parts_trace(&mut trace, "bucket", "obj", "dry_run_matched").await;
assert_eq!(trace_attr_string(&dry_run_trace, "dry_run").as_deref(), Some("true"));
assert_eq!(trace_attr_string(&dry_run_trace, "data_dirs").as_deref(), Some("1"));
assert!(object_dir.join(live.to_string()).exists(), "referenced data dir must be preserved");
assert!(object_dir.join(orphan.to_string()).exists(), "dry-run must not remove orphaned data dir");
set.check_abandoned_parts(
"bucket",
"obj",
&HealOpts {
no_lock: true,
..Default::default()
},
)
.await
.expect("abandoned-parts check should reclaim stale data dir");
let reclaim_trace = recv_abandoned_parts_trace(&mut trace, "bucket", "obj", "reclaimed").await;
assert_eq!(trace_attr_string(&reclaim_trace, "dry_run").as_deref(), Some("false"));
assert_eq!(trace_attr_string(&reclaim_trace, "data_dirs").as_deref(), Some("1"));
assert!(
object_dir.join(live.to_string()).exists(),
"referenced data dir must remain after reclaim"
);
assert!(!object_dir.join(orphan.to_string()).exists(), "orphaned data dir must be removed");
}
#[tokio::test]
async fn reclaim_orphan_data_dirs_recovers_deferred_cleanup_after_restart() {
let (dir, disk) = make_single_local_disk().await;
@@ -12233,11 +12327,18 @@ mod tests {
.expect_err("unsupported copy_object_part should return a typed error");
assert!(matches!(copy_part_err, StorageError::NotImplemented));
let abandoned_err = set_disks
.check_abandoned_parts("bucket", "object", &HealOpts::default())
set_disks
.check_abandoned_parts(
"bucket",
"object",
&HealOpts {
dry_run: true,
no_lock: true,
..Default::default()
},
)
.await
.expect_err("abandoned-parts check should stay in the upper reconciliation layer");
assert!(matches!(abandoned_err, StorageError::NotImplemented));
.expect("abandoned-parts check should be callable on empty disk sets");
}
#[tokio::test]
+275 -5
View File
@@ -16,6 +16,7 @@ use super::super::*;
use crate::disk::disk_store::DiskStoreRenameDataExt;
use crate::io_support::bitrot::object_mmap_read_enabled;
use crate::storage_api_contracts::namespace::NamespaceLocking as _;
use rustfs_common::trace_bus::{TraceEvent, TraceFunc, TraceKind, trace_emit};
use tracing::trace;
const LOG_COMPONENT_ECSTORE: &str = "ecstore";
@@ -2057,11 +2058,61 @@ impl crate::storage_api_contracts::heal::HealOperations for SetDisks {
Err(Error::DiskNotFound)
}
#[tracing::instrument(skip(self))]
async fn check_abandoned_parts(&self, _bucket: &str, _object: &str, _opts: &HealOpts) -> Result<()> {
// Multipart orphan reconciliation is intentionally retained above the set layer
// until there is a concrete caller and a stable lower-level contract to implement.
Err(StorageError::NotImplemented)
#[tracing::instrument(level = "debug", skip(self, opts), fields(bucket = %bucket, object = %object, dry_run = opts.dry_run))]
async fn check_abandoned_parts(&self, bucket: &str, object: &str, opts: &HealOpts) -> Result<()> {
let started_at = std::time::Instant::now();
let _write_lock_guard = if !opts.no_lock {
let ns_lock = self.new_ns_lock(bucket, object).await?;
Some(
ns_lock
.get_write_lock(get_lock_acquire_timeout())
.await
.map_err(|e| self.map_namespace_lock_error(bucket, object, "write", e))?,
)
} else {
None
};
let removed = if opts.dry_run {
self.dry_run_reclaim_orphan_data_dirs(bucket, object).await?
} else {
self.reclaim_orphan_data_dirs(bucket, object).await?
};
let state = if opts.dry_run && removed > 0 {
"dry_run_matched"
} else if removed > 0 {
"reclaimed"
} else {
"checked"
};
let data_dirs = u64::try_from(removed).unwrap_or(u64::MAX);
trace_emit(|| {
TraceEvent::new(TraceKind::Heal, TraceFunc::HealCheckAbandonedParts)
.with_bucket(bucket)
.with_object(object)
.with_duration(started_at.elapsed())
.with_attr("state", state)
.with_attr("dry_run", opts.dry_run)
.with_attr("data_dirs", data_dirs)
});
if removed > 0 {
trace!(
event = "heal_abandoned_parts",
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_HEAL,
state = if opts.dry_run { "dry_run_matched" } else { "reclaimed" },
result = "ok",
bucket,
object,
dry_run = opts.dry_run,
data_dirs = removed,
"Heal abandoned parts checked object data directories"
);
}
Ok(())
}
}
@@ -3246,4 +3297,223 @@ mod heal_result_report_tests {
assert!(result.detail.contains("part 1"));
assert!(result.detail.contains("bitrot_failure=true"));
}
// HS-12 (backlog#1874): a versioned DELETE racing an object heal must never
// resurrect the deleted version. The heal has real reconstruction work (a
// shard of the doomed version is removed), so both sides touch the same
// (bucket, object, data_dir); whichever order the ns write lock serializes
// them in, the committed delete must win.
#[tokio::test]
#[serial_test::serial]
async fn heal_racing_version_delete_never_resurrects_the_deleted_version() {
let (temp_dirs, disks, set) = hermetic_set_disks_isolated(4).await;
let bucket = "heal-race-delete-no-resurrect";
let object = "object.bin";
set.make_bucket(
bucket,
&MakeBucketOptions {
versioning_enabled: true,
..Default::default()
},
)
.await
.expect("versioned bucket should be created");
let mut first_reader = PutObjReader::from_vec(vec![0x11; 1024 * 1024]);
let first_info = set
.put_object(
bucket,
object,
&mut first_reader,
&ObjectOptions {
versioned: true,
..Default::default()
},
)
.await
.expect("first version should be written");
let first_version = first_info
.version_id
.expect("versioned put should return the first version id")
.to_string();
let mut second_reader = PutObjReader::from_vec(vec![0x22; 1024 * 1024]);
let second_info = set
.put_object(
bucket,
object,
&mut second_reader,
&ObjectOptions {
versioned: true,
..Default::default()
},
)
.await
.expect("second version should be written");
let second_version = second_info
.version_id
.expect("versioned put should return the second version id")
.to_string();
// Damage one shard of the doomed version so the racing heal performs an
// actual reconstruction over its data dir instead of an early exit.
let doomed_source = disks[0]
.read_version("", bucket, object, &first_version, &ReadOptions::default())
.await
.expect("doomed version metadata should be readable");
let doomed_data_dir = doomed_source
.data_dir
.expect("non-inline version should have a data directory");
tokio::fs::remove_file(
temp_dirs[1]
.path()
.join(bucket)
.join(object)
.join(doomed_data_dir.to_string())
.join("part.1"),
)
.await
.expect("shard damage should be injected before the race");
let delete_set = set.clone();
let (delete_res, heal_res) = tokio::join!(
async {
delete_set
.delete_object(
bucket,
object,
ObjectOptions {
versioned: true,
version_id: Some(first_version.clone()),
object_lock_config_snapshot: Some(Arc::new(crate::set_disk::ObjectLockConfigSnapshot::new(
crate::bucket::metadata_sys::ObjectLockConfigState::ConfirmedAbsent,
))),
..Default::default()
},
)
.await
},
async {
set.heal_object(
bucket,
object,
"",
&HealOpts {
scan_mode: HealScanMode::Deep,
..Default::default()
},
)
.await
},
);
delete_res.expect("version delete must succeed under lock serialization");
// The heal may legitimately report a transient failure when the version
// it was rebuilding disappears mid-flight; only the end state matters.
drop(heal_res);
let resurrected = set
.get_object_info(
bucket,
object,
&ObjectOptions {
versioned: true,
version_id: Some(first_version.clone()),
..Default::default()
},
)
.await;
assert!(
matches!(&resurrected, Err(Error::FileVersionNotFound) | Err(Error::ObjectNotFound(..))),
"a racing heal must not resurrect the deleted version: {resurrected:?}"
);
let survivor = set
.get_object_info(
bucket,
object,
&ObjectOptions {
versioned: true,
version_id: Some(second_version.clone()),
..Default::default()
},
)
.await
.expect("surviving version must remain readable after the race");
assert_eq!(survivor.size, 1024 * 1024, "survivor size must be intact");
}
// HS-12 (backlog#1874): unversioned overwrite commits race a Deep heal on
// the same object. The overwrite's post-commit tail deletes the replaced
// data dir without the ns lock (object.rs commit tail), which is exactly
// the intersection the audit flagged: the heal must tolerate the tail race
// (retryable outcome) and every committed overwrite must survive — the
// final current version is exactly the last payload written.
#[tokio::test]
#[serial_test::serial]
async fn heal_racing_unversioned_overwrites_preserves_the_last_commit() {
let (temp_dirs, disks, set) = hermetic_set_disks_isolated(4).await;
let bucket = "heal-race-put-overwrite";
let object = "object.bin";
set.make_bucket(bucket, &MakeBucketOptions::default())
.await
.expect("bucket should be created");
const ROUNDS: usize = 8;
const PAYLOAD_SIZE: usize = 256 * 1024;
let mut last_etag = String::new();
for round in 0..ROUNDS {
// Give the heal something to rebuild on alternating rounds: remove a
// shard of the current data dir right before the race.
if round % 2 == 1 {
let current = disks[2]
.read_version("", bucket, object, "", &ReadOptions::default())
.await
.expect("current metadata should be readable");
if let Some(data_dir) = current.data_dir {
let shard = temp_dirs[3]
.path()
.join(bucket)
.join(object)
.join(data_dir.to_string())
.join("part.1");
if shard.exists() {
tokio::fs::remove_file(&shard)
.await
.expect("shard damage should be injectable mid-race");
}
}
}
let payload = vec![round as u8; PAYLOAD_SIZE];
let mut put_reader = PutObjReader::from_vec(payload);
let put_opts = ObjectOptions::default();
let heal_opts = HealOpts {
scan_mode: HealScanMode::Deep,
..Default::default()
};
let (put_res, heal_res) = tokio::join!(
set.put_object(bucket, object, &mut put_reader, &put_opts),
set.heal_object(bucket, object, "", &heal_opts),
);
let put_info = put_res.expect("overwrite must succeed under lock serialization");
last_etag = put_info.etag.clone().unwrap_or_default();
// Heal outcome is unconstrained (may hit the tail race and report a
// retryable error); the invariant is checked on the end state.
drop(heal_res);
}
let final_info = set
.get_object_info(bucket, object, &ObjectOptions::default())
.await
.expect("object must remain readable after the race loop");
assert_eq!(
final_info.size, PAYLOAD_SIZE as i64,
"final current version must be the last committed overwrite"
);
assert_eq!(
final_info.etag.unwrap_or_default(),
last_etag,
"the racing heal loop must never leave a stale or resurrected current version"
);
}
}
+46 -4
View File
@@ -23,6 +23,7 @@
//! per-version `SetDisks::heal_object`.
use super::super::*;
use crate::object_api::ObjectInfo;
use std::collections::HashSet;
use std::sync::Mutex;
use std::sync::atomic::{AtomicBool, AtomicUsize, Ordering};
@@ -39,12 +40,16 @@ const BACKGROUND_WALKDIR_STALL_TIMEOUT: Duration = Duration::from_secs(60);
/// it must not gate healing logic — the delete-marker vs data path is chosen
/// inside `ops/heal.rs` from the resolved latest metadata. `version_id` is
/// normalized (nil/absent UUID => `None`).
#[derive(Debug, Clone, PartialEq, Eq)]
#[derive(Debug, Clone)]
pub struct HealWalkVersion {
/// object key
pub name: String,
/// normalized version id (`None` when the version is nil/absent)
pub version_id: Option<String>,
/// version modification time as Unix nanoseconds
pub mod_time_unix_nanos: Option<i128>,
/// object snapshot for lifecycle evaluation
pub lifecycle_object_info: Option<ObjectInfo>,
/// whether this version is a delete marker (observability only)
pub is_delete_marker: bool,
}
@@ -63,6 +68,7 @@ struct HealWalkCollector {
bucket: String,
batch_objects: usize,
version_budget: usize,
include_lifecycle_object_info: bool,
objects: Mutex<Vec<HealWalkObject>>,
decode_error: Mutex<Option<DiskError>>,
version_total: AtomicUsize,
@@ -116,10 +122,25 @@ impl HealWalkCollector {
let mut versions = Vec::with_capacity(fiv.versions.len() + fiv.free_versions.len());
for fi in fiv.versions.iter().chain(fiv.free_versions.iter()) {
let version_uuid = fi.version_id.filter(|version_id| !version_id.is_nil());
let lifecycle_object_info = if self.include_lifecycle_object_info {
let mut lifecycle_fi = fi.clone();
lifecycle_fi.version_id = version_uuid;
Some(ObjectInfo::from_file_info(
&lifecycle_fi,
&self.bucket,
&entry.name,
version_uuid.is_some(),
))
} else {
None
};
versions.push(HealWalkVersion {
name: entry.name.clone(),
// Normalize: nil/absent version id => None.
version_id: fi.version_id.filter(|u| !u.is_nil()).map(|u| u.to_string()),
version_id: version_uuid.map(|u| u.to_string()),
mod_time_unix_nanos: fi.mod_time.map(|mod_time| mod_time.unix_timestamp_nanos()),
lifecycle_object_info,
is_delete_marker: fi.deleted,
});
}
@@ -173,11 +194,26 @@ impl HealWalkCollector {
}
};
for fi in fiv.versions.iter().chain(fiv.free_versions.iter()) {
let vid = fi.version_id.filter(|u| !u.is_nil()).map(|u| u.to_string());
let version_uuid = fi.version_id.filter(|version_id| !version_id.is_nil());
let vid = version_uuid.map(|u| u.to_string());
if seen.insert(vid.clone()) {
let lifecycle_object_info = if self.include_lifecycle_object_info {
let mut lifecycle_fi = fi.clone();
lifecycle_fi.version_id = version_uuid;
Some(ObjectInfo::from_file_info(
&lifecycle_fi,
&self.bucket,
&entry.name,
version_uuid.is_some(),
))
} else {
None
};
versions.push(HealWalkVersion {
name: entry.name.clone(),
version_id: vid,
mod_time_unix_nanos: fi.mod_time.map(|mod_time| mod_time.unix_timestamp_nanos()),
lifecycle_object_info,
is_delete_marker: fi.deleted,
});
}
@@ -255,6 +291,7 @@ impl SetDisks {
forward_to: Option<&str>,
batch_objects: usize,
version_budget: usize,
include_lifecycle_object_info: bool,
) -> disk::error::Result<(Vec<HealWalkVersion>, Option<String>, bool)> {
assert!(batch_objects >= 2, "heal_walk_versions_page requires batch_objects >= 2");
@@ -264,6 +301,7 @@ impl SetDisks {
bucket: bucket.to_string(),
batch_objects,
version_budget: version_budget.max(1),
include_lifecycle_object_info,
objects: Mutex::new(Vec::new()),
decode_error: Mutex::new(None),
version_total: AtomicUsize::new(0),
@@ -347,6 +385,7 @@ mod tests {
bucket: "bucket".to_string(),
batch_objects: 2,
version_budget: 2,
include_lifecycle_object_info: false,
objects: Mutex::new(Vec::new()),
decode_error: Mutex::new(None),
version_total: AtomicUsize::new(0),
@@ -388,6 +427,8 @@ mod tests {
HealWalkVersion {
name: name.to_string(),
version_id: Some(id.to_string()),
mod_time_unix_nanos: None,
lifecycle_object_info: None,
is_delete_marker: dm,
}
}
@@ -491,6 +532,7 @@ mod tests {
bucket: "bucket".to_string(),
batch_objects: 1000,
version_budget: 10_000,
include_lifecycle_object_info: false,
objects: Mutex::new(Vec::new()),
version_total: AtomicUsize::new(0),
decode_error: Mutex::new(None),
@@ -567,7 +609,7 @@ mod tests {
.expect("corrupt test metadata should be written");
let error = set_disks
.heal_walk_versions_page(bucket, "", None, 2, 2)
.heal_walk_versions_page(bucket, "", None, 2, 2, false)
.await
.expect_err("semantic metadata corruption must fail the heal disk walk");
@@ -5845,6 +5845,14 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
#[tracing::instrument(skip(self))]
async fn add_partial(&self, bucket: &str, object: &str, version_id: &str) -> Result<()> {
// MRF journal intent: partial-write recovery must survive a restart
// (HS-01); the heal request below remains the in-memory fast path.
rustfs_common::mrf_channel::try_send_mrf_intent(
rustfs_common::mrf_channel::MrfKind::PartialWrite,
bucket,
object,
uuid::Uuid::try_parse(version_id).ok(),
);
let mut request = rustfs_common::heal_channel::create_heal_request_with_options(
bucket.to_string(),
Some(object.to_string()),
+9
View File
@@ -1077,6 +1077,15 @@ impl SetDisks {
"Recoverable decode error triggered read repair"
);
let version_id = fi.version_id.as_ref().map(ToString::to_string);
// MRF journal intent: keeps a durable Urgent ECDecode
// request alive across restarts even when the in-memory
// read-repair request is dropped or lost (HS-01).
rustfs_common::mrf_channel::try_send_mrf_intent(
rustfs_common::mrf_channel::MrfKind::DecodeFailure,
bucket,
object,
fi.version_id,
);
submit_read_repair_heal(
bucket,
object,
+35 -7
View File
@@ -18,6 +18,7 @@ use tracing::trace;
const LOG_COMPONENT_ECSTORE: &str = "ecstore";
const LOG_SUBSYSTEM_HEAL: &str = "heal";
const EVENT_HEAL_ABANDONED_PARTS: &str = "heal_abandoned_parts";
const EVENT_HEAL_FORMAT_COMPLETED: &str = "heal_format_completed";
const EVENT_HEAL_OBJECT_STARTED: &str = "heal_object_started";
@@ -256,13 +257,40 @@ impl ECStore {
#[instrument(skip(self))]
pub(super) async fn handle_check_abandoned_parts(&self, bucket: &str, object: &str, opts: &HealOpts) -> Result<()> {
let _ = (bucket, object, opts);
// Stale multipart reconciliation is already owned by the lifecycle-driven
// background cleanup path in `bucket_lifecycle_ops.rs`. There is currently
// no stable object-heal contract that should fan this request out through
// pool/set storage layers, so keep the placeholder explicit at the ECStore
// boundary instead of dispatching into lower layers.
Err(StorageError::NotImplemented)
let object = encode_dir_object(object);
let pools = self.get_pools_for_heal_object(opts)?;
let mut futures = Vec::with_capacity(pools.len());
for pool in pools.iter() {
futures.push(pool.check_abandoned_parts(bucket, &object, opts));
}
let mut first_error = None;
for result in join_all(futures).await {
if let Err(err) = result
&& first_error.is_none()
{
first_error = Some(err);
}
}
if let Some(err) = first_error {
return Err(err);
}
trace!(
event = EVENT_HEAL_ABANDONED_PARTS,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_HEAL,
state = "completed",
result = "ok",
bucket,
object,
dry_run = opts.dry_run,
"Heal abandoned parts completed"
);
Ok(())
}
}
+2 -1
View File
@@ -34,6 +34,7 @@ impl ECStore {
forward_to: Option<&str>,
batch_objects: usize,
version_budget: usize,
include_lifecycle_object_info: bool,
) -> Result<(Vec<HealWalkVersion>, Option<String>, bool)> {
if pool_idx >= self.pools.len() || set_idx >= self.pools[pool_idx].disk_set.len() {
return Err(Error::other(format!(
@@ -43,7 +44,7 @@ impl ECStore {
}
self.pools[pool_idx].disk_set[set_idx]
.heal_walk_versions_page(bucket, prefix, forward_to, batch_objects, version_budget)
.heal_walk_versions_page(bucket, prefix, forward_to, batch_objects, version_budget, include_lifecycle_object_info)
.await
.map_err(Error::from)
}
+10
View File
@@ -216,6 +216,16 @@ impl std::fmt::Debug for ECStore {
/// These delegate to the process-global statics. No local state — the globals
/// remain the single source of truth until the migration is complete.
impl ECStore {
/// Every erasure set across all pools, pool-major order.
///
/// Read-only queries that must consult each set's own copy of a
/// per-bucket object (e.g. the scanner's `.usage-cache.bin`) iterate
/// this instead of the hash-routed store path, which would always land
/// on one set (rustfs/backlog#1872).
pub fn all_set_disks(&self) -> Vec<Arc<crate::set_disk::SetDisks>> {
self.pools.iter().flat_map(|pool| pool.disk_set.iter().cloned()).collect()
}
/// Get server configuration (delegates to global)
pub fn get_server_config(&self) -> Option<Config> {
runtime_sources::server_config()
+3
View File
@@ -42,6 +42,9 @@ const FILEINFO_PART_BITMAP_WORD_BITS: usize = std::mem::size_of::<u64>() * 8;
const FILEINFO_PART_BITMAP_WORDS: usize = MAX_FILEINFO_PARTS.div_ceil(FILEINFO_PART_BITMAP_WORD_BITS);
// Additional constants from Go version
// Intentionally duplicated (S3 wire literal): rustfs-replication and
// rustfs-object-data-cache carry their own independent "null" constants so
// they stay free of a rustfs-filemeta dependency. Keep all three in sync.
pub const NULL_VERSION_ID: &str = "null";
// pub const RUSTFS_ERASURE_UPGRADED: &str = "x-rustfs-internal-erasure-upgraded";
+2
View File
@@ -89,6 +89,8 @@ async-trait = { workspace = true }
futures = { workspace = true }
metrics = { workspace = true }
base64 = { workspace = true }
bytes = { workspace = true }
crc-fast = { workspace = true }
[dev-dependencies]
serde_json = { workspace = true, features = ["raw_value"] }
+100 -17
View File
@@ -66,21 +66,37 @@ struct HealTaskStatusPayload<'a> {
summary: &'a str,
items: &'a [HealResultItem],
truncated: bool,
/// Cursor for incremental consumption (HS-06): sequence of the next item
/// to be produced. Absent on responses without sequencing (0).
#[serde(skip_serializing_if = "u64_is_zero")]
next_seq: u64,
/// Oldest sequence still retained; with `truncated`, tells a lagging
/// client where to restart its cursor.
#[serde(skip_serializing_if = "u64_is_zero")]
min_seq: u64,
#[serde(skip_serializing_if = "Option::is_none")]
progress: Option<&'a HealProgress>,
}
fn u64_is_zero(value: &u64) -> bool {
*value == 0
}
fn encode_heal_task_status_payload(
summary: &str,
mut items: Vec<HealResultItem>,
progress: Option<&HealProgress>,
mut truncated: bool,
next_seq: u64,
min_seq: u64,
) -> Result<(Vec<u8>, bool)> {
loop {
let data = serde_json::to_vec(&HealTaskStatusPayload {
summary,
items: &items,
truncated,
next_seq,
min_seq,
progress,
})
.map_err(|e| Error::Serialization(format!("failed to serialize heal task status: {e}")))?;
@@ -109,8 +125,10 @@ fn encode_heal_status_response(
progress: Option<&HealProgress>,
detail: Option<String>,
truncated: bool,
next_seq: u64,
min_seq: u64,
) -> Result<(Vec<u8>, Option<String>)> {
let (data, truncated) = encode_heal_task_status_payload(summary, items, progress, truncated)?;
let (data, truncated) = encode_heal_task_status_payload(summary, items, progress, truncated, next_seq, min_seq)?;
Ok((data, heal_status_detail(detail, truncated)))
}
@@ -138,8 +156,19 @@ impl HealChannelProcessor {
/// Execute a token query directly against the manager.
pub async fn execute_query_request(&self, heal_path: String, client_token: String) -> Result<HealChannelResponse> {
self.execute_query_request_since(heal_path, client_token, None).await
}
/// Incremental variant of [`Self::execute_query_request`] (HS-06).
pub async fn execute_query_request_since(
&self,
heal_path: String,
client_token: String,
since_seq: Option<u64>,
) -> Result<HealChannelResponse> {
let (response_tx, response_rx) = oneshot::channel();
self.process_query_request(heal_path, client_token, response_tx).await?;
self.process_query_request(heal_path, client_token, since_seq, response_tx)
.await?;
response_rx
.await
.map_err(|err| Error::other(format!("heal query channel closed: {err}")))?
@@ -262,8 +291,12 @@ impl HealChannelProcessor {
HealChannelCommand::Query {
heal_path,
client_token,
since_seq,
response_tx,
} => self.process_query_request(heal_path, client_token, response_tx).await,
} => {
self.process_query_request(heal_path, client_token, since_seq, response_tx)
.await
}
HealChannelCommand::Cancel {
heal_path,
client_token,
@@ -384,6 +417,7 @@ impl HealChannelProcessor {
&self,
heal_path: String,
client_token: String,
since_seq: Option<u64>,
response_tx: oneshot::Sender<std::result::Result<HealChannelResponse, String>>,
) -> Result<()> {
debug!(
@@ -398,72 +432,118 @@ impl HealChannelProcessor {
);
let report = if heal_path.trim_matches('/').is_empty() {
self.heal_manager.get_task_report(&client_token).await
self.heal_manager.get_task_report_since(&client_token, since_seq).await
} else {
self.heal_manager.get_task_report_for_path(&heal_path, &client_token).await
self.heal_manager
.get_task_report_for_path_since(&heal_path, &client_token, since_seq)
.await
};
let (summary, detail, items, truncated, progress) = match report {
let (summary, detail, items, truncated, progress, next_seq, min_seq) = match report {
Ok(HealTaskReport {
status: HealTaskStatus::Pending | HealTaskStatus::Running,
result_items,
result_items_truncated,
progress,
}) => ("running".to_string(), None, result_items, result_items_truncated, progress),
next_seq,
min_seq,
}) => (
"running".to_string(),
None,
result_items,
result_items_truncated,
progress,
next_seq,
min_seq,
),
Ok(HealTaskReport {
status: HealTaskStatus::Retrying { error, retry_attempt },
result_items,
result_items_truncated,
progress,
next_seq,
min_seq,
}) => (
"running".to_string(),
Some(format!("heal task retrying after recoverable failure, attempt {retry_attempt}: {error}")),
result_items,
result_items_truncated,
progress,
next_seq,
min_seq,
),
Ok(HealTaskReport {
status: HealTaskStatus::Completed,
result_items,
result_items_truncated,
progress,
}) => ("finished".to_string(), None, result_items, result_items_truncated, progress),
next_seq,
min_seq,
}) => (
"finished".to_string(),
None,
result_items,
result_items_truncated,
progress,
next_seq,
min_seq,
),
Ok(HealTaskReport {
status: HealTaskStatus::Cancelled,
result_items,
result_items_truncated,
progress,
next_seq,
min_seq,
}) => (
"stopped".to_string(),
Some("heal task cancelled".to_string()),
result_items,
result_items_truncated,
progress,
next_seq,
min_seq,
),
Ok(HealTaskReport {
status: HealTaskStatus::Timeout,
result_items,
result_items_truncated,
progress,
next_seq,
min_seq,
}) => (
"stopped".to_string(),
Some("heal task timed out".to_string()),
result_items,
result_items_truncated,
progress,
next_seq,
min_seq,
),
Ok(HealTaskReport {
status: HealTaskStatus::Failed { error },
result_items,
result_items_truncated,
progress,
}) => ("stopped".to_string(), Some(error), result_items, result_items_truncated, progress),
next_seq,
min_seq,
}) => (
"stopped".to_string(),
Some(error),
result_items,
result_items_truncated,
progress,
next_seq,
min_seq,
),
Err(crate::Error::TaskNotFound { .. }) => (
"notFound".to_string(),
Some("heal task not found or expired".to_string()),
Vec::new(),
false,
None,
0,
0,
),
Err(crate::Error::InvalidClientToken) => {
let response = HealChannelResponse {
@@ -490,7 +570,8 @@ impl HealChannelProcessor {
}
};
let (data, detail) = encode_heal_status_response(&summary, items, progress.as_ref(), detail, truncated)?;
let (data, detail) =
encode_heal_status_response(&summary, items, progress.as_ref(), detail, truncated, next_seq, min_seq)?;
let response = HealChannelResponse {
request_id: client_token,
@@ -612,7 +693,8 @@ impl HealChannelProcessor {
HealRequestSource::Admin
| HealRequestSource::AutoHeal
| HealRequestSource::Internal
| HealRequestSource::ReadRepair => true,
| HealRequestSource::ReadRepair
| HealRequestSource::Mrf => true,
});
// Build HealOptions with all available fields
@@ -767,6 +849,7 @@ mod tests {
_bucket: &str,
_prefix: &str,
_continuation_token: Option<&str>,
_include_lifecycle_object_info: bool,
) -> crate::Result<(Vec<crate::heal::storage::HealListItem>, Option<String>, bool)> {
Ok((vec![], None, false))
}
@@ -803,7 +886,7 @@ mod tests {
..Default::default()
}];
let (data, detail) = encode_heal_status_response("running", items, None, None, false).unwrap();
let (data, detail) = encode_heal_status_response("running", items, None, None, false, 0, 0).unwrap();
assert!(data.len() <= MAX_HEAL_STATUS_PAYLOAD_SIZE);
let payload: serde_json::Value = serde_json::from_slice(&data).unwrap();
@@ -1573,7 +1656,7 @@ mod tests {
let (tx, rx) = oneshot::channel();
processor
.process_query_request("bucket".to_string(), "completed-token".to_string(), tx)
.process_query_request("bucket".to_string(), "completed-token".to_string(), None, tx)
.await
.expect("query should process");
@@ -1608,7 +1691,7 @@ mod tests {
let (tx, rx) = oneshot::channel();
processor
.process_query_request("bucket".to_string(), task_id.clone(), tx)
.process_query_request("bucket".to_string(), task_id.clone(), None, tx)
.await
.expect("query should process");
@@ -1641,7 +1724,7 @@ mod tests {
let (tx, rx) = oneshot::channel();
processor
.process_query_request("bucket".to_string(), "wrong-token".to_string(), tx)
.process_query_request("bucket".to_string(), "wrong-token".to_string(), None, tx)
.await
.expect("query should process");
@@ -1666,7 +1749,7 @@ mod tests {
let (tx, rx) = oneshot::channel();
processor
.process_query_request(String::new(), "wrong-token".to_string(), tx)
.process_query_request(String::new(), "wrong-token".to_string(), None, tx)
.await
.expect("query should process");
@@ -1703,7 +1786,7 @@ mod tests {
let (tx, rx) = oneshot::channel();
processor
.process_query_request(String::new(), task_id.clone(), tx)
.process_query_request(String::new(), task_id.clone(), None, tx)
.await
.expect("query should process");
+317 -27
View File
@@ -23,13 +23,14 @@ use crate::heal::{
};
use crate::{Error, Result};
use futures::{StreamExt, stream::FuturesUnordered};
use metrics::gauge;
use metrics::{counter, gauge};
use rustfs_common::heal_channel::{HealOpts, HealRequestSource, HealScanMode};
use rustfs_madmin::heal_commands::HealResultItem;
use std::sync::{
Arc,
atomic::{AtomicUsize, Ordering},
};
use std::time::{Duration, UNIX_EPOCH};
use tokio::sync::{RwLock, Semaphore};
use tracing::{debug, error, warn};
@@ -47,6 +48,21 @@ enum HealObjectOutcome {
Failed,
}
fn result_object_size_u64(result: &HealResultItem) -> u64 {
u64::try_from(result.object_size).unwrap_or(u64::MAX)
}
const NEW_VERSION_SKIP_GRACE_SECS: u64 = 60;
const NANOS_PER_SECOND: i128 = 1_000_000_000;
fn should_skip_new_version(mod_time_unix_nanos: Option<i128>, started_at_secs: u64) -> bool {
let Some(mod_time_unix_nanos) = mod_time_unix_nanos else {
return false;
};
let cutoff_secs = started_at_secs.saturating_add(NEW_VERSION_SKIP_GRACE_SECS);
mod_time_unix_nanos > i128::from(cutoff_secs).saturating_mul(NANOS_PER_SECOND)
}
struct PageConcurrencyGuard {
in_flight: Arc<AtomicUsize>,
set_label: String,
@@ -492,6 +508,7 @@ impl ErasureSetHealer {
&mut skipped_objects,
resume_manager,
checkpoint_manager,
state.start_time,
)
.await;
@@ -658,6 +675,7 @@ impl ErasureSetHealer {
skipped_objects: &mut u64,
resume_manager: &ResumeManager,
checkpoint_manager: &CheckpointManager,
started_at_secs: u64,
) -> Result<()> {
debug!(
target: "rustfs::heal::erasure_healer",
@@ -710,6 +728,7 @@ impl ErasureSetHealer {
// The end-of-pass summary reports the full failed/skipped counts.
let mut transient_skip_samples_logged = 0_u64;
let mut failure_samples_logged = 0_u64;
let mut bytes_processed = self.progress.read().await.bytes_processed;
// backlog#920: select the per-erasure-set DISK-WALK union enumerator when
// the scan is Deep OR the request came from AutoHeal — these are the paths
@@ -718,17 +737,25 @@ impl ErasureSetHealer {
// which stays the default.
let use_disk_walk =
matches!(self.heal_opts.scan_mode, HealScanMode::Deep) || matches!(self.source, HealRequestSource::AutoHeal);
let lifecycle_expiry_context = self.storage.load_heal_lifecycle_expiry_context(bucket).await?;
let include_lifecycle_object_info = lifecycle_expiry_context.is_some();
loop {
self.verify_replacement_identity_fence("page scan").await?;
// Get one page of object versions
let (objects, next_token, is_truncated) = if use_disk_walk {
self.storage
.list_versions_for_heal_page_disk_walk(set_disk_id, bucket, "", continuation_token.as_deref())
.list_versions_for_heal_page_disk_walk(
set_disk_id,
bucket,
"",
continuation_token.as_deref(),
include_lifecycle_object_info,
)
.await?
} else {
self.storage
.list_objects_for_heal_page(bucket, "", continuation_token.as_deref())
.list_objects_for_heal_page(bucket, "", continuation_token.as_deref(), include_lifecycle_object_info)
.await?
};
let page_is_empty = objects.is_empty();
@@ -736,6 +763,7 @@ impl ErasureSetHealer {
let page_resume_index = *current_object_index;
let semaphore = Arc::new(Semaphore::new(page_concurrency_limit));
let mut page_tasks = FuturesUnordered::new();
let mut completed_in_page = 0usize;
// Capture the last version identity of this page for the anti-loop guard.
let page_last = objects.last().map(|item| (item.name.clone(), item.version_id.clone()));
@@ -751,6 +779,75 @@ impl ErasureSetHealer {
continue;
}
if should_skip_new_version(item.mod_time_unix_nanos, started_at_secs) {
checkpoint_manager.add_processed_object(key).await?;
*processed_objects = processed_objects.saturating_add(1);
completed_in_page = completed_in_page.saturating_add(1);
counter!("rustfs_heal_skipped_new_versions_total").increment(1);
{
let mut progress = self.progress.write().await;
progress.record_skipped_new_version();
progress.set_current_object(Some(format!("skipped_new: {bucket}/{}", item.name)));
progress.update_progress(*processed_objects, *successful_objects, *failed_objects, bytes_processed);
}
debug!(
target: "rustfs::heal::erasure_healer",
event = EVENT_HEAL_ERASURE_OBJECT_STATE,
component = LOG_COMPONENT_HEAL,
subsystem = LOG_SUBSYSTEM_ERASURE_HEALER,
set_disk_id,
bucket,
object = %item.name,
version_id = ?item.version_id,
state = "skipped_new_version",
"Erasure set object version skipped because it was written after heal started"
);
if completed_in_page.is_multiple_of(100) {
checkpoint_manager.update_position(bucket_index, page_resume_index).await?;
}
continue;
}
if let Some(context) = lifecycle_expiry_context.as_ref()
&& self
.storage
.enqueue_heal_lifecycle_expiry(
context,
bucket,
&item.name,
item.version_id.as_deref(),
item.lifecycle_object_info.as_ref(),
)
.await?
{
checkpoint_manager.add_processed_object(key).await?;
*processed_objects = processed_objects.saturating_add(1);
completed_in_page = completed_in_page.saturating_add(1);
counter!("rustfs_heal_skipped_ilm_expired_total").increment(1);
{
let mut progress = self.progress.write().await;
progress.record_skipped_ilm_expired();
progress.set_current_object(Some(format!("skipped_ilm: {bucket}/{}", item.name)));
progress.update_progress(*processed_objects, *successful_objects, *failed_objects, bytes_processed);
}
debug!(
target: "rustfs::heal::erasure_healer",
event = EVENT_HEAL_ERASURE_OBJECT_STATE,
component = LOG_COMPONENT_HEAL,
subsystem = LOG_SUBSYSTEM_ERASURE_HEALER,
set_disk_id,
bucket,
object = %item.name,
version_id = ?item.version_id,
state = "skipped_ilm_expired",
"Erasure set object version skipped because lifecycle expiry was queued"
);
if completed_in_page.is_multiple_of(100) {
checkpoint_manager.update_position(bucket_index, page_resume_index).await?;
}
continue;
}
resume_manager
.set_current_item(Some(bucket.to_string()), Some(item.name.clone()))
.await?;
@@ -777,7 +874,7 @@ impl ErasureSetHealer {
let _permit = match permit {
Ok(permit) => permit,
Err(err) => return (dedup_key, object_name, version_id, Err(err)),
Err(err) => return (dedup_key, object_name, version_id, (0, Err(err))),
};
let _in_flight_guard = PageConcurrencyGuard::new(in_flight, set_label);
@@ -788,7 +885,7 @@ impl ErasureSetHealer {
// recorded as skipped-ok rather than failed. The delete-marker
// vs data path is chosen internally in ops/heal.rs.
let result = if cancel_token.is_cancelled() {
Err(Error::TaskCancelled)
(0, Err(Error::TaskCancelled))
} else {
match storage
.heal_object(&bucket_name, &object_name, version_id.as_deref(), &heal_opts)
@@ -797,8 +894,9 @@ impl ErasureSetHealer {
Ok((result, None))
if target_outcomes_complete(&result, &target_endpoints) =>
{
let object_size = result_object_size_u64(&result);
if !replacement_commit_evidence_required {
Ok(true)
(object_size, Ok(true))
} else {
match storage
.replacement_targets_have_version(
@@ -810,27 +908,42 @@ impl ErasureSetHealer {
)
.await
{
Ok(true) => Ok(true),
Ok(false) => Err(Error::transient_skip(format!(
Ok(true) => (object_size, Ok(true)),
Ok(false) => (object_size, Err(Error::transient_skip(format!(
"Skipped heal for {bucket_name}/{object_name} because replacement target readback did not confirm the committed version"
))),
Err(err) => Err(Error::transient_skip(format!(
)))),
Err(err) => (object_size, Err(Error::transient_skip(format!(
"Skipped heal for {bucket_name}/{object_name} because replacement target readback failed: {err}"
))),
)))),
}
}
}
Ok((_result, None)) if !target_endpoints.is_empty() => Err(Error::transient_skip(format!(
"Skipped heal for {bucket_name}/{object_name} because a replacement target was not committed"
))),
Ok((_result, None)) => Ok(true),
Ok((_, Some(err))) if is_missing_object_dir_heal_result(&object_name, &err) => Ok(false),
Ok((_, Some(err))) | Err(err) => match Self::classify_heal_object_error(&err) {
HealObjectOutcome::Absent => Ok(false),
HealObjectOutcome::Transient => Err(Error::transient_skip(format!(
"Skipped heal for {bucket_name}/{object_name} due to transient error: {err}"
},
Ok((result, None)) if !target_endpoints.is_empty() => (
result_object_size_u64(&result),
Err(Error::transient_skip(format!(
"Skipped heal for {bucket_name}/{object_name} because a replacement target was not committed"
))),
HealObjectOutcome::Failed => Err(err),
),
Ok((result, None)) => (result_object_size_u64(&result), Ok(true)),
Ok((result, Some(err))) if is_missing_object_dir_heal_result(&object_name, &err) => {
(result_object_size_u64(&result), Ok(false))
}
Ok((result, Some(err))) => {
let object_size = result_object_size_u64(&result);
match Self::classify_heal_object_error(&err) {
HealObjectOutcome::Absent => (object_size, Ok(false)),
HealObjectOutcome::Transient => (object_size, Err(Error::transient_skip(format!(
"Skipped heal for {bucket_name}/{object_name} due to transient error: {err}"
)))),
HealObjectOutcome::Failed => (object_size, Err(err)),
}
}
Err(err) => match Self::classify_heal_object_error(&err) {
HealObjectOutcome::Absent => (0, Ok(false)),
HealObjectOutcome::Transient => (0, Err(Error::transient_skip(format!(
"Skipped heal for {bucket_name}/{object_name} due to transient error: {err}"
)))),
HealObjectOutcome::Failed => (0, Err(err)),
},
}
};
@@ -839,11 +952,12 @@ impl ErasureSetHealer {
});
}
let mut completed_in_page = 0usize;
while let Some((key, object, version_id, result)) = page_tasks.next().await {
let (object_size, result) = result;
match result {
Ok(true) => {
*successful_objects += 1;
bytes_processed = bytes_processed.saturating_add(object_size);
checkpoint_manager.add_processed_object(key).await?;
debug!(
target: "rustfs::heal::erasure_healer",
@@ -861,6 +975,7 @@ impl ErasureSetHealer {
Ok(false) => {
checkpoint_manager.add_processed_object(key).await?;
*successful_objects += 1;
bytes_processed = bytes_processed.saturating_add(object_size);
debug!(
target: "rustfs::heal::erasure_healer",
event = EVENT_HEAL_ERASURE_OBJECT_STATE,
@@ -877,6 +992,7 @@ impl ErasureSetHealer {
Err(err @ Error::TaskCancelled) | Err(err @ Error::TaskTimeout) => return Err(err),
Err(Error::TransientSkip { message }) => {
*skipped_objects += 1;
bytes_processed = bytes_processed.saturating_add(object_size);
checkpoint_manager.add_skipped_object(key).await?;
demote_to_debug_when!(!take_failure_log_sample(&mut transient_skip_samples_logged), warn, target: "rustfs::heal::erasure_healer", {
event = EVENT_HEAL_ERASURE_OBJECT_STATE,
@@ -893,6 +1009,7 @@ impl ErasureSetHealer {
}
Err(err) => {
*failed_objects += 1;
bytes_processed = bytes_processed.saturating_add(object_size);
checkpoint_manager.add_failed_object(key).await?;
demote_to_debug_when!(!take_failure_log_sample(&mut failure_samples_logged), warn, target: "rustfs::heal::erasure_healer", {
event = EVENT_HEAL_ERASURE_OBJECT_STATE,
@@ -911,6 +1028,11 @@ impl ErasureSetHealer {
*processed_objects += 1;
completed_in_page += 1;
{
let mut progress = self.progress.write().await;
progress.set_current_object(Some(format!("{bucket}/{object}")));
progress.update_progress(*processed_objects, *successful_objects, *failed_objects, bytes_processed);
}
if completed_in_page.is_multiple_of(100) {
checkpoint_manager.update_position(bucket_index, page_resume_index).await?;
@@ -964,7 +1086,9 @@ impl ErasureSetHealer {
progress.objects_scanned = state.total_objects;
progress.objects_healed = state.successful_objects;
progress.objects_failed = state.failed_objects;
progress.bytes_processed = 0; // set to 0 for now, can be extended later
progress.bytes_processed = 0; // Resume state tracks object counts, not byte counters.
progress.start_time = UNIX_EPOCH.checked_add(Duration::from_secs(state.start_time));
progress.last_update_time = UNIX_EPOCH.checked_add(Duration::from_secs(state.last_update));
progress.set_current_object(state.current_object.clone());
}
}
@@ -1135,13 +1259,15 @@ mod resume_loop_tests {
//! that emits programmable multi-version pages. These exercise the real loop
//! logic (cursor seeding, per-version dedup, anti-loop guard, absence
//! handling) — not merely a mock's own output.
use super::{ErasureSetHealer, target_outcomes_complete};
use super::{
ErasureSetHealer, NANOS_PER_SECOND, NEW_VERSION_SKIP_GRACE_SECS, should_skip_new_version, target_outcomes_complete,
};
use crate::heal::progress::HealProgress;
use crate::heal::resume::{
CheckpointManager, RESUME_CHECKPOINT_FILE, ReplacementTargetIdentity, ResumeDeleteFailure, ResumeManager, ResumeUtils,
compose_key,
};
use crate::heal::storage::{DiskStatus, HealListItem, HealObjectInfo, HealStorageAPI};
use crate::heal::storage::{DiskStatus, HealLifecycleExpiryContext, HealListItem, HealObjectInfo, HealStorageAPI};
use crate::heal::storage_api::status::BucketInfo;
use crate::heal::{
BUCKET_META_PREFIX, DiskOption, DiskStore, EcstoreError, Endpoint, HealDiskExt as _, RUSTFS_META_BUCKET, new_disk,
@@ -1149,7 +1275,7 @@ mod resume_loop_tests {
use crate::{Error, Result};
use rustfs_common::heal_channel::{HealOpts, HealRequestSource};
use rustfs_madmin::heal_commands::{HealDriveInfo, HealResultItem, Infos};
use std::collections::{HashMap, VecDeque};
use std::collections::{HashMap, HashSet, VecDeque};
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex};
use tempfile::TempDir;
@@ -1160,10 +1286,37 @@ mod resume_loop_tests {
HealListItem {
name: name.to_string(),
version_id: version.map(str::to_string),
mod_time_unix_nanos: None,
lifecycle_object_info: None,
is_delete_marker: delete_marker,
}
}
fn item_with_mod_time(name: &str, version: Option<&str>, mod_time_secs: u64) -> HealListItem {
HealListItem {
name: name.to_string(),
version_id: version.map(str::to_string),
mod_time_unix_nanos: Some(i128::from(mod_time_secs).saturating_mul(NANOS_PER_SECOND)),
lifecycle_object_info: None,
is_delete_marker: false,
}
}
#[test]
fn new_version_filter_respects_grace_boundary() {
let started_at = 1_700_000_000;
assert!(!should_skip_new_version(None, started_at));
assert!(!should_skip_new_version(
Some(i128::from(started_at + NEW_VERSION_SKIP_GRACE_SECS).saturating_mul(NANOS_PER_SECOND)),
started_at,
));
assert!(should_skip_new_version(
Some(i128::from(started_at + NEW_VERSION_SKIP_GRACE_SECS + 1).saturating_mul(NANOS_PER_SECOND)),
started_at,
));
}
#[test]
fn target_outcomes_require_each_requested_endpoint_once_and_ok() {
let result = HealResultItem {
@@ -1246,8 +1399,10 @@ mod resume_loop_tests {
/// Target-specific physical readback evidence per `compose_key`; the
/// fake models a healthy backend unless a test explicitly revokes it.
replacement_commit_evidence: Mutex<HashMap<String, ReplacementCommitEvidence>>,
lifecycle_expired: Mutex<HashSet<String>>,
/// every heal_object call recorded as (name, version_id)
heal_calls: Mutex<Vec<(String, Option<String>)>>,
list_include_lifecycle_object_info: Mutex<Vec<bool>>,
replacement_target_identity_sequences: Mutex<VecDeque<Vec<ReplacementTargetIdentity>>>,
fail_listing: AtomicBool,
}
@@ -1274,9 +1429,15 @@ mod resume_loop_tests {
.unwrap()
.insert(compose_key(name, version), ReplacementCommitEvidence::Error(message.to_string()));
}
fn set_lifecycle_expired(&self, name: &str, version: Option<&str>) {
self.lifecycle_expired.lock().unwrap().insert(compose_key(name, version));
}
fn calls(&self) -> Vec<(String, Option<String>)> {
self.heal_calls.lock().unwrap().clone()
}
fn list_include_lifecycle_object_info_calls(&self) -> Vec<bool> {
self.list_include_lifecycle_object_info.lock().unwrap().clone()
}
fn fail_listing(&self) {
self.fail_listing.store(true, Ordering::SeqCst);
}
@@ -1330,6 +1491,23 @@ mod resume_loop_tests {
async fn get_object_checksum(&self, _b: &str, _o: &str) -> Result<Option<String>> {
Ok(None)
}
async fn load_heal_lifecycle_expiry_context(&self, _bucket: &str) -> Result<Option<HealLifecycleExpiryContext>> {
Ok((!self.lifecycle_expired.lock().unwrap().is_empty()).then(HealLifecycleExpiryContext::test))
}
async fn enqueue_heal_lifecycle_expiry(
&self,
_context: &HealLifecycleExpiryContext,
_bucket: &str,
object: &str,
version_id: Option<&str>,
_object_info: Option<&HealObjectInfo>,
) -> Result<bool> {
Ok(self
.lifecycle_expired
.lock()
.unwrap()
.contains(&compose_key(object, version_id)))
}
async fn heal_object(
&self,
_bucket: &str,
@@ -1386,7 +1564,12 @@ mod resume_loop_tests {
_bucket: &str,
_prefix: &str,
continuation_token: Option<&str>,
include_lifecycle_object_info: bool,
) -> Result<(Vec<HealListItem>, Option<String>, bool)> {
self.list_include_lifecycle_object_info
.lock()
.unwrap()
.push(include_lifecycle_object_info);
if self.fail_listing.load(Ordering::SeqCst) {
return Err(Error::other("injected listing failure"));
}
@@ -1476,6 +1659,7 @@ mod resume_loop_tests {
/// Drive one bucket heal pass; returns (processed, successful, failed, skipped, result).
async fn run(env: &Env) -> (u64, u64, u64, u64, Result<()>) {
let state = env.resume.get_state().await;
let mut current_object_index = 0usize;
let mut processed = 0u64;
let mut successful = 0u64;
@@ -1494,6 +1678,7 @@ mod resume_loop_tests {
&mut skipped,
&env.resume,
&env.checkpoint,
state.start_time,
)
.await;
(processed, successful, failed, skipped, result)
@@ -1559,6 +1744,7 @@ mod resume_loop_tests {
let mut successful = 0;
let mut failed = 0;
let mut skipped = 0;
let started_at = env.resume.get_state().await.start_time;
let error = healer
.heal_bucket_with_resume(
@@ -1572,6 +1758,7 @@ mod resume_loop_tests {
&mut skipped,
&env.resume,
&env.checkpoint,
started_at,
)
.await
.expect_err("a remounted target must not begin a new page scan");
@@ -1641,6 +1828,109 @@ mod resume_loop_tests {
assert_eq!(skipped, 0);
}
#[tokio::test]
async fn erasure_set_progress_accumulates_healed_object_bytes() {
let env = make_env().await;
env.storage.set_page(
None,
Page {
items: vec![item("first", Some("v1"), false), item("second", Some("v2"), false)],
next: None,
truncated: false,
},
);
env.storage.set_result(
"first",
Some("v1"),
HealResultItem {
object_size: 1024,
..Default::default()
},
);
env.storage.set_result(
"second",
Some("v2"),
HealResultItem {
object_size: 2048,
..Default::default()
},
);
let (processed, successful, failed, skipped, result) = run(&env).await;
result.expect("page heal should succeed");
assert_eq!(processed, 2);
assert_eq!(successful, 2);
assert_eq!(failed, 0);
assert_eq!(skipped, 0);
let progress = env.healer.progress.read().await;
assert_eq!(progress.objects_scanned, 2);
assert_eq!(progress.objects_healed, 2);
assert_eq!(progress.objects_failed, 0);
assert_eq!(progress.bytes_processed, 3072);
assert!(matches!(progress.current_object.as_deref(), Some("b/first" | "b/second")));
}
#[tokio::test]
async fn erasure_set_skips_versions_written_after_heal_started() {
let env = make_env().await;
let started_at = env.resume.get_state().await.start_time;
env.storage.set_page(
None,
Page {
items: vec![
item_with_mod_time("old", Some("v1"), started_at + NEW_VERSION_SKIP_GRACE_SECS),
item_with_mod_time("new", Some("v2"), started_at + NEW_VERSION_SKIP_GRACE_SECS + 1),
],
next: None,
truncated: false,
},
);
let (processed, successful, failed, skipped, result) = run(&env).await;
result.expect("page heal should succeed");
assert_eq!(processed, 2);
assert_eq!(successful, 1);
assert_eq!(failed, 0);
assert_eq!(skipped, 0);
assert_eq!(env.storage.calls(), vec![("old".to_string(), Some("v1".to_string()))]);
let progress = env.healer.progress.read().await;
assert_eq!(progress.skipped_new_versions, 1);
assert_eq!(progress.objects_scanned, 2);
assert_eq!(progress.objects_healed, 1);
assert_eq!(progress.objects_failed, 0);
}
#[tokio::test]
async fn erasure_set_skips_versions_queued_for_lifecycle_expiry() {
let env = make_env().await;
env.storage.set_page(
None,
Page {
items: vec![item("expired", Some("v1"), false), item("kept", Some("v2"), false)],
next: None,
truncated: false,
},
);
env.storage.set_lifecycle_expired("expired", Some("v1"));
let (processed, successful, failed, skipped, result) = run(&env).await;
result.expect("page heal should succeed");
assert_eq!(processed, 2);
assert_eq!(successful, 1);
assert_eq!(failed, 0);
assert_eq!(skipped, 0);
assert_eq!(env.storage.calls(), vec![("kept".to_string(), Some("v2".to_string()))]);
assert_eq!(env.storage.list_include_lifecycle_object_info_calls(), vec![true]);
let progress = env.healer.progress.read().await;
assert_eq!(progress.skipped_ilm_expired, 1);
assert_eq!(progress.objects_scanned, 2);
assert_eq!(progress.objects_healed, 1);
assert_eq!(progress.objects_failed, 0);
}
#[tokio::test]
async fn bucket_listing_failure_does_not_mark_set_completed() {
let env = make_env().await;
+468 -62
View File
@@ -220,6 +220,11 @@ struct CompletedHealStatus {
result_items: Vec<HealResultItem>,
result_items_truncated: bool,
completed_at: SystemTime,
/// Sequence-stamped retained window, archived with the completion so
/// incremental consumers keep their cursor across the transition (HS-06).
seqed_items: Vec<(u64, HealResultItem)>,
next_seq: u64,
min_seq: u64,
}
#[derive(Debug, Clone)]
@@ -240,6 +245,65 @@ pub struct HealTaskReport {
pub result_items: Vec<HealResultItem>,
pub result_items_truncated: bool,
pub progress: Option<HealProgress>,
/// Cursor for incremental consumption: sequence number of the next item
/// to be produced. `0` on reports from sources without sequencing.
pub next_seq: u64,
/// Oldest sequence still retained (`0` together with `next_seq` when
/// sequencing is unavailable).
pub min_seq: u64,
}
/// Report from a live task, honoring the client's incremental cursor.
async fn active_task_report(task: &HealTask, since: Option<u64>) -> HealTaskReport {
let window = task.get_result_items_since(since).await;
HealTaskReport {
status: task.get_status().await,
result_items: window.items,
// The legacy flag stays set once anything was evicted; a lagging
// incremental cursor additionally marks this response truncated so
// the client knows to restart from `min_seq`.
result_items_truncated: task.result_items_truncated() || window.lagged,
progress: Some(task.get_progress().await),
next_seq: window.next_seq,
min_seq: window.min_seq,
}
}
fn empty_task_report(status: HealTaskStatus) -> HealTaskReport {
HealTaskReport {
status,
result_items: Vec::new(),
result_items_truncated: false,
progress: None,
next_seq: 0,
min_seq: 0,
}
}
fn completed_task_report(completed: &CompletedHealStatus, since: Option<u64>) -> HealTaskReport {
let mut lagged = false;
let result_items = match since {
None => completed.result_items.clone(),
Some(cursor) => {
if cursor + 1 < completed.min_seq {
lagged = true;
}
completed
.seqed_items
.iter()
.filter(|(seq, _)| *seq > cursor)
.map(|(_, item)| item.clone())
.collect()
}
};
HealTaskReport {
status: completed.status.clone(),
result_items,
result_items_truncated: completed.result_items_truncated || lagged,
progress: None,
next_seq: completed.next_seq,
min_seq: completed.min_seq,
}
}
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, serde::Deserialize, serde::Serialize)]
@@ -270,6 +334,8 @@ pub struct HealSourceCounts {
pub auto_heal: u64,
pub internal: u64,
pub read_repair: u64,
#[serde(default)]
pub mrf: u64,
}
impl HealSourceCounts {
@@ -280,6 +346,7 @@ impl HealSourceCounts {
HealRequestSource::AutoHeal => self.auto_heal += 1,
HealRequestSource::Internal => self.internal += 1,
HealRequestSource::ReadRepair => self.read_repair += 1,
HealRequestSource::Mrf => self.mrf += 1,
}
}
}
@@ -528,6 +595,11 @@ impl PriorityHealQueue {
self.dedup_keys.contains_key(&key)
}
/// Iterate queued requests (used by the admin overlap check).
fn requests(&self) -> impl Iterator<Item = &HealRequest> {
self.heap.iter().map(|item| &item.request)
}
fn contains_request_id(&self, request_id: &str) -> bool {
self.heap.iter().any(|item| item.request.id == request_id)
}
@@ -686,6 +758,80 @@ fn recoverable_heal_retry_delay(retry_attempt: u32) -> Duration {
}
/// Heal config
/// HS-06 admin overlap policy.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
pub enum HealOverlapPolicy {
/// Default: overlapping admin starts merge into the existing task
/// (today's dedup semantics).
#[default]
Merge,
/// Return a typed already-running / overlapping-paths rejection like
/// madmin's ErrHealAlreadyRunning / ErrHealOverlappingPaths.
MinioError,
}
/// Path view of a heal type for overlap comparison: a bucket plus a
/// prefix/object path inside it (`None` bucket = cluster-wide, overlaps
/// everything).
fn heal_type_path_view(heal_type: &HealType) -> (Option<&str>, &str) {
match heal_type {
HealType::Cluster => (None, ""),
HealType::Bucket { bucket } => (Some(bucket), ""),
HealType::Prefix { bucket, prefix } => (Some(bucket), prefix),
HealType::Object { bucket, object, .. }
| HealType::Metadata { bucket, object }
| HealType::ECDecode { bucket, object, .. } => (Some(bucket), object),
// MRF/MetaPath heal keys on a meta path; treat the whole set of
// buckets as one namespace so it only overlaps itself exactly.
HealType::MRF { meta_path } => (Some("\u{0}mrf"), meta_path),
// Erasure-set heal: the set id is the overlap dimension.
HealType::ErasureSet { set_disk_id, .. } => (Some("\u{0}set"), set_disk_id),
}
}
/// How two heal paths relate for the admin overlap check (HS-06).
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum OverlapVerdict {
/// Distinct targets: no conflict.
Disjoint,
/// Same target: an identical heal is already in flight.
SameTarget,
/// One target contains the other.
Overlapping,
}
fn prefix_paths_overlap(a: &str, b: &str) -> OverlapVerdict {
if a == b {
return OverlapVerdict::SameTarget;
}
if a.is_empty() || b.is_empty() || a.starts_with(b) || b.starts_with(a) {
return OverlapVerdict::Overlapping;
}
OverlapVerdict::Disjoint
}
fn heal_types_overlap(left: &HealType, right: &HealType) -> OverlapVerdict {
let (left_bucket, left_path) = heal_type_path_view(left);
let (right_bucket, right_path) = heal_type_path_view(right);
match (left_bucket, right_bucket) {
// Cluster-wide overlaps everything (but an exact cluster match is
// SameTarget).
(None, _) | (_, None) => {
if matches!(left, HealType::Cluster) && matches!(right, HealType::Cluster) {
OverlapVerdict::SameTarget
} else {
OverlapVerdict::Overlapping
}
}
(Some(lb), Some(rb)) => {
if lb != rb {
return OverlapVerdict::Disjoint;
}
prefix_paths_overlap(left_path, right_path)
}
}
}
#[derive(Debug, Clone)]
pub struct HealConfig {
/// Whether to enable auto heal
@@ -706,6 +852,9 @@ pub struct HealConfig {
pub low_priority_drop_when_full: bool,
/// Whether notify-driven scheduler wakeups are enabled.
pub event_driven_scheduler_enable: bool,
/// How admin heal starts behave on path overlap (HS-06): merge into the
/// existing task (default) or return a typed already-running rejection.
pub overlap_policy: HealOverlapPolicy,
/// Whether per-set bulkhead scheduling is enabled.
pub set_bulkhead_enable: bool,
/// Whether erasure-set page parallelism is enabled.
@@ -754,6 +903,14 @@ impl Default for HealConfig {
rustfs_config::ENV_HEAL_EVENT_DRIVEN_SCHEDULER_ENABLE,
rustfs_config::DEFAULT_HEAL_EVENT_DRIVEN_SCHEDULER_ENABLE,
);
let overlap_policy =
match rustfs_utils::get_env_str(rustfs_config::ENV_HEAL_OVERLAP_POLICY, rustfs_config::DEFAULT_HEAL_OVERLAP_POLICY)
.to_lowercase()
.as_str()
{
"minio_error" => HealOverlapPolicy::MinioError,
_ => HealOverlapPolicy::Merge,
};
let set_bulkhead_enable = rustfs_utils::get_env_bool(
rustfs_config::ENV_HEAL_SET_BULKHEAD_ENABLE,
rustfs_config::DEFAULT_HEAL_SET_BULKHEAD_ENABLE,
@@ -790,6 +947,7 @@ impl Default for HealConfig {
low_priority_merge_enable,
low_priority_drop_when_full,
event_driven_scheduler_enable,
overlap_policy,
set_bulkhead_enable,
page_parallel_enable,
mainline_throttle_enable,
@@ -1756,6 +1914,50 @@ impl HealManager {
request: HealRequest,
preserve_alias: bool,
) -> Result<HealAdmissionReceipt> {
// HS-06 forceStart semantics (admin only): MinIO stops the old task
// first and then starts the new one. Cancel any active admin task
// overlapping this request's path before entering admission, so the
// fresh task is never merged into the one being replaced.
if request.source == HealRequestSource::Admin && request.force_start {
let overlapping: Vec<String> = {
let active_heals = self.active_heals.lock().await;
active_heals
.iter()
.filter(|(task_id, task)| {
task.source == HealRequestSource::Admin
&& heal_types_overlap(&request.heal_type, &task.heal_type) != OverlapVerdict::Disjoint
&& *task_id != &request.id
})
.map(|(task_id, _)| task_id.clone())
.collect()
};
for task_id in overlapping {
match self.cancel_task(&task_id).await {
Ok(_) => info!(
target: "rustfs::heal::manager",
event = EVENT_HEAL_QUEUE_ADMISSION,
component = LOG_COMPONENT_HEAL,
subsystem = LOG_SUBSYSTEM_MANAGER,
request_id = %request.id,
cancelled_task_id = %task_id,
result = "force_start_cancelled_overlap",
"Admin forceStart cancelled an overlapping heal task"
),
Err(err) => warn!(
target: "rustfs::heal::manager",
event = EVENT_HEAL_QUEUE_ADMISSION,
component = LOG_COMPONENT_HEAL,
subsystem = LOG_SUBSYSTEM_MANAGER,
request_id = %request.id,
cancelled_task_id = %task_id,
error = %err,
result = "force_start_cancel_failed",
"Admin forceStart failed to cancel an overlapping heal task"
),
}
}
}
let config = self.config.read().await;
let dedup_key = PriorityHealQueue::make_dedup_key(&request);
@@ -1778,7 +1980,15 @@ impl HealManager {
.or_else(|| retrying_heal_for_dedup_key(&retrying_heals, &dedup_key).map(|(task_id, _)| (task_id, "retrying")))
});
if let Some((merged_task_id, duplicate_state)) = duplicate.flatten() {
let admission = Self::duplicate_admission_for_request(&request, &config);
// HS-06: under the minio_error overlap policy an exact duplicate
// admin start reports the typed AlreadyRunning rejection instead
// of the silent merge (MinIO's ErrHealAlreadyRunning).
let admission =
if request.source == HealRequestSource::Admin && config.overlap_policy == HealOverlapPolicy::MinioError {
HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning)
} else {
Self::duplicate_admission_for_request(&request, &config)
};
drop(retrying_heals);
drop(queue);
drop(active_heals);
@@ -1824,6 +2034,62 @@ impl HealManager {
});
}
// HS-06 typed overlap rejection (admin only, minio_error policy):
// paths containing or contained by an active/queued task reject with
// AlreadyRunning / OverlappingPaths instead of merging. Exact
// duplicates already merged above; scanner/autoheal/read-repair
// sources never take this path.
if request.source == HealRequestSource::Admin && config.overlap_policy == HealOverlapPolicy::MinioError {
let mut rejection = None;
for (task_id, task) in active_heals.iter() {
match heal_types_overlap(&request.heal_type, &task.heal_type) {
OverlapVerdict::SameTarget => {
rejection = Some((HealAdmissionDropReason::AlreadyRunning, task_id.clone()));
break;
}
OverlapVerdict::Overlapping => {
rejection = Some((HealAdmissionDropReason::OverlappingPaths, task_id.clone()));
}
OverlapVerdict::Disjoint => {}
}
}
if rejection.is_none() {
for queued in queue.requests() {
match heal_types_overlap(&request.heal_type, &queued.heal_type) {
OverlapVerdict::SameTarget => {
rejection = Some((HealAdmissionDropReason::AlreadyRunning, queued.id.clone()));
break;
}
OverlapVerdict::Overlapping => {
rejection = Some((HealAdmissionDropReason::OverlappingPaths, queued.id.clone()));
}
OverlapVerdict::Disjoint => {}
}
}
}
if let Some((reason, overlap_task_id)) = rejection {
drop(retrying_heals);
drop(queue);
drop(active_heals);
Self::record_admission_metric(request.source, HealAdmissionResult::Dropped(reason), "overlap_rejected");
warn!(
target: "rustfs::heal::manager",
event = EVENT_HEAL_QUEUE_ADMISSION,
component = LOG_COMPONENT_HEAL,
subsystem = LOG_SUBSYSTEM_MANAGER,
request_id = %request.id,
overlap_task_id = %overlap_task_id,
reason = reason.as_str(),
result = "overlap_rejected",
"Admin heal start rejected by overlap policy"
);
return Ok(HealAdmissionReceipt {
result: HealAdmissionResult::Dropped(reason),
task_id: overlap_task_id,
});
}
}
let mut task_id = request.id.clone();
let admission = Self::admit_request_to_queue(&mut queue, request, &config, "submit");
if admission == HealAdmissionResult::Merged
@@ -1896,28 +2162,25 @@ impl HealManager {
}
pub async fn get_task_report(&self, task_id: &str) -> Result<HealTaskReport> {
self.get_task_report_since(task_id, None).await
}
/// Incremental variant of [`Self::get_task_report`] (HS-06): `since` is
/// the client's last seen sequence number; `None` keeps the legacy
/// full-snapshot semantics.
pub async fn get_task_report_since(&self, task_id: &str, since: Option<u64>) -> Result<HealTaskReport> {
let canonical_task_id = self.canonical_task_id(task_id).await;
{
let active_heals = self.active_heals.lock().await;
if let Some(task) = active_heals.get(&canonical_task_id) {
return Ok(HealTaskReport {
status: task.get_status().await,
result_items: task.get_result_items().await,
result_items_truncated: task.result_items_truncated(),
progress: Some(task.get_progress().await),
});
return Ok(active_task_report(task, since).await);
}
}
{
let retrying_heals = self.retrying_heals.lock().await;
if let Some(retrying) = retrying_heals.get(&canonical_task_id) {
return Ok(HealTaskReport {
status: retrying.status(),
result_items: Vec::new(),
result_items_truncated: false,
progress: None,
});
return Ok(empty_task_report(retrying.status()));
}
}
@@ -1927,36 +2190,21 @@ impl HealManager {
if let Some(completed) = completed_heals.get(&canonical_task_id)
&& completed_status_is_retrying(&completed.status)
{
return Ok(HealTaskReport {
status: completed.status.clone(),
result_items: completed.result_items.clone(),
result_items_truncated: completed.result_items_truncated,
progress: None,
});
return Ok(completed_task_report(completed, since));
}
}
{
let queue = self.heal_queue.lock().await;
if queue.contains_request_id(&canonical_task_id) {
return Ok(HealTaskReport {
status: HealTaskStatus::Pending,
result_items: Vec::new(),
result_items_truncated: false,
progress: None,
});
return Ok(empty_task_report(HealTaskStatus::Pending));
}
}
let mut completed_heals = self.completed_heals.lock().await;
prune_completed_heal_statuses(&mut completed_heals);
if let Some(completed) = completed_heals.get(&canonical_task_id) {
return Ok(HealTaskReport {
status: completed.status.clone(),
result_items: completed.result_items.clone(),
result_items_truncated: completed.result_items_truncated,
progress: None,
});
return Ok(completed_task_report(completed, since));
}
Err(Error::TaskNotFound {
@@ -1965,18 +2213,23 @@ impl HealManager {
}
pub async fn get_task_report_for_path(&self, heal_path: &str, task_id: &str) -> Result<HealTaskReport> {
self.get_task_report_for_path_since(heal_path, task_id, None).await
}
/// Incremental variant of [`Self::get_task_report_for_path`] (HS-06).
pub async fn get_task_report_for_path_since(
&self,
heal_path: &str,
task_id: &str,
since: Option<u64>,
) -> Result<HealTaskReport> {
let canonical_task_id = self.canonical_task_id(task_id).await;
{
let active_heals = self.active_heals.lock().await;
if let Some(task) = active_heals.get(&canonical_task_id)
&& heal_type_matches_path(&task.heal_type, heal_path)
{
return Ok(HealTaskReport {
status: task.get_status().await,
result_items: task.get_result_items().await,
result_items_truncated: task.result_items_truncated(),
progress: Some(task.get_progress().await),
});
return Ok(active_task_report(task, since).await);
}
}
@@ -1985,12 +2238,7 @@ impl HealManager {
if let Some(retrying) = retrying_heals.get(&canonical_task_id)
&& heal_type_matches_path(&retrying.request.heal_type, heal_path)
{
return Ok(HealTaskReport {
status: retrying.status(),
result_items: Vec::new(),
result_items_truncated: false,
progress: None,
});
return Ok(empty_task_report(retrying.status()));
}
}
@@ -2001,24 +2249,14 @@ impl HealManager {
&& heal_type_matches_path(&completed.heal_type, heal_path)
&& completed_status_is_retrying(&completed.status)
{
return Ok(HealTaskReport {
status: completed.status.clone(),
result_items: completed.result_items.clone(),
result_items_truncated: completed.result_items_truncated,
progress: None,
});
return Ok(completed_task_report(completed, since));
}
}
{
let queue = self.heal_queue.lock().await;
if queue.contains_request_id_matching_path(&canonical_task_id, heal_path) {
return Ok(HealTaskReport {
status: HealTaskStatus::Pending,
result_items: Vec::new(),
result_items_truncated: false,
progress: None,
});
return Ok(empty_task_report(HealTaskStatus::Pending));
}
}
@@ -2028,12 +2266,7 @@ impl HealManager {
if let Some(completed) = completed_heals.get(&canonical_task_id)
&& heal_type_matches_path(&completed.heal_type, heal_path)
{
return Ok(HealTaskReport {
status: completed.status.clone(),
result_items: completed.result_items.clone(),
result_items_truncated: completed.result_items_truncated,
progress: None,
});
return Ok(completed_task_report(completed, since));
}
}
@@ -2385,8 +2618,27 @@ impl HealManager {
snapshot.objects_scanned = snapshot.objects_scanned.saturating_add(progress.objects_scanned);
snapshot.objects_healed = snapshot.objects_healed.saturating_add(progress.objects_healed);
snapshot.objects_failed = snapshot.objects_failed.saturating_add(progress.objects_failed);
snapshot.skipped_new_versions = snapshot.skipped_new_versions.saturating_add(progress.skipped_new_versions);
snapshot.skipped_ilm_expired = snapshot.skipped_ilm_expired.saturating_add(progress.skipped_ilm_expired);
snapshot.objects_total_count = snapshot.objects_total_count.saturating_add(progress.objects_total_count);
snapshot.objects_total_size = snapshot.objects_total_size.saturating_add(progress.objects_total_size);
snapshot.bytes_processed = snapshot.bytes_processed.saturating_add(progress.bytes_processed);
snapshot.start_time = match (snapshot.start_time, progress.start_time) {
(Some(current), Some(next)) => Some(current.min(next)),
(None, next) => next,
(current, None) => current,
};
snapshot.last_update_time = match (snapshot.last_update_time, progress.last_update_time) {
(Some(current), Some(next)) => Some(current.max(next)),
(None, next) => next,
(current, None) => current,
};
if progress.current_object.is_some() {
snapshot.current_object = progress.current_object;
}
}
snapshot.refresh_progress_percentage();
snapshot.refresh_estimated_completion_time();
Some(snapshot)
}
@@ -3208,12 +3460,17 @@ impl HealManager {
} else {
completed_task.get_status().await
};
let completed_progress = completed_task.get_progress().await;
let final_window = completed_task.get_result_items_since(None).await;
let completed_status_entry = CompletedHealStatus {
heal_type: completed_task.heal_type.clone(),
status: completed_status.clone(),
result_items: completed_task.get_result_items().await,
result_items: final_window.items.clone(),
result_items_truncated: completed_task.result_items_truncated(),
completed_at: SystemTime::now(),
seqed_items: completed_task.get_seqed_result_items().await,
next_seq: final_window.next_seq,
min_seq: final_window.min_seq,
};
let mut completed_heals_guard = completed_heals_clone.lock().await;
prune_completed_heal_statuses(&mut completed_heals_guard);
@@ -3223,6 +3480,7 @@ impl HealManager {
match completed_status {
HealTaskStatus::Completed => {
stats.update_task_completion(true);
stats.add_healed_objects(completed_progress.objects_healed, completed_progress.bytes_processed);
}
HealTaskStatus::Retrying { .. } => {}
_ => {
@@ -3749,6 +4007,7 @@ mod tests {
_bucket: &str,
_prefix: &str,
_continuation_token: Option<&str>,
_include_lifecycle_object_info: bool,
) -> Result<(Vec<crate::heal::storage::HealListItem>, Option<String>, bool)> {
Ok((Vec::new(), None, false))
}
@@ -4983,6 +5242,9 @@ mod tests {
},
result_items: Vec::new(),
result_items_truncated: false,
seqed_items: Vec::new(),
next_seq: 0,
min_seq: 0,
completed_at: SystemTime::now(),
},
);
@@ -5264,6 +5526,136 @@ mod tests {
assert_eq!(snapshot.queued_by_source.internal, 0);
}
// HS-06 (backlog#1870): overlap policy + forceStart semantics.
fn manager_with_policy(policy: HealOverlapPolicy) -> HealManager {
let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage);
HealManager::new(
storage,
Some(HealConfig {
overlap_policy: policy,
..Default::default()
}),
)
}
fn admin_prefix_request(bucket: &str, prefix: &str) -> HealRequest {
let mut request = HealRequest::new(
HealType::Prefix {
bucket: bucket.to_string(),
prefix: prefix.to_string(),
},
HealOptions::default(),
HealPriority::Normal,
);
request.source = HealRequestSource::Admin;
request
}
async fn insert_active_task(manager: &HealManager, request: HealRequest) -> String {
let task = Arc::new(HealTask::from_request(request, manager.storage.clone()));
let task_id = task.id.clone();
manager.active_heals.lock().await.insert(task_id.clone(), task);
task_id
}
#[tokio::test]
async fn overlap_policy_minio_error_rejects_same_and_containing_paths() {
let manager = manager_with_policy(HealOverlapPolicy::MinioError);
insert_active_task(&manager, admin_prefix_request("bucket-a", "logs/")).await;
// Same target: typed AlreadyRunning.
let same = manager
.submit_heal_request(admin_prefix_request("bucket-a", "logs/"))
.await
.expect("admission must decide");
assert_eq!(
same,
HealAdmissionResult::Dropped(HealAdmissionDropReason::AlreadyRunning),
"an identical target must reject with already-running"
);
// Contained path: typed OverlappingPaths.
let nested = manager
.submit_heal_request(admin_prefix_request("bucket-a", "logs/app/"))
.await
.expect("admission must decide");
assert_eq!(
nested,
HealAdmissionResult::Dropped(HealAdmissionDropReason::OverlappingPaths),
"a path inside the active task's path must reject with overlapping-paths"
);
// Containing path (bucket-wide vs nested active): also overlapping.
let wide = manager
.submit_heal_request(admin_prefix_request("bucket-a", ""))
.await
.expect("admission must decide");
assert_eq!(
wide,
HealAdmissionResult::Dropped(HealAdmissionDropReason::OverlappingPaths),
"a bucket-wide start overlapping a nested active heal must reject"
);
// Disjoint bucket: unaffected.
let disjoint = manager
.submit_heal_request(admin_prefix_request("bucket-b", "logs/"))
.await
.expect("admission must decide");
assert_eq!(disjoint, HealAdmissionResult::Accepted);
}
#[tokio::test]
async fn overlap_policy_default_merge_keeps_today_semantics() {
let manager = manager_with_policy(HealOverlapPolicy::Merge);
insert_active_task(&manager, admin_prefix_request("bucket-a", "logs/")).await;
// Different-dedup-key overlap still merges under the default policy:
// the nested path dedups to its own key but nothing rejects it.
let nested = manager
.submit_heal_request(admin_prefix_request("bucket-a", "logs/app/"))
.await
.expect("admission must decide");
assert_eq!(nested, HealAdmissionResult::Accepted, "default policy must not reject overlaps");
// Non-admin sources never get overlap rejections even under minio_error.
let manager = manager_with_policy(HealOverlapPolicy::MinioError);
insert_active_task(&manager, admin_prefix_request("bucket-a", "logs/")).await;
let mut scanner_request = admin_prefix_request("bucket-a", "logs/app/");
scanner_request.source = HealRequestSource::Scanner;
let admitted = manager
.submit_heal_request(scanner_request)
.await
.expect("admission must decide");
assert_eq!(admitted, HealAdmissionResult::Accepted, "scanner sources must never be overlap-rejected");
}
#[tokio::test]
async fn admin_force_start_cancels_overlapping_active_task_first() {
let manager = manager_with_policy(HealOverlapPolicy::Merge);
let old_id = insert_active_task(&manager, admin_prefix_request("bucket-a", "logs/")).await;
let mut replacement = admin_prefix_request("bucket-a", "logs/");
replacement.force_start = true;
let receipt = manager
.submit_heal_request_with_receipt(replacement)
.await
.expect("force-start submission must decide");
assert!(receipt.result.is_admitted(), "the new task must be admitted (Accepted or Merged)");
let old_task_gone = {
let active_heals = manager.active_heals.lock().await;
!active_heals.contains_key(&old_id)
};
assert!(
old_task_gone,
"the overlapping admin task must be cancelled (removed from the active table) before the new one starts"
);
assert!(
matches!(manager.get_task_status(&old_id).await, Err(Error::TaskNotFound { .. })),
"a cancelled task must no longer resolve as an active heal"
);
}
#[tokio::test]
async fn test_operations_snapshot_counts_active_by_source_and_priority() {
let storage: Arc<dyn HealStorageAPI> = Arc::new(MockStorage);
@@ -5396,6 +5788,8 @@ mod tests {
));
{
let mut progress = first.progress.write().await;
progress.start_time = Some(SystemTime::now() - Duration::from_secs(20));
progress.set_total_baseline(12, 8192);
progress.update_progress(7, 3, 1, 4096);
}
@@ -5405,6 +5799,8 @@ mod tests {
));
{
let mut progress = second.progress.write().await;
progress.start_time = Some(SystemTime::now() - Duration::from_secs(10));
progress.set_total_baseline(8, 4096);
progress.update_progress(11, 5, 2, 2048);
}
@@ -5419,7 +5815,11 @@ mod tests {
assert_eq!(progress.objects_scanned, 18);
assert_eq!(progress.objects_healed, 8);
assert_eq!(progress.objects_failed, 3);
assert_eq!(progress.objects_total_count, 20);
assert_eq!(progress.objects_total_size, 12288);
assert_eq!(progress.bytes_processed, 6144);
assert!((progress.progress_percentage - 50.0).abs() < 0.001);
assert!(progress.estimated_completion_time.is_some());
}
#[tokio::test]
@@ -5558,6 +5958,9 @@ mod tests {
status: HealTaskStatus::Completed,
result_items: Vec::new(),
result_items_truncated: false,
seqed_items: Vec::new(),
next_seq: 0,
min_seq: 0,
completed_at: SystemTime::now(),
},
);
@@ -5592,6 +5995,9 @@ mod tests {
..Default::default()
}],
result_items_truncated: true,
seqed_items: Vec::new(),
next_seq: 0,
min_seq: 0,
completed_at: SystemTime::now(),
},
);
+1
View File
@@ -16,6 +16,7 @@ pub mod channel;
pub mod erasure_healer;
pub mod event;
pub mod manager;
pub mod mrf_queue;
pub mod progress;
pub(crate) mod replacement_readiness;
pub mod resume;
+682
View File
@@ -0,0 +1,682 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Mission Repair Feed (MRF) queue, journal, and consumer.
//!
//! Intents arriving on the global channel (see `rustfs_common::mrf_channel`)
//! are buffered in a bounded in-memory queue, translated into prioritized
//! heal requests, and — while they are not yet accepted by the heal manager —
//! mirrored into a durable journal so a crash or restart can replay them.
//! This is the RustFS counterpart of MinIO's `.heal/mrf/list.bin` replay,
//! layered on top of (not replacing) read-repair and scanner heal.
//!
//! Durability model: the journal is a snapshot of the *unaccepted* pending
//! set, rewritten on a group-commit cadence (every flush interval or flush
//! threshold new intents). A rewrite is atomic at the record level only — a
//! torn tail simply truncates during replay because every record carries its
//! own CRC32. Losing the last flush window (≤500 ms) is acceptable: replayed
//! duplicates are merged by the manager's dedup key, and read-repair remains
//! the safety net.
use super::{DiskStore, HealDiskExt as _, local_disk_map_read};
use crate::heal::manager::HealManager;
use metrics::{counter, gauge};
use rustfs_common::heal_channel::{HealAdmissionDropReason, HealAdmissionResult};
use rustfs_common::mrf_channel::{MRF_MAX_ATTEMPTS, MrfIntent};
use std::collections::VecDeque;
use std::sync::Arc;
use std::time::Duration;
use tokio::sync::mpsc;
use uuid::Uuid;
use crate::heal::task::{HealOptions, HealPriority, HealRequest, HealType};
/// Journal location inside the metadata bucket, following the resume-state
/// layout.
pub(crate) const MRF_JOURNAL_PATH: &str = "buckets/.heal/mrf/journal.bin";
/// Record format tag.
const MRF_JOURNAL_FORMAT: u8 = 1;
/// Record layout version.
const MRF_JOURNAL_VERSION: u8 = 1;
/// Fixed header size: format, version, kind, attempts, enqueued_at_ms,
/// has_version flag.
const MRF_RECORD_FIXED_HEAD: usize = 1 + 1 + 1 + 1 + 8 + 1;
#[derive(Debug, Clone)]
pub(crate) struct MrfConsumerConfig {
/// In-memory queue capacity in intents.
pub queue_capacity: usize,
/// Journal byte budget; a pending snapshot above this bound is rejected
/// oldest-first so the journal can never grow unbounded.
pub journal_max_bytes: usize,
/// How many journal intents to re-arm per replay round.
pub replay_batch: usize,
/// Group-commit cadence for the journal snapshot.
pub flush_interval: Duration,
/// New intents between flushes that force an early snapshot.
pub flush_threshold: usize,
/// Backoff after the heal manager reports a full admission.
pub admission_backoff: Duration,
}
impl Default for MrfConsumerConfig {
fn default() -> Self {
Self {
queue_capacity: rustfs_utils::get_env_usize(
rustfs_config::ENV_HEAL_MRF_QUEUE_SIZE,
rustfs_config::DEFAULT_HEAL_MRF_QUEUE_SIZE,
),
journal_max_bytes: rustfs_utils::get_env_usize(
rustfs_config::ENV_HEAL_MRF_JOURNAL_MAX_BYTES,
rustfs_config::DEFAULT_HEAL_MRF_JOURNAL_MAX_BYTES,
),
replay_batch: rustfs_utils::get_env_usize(
rustfs_config::ENV_HEAL_MRF_REPLAY_BATCH,
rustfs_config::DEFAULT_HEAL_MRF_REPLAY_BATCH,
),
flush_interval: Duration::from_millis(500),
flush_threshold: 1000,
admission_backoff: Duration::from_secs(5),
}
}
}
/// Bounded pending set with count and byte ceilings. Overflow drops the
/// incoming intent (never a resident one) and counts the loss.
pub(crate) struct MrfQueue {
pending: VecDeque<MrfIntent>,
bytes: usize,
capacity: usize,
byte_budget: usize,
}
impl MrfQueue {
pub(crate) fn new(capacity: usize, byte_budget: usize) -> Self {
Self {
pending: VecDeque::new(),
bytes: 0,
capacity,
byte_budget,
}
}
/// Returns `false` (after counting) when either ceiling would be crossed.
pub(crate) fn try_push(&mut self, intent: MrfIntent) -> bool {
let cost = intent.estimated_bytes();
if self.pending.len() >= self.capacity || self.bytes + cost > self.byte_budget {
counter!("rustfs_heal_mrf_dropped_total", "reason" => "queue_overflow").increment(1);
return false;
}
self.bytes += cost;
self.pending.push_back(intent);
true
}
pub(crate) fn pop_front(&mut self) -> Option<MrfIntent> {
let intent = self.pending.pop_front()?;
self.bytes = self.bytes.saturating_sub(intent.estimated_bytes());
Some(intent)
}
pub(crate) fn push_back(&mut self, intent: MrfIntent) {
self.bytes += intent.estimated_bytes();
self.pending.push_back(intent);
}
pub(crate) fn depth(&self) -> usize {
self.pending.len()
}
pub(crate) fn bytes(&self) -> usize {
self.bytes
}
pub(crate) fn intents(&self) -> impl Iterator<Item = &MrfIntent> {
self.pending.iter()
}
}
// ---------------------------------------------------------------------------
// Journal record codec
// ---------------------------------------------------------------------------
/// Append one encoded record to `out`.
pub(crate) fn encode_intent(intent: &MrfIntent, out: &mut Vec<u8>) {
let start = out.len();
out.push(MRF_JOURNAL_FORMAT);
out.push(MRF_JOURNAL_VERSION);
out.push(match intent.kind {
rustfs_common::mrf_channel::MrfKind::DecodeFailure => 1,
rustfs_common::mrf_channel::MrfKind::MetadataCorruption => 2,
rustfs_common::mrf_channel::MrfKind::PartialWrite => 3,
});
out.push(intent.attempts);
out.extend_from_slice(&intent.enqueued_at_ms.to_le_bytes());
match intent.version_id {
Some(bytes) => {
out.push(1);
out.extend_from_slice(&bytes);
}
None => out.push(0),
}
out.extend_from_slice(&(intent.bucket.len() as u32).to_le_bytes());
out.extend_from_slice(&(intent.object.len() as u32).to_le_bytes());
out.extend_from_slice(intent.bucket.as_bytes());
out.extend_from_slice(intent.object.as_bytes());
let mut hasher = crc_fast::Digest::new(crc_fast::CrcAlgorithm::Crc32IsoHdlc);
hasher.update(&out[start..]);
out.extend_from_slice(&(hasher.finalize() as u32).to_le_bytes());
}
fn decode_one(data: &[u8]) -> Option<(MrfIntent, usize)> {
if data.len() < MRF_RECORD_FIXED_HEAD + 8 {
return None;
}
if data[0] != MRF_JOURNAL_FORMAT || data[1] != MRF_JOURNAL_VERSION {
return None;
}
let kind = match data[2] {
1 => rustfs_common::mrf_channel::MrfKind::DecodeFailure,
2 => rustfs_common::mrf_channel::MrfKind::MetadataCorruption,
3 => rustfs_common::mrf_channel::MrfKind::PartialWrite,
_ => return None,
};
let attempts = data[3];
let enqueued_at_ms = u64::from_le_bytes(data[4..12].try_into().expect("slice length checked"));
let has_version = data[12] != 0;
let mut cursor = MRF_RECORD_FIXED_HEAD;
let version_id = if has_version {
if data.len() < cursor + 16 {
return None;
}
let bytes: [u8; 16] = data[cursor..cursor + 16].try_into().expect("slice length checked");
cursor += 16;
Some(bytes)
} else {
None
};
if data.len() < cursor + 8 {
return None;
}
let bucket_len = u32::from_le_bytes(data[cursor..cursor + 4].try_into().expect("slice length checked")) as usize;
let object_len = u32::from_le_bytes(data[cursor + 4..cursor + 8].try_into().expect("slice length checked")) as usize;
cursor += 8;
let body_end = cursor.checked_add(bucket_len)?.checked_add(object_len)?;
let record_end = body_end.checked_add(4)?;
if data.len() < record_end {
return None;
}
let mut hasher = crc_fast::Digest::new(crc_fast::CrcAlgorithm::Crc32IsoHdlc);
hasher.update(&data[..body_end]);
if (hasher.finalize() as u32) != u32::from_le_bytes(data[body_end..record_end].try_into().expect("slice length checked")) {
return None;
}
let bucket = std::sync::Arc::from(std::str::from_utf8(&data[cursor..cursor + bucket_len]).ok()?);
let object = std::sync::Arc::from(std::str::from_utf8(&data[cursor + bucket_len..body_end]).ok()?);
Some((
MrfIntent {
bucket,
object,
version_id,
kind,
enqueued_at_ms,
attempts,
},
record_end,
))
}
/// Decode a whole journal, stopping at the first torn or corrupt record.
/// Returns the decoded intents and the number of trailing bytes discarded.
pub(crate) fn decode_journal(data: &[u8]) -> (Vec<MrfIntent>, usize) {
let mut intents = Vec::new();
let mut cursor = 0usize;
while cursor < data.len() {
match decode_one(&data[cursor..]) {
Some((intent, consumed)) => {
intents.push(intent);
cursor += consumed;
}
None => break,
}
}
let truncated = data.len() - cursor;
(intents, truncated)
}
// ---------------------------------------------------------------------------
// Journal disk IO (all local disks, first successful read wins)
// ---------------------------------------------------------------------------
async fn journal_disks() -> Vec<DiskStore> {
let map = local_disk_map_read().await;
map.values().flatten().cloned().collect()
}
async fn read_journal() -> Option<Vec<u8>> {
for disk in journal_disks().await {
match disk.read_all(super::RUSTFS_META_BUCKET, MRF_JOURNAL_PATH).await {
Ok(bytes) => return Some(bytes.to_vec()),
Err(_) => continue,
}
}
None
}
async fn write_journal(data: &[u8]) {
let payload = bytes::Bytes::copy_from_slice(data);
for disk in journal_disks().await {
if let Err(err) = disk
.write_all(super::RUSTFS_META_BUCKET, MRF_JOURNAL_PATH, payload.clone())
.await
{
warn_mrf_journal_write(&err);
}
}
if !data.is_empty() {
counter!("rustfs_heal_mrf_journal_fsync_total").increment(1);
}
gauge!("rustfs_heal_mrf_journal_bytes").set(data.len() as f64);
}
async fn delete_journal() {
for disk in journal_disks().await {
let _ = disk
.delete(
super::RUSTFS_META_BUCKET,
MRF_JOURNAL_PATH,
crate::heal::storage_api::owner::EcstoreDeleteOptions::default(),
)
.await;
}
}
fn warn_mrf_journal_write(err: &super::DiskError) {
tracing::warn!(
target: "rustfs::heal::mrf",
error = %err,
"MRF journal write failed; unconsumed intents may be lost on restart"
);
}
// ---------------------------------------------------------------------------
// Consumer
// ---------------------------------------------------------------------------
/// Translate an intent into the prioritized heal request the issue specifies:
/// decode failures go Urgent ECDecode, metadata corruption goes High
/// Metadata, partial writes go Normal object heal.
pub(crate) fn build_heal_request(intent: &MrfIntent) -> HealRequest {
let bucket = intent.bucket.to_string();
let object = intent.object.to_string();
let version_id = intent.version_id.map(|bytes| Uuid::from_bytes(bytes).to_string());
let (heal_type, priority) = match intent.kind {
rustfs_common::mrf_channel::MrfKind::DecodeFailure => (
HealType::ECDecode {
bucket,
object,
version_id,
},
HealPriority::Urgent,
),
rustfs_common::mrf_channel::MrfKind::MetadataCorruption => (HealType::Metadata { bucket, object }, HealPriority::High),
rustfs_common::mrf_channel::MrfKind::PartialWrite => (
HealType::Object {
bucket,
object,
version_id,
},
HealPriority::Normal,
),
};
let mut request = HealRequest::new(heal_type, HealOptions::default(), priority);
request.source = rustfs_common::heal_channel::HealRequestSource::Mrf;
request
}
struct MrfRuntime {
queue: MrfQueue,
config: MrfConsumerConfig,
new_since_flush: usize,
/// True while a journal snapshot exists on disk that no longer reflects
/// an all-consumed pending set; the next idle tick removes it (MinIO
/// deletes its `list.bin` after replay for the same reason).
journal_on_disk: bool,
/// Earliest instant a full-admission retry may proceed.
backoff_until: Option<tokio::time::Instant>,
}
impl MrfRuntime {
fn record_accept(&mut self) {
// Accepted intents leave the pending set; the next flush persists the
// smaller snapshot, which is the journal's compaction.
}
fn snapshot(&self) -> Vec<u8> {
let mut buf = Vec::new();
for intent in self.queue.intents() {
encode_intent(intent, &mut buf);
}
buf
}
async fn flush(&mut self) {
write_journal(&self.snapshot()).await;
self.new_since_flush = 0;
self.journal_on_disk = true;
}
/// Drain pending intents into the heal manager until it is full, the
/// queue empties, or attempts are exhausted.
async fn dispatch(&mut self, manager: &HealManager) {
if let Some(until) = self.backoff_until {
if tokio::time::Instant::now() < until {
return;
}
self.backoff_until = None;
}
while let Some(mut intent) = self.queue.pop_front() {
let request = build_heal_request(&intent);
match manager.submit_heal_request(request).await {
Ok(HealAdmissionResult::Accepted) | Ok(HealAdmissionResult::Merged) => self.record_accept(),
Ok(HealAdmissionResult::Full) | Ok(HealAdmissionResult::Dropped(HealAdmissionDropReason::QueueFull)) => {
intent.attempts = intent.attempts.saturating_add(1);
if intent.attempts >= MRF_MAX_ATTEMPTS {
counter!("rustfs_heal_mrf_dropped_total", "reason" => "attempts_exhausted").increment(1);
continue;
}
self.queue.push_back(intent);
self.backoff_until = Some(tokio::time::Instant::now() + self.config.admission_backoff);
break;
}
Ok(HealAdmissionResult::Dropped(_)) => {
counter!("rustfs_heal_mrf_dropped_total", "reason" => "admission_policy").increment(1);
}
Err(_) => {
intent.attempts = intent.attempts.saturating_add(1);
if intent.attempts >= MRF_MAX_ATTEMPTS {
counter!("rustfs_heal_mrf_dropped_total", "reason" => "attempts_exhausted").increment(1);
continue;
}
self.queue.push_back(intent);
self.backoff_until = Some(tokio::time::Instant::now() + self.config.admission_backoff);
break;
}
}
}
gauge!("rustfs_heal_mrf_queue_depth").set(self.queue.depth() as f64);
gauge!("rustfs_heal_mrf_queue_bytes").set(self.queue.bytes() as f64);
}
}
/// Initialize the global MRF channel (honoring `RUSTFS_HEAL_MRF_ENABLE`) and
/// spawn the consumer task. Called once from the heal runtime bootstrap right
/// after the manager started; a disabled feature or a double call is a no-op.
/// Public for integration tests that drive the real consumer loop.
pub fn spawn_mrf_consumer(manager: Arc<HealManager>) {
let enabled = rustfs_utils::get_env_bool(rustfs_config::ENV_HEAL_MRF_ENABLE, rustfs_config::DEFAULT_HEAL_MRF_ENABLE);
rustfs_common::mrf_channel::set_mrf_delivery_enabled(enabled);
if !enabled {
tracing::info!(
target: "rustfs::heal::mrf",
"MRF intent pipeline disabled by configuration; producers will not deliver"
);
return;
}
let receiver = match rustfs_common::mrf_channel::init_mrf_channel() {
Ok(receiver) => receiver,
Err(err) => {
tracing::warn!(
target: "rustfs::heal::mrf",
error = err,
"MRF channel initialization failed; intents will be dropped at producers"
);
return;
}
};
tokio::spawn(async move {
run_mrf_consumer(manager, receiver).await;
});
tracing::info!(target: "rustfs::heal::mrf", "MRF intent consumer started");
}
/// Replay the durable journal into a fresh pending queue and submit whatever
/// it armed. Returns the number of intact intents replayed. Duplicates are
/// merged by the manager's dedup key; the journal file is removed once read
/// (torn tails truncate via the per-record CRC). Public for integration tests;
/// the live consumer invokes this through [`replay_into`] at startup.
pub async fn replay_journal_once(manager: &Arc<HealManager>) -> usize {
let config = MrfConsumerConfig::default();
let mut queue = MrfQueue::new(config.queue_capacity, config.journal_max_bytes);
let mut backoff_until: Option<tokio::time::Instant> = None;
replay_into(manager, &mut queue, &mut backoff_until).await
}
/// Shared replay core: read + decode + re-arm + delete, then drain what fits.
async fn replay_into(
manager: &Arc<HealManager>,
queue: &mut MrfQueue,
backoff_until: &mut Option<tokio::time::Instant>,
) -> usize {
let Some(data) = read_journal().await else {
return 0;
};
let (intents, truncated) = decode_journal(&data);
if truncated > 0 {
tracing::warn!(
target: "rustfs::heal::mrf",
truncated_bytes = truncated,
"MRF journal had a torn tail; truncated records were discarded"
);
}
counter!("rustfs_heal_mrf_replayed_total").increment(intents.len() as u64);
let replayed = intents.len();
for intent in intents {
queue.try_push(intent);
}
delete_journal().await;
// Drain the replayed intents immediately; whatever the manager refuses
// stays armed in `queue` for the consumer's retry loop.
if backoff_until.is_none() {
while let Some(mut intent) = queue.pop_front() {
let request = build_heal_request(&intent);
match manager.submit_heal_request(request).await {
Ok(HealAdmissionResult::Accepted) | Ok(HealAdmissionResult::Merged) => {}
Ok(HealAdmissionResult::Full) | Ok(HealAdmissionResult::Dropped(HealAdmissionDropReason::QueueFull)) => {
intent.attempts = intent.attempts.saturating_add(1);
if intent.attempts < MRF_MAX_ATTEMPTS {
queue.push_back(intent);
*backoff_until = Some(tokio::time::Instant::now());
}
break;
}
Ok(HealAdmissionResult::Dropped(_)) | Err(_) => {}
}
}
}
replayed
}
/// Replay the journal, then keep draining the channel into the heal manager
/// while persisting the pending snapshot.
async fn run_mrf_consumer(manager: Arc<HealManager>, mut receiver: mpsc::Receiver<MrfIntent>) {
let config = MrfConsumerConfig::default();
let mut runtime = MrfRuntime {
queue: MrfQueue::new(config.queue_capacity, config.journal_max_bytes),
config: config.clone(),
new_since_flush: 0,
journal_on_disk: false,
backoff_until: None,
};
// Replay: read the journal, re-arm intents (duplicates are merged by the
// manager's dedup key), then drop the file so the next flush starts clean.
replay_into(&manager, &mut runtime.queue, &mut runtime.backoff_until).await;
let mut flush_tick = tokio::time::interval(runtime.config.flush_interval);
flush_tick.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
let mut batch: Vec<MrfIntent> = Vec::with_capacity(runtime.config.replay_batch);
loop {
tokio::select! {
received = receiver.recv_many(&mut batch, runtime.config.replay_batch) => {
if received == 0 {
// Channel closed: flush once more and stop.
runtime.flush().await;
tracing::info!(
target: "rustfs::heal::mrf",
"MRF channel closed; consumer stopped after final flush"
);
return;
}
for intent in batch.drain(..) {
runtime.queue.try_push(intent);
runtime.new_since_flush += 1;
}
runtime.dispatch(manager.as_ref()).await;
if runtime.new_since_flush >= runtime.config.flush_threshold {
runtime.flush().await;
}
}
_ = flush_tick.tick() => {
if runtime.new_since_flush > 0 || runtime.queue.depth() > 0 {
runtime.flush().await;
runtime.dispatch(manager.as_ref()).await;
} else if runtime.journal_on_disk {
// All intents consumed: remove the journal so a restart
// replays nothing (mirrors MinIO's post-replay unlink).
delete_journal().await;
runtime.journal_on_disk = false;
gauge!("rustfs_heal_mrf_journal_bytes").set(0.0);
}
gauge!("rustfs_heal_mrf_queue_depth").set(runtime.queue.depth() as f64);
}
}
}
}
#[cfg(test)]
mod tests {
use super::*;
use rustfs_common::mrf_channel::{MrfIntent, MrfKind};
use std::sync::Arc as StdArc;
fn intent(bucket: &str, object: &str, attempts: u8) -> MrfIntent {
MrfIntent {
bucket: StdArc::from(bucket),
object: StdArc::from(object),
version_id: Some([7u8; 16]),
kind: MrfKind::DecodeFailure,
enqueued_at_ms: 1_700_000_000_000,
attempts,
}
}
#[test]
fn queue_enforces_count_and_byte_ceilings() {
let mut queue = MrfQueue::new(2, usize::MAX);
assert!(queue.try_push(intent("b", "o", 0)));
assert!(queue.try_push(intent("b", "o", 0)));
assert!(!queue.try_push(intent("b", "o", 0)), "count ceiling must drop");
let mut tiny = MrfQueue::new(usize::MAX, intent("bucket", "object", 0).estimated_bytes());
assert!(tiny.try_push(intent("bucket", "object", 0)));
assert!(
!tiny.try_push(intent("bucket", "object", 0)),
"byte budget must drop before the second intent fits"
);
}
#[test]
fn journal_roundtrip_preserves_intents() {
let intents = vec![
intent("bucket-a", "object/a", 0),
intent("bucket-b", "object/b", 2),
MrfIntent {
bucket: StdArc::from("bucket-c"),
object: StdArc::from("object/c"),
version_id: None,
kind: MrfKind::MetadataCorruption,
enqueued_at_ms: 5,
attempts: 1,
},
];
let mut buf = Vec::new();
for intent in &intents {
encode_intent(intent, &mut buf);
}
let (decoded, truncated) = decode_journal(&buf);
assert_eq!(truncated, 0);
assert_eq!(decoded.len(), intents.len());
for (left, right) in decoded.iter().zip(intents.iter()) {
assert_eq!(left.bucket, right.bucket);
assert_eq!(left.object, right.object);
assert_eq!(left.version_id, right.version_id);
assert_eq!(left.kind, right.kind);
assert_eq!(left.attempts, right.attempts);
}
}
#[test]
fn journal_torn_tail_is_truncated() {
let mut buf = Vec::new();
encode_intent(&intent("b", "o", 0), &mut buf);
let mut torn = buf.clone();
torn.extend_from_slice(&buf[..buf.len() / 2]);
let (decoded, truncated) = decode_journal(&torn);
assert_eq!(decoded.len(), 1, "the intact record must survive");
assert!(truncated > 0, "the partial tail must be discarded");
// A corrupted body (CRC mismatch) also truncates from that record on.
let mut corrupt = buf.clone();
let mid = MRF_RECORD_FIXED_HEAD + 4;
corrupt[mid] ^= 0xff;
let (decoded, truncated) = decode_journal(&corrupt);
assert!(decoded.is_empty());
assert_eq!(truncated, corrupt.len());
}
#[test]
fn heal_request_mapping_follows_priority_matrix() {
let decode = build_heal_request(&intent("b", "o", 0));
assert!(matches!(decode.heal_type, HealType::ECDecode { .. }));
assert_eq!(decode.priority, HealPriority::Urgent);
let metadata = build_heal_request(&MrfIntent {
bucket: StdArc::from("b"),
object: StdArc::from("o"),
version_id: None,
kind: MrfKind::MetadataCorruption,
enqueued_at_ms: 0,
attempts: 0,
});
assert!(matches!(metadata.heal_type, HealType::Metadata { .. }));
assert_eq!(metadata.priority, HealPriority::High);
let partial = build_heal_request(&MrfIntent {
bucket: StdArc::from("b"),
object: StdArc::from("o"),
version_id: None,
kind: MrfKind::PartialWrite,
enqueued_at_ms: 0,
attempts: 0,
});
assert!(matches!(partial.heal_type, HealType::Object { .. }));
assert_eq!(partial.priority, HealPriority::Normal);
}
}
+160 -6
View File
@@ -13,7 +13,7 @@
// limitations under the License.
use serde::{Deserialize, Serialize};
use std::time::SystemTime;
use std::time::{Duration, SystemTime};
#[derive(Debug, Default, Clone, Serialize, Deserialize)]
#[serde(rename_all = "camelCase")]
@@ -24,6 +24,14 @@ pub struct HealProgress {
pub objects_healed: u64,
/// Objects failed
pub objects_failed: u64,
/// Versions skipped because they were written after this heal started
pub skipped_new_versions: u64,
/// Versions skipped because lifecycle already selected them for expiry
pub skipped_ilm_expired: u64,
/// Baseline object count from the latest complete usage snapshot
pub objects_total_count: u64,
/// Baseline object bytes from the latest complete usage snapshot
pub objects_total_size: u64,
/// Bytes processed
pub bytes_processed: u64,
/// Current object
@@ -54,10 +62,56 @@ impl HealProgress {
self.bytes_processed = bytes;
self.last_update_time = Some(SystemTime::now());
// calculate progress percentage
let total = scanned + healed + failed;
self.refresh_progress_percentage();
self.refresh_estimated_completion_time();
}
pub fn set_total_baseline(&mut self, objects_total_count: u64, objects_total_size: u64) {
self.objects_total_count = objects_total_count;
self.objects_total_size = objects_total_size;
self.last_update_time = Some(SystemTime::now());
self.refresh_progress_percentage();
self.refresh_estimated_completion_time();
}
pub fn record_skipped_new_version(&mut self) {
self.skipped_new_versions = self.skipped_new_versions.saturating_add(1);
self.last_update_time = Some(SystemTime::now());
self.refresh_progress_percentage();
self.refresh_estimated_completion_time();
}
pub fn record_skipped_ilm_expired(&mut self) {
self.skipped_ilm_expired = self.skipped_ilm_expired.saturating_add(1);
self.last_update_time = Some(SystemTime::now());
self.refresh_progress_percentage();
self.refresh_estimated_completion_time();
}
fn completed_for_baseline(&self) -> u64 {
self.objects_healed
.saturating_add(self.objects_failed)
.saturating_add(self.skipped_new_versions)
.saturating_add(self.skipped_ilm_expired)
}
pub(crate) fn refresh_progress_percentage(&mut self) {
if self.objects_total_size > 0 {
self.progress_percentage = ((self.bytes_processed as f64 / self.objects_total_size as f64) * 100.0).min(100.0);
return;
}
if self.objects_total_count > 0 {
let completed = self.completed_for_baseline();
self.progress_percentage = ((completed as f64 / self.objects_total_count as f64) * 100.0).min(100.0);
return;
}
let total = self
.objects_scanned
.saturating_add(self.objects_healed)
.saturating_add(self.objects_failed);
if total > 0 {
self.progress_percentage = (healed as f64 / total as f64) * 100.0;
self.progress_percentage = (self.objects_healed as f64 / total as f64) * 100.0;
}
}
@@ -66,9 +120,36 @@ impl HealProgress {
self.last_update_time = Some(SystemTime::now());
}
pub fn refresh_estimated_completion_time(&mut self) {
let Some(start_time) = self.start_time else {
self.estimated_completion_time = None;
return;
};
if self.is_completed() || !(0.0..100.0).contains(&self.progress_percentage) || self.bytes_processed == 0 {
self.estimated_completion_time = None;
return;
}
let elapsed = match SystemTime::now().duration_since(start_time) {
Ok(elapsed) if !elapsed.is_zero() => elapsed,
_ => {
self.estimated_completion_time = None;
return;
}
};
let estimated_total_secs = elapsed.as_secs_f64() * 100.0 / self.progress_percentage;
self.estimated_completion_time = start_time.checked_add(Duration::from_secs_f64(estimated_total_secs));
}
pub fn is_completed(&self) -> bool {
self.progress_percentage >= 100.0
|| self.objects_scanned > 0 && self.objects_healed + self.objects_failed >= self.objects_scanned
if self.progress_percentage >= 100.0 {
return true;
}
if self.objects_total_count > 0 || self.objects_total_size > 0 {
return false;
}
self.objects_scanned > 0 && self.objects_healed.saturating_add(self.objects_failed) >= self.objects_scanned
}
pub fn get_success_rate(&self) -> f64 {
@@ -158,6 +239,10 @@ mod tests {
assert_eq!(progress.objects_scanned, 0);
assert_eq!(progress.objects_healed, 0);
assert_eq!(progress.objects_failed, 0);
assert_eq!(progress.skipped_new_versions, 0);
assert_eq!(progress.skipped_ilm_expired, 0);
assert_eq!(progress.objects_total_count, 0);
assert_eq!(progress.objects_total_size, 0);
assert_eq!(progress.bytes_processed, 0);
assert_eq!(progress.progress_percentage, 0.0);
assert!(progress.start_time.is_some());
@@ -181,6 +266,73 @@ mod tests {
assert!(progress.last_update_time.is_some());
}
#[test]
fn test_heal_progress_estimates_completion_time_from_progress() {
let mut progress = HealProgress::new();
progress.start_time = Some(SystemTime::now() - Duration::from_secs(10));
progress.update_progress(100, 25, 0, 4096);
let eta = progress
.estimated_completion_time
.expect("partial byte progress should estimate completion");
assert!(eta > SystemTime::now());
}
#[test]
fn test_heal_progress_uses_byte_baseline_for_percentage() {
let mut progress = HealProgress::new();
progress.set_total_baseline(10, 8192);
progress.update_progress(100, 25, 0, 4096);
assert!((progress.progress_percentage - 50.0).abs() < 0.001);
}
#[test]
fn test_heal_progress_uses_object_baseline_when_bytes_unknown() {
let mut progress = HealProgress::new();
progress.set_total_baseline(10, 0);
progress.update_progress(100, 3, 2, 0);
assert!((progress.progress_percentage - 50.0).abs() < 0.001);
}
#[test]
fn test_heal_progress_counts_skipped_versions_for_object_baseline() {
let mut progress = HealProgress::new();
progress.set_total_baseline(10, 0);
progress.update_progress(100, 3, 2, 0);
progress.record_skipped_new_version();
assert_eq!(progress.skipped_new_versions, 1);
assert!((progress.progress_percentage - 60.0).abs() < 0.001);
}
#[test]
fn test_heal_progress_does_not_estimate_completion_without_bytes() {
let mut progress = HealProgress::new();
progress.start_time = Some(SystemTime::now() - Duration::from_secs(10));
progress.update_progress(100, 25, 0, 0);
assert!(progress.estimated_completion_time.is_none());
}
#[test]
fn test_heal_progress_with_baseline_is_not_completed_by_processed_count() {
let mut progress = HealProgress::new();
progress.start_time = Some(SystemTime::now() - Duration::from_secs(10));
progress.set_total_baseline(10, 8192);
progress.update_progress(1, 1, 0, 1024);
assert!(!progress.is_completed());
assert!(progress.estimated_completion_time.is_some());
}
#[test]
fn test_heal_progress_update_progress_zero_total() {
let mut progress = HealProgress::new();
@@ -251,6 +403,8 @@ mod tests {
assert_eq!(json["objectsScanned"], 10);
assert_eq!(json["objectsHealed"], 8);
assert_eq!(json["objectsFailed"], 2);
assert_eq!(json["skippedNewVersions"], 0);
assert_eq!(json["skippedIlmExpired"], 0);
assert_eq!(json["bytesProcessed"], 1024);
assert_eq!(json["currentObject"], "test-bucket/test-object");
assert!(json["progressPercentage"].is_number());
+169 -7
View File
@@ -22,6 +22,7 @@ use serde::{Deserialize, Serialize};
use std::sync::Arc;
use tracing::{debug, error, warn};
use super::storage_api::owner::{EcstoreHealLifecycleExpiryContext, ecstore_load_admin_data_usage_from_backend_cached};
use super::storage_api::storage::{
BucketInfo, BucketOperations, DiskSetSelector, HealOperations as _, ListOperations as _, ObjectIO as _,
ObjectOperations as _, StorageAdminApi,
@@ -29,6 +30,37 @@ use super::storage_api::storage::{
use super::{DiskStore, ECStore, Endpoint, HealDiskExt as _, StorageError, resume::ReplacementTargetIdentity};
pub use super::{HealObjectInfo, HealObjectOptions, HealPutObjReader};
#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
pub struct HealBucketUsageBaseline {
pub objects_count: u64,
pub bytes: u64,
}
pub struct HealLifecycleExpiryContext {
inner: HealLifecycleExpiryContextInner,
}
enum HealLifecycleExpiryContextInner {
Ecstore(EcstoreHealLifecycleExpiryContext),
#[allow(dead_code)]
Test,
}
impl HealLifecycleExpiryContext {
fn ecstore(inner: EcstoreHealLifecycleExpiryContext) -> Self {
Self {
inner: HealLifecycleExpiryContextInner::Ecstore(inner),
}
}
#[cfg(test)]
pub(crate) fn test() -> Self {
Self {
inner: HealLifecycleExpiryContextInner::Test,
}
}
}
const LOG_COMPONENT_HEAL: &str = "heal";
const LOG_SUBSYSTEM_STORAGE: &str = "storage";
const EVENT_HEAL_STORAGE_OBJECT_IO: &str = "heal_storage_object_io";
@@ -272,6 +304,10 @@ pub struct HealListItem {
pub name: String,
/// normalized version id (`None` when the version is nil/absent)
pub version_id: Option<String>,
/// version modification time as Unix nanoseconds
pub mod_time_unix_nanos: Option<i128>,
/// object snapshot for lifecycle evaluation
pub lifecycle_object_info: Option<HealObjectInfo>,
/// whether this version is a delete marker (observability only)
pub is_delete_marker: bool,
}
@@ -329,6 +365,28 @@ pub trait HealStorageAPI: Send + Sync {
/// Get bucket info
async fn get_bucket_info(&self, bucket: &str) -> Result<Option<BucketInfo>>;
/// Aggregate usage-cache baselines for the requested buckets.
async fn erasure_set_usage_baseline(&self, _buckets: &[String]) -> Result<Option<HealBucketUsageBaseline>> {
Ok(None)
}
/// Load per-bucket lifecycle expiry context for heal skips.
async fn load_heal_lifecycle_expiry_context(&self, _bucket: &str) -> Result<Option<HealLifecycleExpiryContext>> {
Ok(None)
}
/// Queue lifecycle expiry for a version that heal can skip.
async fn enqueue_heal_lifecycle_expiry(
&self,
_context: &HealLifecycleExpiryContext,
_bucket: &str,
_object: &str,
_version_id: Option<&str>,
_object_info: Option<&HealObjectInfo>,
) -> Result<bool> {
Ok(false)
}
/// Fix bucket metadata
async fn heal_bucket_metadata(&self, bucket: &str) -> Result<()>;
@@ -409,6 +467,7 @@ pub trait HealStorageAPI: Send + Sync {
bucket: &str,
prefix: &str,
continuation_token: Option<&str>,
include_lifecycle_object_info: bool,
) -> Result<(Vec<HealListItem>, Option<String>, bool)>;
/// List versions for healing via a per-erasure-set DISK-WALK union enumerator
@@ -427,8 +486,10 @@ pub trait HealStorageAPI: Send + Sync {
bucket: &str,
prefix: &str,
continuation_token: Option<&str>,
include_lifecycle_object_info: bool,
) -> Result<(Vec<HealListItem>, Option<String>, bool)> {
self.list_objects_for_heal_page(bucket, prefix, continuation_token).await
self.list_objects_for_heal_page(bucket, prefix, continuation_token, include_lifecycle_object_info)
.await
}
/// Get disk for resume functionality.
@@ -1021,6 +1082,85 @@ impl HealStorageAPI for ECStoreHealStorage {
}
}
async fn erasure_set_usage_baseline(&self, buckets: &[String]) -> Result<Option<HealBucketUsageBaseline>> {
if buckets.is_empty() {
return Ok(None);
}
let info = match ecstore_load_admin_data_usage_from_backend_cached(self.ecstore.clone()).await {
Ok(info) if info.is_complete_bucket_usage_snapshot() => info,
Ok(_) | Err(_) => return Ok(None),
};
let mut baseline = HealBucketUsageBaseline::default();
for bucket in buckets {
if let Some(usage) = info.buckets_usage.get(bucket) {
baseline.objects_count = baseline.objects_count.saturating_add(usage.objects_count);
baseline.bytes = baseline.bytes.saturating_add(usage.size);
}
}
Ok(Some(baseline))
}
async fn load_heal_lifecycle_expiry_context(&self, bucket: &str) -> Result<Option<HealLifecycleExpiryContext>> {
match self.ecstore.load_heal_lifecycle_expiry_context(bucket).await {
Ok(Some(context)) => Ok(Some(HealLifecycleExpiryContext::ecstore(context))),
Ok(None) => Ok(None),
Err(err) => {
debug!(
target: "rustfs::heal::storage",
event = EVENT_HEAL_STORAGE_ADMIN_OP,
component = LOG_COMPONENT_HEAL,
subsystem = LOG_SUBSYSTEM_STORAGE,
operation = "load_heal_lifecycle_expiry_context",
bucket,
result = "failed",
error = %err,
"Heal storage lifecycle expiry context load failed"
);
Ok(None)
}
}
}
async fn enqueue_heal_lifecycle_expiry(
&self,
context: &HealLifecycleExpiryContext,
bucket: &str,
object: &str,
version_id: Option<&str>,
object_info: Option<&HealObjectInfo>,
) -> Result<bool> {
let context = match &context.inner {
HealLifecycleExpiryContextInner::Ecstore(context) => context,
HealLifecycleExpiryContextInner::Test => return Ok(false),
};
match self
.ecstore
.enqueue_heal_lifecycle_expiry(context, bucket, object, version_id, object_info)
.await
{
Ok(queued) => Ok(queued),
Err(err) => {
debug!(
target: "rustfs::heal::storage",
event = EVENT_HEAL_STORAGE_ADMIN_OP,
component = LOG_COMPONENT_HEAL,
subsystem = LOG_SUBSYSTEM_STORAGE,
operation = "enqueue_heal_lifecycle_expiry",
bucket,
object,
version_id = ?version_id,
result = "failed",
error = %err,
"Heal storage lifecycle expiry check failed"
);
Ok(false)
}
}
}
async fn heal_bucket_metadata(&self, bucket: &str) -> Result<()> {
debug!(
target: "rustfs::heal::storage",
@@ -1436,7 +1576,7 @@ impl HealStorageAPI for ECStoreHealStorage {
loop {
let (page_objects, next_token, is_truncated) = self
.list_objects_for_heal_page(bucket, prefix, continuation_token.as_deref())
.list_objects_for_heal_page(bucket, prefix, continuation_token.as_deref(), false)
.await?;
all_objects.extend(page_objects);
@@ -1471,6 +1611,7 @@ impl HealStorageAPI for ECStoreHealStorage {
bucket: &str,
prefix: &str,
continuation_token: Option<&str>,
include_lifecycle_object_info: bool,
) -> Result<(Vec<HealListItem>, Option<String>, bool)> {
debug!(
target: "rustfs::heal::storage",
@@ -1522,10 +1663,19 @@ impl HealStorageAPI for ECStoreHealStorage {
let page_objects: Vec<HealListItem> = list_info
.objects
.into_iter()
.map(|obj| HealListItem {
name: obj.name,
version_id: obj.version_id.filter(|u| !u.is_nil()).map(|u| u.to_string()),
is_delete_marker: obj.delete_marker,
.map(|mut obj| {
obj.version_id = obj.version_id.filter(|u| !u.is_nil());
let version_id = obj.version_id.map(|u| u.to_string());
let mod_time_unix_nanos = obj.mod_time.map(|mod_time| mod_time.unix_timestamp_nanos());
let is_delete_marker = obj.delete_marker;
let lifecycle_object_info = include_lifecycle_object_info.then(|| obj.clone());
HealListItem {
name: obj.name,
version_id,
mod_time_unix_nanos,
lifecycle_object_info,
is_delete_marker,
}
})
.collect();
let page_count = page_objects.len();
@@ -1562,6 +1712,7 @@ impl HealStorageAPI for ECStoreHealStorage {
bucket: &str,
prefix: &str,
continuation_token: Option<&str>,
include_lifecycle_object_info: bool,
) -> Result<(Vec<HealListItem>, Option<String>, bool)> {
// Per-page bounds for the disk-walk union enumerator. Objects are atomic
// (never split across pages), so version_budget only bounds how many
@@ -1590,7 +1741,16 @@ impl HealStorageAPI for ECStoreHealStorage {
let (versions, next_forward, is_truncated) = self
.ecstore
.heal_walk_versions_page(pool_idx, set_idx, bucket, prefix, forward_to.as_deref(), BATCH_OBJECTS, VERSION_BUDGET)
.heal_walk_versions_page(
pool_idx,
set_idx,
bucket,
prefix,
forward_to.as_deref(),
BATCH_OBJECTS,
VERSION_BUDGET,
include_lifecycle_object_info,
)
.await
.map_err(|e| {
error!(
@@ -1614,6 +1774,8 @@ impl HealStorageAPI for ECStoreHealStorage {
.map(|v| HealListItem {
name: v.name,
version_id: v.version_id,
mod_time_unix_nanos: v.mod_time_unix_nanos,
lifecycle_object_info: v.lifecycle_object_info,
is_delete_marker: v.is_delete_marker,
})
.collect();
+9 -4
View File
@@ -12,7 +12,10 @@
// See the License for the specific language governing permissions and
// limitations under the License.
pub(crate) use rustfs_ecstore::api::data_usage::DATA_USAGE_CACHE_NAME as ECSTORE_DATA_USAGE_CACHE_NAME;
pub(crate) use rustfs_ecstore::api::data_usage::{
DATA_USAGE_CACHE_NAME as ECSTORE_DATA_USAGE_CACHE_NAME,
load_admin_data_usage_from_backend_cached as ecstore_load_admin_data_usage_from_backend_cached,
};
pub(crate) use rustfs_ecstore::api::disk::endpoint::Endpoint as EcstoreEndpoint;
pub(crate) use rustfs_ecstore::api::disk::error::{DiskError as EcstoreDiskError, Result as EcstoreDiskResult};
pub(crate) use rustfs_ecstore::api::disk::{
@@ -25,7 +28,9 @@ pub(crate) use rustfs_ecstore::api::disk::{
pub(crate) use rustfs_ecstore::api::disk::{DiskOption as EcstoreDiskOption, new_disk as ecstore_new_disk};
pub(crate) use rustfs_ecstore::api::error::{Error as EcstoreErrorType, StorageError as EcstoreStorageError};
pub(crate) use rustfs_ecstore::api::runtime::local_disk_map_read as ecstore_local_disk_map_read;
pub(crate) use rustfs_ecstore::api::storage::ECStore as EcstoreStore;
pub(crate) use rustfs_ecstore::api::storage::{
ECStore as EcstoreStore, HealLifecycleExpiryContext as EcstoreHealLifecycleExpiryContext,
};
use rustfs_storage_api as storage_contracts;
pub(crate) mod owner {
@@ -34,8 +39,8 @@ pub(crate) mod owner {
pub(crate) use super::{
ECSTORE_BUCKET_META_PREFIX, ECSTORE_DATA_USAGE_CACHE_NAME, ECSTORE_HEALING_MARKER_PATH, ECSTORE_RUSTFS_META_BUCKET,
EcstoreConditionalFileUpdate, EcstoreDeleteOptions, EcstoreDiskAPI, EcstoreDiskBytes, EcstoreDiskError,
EcstoreDiskResult, EcstoreDiskStore, EcstoreEndpoint, EcstoreErrorType, EcstoreStorageError, EcstoreStore,
ecstore_local_disk_map_read,
EcstoreDiskResult, EcstoreDiskStore, EcstoreEndpoint, EcstoreErrorType, EcstoreHealLifecycleExpiryContext,
EcstoreStorageError, EcstoreStore, ecstore_load_admin_data_usage_from_backend_cached, ecstore_local_disk_map_read,
};
#[cfg(test)]
+371 -7
View File
@@ -19,11 +19,12 @@ use crate::heal::{
resume::{
CheckpointManager, ReplacementPhase, ReplacementTargetIdentity, ResumeManager, replacement_target_identities_match,
},
storage::{HealStorageAPI, next_heal_listing_token},
storage::{HealBucketUsageBaseline, HealStorageAPI, next_heal_listing_token},
};
use crate::{Error, Result};
use metrics::{counter, histogram};
use rustfs_common::heal_channel::{HealOpts, HealRequestSource, HealScanMode};
use rustfs_common::trace_bus::{TraceEvent, TraceFunc, TraceKind, trace_emit};
use rustfs_madmin::heal_commands::HealResultItem;
use rustfs_utils::path::SLASH_SEPARATOR;
use serde::{Deserialize, Serialize};
@@ -31,7 +32,7 @@ use std::{
future::Future,
sync::{
Arc,
atomic::{AtomicBool, Ordering},
atomic::{AtomicBool, AtomicU64, Ordering},
},
time::{Duration, Instant, SystemTime},
};
@@ -178,6 +179,17 @@ pub enum HealPriority {
Urgent = 3,
}
impl HealPriority {
fn as_str(self) -> &'static str {
match self {
Self::Low => "low",
Self::Normal => "normal",
Self::High => "high",
Self::Urgent => "urgent",
}
}
}
/// Heal options
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct HealOptions {
@@ -339,6 +351,20 @@ impl HealRequest {
}
/// Heal task
/// Incremental view over a task's retained result items (HS-06).
///
/// `next_seq` is the cursor a client should pass on its next poll; `min_seq`
/// is the oldest sequence still retained; `lagged` means the client's cursor
/// fell behind `min_seq` and items were skipped — the client should restart
/// from `min_seq`.
#[derive(Debug, Clone)]
pub struct HealResultWindow {
pub items: Vec<HealResultItem>,
pub next_seq: u64,
pub min_seq: u64,
pub lagged: bool,
}
pub struct HealTask {
/// Task ID
pub id: String,
@@ -361,8 +387,16 @@ pub struct HealTask {
pub status: Arc<RwLock<HealTaskStatus>>,
/// Progress tracking
pub progress: Arc<RwLock<HealProgress>>,
/// Result items collected from storage heal calls.
pub result_items: Arc<RwLock<Vec<HealResultItem>>>,
/// Result items collected from storage heal calls, each stamped with a
/// monotonically increasing sequence number for incremental consumption
/// (the client passes the last seen seq back and receives only newer
/// items; see `get_result_items_since`).
pub result_items: Arc<RwLock<Vec<(u64, HealResultItem)>>>,
/// Next sequence number to assign; starts at 1.
next_item_seq: Arc<AtomicU64>,
/// Sequence number of the oldest item still inside the retention window;
/// equals `next_item_seq` while the window is empty.
min_available_seq: Arc<AtomicU64>,
result_items_truncated: Arc<AtomicBool>,
batch_failure: Arc<RwLock<Option<BatchHealFailure>>>,
batch_failure_recorded: Arc<AtomicBool>,
@@ -414,6 +448,8 @@ impl HealTask {
status: Arc::new(RwLock::new(HealTaskStatus::Pending)),
progress: Arc::new(RwLock::new(HealProgress::new())),
result_items: Arc::new(RwLock::new(Vec::new())),
next_item_seq: Arc::new(AtomicU64::new(1)),
min_available_seq: Arc::new(AtomicU64::new(1)),
result_items_truncated: Arc::new(AtomicBool::new(false)),
batch_failure: Arc::new(RwLock::new(None)),
batch_failure_recorded: Arc::new(AtomicBool::new(false)),
@@ -498,6 +534,61 @@ impl HealTask {
}
}
fn emit_trace_task_state(&self, state: &'static str, duration: Duration, error: Option<&Error>) {
trace_emit(|| {
let mut event = TraceEvent::new(TraceKind::Heal, TraceFunc::HealTask)
.with_duration(duration)
.with_attr("task_id", self.id.as_str())
.with_attr("heal_type", self.heal_type.log_kind())
.with_attr("state", state)
.with_attr("source", self.source.as_str())
.with_attr("priority", self.priority.as_str())
.with_attr("retry_attempts", u64::from(self.retry_attempts))
.with_attr("dry_run", self.options.dry_run);
event = match &self.heal_type {
HealType::Cluster => event,
HealType::Object {
bucket,
object,
version_id,
} => {
let event = event.with_bucket(bucket.as_str()).with_object(object.as_str());
match version_id {
Some(version_id) => event.with_attr("version_id", version_id.as_str()),
None => event,
}
}
HealType::Bucket { bucket } => event.with_bucket(bucket.as_str()),
HealType::Prefix { bucket, prefix } => event.with_bucket(bucket.as_str()).with_object(prefix.as_str()),
HealType::ErasureSet { buckets, set_disk_id } => {
let bucket_count = u64::try_from(buckets.len()).unwrap_or(u64::MAX);
event
.with_attr("set_disk_id", set_disk_id.as_str())
.with_attr("bucket_count", bucket_count)
}
HealType::Metadata { bucket, object } => event.with_bucket(bucket.as_str()).with_object(object.as_str()),
HealType::ECDecode {
bucket,
object,
version_id,
} => {
let event = event.with_bucket(bucket.as_str()).with_object(object.as_str());
match version_id {
Some(version_id) => event.with_attr("version_id", version_id.as_str()),
None => event,
}
}
HealType::MRF { meta_path } => event.with_object(meta_path.as_str()),
};
match error {
Some(error) => event.with_attr("error", error.to_string()),
None => event,
}
});
}
async fn remaining_timeout(&self) -> Result<Option<Duration>> {
if let Some(total) = self.options.timeout {
let start_instant = { *self.task_start_instant.read().await };
@@ -717,6 +808,7 @@ impl HealTask {
queue_delay = ?queue_delay,
"Heal task started"
});
self.emit_trace_task_state("started", Duration::ZERO, None);
let result = match &self.heal_type {
HealType::Cluster => self.heal_cluster().await,
@@ -805,6 +897,14 @@ impl HealTask {
}
}
let terminal_state = match &result {
Ok(_) => "completed",
Err(Error::TaskCancelled) => "cancelled",
Err(Error::TaskTimeout) => "timed_out",
Err(_) => "failed",
};
self.emit_trace_task_state(terminal_state, start_instant.elapsed(), result.as_ref().err());
result
}
@@ -835,18 +935,63 @@ impl HealTask {
}
pub async fn get_result_items(&self) -> Vec<HealResultItem> {
self.result_items.read().await.iter().map(|(_, item)| item.clone()).collect()
}
/// Sequence-stamped retained window, used when archiving a completed
/// task so incremental cursors survive the transition (HS-06).
pub async fn get_seqed_result_items(&self) -> Vec<(u64, HealResultItem)> {
self.result_items.read().await.clone()
}
/// Incremental result window (HS-06): `since = None` returns the full
/// retained window (legacy snapshot semantics); `since = Some(seq)`
/// returns only items stamped with a sequence greater than `seq`.
/// `lagged` warns that the caller's cursor fell behind the window start
/// and items were skipped (the response carries `min_seq` as the catch-up
/// cursor).
pub async fn get_result_items_since(&self, since: Option<u64>) -> HealResultWindow {
let result_items = self.result_items.read().await;
let next_seq = self.next_item_seq.load(Ordering::Relaxed);
let min_seq = self.min_available_seq.load(Ordering::Relaxed);
let mut lagged = false;
let items = match since {
None => result_items.iter().map(|(_, item)| item.clone()).collect::<Vec<_>>(),
Some(cursor) => {
if cursor + 1 < min_seq {
lagged = true;
}
result_items
.iter()
.filter(|(seq, _)| *seq > cursor)
.map(|(_, item)| item.clone())
.collect::<Vec<_>>()
}
};
HealResultWindow {
items,
next_seq,
min_seq,
lagged,
}
}
pub fn result_items_truncated(&self) -> bool {
self.result_items_truncated.load(Ordering::Relaxed)
}
async fn record_result_item(&self, result: HealResultItem) {
let seq = self.next_item_seq.fetch_add(1, Ordering::Relaxed);
let mut result_items = self.result_items.write().await;
if result_items.len() < MAX_RETAINED_HEAL_RESULT_ITEMS {
result_items.push(result);
result_items.push((seq, result));
} else {
// Slide the window: the oldest item leaves and the cursor for the
// oldest still-available item moves forward with it.
result_items.remove(0);
self.min_available_seq
.store(result_items.first().map_or(seq, |(oldest, _)| *oldest), Ordering::Relaxed);
result_items.push((seq, result));
self.result_items_truncated.store(true, Ordering::Relaxed);
}
}
@@ -1535,7 +1680,7 @@ impl HealTask {
let (objects, next_token, is_truncated) = self
.await_with_control(
self.storage
.list_objects_for_heal_page(bucket, prefix, continuation_token.as_deref()),
.list_objects_for_heal_page(bucket, prefix, continuation_token.as_deref(), false),
)
.await?;
@@ -1697,6 +1842,23 @@ impl HealTask {
Ok(())
}
async fn apply_erasure_set_usage_baseline(&self, buckets: &[String]) -> Result<()> {
let baseline = match self
.await_with_control(self.storage.erasure_set_usage_baseline(buckets))
.await
{
Ok(Some(baseline)) => baseline,
Ok(None) => return Ok(()),
Err(err @ Error::TaskCancelled) | Err(err @ Error::TaskTimeout) => return Err(err),
Err(_) => return Ok(()),
};
let HealBucketUsageBaseline { objects_count, bytes } = baseline;
let mut progress = self.progress.write().await;
progress.set_total_baseline(objects_count, bytes);
Ok(())
}
async fn heal_metadata(&self, bucket: &str, object: &str) -> Result<()> {
debug!(
target: "rustfs::heal::task",
@@ -2298,6 +2460,8 @@ impl HealTask {
None
};
self.apply_erasure_set_usage_baseline(&buckets).await?;
let healing_marker = format!("{set_disk_id}:{}", self.id);
if let Some((disk, resume_manager, _)) = replacement_resume.as_ref() {
let state = resume_manager.get_state().await;
@@ -2602,7 +2766,8 @@ impl HealTask {
{
let mut progress = self.progress.write().await;
progress.update_progress(4, 4, 0, 0);
let bytes_processed = progress.bytes_processed;
progress.update_progress(4, 4, 0, bytes_processed);
}
match result {
@@ -2658,6 +2823,7 @@ mod tests {
use super::super::{DiskOption, DiskStore, Endpoint, HealDiskExt as _, new_disk};
use super::*;
use crate::heal::storage::{DiskStatus, HealListItem, HealObjectInfo};
use rustfs_common::trace_bus::{TraceEvent, TraceFunc, TraceKind, TraceSubscription, TraceVal, subscribe_trace_events};
use rustfs_madmin::heal_commands::{HealDriveInfo, HealResultItem, Infos};
use std::collections::{HashMap, VecDeque};
use std::sync::Mutex;
@@ -3203,6 +3369,8 @@ mod tests {
block_heal_object: Mutex<bool>,
resume_disk: Mutex<Option<DiskStore>>,
replacement_resume_disk: Mutex<Option<DiskStore>>,
usage_baseline: Mutex<Option<HealBucketUsageBaseline>>,
usage_baseline_error: Mutex<bool>,
}
#[test]
@@ -3265,11 +3433,69 @@ mod tests {
assert_eq!(samples_logged, MAX_BUCKET_FAILURE_LOG_SAMPLES);
}
#[tokio::test]
async fn execute_emits_heal_trace_task_state() {
let mut trace = subscribe_trace_events();
let storage = Arc::new(MockStorage::default());
let task = HealTask::from_request(
HealRequest::object("bucket-a".to_string(), "object-a".to_string(), Some("version-a".to_string())),
storage,
);
task.execute().await.expect("mock object heal should complete");
let started = recv_trace_task_state(&mut trace, &task.id, "started").await;
assert_eq!(started.kind, TraceKind::Heal);
assert_eq!(started.func, TraceFunc::HealTask);
assert_eq!(started.bucket.as_deref(), Some("bucket-a"));
assert_eq!(started.object.as_deref(), Some("object-a"));
assert_eq!(trace_attr_string(&started, "heal_type").as_deref(), Some("object"));
assert_eq!(trace_attr_string(&started, "source").as_deref(), Some("internal"));
assert_eq!(trace_attr_string(&started, "version_id").as_deref(), Some("version-a"));
let completed = recv_trace_task_state(&mut trace, &task.id, "completed").await;
assert_eq!(completed.kind, TraceKind::Heal);
assert_eq!(completed.func, TraceFunc::HealTask);
assert_eq!(trace_attr_string(&completed, "state").as_deref(), Some("completed"));
}
async fn recv_trace_task_state(trace: &mut TraceSubscription, task_id: &str, state: &str) -> TraceEvent {
for _ in 0..32 {
let event = tokio::time::timeout(Duration::from_secs(1), trace.recv())
.await
.expect("trace event should arrive")
.expect("trace bus should stay open");
if trace_attr_string(&event, "task_id").as_deref() == Some(task_id)
&& trace_attr_string(&event, "state").as_deref() == Some(state)
{
return (*event).clone();
}
}
panic!("expected trace state {state} for task {task_id}");
}
fn trace_attr_string(event: &TraceEvent, key: &str) -> Option<String> {
event.attrs.iter().find_map(|attr| {
if attr.key != key {
return None;
}
Some(match &attr.value {
TraceVal::Bool(value) => value.to_string(),
TraceVal::U64(value) => value.to_string(),
TraceVal::I64(value) => value.to_string(),
TraceVal::Str(value) => value.to_string(),
})
})
}
/// Build a latest, non-delete-marker heal list item with no version id.
fn heal_item(name: &str) -> HealListItem {
HealListItem {
name: name.to_string(),
version_id: None,
mod_time_unix_nanos: None,
lifecycle_object_info: None,
is_delete_marker: false,
}
}
@@ -3357,6 +3583,13 @@ mod tests {
}))
}
async fn erasure_set_usage_baseline(&self, _buckets: &[String]) -> Result<Option<HealBucketUsageBaseline>> {
if *self.usage_baseline_error.lock().unwrap() {
return Err(Error::Other("usage baseline unavailable".to_string()));
}
Ok(*self.usage_baseline.lock().unwrap())
}
async fn heal_bucket_metadata(&self, _bucket: &str) -> Result<()> {
Ok(())
}
@@ -3540,6 +3773,7 @@ mod tests {
bucket: &str,
prefix: &str,
continuation_token: Option<&str>,
_include_lifecycle_object_info: bool,
) -> Result<(Vec<HealListItem>, Option<String>, bool)> {
self.listed_prefixes.lock().unwrap().push(prefix.to_string());
if *self.truncate_without_token.lock().unwrap() {
@@ -3715,6 +3949,69 @@ mod tests {
assert!(task.result_items_truncated());
}
// HS-06 (backlog#1870): incremental result windows.
#[tokio::test]
async fn result_items_seq_is_monotonic_and_incremental_slices_work() {
let storage = Arc::new(MockStorage::default());
let task = HealTask::from_request(HealRequest::bucket("bucket-a".to_string()), storage);
for round in 0..5u64 {
let item = HealResultItem {
object_size: round as usize,
..Default::default()
};
task.record_result_item(item).await;
}
let full = task.get_result_items_since(None).await;
assert_eq!(full.items.len(), 5, "None keeps the full-snapshot semantics");
assert_eq!(full.next_seq, 6, "next_seq is one past the last assigned");
assert_eq!(full.min_seq, 1, "nothing was evicted yet");
assert!(!full.lagged);
// Incremental: only items newer than the cursor.
let incremental = task.get_result_items_since(Some(3)).await;
assert_eq!(
incremental.items.iter().map(|item| item.object_size).collect::<Vec<_>>(),
vec![3, 4],
"only sequences greater than the cursor are returned"
);
assert_eq!(incremental.next_seq, 6);
// A cursor at the head is not lagging.
assert!(!task.get_result_items_since(Some(0)).await.lagged);
}
#[tokio::test]
async fn result_items_window_slide_moves_min_seq_and_flags_lagging_cursors() {
let storage = Arc::new(MockStorage::default());
let task = HealTask::from_request(HealRequest::bucket("bucket-a".to_string()), storage);
// Fill the window completely, then push two more items: seq 1 and 2
// are evicted by the slide.
for _ in 0..(MAX_RETAINED_HEAL_RESULT_ITEMS + 2) {
task.record_result_item(HealResultItem::default()).await;
}
let full = task.get_result_items_since(None).await;
assert_eq!(full.items.len(), MAX_RETAINED_HEAL_RESULT_ITEMS);
assert_eq!(full.min_seq, 3, "each evicted head item moved the oldest-available cursor");
assert!(task.result_items_truncated());
// A client still polling from before the eviction is lagging.
let lagging = task.get_result_items_since(Some(0)).await;
assert!(lagging.lagged, "a cursor behind min_seq must be flagged");
assert_eq!(lagging.min_seq, 3, "the response tells the client where to restart");
// A cursor inside the window is fine.
assert!(!task.get_result_items_since(Some(3)).await.lagged);
// The lagging client restarts from min_seq and gets the full window.
let catch_up = task.get_result_items_since(Some(3)).await;
assert_eq!(catch_up.items.len(), MAX_RETAINED_HEAL_RESULT_ITEMS - 1);
assert!(!catch_up.lagged);
}
#[tokio::test]
async fn test_recursive_bucket_heal_skips_object_dir_candidates() {
let storage = Arc::new(MockStorage {
@@ -4654,6 +4951,73 @@ mod tests {
assert!(storage.object_heal_opts.lock().unwrap().is_empty());
}
#[tokio::test]
async fn erasure_set_heal_applies_usage_baseline_to_progress() {
let temp = TempDir::new().expect("temporary directory should be created");
let disk = make_resume_disk(&temp).await;
let storage = Arc::new(MockStorage {
resume_disk: Mutex::new(Some(disk)),
usage_baseline: Mutex::new(Some(HealBucketUsageBaseline {
objects_count: 10,
bytes: 8,
})),
..Default::default()
});
let request = HealRequest::new(
HealType::ErasureSet {
buckets: vec!["bucket-a".to_string()],
set_disk_id: "pool_0_set_0".to_string(),
},
HealOptions {
timeout: None,
..Default::default()
},
HealPriority::Normal,
);
let task = HealTask::from_request(request, storage);
task.heal_erasure_set(vec!["bucket-a".to_string()], "pool_0_set_0".to_string())
.await
.expect("erasure set heal should complete");
let progress = task.get_progress().await;
assert_eq!(progress.objects_total_count, 10);
assert_eq!(progress.objects_total_size, 8);
assert_eq!(progress.bytes_processed, 2);
assert!((progress.progress_percentage - 25.0).abs() < 0.001);
}
#[tokio::test]
async fn erasure_set_heal_ignores_usage_baseline_errors() {
let temp = TempDir::new().expect("temporary directory should be created");
let disk = make_resume_disk(&temp).await;
let storage = Arc::new(MockStorage {
resume_disk: Mutex::new(Some(disk)),
usage_baseline_error: Mutex::new(true),
..Default::default()
});
let request = HealRequest::new(
HealType::ErasureSet {
buckets: vec!["bucket-a".to_string()],
set_disk_id: "pool_0_set_0".to_string(),
},
HealOptions {
timeout: None,
..Default::default()
},
HealPriority::Normal,
);
let task = HealTask::from_request(request, storage);
task.heal_erasure_set(vec!["bucket-a".to_string()], "pool_0_set_0".to_string())
.await
.expect("usage baseline failures should not fail erasure set heal");
let progress = task.get_progress().await;
assert_eq!(progress.objects_total_count, 0);
assert_eq!(progress.objects_total_size, 0);
}
#[tokio::test]
async fn resumable_erasure_set_execution_is_cancelled_while_object_heal_is_pending() {
let temp = TempDir::new().expect("temporary directory should be created");
+5
View File
@@ -158,6 +158,10 @@ pub async fn init_heal_manager_with_workload_provider(
return Err(err);
}
// Start the MRF intent consumer (error-path repair intents + durable
// journal replay) now that the manager can accept submissions.
heal::mrf_queue::spawn_mrf_consumer(heal_manager.clone());
#[cfg(test)]
test_hook_after_manager_start().await;
@@ -445,6 +449,7 @@ mod tests {
_bucket: &str,
_prefix: &str,
_continuation_token: Option<&str>,
_include_lifecycle_object_info: bool,
) -> Result<(Vec<HealListItem>, Option<String>, bool), Error> {
Ok((Vec::new(), None, false))
}
@@ -176,7 +176,7 @@ async fn enumerate_all_versions(heal_storage: &Arc<ECStoreHealStorage>, bucket:
let mut token: Option<String> = None;
loop {
let (page, next, truncated) = heal_storage
.list_objects_for_heal_page(bucket, "", token.as_deref())
.list_objects_for_heal_page(bucket, "", token.as_deref(), false)
.await
.expect("list_objects_for_heal_page failed");
items.extend(page);
@@ -166,7 +166,7 @@ async fn enumerate_b5(heal_storage: &Arc<ECStoreHealStorage>, bucket: &str) -> V
let mut token: Option<String> = None;
loop {
let (page, next, truncated) = heal_storage
.list_objects_for_heal_page(bucket, "", token.as_deref())
.list_objects_for_heal_page(bucket, "", token.as_deref(), false)
.await
.expect("b5 list page failed");
items.extend(page);
@@ -187,7 +187,7 @@ async fn enumerate_disk_walk(heal_storage: &Arc<ECStoreHealStorage>, bucket: &st
let mut token: Option<String> = None;
loop {
let (page, next, truncated) = heal_storage
.list_versions_for_heal_page_disk_walk(SET_DISK_ID, bucket, "", token.as_deref())
.list_versions_for_heal_page_disk_walk(SET_DISK_ID, bucket, "", token.as_deref(), false)
.await
.expect("disk-walk list page failed");
items.extend(page);
@@ -418,7 +418,7 @@ mod serial_tests {
let mut pages = 0usize;
loop {
let (versions, next_forward, truncated) = ecstore
.heal_walk_versions_page(0, 0, bucket, "", forward.as_deref(), 2, 100_000)
.heal_walk_versions_page(0, 0, bucket, "", forward.as_deref(), 2, 100_000, false)
.await
.expect("heal_walk_versions_page failed");
pages += 1;
+2
View File
@@ -242,6 +242,7 @@ fn test_heal_task_status_atomic_update() {
_bucket: &str,
_prefix: &str,
_continuation_token: Option<&str>,
_include_lifecycle_object_info: bool,
) -> rustfs_heal::Result<(Vec<HealListItem>, Option<String>, bool)> {
Ok((vec![], None, false))
}
@@ -385,6 +386,7 @@ async fn test_heal_task_transient_object_exists_skip_avoids_recreate() {
_bucket: &str,
_prefix: &str,
_continuation_token: Option<&str>,
_include_lifecycle_object_info: bool,
) -> rustfs_heal::Result<(Vec<HealListItem>, Option<String>, bool)> {
Ok((Vec::new(), None, false))
}
+189
View File
@@ -0,0 +1,189 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! HS-01 (rustfs/backlog#1865): MRF intent pipeline integration tests.
//!
//! Drives the real consumer loop (`spawn_mrf_consumer`) against a real
//! 4-disk `ECStore` heal storage and a `HealManager` that has not started its
//! scheduler, so submitted intents stay observable in the admission queue.
//! Under `cargo nextest` each test runs in its own process, which keeps the
//! process-global MRF channel singleton safe.
use rustfs_common::mrf_channel::{self, MrfKind};
use rustfs_heal::heal::{
manager::{HealConfig, HealManager},
mrf_queue,
storage::{ECStoreHealStorage, HealStorageAPI},
};
use serial_test::serial;
use std::{path::Path, sync::Arc, time::Duration};
mod storage_api;
use storage_api::endpoint_index::{Endpoint, EndpointServerPools, Endpoints, PoolEndpoints, init_local_disks};
const META_BUCKET: &str = ".rustfs.sys";
const JOURNAL_REL: &str = "buckets/.heal/mrf/journal.bin";
async fn heal_env() -> (Vec<std::path::PathBuf>, Arc<dyn HealStorageAPI>) {
let env = rustfs_test_utils::TestECStoreEnv::builder()
.prefix("rustfs_heal_mrf_test")
.build()
.await;
let heal_storage: Arc<dyn HealStorageAPI> = Arc::new(ECStoreHealStorage::new(env.ecstore.clone()));
(env.disk_paths, heal_storage)
}
fn make_manager(storage: Arc<dyn HealStorageAPI>) -> Arc<HealManager> {
Arc::new(HealManager::new(
storage,
Some(HealConfig {
// Keep the scheduler from draining the queue before assertions.
heal_interval: Duration::from_secs(3600),
enable_auto_heal: false,
..Default::default()
}),
))
}
/// Encode one journal record independently of the implementation, so a format
/// drift between writer and this fixture fails loudly here.
fn journal_record(kind: u8, bucket: &str, object: &str, version: Option<[u8; 16]>, attempts: u8) -> Vec<u8> {
let mut body = vec![1u8, 1, kind, attempts];
body.extend_from_slice(&1_700_000_000_000u64.to_le_bytes());
match version {
Some(bytes) => {
body.push(1);
body.extend_from_slice(&bytes);
}
None => body.push(0),
}
body.extend_from_slice(&(bucket.len() as u32).to_le_bytes());
body.extend_from_slice(&(object.len() as u32).to_le_bytes());
body.extend_from_slice(bucket.as_bytes());
body.extend_from_slice(object.as_bytes());
let mut hasher = crc_fast::Digest::new(crc_fast::CrcAlgorithm::Crc32IsoHdlc);
hasher.update(&body);
body.extend_from_slice(&(hasher.finalize() as u32).to_le_bytes());
body
}
fn write_journal_to_disks(disk_paths: &[std::path::PathBuf], data: &[u8]) {
for path in disk_paths {
let journal = path.join(META_BUCKET).join(JOURNAL_REL);
std::fs::create_dir_all(journal.parent().expect("journal parent")).expect("create journal dir");
std::fs::write(&journal, data).expect("write journal fixture");
}
}
async fn wait_until<F, Fut>(deadline: Duration, mut probe: F) -> bool
where
F: FnMut() -> Fut,
Fut: std::future::Future<Output = bool>,
{
let start = std::time::Instant::now();
while start.elapsed() < deadline {
if probe().await {
return true;
}
tokio::time::sleep(Duration::from_millis(50)).await;
}
false
}
/// A decode-failure intent delivered on the global channel must surface in the
/// heal manager as an Urgent request attributed to the MRF source.
#[tokio::test]
#[serial]
async fn decode_failure_intent_maps_to_urgent_mrf_heal_request() {
let (_disk_paths, storage) = heal_env().await;
let manager = make_manager(storage);
mrf_queue::spawn_mrf_consumer(manager.clone());
assert!(
mrf_channel::try_send_mrf_intent(MrfKind::DecodeFailure, "mrf-bucket", "mrf-object", None),
"intent should be accepted while the consumer holds the channel"
);
let appeared = wait_until(Duration::from_secs(10), || async {
let snapshot = manager.operations_snapshot().await;
snapshot.queued_by_source.mrf >= 1 && snapshot.queued_by_priority.urgent >= 1
})
.await;
assert!(
appeared,
"MRF intent must reach the manager queue as an Urgent request (snapshot: {:?})",
manager.operations_snapshot().await
);
}
/// A journal left behind by a previous process must be replayed into the
/// manager queue and then removed, and a torn tail must not block replay of
/// the intact records.
#[tokio::test(flavor = "multi_thread", worker_threads = 4)]
#[serial]
async fn journal_replay_arms_intents_and_deletes_the_file() {
let (disk_paths, storage) = heal_env().await;
// The journal reader resolves disks through the process-local disk map;
// register the environment's disks the same way server startup does.
let mut endpoints: Vec<Endpoint> = disk_paths
.iter()
.map(|p| Endpoint::try_from(p.to_string_lossy().as_ref()).expect("endpoint from disk path"))
.collect();
for (i, endpoint) in endpoints.iter_mut().enumerate() {
endpoint.set_pool_index(0);
endpoint.set_set_index(0);
endpoint.set_disk_index(i);
}
let pool = PoolEndpoints {
legacy: false,
set_count: 1,
drives_per_set: endpoints.len(),
endpoints: Endpoints::from(endpoints),
cmd_line: "mrf-test".to_string(),
platform: String::new(),
};
init_local_disks(EndpointServerPools::from(vec![pool]))
.await
.expect("local disks should register");
let mut journal = journal_record(1, "replay-bucket", "replay-object", Some([9u8; 16]), 0);
journal.extend(journal_record(3, "replay-bucket", "partial-object", None, 1));
// Torn tail: a third record truncated mid-way must not block the two
// intact records above.
journal.extend_from_slice(&journal_record(2, "replay-bucket", "metadata-object", None, 0)[..8]);
write_journal_to_disks(&disk_paths, &journal);
let manager = make_manager(storage);
// Replay directly (not via the process-global channel consumer, which the
// sibling test already claimed in this process under plain `cargo test`).
let replayed = mrf_queue::replay_journal_once(&manager).await;
assert_eq!(replayed, 2, "the two intact records must be replayed");
let snapshot = manager.operations_snapshot().await;
assert_eq!(snapshot.queued_by_source.mrf, 2, "replayed intents must be attributed to the MRF source");
assert!(
disk_paths
.iter()
.all(|path| !Path::new(path).join(META_BUCKET).join(JOURNAL_REL).exists()),
"the journal file must be removed after a successful replay"
);
let snapshot = manager.operations_snapshot().await;
assert_eq!(snapshot.queued_by_priority.urgent, 1, "the decode-failure record must replay as Urgent");
assert!(snapshot.queued_by_priority.normal >= 1, "the partial-write record must replay as Normal");
}
+3
View File
@@ -9,6 +9,9 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Removed
#### rustfs-io-core
- **Zero-consumer modules** (added in 0.0.5): `reader`, `writer`, `bufreader_optimizer`, `shared_memory`, `direct_io`, `timeout_wrapper`, `io_priority_queue`, and `scheduler` had no caller in the workspace and were removed (rustfs/backlog#1824). The scheduling algorithm and the request timeout wrapper that RustFS actually runs live in `rustfs/src/storage/`; this crate keeps the config shapes they project into. `OperationProgress` moved to the new `progress` module and is still exported as `rustfs_io_core::OperationProgress`.
#### rustfs-io-metrics
- **Unified configuration** (added in 0.0.5): the zero-consumer `IoConfig`, `CacheSettings`, `IoSchedulerSettings`, `BackpressureSettings`, `TimeoutSettings`, `DeadlockDetectionSettings` types and their `DEFAULT_*` constants were removed (rustfs/rustfs#6008); rustfs-io-core's `IoSchedulerConfig`/`BackpressureConfig` remain the canonical configuration types.
+2 -3
View File
@@ -20,8 +20,8 @@ license.workspace = true
repository.workspace = true
rust-version.workspace = true
homepage.workspace = true
description = "Buffered I/O reader and writer implementations for RustFS (mmap-then-copy, aligned pread)"
keywords = ["io", "reader", "writer", "rustfs", "mmap"]
description = "Shared I/O primitives for RustFS (buffer pool, storage profiling, backpressure, deadlock detection)"
keywords = ["io", "buffer", "pool", "rustfs", "backpressure"]
categories = ["development-tools", "filesystem"]
[lints]
@@ -38,7 +38,6 @@ hotpath.workspace = true
bytes = { workspace = true, features = ["serde"] }
thiserror = { workspace = true }
tokio = { workspace = true, features = ["io-util", "fs", "sync", "rt-multi-thread"] }
memmap2 = { workspace = true }
rustfs-io-metrics = { workspace = true }
tracing = { workspace = true }
+18 -120
View File
@@ -23,67 +23,20 @@
## Overview
**rustfs-io-core** is the core I/O scheduling module for [RustFS](https://rustfs.com), a distributed object storage system. It provides:
**rustfs-io-core** holds the shared I/O primitives for [RustFS](https://rustfs.com), a distributed object storage system. It provides:
- **I/O Scheduler**: Adaptive buffer size calculation and load management
- **Priority Queue**: Request priority scheduling with starvation prevention
- **Buffer Pool**: Tiered `BytesPool` for buffer reuse
- **Storage Profiling**: Storage-media and access-pattern model (`io_profile`)
- **Scheduler Configuration**: The `IoSchedulerConfig` / `IoPriorityQueueConfig` shapes the storage layer projects into
- **Backpressure Control**: System overload protection with graceful degradation
- **Deadlock Detection**: Wait-for graph based deadlock detection algorithm
- **Lock Optimizer**: Adaptive spin lock optimization
- **Timeout Wrapper**: Dynamic timeout calculation and operation progress tracking
- **Progress Tracking**: Byte progress and staleness for long-running operations
The scheduling algorithm itself lives in `rustfs/src/storage/concurrency/io_schedule.rs`; this crate carries the configuration shapes it projects into, not a second implementation.
## Features
### I/O Scheduler
Adaptive I/O scheduling with dynamic buffer size calculation based on file size, access pattern, and system load:
```rust
use rustfs_io_core::{IoScheduler, IoSchedulerConfig, IoLoadLevel};
use rustfs_io_core::io_profile::{StorageMedia, AccessPattern};
// Create scheduler
let config = IoSchedulerConfig {
max_concurrent_reads: 64,
base_buffer_size: 64 * 1024, // 64 KB
max_buffer_size: 1024 * 1024, // 1 MB
..Default::default()
};
let scheduler = IoScheduler::new(config);
// Calculate optimal buffer size
let buffer_size = calculate_optimal_buffer_size(
10 * 1024 * 1024, // 10 MB file
64 * 1024, // base buffer
true, // sequential access
4, // concurrent requests
StorageMedia::Ssd,
IoLoadLevel::Low,
);
```
### Priority Queue
Priority queue with starvation prevention:
```rust
use rustfs_io_core::{IoPriorityQueue, IoPriority, IoQueueStatus};
let queue = IoPriorityQueue::<()>::new(100);
// Enqueue request
let request_id = queue.enqueue(IoPriority::High, (), 1024);
// Dequeue request
if let Some((priority, data)) = queue.dequeue() {
println!("Processing priority {:?} request", priority);
}
// Check queue status
let status = queue.status();
println!("High priority waiting: {}", status.high_priority_waiting);
```
### Backpressure Control
System overload protection:
@@ -148,71 +101,23 @@ let stats = optimizer.stats();
println!("Locks acquired: {}", stats.total_acquired());
```
### Timeout Wrapper
### Progress Tracking
Dynamic timeout calculation:
Byte progress and staleness for long-running operations:
```rust
use rustfs_io_core::{RequestTimeoutWrapper, TimeoutConfig};
use rustfs_io_core::OperationProgress;
use std::time::Duration;
let config = TimeoutConfig {
base_timeout: Duration::from_secs(5),
timeout_per_mb: Duration::from_millis(100),
max_timeout: Duration::from_secs(300),
..Default::default()
};
let wrapper = RequestTimeoutWrapper::new(config);
let progress = OperationProgress::new(Some(1000), Duration::from_secs(5));
// Calculate operation timeout
let timeout = wrapper.calculate_timeout(10 * 1024 * 1024); // 10 MB
```
## Buffer Size Calculation
Multiple buffer size calculation functions are provided:
```rust
use rustfs_io_core::{
get_concurrency_aware_buffer_size,
get_advanced_buffer_size,
get_buffer_size_for_media,
calculate_optimal_buffer_size,
KI_B, MI_B,
};
use rustfs_io_core::io_profile::StorageMedia;
// Basic calculation
let size1 = get_concurrency_aware_buffer_size(1024 * 1024, 64 * 1024);
// Advanced calculation (considering access pattern)
let size2 = get_advanced_buffer_size(10 * 1024 * 1024, 64 * 1024, true);
// Media type optimization
let size3 = get_buffer_size_for_media(64 * 1024, StorageMedia::Ssd);
// Comprehensive calculation
let size4 = calculate_optimal_buffer_size(
100 * 1024 * 1024, // 100 MB file
64 * 1024, // base buffer
true, // sequential access
4, // concurrent requests
StorageMedia::Nvme,
IoLoadLevel::Low,
);
progress.update(500);
assert_eq!(progress.progress_percent(), Some(50.0));
assert!(!progress.is_stale());
```
## Configuration
### Environment Variables
| Variable | Description | Default |
|----------|-------------|---------|
| `RUSTFS_MAX_CONCURRENT_READS` | Max concurrent reads | 64 |
| `RUSTFS_BASE_BUFFER_SIZE` | Base buffer size | 65536 |
| `RUSTFS_MAX_BUFFER_SIZE` | Max buffer size | 1048576 |
| `RUSTFS_IO_TIMEOUT_SECS` | I/O timeout seconds | 30 |
### Code Configuration
```rust
@@ -240,12 +145,11 @@ rustfs-io-core/
├── src/
│ ├── lib.rs # Module entry
│ ├── config.rs # Configuration types
│ ├── scheduler.rs # I/O scheduler
│ ├── io_priority_queue.rs # Priority queue
│ ├── pool.rs # Tiered buffer pool
│ ├── backpressure.rs # Backpressure control
│ ├── deadlock_detector.rs # Deadlock detection
│ ├── lock_optimizer.rs # Lock optimization
│ ├── timeout_wrapper.rs # Timeout wrapper
│ ├── progress.rs # Operation progress tracking
│ └── io_profile.rs # I/O profile
└── Cargo.toml
```
@@ -254,21 +158,15 @@ rustfs-io-core/
```bash
# Run all tests
cargo test --package rustfs-io-core
cargo nextest run --package rustfs-io-core
# Run specific tests
cargo test --package rustfs-io-core --lib scheduler
# Run benchmarks
cargo bench --package rustfs-io-core
cargo nextest run --package rustfs-io-core -E 'test(backpressure)'
```
## Documentation
- [API Documentation](https://docs.rs/rustfs-io-core)
- [I/O Scheduler Design](./docs/scheduler-design.md)
- [Backpressure Control Design](./docs/backpressure-design.md)
- [Deadlock Detection Algorithm](./docs/deadlock-detection.md)
## Related Modules
+18 -131
View File
@@ -23,71 +23,20 @@
## 📖 概述
**rustfs-io-core** 是 [RustFS](https://rustfs.com) 分布式对象存储系统的核心 I/O 调度模块。它提供了:
**rustfs-io-core** 是 [RustFS](https://rustfs.com) 分布式对象存储系统的共享 I/O 基础组件。它提供了:
- **I/O 调度器**:自适应缓冲区大小计算和负载管理
- **优先级队列**支持饥饿预防的请求优先级调度
- **缓冲池**:分级复用的 `BytesPool`
- **存储画像**存储介质与访问模式模型(`io_profile`
- **调度配置**:存储层投影使用的 `IoSchedulerConfig` / `IoPriorityQueueConfig`
- **背压控制**:系统过载保护和优雅降级
- **死锁检测**:基于等待图的死锁检测算法
- **锁优化**:自适应自旋锁优化
- **超时包装器**动态超时计算和操作进度追踪
- **进度追踪**长耗时操作的字节进度与停滞判定
调度算法本身位于 `rustfs/src/storage/concurrency/io_schedule.rs`;本 crate 只承载它投影使用的配置形状,不是第二套实现。
## ✨ 核心功能
### I/O 调度器 (IoScheduler)
自适应 I/O 调度,根据文件大小、访问模式和系统负载动态调整缓冲区大小:
```rust
use rustfs_io_core::{IoScheduler, IoSchedulerConfig, IoLoadLevel};
use rustfs_io_core::io_profile::{StorageMedia, AccessPattern};
// 创建调度器
let config = IoSchedulerConfig {
max_concurrent_reads: 64,
base_buffer_size: 64 * 1024, // 64 KB
max_buffer_size: 1024 * 1024, // 1 MB
..Default::default()
};
let scheduler = IoScheduler::new(config);
// 计算最优缓冲区大小
let buffer_size = scheduler.calculate_buffer_size(
10 * 1024 * 1024, // 10 MB 文件
true, // 顺序访问
StorageMedia::Ssd,
IoLoadLevel::Low,
);
println!("缓冲区大小: {} bytes", buffer_size);
```
### 优先级队列 (IoPriorityQueue)
支持饥饿预防的优先级队列:
```rust
use rustfs_io_core::{IoPriorityQueue, IoPriority, IoQueueStatus};
let queue = IoPriorityQueue::<()>::new(100);
// 入队请求
let request_id = queue.enqueue(
IoPriority::High,
(), // 请求数据
1024, // 请求大小
);
// 出队请求
if let Some((priority, data)) = queue.dequeue() {
println!("处理优先级 {:?} 的请求", priority);
}
// 检查队列状态
let status = queue.status();
println!("高优先级等待: {}", status.high_priority_waiting);
println!("低优先级等待: {}", status.low_priority_waiting);
```
### 背压控制 (BackpressureMonitor)
系统过载保护:
@@ -165,78 +114,23 @@ let stats = optimizer.stats();
println!("获取锁次数: {}", stats.locks_acquired.load(std::sync::atomic::Ordering::Relaxed));
```
### 超时包装器 (RequestTimeoutWrapper)
### 进度追踪 (OperationProgress)
动态超时计算
长耗时操作的字节进度与停滞判定
```rust
use rustfs_io_core::{RequestTimeoutWrapper, TimeoutConfig};
use rustfs_io_core::OperationProgress;
use std::time::Duration;
let config = TimeoutConfig {
base_timeout: Duration::from_secs(5),
timeout_per_mb: Duration::from_millis(100),
max_timeout: Duration::from_secs(300),
..Default::default()
};
let wrapper = RequestTimeoutWrapper::new(config);
let progress = OperationProgress::new(Some(1000), Duration::from_secs(5));
// 计算操作超时
let timeout = wrapper.calculate_timeout(10 * 1024 * 1024); // 10 MB
println!("超时时间: {:?}", timeout);
// 执行带超时的操作
let result = wrapper.execute_with_timeout(async {
// 异步操作
Ok::<_, std::io::Error>(())
}, timeout).await;
```
## 📊 缓冲区大小计算
模块提供了多种缓冲区大小计算函数:
```rust
use rustfs_io_core::{
get_concurrency_aware_buffer_size,
get_advanced_buffer_size,
get_buffer_size_for_media,
calculate_optimal_buffer_size,
KI_B, MI_B,
};
use rustfs_io_core::io_profile::StorageMedia;
// 基础计算
let size1 = get_concurrency_aware_buffer_size(1024 * 1024, 64 * 1024);
// 高级计算(考虑访问模式)
let size2 = get_advanced_buffer_size(10 * 1024 * 1024, 64 * 1024, true);
// 媒体类型优化
let size3 = get_buffer_size_for_media(64 * 1024, StorageMedia::Ssd);
// 综合计算
let size4 = calculate_optimal_buffer_size(
100 * 1024 * 1024, // 100 MB 文件
64 * 1024, // 基础缓冲区
true, // 顺序访问
4, // 并发请求数
StorageMedia::Nvme,
IoLoadLevel::Low,
);
progress.update(500);
assert_eq!(progress.progress_percent(), Some(50.0));
assert!(!progress.is_stale());
```
## 🔧 配置
### 环境变量
| 变量名 | 描述 | 默认值 |
|--------|------|--------|
| `RUSTFS_MAX_CONCURRENT_READS` | 最大并发读数 | 64 |
| `RUSTFS_BASE_BUFFER_SIZE` | 基础缓冲区大小 | 65536 |
| `RUSTFS_MAX_BUFFER_SIZE` | 最大缓冲区大小 | 1048576 |
| `RUSTFS_IO_TIMEOUT_SECS` | I/O 超时秒数 | 30 |
### 代码配置
```rust
@@ -264,12 +158,11 @@ rustfs-io-core/
├── src/
│ ├── lib.rs # 模块入口
│ ├── config.rs # 配置类型
│ ├── scheduler.rs # I/O 调度器
│ ├── io_priority_queue.rs # 优先级队列
│ ├── pool.rs # 分级缓冲池
│ ├── backpressure.rs # 背压控制
│ ├── deadlock_detector.rs # 死锁检测
│ ├── lock_optimizer.rs # 锁优化
│ ├── timeout_wrapper.rs # 超时包装器
│ ├── progress.rs # 操作进度追踪
│ └── io_profile.rs # I/O 配置文件
└── Cargo.toml
```
@@ -278,21 +171,15 @@ rustfs-io-core/
```bash
# 运行所有测试
cargo test --package rustfs-io-core
cargo nextest run --package rustfs-io-core
# 运行特定测试
cargo test --package rustfs-io-core --lib scheduler
# 运行基准测试
cargo bench --package rustfs-io-core
cargo nextest run --package rustfs-io-core -E 'test(backpressure)'
```
## 📚 文档
- [API 文档](https://docs.rs/rustfs-io-core)
- [I/O 调度器设计](./docs/scheduler-design.md)
- [背压控制原理](./docs/backpressure-design.md)
- [死锁检测算法](./docs/deadlock-detection.md)
## 🔗 相关模块
@@ -1,190 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Example demonstrating I/O scheduler usage.
use rustfs_io_core::io_profile::StorageMedia;
use rustfs_io_core::{
BackpressureMonitor, BackpressureState, DeadlockDetector, IoLoadLevel, IoScheduler, IoSchedulerConfig, KI_B, LockOptimizer,
LockType, MI_B, calculate_optimal_buffer_size, get_buffer_size_for_media,
};
use std::time::Duration;
fn main() {
println!("=== rustfs-io-core Example ===\n");
// 1. I/O scheduler example
io_scheduler_example();
// 2. Buffer size calculation example
buffer_size_example();
// 3. Backpressure control example
backpressure_example();
// 4. Deadlock detection example
deadlock_detection_example();
// 5. Lock optimizer example
lock_optimizer_example();
}
fn io_scheduler_example() {
println!("--- I/O Scheduler ---");
// Create scheduler with configuration
let config = IoSchedulerConfig {
max_concurrent_reads: 64,
base_buffer_size: 64 * KI_B,
max_buffer_size: MI_B,
..Default::default()
};
let scheduler = IoScheduler::new(config);
println!(" Max concurrent reads: {}", scheduler.config().max_concurrent_reads);
println!(" Base buffer size: {} KB", scheduler.config().base_buffer_size / KI_B);
println!(" Max buffer size: {} KB", scheduler.config().max_buffer_size / KI_B);
// Calculate buffer sizes for different scenarios
let scenarios = [
("Small file", 10 * KI_B as i64, true, StorageMedia::Ssd),
("Medium file", MI_B as i64, true, StorageMedia::Ssd),
("Large sequential", 100 * MI_B as i64, true, StorageMedia::Ssd),
("Large random", 100 * MI_B as i64, false, StorageMedia::Ssd),
("NVMe large", 100 * MI_B as i64, true, StorageMedia::Nvme),
("HDD large", 100 * MI_B as i64, true, StorageMedia::Hdd),
];
for (name, size, sequential, media) in scenarios {
let buffer = calculate_optimal_buffer_size(size, 64 * KI_B, sequential, 4, media, IoLoadLevel::Low);
println!(" {}: {} bytes ({} KB)", name, buffer, buffer / KI_B);
}
println!();
}
fn buffer_size_example() {
println!("--- Buffer Size Calculation ---");
// Comprehensive calculation
let size1 = calculate_optimal_buffer_size(10 * MI_B as i64, 64 * KI_B, true, 4, StorageMedia::Ssd, IoLoadLevel::Low);
println!(" Comprehensive (10MB, sequential, SSD): {} KB", size1 / KI_B);
// Media type optimization
let media_types = [
StorageMedia::Nvme,
StorageMedia::Ssd,
StorageMedia::Hdd,
StorageMedia::Unknown,
];
for media in media_types {
let size = get_buffer_size_for_media(64 * KI_B, media);
println!(" {} optimized: {} KB", media.as_str(), size / KI_B);
}
println!();
}
fn backpressure_example() {
println!("--- Backpressure Control ---");
let monitor = BackpressureMonitor::with_defaults();
// Check initial state
let state = monitor.state();
let state_str = match state {
BackpressureState::Normal => "Normal",
BackpressureState::Warning => "Warning",
BackpressureState::Critical => "Critical",
};
println!(" Initial state: {}", state_str);
// Check if active
let is_active = monitor.is_active();
println!(" Backpressure active: {}", is_active);
// Try to acquire permit
if monitor.try_acquire() {
println!(" Successfully acquired permit");
monitor.release();
println!(" Released permit");
}
// View statistics
println!(" Total processed: {}", monitor.total_processed());
println!(" Total rejected: {}", monitor.total_rejected());
println!();
}
fn deadlock_detection_example() {
println!("--- Deadlock Detection ---");
let detector = DeadlockDetector::with_defaults();
// Register locks
let mutex1 = detector.register_lock(LockType::Mutex);
let mutex2 = detector.register_lock(LockType::Mutex);
println!(" Registered locks: mutex1={}, mutex2={}", mutex1, mutex2);
// Simulate normal operation
detector.record_acquire(mutex1, 1); // Thread 1 acquires mutex1
detector.record_acquire(mutex2, 2); // Thread 2 acquires mutex2
println!(" Normal operation: no deadlock");
// Detect deadlock
if detector.detect_deadlock().is_none() {
println!(" Detection result: no deadlock");
}
// Simulate deadlock scenario
detector.record_wait(mutex2, 1); // Thread 1 waits for mutex2
detector.record_wait(mutex1, 2); // Thread 2 waits for mutex1
// Detect deadlock
if let Some(deadlock) = detector.detect_deadlock() {
println!(" Detection result: deadlock found {:?}", deadlock);
}
// Cleanup
detector.unregister_lock(mutex1);
detector.unregister_lock(mutex2);
println!();
}
fn lock_optimizer_example() {
println!("--- Lock Optimizer ---");
let optimizer = LockOptimizer::with_defaults();
// Simulate lock operations
for _i in 0..5 {
optimizer.on_acquire();
// Simulate work
std::thread::sleep(Duration::from_millis(10));
optimizer.on_release(Duration::from_millis(10));
}
// View statistics
let stats = optimizer.stats();
let acquired = stats.total_acquired();
let avg_hold = stats.avg_hold_time();
let contention = stats.contention_rate();
println!(" Locks acquired: {}", acquired);
println!(" Average hold time: {:?}", avg_hold);
println!(" Contention rate: {:.2}%", contention * 100.0);
println!();
}
-227
View File
@@ -1,227 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! BufReader layer optimizer for minimizing redundant buffering layers.
//!
//! This module provides optimization for BufReader usage in data paths,
//! including layer count limiting and dynamic buffer size adjustment.
use std::sync::atomic::{AtomicU64, Ordering};
/// BufReader optimization configuration.
#[derive(Debug, Clone)]
pub struct BufReaderConfig {
/// Maximum number of nested BufReader layers (default: 2)
pub max_layers: u32,
/// Buffer size for small files (default: 8KB)
pub small_file_buffer: usize,
/// Buffer size for large files (default: 64KB)
pub large_file_buffer: usize,
/// Threshold for large file classification (default: 1MB)
pub large_file_threshold: usize,
}
impl Default for BufReaderConfig {
fn default() -> Self {
Self {
max_layers: 2,
small_file_buffer: 8 * 1024, // 8KB
large_file_buffer: 64 * 1024, // 64KB
large_file_threshold: 1024 * 1024, // 1MB
}
}
}
/// BufReader optimization statistics.
#[derive(Debug, Default)]
pub struct BufReaderStats {
/// Total number of readers created
pub total_readers: AtomicU64,
/// Number of redundant layers eliminated
pub eliminated_layers: AtomicU64,
/// Number of buffer size adjustments
pub buffer_size_adjustments: AtomicU64,
}
/// BufReader layer optimizer.
///
/// Analyzes and optimizes BufReader nesting in data paths,
/// dynamically adjusting buffer sizes based on data characteristics.
pub struct BufReaderOptimizer {
config: BufReaderConfig,
stats: BufReaderStats,
}
impl BufReaderOptimizer {
/// Create a new BufReader optimizer with the given configuration.
pub fn new(config: BufReaderConfig) -> Self {
Self {
config,
stats: BufReaderStats::default(),
}
}
/// Create a new BufReader optimizer with default configuration.
pub fn with_defaults() -> Self {
Self::new(BufReaderConfig::default())
}
/// Calculate the optimal buffer size based on data size.
///
/// Returns the appropriate buffer size based on whether the data
/// is classified as a small or large file.
pub fn optimal_buffer_size(&self, data_size: Option<usize>) -> usize {
match data_size {
Some(size) if size >= self.config.large_file_threshold => self.config.large_file_buffer,
Some(_) => self.config.small_file_buffer,
None => self.config.small_file_buffer,
}
}
/// Optimize a reader by wrapping it with an appropriately sized BufReader.
///
/// This method applies the optimal buffer size based on the expected
/// data size and tracks statistics.
pub fn optimize<R: tokio::io::AsyncRead + Unpin>(&self, reader: R, data_size: Option<usize>) -> tokio::io::BufReader<R> {
let buffer_size = self.optimal_buffer_size(data_size);
self.stats.total_readers.fetch_add(1, Ordering::Relaxed);
tokio::io::BufReader::with_capacity(buffer_size, reader)
}
/// Get the statistics for this optimizer.
pub fn stats(&self) -> &BufReaderStats {
&self.stats
}
/// Get the configuration for this optimizer.
pub fn config(&self) -> &BufReaderConfig {
&self.config
}
}
/// Marker trait for buffered sources.
///
/// Types implementing this trait are considered already buffered
/// and should not be wrapped with additional BufReader layers.
pub trait BufferedSource: tokio::io::AsyncRead {}
impl BufReaderOptimizer {
/// Check if a reader is already a buffered source.
///
/// Returns true if the reader implements `BufferedSource`,
/// indicating it should not be wrapped with BufReader.
pub fn is_buffered_source<R: BufferedSource + ?Sized>(&self, _reader: &R) -> bool {
true
}
/// Eliminate redundant BufReader layers if possible.
///
/// This method attempts to reduce the nesting depth of BufReader
/// layers to improve performance.
pub fn eliminate_redundant_layers<R: tokio::io::AsyncRead + Unpin>(&self, reader: R) -> R {
// For now, just return the reader as-is
// Future implementation could detect and unwrap nested BufReaders
self.stats.eliminated_layers.fetch_add(0, Ordering::Relaxed);
reader
}
}
#[cfg(test)]
mod tests {
use super::*;
use tokio::io::AsyncReadExt;
#[test]
fn test_default_config() {
let config = BufReaderConfig::default();
assert_eq!(config.max_layers, 2);
assert_eq!(config.small_file_buffer, 8 * 1024);
assert_eq!(config.large_file_buffer, 64 * 1024);
assert_eq!(config.large_file_threshold, 1024 * 1024);
}
#[test]
fn test_optimal_buffer_size_small_file() {
let optimizer = BufReaderOptimizer::with_defaults();
// Small file (< 1MB)
assert_eq!(optimizer.optimal_buffer_size(Some(100)), 8 * 1024);
assert_eq!(optimizer.optimal_buffer_size(Some(1024)), 8 * 1024);
assert_eq!(optimizer.optimal_buffer_size(Some(512 * 1024)), 8 * 1024);
}
#[test]
fn test_optimal_buffer_size_large_file() {
let optimizer = BufReaderOptimizer::with_defaults();
// Large file (>= 1MB)
assert_eq!(optimizer.optimal_buffer_size(Some(1024 * 1024)), 64 * 1024);
assert_eq!(optimizer.optimal_buffer_size(Some(10 * 1024 * 1024)), 64 * 1024);
}
#[test]
fn test_optimal_buffer_size_unknown() {
let optimizer = BufReaderOptimizer::with_defaults();
// Unknown size
assert_eq!(optimizer.optimal_buffer_size(None), 8 * 1024);
}
#[tokio::test]
async fn test_optimize_creates_bufreader() {
let optimizer = BufReaderOptimizer::with_defaults();
let data = vec![1u8, 2, 3, 4, 5];
let cursor = std::io::Cursor::new(data.clone());
let mut reader = optimizer.optimize(cursor, Some(5));
let mut buf = vec![0u8; 5];
let n = reader.read(&mut buf).await.unwrap();
assert_eq!(n, 5);
assert_eq!(buf, data);
}
#[test]
fn test_stats_tracking() {
let optimizer = BufReaderOptimizer::with_defaults();
assert_eq!(optimizer.stats().total_readers.load(Ordering::Relaxed), 0);
let cursor = std::io::Cursor::new(vec![1u8, 2, 3]);
let _reader = optimizer.optimize(cursor, Some(3));
assert_eq!(optimizer.stats().total_readers.load(Ordering::Relaxed), 1);
}
#[test]
fn test_custom_config() {
let config = BufReaderConfig {
max_layers: 3,
small_file_buffer: 4 * 1024,
large_file_buffer: 128 * 1024,
large_file_threshold: 2 * 1024 * 1024,
};
let optimizer = BufReaderOptimizer::new(config);
assert_eq!(optimizer.optimal_buffer_size(Some(1024 * 1024)), 4 * 1024);
assert_eq!(optimizer.optimal_buffer_size(Some(3 * 1024 * 1024)), 128 * 1024);
}
}
-332
View File
@@ -1,332 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! Aligned pread-based file reader.
//!
//! This module provides an aligned, position-based file reader that uses
//! `pread`/`FileExt::read_at` for I/O operations. It performs reads at
//! 512-byte-aligned offsets and sizes, making it suitable as a foundation
//! for workloads where alignment matters.
//!
//! Note: This reader does **not** set the `O_DIRECT` flag and therefore does
//! not bypass the OS page cache. It is an aligned `pread`-based reader, not
//! true Direct I/O. To implement true O_DIRECT on Linux, the file must be
//! opened with `O_DIRECT` via `libc::open`.
//!
//! # Platform Support
//!
//! The `read_at` implementation is only available on Unix-like platforms.
//! On other platforms, this reader will return an error.
use std::io::{self};
use std::pin::Pin;
use std::task::{Context, Poll};
use tokio::io::{AsyncRead, ReadBuf};
/// Errors that can occur during aligned pread operations.
#[derive(Debug, Clone)]
pub enum AlignedPreadError {
/// Platform doesn't support `read_at`-based I/O
UnsupportedPlatform,
/// File descriptor doesn't support this reader
UnsupportedFile,
/// I/O error occurred
Io(String),
/// Invalid alignment (reads require 512-byte-aligned offset and size)
AlignmentError { offset: u64, size: usize },
}
impl std::fmt::Display for AlignedPreadError {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
match self {
Self::UnsupportedPlatform => write!(f, "Aligned pread not supported on this platform"),
Self::UnsupportedFile => write!(f, "File doesn't support this reader"),
Self::Io(msg) => write!(f, "I/O error: {}", msg),
Self::AlignmentError { offset, size } => {
write!(f, "Alignment error: offset={}, size={}", offset, size)
}
}
}
}
impl std::error::Error for AlignedPreadError {}
impl From<io::Error> for AlignedPreadError {
fn from(err: io::Error) -> Self {
Self::Io(err.to_string())
}
}
/// Aligned pread-based file reader for Unix platforms.
///
/// This reader performs I/O using `pread`/`FileExt::read_at` at
/// 512-byte-aligned offsets and sizes, without modifying the file's
/// current position.
///
/// **Note:** This reader does **not** set the `O_DIRECT` flag and therefore
/// does **not** bypass the OS page cache. It is an aligned `pread`-based
/// reader. To implement true O_DIRECT, the file must be opened with
/// `O_DIRECT` via `libc::open`.
///
/// # Platform Support
///
/// Only available on Linux (uses `FileExt::read_at`). On other platforms,
/// use `BytesBufferedReader` instead.
///
/// # Alignment Requirements
///
/// Reads have strict alignment requirements:
/// - File offset must be aligned to 512 bytes
/// - Buffer size must be a multiple of 512 bytes
/// - Buffer address must be aligned (handled internally)
///
/// # Example
///
/// ```ignore
/// use rustfs_io_core::AlignedPreadReader;
///
/// // Linux only
/// #[cfg(target_os = "linux")]
/// let reader = AlignedPreadReader::new(file, offset, size)?;
/// ```
#[cfg(target_os = "linux")]
pub struct AlignedPreadReader {
/// Underlying file handle used for aligned pread I/O
file: std::fs::File,
/// Current read position
pos: u64,
/// Remaining bytes to read
remaining: usize,
/// Buffer for aligned reads
buffer: Vec<u8>,
/// Current position in the buffer
buffer_pos: usize,
/// Amount of data in the buffer
buffer_len: usize,
}
#[cfg(target_os = "linux")]
impl AlignedPreadReader {
/// Alignment requirement for reads (512 bytes for most systems)
pub const ALIGNMENT: usize = 512;
/// Create a new aligned pread-based reader.
///
/// # Arguments
///
/// * `file` - File to read from
/// * `offset` - Starting offset in the file (must be 512-byte aligned)
/// * `size` - Number of bytes to read (must be 512-byte aligned)
///
/// # Returns
///
/// An `AlignedPreadReader` that reads the file at the given offset.
///
/// # Errors
///
/// Returns an error if offset or size are not 512-byte aligned.
pub fn new(file: std::fs::File, offset: u64, size: usize) -> Result<Self, AlignedPreadError> {
// Check alignment
if !offset.is_multiple_of(Self::ALIGNMENT as u64) {
return Err(AlignedPreadError::AlignmentError { offset, size });
}
if !size.is_multiple_of(Self::ALIGNMENT) {
return Err(AlignedPreadError::AlignmentError { offset, size });
}
Ok(Self {
file,
pos: offset,
remaining: size,
buffer: Vec::new(),
buffer_pos: 0,
buffer_len: 0,
})
}
/// Read a chunk of data using aligned pread.
///
/// This method performs aligned reads and handles the buffering required
/// by this aligned pread implementation. It does not use `O_DIRECT`.
fn read_chunk(&mut self, buf: &mut [u8]) -> io::Result<usize> {
// If buffer is exhausted, read more data
if self.buffer_pos >= self.buffer_len {
if self.remaining == 0 {
return Ok(0);
}
// Allocate aligned buffer
let chunk_size = (self.remaining).min(64 * 1024); // 64KB chunks
let aligned_size = chunk_size.div_ceil(Self::ALIGNMENT) * Self::ALIGNMENT;
self.buffer = vec![0u8; aligned_size];
// Use pread for atomic read at position (no file offset modification)
use std::os::unix::fs::FileExt;
let n = self.file.read_at(&mut self.buffer, self.pos)?;
self.buffer_pos = 0;
self.buffer_len = n;
self.pos += n as u64;
self.remaining -= n;
if n == 0 {
return Ok(0);
}
}
// Copy from buffer to user buffer
let available = self.buffer_len - self.buffer_pos;
let to_copy = buf.len().min(available);
buf[..to_copy].copy_from_slice(&self.buffer[self.buffer_pos..self.buffer_pos + to_copy]);
self.buffer_pos += to_copy;
Ok(to_copy)
}
}
#[cfg(target_os = "linux")]
impl AsyncRead for AlignedPreadReader {
fn poll_read(mut self: Pin<&mut Self>, _cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
let filled = buf.filled().len();
let mut remaining = buf.initialize_unfilled();
while !remaining.is_empty() {
match self.read_chunk(remaining) {
Ok(0) => break,
Ok(n) => {
remaining = &mut remaining[n..];
}
Err(e) => return Poll::Ready(Err(e)),
}
}
let _n_read = buf.filled().len() - filled;
Poll::Ready(Ok(()))
}
}
/// Aligned pread reader stub for non-Linux platforms.
///
/// On non-Linux platforms, `read_at`-based I/O is not available through this
/// type. This stub exists to provide a consistent API across platforms.
#[cfg(not(target_os = "linux"))]
pub struct AlignedPreadReader {
_priv: (),
}
#[cfg(not(target_os = "linux"))]
impl AlignedPreadReader {
/// Create a new aligned pread reader (not supported on this platform).
///
/// Always returns an error on non-Linux platforms.
pub fn new(_file: std::fs::File, _offset: u64, _size: usize) -> Result<Self, AlignedPreadError> {
Err(AlignedPreadError::UnsupportedPlatform)
}
}
#[cfg(not(target_os = "linux"))]
impl AsyncRead for AlignedPreadReader {
fn poll_read(self: Pin<&mut Self>, _cx: &mut Context<'_>, _buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
Poll::Ready(Err(io::Error::new(
io::ErrorKind::Unsupported,
"Aligned pread-based I/O not supported on this platform",
)))
}
}
impl std::fmt::Debug for AlignedPreadReader {
fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
#[cfg(target_os = "linux")]
{
f.debug_struct("AlignedPreadReader")
.field("pos", &self.pos)
.field("remaining", &self.remaining)
.field("buffer_len", &self.buffer_len)
.finish()
}
#[cfg(not(target_os = "linux"))]
{
f.debug_struct("AlignedPreadReader")
.field("platform", &"unsupported")
.finish()
}
}
}
/// Historical name for aligned pread errors.
#[deprecated(since = "1.0.0-beta.8", note = "use AlignedPreadError; this reader does not set O_DIRECT")]
pub type DirectIoError = AlignedPreadError;
/// Historical name for the aligned pread-based reader.
#[deprecated(since = "1.0.0-beta.8", note = "use AlignedPreadReader; this reader does not set O_DIRECT")]
pub type DirectIoReader = AlignedPreadReader;
#[cfg(test)]
mod tests {
use super::*;
#[test]
fn test_alignment_check() {
#[cfg(target_os = "linux")]
{
// Valid alignment
let file = std::fs::File::open("/dev/zero").unwrap();
assert!(
AlignedPreadReader::new(file, 0, 512).is_ok(),
"Should succeed with aligned offset and size"
);
let file = std::fs::File::open("/dev/zero").expect("open /dev/zero for alias");
assert!(
AlignedPreadReader::new(file, 0, 512).is_ok(),
"Should succeed through aligned pread alias"
);
// Invalid offset
let file = std::fs::File::open("/dev/zero").unwrap();
assert!(AlignedPreadReader::new(file, 1, 512).is_err(), "Should fail with unaligned offset");
// Invalid size
let file = std::fs::File::open("/dev/zero").unwrap();
assert!(AlignedPreadReader::new(file, 0, 511).is_err(), "Should fail with unaligned size");
}
#[cfg(not(target_os = "linux"))]
{
// Non-Linux should return UnsupportedPlatform
let file = std::fs::File::open(std::env::current_exe().unwrap()).unwrap();
assert!(matches!(
AlignedPreadReader::new(file, 0, 512),
Err(AlignedPreadError::UnsupportedPlatform)
));
}
}
#[test]
#[allow(deprecated)]
fn test_legacy_direct_io_alias() {
#[cfg(target_os = "linux")]
{
let file = std::fs::File::open("/dev/zero").unwrap();
assert!(DirectIoReader::new(file, 0, 512).is_ok());
}
#[cfg(not(target_os = "linux"))]
{
let file = std::fs::File::open(std::env::current_exe().unwrap()).unwrap();
assert!(matches!(DirectIoReader::new(file, 0, 512), Err(AlignedPreadError::UnsupportedPlatform)));
}
}
}

Some files were not shown because too many files have changed in this diff Show More