Compare commits

...

66 Commits

Author SHA1 Message Date
overtrue 728b0488c4 chore(ecstore): drop the data_usage dead_code blanket
Removing the blanket exposes eighteen items. Six are deleted; the rest belong to one feature that was never wired up.

crates/ecstore/src/data_usage/local_snapshot.rs and its aggregation entry point, aggregate_local_snapshots, form a complete per-disk usage-snapshot feature: read/write of snapshot files under the metadata bucket, cross-disk aggregation, and tests. Nothing calls it. It landed in #5307 on 2026-07-27 and git log -S over the whole history shows the entry point has never had a caller; the live data-usage path is load_data_usage_from_backend / store_data_usage_in_backend. Rather than delete a recent, tested feature on cleanup grounds, the module gets a header explaining the situation and its items carry individual allows, so the gap stays greppable without reintroducing a blanket. One commit flips this to deletion if that is preferred.

Deleted:

- DATA_USAGE_BLOOM_NAME together with DATA_USAGE_BLOOM_NAME_PATH. Both are a dead duplicate of the pair in crates/scanner/src/data_usage_define.rs, which has roughly fifty consumers across scanner.rs and remote_scanner.rs; the ecstore copies have none.
- increment_bucket_usage_memory and decrement_bucket_usage_memory, thin wrappers over the live record_bucket_object_write_memory and record_bucket_object_delete_memory.
- sync_memory_cache_with_backend, whose doc comment says it is "called by scanner" — nothing calls it.
- create_cache_entry_from_summary and cache_to_data_usage_info, neither with a consumer in any lane.

resolve_loaded_snapshot is kept with an allow: nine tests in the same file assert its primary/backup fallback.

Verification, four lanes warning-free: default, --tests, --features rio-v2 --tests, --features test-util --tests. rustfs-scanner still compiles clean. cargo nextest run -p rustfs-ecstore 4041 passed; clippy --lib --tests -D warnings clean; make pre-commit exit 0.

Ref rustfs/backlog#1823 (step 2).
2026-08-14 10:24:09 +08:00
cxymds 8ac2ff5c61 docs(architecture): sync migration guard docs (#6092) 2026-08-14 10:02:25 +08:00
Zhengchao An eb41f45175 chore(ecstore): drop the cluster and erasure dead_code blankets (#6088)
Removing both blankets exposes 23 items, of which only four are deleted. The ratio is the point: close to the core data path the blankets were hiding test assertions and migration seams, not dead code.

A cfg-split function is the reason two symbols in the internode transport look dead when neither is. build_internode_data_transport_from_env has two bodies, one under #[cfg(test)] that calls build_internode_data_transport directly and one under #[cfg(not(test))] that goes through the INTERNODE_DATA_TRANSPORT static so tests do not share process-global transport state. Each half's helper is live in exactly one build, and because cargo check --tests compiles both the lib target and the test harness, both symbols appear in one warning list. Deleting either one breaks the other lane. Both are kept with allows naming their half.

Three deletion candidates were withdrawn after a per-name grep: ParallelReader::new, ErasureDecodeReader::new and SyncErasureDecodeReader::new all have test callers. The last two are exactly the shape of the dead wrapper deleted in #6084 — a thin forward to a new_with_metrics_path sibling — except that sibling is live in production (set_disk/read.rs) and the wrappers are used by tests.

Deleted:

- RemotePeerS3Client::get_addr and RemoteLocker::from_url, neither with a consumer in any lane.
- RemotePeerS3Client's node field, which new writes after using it to derive addr and nothing ever reads. Its only other writer was a test helper that built a whole Node solely to fill the field; that block goes too.
- ParallelReader::can_decode, superseded by an inlined copy. The copy's comment named the method it replaced, so deleting the method alone would have left a dangling reference; the comment now describes the check instead of pointing at a method that no longer exists.

Kept with allows: the erasure items are decode/encode invariants asserted by their own files' tests (shard_read_launch_order, decode_with_read_costs, emit_data_shards, queued_block_bytes, the engine trait facets, the ParallelReader and decode-reader constructors, encode_stream_callback_async). On the cluster side, peer_replay_state, heal_bucket_local and clone_drives are test-only, InternodeDataTransportCapabilities and tcp_http are constructed only by transport test doubles, and the InternodeDataTransport trait's name/capabilities pair is an unused capability-negotiation facet kept for the transport split (backlog#1350) — six impls provide them and no caller negotiates on them yet.

Verification, four lanes warning-free: default, --tests, --features rio-v2 --tests, --features test-util --tests. cargo nextest run -p rustfs-ecstore 4041 passed; clippy --lib --tests -D warnings clean; make pre-commit exit 0.

Ref rustfs/backlog#1823 (step 2).
2026-08-14 00:57:12 +00:00
Zhengchao An 122d200675 ci: gate rio-v2 full-suite jobs on schedule; document lifecycle (#6036)
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-08-14 08:19:49 +08:00
Zhengchao An 161e515c72 chore(heal): remove seven dead error variants (#6031)
heal::Error carried six variants with zero construction and zero match sites (ConfigurationError, NotFound, TaskAlreadyExists, ManagerNotRunning, EventProcessingFailed, ProgressTrackingFailed) plus IO(String), which was never constructed either — its only appearances were two or-pattern match arms that could never fire (task.rs's demotion match and the recoverability classifier). All seven are deleted and the two or-patterns lose their dead alternative.

Config(String) stays (live, four construction sites); Io(std::io::Error) stays; the retry classifier's behavior is untouched per the issue constraint — removing an arm that can never match is not a classification change.

Ref rustfs/backlog#1831 (PR3).

Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
2026-08-14 08:18:47 +08:00
Zhengchao An 83cf063b45 chore(rustfs): drop two dead_code allows sitting on live code (#6081) 2026-08-14 08:14:22 +08:00
Zhengchao An f8bbfcbeb1 chore(ecstore): drop the io_support dead_code blanket (#6082) 2026-08-14 08:14:08 +08:00
Zhengchao An 710dcb4865 chore(ecstore): drop the layout dead_code blanket (#6084) 2026-08-14 08:13:44 +08:00
Zhengchao An f5cced910a chore(ecstore): drop the diagnostics dead_code blanket (#6083) 2026-08-14 08:13:19 +08:00
Zhengchao An 8c9249054f chore(ecstore): drop the runtime and error dead_code blankets (#6085) 2026-08-14 08:12:47 +08:00
Zhengchao An 7c2b513613 chore(obs): drop 44 dead_code blankets from the metrics tree (#6086) 2026-08-14 08:10:49 +08:00
Zhengchao An 00844721ff chore(ecstore): drop the config, core, data_movement, object_api and event blankets (#6087) 2026-08-14 08:09:56 +08:00
houseme 068a0c2b8c perf(ecstore): shorten multipart commit lock tail (#6080)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-14 04:48:05 +08:00
houseme 5b54c4303d fix(ecstore): reconcile object cleanup receipts (#6077)
* fix(s3): keep multipart completion publication owned

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(s3): keep put publication owned

Co-Authored-By: heihutu <heihutu@gmail.com>

* chore(app): route multipart context through facade

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ecstore): gate object transaction fencing

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ecstore): fence object transaction epochs

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ecstore): reconcile old data cleanup receipts

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-13 18:08:50 +00:00
houseme 6178083985 perf(ecstore): reuse erasure codecs on GET paths (#6074)
* perf(ecstore): share legacy SIMD workspaces

Reuse legacy Reed-Solomon encoder and decoder workspaces across Erasure instances with the same shard layout while keeping active codecs request-exclusive.

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(ecstore): reuse GET erasure shells and scratch buffers

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): satisfy concurrent codec lint

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(ecstore): bound cached legacy workspaces

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(ecstore): cap retained legacy codec memory

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-13 17:51:42 +00:00
houseme e16c07b9cd perf(ecstore): scale inline threshold by EC layout (#6075)
* perf(ecstore): scale inline threshold by EC layout

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): preserve inline budget semantics

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-13 17:00:31 +00:00
houseme 1ac28d6459 feat(ecstore): expose read version stage metrics (#6073)
Record local read_version path resolution, path length check, xl.meta read, and metadata decode durations through the existing GET stage metrics channel. The new samples are gated by GET stage metrics so metrics-off reads avoid timer and recorder work.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-13 16:33:46 +00:00
Zhengchao An 9b66040a02 refactor(sse): sink managed-SSE attribution into the shared encryption-keys module (#6017)
* refactor(sse): sink managed-SSE attribution into the shared encryption-keys module

Moves the managed-SSE classifier — stored_managed_encryption_key, contains_managed_encryption_metadata, normalize_managed_metadata — and the SSEType enum from rustfs/src/storage/sse.rs into crates/utils/src/http/object_encryption_keys.rs, the module that already owns every constant they read. This is PR-B0 of rustfs/backlog#1643: crates/scanner must never depend on the rustfs binary crate, so encryption attribution has to live in a shared lower layer before the scanner can report per-scheme coverage without growing a second classifier.

SSEType moves wholesale (option a): its only impl is the dependency-free audit_label(), so the enum relocates verbatim (audit_label becomes pub) and rustfs::storage::sse re-exports it, keeping every existing path compiling. The one piece that cannot move verbatim is normalize_managed_metadata's KMS-context branch, which needs base64 and serde_json — dependencies rustfs-utils does not have and does not gain here. The shared normalizer instead takes an injected Option<fn(&str) -> Option<String>> context recoder; sse.rs passes recode_minio_kms_context, the old inline chain verbatim including the silent skip on decode failure. stored_managed_encryption_key passes no recoder because the context mapping only ever inserts the context key, which the key-id lookup never reads, so its output is identical.

Every metadata lookup stays a case-sensitive exact match (lowercase x-amz-* stored forms, TitleCase MinIO-internal names) per the backlog#1775 trap; new shared-module tests pin that, and a source-scan test in sse.rs asserts the classifier has exactly one definition so a second copy cannot silently return.

* fix(utils): satisfy encryption key test clippy

---------

Co-authored-by: cxymds <cxymds@gmail.com>
2026-08-13 16:08:18 +00:00
Zhengchao An 7710f70fda feat(kms): report a key as due for rotation once its wrap budget is spent (#6059)
* fix(kms): construct wrap_budget_reserved in the VaultKeyData deserializer

main does not compile: #6019 added VaultKeyData.wrap_budget_reserved on a base that predated #6003's hand-written Deserialize, so the visitor's struct literal never learned about the field. Each PR was green on its own base; the breakage only exists in their merge.

The field joins the other three lists the hand-written impl maintains (Field enum, match arm, struct literal, FIELDS) and defaults to 0 when absent — the value a record written before wrap accounting, or rewritten by an older build, carries; zero restarts the reservation rather than blocking a wrap.

vault_key_data_deserializer_covers_every_serialized_field turns this class of mistake into a test failure instead of a merge-order accident: it serializes a fully populated record and asserts the deserializer recognizes every emitted key (unknown-field counter stays zero) and reads every value back. Mutation-verified by dropping the new match arm.

* feat(kms): report a key as due for rotation once its wrap budget is spent

The rotation readiness verdict only knew about age; the wrap accounting landed by #6019 counted wraps and published an aggregate gauge but never fed the per-key verdict, leaving the criterion backlog#1636 asks for unimplemented.

RUSTFS_KMS_ROTATION_MAX_WRAPS adds the second, independent threshold, parsed with the same discipline as the age one: unset or unparsable leaves the verdict unreported rather than inventing a policy, and values below one million are raised to it because wraps are reserved in blocks of that size and a smaller threshold would trip on the first reservation regardless of how many wraps happened.

The wrap check runs before the age check so that a key crossing both reports 'wraps': the AES-GCM random-nonce ceiling is a cryptographic bound an operator cannot negotiate, while the age period is a policy they chose. Backends that report no count — Transit and AWS wrap externally, and pre-accounting records carry nothing — leave the wrap half silent instead of guessing, and a backend that cannot rotate is still never told to.

Refs rustfs/backlog#1636 (PR-3 acceptance criterion), rustfs/backlog#1562.

---------

Co-authored-by: houseme <housemecn@gmail.com>
2026-08-13 23:21:37 +08:00
houseme aa4d3317ed perf(ecstore): guard inline data-read metadata early-stop (#6069)
Add a default-off inline-only data-read metadata early-stop gate that verifies inline plaintext before cancelling pending metadata tasks.

Keep non-inline, prepared, and request-shape-sensitive reads on full fanout, and record scheduled/completed/cancelled ReadVersion lifecycle metrics for normal fanout completion.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-13 21:40:14 +08:00
houseme f704d015d6 fix(copy): keep copy commit owner alive (#6070)
Keep S3 CopyObject's real outer owner task alive across caller cancellation so the source/destination bucket guards, same-key copy guard, storage commit, and post-commit publication hooks complete as one request-owned transaction boundary.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-13 20:19:33 +08:00
houseme 6b86d44cac fix(ecstore): retain commit owners across cancellation (#6068) 2026-08-13 18:08:58 +08:00
Zhengchao An e3c15f012c test(table-catalog): extract the shared avro/json fixture constructors (#6066)
The two table_catalog test files (27.5K lines combined) each maintained a parallel constructor stack for Iceberg metadata JSON and avro manifest-list/manifest bytes. Per the issue's adversarial ruling the parameterized admin variants are canonical (the store file hardcoded sequence 7 / snapshot 20); the two stacks were verified structurally identical first — schemas byte-equal, field lists and values aligned.

New #[cfg(test)] table_catalog/test_support.rs owns the seven constructors (metadata JSON, three manifest-list variants, two manifest variants, nullable_long). The admin tests import them under their old names; the store tests keep their historical signatures as thin delegates passing the fixed values explicitly — every produced byte is identical to the pre-extraction fixtures (the delegate's argument order was cross-checked against the canonical destructuring after an initial swap surfaced as five sequence-bound validation failures).

Ref rustfs/backlog#1837 (PR1).
2026-08-13 09:45:47 +00:00
houseme d2b1003612 perf(storage): converge Wave 2 hot-path optimizations (#6065)
* perf(get): share inline shards and lock clients

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(ecstore): converge PUT encoding on contiguous blocks

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(get): cache codec streaming gate config

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(sse): redact projected customer headers

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(ecstore): collapse GET metadata snapshots

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(ecstore): reuse decode stripe scratch

Co-Authored-By: heihutu <heihutu@gmail.com>

* refactor(ecstore): trim decode scratch adapters

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): adapt transition checks to metadata snapshots

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(get): release metadata snapshots at ownership boundary

Co-Authored-By: heihutu <heihutu@gmail.com>

* refactor(ecstore): close cumulative fast-path findings

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(storage): preserve lock and header invariants

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): adapt cumulative paths after rebase

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(rio-v2): adapt generated metadata fixture

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-13 16:34:28 +08:00
GatewayJ 36deab8670 perf(ecstore): retain remote shard HTTP chunks (#5991)
* perf(ecstore): retain remote shard HTTP chunks

* fix(ecstore): bound remote shard chunk retention

* fix(rio): persist empty chunk limit across polls
2026-08-13 15:00:44 +08:00
cxymds e11fcfbd08 fix(rebalance): converge multipart data movement retries (#6057)
* fix(rebalance): converge multipart data movement retries

* fix(rebalance): harden multipart retry replacement

* fix(rebalance): isolate internal multipart uploads

* test(ecstore): adapt metadata mutation fixtures

* fix(rebalance): preserve transition metadata semantics

* refactor(ecstore): reuse internal metadata matcher

* Revert "refactor(ecstore): reuse internal metadata matcher"

This reverts commit c87ca0328f.

* refactor(rebalance): reuse data movement log constants

* fix(rebalance): isolate migration-owned state

* fix(rebalance): preserve pre-gate retry compatibility
2026-08-13 06:12:26 +00:00
houseme 11eecdc888 perf(put): avoid eager body zero fill (#6063)
Use BytesMut spare capacity for direct and pooled small PUT body reads while preserving exact-length validation.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-13 05:30:00 +00:00
houseme 80eb4244a3 chore(deps): refresh mimalloc revision (#6062) 2026-08-13 12:31:01 +08:00
Zhengchao An e4da9bd718 refactor(rustfs): move layer-neutral shared types out of server (#6061)
RemoteAddr, the DependencyReadiness family (DependencyReadiness, ReadinessDegradedReason, DependencyReadinessReport), and convert_ecstore_object_info (with its offset_date_time_to_timestamp helper) are consumed across app, infra, and interface layers but lived under server, so every lower-layer import was an upward app->interface or infra->interface edge the layer guard had to baseline.

They now live in a new layer-neutral rustfs/src/shared_types.rs (classified infra by the guard, making all consumer imports downward or lateral). server::readiness and server::event re-export for their own internals; the eight consumer sites (admin_usecase, bucket_usecase, object_usecase, cluster_snapshot, storage/access, storage/helper, plus the admin handler tests) import from the new home. Pure move: no type, impl, or behavior change.

The regenerated layer-dependency baseline shrinks by exactly eight lines with zero additions — the ratchet's intended direction. The two remaining readiness entries (collect/snapshot fn imports) need the collection machinery itself extracted from server and are left for the issue's PR5 scope.

Ref rustfs/backlog#1834 (PR4).
2026-08-13 12:29:49 +08:00
Zhengchao An e28430ab3d test(rustfs): un-ignore the fourteen ecfs_test global-state tests (#6046)
The 14 tests carried #[ignore = "requires isolated global object layer state"], and the only CI lane that runs ignored tests filters for lifecycle tests — so they executed nowhere. Under nextest, the authoritative runner, every test owns its process and the stale reason no longer applies; all 14 pass.

Ten of them assert the InternalError path taken while the global object layer is uninitialized, a premise a sibling test can destroy under the documented shared-process cargo test fallback. Those ten now start with an explicit premise guard: when a sibling already initialized the store the test skips with a message instead of asserting against a scenario it does not describe. Under nextest the guard never fires and the assertions always run.

Dual-runner evidence: nextest 79 passed; cargo test module-scoped 79 passed; the full storage-tree cargo test sweep returns to its pre-existing baseline (8 unrelated in-process failures, none introduced or worsened here). No test deleted.

Ref rustfs/backlog#1830 (PR1).
2026-08-13 12:29:06 +08:00
Zhengchao An db4707f187 chore(io-metrics): make the server label injected, drop two leaf-violating deps (#6051) 2026-08-13 03:20:18 +00:00
houseme 3a0dbccc2e perf(ecstore): reduce inline PUT commit overhead (#6033)
* perf(metrics): attribute PUT stage costs

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(ecstore): move PUT metadata during shuffle

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(s3): reuse PUT object lock state

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(ecstore): trim PUT metadata fanout clones

Build per-disk PUT metadata only for committed writer slots, move the response metadata out of the fanout vector, and preserve fresh FileInfo shuffle semantics.

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(metrics): make PUT stage attribution opt-in

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(ecstore): commit inline PUT shards directly

Co-Authored-By: heihutu <heihutu@gmail.com>

* perf(ecstore): streamline rename staging cleanup

Use the directory-specific removal operation for rename_data staging parents. This avoids a guaranteed failed file-removal probe on Unix-like hosts and lets Windows remove the empty directory directly while preserving best-effort non-empty handling.

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): cover inline PUT rename failures

Cache the detailed stage metrics gate once per PUT and exercise exact-quorum and quorum-minus-one failures after inline shard encoding.

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-13 02:04:20 +00:00
houseme 846517625b fix(ecstore): bound remote metadata read retries (#6035)
* fix(ecstore): bound remote metadata read retries

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(ecstore): handle oversized read timeouts

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-13 02:01:05 +00:00
houseme f21e88b112 perf(ecstore): avoid per-block shard vector allocation (#6037)
Keep encoded shards in one contiguous Bytes buffer while they cross the streaming write queue, and materialize Vec<Bytes> only for the existing public APIs.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-13 01:55:24 +00:00
Zhengchao An a5594c3d89 fix(kms): construct wrap_budget_reserved in the VaultKeyData deserializer (#6055) 2026-08-13 00:59:13 +00:00
Zhengchao An fc927caadd docs(guards): make the layer-dependency baseline an explicit shrink-only ratchet (#6053) 2026-08-13 00:27:19 +00:00
Henry Guo b7e6334c13 fix(table-catalog): isolate metadata after table rename (#6050) 2026-08-13 00:15:31 +00:00
Zhengchao An 65091aa6a8 test: give the twenty-one bare #[ignore] attributes their reasons (#6049) 2026-08-13 08:10:47 +08:00
Zhengchao An 3c78a56ab0 test: un-ignore the remaining seven global-state tests, drop one stale premise (#6048) 2026-08-13 08:10:28 +08:00
Zhengchao An bdd7ecd205 test(rustfs): un-ignore the nine node_service global-state tests (#6047) 2026-08-13 08:10:11 +08:00
Zhengchao An a70a3787d8 fix(kms): tell an empty KV2 prefix from a missing mount on Vault's 404 (#6043) 2026-08-13 08:09:17 +08:00
Zhengchao An b2ae430805 chore(ecstore): compile the list-objects chaos injector out of production builds (#6042) 2026-08-13 08:09:03 +08:00
Zhengchao An 299eb0d965 docs(operations): record the two experimental GET-path switches (#6041) 2026-08-13 08:08:48 +08:00
Zhengchao An f5a780099b fix(kms): let the Vault Transit backend start against an empty transit engine (#6040) 2026-08-13 08:08:30 +08:00
Zhengchao An 5cfafcf39b chore(rustfs): remove the orphan starshard bucket-cache backend (#6038) 2026-08-13 08:08:15 +08:00
Zhengchao An a49243c671 test(kms): pin ILM behavior on SSE-KMS buckets under key-policy enforcement (#6027) 2026-08-13 08:06:42 +08:00
Zhengchao An c2a15f5214 refactor(utils): add shared retry_with_backoff and migrate target_descriptor (#6026) 2026-08-13 08:06:21 +08:00
Zhengchao An ee54f1e618 docs(checksums): cross-reference the three checksum registries (#6024) 2026-08-13 08:06:00 +08:00
Zhengchao An e6b85b60a8 test(io-metrics): assert metric emission in six modules of record_* smoke tests (#6021) 2026-08-13 08:05:43 +08:00
Zhengchao An 8c1e3c09ff refactor(admin): add authorize_admin_request and fold four local wrappers (#6020) 2026-08-13 08:05:28 +08:00
Zhengchao An ca06c7ec2c feat(kms): reserve and expose KV2 wrap-budget consumption (#6019) 2026-08-13 08:05:00 +08:00
Zhengchao An 4a41325d1a feat(sse): report the wrapping master-key version on S3 audit entries (#6005) 2026-08-13 08:04:30 +08:00
Zhengchao An 45e2bd0c28 chore(ecstore): collapse the expiry worker knobs to one documented env var (#6034)
init_background_expiry resolved its worker count through three env vars, none documented, none set in any known deployment: RUSTFS_MAX_EXPIRY_WORKERS, silently overridden by the underscore-prefixed _RUSTFS_ILM_EXPIRATION_WORKERS (a MinIO fossil, comment included), with a zero value then falling through to RUSTFS_DEFAULT_EXPIRY_WORKERS.

RUSTFS_MAX_EXPIRY_WORKERS stays as the canonical name per the rc constraint (no new env var names): the count is now resolved once — a set, parseable, non-zero value wins, anything else falls back to min(cpus, 16). The constant moves to rustfs-config's runtime constants alongside ENV_TRANSITION_WORKERS, the two dead names are gone repo-wide (rg-verified), and a serial four-state unit test (unset/zero/valid/garbage) pins the resolution, modeled on the transition-worker env harness.

Ref rustfs/backlog#1832 (PR2).
2026-08-12 20:46:06 +00:00
Zhengchao An e2fb0427f9 docs(io-metrics): remove stale unified-config docs left by #6008 (#6039)
PR #6008 deleted crates/io-metrics/src/config.rs but the crate docs still taught the deleted API: both READMEs kept the Unified Configuration sections, module-tree entries and ./src/config.rs links, the example kept an orphaned numbered comment, and the io-core/io-metrics changelog had no removal record. Scrub all of it; keep cache_config.rs references, which are still real.
2026-08-12 20:41:40 +00:00
Zhengchao An a825326ede test(e2e): fold seven identical POST-policy exact-mismatch tests into one table (#6016)
* test(e2e): fold seven identical POST-policy exact-mismatch tests into one table

The seven *_policy_mismatch tests in multipart_auth_test.rs were body-identical after literal normalization: the policy pins one field to an exact value, the form sends a different value, and the upload must be rejected with 400 InvalidPolicyDocument naming the field. Each test booted its own full server.

They fold into one table-driven test on the run_post_object_policy_case helper introduced by the PR1 fold. Every row keeps its original test's exact bucket, key, field name, policy value, mismatched form value, body bytes, and expected error strings — including the three rows that asserted the stronger <Code>InvalidPolicyDocument</Code> form. The pinned condition is built with an explicit serde_json::Map since the field name is now a table parameter.

cargo nextest list reports 97 tests for this module; the inventory row is updated in the same diff (103 -> 97).

Ref rustfs/backlog#1838 (PR2).

* test(e2e): fold the remaining POST-policy duplicate groups (15 tests) (#6018)

Completes the multipart_auth table-driven fold: the six remaining body-identical groups collapse onto the shared run_post_object_policy_case helper.

- Seven single-field exact-mismatch tests (cache-control, expires, tagging, storage-class, content-type, success_action_status, metadata-field-exact) join the existing exact-condition mismatch table as rows — same shape as the PR2 fold.
- The two object-lock mismatch tests become a two-row table (policy pins mode + retain-until-date, one form field mismatches).
- The three SSE-KMS parameter mismatch tests become a three-row table (policy pins the SSE mode plus one KMS parameter, form differs).
- The three SSE-KMS outside-policy tests become a three-row table pinning the distinct contract: an undeclared KMS parameter sails past policy validation and is rejected at runtime with 501 NotImplemented, not a policy error.

Every row keeps its original test's exact bucket, key, field names, values, body bytes, and expected status/code strings. cargo nextest list reports 85 tests for the module; the inventory row is updated in the same diff (97 -> 85).

Ref rustfs/backlog#1838 (PR3).
2026-08-12 20:18:36 +00:00
Zhengchao An e313276e49 fix(sse): align copy-path unknown-algorithm fallback with put path (#6022) 2026-08-13 03:42:03 +08:00
Zhengchao An 66af487978 docs(policy): pin the deliberate slash-only path.Clean duplication (#6013)
The policy crate's Go path.Clean port and rustfs-utils' Windows-aware clean look like duplicates but are not interchangeable: S3 ARN/resource matching must treat backslashes as object-name data, never as separators, so adopting the utils version would change policy evaluation semantics on Windows — a security-adjacent behavior change. Record that judgment as bidirectional do-not-merge notes on both implementations, per the issue's adversarial ruling.

Comment-only change.

Ref rustfs/backlog#1833 (PR7).
2026-08-13 03:41:09 +08:00
Zhengchao An d668a9293f chore(ecstore): remove test-only BitrotErrorType and pin wire-only disk variants (#6032)
BitrotErrorType (disk/error.rs) was constructed only by its own unit test: production bitrot mismatches never flow through it (they surface as DiskError::other strings). Delete the enum, its From<BitrotErrorType> for DiskError impl, the self-test, and the api facade re-export. The facade inventory doc does not name the type, so no doc change is needed.

DiskError::SourceStalled and DiskError::CrossDeviceLink are never constructed locally — they are reachable only through wire decoding and no current node sends them. Their decode arms stay per the cross-version compatibility constraint; each variant now carries a doc comment saying exactly that so the next dead-code sweep does not re-litigate them. Their consumer arms (heal classifier, batch processor) are left untouched — the values cannot appear, so removing the arms would be unobservable, and the heal classifier is pinned by the issue as do-not-touch.

Ref rustfs/backlog#1831 (PR4).
2026-08-12 19:37:00 +00:00
Zhengchao An ace28c1f85 chore(common): remove dead bucket_stats module and LastMinuteHistogram (#6011)
crates/common/src/bucket_stats.rs (ReplicationLatency plus a commented-out ReplicationLastMinute corpse) had zero consumers anywhere in the workspace — the live replication statistics implementation is crates/replication/src/stats.rs. LastMinuteHistogram in last_minute.rs (already carrying allow(dead_code)) was equally unreferenced, and size_to_tag / SIZE_LAST_ELEM_MARKER had no user besides the histogram, so the whole block goes with it. LastMinuteLatency and AccElem stay: common's metrics.rs uses them.

Ref rustfs/backlog#1833 (PR5).
2026-08-13 03:27:17 +08:00
Zhengchao An 2ad8ab534e fix(site-replication): admit same-generation peer-edit fan-out bodies (#6007)
The peer-edit delivery fence from #5882 treated an equal applied generation as stale. One edit legitimately fans out one delivery per peer record under a single generation (the ILM-expiry edit sends every peer's record), so the receiver applied only the first body, raised its high-water mark, and silently acked-success while dropping the rest — enableILMExpiryReplication never converged on receiving sites and the three-node nightly e2e failed deterministically (issue #5767).

Only a strictly newer applied generation is stale now. Equal generation implies the same logical edit and re-applying a delivery is idempotent (update_peer overwrites the peer record; the mark is raised with max), while strictly older deliveries — the cross-node ordering case the fence exists for — stay rejected.

Adds a composed unit test driving three same-generation bodies through the receiver's fenced sequence, and widens the replication e2e's two site-replication wait helpers from a 10s polling ceiling to the 30s deadline the file's other waits use.
2026-08-13 03:26:34 +08:00
Zhengchao An f7df4fa62a fix(versioning): reject suspending versioning while a replication config exists (#6006)
PutBucketVersioning with Status=Suspended on a bucket that carries a replication configuration now fails with InvalidBucketState, matching AWS S3 and MinIO. Suspension would start minting null versions that the versioned replication engine can never converge — the state is unreachable on AWS and MinIO, and the nightly acceptance-matrix e2e that tried to exercise it failed every night since it landed (issue #5767).

The acceptance-matrix test tail now pins the rejection contract (InvalidBucketState) and verifies a fresh matched PUT still replicates with a real version id after the rejected suspension.
2026-08-13 03:26:17 +08:00
Zhengchao An ca4e66daab chore(io-metrics): remove the zero-consumer config module (391 lines) (#6008)
crates/io-metrics/src/config.rs was a near-copy of io-core's Backpressure/Deadlock configuration with already-drifted field names (high_watermark vs io-core's high_water_mark) and had no consumer outside the crate's own example: the canonical BackpressureConfig lives in crates/io-core/src/backpressure.rs. Delete the module, its lib.rs re-exports, and the example's unified-config section, and settle the corresponding ARCHITECTURE.md ledger line that tracked this copy's removal.

Ref rustfs/backlog#1833 (PR4).
2026-08-13 03:25:30 +08:00
Zhengchao An 5b9c5289c2 fix(iam): remove eight dead error variants and make Clone variant-preserving (#6030)
* chore(iam): remove eight dead error variants

iam::Error mirrored policy::Error variant-for-variant, and eight of the twins had zero construction and zero match sites anywhere in the workspace: InvalidServiceType, ErrCredMalformed, CredNotInitialized, JWTError, NoAccessKey, InvalidToken, InvalidAccessKey, InvalidExpiration (each verified by repo-wide sweep; the InvalidToken hits elsewhere are KeystoneError's unrelated variant). Delete the variants along with their Clone and PartialEq arms.

The From<policy::Error> mapping keeps its exhaustive match: the eight orphaned arms now route through a grouped binding to Error::StringError(err.to_string()), so the rendered message is preserved; nothing could observe the old discriminants because no site ever matched on them.

Ref rustfs/backlog#1831 (PR1).

* fix(iam): make Error clone variant-preserving via Arc payloads

iam::Error's hand-written Clone demoted PolicyError and CryptoError to StringError because their payloads are not cloneable — a clone changed the variant identity. There is no production clone site today (the issue's refuter confirmed this is preventive hardening, not a live bug), but any future holder of a cloned error would match the wrong variant.

The two payloads are now Arc-wrapped, so Clone is a cheap reference bump that keeps the variant. Display strings are unchanged ({0} and crypto: {0}); the #[from] derives become manual From impls wrapping in Arc; the one behavioral trade-off is that source() is no longer forwarded for these two variants (Arc<E> does not implement std::error::Error), which nothing in the workspace consumed. A regression test pins discriminant and rendered message across clone for the hard-to-clone variants.

Ref rustfs/backlog#1831 (PR2).
2026-08-12 19:14:07 +00:00
Zhengchao An 3f9b84ec70 feat(kms): observe unknown fields in the last three silent persisted formats (#6003) 2026-08-13 03:08:49 +08:00
houseme 73bd5d9d95 perf(get): reduce request entry allocations (#6029)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-12 19:07:59 +00:00
houseme 398d2d87c8 fix(ecstore): retry manual ILM job CAS updates (#6012)
Co-authored-by: heihutu <heihutu@gmail.com>
2026-08-12 18:40:10 +00:00
256 changed files with 22764 additions and 6283 deletions
+1 -1
View File
@@ -85,7 +85,7 @@ runs:
repo-token: ${{ github.token }}
- name: Install flatc
uses: Nugine/setup-flatc@e7855e994773ce90094a3f1626d4afc9080c23ae # v1
uses: Nugine/setup-flatc@698800de72a96bfb22cf60431dc21a2ff9a7e07b # v1
with:
version: "25.12.19"
+6 -1
View File
@@ -182,7 +182,12 @@ jobs:
echo '```'
} >> "$GITHUB_STEP_SUMMARY"
# Readers: test-and-lint-rio-v2, build-rustfs-debug-binary-rio-v2.
# Readers: test-and-lint-rio-v2 (per-PR), build-rustfs-debug-binary-rio-v2
# (weekly schedule / manual dispatch only — dormant rio-v2 variant, see
# rustfs/backlog#1835 and docs/architecture/minio-file-format-compat.md).
# The second build below stays despite the reduced cadence: it warms the
# rio-v2,e2e-test-hooks feature resolution the scheduled build restores,
# which keeps that lane inside its 30-minute timeout.
warm-ci-feat-rio:
name: Warm ci-feat-rio
runs-on: sm-standard-4
+9 -1
View File
@@ -533,7 +533,12 @@ jobs:
build-rustfs-debug-binary-rio-v2:
name: Build RustFS Debug Binary (rio-v2)
if: github.event_name != 'pull_request' || github.event.action != 'closed'
# Dormant rio-v2 variant (rustfs/backlog#1835): the feature ships in no
# default build, so this full-suite lane runs only on the weekly schedule
# and manual dispatch. Per-PR cfg-seam coverage stays with
# test-and-lint-rio-v2. Lifecycle and the promote-or-delete condition:
# docs/architecture/minio-file-format-compat.md ("rio-v2 variant lifecycle").
if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
needs: [ quick-checks ]
runs-on: sm-standard-4
timeout-minutes: 30
@@ -824,6 +829,9 @@ jobs:
e2e-tests-rio-v2:
name: End-to-End Tests (rio-v2)
# Inherits the schedule/dispatch-only gate through needs: on every other
# event build-rustfs-debug-binary-rio-v2 is skipped, so this job skips
# with it (see the dormant-variant comment on that job).
needs: [ build-rustfs-debug-binary-rio-v2 ]
runs-on: sm-standard-2
timeout-minutes: 30
+7 -4
View File
@@ -101,7 +101,10 @@ refactors.
The `rustfs` binary crate composes these libraries into the running server.
`ecstore` remains the storage engine at the architectural center; its internal
module split is tracked under `docs/architecture/`.
module split is tracked under `docs/architecture/`. `rio-v2` is the
feature-gated MinIO on-disk format compatibility I/O layer; it ships in no
default build (lifecycle:
[docs/architecture/minio-file-format-compat.md](docs/architecture/minio-file-format-compat.md)).
## Architecture Invariants
@@ -131,9 +134,9 @@ module split is tracked under `docs/architecture/`.
why it stays local).
- ✅ RESOLVED: `BackpressureConfig` and `DataUsageInfo` each have exactly one
definition (`crates/io-core/src/backpressure.rs`,
`crates/data-usage/src/data_usage.rs`). A zero-consumer
`BackpressureSettings` copy lingers in `crates/io-metrics/src/config.rs`;
its removal is tracked in rustfs/backlog#1833.
`crates/data-usage/src/data_usage.rs`). The zero-consumer
`BackpressureSettings` copy that lingered in io-metrics was removed
(rustfs/backlog#1833).
4. **ecstore does not know about HTTP or S3 protocol details.** It operates on
storage-level abstractions (objects, buckets, disks, pools).
Generated
+39 -31
View File
@@ -1162,9 +1162,9 @@ dependencies = [
[[package]]
name = "aws-smithy-eventstream"
version = "0.61.1"
version = "0.61.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "5a9381123ab62d20c13082b151f30f962a3b112b727345394536dfa39a482944"
checksum = "6de526c7b567420a31bc283657a7921b45c4cafe0827fdf2490713dcc770c28f"
dependencies = [
"aws-smithy-types",
"bytes",
@@ -1195,9 +1195,9 @@ dependencies = [
[[package]]
name = "aws-smithy-http-client"
version = "1.2.0"
version = "1.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "635d23afda0a6ab48d666c4d447c4873e8d1e83518a2be2093122397e50b838e"
checksum = "3c1c8a04cb31ba74d0115af5a890bb8c0d48fba64b52812fa13929a6ef0cc83c"
dependencies = [
"aws-smithy-async",
"aws-smithy-protocol-test",
@@ -1277,9 +1277,9 @@ dependencies = [
[[package]]
name = "aws-smithy-runtime"
version = "1.12.1"
version = "1.13.1"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "07505b34e8f4b3591a4fa69e9792b52289b95488dbbc68c3c0075b7bedb245e1"
checksum = "483b858ff67522011c4786310c5cd8fd88d0be7ea3d5f1a48328446300c4269e"
dependencies = [
"aws-smithy-async",
"aws-smithy-http",
@@ -1343,9 +1343,9 @@ dependencies = [
[[package]]
name = "aws-smithy-types"
version = "1.6.1"
version = "1.6.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d6dc683efb34b9e755675b37fedbe0103141e5b6df7bdc9eb6967756a8c167d8"
checksum = "fce83ce9abbb198d25bc7131e468d0f9fe1257125e58c39f3f9fc9f5098c9647"
dependencies = [
"base64-simd",
"bytes",
@@ -5133,9 +5133,9 @@ dependencies = [
[[package]]
name = "http-body-util"
version = "0.1.4"
version = "0.1.5"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e9f41fd6a08e4d4ec69df65976da761afd5ad5e58a9d4acb46bd1c953a9e3ff2"
checksum = "23169fe34a5fbcdd3f3862e78fb9b6fccd5f02a6dc6f732547005d45631ce71c"
dependencies = [
"bytes",
"futures-core",
@@ -5957,7 +5957,7 @@ checksum = "b6d2cec3eae94f9f509c767b45932f1ada8350c4bdb85af2fcab4a3c14807981"
[[package]]
name = "libmimalloc-sys"
version = "0.1.49"
source = "git+https://github.com/xonatius/mimalloc_rust.git?rev=ce6338661179c8be22e516b00af7483f151485a7#ce6338661179c8be22e516b00af7483f151485a7"
source = "git+https://github.com/xonatius/mimalloc_rust.git?rev=6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11#6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11"
dependencies = [
"cc",
"cty",
@@ -6259,9 +6259,9 @@ dependencies = [
[[package]]
name = "metrique"
version = "0.1.29"
version = "0.1.30"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d2e394c63e2d1a30aeb3b9392ecf3439d8475d2df810a8f4f6e66d6866754017"
checksum = "dedbf06ffeef4c37990c73636fbd993aa34fb1948afd736e6114f239220993db"
dependencies = [
"itoa",
"jiff",
@@ -6289,9 +6289,9 @@ dependencies = [
[[package]]
name = "metrique-macro"
version = "0.1.20"
version = "0.1.21"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "786df1fd0abebd0db685f7e9a353c78756d4b370fb98a52376c2015fa55f141f"
checksum = "f4fb1f30185f53f7f6e4c9e46745c1a1350af8e77fda5a88aded44b0637a82e0"
dependencies = [
"Inflector",
"darling 0.23.0",
@@ -6318,9 +6318,9 @@ checksum = "2faca4e4480069ff02b1763b3b79f5cec7e8628e24d9dc5b6073f53d2577a4d9"
[[package]]
name = "metrique-writer"
version = "0.1.25"
version = "0.1.26"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "82cdde44d241dab7fc8b7a32e0eb5dae6cd28f8de80b59f9a1e9f2f0b05e485e"
checksum = "20bd17c1a3ca2719e31f19ce77a853948dc2102f35976b92276c42a64fdc5f3f"
dependencies = [
"ahash",
"crossbeam-queue",
@@ -6339,9 +6339,9 @@ dependencies = [
[[package]]
name = "metrique-writer-core"
version = "0.1.19"
version = "0.1.20"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e57379b7ee2272efaeaaa6de062503563e57333b24aadc7f2255b3d602899e8b"
checksum = "f1a55b6aae1d85c557c729564c4e2b32a26dc65ba2d90d9647ca01f2bd4854c4"
dependencies = [
"derive-where",
"itertools 0.14.0",
@@ -6366,7 +6366,7 @@ dependencies = [
[[package]]
name = "mimalloc"
version = "0.1.52"
source = "git+https://github.com/xonatius/mimalloc_rust.git?rev=ce6338661179c8be22e516b00af7483f151485a7#ce6338661179c8be22e516b00af7483f151485a7"
source = "git+https://github.com/xonatius/mimalloc_rust.git?rev=6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11#6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11"
dependencies = [
"libmimalloc-sys",
]
@@ -6882,7 +6882,7 @@ version = "5.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "51e219e79014df21a225b1860a479e2dcd7cbd9130f4defd4bd0e191ea31d67d"
dependencies = [
"base64 0.22.1",
"base64 0.21.7",
"chrono",
"getrandom 0.2.17",
"http 1.5.0",
@@ -8043,7 +8043,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "be769465445e8c1474e9c5dac2018218498557af32d9ed057325ec9a41ae81bf"
dependencies = [
"heck 0.5.0",
"itertools 0.14.0",
"itertools 0.10.5",
"log",
"multimap",
"once_cell",
@@ -8063,7 +8063,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "03da047801ff44bb6a4d407d4860c05fd70bb81714e6b2f3812603d5b145b042"
dependencies = [
"heck 0.5.0",
"itertools 0.14.0",
"itertools 0.10.5",
"log",
"multimap",
"petgraph 0.8.3",
@@ -8084,7 +8084,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8a56d757972c98b346a9b766e3f02746cde6dd1cd1d1d563472929fdd74bec4d"
dependencies = [
"anyhow",
"itertools 0.14.0",
"itertools 0.10.5",
"proc-macro2",
"quote",
"syn 2.0.119",
@@ -8097,7 +8097,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "b570b25f7617e43d59005d0990ccb79e950a423952cea19671b7a876da390adf"
dependencies = [
"anyhow",
"itertools 0.14.0",
"itertools 0.10.5",
"proc-macro2",
"quote",
"syn 2.0.119",
@@ -9201,7 +9201,6 @@ dependencies = [
"sha2 0.11.0",
"shadow-rs",
"socket2",
"starshard",
"subtle",
"sysinfo",
"temp-env",
@@ -9621,9 +9620,7 @@ dependencies = [
"metrics",
"metrics-util",
"num_cpus",
"rustfs-common",
"rustfs-s3-ops",
"rustfs-utils",
"sysinfo",
"thiserror 2.0.20",
"tokio",
@@ -9737,6 +9734,7 @@ dependencies = [
"rustfs-utils",
"rustify",
"serde",
"serde_ignored",
"serde_json",
"sha2 0.11.0",
"subtle",
@@ -10561,9 +10559,9 @@ dependencies = [
[[package]]
name = "rustls-connector"
version = "0.23.7"
version = "0.23.8"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "09a5abe04eec18f8b9fbe87885bcaee6426de80bbc579958c0bc064b728ee617"
checksum = "1babecfcc65b139b812e74bcc7f9ec7b4e00db659fd42d99567b7e77f0c714c6"
dependencies = [
"futures-io",
"futures-rustls",
@@ -10939,6 +10937,16 @@ dependencies = [
"syn 3.0.3",
]
[[package]]
name = "serde_ignored"
version = "0.1.14"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "115dffd5f3853e06e746965a20dcbae6ee747ae30b543d91b0e089668bb07798"
dependencies = [
"serde",
"serde_core",
]
[[package]]
name = "serde_json"
version = "1.0.151"
@@ -11792,7 +11800,7 @@ source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "32497e9a4c7b38532efcdebeef879707aa9f794296a4f0244f6f69e9bc8574bd"
dependencies = [
"fastrand",
"getrandom 0.4.3",
"getrandom 0.3.4",
"once_cell",
"rustix",
"windows-sys 0.61.2",
+7 -6
View File
@@ -41,7 +41,7 @@ members = [
"crates/protocols", # Protocol implementations (FTPS, SFTP, etc.)
"crates/protos", # Protocol buffer definitions
"crates/rio", # Rust I/O utilities and abstractions
"crates/rio-v2", # Next-generation Rust I/O compatibility layer
"crates/rio-v2", # MinIO on-disk format compatibility I/O layer (feature-gated, ships in no default build)
"crates/replication", # Replication contracts and wire formats
"crates/concurrency", # Concurrency management for RustFS - timeout, locking, backpressure, and I/O scheduling
"crates/s3-types", # S3 event type definitions
@@ -154,7 +154,7 @@ hyper-rustls = { default-features = false, version = "0.27.9" }
hyper-util = { version = "0.1.20" }
http = "1.5.0"
http-body = "1.1.0"
http-body-util = "0.1.4"
http-body-util = "0.1.5"
minlz = "1.2.3"
reqwest = "0.13.4"
rustfs-kafka-async = { version = "1.2.0" }
@@ -182,6 +182,7 @@ quick-xml = "0.41.0"
rmp = { version = "0.8.15" }
rmp-serde = { version = "1.3.1" }
serde = { version = "1.0.229" }
serde_ignored = { version = "0.1" }
serde_json = { version = "1.0.151" }
serde_urlencoded = "0.7.1"
@@ -230,9 +231,9 @@ aws-credential-types = { version = "1.3.0" }
aws-sdk-kms = { default-features = false, version = "1.114.0" }
aws-sdk-s3 = { default-features = false, version = "1.141.0" }
aws-sdk-sts = { default-features = false, version = "1.110.0" }
aws-smithy-http-client = { default-features = false, version = "1.2.0" }
aws-smithy-http-client = { default-features = false, version = "1.3.0" }
aws-smithy-runtime-api = { version = "1.14.0" }
aws-smithy-types = { version = "1.6.1" }
aws-smithy-types = { version = "1.6.2" }
base64 = "0.23.1"
base64-simd = "0.8.0"
brotli = "8.0.4"
@@ -347,8 +348,8 @@ russh-sftp = "2.4.0"
dav-server = "0.11.0"
# Performance Analysis and Memory Profiling
mimalloc = { version = "0.1.52", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "ce6338661179c8be22e516b00af7483f151485a7" }
libmimalloc-sys = { version = "0.1.49", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "ce6338661179c8be22e516b00af7483f151485a7", features = ["extended"] }
mimalloc = { version = "0.1.52", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11" }
libmimalloc-sys = { version = "0.1.49", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11", features = ["extended"] }
hotpath = { version = "0.23.2", default-features = false }
# Snapshot testing for output format regression detection
insta = { version = "1.48" }
+7
View File
@@ -21,6 +21,13 @@ use crate::{
Xxhash3, Xxhash64, Xxhash128,
};
// DELIBERATE DUPLICATION of the x-amz-checksum-* names that also exist as
// AMZ_CHECKSUM_* in rustfs-utils' headers module (crates/utils/src/http/
// headers.rs): this crate is a zero-internal-dependency leaf, so it cannot
// import them, and it additionally owns the RustFS extension names
// (sha512/xxhash*) that utils does not carry. Values are pinned by the S3
// wire protocol; do not merge without a maintainer decision on the leaf
// boundary (backlog#1833).
pub const CRC_32_HEADER_NAME: &str = "x-amz-checksum-crc32";
pub const CRC_32_C_HEADER_NAME: &str = "x-amz-checksum-crc32c";
pub const SHA_1_HEADER_NAME: &str = "x-amz-checksum-sha1";
+8
View File
@@ -41,6 +41,14 @@ pub const XXHASH_64_NAME: &str = "xxhash64";
pub const XXHASH_128_NAME: &str = "xxhash128";
pub const MD5_NAME: &str = "md5";
/// One of three deliberately separate checksum registries (backlog#1833):
/// this enum owns the **streaming-hash algorithm registry**, including the
/// RustFS extensions (sha512, xxhash3/64/128). The on-disk xl.meta bitset
/// lives in `rustfs_rio::ChecksumType` (crates/rio/src/checksum.rs, varint
/// bits are append-only), and the MinIO-port client keeps its own
/// `ChecksumMode` (crates/ecstore/src/client/checksum.rs). When adding an
/// algorithm, extend all three (or record why not) — they do not derive from
/// each other.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
#[non_exhaustive]
pub enum ChecksumAlgorithm {
-87
View File
@@ -1,87 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::last_minute::{self};
use std::collections::HashMap;
pub struct ReplicationLatency {
// Delays for single and multipart PUT requests
upload_histogram: last_minute::LastMinuteHistogram,
}
impl ReplicationLatency {
// Merge two ReplicationLatency
pub fn merge(&mut self, other: &mut ReplicationLatency) -> &ReplicationLatency {
self.upload_histogram.merge(&other.upload_histogram);
self
}
// Get upload delay (categorized by object size interval)
pub fn get_upload_latency(&mut self) -> HashMap<String, u64> {
let mut ret = HashMap::new();
let avg = self.upload_histogram.get_avg_data();
for (i, v) in avg.iter().enumerate() {
let avg_duration = v.avg();
ret.insert(self.size_tag_to_string(i), avg_duration.as_millis() as u64);
}
ret
}
pub fn update(&mut self, size: i64, during: std::time::Duration) {
self.upload_histogram.add(size, during);
}
// Simulate the conversion from size tag to string
fn size_tag_to_string(&self, tag: usize) -> String {
match tag {
0 => String::from("Size < 1 KiB"),
1 => String::from("Size < 1 MiB"),
2 => String::from("Size < 10 MiB"),
3 => String::from("Size < 100 MiB"),
4 => String::from("Size < 1 GiB"),
_ => String::from("Size > 1 GiB"),
}
}
}
// #[derive(Debug, Clone, Default)]
// pub struct ReplicationLastMinute {
// pub last_minute: LastMinuteLatency,
// }
// impl ReplicationLastMinute {
// pub fn merge(&mut self, other: ReplicationLastMinute) -> ReplicationLastMinute {
// let mut nl = ReplicationLastMinute::default();
// nl.last_minute = self.last_minute.merge(&mut other.last_minute);
// nl
// }
// pub fn add_size(&mut self, n: i64) {
// let t = SystemTime::now()
// .duration_since(UNIX_EPOCH)
// .expect("Time went backwards")
// .as_secs();
// self.last_minute.add_all(t - 1, &AccElem { total: t - 1, size: n as u64, n: 1 });
// }
// pub fn get_total(&self) -> AccElem {
// self.last_minute.get_total()
// }
// }
// impl fmt::Display for ReplicationLastMinute {
// fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
// let t = self.last_minute.get_total();
// write!(f, "ReplicationLastMinute sz= {}, n= {}, dur= {}", t.size, t.n, t.total)
// }
// }
-41
View File
@@ -572,44 +572,3 @@ mod tests {
assert_eq!(total.n, 6);
}
}
const SIZE_LAST_ELEM_MARKER: usize = 10; // Assumed marker size is 10, modify according to actual situation
#[allow(dead_code)]
#[derive(Debug, Default)]
pub struct LastMinuteHistogram {
histogram: Vec<LastMinuteLatency>,
size: u32,
}
impl LastMinuteHistogram {
pub fn merge(&mut self, other: &LastMinuteHistogram) {
for i in 0..self.histogram.len() {
self.histogram[i].merge(&other.histogram[i]);
}
}
pub fn add(&mut self, size: i64, t: Duration) {
let index = size_to_tag(size);
self.histogram[index].add(&t);
}
pub fn get_avg_data(&mut self) -> [AccElem; SIZE_LAST_ELEM_MARKER] {
let mut res = [AccElem::default(); SIZE_LAST_ELEM_MARKER];
for (i, elem) in self.histogram.iter_mut().enumerate() {
res[i] = elem.get_total();
}
res
}
}
fn size_to_tag(size: i64) -> usize {
match size {
_ if size < 1024 => 0, // sizeLessThan1KiB
_ if size < 1024 * 1024 => 1, // sizeLessThan1MiB
_ if size < 10 * 1024 * 1024 => 2, // sizeLessThan10MiB
_ if size < 100 * 1024 * 1024 => 3, // sizeLessThan100MiB
_ if size < 1024 * 1024 * 1024 => 4, // sizeLessThan1GiB
_ => 5, // sizeGreaterThan1GiB
}
}
-1
View File
@@ -12,7 +12,6 @@
// See the License for the specific language governing permissions and
// limitations under the License.
pub mod bucket_stats;
// pub mod error;
pub mod globals;
pub mod heal_channel;
+5
View File
@@ -353,6 +353,11 @@ pub const DEFAULT_OBS_TRACES_EXPORT_ENABLED: bool = true;
/// Environment variable: RUSTFS_OBS_METRICS_EXPORT_ENABLED
pub const DEFAULT_OBS_METRICS_EXPORT_ENABLED: bool = true;
/// Default detailed PUT stage metrics enabled
/// Default value: false
/// Environment variable: RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED
pub const DEFAULT_OBS_PUT_STAGE_METRICS_ENABLED: bool = false;
/// Default logs export enabled
/// It is used to enable or disable exporting logs
/// Default value: true
+49
View File
@@ -137,6 +137,37 @@ pub const DEFAULT_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED: bool = false;
const _: () = assert!(!DEFAULT_TIER_REMOTE_VERSION_STATE_WRITE);
const _: () = assert!(!DEFAULT_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED);
/// Request the object-transaction fencing contract used by storage-owned
/// cleanup receipts and lock-window optimizations.
///
/// This is fail-closed: enabling the writer without a live fleet proof rejects
/// the commit rather than silently using a legacy-safe path.
pub const ENV_OBJECT_TRANSACTION_FENCING_WRITE: &str = "RUSTFS_OBJECT_TRANSACTION_FENCING_WRITE";
pub const DEFAULT_OBJECT_TRANSACTION_FENCING_WRITE: bool = false;
/// Operator-attested confirmation that every serving node understands the
/// object transaction fencing contract.
pub const ENV_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED: &str = "RUSTFS_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED";
pub const DEFAULT_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED: bool = false;
const _: () = assert!(!DEFAULT_OBJECT_TRANSACTION_FENCING_WRITE);
const _: () = assert!(!DEFAULT_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED);
/// Request preserving legacy per-part checksum metadata during data movement.
///
/// This remains ineffective until
/// [`ENV_DATA_MOVEMENT_PART_CHECKSUMS_FLEET_CONFIRMED`] is also enabled.
pub const ENV_DATA_MOVEMENT_PART_CHECKSUMS_WRITE: &str = "RUSTFS_DATA_MOVEMENT_PART_CHECKSUMS_WRITE";
pub const DEFAULT_DATA_MOVEMENT_PART_CHECKSUMS_WRITE: bool = false;
/// Operator-attested confirmation that every serving node understands the
/// data-movement per-part checksum sidecar.
pub const ENV_DATA_MOVEMENT_PART_CHECKSUMS_FLEET_CONFIRMED: &str = "RUSTFS_DATA_MOVEMENT_PART_CHECKSUMS_FLEET_CONFIRMED";
pub const DEFAULT_DATA_MOVEMENT_PART_CHECKSUMS_FLEET_CONFIRMED: bool = false;
const _: () = assert!(!DEFAULT_DATA_MOVEMENT_PART_CHECKSUMS_WRITE);
const _: () = assert!(!DEFAULT_DATA_MOVEMENT_PART_CHECKSUMS_FLEET_CONFIRMED);
// =============================================================================
// Concurrent Request Fix - Timeout and Backpressure Configuration
// =============================================================================
@@ -649,4 +680,22 @@ mod remote_version_state_tests {
"RUSTFS_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED"
);
}
#[test]
fn data_movement_part_checksum_gate_uses_stable_environment_names() {
assert_eq!(super::ENV_DATA_MOVEMENT_PART_CHECKSUMS_WRITE, "RUSTFS_DATA_MOVEMENT_PART_CHECKSUMS_WRITE");
assert_eq!(
super::ENV_DATA_MOVEMENT_PART_CHECKSUMS_FLEET_CONFIRMED,
"RUSTFS_DATA_MOVEMENT_PART_CHECKSUMS_FLEET_CONFIRMED"
);
}
#[test]
fn object_transaction_fencing_gate_uses_stable_environment_names() {
assert_eq!(super::ENV_OBJECT_TRANSACTION_FENCING_WRITE, "RUSTFS_OBJECT_TRANSACTION_FENCING_WRITE");
assert_eq!(
super::ENV_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED,
"RUSTFS_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED"
);
}
}
+3
View File
@@ -81,6 +81,9 @@ pub const ENV_TEST_IAM_FAIL_INIT_ATTEMPTS: &str = "RUSTFS_TEST_IAM_FAIL_INIT_ATT
pub const ENV_TEST_IAM_RETRY_INTERVAL_MS: &str = "RUSTFS_TEST_IAM_RETRY_INTERVAL_MS";
/// Runtime env var controlling the transition worker count.
pub const ENV_TRANSITION_WORKERS: &str = "RUSTFS_MAX_TRANSITION_WORKERS";
/// Runtime env var controlling the ILM expiry worker count. A set, parsable,
/// non-zero value wins; anything else falls back to `min(cpus, 16)`.
pub const ENV_MAX_EXPIRY_WORKERS: &str = "RUSTFS_MAX_EXPIRY_WORKERS";
/// Runtime env var controlling the absolute maximum transition workers.
pub const ENV_TRANSITION_WORKERS_ABSOLUTE_MAX: &str = "RUSTFS_ABSOLUTE_MAX_WORKERS";
/// Runtime env var controlling the transition queue capacity.
+5
View File
@@ -44,6 +44,10 @@ pub const ENV_OBS_METRICS_EXPORT_ENABLED: &str = "RUSTFS_OBS_METRICS_EXPORT_ENAB
pub const ENV_OBS_LOGS_EXPORT_ENABLED: &str = "RUSTFS_OBS_LOGS_EXPORT_ENABLED";
pub const ENV_OBS_PROFILING_EXPORT_ENABLED: &str = "RUSTFS_OBS_PROFILING_EXPORT_ENABLED";
/// Enables detailed per-stage PUT metrics. Disabled by default because each
/// PUT records multiple timers and histograms when attribution is active.
pub const ENV_OBS_PUT_STAGE_METRICS_ENABLED: &str = "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED";
pub const ENV_OBS_LOGGER_LEVEL: &str = "RUSTFS_OBS_LOGGER_LEVEL";
pub const ENV_OBS_LOG_STDOUT_ENABLED: &str = "RUSTFS_OBS_LOG_STDOUT_ENABLED";
pub const ENV_OBS_LOG_DIRECTORY: &str = "RUSTFS_OBS_LOG_DIRECTORY";
@@ -141,6 +145,7 @@ mod tests {
assert_eq!(ENV_OBS_METRICS_EXPORT_ENABLED, "RUSTFS_OBS_METRICS_EXPORT_ENABLED");
assert_eq!(ENV_OBS_LOGS_EXPORT_ENABLED, "RUSTFS_OBS_LOGS_EXPORT_ENABLED");
assert_eq!(ENV_OBS_PROFILING_EXPORT_ENABLED, "RUSTFS_OBS_PROFILING_EXPORT_ENABLED");
assert_eq!(ENV_OBS_PUT_STAGE_METRICS_ENABLED, "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED");
// Test log cleanup related env keys
assert_eq!(ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES");
assert_eq!(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES");
@@ -0,0 +1,612 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! ILM on SSE-KMS buckets while per-key SSE authorization is enforced (backlog#1582).
//!
//! Per-key KMS authorization (`RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY=true`) scopes the
//! SSE-KMS data path to the requesting principal's `kms:GenerateDataKey` /
//! `kms:Decrypt` grants. Internal callers — the lifecycle scanner's expiry deletes
//! and the tier transition worker's reads — carry no request principal, and
//! `authorize_sse_kms_key` (rustfs/src/storage/sse.rs) exempts a `None` principal
//! so background maintenance keeps working on encrypted buckets.
//!
//! These tests pin that exemption end to end. If enforcement ever starts applying
//! to the scanner's internal operations, expiry stops happening on SSE-KMS buckets
//! and [`ilm_expiration_on_sse_kms_bucket_under_enforcement`] times out; if it
//! starts applying to the transition worker or the read-through path,
//! [`ilm_transition_on_sse_kms_bucket_under_enforcement_reads_back`] fails at the
//! transition wait or the plaintext round-trip.
//!
//! The replication half of the same acceptance item lives in
//! `crates/e2e_test/src/replication_extension_test.rs`
//! (`test_bucket_replication_sse_kms_failure_contract`); ILM had no coverage
//! before this file.
//!
//! Deployment constraint pinned by the transition test's setup: the RustFS warm
//! backend forwards the object's stored `x-amz-server-side-encryption*` metadata
//! as raw headers on the tier data PUT (`build_transition_put_options` +
//! `api_put_object.rs` header mapping), so a RustFS tier target must itself have
//! KMS enabled and hold the named key or it rejects every transition upload with
//! 400 InvalidRequest. That rejection is independent of the enforcement switch;
//! the cold server here therefore runs its own Local KMS with the same key id.
use super::common::{LocalKMSTestEnvironment, create_key_with_specific_id};
use crate::common::{RustFSTestEnvironment, admin_request, init_logging};
use aws_sdk_s3::Client;
use aws_sdk_s3::primitives::ByteStream;
use aws_sdk_s3::types::{
BucketLifecycleConfiguration, ExpirationStatus, LifecycleExpiration, LifecycleRule, LifecycleRuleFilter, RestoreRequest,
ServerSideEncryption, ServerSideEncryptionByDefault, ServerSideEncryptionConfiguration, ServerSideEncryptionRule, Transition,
TransitionStorageClass,
};
use serde::Deserialize;
use serial_test::serial;
use std::time::{Duration as StdDuration, Instant};
use tracing::info;
type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
const SSE_KEY: &str = "kms-ilm-sse-key";
const PAYLOAD: &[u8] = b"kms ilm sse payload: survives enforcement, expires and transitions on schedule";
const EXPIRY_BUCKET: &str = "kms-ilm-expiry";
const EXPIRE_KEY: &str = "expire/object.bin";
const SURVIVOR_KEY: &str = "keep/object.bin";
const TIER_NAME: &str = "KMSCOLD";
const TIER_BUCKET: &str = "kms-ilm-cold-tier";
const TIER_PREFIX: &str = "tiered";
const TRANSITION_BUCKET: &str = "kms-ilm-transition";
const TRANSITION_KEY: &str = "tier/object.bin";
/// Generous CI safety net; with a 1s scanner cycle and 2s lifecycle days the
/// terminal state normally lands within a few seconds.
const ILM_DEADLINE: StdDuration = StdDuration::from_secs(90);
/// Start a Local-KMS server with per-key SSE authorization enforced and the
/// lifecycle clock accelerated.
///
/// KMS wiring matches `kms_authorization_negative_matrix_test.rs` (local backend,
/// `--kms-default-key-id`, insecure dev defaults). The lifecycle env matches
/// `reliant/lifecycle.rs::fast_lifecycle_env` plus `RUSTFS_ILM_DEBUG_DAY_SECS=2`,
/// so a `Days=1` rule is due about two seconds after the write.
async fn start_enforcing_ilm_server(env: &mut LocalKMSTestEnvironment) -> TestResult {
create_key_with_specific_id(&env.kms_keys_dir, SSE_KEY).await?;
let key_dir = env.kms_keys_dir.clone();
let args = vec![
"--kms-enable",
"--kms-backend",
"local",
"--kms-key-dir",
key_dir.as_str(),
"--kms-default-key-id",
SSE_KEY,
];
let envs = [
("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true"),
("RUSTFS_KMS_ENFORCE_SSE_KEY_POLICY", "false"),
("RUSTFS_SCANNER_CYCLE", "1"),
("RUSTFS_ILM_PROCESS_TIME", "1"),
("RUSTFS_ILM_DEBUG_DAY_SECS", "2"),
];
env.base_env.start_rustfs_server_with_env(args, &envs).await?;
Ok(())
}
/// Set the bucket's default encryption to SSE-KMS under [`SSE_KEY`], so plain
/// PUTs (and internal rewrites) are encrypted without per-request SSE headers.
async fn set_bucket_default_sse_kms(client: &Client, bucket: &str) -> TestResult {
let encryption_config = ServerSideEncryptionConfiguration::builder()
.rules(
ServerSideEncryptionRule::builder()
.apply_server_side_encryption_by_default(
ServerSideEncryptionByDefault::builder()
.sse_algorithm(ServerSideEncryption::AwsKms)
.kms_master_key_id(SSE_KEY)
.build()?,
)
.build(),
)
.build()?;
client
.put_bucket_encryption()
.bucket(bucket)
.server_side_encryption_configuration(encryption_config)
.send()
.await?;
Ok(())
}
/// Assert via `HeadObject` that the stored object is SSE-KMS encrypted under
/// [`SSE_KEY`]. Without this, a bucket-default misconfiguration would let the
/// tests pass on an unencrypted object and prove nothing about KMS.
async fn assert_head_sse_kms(client: &Client, bucket: &str, key: &str) -> TestResult {
let head = client.head_object().bucket(bucket).key(key).send().await?;
assert_eq!(
head.server_side_encryption(),
Some(&ServerSideEncryption::AwsKms),
"{bucket}/{key} must be SSE-KMS encrypted via the bucket default"
);
assert_eq!(
head.ssekms_key_id(),
Some(SSE_KEY),
"{bucket}/{key} must be wrapped under the configured KMS key"
);
Ok(())
}
/// Returns `true` once `GET bucket/key` fails with `NoSuchKey`, `false` while it
/// still succeeds. Any other error is surfaced. (Copied from
/// `reliant/lifecycle.rs`; that helper is private to the reliant module.)
async fn object_is_gone(client: &Client, bucket: &str, key: &str) -> Result<bool, Box<dyn std::error::Error + Send + Sync>> {
match client.get_object().bucket(bucket).key(key).send().await {
Ok(output) => {
output.body.collect().await?;
Ok(false)
}
Err(e) => {
if let Some(service_error) = e.as_service_error() {
if service_error.is_no_such_key() {
return Ok(true);
}
return Err(format!("expected NoSuchKey, got: {e:?}").into());
}
Err(format!("expected a service error, got: {e:?}").into())
}
}
}
/// Poll until `GET bucket/key` returns `NoSuchKey`, or fail after `deadline`.
async fn wait_for_object_expired(client: &Client, bucket: &str, key: &str, deadline: StdDuration) -> TestResult {
let start = Instant::now();
loop {
if object_is_gone(client, bucket, key).await? {
return Ok(());
}
if start.elapsed() >= deadline {
return Err(format!(
"object {bucket}/{key} was not expired by the lifecycle scanner within {}s; \
SSE key-policy enforcement may have started blocking the scanner's internal deletes",
deadline.as_secs()
)
.into());
}
tokio::time::sleep(StdDuration::from_millis(500)).await;
}
}
/// Install a prefix-scoped `Days`-based expiration rule.
async fn put_expiration_rule(client: &Client, bucket: &str, id: &str, prefix: &str, days: i32) -> TestResult {
let rule = LifecycleRule::builder()
.id(id)
.filter(LifecycleRuleFilter::builder().prefix(prefix).build())
.expiration(LifecycleExpiration::builder().days(days).build())
.status(ExpirationStatus::Enabled)
.build()?;
let lifecycle = BucketLifecycleConfiguration::builder().rules(rule).build()?;
client
.put_bucket_lifecycle_configuration()
.bucket(bucket)
.lifecycle_configuration(lifecycle)
.send()
.await?;
Ok(())
}
/// Install a prefix-scoped `Days`-based transition rule targeting [`TIER_NAME`].
async fn put_transition_rule(client: &Client, bucket: &str, id: &str, prefix: &str, days: i32) -> TestResult {
let rule = LifecycleRule::builder()
.id(id)
.filter(LifecycleRuleFilter::builder().prefix(prefix).build())
.transitions(
Transition::builder()
.days(days)
.storage_class(TransitionStorageClass::from(TIER_NAME))
.build(),
)
.status(ExpirationStatus::Enabled)
.build()?;
let lifecycle = BucketLifecycleConfiguration::builder().rules(rule).build()?;
client
.put_bucket_lifecycle_configuration()
.bucket(bucket)
.lifecycle_configuration(lifecycle)
.send()
.await?;
Ok(())
}
/// Start a plain Local-KMS server (no enforcement, no lifecycle acceleration)
/// holding [`SSE_KEY`], to serve as the cold tier target.
///
/// The RustFS warm backend forwards the stored SSE-KMS headers on the tier data
/// PUT, so the target re-applies managed SSE-KMS under the named key and must
/// be able to resolve it; without KMS it answers 400 InvalidRequest and the
/// transition can never complete. Enforcement stays off here: the tier writes
/// arrive under `cold`'s root credentials, and one enforcing side is enough to
/// pin the exemption.
async fn start_cold_tier_kms_server(env: &mut LocalKMSTestEnvironment) -> TestResult {
create_key_with_specific_id(&env.kms_keys_dir, SSE_KEY).await?;
let key_dir = env.kms_keys_dir.clone();
let args = vec![
"--kms-enable",
"--kms-backend",
"local",
"--kms-key-dir",
key_dir.as_str(),
"--kms-default-key-id",
SSE_KEY,
];
env.base_env
.start_rustfs_server_with_env(args, &[("RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS", "true")])
.await?;
Ok(())
}
/// The subset of the manual transition run report these tests assert on.
///
/// Unknown fields are ignored, so this stays compatible with report growth; the
/// full shape is pinned by `reliant/tiering.rs`.
#[derive(Debug, Deserialize)]
struct ManualTransitionRunReport {
#[serde(default)]
scanned: u64,
#[serde(default)]
enqueued: u64,
#[serde(default)]
skipped_already_in_flight: u64,
#[serde(default)]
skipped_tier: u64,
}
#[derive(Debug, Deserialize)]
struct ManualTransitionRunResponse {
state: String,
report: ManualTransitionRunReport,
}
/// One synchronous (enqueue-only) manual transition run over `bucket/prefix`,
/// via the same admin endpoint `reliant/tiering.rs` drives.
async fn manual_transition_run(
hot: &RustFSTestEnvironment,
bucket: &str,
prefix: &str,
) -> Result<ManualTransitionRunResponse, Box<dyn std::error::Error + Send + Sync>> {
let bucket = urlencoding::encode(bucket);
let prefix = urlencoding::encode(prefix);
let tier = urlencoding::encode(TIER_NAME);
let path =
format!("/rustfs/admin/v3/ilm/transition/run?bucket={bucket}&prefix={prefix}&tier={tier}&dryRun=false&maxObjects=10");
let (status, body) = admin_request(&hot.url, http::Method::POST, &path, None, &hot.access_key, &hot.secret_key).await?;
if !status.is_success() {
return Err(format!("manual transition run failed: status={status}, body={body}").into());
}
Ok(serde_json::from_str(&body)?)
}
/// Drive manual transition runs until one reports the object as processed.
///
/// The `Days=1` rule becomes due about two seconds after the write
/// (`RUSTFS_ILM_DEBUG_DAY_SECS=2`), so early runs may legitimately report the
/// object as not yet eligible; the loop keeps running the endpoint until it
/// either enqueues the transition, sees it already in flight (the 1s scanner
/// backstop got there first), or finds it already on the tier.
async fn run_manual_transition_until_processed(
hot: &RustFSTestEnvironment,
bucket: &str,
prefix: &str,
deadline: StdDuration,
) -> TestResult {
let start = Instant::now();
loop {
let run = manual_transition_run(hot, bucket, prefix).await?;
assert_eq!(run.report.scanned, 1, "manual transition run must scan the object: {run:#?}");
if run.report.enqueued + run.report.skipped_already_in_flight + run.report.skipped_tier >= 1 {
info!(state = %run.state, report = ?run.report, "manual transition run processed the SSE-KMS object");
return Ok(());
}
if start.elapsed() >= deadline {
return Err(format!(
"manual transition runs never processed {bucket}/{prefix} within {}s; last report: {run:#?}",
deadline.as_secs()
)
.into());
}
tokio::time::sleep(StdDuration::from_millis(500)).await;
}
}
/// Wire `hot` -> `cold` as a `TierType::RustFS` remote tier via `AddTier`.
///
/// No `force`, so the server runs the real connectivity probe against `cold`
/// (the tier bucket must already exist there). Mirrors
/// `reliant/tiering.rs::add_rustfs_tier`, which is private to that module.
async fn add_rustfs_tier(hot: &RustFSTestEnvironment, cold: &RustFSTestEnvironment) -> TestResult {
let body = serde_json::json!({
"type": "rustfs",
"rustfs": {
"name": TIER_NAME,
"endpoint": cold.url.as_str(),
"accessKey": cold.access_key.as_str(),
"secretKey": cold.secret_key.as_str(),
"bucket": TIER_BUCKET,
"prefix": TIER_PREFIX,
"region": "us-east-1",
"storageClass": ""
}
})
.to_string();
let (status, resp) = admin_request(
&hot.url,
http::Method::PUT,
"/rustfs/admin/v3/tier",
Some(body),
&hot.access_key,
&hot.secret_key,
)
.await?;
if !status.is_success() {
return Err(format!("AddTier(RustFS) failed: status={status}, body={resp}").into());
}
Ok(())
}
/// Poll `HEAD` until the object's storage class is the tier name (transition
/// complete), or fail after `deadline`. (From `reliant/tiering.rs`.)
async fn wait_for_transition(client: &Client, bucket: &str, key: &str, deadline: StdDuration) -> TestResult {
let start = Instant::now();
loop {
let head = client.head_object().bucket(bucket).key(key).send().await?;
if head.storage_class().map(|sc| sc.as_str()) == Some(TIER_NAME) {
return Ok(());
}
if start.elapsed() >= deadline {
return Err(format!(
"object {bucket}/{key} was not transitioned to {TIER_NAME} within {}s (storage_class={:?}); \
SSE key-policy enforcement may have started blocking the transition worker's internal reads",
deadline.as_secs(),
head.storage_class()
)
.into());
}
tokio::time::sleep(StdDuration::from_millis(500)).await;
}
}
/// Poll `HEAD` until `x-amz-restore` reports a finished restore
/// (`ongoing-request="false"`), or fail after `deadline`.
async fn wait_for_restore_complete(client: &Client, bucket: &str, key: &str, deadline: StdDuration) -> TestResult {
let start = Instant::now();
loop {
let head = client.head_object().bucket(bucket).key(key).send().await?;
if head.restore().is_some_and(|r| r.contains("ongoing-request=\"false\"")) {
return Ok(());
}
if start.elapsed() >= deadline {
return Err(format!(
"object {bucket}/{key} restore did not complete within {}s (restore={:?}); \
SSE key-policy enforcement may have started blocking the restore copy-back's internal reads",
deadline.as_secs(),
head.restore()
)
.into());
}
tokio::time::sleep(StdDuration::from_millis(500)).await;
}
}
/// ILM expiration keeps working on an SSE-KMS bucket while per-key SSE
/// authorization is enforced.
///
/// The lifecycle scanner deletes expired objects with an internal (no-principal)
/// identity that holds no `kms` grant. If enforcement ever starts applying to
/// those internal deletes (or to the scanner's metadata reads) on encrypted
/// buckets, expiry stops happening and this test times out.
///
/// A survivor object under a non-matching prefix isolates the rule's prefix
/// filter as the cause of the deletion and proves the encrypted bucket stays
/// readable end to end after the scanner has run.
#[tokio::test]
#[serial]
async fn ilm_expiration_on_sse_kms_bucket_under_enforcement() -> TestResult {
init_logging();
let mut env = LocalKMSTestEnvironment::new().await?;
start_enforcing_ilm_server(&mut env).await?;
env.base_env.create_test_bucket(EXPIRY_BUCKET).await?;
let client = env.base_env.create_s3_client();
set_bucket_default_sse_kms(&client, EXPIRY_BUCKET).await?;
for key in [EXPIRE_KEY, SURVIVOR_KEY] {
client
.put_object()
.bucket(EXPIRY_BUCKET)
.key(key)
.body(ByteStream::from_static(PAYLOAD))
.send()
.await?;
assert_head_sse_kms(&client, EXPIRY_BUCKET, key).await?;
}
info!("both objects stored SSE-KMS encrypted under enforcement");
put_expiration_rule(&client, EXPIRY_BUCKET, "kms-ilm-expire", "expire/", 1).await?;
// The regression this pins: the scanner's internal delete must stay exempt
// from per-key SSE authorization, so the encrypted object actually expires.
wait_for_object_expired(&client, EXPIRY_BUCKET, EXPIRE_KEY, ILM_DEADLINE).await?;
info!("SSE-KMS object expired by the lifecycle scanner under enforcement");
// Negative control: same bucket, same encryption, non-matching prefix. It
// must survive the scanner and still decrypt for the requesting principal.
assert!(
!object_is_gone(&client, EXPIRY_BUCKET, SURVIVOR_KEY).await?,
"non-matching-prefix object must not be expired by a prefix-scoped rule"
);
let survivor = client.get_object().bucket(EXPIRY_BUCKET).key(SURVIVOR_KEY).send().await?;
assert_eq!(
survivor.body.collect().await?.into_bytes().as_ref(),
PAYLOAD,
"surviving SSE-KMS object must still decrypt after the scanner has run"
);
Ok(())
}
/// ILM transition to a remote tier keeps working on an SSE-KMS bucket while
/// per-key SSE authorization is enforced, and the transitioned object reads
/// back as plaintext.
///
/// The transition worker moves the stored (encrypted) bytes to the cold tier
/// with an internal (no-principal) identity; the read-through `GET` then
/// decrypts the envelope for the requesting principal. If enforcement ever
/// starts applying to the worker's internal reads, the transition wait times
/// out; if the stored envelope is mishandled across the tier round trip, the
/// plaintext comparison fails.
///
/// The transition is driven through the manual transition-run admin endpoint
/// (the mechanism `reliant/tiering.rs` established), so the test does not
/// depend on scanner scheduling; the 1s scanner cycle stays on as a backstop.
#[tokio::test]
#[serial]
#[ignore = "pins rustfs/rustfs#6025: GET on a transitioned managed-SSE object silently returns corrupt bytes (fails with enforcement on AND off, so it is not an authorization regression); un-ignore with the fix"]
async fn ilm_transition_on_sse_kms_bucket_under_enforcement_reads_back() -> TestResult {
init_logging();
// Cold-tier server: independent credentials, its own Local KMS holding the
// same key id (see the module docs for why the tier target needs KMS).
// Started first; each server's startup cleanup only matches its own unique
// address and temp dir, so the two instances coexist.
let mut cold = LocalKMSTestEnvironment::new().await?;
cold.base_env.access_key = "kmscoldtieradmin".to_string();
cold.base_env.secret_key = "kmscoldtiersecret".to_string();
start_cold_tier_kms_server(&mut cold).await?;
let cold_client = cold.base_env.create_s3_client();
cold_client.create_bucket().bucket(TIER_BUCKET).send().await?;
// Hot server: Local KMS + enforcement + accelerated lifecycle clock.
let mut env = LocalKMSTestEnvironment::new().await?;
start_enforcing_ilm_server(&mut env).await?;
let hot_client = env.base_env.create_s3_client();
add_rustfs_tier(&env.base_env, &cold.base_env).await?;
env.base_env.create_test_bucket(TRANSITION_BUCKET).await?;
set_bucket_default_sse_kms(&hot_client, TRANSITION_BUCKET).await?;
hot_client
.put_object()
.bucket(TRANSITION_BUCKET)
.key(TRANSITION_KEY)
.body(ByteStream::from_static(PAYLOAD))
.send()
.await?;
assert_head_sse_kms(&hot_client, TRANSITION_BUCKET, TRANSITION_KEY).await?;
info!("object stored SSE-KMS encrypted under enforcement");
// Days=1 is due ~2s after the write with RUSTFS_ILM_DEBUG_DAY_SECS=2.
put_transition_rule(&hot_client, TRANSITION_BUCKET, "kms-ilm-transition", "tier/", 1).await?;
// Drive the transition deterministically via the manual run endpoint, then
// wait for HEAD to report the tier as the object's storage class.
run_manual_transition_until_processed(&env.base_env, TRANSITION_BUCKET, "tier/", ILM_DEADLINE).await?;
wait_for_transition(&hot_client, TRANSITION_BUCKET, TRANSITION_KEY, ILM_DEADLINE).await?;
info!("SSE-KMS object transitioned to the remote tier under enforcement");
let head = hot_client
.head_object()
.bucket(TRANSITION_BUCKET)
.key(TRANSITION_KEY)
.send()
.await?;
assert!(
head.restore().is_none(),
"a freshly transitioned object must not advertise x-amz-restore, got {:?}",
head.restore()
);
// The remote copy exists on the cold tier. The payload the tier holds is the
// hot server's stored ciphertext, wrapped once more under the cold server's
// own managed SSE-KMS layer (the forwarded headers re-request encryption).
let remote = cold_client.list_objects_v2().bucket(TIER_BUCKET).send().await?;
assert!(!remote.contents().is_empty(), "cold-tier bucket must hold the transitioned object's data");
// Read-through GET under enforcement must succeed (not AccessDenied) and
// keep advertising SSE-KMS. Its BODY is deliberately not compared here:
// the transitioned read path skips managed-SSE decryption — a product gap
// unrelated to enforcement — so a direct GET streams the stored ciphertext
// (`new_getobjectreader` in crates/ecstore/src/client/object_api_utils.rs
// hardcodes `is_encrypted = false` and never applies the
// `ReadTransform::Encrypted` wrapping the hot-read path builds in
// crates/ecstore/src/object_api/readers.rs). Plaintext recovery is pinned
// through restore semantics below; when the read-through gap is fixed, a
// byte assertion can be added here too.
let read_through = hot_client
.get_object()
.bucket(TRANSITION_BUCKET)
.key(TRANSITION_KEY)
.send()
.await?;
assert_eq!(
read_through.server_side_encryption(),
Some(&ServerSideEncryption::AwsKms),
"transitioned object must still report SSE-KMS on read-through"
);
let read_through_body = read_through.body.collect().await?.into_bytes();
assert_eq!(
read_through_body.len(),
PAYLOAD.len(),
"read-through GET must stream the object's full logical size under enforcement"
);
// RestoreObject copies the ciphertext back from the tier under the original
// envelope metadata; the restored copy is then served by the normal
// decrypting read path. The copy-back runs with an internal (no-principal)
// identity, so this also pins the exemption on the restore path. Days=300
// because RUSTFS_ILM_DEBUG_DAY_SECS=2 accelerates the restored copy's
// expiry as well (300 accelerated days == 600s of validity).
hot_client
.restore_object()
.bucket(TRANSITION_BUCKET)
.key(TRANSITION_KEY)
.restore_request(RestoreRequest::builder().days(300).build())
.send()
.await?;
wait_for_restore_complete(&hot_client, TRANSITION_BUCKET, TRANSITION_KEY, ILM_DEADLINE).await?;
info!("SSE-KMS object restored from the remote tier under enforcement");
// The KMS-relevant half: the restored envelope decrypts back to the exact
// plaintext for the requesting principal.
let restored = hot_client
.get_object()
.bucket(TRANSITION_BUCKET)
.key(TRANSITION_KEY)
.send()
.await?;
assert_eq!(
restored.server_side_encryption(),
Some(&ServerSideEncryption::AwsKms),
"restored object must still report SSE-KMS"
);
let body = restored.body.collect().await?.into_bytes();
assert_eq!(body.as_ref(), PAYLOAD, "restored SSE-KMS object must round-trip byte-identical plaintext");
Ok(())
}
+3
View File
@@ -59,3 +59,6 @@ mod configured_roundtrip_test;
#[cfg(test)]
mod kms_authorization_negative_matrix_test;
#[cfg(test)]
mod kms_ilm_sse_kms_test;
File diff suppressed because it is too large Load Diff
@@ -2854,7 +2854,7 @@ pub(crate) mod cmptst_30 {
result
}
#[ignore]
#[ignore = "timing-sensitive backend-pressure latency probe; run explicitly with --ignored"]
#[tokio::test]
async fn regression() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
crate::common::init_logging();
@@ -2401,15 +2401,20 @@ async fn wait_for_site_replication_info<F>(
where
F: Fn(&SiteReplicationInfo) -> bool,
{
for _ in 0..40 {
// 30s to match wait_for_replication_state: the three-node site tests run
// several full rustfs processes on one runner, so peer-state propagation
// can take well over 10s under CI load.
let deadline = tokio::time::Instant::now() + Duration::from_secs(30);
loop {
let info = site_replication_info(env).await?;
if predicate(&info) {
return Ok(info);
}
if tokio::time::Instant::now() >= deadline {
return Err(format!("site replication info did not reach expected state on {}", env.address).into());
}
sleep(Duration::from_millis(250)).await;
}
Err(format!("site replication info did not reach expected state on {}", env.address).into())
}
async fn wait_for_site_replication_status<F>(
@@ -2420,15 +2425,19 @@ async fn wait_for_site_replication_status<F>(
where
F: Fn(&SRStatusInfo) -> bool,
{
for _ in 0..40 {
// Same 30s ceiling as wait_for_site_replication_info: the status probes
// fan out to every peer, so they see the same multi-process CI load.
let deadline = tokio::time::Instant::now() + Duration::from_secs(30);
loop {
let status = site_replication_status(env, query).await?;
if predicate(&status) {
return Ok(status);
}
if tokio::time::Instant::now() >= deadline {
return Err(format!("site replication status did not reach expected state on {}", env.address).into());
}
sleep(Duration::from_millis(250)).await;
}
Err(format!("site replication status did not reach expected state on {}", env.address).into())
}
async fn wait_for_replication_reset_target<F>(
@@ -4235,37 +4244,49 @@ async fn test_bucket_replication_acceptance_matrix_local_dual_targets() -> TestR
"tag rule with disabled delete-marker replication created a marker: {tagged_state:?}"
);
set_bucket_versioning(&source_env, source_bucket, BucketVersioningStatus::Suspended).await?;
set_bucket_versioning(&target_env_a, target_bucket_a, BucketVersioningStatus::Suspended).await?;
let null_put = source_client
// AWS S3 and MinIO both reject suspending versioning on a bucket that
// carries a replication configuration (InvalidBucketState): suspension
// would mint null versions that versioned replication can never converge.
let suspend_err = source_client
.put_bucket_versioning()
.bucket(source_bucket)
.versioning_configuration(
VersioningConfiguration::builder()
.status(BucketVersioningStatus::Suspended)
.build(),
)
.send()
.await
.expect_err("suspending versioning on a replication source must be rejected");
assert_eq!(
suspend_err.as_service_error().and_then(|error| error.code()),
Some("InvalidBucketState"),
"suspension on a replication source must fail with InvalidBucketState: {suspend_err:?}"
);
// The rejected suspension must leave the versioning + replication state
// fully intact: a fresh matched PUT still replicates with a real version.
let post_reject_put = source_client
.put_object()
.bucket(source_bucket)
.key("prefix/null.txt")
.body(ByteStream::from_static(b"null version"))
.key("prefix/after-rejected-suspend.txt")
.body(ByteStream::from_static(b"still replicating"))
.send()
.await?;
assert!(null_put.version_id().is_none(), "suspended source PUT must create a null version");
wait_for_replication_state(&target_client_a, target_bucket_a, "null version did not replicate", |state| {
state
.iter()
.any(|entry| entry.key == "prefix/null.txt" && entry.version_id == "null" && !entry.delete_marker)
})
.await?;
let null_delete = source_client
.delete_object()
.bucket(source_bucket)
.key("prefix/null.txt")
.send()
.await?;
assert!(
null_delete.version_id().is_none(),
"suspended source DELETE must create a null delete marker"
);
wait_for_replication_state(&target_client_a, target_bucket_a, "null delete marker did not replicate", |state| {
state
.iter()
.any(|entry| entry.key == "prefix/null.txt" && entry.version_id == "null" && entry.delete_marker)
})
let post_reject_version_id = post_reject_put
.version_id()
.ok_or("PUT after rejected suspension omitted version ID")?
.to_string();
wait_for_replication_state(
&target_client_a,
target_bucket_a,
"replication stopped after rejected versioning suspension",
|state| {
state
.iter()
.any(|entry| entry.key == "prefix/after-rejected-suspend.txt" && entry.version_id == post_reject_version_id)
},
)
.await?;
Ok(())
+5
View File
@@ -32,6 +32,11 @@ workspace = true
[features]
default = []
# Compiles the controlled list-objects namespace-journal chaos injector into a
# production binary (it is always available to tests). Off by default so the
# RUSTFS_LIST_OBJECTS_NAMESPACE_JOURNAL_CHAOS_* env vars cannot rewrite journal
# state in a stock build (backlog#1832).
list-chaos = []
rio-v2 = ["dep:rustfs-rio-v2"]
hotpath = [
"hotpath/hotpath",
+6 -4
View File
@@ -61,9 +61,11 @@ pub mod bucket {
delete_manual_transition_scope_admission_if_current, load_manual_transition_job_record,
load_manual_transition_job_record_with_etag, load_manual_transition_scope_admission,
manual_transition_job_lease_expired, manual_transition_scope_admission_lease_expired,
manual_transition_scope_key, persist_manual_transition_job_progress, renew_manual_transition_job_lease,
request_manual_transition_job_cancel, save_manual_transition_job_record,
save_manual_transition_job_record_if_current, save_manual_transition_scope_admission_if_absent,
manual_transition_scope_key, persist_manual_transition_job_progress,
persist_manual_transition_job_progress_if_owned, renew_manual_transition_job_lease,
renew_manual_transition_job_lease_if_owned, request_manual_transition_job_cancel,
save_manual_transition_job_record, save_manual_transition_job_record_if_current,
save_manual_transition_scope_admission_if_absent, update_manual_transition_job_record,
};
}
@@ -344,7 +346,7 @@ pub mod disk {
}
pub mod error {
pub use crate::disk::error::{BitrotErrorType, DiskError, Error, FileAccessDeniedWithContext, Result};
pub use crate::disk::error::{DiskError, Error, FileAccessDeniedWithContext, Result};
}
pub mod error_reduce {
File diff suppressed because it is too large Load Diff
@@ -86,6 +86,21 @@ where
com::save_config_with_opts(api, file, data, opts).await
}
pub(crate) async fn save_config_with_opts_quiet<S>(api: Arc<S>, file: &str, data: Vec<u8>, opts: &ObjectOptions) -> Result<()>
where
S: ObjectIO<
Error = Error,
RangeSpec = HTTPRangeSpec,
HeaderMap = HeaderMap,
ObjectOptions = ObjectOptions,
ObjectInfo = ObjectInfo,
GetObjectReader = GetObjectReader,
PutObjectReader = PutObjReader,
>,
{
com::save_config_with_opts_quiet(api, file, data, opts).await
}
pub(crate) async fn delete_config<S>(api: Arc<S>, file: &str) -> Result<()>
where
S: ObjectOperations<
@@ -45,6 +45,104 @@ const MANUAL_TRANSITION_JOB_LEASE_SECONDS: i128 = 60;
const MANUAL_TRANSITION_LEGACY_SCOPE_SCAN_LIMIT: i32 = 1000;
const MANUAL_TRANSITION_TASK_SCAN_LIMIT: i32 = 1000;
const MANUAL_TRANSITION_WORKER_RESULT_SCAN_LIMIT: i32 = 1000;
const MANUAL_TRANSITION_JOB_CAS_RETRIES: usize = 4;
#[cfg(test)]
struct ManualTransitionJobCasBarrierState {
job_id: Uuid,
paused: std::sync::atomic::AtomicBool,
arrived: tokio::sync::Notify,
release: tokio::sync::Semaphore,
}
#[cfg(test)]
pub(crate) struct ManualTransitionJobCasBarrier {
state: Arc<ManualTransitionJobCasBarrierState>,
}
#[cfg(test)]
static MANUAL_TRANSITION_JOB_CAS_BARRIER: std::sync::OnceLock<std::sync::Mutex<Option<Arc<ManualTransitionJobCasBarrierState>>>> =
std::sync::OnceLock::new();
#[cfg(test)]
impl ManualTransitionJobCasBarrier {
pub(crate) fn install(job_id: Uuid) -> Self {
let state = Arc::new(ManualTransitionJobCasBarrierState {
job_id,
paused: std::sync::atomic::AtomicBool::new(false),
arrived: tokio::sync::Notify::new(),
release: tokio::sync::Semaphore::new(0),
});
let mut slot = MANUAL_TRANSITION_JOB_CAS_BARRIER
.get_or_init(|| std::sync::Mutex::new(None))
.lock()
.expect("manual transition progress CAS barrier mutex should not poison");
assert!(
slot.is_none(),
"manual transition job CAS barrier must be installed by one test at a time"
);
*slot = Some(Arc::clone(&state));
drop(slot);
Self { state }
}
pub(crate) async fn wait_until_paused(&self) {
tokio::time::timeout(std::time::Duration::from_secs(30), async {
loop {
let arrived = self.state.arrived.notified();
if self.state.paused.load(std::sync::atomic::Ordering::Acquire) {
return;
}
arrived.await;
}
})
.await
.expect("manual transition job update should reach the deterministic CAS barrier");
}
pub(crate) fn release(&self) {
self.state.release.add_permits(1);
}
}
#[cfg(test)]
impl Drop for ManualTransitionJobCasBarrier {
fn drop(&mut self) {
self.release();
let mut slot = MANUAL_TRANSITION_JOB_CAS_BARRIER
.get_or_init(|| std::sync::Mutex::new(None))
.lock()
.expect("manual transition progress CAS barrier mutex should not poison");
if slot.as_ref().is_some_and(|state| Arc::ptr_eq(state, &self.state)) {
*slot = None;
}
}
}
#[cfg(test)]
async fn pause_manual_transition_job_before_first_cas(job_id: Uuid) {
let barrier = MANUAL_TRANSITION_JOB_CAS_BARRIER
.get_or_init(|| std::sync::Mutex::new(None))
.lock()
.expect("manual transition progress CAS barrier mutex should not poison")
.as_ref()
.filter(|barrier| barrier.job_id == job_id)
.cloned();
if let Some(barrier) = barrier
&& barrier
.paused
.compare_exchange(false, true, std::sync::atomic::Ordering::AcqRel, std::sync::atomic::Ordering::Acquire)
.is_ok()
{
barrier.arrived.notify_one();
barrier
.release
.acquire()
.await
.expect("manual transition job CAS barrier should remain open")
.forget();
}
}
fn is_false(value: &bool) -> bool {
!*value
@@ -148,7 +246,6 @@ impl ManualTransitionJobRecord {
pub fn fail(&mut self, error: impl Into<String>) {
self.state = ManualTransitionJobState::Failed;
self.report.tier_failure = self.report.tier_failure.saturating_add(1);
self.error = Some(error.into());
self.mark_updated_terminal();
}
@@ -1040,7 +1137,7 @@ pub async fn save_manual_transition_job_record_if_current(
}
let object = manual_transition_job_record_object_name(job.job_id).map_err(manual_transition_job_store_error)?;
let data = job.encode().map_err(manual_transition_job_store_error)?;
config_boundary::save_config_with_opts(
config_boundary::save_config_with_opts_quiet(
api,
&object,
data,
@@ -1056,6 +1153,54 @@ pub async fn save_manual_transition_job_record_if_current(
.await
}
/// Applies a job-record mutation with optimistic concurrency control.
///
/// The mutation returns whether the record needs to be persisted. When a lease
/// is supplied, ownership is checked again after every conflicting write.
pub async fn update_manual_transition_job_record<F>(
api: Arc<ECStore>,
job_id: Uuid,
expected_lease_id: Option<Uuid>,
update: F,
) -> EcstoreResult<ManualTransitionJobRecord>
where
F: FnMut(&mut ManualTransitionJobRecord) -> bool,
{
update_manual_transition_job_record_from(api, job_id, expected_lease_id, None, update).await
}
async fn update_manual_transition_job_record_from<F>(
api: Arc<ECStore>,
job_id: Uuid,
expected_lease_id: Option<Uuid>,
mut current: Option<(ManualTransitionJobRecord, String)>,
mut update: F,
) -> EcstoreResult<ManualTransitionJobRecord>
where
F: FnMut(&mut ManualTransitionJobRecord) -> bool,
{
for _ in 0..MANUAL_TRANSITION_JOB_CAS_RETRIES {
let (mut record, etag) = match current.take() {
Some(current) => current,
None => load_manual_transition_job_record_with_etag(api.clone(), job_id).await?,
};
if expected_lease_id.is_some_and(|lease_id| record.lease_id != lease_id) {
return Err(Error::PreconditionFailed);
}
if !update(&mut record) {
return Ok(record);
}
#[cfg(test)]
pause_manual_transition_job_before_first_cas(job_id).await;
match save_manual_transition_job_record_if_current(api.clone(), &record, &etag).await {
Ok(()) => return Ok(record),
Err(Error::PreconditionFailed) => continue,
Err(err) => return Err(err),
}
}
Err(Error::PreconditionFailed)
}
pub(crate) async fn save_manual_transition_worker_result_if_absent(
api: Arc<ECStore>,
record: &ManualTransitionWorkerResultRecord,
@@ -1314,99 +1459,113 @@ pub async fn reconcile_manual_transition_worker_results(
api: Arc<ECStore>,
job_id: Uuid,
queue_snapshot: ManualTransitionQueueSnapshot,
) -> EcstoreResult<ManualTransitionJobRecord> {
reconcile_manual_transition_worker_results_inner(api, job_id, None, queue_snapshot, false).await
}
pub(crate) async fn reconcile_manual_transition_worker_results_if_owned(
api: Arc<ECStore>,
job_id: Uuid,
expected_lease_id: Uuid,
queue_snapshot: ManualTransitionQueueSnapshot,
) -> EcstoreResult<ManualTransitionJobRecord> {
reconcile_manual_transition_worker_results_inner(api, job_id, Some(expected_lease_id), queue_snapshot, false).await
}
async fn reconcile_manual_transition_worker_results_inner(
api: Arc<ECStore>,
job_id: Uuid,
expected_lease_id: Option<Uuid>,
queue_snapshot: ManualTransitionQueueSnapshot,
mark_missing_results_unknown: bool,
) -> EcstoreResult<ManualTransitionJobRecord> {
let task_stats = match scan_manual_transition_task_journal(api.clone(), job_id).await? {
ManualTransitionTaskJournal::Stats(stats) => stats,
ManualTransitionTaskJournal::Corrupt(error) => {
return mark_manual_transition_job_unknown_for_task_journal_error(api, job_id, error, queue_snapshot).await;
return mark_manual_transition_job_unknown_for_task_journal_error(
api,
job_id,
expected_lease_id,
error,
queue_snapshot,
)
.await;
}
};
let stats = match scan_manual_transition_worker_result_journal(api.clone(), job_id).await? {
ManualTransitionWorkerResultJournal::Stats(stats) => stats,
ManualTransitionWorkerResultJournal::Corrupt(error) => {
return mark_manual_transition_job_unknown_for_worker_result_journal_error(api, job_id, error, queue_snapshot).await;
return mark_manual_transition_job_unknown_for_worker_result_journal_error(
api,
job_id,
expected_lease_id,
error,
queue_snapshot,
)
.await;
}
};
for _ in 0..4 {
let (mut record, etag) = load_manual_transition_job_record_with_etag(api.clone(), job_id).await?;
let changed = record.apply_worker_result_counts(
let mut changed = false;
let record = update_manual_transition_job_record(api.clone(), job_id, expected_lease_id, |record| {
let counts_changed = record.apply_worker_result_counts(
stats.stats.completed,
stats.stats.failed,
&stats.stats.tier_failure_by_reason,
task_stats.queued,
queue_snapshot,
);
if !changed {
return Ok(record);
}
match save_manual_transition_job_record_if_current(api.clone(), &record, &etag).await {
Ok(()) => {
if record.is_terminal() {
delete_manual_transition_scope_admission_if_current(
api.clone(),
&record.scope_key,
record.job_id,
record.lease_id,
)
.await?;
} else {
renew_manual_transition_scope_admission_from_job(api, &record).await?;
}
return Ok(record);
}
Err(Error::PreconditionFailed) => continue,
Err(err) => return Err(err),
}
let became_unknown = mark_missing_results_unknown && record.mark_unknown_if_worker_results_lost(queue_snapshot);
changed = counts_changed || became_unknown;
changed
})
.await?;
if !changed {
return Ok(record);
}
Err(Error::PreconditionFailed)
if record.is_terminal() {
delete_manual_transition_scope_admission_if_current(api, &record.scope_key, record.job_id, record.lease_id).await?;
} else {
renew_manual_transition_scope_admission_from_job(api, &record).await?;
}
Ok(record)
}
async fn mark_manual_transition_job_unknown_for_task_journal_error(
api: Arc<ECStore>,
job_id: Uuid,
expected_lease_id: Option<Uuid>,
error: String,
queue_snapshot: ManualTransitionQueueSnapshot,
) -> EcstoreResult<ManualTransitionJobRecord> {
for _ in 0..4 {
let (mut record, etag) = load_manual_transition_job_record_with_etag(api.clone(), job_id).await?;
if !record.mark_unknown_for_task_journal_error(error.clone(), queue_snapshot) {
return Ok(record);
}
match save_manual_transition_job_record_if_current(api.clone(), &record, &etag).await {
Ok(()) => {
delete_manual_transition_scope_admission_if_current(api, &record.scope_key, record.job_id, record.lease_id)
.await?;
return Ok(record);
}
Err(Error::PreconditionFailed) => continue,
Err(err) => return Err(err),
}
let mut changed = false;
let record = update_manual_transition_job_record(api.clone(), job_id, expected_lease_id, |record| {
changed = record.mark_unknown_for_task_journal_error(error.clone(), queue_snapshot);
changed
})
.await?;
if changed && record.is_terminal() {
delete_manual_transition_scope_admission_if_current(api, &record.scope_key, record.job_id, record.lease_id).await?;
}
Err(Error::PreconditionFailed)
Ok(record)
}
async fn mark_manual_transition_job_unknown_for_worker_result_journal_error(
api: Arc<ECStore>,
job_id: Uuid,
expected_lease_id: Option<Uuid>,
error: String,
queue_snapshot: ManualTransitionQueueSnapshot,
) -> EcstoreResult<ManualTransitionJobRecord> {
for _ in 0..4 {
let (mut record, etag) = load_manual_transition_job_record_with_etag(api.clone(), job_id).await?;
if !record.mark_unknown_for_worker_result_journal_error(error.clone(), queue_snapshot) {
return Ok(record);
}
match save_manual_transition_job_record_if_current(api.clone(), &record, &etag).await {
Ok(()) => {
delete_manual_transition_scope_admission_if_current(api, &record.scope_key, record.job_id, record.lease_id)
.await?;
return Ok(record);
}
Err(Error::PreconditionFailed) => continue,
Err(err) => return Err(err),
}
let mut changed = false;
let record = update_manual_transition_job_record(api.clone(), job_id, expected_lease_id, |record| {
changed = record.mark_unknown_for_worker_result_journal_error(error.clone(), queue_snapshot);
changed
})
.await?;
if changed && record.is_terminal() {
delete_manual_transition_scope_admission_if_current(api, &record.scope_key, record.job_id, record.lease_id).await?;
}
Err(Error::PreconditionFailed)
Ok(record)
}
pub async fn save_manual_transition_scope_admission_if_absent(
@@ -1603,19 +1762,14 @@ async fn find_active_legacy_manual_transition_scope_conflict(
}
pub async fn request_manual_transition_job_cancel(api: Arc<ECStore>, job_id: Uuid) -> EcstoreResult<ManualTransitionJobRecord> {
for _ in 0..4 {
let (mut record, etag) = load_manual_transition_job_record_with_etag(api.clone(), job_id).await?;
update_manual_transition_job_record(api, job_id, None, |record| {
if record.is_terminal() || record.cancel_requested {
return Ok(record);
return false;
}
record.mark_cancel_requested();
match save_manual_transition_job_record_if_current(api.clone(), &record, &etag).await {
Ok(()) => return Ok(record),
Err(Error::PreconditionFailed) => continue,
Err(err) => return Err(err),
}
}
Err(Error::PreconditionFailed)
true
})
.await
}
pub async fn persist_manual_transition_job_progress(
@@ -1624,10 +1778,39 @@ pub async fn persist_manual_transition_job_progress(
report: &ManualTransitionRunReport,
queue_snapshot: ManualTransitionQueueSnapshot,
) -> EcstoreResult<ManualTransitionJobRecord> {
let (mut record, etag) = load_manual_transition_job_record_with_etag(api.clone(), job_id).await?;
record.update_running_progress(report.clone(), queue_snapshot);
save_manual_transition_job_record_if_current(api.clone(), &record, &etag).await?;
renew_manual_transition_scope_admission_from_job(api, &record).await?;
let current = load_manual_transition_job_record_with_etag(api.clone(), job_id).await?;
persist_manual_transition_job_progress_inner(api, job_id, current.0.lease_id, Some(current), report, queue_snapshot).await
}
pub async fn persist_manual_transition_job_progress_if_owned(
api: Arc<ECStore>,
job_id: Uuid,
expected_lease_id: Uuid,
report: &ManualTransitionRunReport,
queue_snapshot: ManualTransitionQueueSnapshot,
) -> EcstoreResult<ManualTransitionJobRecord> {
persist_manual_transition_job_progress_inner(api, job_id, expected_lease_id, None, report, queue_snapshot).await
}
async fn persist_manual_transition_job_progress_inner(
api: Arc<ECStore>,
job_id: Uuid,
expected_lease_id: Uuid,
current: Option<(ManualTransitionJobRecord, String)>,
report: &ManualTransitionRunReport,
queue_snapshot: ManualTransitionQueueSnapshot,
) -> EcstoreResult<ManualTransitionJobRecord> {
let record = update_manual_transition_job_record_from(api.clone(), job_id, Some(expected_lease_id), current, |record| {
if record.state != ManualTransitionJobState::Running {
return false;
}
record.update_running_progress(report.clone(), queue_snapshot);
true
})
.await?;
if record.state == ManualTransitionJobState::Running {
renew_manual_transition_scope_admission_from_job(api, &record).await?;
}
Ok(record)
}
@@ -1661,25 +1844,58 @@ pub async fn renew_manual_transition_job_lease(
job_id: Uuid,
queue_snapshot: ManualTransitionQueueSnapshot,
) -> EcstoreResult<ManualTransitionJobRecord> {
let (mut record, mut etag) = load_manual_transition_job_record_with_etag(api.clone(), job_id).await?;
if record.state == ManualTransitionJobState::Running {
if record.scan_completed && queue_snapshot.queued == 0 && queue_snapshot.active == 0 {
record = reconcile_manual_transition_worker_results(api.clone(), job_id, queue_snapshot).await?;
if record.is_terminal() || !record.report.worker_transition_pending() {
return Ok(record);
let current = load_manual_transition_job_record_with_etag(api.clone(), job_id).await?;
renew_manual_transition_job_lease_inner(api, job_id, current.0.lease_id, Some(current), queue_snapshot).await
}
pub async fn renew_manual_transition_job_lease_if_owned(
api: Arc<ECStore>,
job_id: Uuid,
expected_lease_id: Uuid,
queue_snapshot: ManualTransitionQueueSnapshot,
) -> EcstoreResult<ManualTransitionJobRecord> {
renew_manual_transition_job_lease_inner(api, job_id, expected_lease_id, None, queue_snapshot).await
}
async fn renew_manual_transition_job_lease_inner(
api: Arc<ECStore>,
job_id: Uuid,
expected_lease_id: Uuid,
current: Option<(ManualTransitionJobRecord, String)>,
queue_snapshot: ManualTransitionQueueSnapshot,
) -> EcstoreResult<ManualTransitionJobRecord> {
let (current, current_etag) = match current {
Some(current) => current,
None => load_manual_transition_job_record_with_etag(api.clone(), job_id).await?,
};
if current.lease_id != expected_lease_id {
return Err(Error::PreconditionFailed);
}
if current.state != ManualTransitionJobState::Running {
return Ok(current);
}
if current.scan_completed && queue_snapshot.queued == 0 && queue_snapshot.active == 0 {
return reconcile_manual_transition_worker_results_inner(api, job_id, Some(expected_lease_id), queue_snapshot, true)
.await;
}
let record = update_manual_transition_job_record_from(
api.clone(),
job_id,
Some(expected_lease_id),
Some((current, current_etag)),
|record| {
if record.state != ManualTransitionJobState::Running {
return false;
}
(record, etag) = load_manual_transition_job_record_with_etag(api.clone(), job_id).await?;
}
let became_terminal = record.mark_unknown_if_worker_results_lost(queue_snapshot);
if !became_terminal {
record.renew_lease(queue_snapshot);
}
save_manual_transition_job_record_if_current(api.clone(), &record, &etag).await?;
if became_terminal {
delete_manual_transition_scope_admission_if_current(api, &record.scope_key, record.job_id, record.lease_id).await?;
} else {
renew_manual_transition_scope_admission_from_job(api, &record).await?;
}
true
},
)
.await?;
if record.is_terminal() {
delete_manual_transition_scope_admission_if_current(api, &record.scope_key, record.job_id, record.lease_id).await?;
} else if record.state == ManualTransitionJobState::Running {
renew_manual_transition_scope_admission_from_job(api, &record).await?;
}
Ok(record)
}
@@ -1688,15 +1904,31 @@ async fn renew_manual_transition_scope_admission_from_job(
api: Arc<ECStore>,
record: &ManualTransitionJobRecord,
) -> EcstoreResult<()> {
if let Ok((admission, admission_etag)) =
load_manual_transition_scope_admission_with_etag(api.clone(), &record.scope_key).await
&& admission.job_id == record.job_id
&& admission.lease_id == record.lease_id
{
let renewed_admission = ManualTransitionScopeAdmission::from_job(record);
save_manual_transition_scope_admission_if_current(api, &renewed_admission, &admission_etag).await?;
for _ in 0..MANUAL_TRANSITION_JOB_CAS_RETRIES {
let (admission, admission_etag) =
match load_manual_transition_scope_admission_with_etag(api.clone(), &record.scope_key).await {
Ok(admission) => admission,
Err(Error::ConfigNotFound) => return Ok(()),
Err(err) => return Err(err),
};
if admission.job_id != record.job_id || admission.lease_id != record.lease_id {
return Err(Error::PreconditionFailed);
}
let mut renewed_admission = ManualTransitionScopeAdmission::from_job(record);
renewed_admission.lease_expires_at_unix_nanos = renewed_admission
.lease_expires_at_unix_nanos
.max(admission.lease_expires_at_unix_nanos);
renewed_admission.updated_at_unix_nanos = renewed_admission.updated_at_unix_nanos.max(admission.updated_at_unix_nanos);
if renewed_admission == admission {
return Ok(());
}
match save_manual_transition_scope_admission_if_current(api.clone(), &renewed_admission, &admission_etag).await {
Ok(()) => return Ok(()),
Err(Error::PreconditionFailed) => continue,
Err(err) => return Err(err),
}
}
Ok(())
Err(Error::PreconditionFailed)
}
pub async fn delete_manual_transition_scope_admission_if_current(
@@ -2386,14 +2618,14 @@ mod tests {
}
#[test]
fn manual_transition_job_record_failure_counts_tier_failure() {
fn manual_transition_job_record_control_plane_failure_does_not_count_tier_failure() {
let options = ManualTransitionRunOptions::default();
let mut record = ManualTransitionJobRecord::new(Uuid::new_v4(), "bucket", &options, TEST_OWNER);
record.fail("missing tier");
assert_eq!(record.state, ManualTransitionJobState::Failed);
assert_eq!(record.report.tier_failure, 1);
assert_eq!(record.report.tier_failure, 0);
assert_eq!(record.error.as_deref(), Some("missing tier"));
}
+12 -2
View File
@@ -27,12 +27,24 @@ use crate::client::utils::base64_decode;
use crate::client::utils::base64_encode;
use crate::client::{api_put_object::PutObjectOptions, api_s3_datatypes::ObjectPart};
use crate::{disk::DiskAPI, object_api::GetObjectReader};
// s3s::header has no CRC64NVME constant yet; the canonical RustFS copy lives
// in rustfs-utils' headers module.
use rustfs_utils::http::headers::AMZ_CHECKSUM_CRC64NVME;
use s3s::header::{
X_AMZ_CHECKSUM_ALGORITHM, X_AMZ_CHECKSUM_CRC32, X_AMZ_CHECKSUM_CRC32C, X_AMZ_CHECKSUM_SHA1, X_AMZ_CHECKSUM_SHA256,
};
use enumset::{EnumSet, EnumSetType, enum_set};
/// One of three deliberately separate checksum registries (backlog#1833):
/// this enum is the MinIO-port client's wire vocabulary and stops at the
/// standard S3 set (CRC64NVME is its newest member; the RustFS extensions do
/// not exist on this client path). The streaming-hash registry lives in
/// `rustfs_checksums::ChecksumAlgorithm` (crates/checksums/src/lib.rs) and
/// the on-disk xl.meta bitset in `rustfs_rio::ChecksumType`
/// (crates/rio/src/checksum.rs, varint bits are append-only). When adding an
/// algorithm, extend all three (or record why not) — they do not derive from
/// each other.
#[derive(Debug, EnumSetType, Default)]
#[enumset(repr = "u8")]
pub enum ChecksumMode {
@@ -57,8 +69,6 @@ lazy_static! {
static ref C_ChecksumFullObjectCRC32C: EnumSet<ChecksumMode> =
enum_set!(ChecksumMode::ChecksumCRC32C | ChecksumMode::ChecksumFullObject);
}
const AMZ_CHECKSUM_CRC64NVME: &str = "x-amz-checksum-crc64nvme";
impl ChecksumMode {
//pub const CRC64_NVME_POLYNOMIAL: i64 = 0xad93d23594c93659;
-1
View File
@@ -13,7 +13,6 @@
// limitations under the License.
// #730: cluster/RPC migration leaves transport capabilities staged for upcoming owners.
#![allow(dead_code)]
mod control_plane;
pub(crate) mod rpc;
+1
View File
@@ -256,6 +256,7 @@ impl<S> ReplayScopeChannel<S> {
}
}
#[allow(dead_code, reason = "replay-state probe asserted by this file's tests (backlog#1823)")]
fn peer_replay_state(audience: &str) -> PeerReplayState {
PEER_REPLAY_STATES
.lock()
@@ -31,7 +31,7 @@ use rustfs_config::{
DEFAULT_INTERNODE_DATA_TRANSPORT, ENV_RUSTFS_INTERNODE_DATA_TRANSPORT, INTERNODE_DATA_TRANSPORT_TCP,
KNOWN_INTERNODE_DATA_TRANSPORT_BACKENDS,
};
use rustfs_rio::{HttpReader, HttpWriter};
use rustfs_rio::{ChunkReaderBox, HttpChunkReader, HttpReader, HttpWriter};
use sha2::{Digest, Sha256};
use std::collections::HashMap;
use std::future::Future;
@@ -43,6 +43,10 @@ use tokio::io::{AsyncReadExt, AsyncWrite};
use tokio::sync::OnceCell;
use uuid::Uuid;
#[allow(
dead_code,
reason = "live in the cfg(not(test)) half of build_internode_data_transport_from_env (backlog#1823)"
)]
static INTERNODE_DATA_TRANSPORT: OnceLock<std::result::Result<Arc<dyn InternodeDataTransport>, String>> = OnceLock::new();
const READ_FILE_STREAM_PATH: &str = "/rustfs/rpc/read_file_stream";
@@ -134,6 +138,10 @@ fn put_file_capability_status_is_legacy(status: u16) -> bool {
}
#[derive(Debug, Clone, Copy, Eq, PartialEq)]
#[allow(
dead_code,
reason = "capability-negotiation seam; constructed only by transport test doubles (backlog#1823)"
)]
pub struct InternodeDataTransportCapabilities {
/// Backend can open a streaming remote disk reader.
pub streaming_read: bool,
@@ -150,6 +158,10 @@ pub struct InternodeDataTransportCapabilities {
}
impl InternodeDataTransportCapabilities {
#[allow(
dead_code,
reason = "capability-negotiation seam; used by transport test doubles (backlog#1823)"
)]
pub const fn tcp_http() -> Self {
Self {
streaming_read: true,
@@ -221,6 +233,11 @@ pub struct NsScannerCapabilityRequest {
#[async_trait]
pub trait InternodeDataTransport: Send + Sync + std::fmt::Debug {
async fn open_read(&self, request: ReadStreamRequest) -> Result<FileReader>;
/// Opens an owned-chunk stream when this transport can retain receive-buffer
/// ownership. `None` preserves the established `open_read` fallback.
async fn open_read_chunks(&self, _request: ReadStreamRequest) -> Result<Option<ChunkReaderBox>> {
Ok(None)
}
async fn open_write(&self, request: WriteStreamRequest) -> Result<FileWriter>;
async fn open_walk_dir(&self, request: WalkDirStreamRequest) -> Result<FileReader>;
async fn open_ns_scanner(&self, _request: NsScannerStreamRequest) -> Result<FileReader> {
@@ -229,7 +246,12 @@ pub trait InternodeDataTransport: Send + Sync + std::fmt::Debug {
async fn probe_ns_scanner(&self, _request: NsScannerCapabilityRequest) -> Result<Uuid> {
Err(Error::MethodNotAllowed)
}
// Interface facet nobody calls yet: every transport implements both, but no
// caller negotiates on them. Kept for the internode transport split
// (backlog#1350); deleting them would delete the seam and six impls.
#[allow(dead_code, reason = "unused capability-negotiation facet (backlog#1823)")]
fn name(&self) -> &'static str;
#[allow(dead_code, reason = "unused capability-negotiation facet (backlog#1823)")]
fn capabilities(&self) -> InternodeDataTransportCapabilities;
}
@@ -247,6 +269,15 @@ impl InternodeDataTransport for TcpHttpInternodeDataTransport {
))
}
async fn open_read_chunks(&self, request: ReadStreamRequest) -> Result<Option<ChunkReaderBox>> {
let url = build_read_file_stream_url(&request);
let mut headers = json_headers();
build_auth_headers(&url, &Method::GET, &mut headers)?;
Ok(Some(Box::new(
HttpChunkReader::new_with_stall_timeout(url, Method::GET, headers, None, request.stall_timeout).await?,
)))
}
async fn open_write(&self, request: WriteStreamRequest) -> Result<FileWriter> {
let server_epoch = self.put_file_auth_capability(&request.endpoint).await?;
let nonce = server_epoch.map(|_| Uuid::new_v4());
@@ -656,6 +687,10 @@ fn build_internode_data_transport_result(
}
}
#[allow(
dead_code,
reason = "live in the cfg(test) half of build_internode_data_transport_from_env, which bypasses the process static (backlog#1823)"
)]
pub fn build_internode_data_transport(configured_transport: Option<&str>) -> Result<Arc<dyn InternodeDataTransport>> {
build_internode_data_transport_result(configured_transport).map_err(Error::other)
}
@@ -854,7 +854,6 @@ impl PeerS3Client for LocalPeerS3Client {
#[derive(Debug)]
pub struct RemotePeerS3Client {
pub node: Option<Node>,
pub pools: Option<Vec<usize>>,
addr: String,
/// Health tracker for connection monitoring
@@ -886,7 +885,6 @@ impl RemotePeerS3Client {
pub fn new(node: Option<Node>, pools: Option<Vec<usize>>) -> Self {
let addr = node.as_ref().map(|v| v.url.to_string()).unwrap_or_default();
let client = Self {
node,
pools,
addr,
health: Arc::new(DiskHealthTracker::new()),
@@ -905,10 +903,6 @@ impl RemotePeerS3Client {
.map_err(|err| Error::other(format!("can not get client, err: {err}")))
}
pub fn get_addr(&self) -> String {
self.addr.clone()
}
/// Start health monitoring for the remote peer
fn start_health_monitoring(&self) {
let health = Arc::clone(&self.health);
@@ -1208,6 +1202,10 @@ impl PeerS3Client for RemotePeerS3Client {
}
}
#[allow(
dead_code,
reason = "local bucket-heal path reached only by this file's tests (backlog#1823)"
)]
pub async fn heal_bucket_local(bucket: &str, opts: &HealOpts) -> Result<HealResultItem> {
let disks = clone_drives().await;
heal_bucket_local_on_disks(bucket, opts, disks).await
@@ -1404,6 +1402,10 @@ pub(crate) async fn heal_bucket_local_on_disks(
}
}
#[allow(
dead_code,
reason = "reached only through heal_bucket_local, which only tests call (backlog#1823)"
)]
async fn clone_drives() -> Vec<Option<DiskStore>> {
runtime_sources::local_disk_entries().await
}
@@ -1585,15 +1587,7 @@ mod tests {
}
fn test_remote_peer(addr: &str) -> RemotePeerS3Client {
let node = Node {
url: url::Url::parse(addr).expect("test peer URL should parse"),
pools: vec![0],
is_local: false,
grid_host: addr.to_string(),
};
RemotePeerS3Client {
node: Some(node),
pools: Some(vec![0]),
addr: addr.to_string(),
health: Arc::new(DiskHealthTracker::new()),
+594 -26
View File
@@ -522,6 +522,33 @@ impl RemoteDisk {
}
}
async fn open_read_chunks_with_retry(&self, request: ReadStreamRequest) -> Result<Option<rustfs_rio::ChunkReaderBox>> {
let mut attempt = 1;
let mut last_retry_classification = None;
loop {
match self.data_transport.open_read_chunks(request.clone()).await {
Ok(reader) => {
if attempt > 1
&& let Some(classification) = last_retry_classification
{
crate::cluster::rpc::runtime_sources::record_remote_disk_open_read_retry_success(classification);
}
return Ok(reader);
}
Err(err) if attempt < REMOTE_DISK_OPEN_READ_MAX_ATTEMPTS && Self::is_retryable_open_read_error(&err) => {
if let Some(classification) = err.internode_http_error_kind() {
let classification = classification.metric_label();
crate::cluster::rpc::runtime_sources::record_remote_disk_open_read_retry(classification);
last_retry_classification = Some(classification);
}
tokio::time::sleep(REMOTE_DISK_OPEN_READ_RETRY_BACKOFF).await;
attempt += 1;
}
Err(err) => return Err(err),
}
}
}
pub fn record_capacity_probe(&self, total: u64, used: u64, free: u64) {
self.health.record_capacity_probe(total, used, free);
}
@@ -846,31 +873,49 @@ impl RemoteDisk {
/// default to 1 (see [`internode_idempotent_read_retries`]). MUST NOT be used for write/lock
/// RPCs — those must never auto-retry (quorum/idempotency safety). The `operation` closure is
/// re-invoked per attempt, so it must be `Fn` (rebuild the request from borrowed inputs, do not
/// move captured state out).
/// move captured state out). Attempts and backoff share one total timeout budget.
async fn execute_read_with_retry<T, F, Fut>(&self, op: &'static str, operation: F, timeout_duration: Duration) -> Result<T>
where
F: Fn() -> Fut,
Fut: std::future::Future<Output = Result<T>>,
{
let deadline = (!timeout_duration.is_zero()).then(|| {
time::Instant::now()
.checked_add(timeout_duration)
.unwrap_or_else(|| time::sleep(timeout_duration).deadline())
});
let max_retries = internode_idempotent_read_retries();
let mut attempt = 0usize;
loop {
// Only the final attempt marks the disk faulty / evicts the channel. Earlier retries
// ignore the failure, so a transient error cannot flip the disk into a faulty
// short-circuit (which would defeat the retry) or over-count failures.
let attempt_timeout = deadline
.map(|deadline| deadline.saturating_duration_since(time::Instant::now()))
.unwrap_or(Duration::ZERO);
if deadline.is_some() && attempt_timeout.is_zero() {
self.record_timeout(op, timeout_duration);
return Err(DiskError::Timeout);
}
let health_action = if attempt >= max_retries {
FailureHealthAction::MarkFailure
} else {
FailureHealthAction::IgnoreFailure
};
match self
.execute_with_timeout_for_op_and_health_action(op, &operation, timeout_duration, health_action)
.execute_with_timeout_for_op_and_health_action(op, &operation, attempt_timeout, health_action)
.await
{
Err(err) if attempt < max_retries && is_network_like_disk_error(&err) => {
if matches!(err, DiskError::Timeout) && deadline.is_some_and(|deadline| time::Instant::now() >= deadline) {
self.mark_faulty("read_operation_deadline");
return Err(err);
}
attempt += 1;
let backoff = REMOTE_DISK_READ_RETRY_BASE_BACKOFF
.saturating_mul(1u32 << u32::try_from(attempt - 1).unwrap_or(4).min(4));
if deadline.is_some_and(|deadline| deadline.saturating_duration_since(time::Instant::now()) <= backoff) {
attempt = max_retries;
continue;
}
debug!(
endpoint = %self.endpoint,
addr = %self.addr,
@@ -878,7 +923,17 @@ impl RemoteDisk {
attempt,
"retrying idempotent read-only RPC after transient network error"
);
tokio::time::sleep(backoff).await;
if let Some(deadline) = deadline {
if time::timeout_at(deadline, time::sleep(backoff)).await.is_err() {
self.record_timeout(op, timeout_duration);
return Err(DiskError::Timeout);
}
} else {
time::sleep(backoff).await;
}
if self.health.is_faulty() {
return Err(DiskError::FaultyDisk);
}
}
other => return other,
}
@@ -957,32 +1012,35 @@ impl RemoteDisk {
operation_result
}
Err(_) => {
// Timeout occurred, mark disk as potentially faulty
counter!(
"rustfs_drive_op_timeout_total",
"endpoint" => self.endpoint.to_string(),
"op" => op.to_string()
)
.increment(1);
self.record_timeout(op, timeout_duration);
if failure_health_action == FailureHealthAction::MarkFailure {
self.mark_faulty_and_evict("operation_timeout").await;
}
warn!(
event = EVENT_REMOTE_DISK_RPC,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REMOTE_DISK,
endpoint = %self.endpoint,
addr = %self.addr,
op,
timeout_ms = timeout_duration.as_millis(),
state = "timeout",
"Remote disk operation timed out"
);
Err(DiskError::Timeout)
}
}
}
fn record_timeout(&self, op: &'static str, timeout_duration: Duration) {
counter!(
"rustfs_drive_op_timeout_total",
"endpoint" => self.endpoint.to_string(),
"op" => op.to_string()
)
.increment(1);
warn!(
event = EVENT_REMOTE_DISK_RPC,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_REMOTE_DISK,
endpoint = %self.endpoint,
addr = %self.addr,
op,
timeout_ms = timeout_duration.as_millis(),
state = "timeout",
"Remote disk operation timed out"
);
}
async fn handle_network_like_error<T>(
&self,
op: &'static str,
@@ -1016,7 +1074,7 @@ impl RemoteDisk {
}
}
async fn mark_faulty_and_evict(&self, reason: &'static str) {
fn mark_faulty(&self, reason: &'static str) -> bool {
let previous_state = self.runtime_state();
let transitioned_to_offline = self.mark_suspect_or_offline(reason);
let state = self.runtime_state();
@@ -1053,6 +1111,12 @@ impl RemoteDisk {
"Remote disk marked suspect"
);
}
}
state != previous_state
}
async fn mark_faulty_and_evict(&self, reason: &'static str) {
if self.mark_faulty(reason) {
counter!(
"rustfs_drive_connection_evict_total",
"endpoint" => self.endpoint.to_string(),
@@ -2068,7 +2132,7 @@ impl DiskAPI for RemoteDisk {
Ok(file_info)
},
get_max_timeout_duration(),
get_drive_metadata_timeout(),
)
.await
}
@@ -2417,6 +2481,30 @@ impl DiskAPI for RemoteDisk {
.await
}
async fn read_file_stream_chunks(
&self,
volume: &str,
path: &str,
offset: usize,
length: usize,
) -> Result<Option<rustfs_rio::ChunkReaderBox>> {
if self.health.is_faulty() {
return Err(DiskError::FaultyDisk);
}
let disk = self.disk_ref().await;
let stall_timeout = get_object_disk_read_timeout();
self.open_read_chunks_with_retry(ReadStreamRequest {
endpoint: self.endpoint.grid_host(),
disk,
volume: volume.to_string(),
path: path.to_string(),
offset,
length,
stall_timeout: (!stall_timeout.is_zero()).then_some(stall_timeout),
})
.await
}
/// Buffered read for remote disks.
/// The transport stream is collected into owned Bytes for caller sharing.
#[tracing::instrument(level = "trace", skip_all)]
@@ -5094,6 +5182,452 @@ mod tests {
);
}
#[tokio::test(start_paused = true)]
#[serial(remote_disk_read_retry)]
async fn execute_read_with_retry_reset_during_backoff_preserves_recovery() {
let remote_disk = Arc::new(new_remote_disk_with_transport(Arc::new(RecordingInternodeDataTransport::default())).await);
let attempts = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let first_attempt = Arc::new(tokio::sync::Notify::new());
let started = time::Instant::now();
let task_disk = Arc::clone(&remote_disk);
let task_attempts = Arc::clone(&attempts);
let task_first_attempt = Arc::clone(&first_attempt);
let task = tokio::spawn(async move {
task_disk
.execute_read_with_retry(
"read_version",
move || {
let attempt = task_attempts.fetch_add(1, Ordering::SeqCst);
let first_attempt = Arc::clone(&task_first_attempt);
async move {
if attempt == 0 {
time::sleep(Duration::from_millis(20)).await;
first_attempt.notify_one();
return Err::<(), Error>(DiskError::Io(std_io::Error::new(
std_io::ErrorKind::ConnectionRefused,
"connection refused",
)));
}
Ok(())
}
},
Duration::from_millis(100),
)
.await
});
first_attempt.notified().await;
tokio::task::yield_now().await;
remote_disk.health.reset_for_store_init_retry(&remote_disk.endpoint);
let channel = TonicEndpoint::from_shared(remote_disk.addr.clone())
.expect("remote disk address should parse")
.connect_lazy();
runtime_sources::cache_test_node_channel(remote_disk.addr.clone(), channel).await;
task.await
.expect("retry task should finish")
.expect("the retry should succeed after the health reset");
assert_eq!(attempts.load(Ordering::SeqCst), 2);
assert_eq!(started.elapsed(), Duration::from_millis(70));
assert_eq!(
remote_disk.health.waiting_count(),
0,
"health reset must not underflow the waiting counter"
);
assert_eq!(remote_disk.runtime_state(), RuntimeDriveHealthState::Online);
assert!(
runtime_sources::test_node_channel_is_cached(&remote_disk.addr).await,
"a recovered channel must survive the retry backoff"
);
remote_disk.cancel_token.cancel();
}
#[tokio::test(start_paused = true)]
#[serial(remote_disk_read_retry)]
async fn execute_read_with_retry_still_retries_within_shared_deadline() {
let remote_disk = new_remote_disk_with_transport(Arc::new(RecordingInternodeDataTransport::default())).await;
let attempts = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let channel = TonicEndpoint::from_shared(remote_disk.addr.clone())
.expect("remote disk address should parse")
.connect_lazy();
runtime_sources::cache_test_node_channel(remote_disk.addr.clone(), channel).await;
remote_disk
.execute_read_with_retry(
"read_version",
|| {
let attempt = attempts.fetch_add(1, Ordering::SeqCst);
async move {
if attempt == 0 {
return Err::<(), Error>(DiskError::Io(std_io::Error::new(
std_io::ErrorKind::ConnectionReset,
"connection reset",
)));
}
Ok(())
}
},
Duration::from_millis(100),
)
.await
.expect("a retry that fits the shared deadline should succeed");
assert_eq!(attempts.load(Ordering::SeqCst), 2);
assert_eq!(remote_disk.runtime_state(), RuntimeDriveHealthState::Online);
assert!(runtime_sources::test_node_channel_is_cached(&remote_disk.addr).await);
remote_disk.cancel_token.cancel();
}
#[tokio::test(start_paused = true)]
#[serial(remote_disk_read_retry)]
async fn execute_read_with_retry_uses_remaining_budget_for_final_attempt() {
let remote_disk = new_remote_disk_with_transport(Arc::new(RecordingInternodeDataTransport::default())).await;
let attempts = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let started = time::Instant::now();
let channel = TonicEndpoint::from_shared(remote_disk.addr.clone())
.expect("remote disk address should parse")
.connect_lazy();
runtime_sources::cache_test_node_channel(remote_disk.addr.clone(), channel).await;
let err = remote_disk
.execute_read_with_retry(
"read_version",
|| {
let attempt = attempts.fetch_add(1, Ordering::SeqCst);
async move {
if attempt == 0 {
time::sleep(Duration::from_millis(20)).await;
return Err::<(), Error>(DiskError::Io(std_io::Error::new(
std_io::ErrorKind::ConnectionRefused,
"connection refused",
)));
}
std::future::pending::<Result<()>>().await
}
},
Duration::from_millis(100),
)
.await
.expect_err("the final retry should consume only the remaining total budget");
assert_eq!(err, DiskError::Timeout);
assert_eq!(attempts.load(Ordering::SeqCst), 2);
assert_eq!(started.elapsed(), Duration::from_millis(100));
assert_eq!(remote_disk.runtime_state(), RuntimeDriveHealthState::Suspect);
assert!(!runtime_sources::test_node_channel_is_cached(&remote_disk.addr).await);
remote_disk.cancel_token.cancel();
}
#[tokio::test(start_paused = true)]
#[serial(remote_disk_read_retry)]
async fn execute_read_with_retry_uses_final_attempt_at_exact_backoff_boundary() {
let remote_disk = new_remote_disk_with_transport(Arc::new(RecordingInternodeDataTransport::default())).await;
let attempts = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let started = time::Instant::now();
let err = remote_disk
.execute_read_with_retry(
"read_version",
|| {
let attempt = attempts.fetch_add(1, Ordering::SeqCst);
async move {
if attempt == 0 {
time::sleep(Duration::from_millis(50)).await;
return Err::<(), Error>(DiskError::Io(std_io::Error::new(
std_io::ErrorKind::ConnectionRefused,
"connection refused",
)));
}
std::future::pending::<Result<()>>().await
}
},
Duration::from_millis(100),
)
.await
.expect_err("the exact backoff boundary should be reserved for a final attempt");
assert_eq!(err, DiskError::Timeout);
assert_eq!(attempts.load(Ordering::SeqCst), 2);
assert_eq!(started.elapsed(), Duration::from_millis(100));
assert_eq!(remote_disk.runtime_state(), RuntimeDriveHealthState::Suspect);
remote_disk.cancel_token.cancel();
}
#[tokio::test(start_paused = true)]
#[serial(remote_disk_read_retry)]
async fn execute_read_with_retry_uses_final_attempt_below_backoff_budget() {
let remote_disk = new_remote_disk_with_transport(Arc::new(RecordingInternodeDataTransport::default())).await;
let attempts = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let started = time::Instant::now();
let err = remote_disk
.execute_read_with_retry(
"read_version",
|| {
let attempt = attempts.fetch_add(1, Ordering::SeqCst);
async move {
if attempt == 0 {
time::sleep(Duration::from_millis(80)).await;
return Err::<(), Error>(DiskError::Io(std_io::Error::new(
std_io::ErrorKind::ConnectionRefused,
"connection refused",
)));
}
std::future::pending::<Result<()>>().await
}
},
Duration::from_millis(100),
)
.await
.expect_err("remaining budget below backoff should be reserved for a final attempt");
assert_eq!(err, DiskError::Timeout);
assert_eq!(attempts.load(Ordering::SeqCst), 2);
assert_eq!(started.elapsed(), Duration::from_millis(100));
assert_eq!(remote_disk.runtime_state(), RuntimeDriveHealthState::Suspect);
remote_disk.cancel_token.cancel();
}
#[tokio::test(start_paused = true)]
#[serial(remote_disk_read_retry)]
async fn execute_read_with_retry_zero_timeout_disables_the_deadline() {
let remote_disk = new_remote_disk_with_transport(Arc::new(RecordingInternodeDataTransport::default())).await;
let attempts = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let started = time::Instant::now();
remote_disk
.execute_read_with_retry(
"read_version",
|| {
let attempt = attempts.fetch_add(1, Ordering::SeqCst);
async move {
if attempt == 0 {
return Err::<(), Error>(DiskError::Io(std_io::Error::new(
std_io::ErrorKind::ConnectionReset,
"connection reset",
)));
}
Ok(())
}
},
Duration::ZERO,
)
.await
.expect("zero timeout should allow a retry without a deadline");
assert_eq!(attempts.load(Ordering::SeqCst), 2);
assert_eq!(started.elapsed(), REMOTE_DISK_READ_RETRY_BASE_BACKOFF);
remote_disk.cancel_token.cancel();
}
#[tokio::test]
#[serial(remote_disk_read_retry)]
async fn execute_read_with_retry_accepts_max_metadata_timeout() {
temp_env::async_with_vars([(rustfs_config::ENV_DRIVE_METADATA_TIMEOUT_SECS, Some(u64::MAX.to_string()))], async {
let remote_disk = new_remote_disk_with_transport(Arc::new(RecordingInternodeDataTransport::default())).await;
remote_disk
.execute_read_with_retry("read_version", || async { Ok::<(), Error>(()) }, get_drive_metadata_timeout())
.await
.expect("the maximum configured metadata timeout must not panic");
remote_disk.cancel_token.cancel();
})
.await;
}
#[tokio::test(start_paused = true)]
#[serial(remote_disk_read_retry)]
async fn execute_read_with_retry_zero_retries_runs_once() {
temp_env::async_with_vars([(rustfs_config::ENV_INTERNODE_IDEMPOTENT_READ_RETRIES, Some("0"))], async {
let remote_disk = new_remote_disk_with_transport(Arc::new(RecordingInternodeDataTransport::default())).await;
let attempts = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let started = time::Instant::now();
let channel = TonicEndpoint::from_shared(remote_disk.addr.clone())
.expect("remote disk address should parse")
.connect_lazy();
runtime_sources::cache_test_node_channel(remote_disk.addr.clone(), channel).await;
let err = remote_disk
.execute_read_with_retry(
"read_version",
|| {
attempts.fetch_add(1, Ordering::SeqCst);
async {
Err::<(), Error>(DiskError::Io(std_io::Error::new(
std_io::ErrorKind::ConnectionReset,
"connection reset",
)))
}
},
Duration::from_secs(1),
)
.await
.expect_err("zero retries should return the first network error");
assert!(matches!(err, DiskError::Io(ref io_err) if io_err.kind() == std_io::ErrorKind::ConnectionReset));
assert_eq!(attempts.load(Ordering::SeqCst), 1);
assert_eq!(started.elapsed(), Duration::ZERO);
assert_eq!(remote_disk.runtime_state(), RuntimeDriveHealthState::Suspect);
assert!(!runtime_sources::test_node_channel_is_cached(&remote_disk.addr).await);
remote_disk.cancel_token.cancel();
})
.await;
}
#[tokio::test(start_paused = true)]
#[serial(remote_disk_read_retry)]
async fn execute_read_with_retry_attempt_timeout_marks_health_without_evicting() {
let remote_disk = new_remote_disk_with_transport(Arc::new(RecordingInternodeDataTransport::default())).await;
let recorder = crate::test_metrics::CapturingRecorder::default();
let _recorder_guard = metrics::set_default_local_recorder(&recorder);
let attempts = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let channel = TonicEndpoint::from_shared(remote_disk.addr.clone())
.expect("remote disk address should parse")
.connect_lazy();
runtime_sources::cache_test_node_channel(remote_disk.addr.clone(), channel).await;
let err = remote_disk
.execute_read_with_retry(
"read_version",
|| {
attempts.fetch_add(1, Ordering::SeqCst);
std::future::pending::<Result<()>>()
},
Duration::from_millis(100),
)
.await
.expect_err("an in-flight attempt that consumes the deadline should time out");
assert_eq!(err, DiskError::Timeout);
assert_eq!(attempts.load(Ordering::SeqCst), 1);
assert_eq!(remote_disk.runtime_state(), RuntimeDriveHealthState::Suspect);
assert!(runtime_sources::test_node_channel_is_cached(&remote_disk.addr).await);
assert_eq!(
recorder.counter_value(
"rustfs_drive_op_timeout_total",
&[
("endpoint", remote_disk.endpoint.to_string().as_str()),
("op", "read_version")
]
),
1
);
remote_disk.cancel_token.cancel();
}
#[tokio::test(start_paused = true)]
#[serial(remote_disk_read_retry)]
async fn execute_read_with_retry_does_not_retry_business_errors() {
let remote_disk = new_remote_disk_with_transport(Arc::new(RecordingInternodeDataTransport::default())).await;
let attempts = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let err = remote_disk
.execute_read_with_retry(
"read_version",
|| {
attempts.fetch_add(1, Ordering::SeqCst);
async { Err::<(), Error>(DiskError::FileNotFound) }
},
Duration::from_secs(1),
)
.await
.expect_err("business errors should be returned directly");
assert_eq!(err, DiskError::FileNotFound);
assert_eq!(attempts.load(Ordering::SeqCst), 1);
remote_disk.cancel_token.cancel();
}
#[tokio::test(start_paused = true)]
#[serial(remote_disk_read_retry)]
async fn execute_read_with_retry_honors_configured_retry_count() {
temp_env::async_with_vars([(rustfs_config::ENV_INTERNODE_IDEMPOTENT_READ_RETRIES, Some("2"))], async {
let remote_disk = new_remote_disk_with_transport(Arc::new(RecordingInternodeDataTransport::default())).await;
let attempts = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let started = time::Instant::now();
let channel = TonicEndpoint::from_shared(remote_disk.addr.clone())
.expect("remote disk address should parse")
.connect_lazy();
runtime_sources::cache_test_node_channel(remote_disk.addr.clone(), channel).await;
let err = remote_disk
.execute_read_with_retry(
"read_version",
|| {
attempts.fetch_add(1, Ordering::SeqCst);
async {
Err::<(), Error>(DiskError::Io(std_io::Error::new(
std_io::ErrorKind::ConnectionReset,
"connection reset",
)))
}
},
Duration::from_secs(1),
)
.await
.expect_err("exhausted retries should return the last network error");
assert!(matches!(err, DiskError::Io(ref io_err) if io_err.kind() == std_io::ErrorKind::ConnectionReset));
assert_eq!(attempts.load(Ordering::SeqCst), 3);
assert_eq!(started.elapsed(), Duration::from_millis(150));
assert_eq!(remote_disk.runtime_state(), RuntimeDriveHealthState::Suspect);
assert!(!runtime_sources::test_node_channel_is_cached(&remote_disk.addr).await);
remote_disk.cancel_token.cancel();
})
.await;
}
#[tokio::test(start_paused = true)]
#[serial(remote_disk_read_retry)]
async fn execute_read_with_retry_stops_when_disk_turns_offline_during_backoff() {
let remote_disk = Arc::new(new_remote_disk_with_transport(Arc::new(RecordingInternodeDataTransport::default())).await);
let attempts = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let first_attempt = Arc::new(tokio::sync::Notify::new());
let task_disk = Arc::clone(&remote_disk);
let task_attempts = Arc::clone(&attempts);
let task_first_attempt = Arc::clone(&first_attempt);
let task = tokio::spawn(async move {
task_disk
.execute_read_with_retry(
"read_version",
move || {
let attempt = task_attempts.fetch_add(1, Ordering::SeqCst);
let first_attempt = Arc::clone(&task_first_attempt);
async move {
if attempt == 0 {
first_attempt.notify_one();
return Err::<(), Error>(DiskError::Io(std_io::Error::new(
std_io::ErrorKind::ConnectionReset,
"connection reset",
)));
}
Ok(())
}
},
Duration::from_secs(1),
)
.await
});
first_attempt.notified().await;
tokio::task::yield_now().await;
remote_disk
.health
.force_runtime_state_for_test(RuntimeDriveHealthState::Offline);
time::advance(REMOTE_DISK_READ_RETRY_BASE_BACKOFF).await;
let err = task
.await
.expect("retry task should finish")
.expect_err("an offline disk must stop before the next attempt");
assert_eq!(err, DiskError::FaultyDisk);
assert_eq!(attempts.load(Ordering::SeqCst), 1);
remote_disk.cancel_token.cancel();
}
#[tokio::test]
async fn test_execute_with_timeout_evicts_cached_connection() {
let addr = "http://127.0.0.1:59991".to_string();
@@ -5603,6 +6137,40 @@ mod tests {
accept_task.abort();
}
#[tokio::test]
async fn read_version_uses_the_metadata_timeout_on_a_stalled_peer() {
runtime_sources::ensure_test_rpc_secret();
let Some((base_addr, accept_task)) = spawn_stalled_grpc_peer().await else {
return;
};
let remote_disk = remote_disk_for_addr(&base_addr).await;
temp_env::async_with_vars(
[
(rustfs_config::ENV_DRIVE_METADATA_TIMEOUT_SECS, Some("1")),
(rustfs_config::ENV_DRIVE_MAX_TIMEOUT_DURATION, Some("10")),
],
async {
let started = time::Instant::now();
let err = tokio::time::timeout(
Duration::from_secs(5),
remote_disk.read_version("bucket", "bucket", "object", "", &ReadOptions::default()),
)
.await
.expect("read_version must use the shorter metadata deadline")
.expect_err("a stalled peer must fail read_version");
assert!(matches!(err, DiskError::Timeout), "expected the metadata deadline to fire, got {err:?}");
assert!(started.elapsed() >= Duration::from_millis(900));
assert!(started.elapsed() < Duration::from_secs(2));
},
)
.await;
remote_disk.cancel_token.cancel();
accept_task.abort();
}
#[tokio::test]
async fn delete_volume_bounds_the_wait_on_a_stalled_peer() {
runtime_sources::ensure_test_rpc_secret();
@@ -48,10 +48,6 @@ impl RemoteClient {
Self { addr: endpoint }
}
pub fn from_url(url: url::Url) -> Self {
Self { addr: url.to_string() }
}
fn build_ping_request() -> PingRequest {
let mut fbb = flatbuffers::FlatBufferBuilder::new();
let payload = fbb.create_vector(b"health-check");
-3
View File
@@ -46,7 +46,6 @@ use rustfs_config::{
SCANNER_SUB_SYS,
};
use rustfs_filemeta::FileInfo;
use rustfs_utils::path::SLASH_SEPARATOR;
use serde_json::{Map, Value};
use std::collections::{HashMap, HashSet};
use std::sync::LazyLock;
@@ -200,8 +199,6 @@ pub const STORAGE_CLASS_SUB_SYS: &str = "storage_class";
pub const COMMA_SEPARATED_LISTS: &[&str] = &[rustfs_config::oidc::OIDC_SCOPES, rustfs_config::oidc::OIDC_OTHER_AUDIENCES];
static CONFIG_BUCKET: LazyLock<String> = LazyLock::new(|| format!("{RUSTFS_META_BUCKET}{SLASH_SEPARATOR}{CONFIG_PREFIX}"));
type ServerConfigDecryptFn = crate::bucket::migration::LegacyBlobDecryptFn;
static SERVER_CONFIG_DECRYPT_FN: LazyLock<RwLock<Option<ServerConfigDecryptFn>>> = LazyLock::new(|| RwLock::new(None));
-1
View File
@@ -13,7 +13,6 @@
// limitations under the License.
// #730: configuration migration keeps legacy subsystem definitions available behind this module.
#![allow(dead_code)]
mod audit;
pub mod com;
+70 -20
View File
@@ -101,6 +101,7 @@ const DEFAULT_RRS_STORAGE_CLASS: &str = "EC:1";
const ZERO_SET_DRIVE_COUNT_ERROR: &str = "set drive count must be greater than zero";
pub static DEFAULT_INLINE_BLOCK: usize = 128 * 1024;
const DEFAULT_INLINE_OBJECT_BUDGET: usize = 2 * DEFAULT_INLINE_BLOCK;
pub static DEFAULT_KVS: LazyLock<KVS> = LazyLock::new(|| {
let kvs = vec![
@@ -150,6 +151,8 @@ pub struct Config {
optimize: Option<String>,
inline_block: usize,
initialized: bool,
#[serde(default, skip_serializing_if = "std::ops::Not::not")]
inline_block_explicit: bool,
#[serde(skip)]
standard_parities: Vec<PoolParity>,
#[serde(skip)]
@@ -186,6 +189,10 @@ impl Config {
/// A topology-bound lookup fails closed for unknown drive counts and for
/// deserialized legacy configurations that have no pool topology. Legacy
/// callers retain scalar compatibility through [`Self::get_parity_for_sc`].
#[allow(
dead_code,
reason = "per-set parity resolution asserted by this file's tests (backlog#1823)"
)]
pub(crate) fn parity_for_sc(&self, sc: &str, drives_per_set: usize) -> Option<usize> {
if !self.initialized {
return None;
@@ -233,17 +240,19 @@ impl Config {
.map(|(pool_index, pool)| (pool_index, pool.drives_per_set))
}
pub fn should_inline(&self, shard_size: i64, versioned: bool) -> bool {
if shard_size < 0 {
pub fn should_inline(&self, shard_size: i64, data_shards: usize, versioned: bool) -> bool {
if shard_size < 0 || data_shards == 0 {
return false;
}
let shard_size = shard_size as usize;
let mut inline_block = DEFAULT_INLINE_BLOCK;
if self.initialized {
inline_block = self.inline_block;
}
// Keep the historical two-data-shard object budget while preventing
// wider EC layouts from multiplying the maximum inline object size.
let inline_block = if self.initialized && self.inline_block_explicit {
self.inline_block
} else {
(DEFAULT_INLINE_OBJECT_BUDGET / data_shards).min(DEFAULT_INLINE_BLOCK)
};
if versioned {
shard_size <= inline_block / 8
@@ -392,6 +401,7 @@ fn lookup_config_for_pools_with_env(
}
let optimize = overrides.optimize;
let inline_block_explicit = overrides.inline_block.is_some();
let inline_block = if let Some(value) = overrides.inline_block {
let block = value
.parse::<bytesize::ByteSize>()
@@ -424,6 +434,7 @@ fn lookup_config_for_pools_with_env(
optimize,
inline_block,
initialized: true,
inline_block_explicit,
standard_parities,
rrs_parities,
})
@@ -541,22 +552,26 @@ mod tests {
}
#[test]
fn should_inline_preserves_exact_default_shard_boundaries() {
let config = Config::default();
fn should_inline_scales_default_threshold_by_data_shards() {
let config = lookup_config_for_pools_with_env(&KVS::new(), &[3, 12], no_env_overrides())
.expect("default inline policy should resolve for EC2+1 and EC8+4");
for (case, shard_size, versioned, expected) in [
("unversioned below", 128 * 1024 - 1, false, true),
("unversioned exact", 128 * 1024, false, true),
("unversioned above", 128 * 1024 + 1, false, false),
("versioned below", 16 * 1024 - 1, true, true),
("versioned exact", 16 * 1024, true, true),
("versioned above", 16 * 1024 + 1, true, false),
("negative", -1, false, false),
for (case, shard_size, data_shards, versioned, expected) in [
("EC2+1 unversioned exact", 128 * 1024, 2, false, true),
("EC2+1 unversioned above", 128 * 1024 + 1, 2, false, false),
("EC2+1 versioned exact", 16 * 1024, 2, true, true),
("EC2+1 versioned above", 16 * 1024 + 1, 2, true, false),
("EC8+4 unversioned exact", 32 * 1024, 8, false, true),
("EC8+4 unversioned above", 32 * 1024 + 1, 8, false, false),
("EC8+4 versioned exact", 4 * 1024, 8, true, true),
("EC8+4 versioned above", 4 * 1024 + 1, 8, true, false),
("negative", -1, 2, false, false),
("zero data shards", 0, 0, false, false),
] {
assert_eq!(
config.should_inline(shard_size, versioned),
config.should_inline(shard_size, data_shards, versioned),
expected,
"{case}: shard_size={shard_size}, versioned={versioned}"
"{case}: shard_size={shard_size}, data_shards={data_shards}, versioned={versioned}"
);
}
}
@@ -577,13 +592,28 @@ mod tests {
let shard_size = erasure.shard_file_size(object_size);
assert_eq!(shard_size, expected_shard_size, "{case}: object_size={object_size}");
assert_eq!(
config.should_inline(shard_size, versioned),
config.should_inline(shard_size, erasure.data_shards, versioned),
expected,
"{case}: object_size={object_size}, shard_size={shard_size}, versioned={versioned}"
);
}
}
#[test]
fn explicit_inline_block_preserves_fixed_per_shard_rollback() {
let overrides = StorageClassEnvOverrides {
inline_block: Some("128KiB".to_string()),
..Default::default()
};
let config = lookup_config_for_pools_with_env(&KVS::new(), &[12], overrides)
.expect("explicit inline block should resolve for EC8+4");
assert!(config.should_inline(128 * 1024, 8, false));
assert!(!config.should_inline(128 * 1024 + 1, 8, false));
assert!(config.should_inline(16 * 1024, 8, true));
assert!(!config.should_inline(16 * 1024 + 1, 8, true));
}
#[test]
fn write_capability_contract_only_accepts_implemented_layouts() {
assert_eq!(SUPPORTED_WRITE_CLASSES, [STANDARD, RRS]);
@@ -777,6 +807,7 @@ mod tests {
let encoded = serde_json::to_string(&cfg).expect("config should serialize");
assert!(!encoded.contains("standard_parities"));
assert!(!encoded.contains("rrs_parities"));
assert!(!encoded.contains("inline_block_explicit"));
let decoded: Config = serde_json::from_str(&encoded).expect("legacy scalar config should deserialize");
assert_eq!(decoded.get_parity_for_sc(STANDARD), Some(2));
@@ -786,6 +817,25 @@ mod tests {
assert!(validate_parity(0, 0).is_err());
}
#[test]
fn explicit_inline_block_survives_config_round_trip() {
let cfg = lookup_config_for_pools_with_env(
&KVS::new(),
&[12],
StorageClassEnvOverrides {
inline_block: Some("128KiB".to_string()),
..Default::default()
},
)
.expect("explicit inline block should resolve");
assert!(cfg.should_inline(100 * 1024, 8, false));
let encoded = serde_json::to_string(&cfg).expect("config should serialize");
assert!(encoded.contains("\"inline_block_explicit\":true"));
let decoded: Config = serde_json::from_str(&encoded).expect("explicit inline config should deserialize");
assert!(decoded.should_inline(100 * 1024, 8, false));
}
#[test]
fn lookup_config_reads_rrs_from_class_rrs_key() {
// Regression: kvs.get(RRS) used RRS="REDUCED_REDUNDANCY" instead of
-1
View File
@@ -13,7 +13,6 @@
// limitations under the License.
// #730: pool coordination helpers are being migrated behind runtime owners.
#![allow(dead_code)]
pub(crate) mod pools;
pub(crate) mod sets;
+85 -9
View File
@@ -226,6 +226,7 @@ fn ensure_decommission_start_rebalance_meta_allowed(meta: Option<&RebalanceMeta>
ensure_decommission_not_rebalancing(meta.is_some_and(is_rebalance_conflicting_with_decommission))
}
#[allow(dead_code, reason = "leader precondition asserted by this file's tests (backlog#1823)")]
fn ensure_local_decommission_pool_leaders(endpoints: &EndpointServerPools, indices: &[usize]) -> Result<()> {
for idx in indices {
ensure_local_decommission_pool_leader(endpoints, *idx)?;
@@ -1058,11 +1059,19 @@ fn should_cleanup_decommission_source_entry(decommissioned: usize, total_version
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[allow(
dead_code,
reason = "terminal-state classification asserted by this file's tests (backlog#1823)"
)]
enum DecommissionTerminalState {
Completed,
Failed,
}
#[allow(
dead_code,
reason = "terminal-state classification asserted by this file's tests (backlog#1823)"
)]
fn classify_decommission_terminal_state(failed_items_present: bool) -> DecommissionTerminalState {
if failed_items_present {
DecommissionTerminalState::Failed
@@ -2266,15 +2275,19 @@ fn decommission_delete_marker_opts(
version: &rustfs_filemeta::FileInfo,
version_id: Option<String>,
src_pool_idx: usize,
expected_bucket_incarnation_id: Option<uuid::Uuid>,
) -> ObjectOptions {
let version_suspended = version.version_id.is_none() && version_id.is_none();
ObjectOptions {
versioned: true,
version_id,
versioned: !version_suspended,
version_suspended,
version_id: version_id.or_else(|| version_suspended.then(|| uuid::Uuid::nil().to_string())),
mod_time: version.mod_time,
src_pool_idx,
data_movement: true,
delete_marker: true,
skip_decommissioned: true,
expected_bucket_incarnation_id,
delete_replication: version
.replication_state_internal
.as_ref()
@@ -2299,6 +2312,7 @@ fn decommission_remote_tiered_opts(
version: &rustfs_filemeta::FileInfo,
version_id: Option<String>,
src_pool_idx: usize,
expected_bucket_incarnation_id: Option<uuid::Uuid>,
) -> ObjectOptions {
ObjectOptions {
versioned: version_id.is_some(),
@@ -2307,6 +2321,9 @@ fn decommission_remote_tiered_opts(
user_defined: version.metadata.clone(),
src_pool_idx,
data_movement: true,
include_part_checksums: true,
http_preconditions: Some(crate::data_movement::data_movement_target_precondition()),
expected_bucket_incarnation_id,
..Default::default()
}
}
@@ -2805,6 +2822,7 @@ impl ECStore {
lifecycle_config: Option<BucketLifecycleConfiguration>,
object_lock_config: Option<ObjectLockConfiguration>,
replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
expected_bucket_incarnation_id: Option<uuid::Uuid>,
) -> Result<()> {
debug!(
event = EVENT_DECOMMISSION_ENTRY,
@@ -2834,6 +2852,11 @@ impl ECStore {
}
decommission_cancel_signal_result(rx.is_cancelled())?;
let bucket_incarnation_fence = match expected_bucket_incarnation_id {
Some(expected) => Some(self.acquire_bucket_incarnation_fence(&bucket, expected).await?),
None => None,
};
let mut fivs = load_decommission_entry_exact_versions(&set, &entry, &bucket, "file_info_versions").await?;
fivs.versions
@@ -2894,7 +2917,7 @@ impl ECStore {
.delete_object(
bucket.as_str(),
&version.name,
decommission_delete_marker_opts(version, version_id.clone(), idx),
decommission_delete_marker_opts(version, version_id.clone(), idx, expected_bucket_incarnation_id),
)
.await
{
@@ -2984,7 +3007,7 @@ impl ECStore {
bucket.as_str(),
&version.name,
version,
&decommission_remote_tiered_opts(version, version_id.clone(), idx),
&decommission_remote_tiered_opts(version, version_id.clone(), idx, expected_bucket_incarnation_id),
)
.await
{
@@ -3056,7 +3079,11 @@ impl ECStore {
)
.await?;
if let Err(err) = self.clone().decommission_object(idx, bucket, rd).await {
if let Err(err) = self
.clone()
.decommission_object(idx, bucket, rd, expected_bucket_incarnation_id)
.await
{
if is_decommission_copy_cleanup_safe_error(&err) {
ignore = true;
cleanup_ignored = true;
@@ -3133,6 +3160,9 @@ impl ECStore {
}
if should_cleanup_decommission_source_entry(decommissioned, fivs.versions.len(), expired) {
if bucket_incarnation_fence.as_ref().is_some_and(|guard| guard.is_lock_lost()) {
return Err(Error::other("decommission bucket incarnation fence was lost before source cleanup"));
}
decommission_cancel_signal_result(rx.is_cancelled())?;
self.save_decommission_entry_progress_stage(
@@ -3157,6 +3187,12 @@ impl ECStore {
entry.name.as_str(),
&fivs,
&cleanup_preflight_allowed_missing,
data_movement::SourceCleanupBucketFence {
expected_incarnation_id: expected_bucket_incarnation_id,
lifecycle_guard: bucket_incarnation_fence
.as_ref()
.and_then(|guard| guard.namespace_lock_guard()),
},
"decommission",
)
.await
@@ -3268,6 +3304,11 @@ impl ECStore {
let mut lifecycle_config = None;
let mut object_lock_config = None;
let mut replication_config = None;
let expected_bucket_incarnation_id = if bi.name == RUSTFS_META_BUCKET {
None
} else {
Some(self.bucket_incarnation_id_from_disk(&bi.name).await?)
};
if bi.name != RUSTFS_META_BUCKET {
let _ = resolve_decommission_optional_bucket_config_result(
@@ -3321,6 +3362,7 @@ impl ECStore {
let lifecycle_config = lifecycle_config.clone();
let object_lock_config = object_lock_config.clone();
let replication_config = replication_config.clone();
let expected_bucket_incarnation_id = expected_bucket_incarnation_id;
let entry_error = entry_error.clone();
let callback_rx = callback_rx.clone();
@@ -3383,6 +3425,7 @@ impl ECStore {
lifecycle_config,
object_lock_config,
replication_config,
expected_bucket_incarnation_id,
)
.await
{
@@ -4168,10 +4211,24 @@ impl ECStore {
}
#[tracing::instrument(skip(self, rd))]
async fn decommission_object(self: Arc<Self>, pool_idx: usize, bucket: String, rd: GetObjectReader) -> Result<()> {
async fn decommission_object(
self: Arc<Self>,
pool_idx: usize,
bucket: String,
rd: GetObjectReader,
expected_bucket_incarnation_id: Option<uuid::Uuid>,
) -> Result<()> {
warn!("decommission_object: start {} {}", &bucket, &rd.object_info.name);
let object_name = rd.object_info.name.clone();
let result = data_movement::migrate_object(self, pool_idx, bucket.clone(), rd, "decommission_object").await;
let result = data_movement::migrate_object(
self,
pool_idx,
bucket.clone(),
rd,
expected_bucket_incarnation_id,
"decommission_object",
)
.await;
if result.is_ok() {
warn!("decommission_object: migrated {} {}", &bucket, &object_name);
}
@@ -4347,7 +4404,8 @@ mod tests {
..Default::default()
};
let opts = decommission_delete_marker_opts(&version, Some("version-id".to_string()), 7);
let incarnation = uuid::Uuid::new_v4();
let opts = decommission_delete_marker_opts(&version, Some("version-id".to_string()), 7, Some(incarnation));
let replication = opts.delete_replication.expect("replication state should be preserved");
assert!(opts.versioned);
@@ -4357,11 +4415,25 @@ mod tests {
assert_eq!(opts.src_pool_idx, 7);
assert_eq!(opts.version_id.as_deref(), Some("version-id"));
assert_eq!(opts.mod_time, Some(mod_time));
assert_eq!(opts.expected_bucket_incarnation_id, Some(incarnation));
assert_eq!(replication.replica_status, ReplicationStatusType::Replica);
assert!(replication.delete_marker);
assert_eq!(replication.replicate_decision_str, "existing");
}
#[test]
fn decommission_delete_marker_opts_preserves_suspended_null_version() {
let version = rustfs_filemeta::FileInfo {
deleted: true,
..Default::default()
};
let opts = decommission_delete_marker_opts(&version, None, 7, None);
assert!(!opts.versioned);
assert!(opts.version_suspended);
assert_eq!(opts.version_id.as_deref(), Some(uuid::Uuid::nil().to_string().as_str()));
}
#[test]
fn test_decommission_object_migration_read_opts_are_raw_data_movement() {
let opts = decommission_object_migration_read_opts(Some("vid-1".to_string()));
@@ -4383,7 +4455,8 @@ mod tests {
..Default::default()
};
let opts = decommission_remote_tiered_opts(&version, Some("version-id".to_string()), 9);
let incarnation = uuid::Uuid::new_v4();
let opts = decommission_remote_tiered_opts(&version, Some("version-id".to_string()), 9, Some(incarnation));
assert!(opts.versioned);
assert!(opts.data_movement);
@@ -4391,6 +4464,9 @@ mod tests {
assert_eq!(opts.version_id.as_deref(), Some("version-id"));
assert_eq!(opts.mod_time, Some(mod_time));
assert_eq!(opts.user_defined.get("x-amz-meta-key").map(String::as_str), Some("value"));
assert!(opts.include_part_checksums);
assert!(opts.http_preconditions.is_some());
assert_eq!(opts.expected_bucket_incarnation_id, Some(incarnation));
}
#[test]
File diff suppressed because it is too large Load Diff
@@ -12,6 +12,16 @@
// See the License for the specific language governing permissions and
// limitations under the License.
//! Per-disk usage snapshots persisted under the metadata bucket.
//!
//! **Nothing calls into this module.** It landed complete with tests in #5307
//! (2026-07-27) and its aggregation entry point,
//! [`crate::data_usage::aggregate_local_snapshots`], has never had a caller in
//! the tree's history. The live data-usage path is
//! `load_data_usage_from_backend` / `store_data_usage_in_backend`. The items
//! below therefore carry individual `dead_code` allows rather than a module
//! blanket, so the gap stays greppable until it is either wired up or removed.
use crate::data_usage::BucketUsageInfo;
use crate::disk::RUSTFS_META_BUCKET;
use crate::error::{Error, Result};
@@ -26,10 +36,12 @@ pub const DATA_USAGE_DIR: &str = "datausage";
/// Directory used to store incremental scan state files under the metadata bucket.
pub const DATA_USAGE_STATE_DIR: &str = "datausage/state";
/// Snapshot file format version, allows forward compatibility if the structure evolves.
#[allow(dead_code, reason = "unwired local usage-snapshot feature; see module docs (backlog#1823)")]
pub const LOCAL_USAGE_SNAPSHOT_VERSION: u32 = 1;
/// Additional metadata describing which disk produced the snapshot.
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
#[allow(dead_code, reason = "unwired local usage-snapshot feature; see module docs (backlog#1823)")]
pub struct LocalUsageSnapshotMeta {
/// Disk UUID stored as a string for simpler serialization.
pub disk_id: String,
@@ -43,6 +55,7 @@ pub struct LocalUsageSnapshotMeta {
/// Usage snapshot produced by a single disk.
#[derive(Debug, Clone, Serialize, Deserialize, Default)]
#[allow(dead_code, reason = "unwired local usage-snapshot feature; see module docs (backlog#1823)")]
pub struct LocalUsageSnapshot {
/// Format version recorded in the snapshot.
pub format_version: u32,
@@ -64,6 +77,7 @@ pub struct LocalUsageSnapshot {
pub objects_total_size: u64,
}
#[allow(dead_code, reason = "unwired local usage-snapshot feature; see module docs (backlog#1823)")]
impl LocalUsageSnapshot {
/// Create an empty snapshot with the default format version filled in.
pub fn new(meta: LocalUsageSnapshotMeta) -> Self {
@@ -99,11 +113,13 @@ impl LocalUsageSnapshot {
}
/// Build the snapshot file name `<disk-id>.json`.
#[allow(dead_code, reason = "unwired local usage-snapshot feature; see module docs (backlog#1823)")]
pub fn snapshot_file_name(disk_id: &str) -> String {
format!("{disk_id}.json")
}
/// Build the object path relative to `RUSTFS_META_BUCKET`, e.g. `datausage/<disk-id>.json`.
#[allow(dead_code, reason = "unwired local usage-snapshot feature; see module docs (backlog#1823)")]
pub fn snapshot_object_path(disk_id: &str) -> String {
format!("{}/{}", DATA_USAGE_DIR, snapshot_file_name(disk_id))
}
@@ -119,11 +135,13 @@ pub fn data_usage_state_dir(root: &Path) -> PathBuf {
}
/// Build the absolute path to the snapshot file for the provided disk ID.
#[allow(dead_code, reason = "unwired local usage-snapshot feature; see module docs (backlog#1823)")]
pub fn snapshot_path(root: &Path, disk_id: &str) -> PathBuf {
data_usage_dir(root).join(snapshot_file_name(disk_id))
}
/// Read a snapshot from disk if it exists.
#[allow(dead_code, reason = "unwired local usage-snapshot feature; see module docs (backlog#1823)")]
pub async fn read_snapshot(root: &Path, disk_id: &str) -> Result<Option<LocalUsageSnapshot>> {
let path = snapshot_path(root, disk_id);
match fs::read(&path).await {
@@ -138,6 +156,7 @@ pub async fn read_snapshot(root: &Path, disk_id: &str) -> Result<Option<LocalUsa
}
/// Persist a snapshot to disk, creating directories as needed and overwriting any existing file.
#[allow(dead_code, reason = "unwired local usage-snapshot feature; see module docs (backlog#1823)")]
pub async fn write_snapshot(root: &Path, disk_id: &str, snapshot: &LocalUsageSnapshot) -> Result<()> {
let dir = data_usage_dir(root);
fs::create_dir_all(&dir).await.map_err(Error::other)?;
+14 -104
View File
@@ -13,7 +13,6 @@
// limitations under the License.
// #730: scanner/data-usage state is partially migrated and still owns staged cache helpers.
#![allow(dead_code)]
pub mod local_snapshot;
@@ -34,8 +33,8 @@ use crate::{
pub use local_snapshot::{LocalUsageSnapshot, read_snapshot as read_local_snapshot, snapshot_path};
use rustfs_data_usage::{
BucketTargetUsageInfo, BucketUsageInfo, CompressionTotalInfo, DATA_USAGE_OBJECT_NAME, DATA_USAGE_OBSERVED_OBJECT_NAME,
DataUsageCache, DataUsageEntry, DataUsageInfo, DiskUsageStatus, LEGACY_DATA_USAGE_OBJECT_NAME, SizeHistogram, SizeSummary,
VersionsHistogram, observed_data_usage_is_newer,
DataUsageCache, DataUsageInfo, DiskUsageStatus, LEGACY_DATA_USAGE_OBJECT_NAME, SizeHistogram, VersionsHistogram,
observed_data_usage_is_newer,
};
use rustfs_io_metrics::record_system_path_failure;
use rustfs_utils::path::SLASH_SEPARATOR;
@@ -55,7 +54,6 @@ use tracing::{debug, error, info, instrument};
// Data usage storage constants
pub const DATA_USAGE_ROOT: &str = SLASH_SEPARATOR;
const DATA_COMPRESSION_TOTAL_NAME: &str = ".compression.json";
const DATA_USAGE_BLOOM_NAME: &str = ".bloomcycle.bin";
pub const DATA_USAGE_CACHE_NAME: &str = ".usage-cache.bin";
const DATA_USAGE_CACHE_TTL_SECS: u64 = 30;
const LIVE_BUCKET_USAGE_MAX_ENTRIES: u64 = 1024;
@@ -313,11 +311,6 @@ lazy_static::lazy_static! {
LEGACY_DATA_USAGE_OBJECT_NAME
);
static ref LEGACY_DATA_USAGE_OBJ_BACKUP_PATH: String = format!("{}.bkp", LEGACY_DATA_USAGE_OBJ_NAME_PATH.as_str());
pub static ref DATA_USAGE_BLOOM_NAME_PATH: String = format!("{}{}{}",
crate::disk::BUCKET_META_PREFIX,
SLASH_SEPARATOR,
DATA_USAGE_BLOOM_NAME
);
pub static ref DATA_COMPRESSION_TOTAL_NAME_PATH: String = format!("{}{}{}",
crate::disk::BUCKET_META_PREFIX,
SLASH_SEPARATOR,
@@ -858,6 +851,10 @@ async fn resolve_loaded_snapshot_pair_with_source(
}
}
#[allow(
dead_code,
reason = "primary/backup snapshot fallback asserted by this file's tests (backlog#1823)"
)]
async fn resolve_loaded_snapshot(
primary: Result<Vec<u8>, Error>,
backup: impl Future<Output = Result<Vec<u8>, Error>>,
@@ -1187,6 +1184,10 @@ pub async fn invalidate_admin_data_usage_snapshot_cache() {
}
/// Aggregate usage information from local disk snapshots.
#[allow(
dead_code,
reason = "reached only through aggregate_local_snapshots, which has no caller (backlog#1823)"
)]
fn merge_snapshot(aggregated: &mut DataUsageInfo, mut snapshot: LocalUsageSnapshot, latest_update: &mut Option<SystemTime>) {
if let Some(update) = snapshot.last_update
&& latest_update.is_none_or(|current| update > current)
@@ -1220,6 +1221,10 @@ fn merge_snapshot(aggregated: &mut DataUsageInfo, mut snapshot: LocalUsageSnapsh
}
}
#[allow(
dead_code,
reason = "entry point of the local usage-snapshot feature, which has had no caller since it landed in #5307 (backlog#1823)"
)]
pub async fn aggregate_local_snapshots(store: Arc<ECStore>) -> Result<(Vec<DiskUsageStatus>, DataUsageInfo), Error> {
let mut aggregated = DataUsageInfo::default();
let mut latest_update: Option<SystemTime> = None;
@@ -1742,11 +1747,6 @@ pub async fn record_bucket_object_write_unknown_previous_memory(bucket: &str, ne
entry.pending_scanner_position = None;
}
/// Fast in-memory increment for immediate quota consistency.
pub async fn increment_bucket_usage_memory(bucket: &str, size_increment: u64) {
record_bucket_object_write_memory(bucket, None, size_increment).await;
}
/// Fast in-memory update for successful object deletes.
pub async fn record_bucket_object_delete_memory(bucket: &str, deleted_size: u64, removed_current_object: bool) {
ensure_bucket_usage_cached(bucket).await;
@@ -1789,11 +1789,6 @@ pub async fn record_bucket_delete_marker_memory(bucket: &str) {
entry.pending_scanner_position = None;
}
/// Fast in-memory decrement for immediate quota consistency
pub async fn decrement_bucket_usage_memory(bucket: &str, size_decrement: u64) {
record_bucket_object_delete_memory(bucket, size_decrement, size_decrement > 0).await;
}
/// Get bucket usage from the authoritative cache for this topology.
async fn get_persisted_bucket_usage(bucket: &str) -> Option<u64> {
let store = runtime_sources::object_store_handle()?;
@@ -1988,91 +1983,6 @@ pub async fn apply_bucket_usage_memory_overlay(data_usage_info: &mut DataUsageIn
apply_bucket_usage_memory_overlay_if_authoritative(data_usage_info, authoritative).await;
}
/// Sync memory cache with backend data (called by scanner)
pub async fn sync_memory_cache_with_backend() -> Result<(), Error> {
if let Some(store) = runtime_sources::object_store_handle() {
match load_data_usage_from_backend(store.clone()).await {
Ok(data_usage_info) => {
replace_bucket_usage_memory_from_info(&data_usage_info).await;
}
Err(e) => {
debug!("Failed to sync memory cache with backend: {}", e);
}
}
}
Ok(())
}
/// Create a data usage cache entry from size summary
pub fn create_cache_entry_from_summary(summary: &SizeSummary) -> DataUsageEntry {
let mut entry = DataUsageEntry::default();
entry.add_sizes(summary);
entry
}
/// Convert data usage cache to DataUsageInfo
pub fn cache_to_data_usage_info(
cache: &DataUsageCache,
path: &str,
buckets: &[crate::storage_api_contracts::bucket::BucketInfo],
) -> DataUsageInfo {
let e = match cache.find(path) {
Some(e) => e,
None => return DataUsageInfo::default(),
};
let flat = cache.flatten(&e);
let mut buckets_usage = HashMap::new();
for bucket in buckets.iter() {
let e = match cache.find(&bucket.name) {
Some(e) => e,
None => continue,
};
let flat = cache.flatten(&e);
let mut bui = BucketUsageInfo {
size: flat.size as u64,
versions_count: flat.versions as u64,
objects_count: flat.objects as u64,
delete_markers_count: flat.delete_markers as u64,
object_size_histogram: flat.obj_sizes.to_map(),
object_versions_histogram: flat.obj_versions.to_map(),
..Default::default()
};
if let Some(rs) = &flat.replication_stats {
bui.replica_size = rs.replica_size;
bui.replica_count = rs.replica_count;
for (arn, stat) in rs.targets.iter() {
bui.replication_info.insert(
arn.clone(),
BucketTargetUsageInfo {
replication_pending_size: stat.pending_size,
replicated_size: stat.replicated_size,
replication_failed_size: stat.failed_size,
replication_pending_count: stat.pending_count,
replication_failed_count: stat.failed_count,
replicated_count: stat.replicated_count,
..Default::default()
},
);
}
}
buckets_usage.insert(bucket.name.clone(), bui);
}
DataUsageInfo {
last_update: cache.info.last_update,
objects_total_count: flat.objects as u64,
versions_total_count: flat.versions as u64,
delete_markers_total_count: flat.delete_markers as u64,
objects_total_size: flat.size as u64,
buckets_count: e.children.len() as u64,
buckets_usage,
..Default::default()
}
}
// Helper functions for DataUsageCache operations
pub async fn load_data_usage_cache(store: &crate::set_disk::SetDisks, name: &str) -> crate::error::Result<DataUsageCache> {
use crate::disk::{BUCKET_META_PREFIX, RUSTFS_META_BUCKET};
@@ -653,6 +653,7 @@ fn reconcile_servers_with_endpoint_topology(
(added, report)
}
#[allow(dead_code, reason = "exercised by this file's topology tests (backlog#1823)")]
fn server_topology_completeness_report(
servers: &[ServerProperties],
endpoints: &EndpointServerPools,
+62
View File
@@ -46,21 +46,49 @@ pub(crate) const GET_CODEC_STREAMING_OBJECT_CLASS_MULTIPART: &str = "multipart";
pub(crate) const GET_STAGE_DECODE: &str = "decode";
pub(crate) const GET_STAGE_EMIT: &str = "emit";
pub(crate) const GET_STAGE_FILL: &str = "fill";
#[allow(
dead_code,
reason = "GET stage vocabulary; value pinned by this file's tests, no writer yet (backlog#1823)"
)]
pub(crate) const GET_STAGE_FIRST_BYTE: &str = "first_byte";
#[allow(
dead_code,
reason = "GET stage vocabulary; value pinned by this file's tests, no writer yet (backlog#1823)"
)]
pub(crate) const GET_STAGE_FIRST_METADATA_RESPONSE: &str = "first_metadata_response";
#[allow(
dead_code,
reason = "GET stage vocabulary; value pinned by this file's tests, no writer yet (backlog#1823)"
)]
pub(crate) const GET_STAGE_FIRST_VALID_METADATA_RESPONSE: &str = "first_valid_metadata_response";
#[allow(
dead_code,
reason = "GET stage vocabulary; value pinned by this file's tests, no writer yet (backlog#1823)"
)]
pub(crate) const GET_STAGE_FIRST_SHARD_READ: &str = "first_shard_read";
#[allow(
dead_code,
reason = "GET stage vocabulary; value pinned by this file's tests, no writer yet (backlog#1823)"
)]
pub(crate) const GET_STAGE_FULL_BODY: &str = "full_body";
pub(crate) const GET_STAGE_INLINE_PREPARE: &str = "inline_prepare";
pub(crate) const GET_STAGE_LOCK_ACQUIRE: &str = "lock_acquire";
pub(crate) const GET_STAGE_METADATA: &str = "metadata";
pub(crate) const GET_STAGE_METADATA_CACHE_LOOKUP: &str = "metadata_cache_lookup";
#[allow(
dead_code,
reason = "GET stage vocabulary; value pinned by this file's tests, no writer yet (backlog#1823)"
)]
pub(crate) const GET_STAGE_METADATA_FANOUT: &str = "metadata_fanout";
pub(crate) const GET_STAGE_METADATA_RESOLVE: &str = "metadata_resolve";
pub(crate) const GET_STAGE_OBJECT_INFO: &str = "object_info";
pub(crate) const GET_STAGE_OUTPUT_LOCK_WAIT: &str = "output_lock_wait";
pub(crate) const GET_STAGE_OUTPUT_POLL: &str = "output_poll";
pub(crate) const GET_STAGE_PATH_DECISION: &str = "path_decision";
#[allow(
dead_code,
reason = "GET stage vocabulary; value pinned by this file's tests, no writer yet (backlog#1823)"
)]
pub(crate) const GET_STAGE_QUORUM_REACHED: &str = "quorum_reached";
pub(crate) const GET_STAGE_RANGE: &str = "range";
pub(crate) const GET_STAGE_READER_SETUP: &str = "reader_setup";
@@ -84,12 +112,28 @@ pub(crate) const GET_STAGE_READER_STREAM_FIRST_READ: &str = "reader_stream_first
pub(crate) const GET_STAGE_READER_TASK_BITROT_READER_INIT: &str = "reader_task_bitrot_reader_init";
pub(crate) const GET_STAGE_READER_TASK_FILE_OPEN: &str = "reader_task_file_open";
pub(crate) const GET_STAGE_READER_TASK_READER_CONSTRUCTION: &str = "reader_task_reader_construction";
pub(crate) const GET_STAGE_READ_VERSION_DECODE: &str = "read_version_decode";
pub(crate) const GET_STAGE_READ_VERSION_PATH_CHECK: &str = "read_version_path_check";
pub(crate) const GET_STAGE_READ_VERSION_PATH_RESOLVE: &str = "read_version_path_resolve";
pub(crate) const GET_STAGE_READ_VERSION_XLMETA_READ: &str = "read_version_xlmeta_read";
pub(crate) const GET_STAGE_RECONSTRUCT: &str = "reconstruct";
#[allow(
dead_code,
reason = "GET stage vocabulary; value pinned by this file's tests, no writer yet (backlog#1823)"
)]
pub(crate) const GET_STAGE_RESPONSE_HANDOFF: &str = "response_handoff";
#[allow(
dead_code,
reason = "GET stage vocabulary; value pinned by this file's tests, no writer yet (backlog#1823)"
)]
pub(crate) const GET_STAGE_SLOWEST_METADATA_RESPONSE: &str = "slowest_metadata_response";
pub(crate) const GET_STAGE_STRIPE_READ: &str = "stripe_read";
pub(crate) const GET_STAGE_STRIPE_READ_FIRST_SHARD: &str = "stripe_read_first_shard";
pub(crate) const GET_STAGE_STRIPE_READ_QUORUM: &str = "stripe_read_quorum";
#[allow(
dead_code,
reason = "GET stage vocabulary; value pinned by this file's tests, no writer yet (backlog#1823)"
)]
pub(crate) const GET_STAGE_BITROT_VERIFY: &str = "bitrot_verify";
pub(crate) const GET_READER_BUFFER_OUTPUT: &str = "output";
@@ -137,6 +181,7 @@ pub(crate) const GET_METADATA_CACHE_REASON_NO_LOCK: &str = "no_lock";
pub(crate) const GET_METADATA_CACHE_REASON_NOT_FOUND_OR_EXPIRED: &str = "not_found_or_expired";
pub(crate) const GET_METADATA_CACHE_REASON_NOT_READ_DATA: &str = "not_read_data";
pub(crate) const GET_METADATA_CACHE_REASON_PART_NUMBER: &str = "part_number";
pub(crate) const GET_METADATA_CACHE_REASON_PART_CHECKSUMS: &str = "part_checksums";
pub(crate) const GET_METADATA_CACHE_REASON_RAW_DATA_MOVEMENT_READ: &str = "raw_data_movement_read";
pub(crate) const GET_METADATA_CACHE_REASON_STALE_PUBLICATION: &str = "stale_publication";
pub(crate) const GET_METADATA_CACHE_REASON_USABLE: &str = "usable";
@@ -154,8 +199,20 @@ pub(crate) const GET_METADATA_EARLY_STOP_REASON_VERSION_NOT_FOUND: &str = "versi
pub(crate) const GET_METADATA_EARLY_STOP_REASON_VERSION_MATCH_QUORUM: &str = "version_match_quorum";
/// Early-stop active state labels
#[allow(
dead_code,
reason = "GET stage vocabulary; value pinned by this file's tests, no writer yet (backlog#1823)"
)]
pub(crate) const EARLY_STOP_ACTIVE_HIT: &str = "hit";
#[allow(
dead_code,
reason = "GET stage vocabulary; value pinned by this file's tests, no writer yet (backlog#1823)"
)]
pub(crate) const EARLY_STOP_ACTIVE_MISS: &str = "miss";
#[allow(
dead_code,
reason = "GET stage vocabulary; value pinned by this file's tests, no writer yet (backlog#1823)"
)]
pub(crate) const EARLY_STOP_ACTIVE_DISABLED: &str = "disabled";
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
@@ -441,6 +498,10 @@ mod tests {
assert_eq!(GET_STAGE_QUORUM_REACHED, "quorum_reached");
assert_eq!(GET_STAGE_RANGE, "range");
assert_eq!(GET_STAGE_READER_SETUP, "reader_setup");
assert_eq!(GET_STAGE_READ_VERSION_DECODE, "read_version_decode");
assert_eq!(GET_STAGE_READ_VERSION_PATH_CHECK, "read_version_path_check");
assert_eq!(GET_STAGE_READ_VERSION_PATH_RESOLVE, "read_version_path_resolve");
assert_eq!(GET_STAGE_READ_VERSION_XLMETA_READ, "read_version_xlmeta_read");
assert_eq!(GET_STAGE_RECONSTRUCT, "reconstruct");
assert_eq!(GET_STAGE_RESPONSE_HANDOFF, "response_handoff");
assert_eq!(GET_STAGE_SLOWEST_METADATA_RESPONSE, "slowest_metadata_response");
@@ -480,6 +541,7 @@ mod tests {
assert_eq!(GET_METADATA_CACHE_REASON_NO_LOCK, "no_lock");
assert_eq!(GET_METADATA_CACHE_REASON_NOT_FOUND_OR_EXPIRED, "not_found_or_expired");
assert_eq!(GET_METADATA_CACHE_REASON_NOT_READ_DATA, "not_read_data");
assert_eq!(GET_METADATA_CACHE_REASON_PART_CHECKSUMS, "part_checksums");
assert_eq!(GET_METADATA_CACHE_REASON_PART_NUMBER, "part_number");
assert_eq!(GET_METADATA_CACHE_REASON_RAW_DATA_MOVEMENT_READ, "raw_data_movement_read");
assert_eq!(GET_METADATA_CACHE_REASON_STALE_PUBLICATION, "stale_publication");
-2
View File
@@ -13,8 +13,6 @@
// limitations under the License.
// #730: diagnostics constants are staged for request-path telemetry migration.
#![allow(dead_code)]
pub(crate) mod admin_server_info;
pub(crate) mod get;
pub(crate) mod pool;
-30
View File
@@ -1,30 +0,0 @@
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
//! BytesPool metric label constants.
//!
//! These constants are used when recording pool acquisition and return
//! metrics to avoid string allocations and ensure label consistency.
/// BytesPool tier labels
pub const POOL_TIER_SMALL: &str = "small";
pub const POOL_TIER_MEDIUM: &str = "medium";
pub const POOL_TIER_LARGE: &str = "large";
pub const POOL_TIER_XLARGE: &str = "xlarge";
/// BytesPool outcome labels
pub const POOL_OUTCOME_HIT: &str = "hit";
pub const POOL_OUTCOME_MISS: &str = "miss";
pub const POOL_OUTCOME_RECYCLED: &str = "recycled";
pub const POOL_OUTCOME_DROPPED: &str = "dropped";
+15
View File
@@ -2022,6 +2022,21 @@ impl DiskAPI for LocalDiskWrapper {
.await
}
async fn read_file_stream_chunks(
&self,
volume: &str,
path: &str,
offset: usize,
length: usize,
) -> Result<Option<rustfs_rio::ChunkReaderBox>> {
self.track_disk_health_with_op(
"read_file_stream_chunks",
|| async { self.disk.read_file_stream_chunks(volume, path, offset, length).await },
get_max_timeout_duration(),
)
.await
}
async fn read_file_mmap_copy(&self, volume: &str, path: &str, offset: usize, length: usize) -> Result<bytes::Bytes> {
self.track_disk_health_with_op(
"read_file_mmap_copy",
+6 -26
View File
@@ -113,6 +113,9 @@ pub enum DiskError {
#[error("bit-rot hash algorithm is invalid")]
BitrotHashAlgoInvalid,
/// Never constructed locally by RustFS (only reachable through wire
/// decoding, and no current node sends it). The wire code is kept for
/// cross-version compatibility — do not renumber or remove (backlog#1831).
#[error("Rename across devices not allowed, please fix your backend configuration")]
CrossDeviceLink,
@@ -143,6 +146,9 @@ pub enum DiskError {
#[error("io error {0}")]
Io(#[source] io::Error),
/// Never constructed locally by RustFS (only reachable through wire
/// decoding, and no current node sends it). The wire code is kept for
/// cross-version compatibility — do not renumber or remove (backlog#1831).
#[error("source stalled")]
SourceStalled,
@@ -642,19 +648,6 @@ impl Hash for DiskError {
// is currently commented out to avoid complexity. These can be re-enabled
// when needed for specific disk quorum checking and error aggregation logic.
/// Bitrot errors
#[derive(Debug, thiserror::Error)]
pub enum BitrotErrorType {
#[error("bitrot checksum verification failed")]
BitrotChecksumMismatch { expected: String, got: String },
}
impl From<BitrotErrorType> for DiskError {
fn from(e: BitrotErrorType) -> Self {
DiskError::other(e)
}
}
/// Context wrapper for file access errors
#[derive(Debug, thiserror::Error)]
pub struct FileAccessDeniedWithContext {
@@ -869,19 +862,6 @@ mod tests {
let _disk_error: DiskError = json_error.into();
}
#[test]
fn test_bitrot_error_type() {
let bitrot_error = BitrotErrorType::BitrotChecksumMismatch {
expected: "abc123".to_string(),
got: "def456".to_string(),
};
assert!(bitrot_error.to_string().contains("bitrot checksum verification failed"));
let disk_error: DiskError = bitrot_error.into();
assert!(matches!(disk_error, DiskError::Io(_)));
}
#[test]
fn test_file_access_denied_with_context() {
let path = PathBuf::from("/test/path");
+191 -25
View File
@@ -15,6 +15,11 @@
use crate::config::storageclass::DEFAULT_INLINE_BLOCK;
use crate::crash_inject::{self, CrashPoint};
use crate::data_usage::local_snapshot::ensure_data_usage_layout;
use crate::diagnostics::get::{
GET_OBJECT_PATH_INTERNAL_META, GET_OBJECT_PATH_LEGACY_DUPLEX, GET_STAGE_READ_VERSION_DECODE,
GET_STAGE_READ_VERSION_PATH_CHECK, GET_STAGE_READ_VERSION_PATH_RESOLVE, GET_STAGE_READ_VERSION_XLMETA_READ,
get_stage_timer_if_enabled, record_get_stage_duration_if_enabled,
};
#[cfg(test)]
use crate::disk::HEALING_MARKER_PATH;
use crate::disk::disk_store::{get_drive_walkdir_stall_timeout, get_object_disk_read_timeout};
@@ -9174,7 +9179,7 @@ impl DiskAPI for LocalDisk {
if let Some(src_file_path_parent) = src_file_path.parent() {
if src_volume != super::RUSTFS_META_MULTIPART_BUCKET {
let _ = remove_std(src_file_path_parent);
let _ = std::fs::remove_dir(src_file_path_parent);
} else {
let _ = self
.delete_file(&dst_volume_dir, &src_file_path_parent.to_path_buf(), true, false)
@@ -9499,7 +9504,7 @@ impl DiskAPI for LocalDisk {
if let Some(ref cleanup) = cleanup_path {
let _ = self.delete_file(&dst_volume_dir, cleanup, true, false).await;
} else if let Some(parent) = src_file_path.parent() {
let _ = remove_std(parent);
let _ = std::fs::remove_dir(parent);
}
// Heal reuses a version's `data_dir` and lands the rebuilt shard on
@@ -9840,6 +9845,12 @@ impl DiskAPI for LocalDisk {
opts: &ReadOptions,
) -> Result<FileInfo> {
crate::hp_guard!("LocalDisk::read_version");
let stage_metrics_enabled = rustfs_io_metrics::get_stage_metrics_enabled();
let metrics_path = if stage_metrics_enabled && crate::bucket::utils::is_meta_bucketname(volume) {
GET_OBJECT_PATH_INTERNAL_META
} else {
GET_OBJECT_PATH_LEGACY_DUPLEX
};
if !org_volume.is_empty() {
let org_volume_path = self.io_get_bucket_path(org_volume)?;
if !skip_access_checks(org_volume) {
@@ -9849,36 +9860,46 @@ impl DiskAPI for LocalDisk {
}
}
let path_resolve_start = get_stage_timer_if_enabled(stage_metrics_enabled);
let file_path = self.io_get_object_path(volume, path)?;
let volume_dir = self.io_get_bucket_path(volume)?;
record_get_stage_duration_if_enabled(metrics_path, GET_STAGE_READ_VERSION_PATH_RESOLVE, path_resolve_start);
let path_check_start = get_stage_timer_if_enabled(stage_metrics_enabled);
check_path_length(file_path.to_string_lossy().as_ref())?;
record_get_stage_duration_if_enabled(metrics_path, GET_STAGE_READ_VERSION_PATH_CHECK, path_check_start);
let read_data = opts.read_data;
let (data, _) = self
.read_raw(volume, volume_dir.clone(), file_path, read_data)
.await
.map_err(|e| {
if e == DiskError::FileNotFound && !version_id.is_empty() {
DiskError::FileVersionNotFound
} else {
e
}
})?;
let xlmeta_read_start = get_stage_timer_if_enabled(stage_metrics_enabled);
let raw_read_result = self.read_raw(volume, volume_dir.clone(), file_path, read_data).await;
record_get_stage_duration_if_enabled(metrics_path, GET_STAGE_READ_VERSION_XLMETA_READ, xlmeta_read_start);
let (data, _) = raw_read_result.map_err(|e| {
if e == DiskError::FileNotFound && !version_id.is_empty() {
DiskError::FileVersionNotFound
} else {
e
}
})?;
let mut fi = get_file_info(
&data,
volume,
path,
version_id,
FileInfoOpts {
data: read_data,
include_free_versions: opts.incl_free_versions,
},
)?;
fi.validate_for_metadata_read()?;
let decode_start = get_stage_timer_if_enabled(stage_metrics_enabled);
let file_info_result: Result<FileInfo> = (|| {
let fi = get_file_info(
&data,
volume,
path,
version_id,
FileInfoOpts {
data: read_data,
include_free_versions: opts.incl_free_versions,
include_part_checksums: false,
},
)?;
fi.validate_for_metadata_read()?;
Ok(fi)
})();
record_get_stage_duration_if_enabled(metrics_path, GET_STAGE_READ_VERSION_DECODE, decode_start);
let mut fi = file_info_result?;
if fi.is_canonical_delete_marker() {
return Ok(fi);
}
@@ -10561,6 +10582,108 @@ mod test {
meta.marshal_msg().expect("test metadata should encode")
}
#[test]
#[serial_test::serial]
fn read_version_records_local_metadata_stage_breakdown() {
let runtime = tokio::runtime::Builder::new_current_thread()
.enable_all()
.build()
.expect("test runtime should be created");
let recorder = crate::test_metrics::CapturingRecorder::default();
let previous_gate = rustfs_io_metrics::get_stage_metrics_enabled();
rustfs_io_metrics::set_get_stage_metrics_enabled(true);
metrics::with_local_recorder(&recorder, || {
runtime.block_on(async {
let dir = tempfile::tempdir().expect("temp dir should be created");
let endpoint =
Endpoint::try_from(dir.path().to_str().expect("temp dir should be utf8")).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
let bucket = "bucket";
let object = "stage-breakdown";
ensure_test_volume(&disk, bucket).await;
let object_dir = dir.path().join(bucket).join(object);
fs::create_dir_all(&object_dir)
.await
.expect("object directory should be created");
fs::write(
object_dir.join(STORAGE_FORMAT_FILE),
test_meta(test_file_info(object, Uuid::new_v4(), None, Some(Bytes::from_static(b"inline")))),
)
.await
.expect("object metadata should be written");
disk.read_version(
"",
bucket,
object,
"",
&ReadOptions {
read_data: true,
..Default::default()
},
)
.await
.expect("read_version should succeed");
let meta_object = "stage-breakdown-meta";
let meta_object_dir = dir.path().join(RUSTFS_META_BUCKET).join(meta_object);
fs::create_dir_all(&meta_object_dir)
.await
.expect("internal metadata object directory should be created");
fs::write(
meta_object_dir.join(STORAGE_FORMAT_FILE),
test_meta(test_file_info(meta_object, Uuid::new_v4(), None, Some(Bytes::from_static(b"meta")))),
)
.await
.expect("internal metadata should be written");
disk.read_version(
"",
RUSTFS_META_BUCKET,
meta_object,
"",
&ReadOptions {
read_data: true,
..Default::default()
},
)
.await
.expect("internal metadata read_version should succeed");
});
});
rustfs_io_metrics::set_get_stage_metrics_enabled(previous_gate);
for stage in [
GET_STAGE_READ_VERSION_PATH_RESOLVE,
GET_STAGE_READ_VERSION_PATH_CHECK,
GET_STAGE_READ_VERSION_XLMETA_READ,
GET_STAGE_READ_VERSION_DECODE,
] {
assert_eq!(
recorder
.histogram_values(
"rustfs_io_get_object_stage_duration_seconds",
&[("path", GET_OBJECT_PATH_LEGACY_DUPLEX), ("stage", stage)]
)
.len(),
1,
"{stage} should be recorded once for user-bucket LocalDisk::read_version"
);
assert_eq!(
recorder
.histogram_values(
"rustfs_io_get_object_stage_duration_seconds",
&[("path", GET_OBJECT_PATH_INTERNAL_META), ("stage", stage)]
)
.len(),
1,
"{stage} should be recorded once for internal-meta LocalDisk::read_version"
);
}
}
#[test]
fn inline_metadata_rollback_dir_avoids_real_data_dir_collision() {
let target_version = Uuid::parse_str("11111111-2222-3333-4444-555555555555").expect("version id should parse");
@@ -12439,6 +12562,10 @@ mod test {
.join(RUSTFS_META_TMP_BUCKET)
.join(tmp_object)
.join(new_data_dir.to_string());
let tmp_parent = tmp_data_dir
.parent()
.expect("tmp data dir should have a parent")
.to_path_buf();
fs::create_dir_all(&tmp_data_dir)
.await
.expect("new tmp data dir should be created");
@@ -12450,6 +12577,10 @@ mod test {
disk.rename_data(RUSTFS_META_TMP_BUCKET, tmp_object, new_fi, bucket, object)
.await
.expect("rename_data should commit");
assert!(
!tmp_parent.exists(),
"successful non-inline commit should remove the empty staging parent"
);
// The tmp xl.meta write point uses SyncMode::FileOnly: its parent dir
// ({tmp}/{tmp_object}) must not be fsynced.
@@ -12654,6 +12785,9 @@ mod test {
let tmp_object = "tmp-new-inline";
ensure_test_volume(&disk, bucket).await;
ensure_test_volume(&disk, RUSTFS_META_TMP_BUCKET).await;
let tmp_parent = disk
.get_object_path(RUSTFS_META_TMP_BUCKET, tmp_object)
.expect("tmp parent should resolve");
let _mode = durability_mode_override::set(DurabilityMode::Strict);
let version_id = Uuid::parse_str("99999999-9999-9999-9999-999999999999").expect("version id should parse");
@@ -12662,6 +12796,7 @@ mod test {
disk.rename_data(RUSTFS_META_TMP_BUCKET, tmp_object, new_fi, bucket, object)
.await
.expect("inline rename_data should commit the new object");
assert!(!tmp_parent.exists(), "successful inline commit should remove the empty staging parent");
let bucket_dir = disk.get_bucket_path(bucket).expect("bucket path should resolve");
let prefix_dir = disk.get_object_path(bucket, "prefix").expect("prefix path should resolve");
@@ -12685,6 +12820,34 @@ mod test {
);
}
#[tokio::test]
async fn rename_data_inline_preserves_non_empty_staging_parent() {
use tempfile::tempdir;
let dir = tempdir().expect("temp dir should be created");
let endpoint = Endpoint::try_from(dir.path().to_str().expect("temp dir should be utf8")).expect("endpoint should parse");
let disk = LocalDisk::new(&endpoint, false).await.expect("local disk should be created");
let bucket = "inline-staging-sentinel-bucket";
let object = "inline-object";
let tmp_object = "inline-stage-with-sentinel";
ensure_test_volume(&disk, bucket).await;
ensure_test_volume(&disk, RUSTFS_META_TMP_BUCKET).await;
let tmp_parent = disk
.get_object_path(RUSTFS_META_TMP_BUCKET, tmp_object)
.expect("tmp parent should resolve");
fs::create_dir_all(&tmp_parent).await.expect("tmp parent should be created");
let sentinel = tmp_parent.join("sentinel");
fs::write(&sentinel, b"keep").await.expect("sentinel should be written");
let fi = test_file_info(object, Uuid::new_v4(), None, Some(Bytes::from_static(b"inline-payload")));
disk.rename_data(RUSTFS_META_TMP_BUCKET, tmp_object, fi, bucket, object)
.await
.expect("non-empty staging cleanup must not negate the committed object");
assert_eq!(fs::read(&sentinel).await.expect("sentinel should remain"), b"keep");
}
#[cfg(unix)]
#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
#[allow(clippy::await_holding_lock)]
@@ -12859,7 +13022,10 @@ mod test {
.expect("non-inline rename_data should commit");
assert!(!replacement_dir.exists(), "the destination object directory must not be replaced");
assert!(staging_parent.exists(), "the guarded staging parent must retain its identity");
assert!(
!staging_parent.exists(),
"successful commit should remove the empty staging parent after releasing its guard"
);
assert!(
!replacement_staging_parent.exists(),
"the staging parent must not be replaced between data and metadata publication"
+26
View File
@@ -65,6 +65,7 @@ use error::{Error, Result};
use local::LocalDisk;
use rustfs_filemeta::{FileInfo, ObjectPartInfo, RawFileInfo};
use rustfs_madmin::info_commands::DiskMetrics;
use rustfs_rio::ChunkReaderBox;
use serde::{Deserialize, Serialize};
use std::{fmt::Debug, path::PathBuf, sync::Arc, time::Duration};
use time::OffsetDateTime;
@@ -427,6 +428,19 @@ impl DiskAPI for Disk {
}
}
async fn read_file_stream_chunks(
&self,
volume: &str,
path: &str,
offset: usize,
length: usize,
) -> Result<Option<ChunkReaderBox>> {
match self {
Disk::Local(_) => Ok(None),
Disk::Remote(remote_disk) => remote_disk.read_file_stream_chunks(volume, path, offset, length).await,
}
}
#[tracing::instrument(level = "trace", skip_all)]
async fn read_file_mmap_copy(&self, volume: &str, path: &str, offset: usize, length: usize) -> Result<Bytes> {
match self {
@@ -865,6 +879,18 @@ pub trait DiskAPI: Debug + Send + Sync + 'static {
async fn read_file(&self, volume: &str, path: &str) -> Result<FileReader>;
async fn read_file_stream(&self, volume: &str, path: &str, offset: usize, length: usize) -> Result<FileReader>;
/// Returns an owned-chunk stream when the backing transport can preserve
/// receive-buffer ownership. `None` retains the ordinary reader path.
async fn read_file_stream_chunks(
&self,
_volume: &str,
_path: &str,
_offset: usize,
_length: usize,
) -> Result<Option<ChunkReaderBox>> {
Ok(None)
}
/// File read using mmap-then-copy on Unix or an efficient read on non-Unix.
async fn read_file_mmap_copy(&self, volume: &str, path: &str, offset: usize, length: usize) -> Result<Bytes>;
@@ -26,6 +26,7 @@ pub(crate) const GET_RECONSTRUCT_OUTCOME_SKIP_DATA_COMPLETE: &str = "skip_data_c
pub(crate) const GET_RECONSTRUCT_OUTCOME_SKIP_EMPTY_PAYLOAD: &str = "skip_empty_payload";
pub(crate) trait DecodeWorkspace: Send + Sync + 'static {
#[allow(dead_code, reason = "workspace width asserted by decode_reader tests (backlog#1823)")]
fn shard_len(&self) -> usize;
}
@@ -33,11 +34,14 @@ pub(crate) trait ErasureDecodeEngine: Send + Sync + 'static {
type Workspace: DecodeWorkspace;
fn data_shards(&self) -> usize;
#[allow(dead_code, reason = "engine trait facet asserted by decode_reader tests (backlog#1823)")]
fn parity_shards(&self) -> usize;
fn block_size(&self) -> usize;
fn engine_name(&self) -> &'static str;
#[allow(dead_code, reason = "engine trait facet asserted by decode_reader tests (backlog#1823)")]
fn supports_progressive_decode(&self) -> bool;
#[allow(dead_code, reason = "engine trait facet asserted by decode_reader tests (backlog#1823)")]
fn supports_aligned_shards(&self) -> bool;
fn prepare_workspace(&self, shard_len: usize) -> io::Result<Self::Workspace>;
@@ -24,6 +24,7 @@ impl RustfsCodecDecodeWorkspace {
}
#[inline]
#[allow(dead_code, reason = "workspace width asserted by decode_reader tests (backlog#1823)")]
pub(crate) fn shard_len(&self) -> usize {
self.shard_len
}
@@ -76,6 +77,13 @@ impl ShardBufferPool {
self.buffers[index] = Some(buf);
}
#[cfg(test)]
pub(crate) fn stored_allocation(&self, index: usize) -> Option<(*const u8, usize)> {
self.buffers
.get(index)
.and_then(|buf| buf.as_ref().map(|buf| (buf.as_ptr(), buf.capacity())))
}
#[cfg(test)]
fn stored_capacity(&self, index: usize) -> Option<usize> {
self.buffers.get(index).and_then(|buf| buf.as_ref().map(Vec::capacity))
+526 -9
View File
@@ -14,7 +14,10 @@
use pin_project_lite::pin_project;
use rustfs_utils::HashAlgorithm;
use std::future::poll_fn;
use std::io::IoSlice;
use std::pin::Pin;
use std::task::{Context, Poll};
use std::time::Duration;
use tokio::io::{AsyncRead, AsyncReadExt, AsyncWrite, AsyncWriteExt};
use tracing::error;
@@ -23,6 +26,18 @@ const LOG_COMPONENT_ECSTORE: &str = "ecstore";
const LOG_SUBSYSTEM_ERASURE: &str = "erasure";
const EVENT_BITROT_SHORT_SHARD_READ: &str = "bitrot_short_shard_read";
const EVENT_BITROT_HASH_MISMATCH: &str = "bitrot_hash_mismatch";
const MAX_RETAINED_CHUNKS_PER_BLOCK: usize = 64;
const MAX_CHUNK_POLLS_PER_YIELD: usize = MAX_RETAINED_CHUNKS_PER_BLOCK + 1;
/// Result of polling an optional owned-chunk handoff.
pub enum ShardChunkRead {
/// The source does not support owned-chunk handoff and remains untouched.
Unsupported,
/// The source reached EOF.
Eof,
/// A non-empty chunk containing at most the requested number of bytes.
Chunk(bytes::Bytes),
}
/// A shard source that may already hold its bytes in memory.
///
@@ -42,6 +57,12 @@ pub trait ShardSource: AsyncRead + Send + Sync + Unpin {
fn try_take_block(&mut self, _n: usize) -> Option<bytes::Bytes> {
None
}
/// Polls one owned chunk when the source supports chunk handoff.
/// `Unsupported` must leave the source untouched.
fn poll_read_chunk(self: Pin<&mut Self>, _cx: &mut Context<'_>, _max: usize) -> Poll<std::io::Result<ShardChunkRead>> {
Poll::Ready(Ok(ShardChunkRead::Unsupported))
}
}
/// Borrowed and owned byte slices are ordinary streaming sources: they carry no
@@ -75,6 +96,9 @@ pin_project! {
// contiguous on-disk `[hash][data]` block so both are pulled in a single
// pass; grown lazily and never shrunk.
buf: Vec<u8>,
// Reused owned chunk vector for the remote HTTP fast path. Keeping the
// allocation with the reader avoids allocating once per bitrot block.
chunks: Vec<bytes::Bytes>,
skip_verify: bool,
last_verify_duration: Duration,
}
@@ -91,6 +115,7 @@ where
hash_algo: algo,
shard_size,
buf: Vec::new(),
chunks: Vec::new(),
skip_verify,
last_verify_duration: Duration::ZERO,
}
@@ -100,6 +125,11 @@ where
self.last_verify_duration
}
#[cfg(test)]
pub(crate) fn inner_ref(&self) -> &R {
&self.inner
}
/// Read a single (hash+data) block, verify hash, and copy `out.len()` bytes
/// into `out`. Returns an error if the shard is short, the hash mismatches,
/// or `out` is larger than one shard. On error `out`'s contents are
@@ -260,11 +290,6 @@ where
let need = hash_size + want;
// In-memory fast path: the block is already resident, so slice it instead
// of copying it into the scratch buffer first (rustfs/backlog#1159). One
// copy (`extend_from_slice`) instead of two. A source that cannot serve
// `need` bytes returns `None` and falls through to the scratch path,
// keeping the short-read contract.
if let Some(block) = self.inner.try_take_block(need) {
let (data, verify) = split_and_verify(&self.hash_algo, self.skip_verify, &block)?;
out.extend_from_slice(data);
@@ -272,6 +297,126 @@ where
return Ok(want);
}
self.chunks.clear();
let handed_off = {
let inner = &mut self.inner;
let chunks = &mut self.chunks;
let tail_buf = &mut self.buf;
let mut received = 0usize;
poll_fn(|cx| {
for _ in 0..MAX_CHUNK_POLLS_PER_YIELD {
let next = match Pin::new(&mut *inner).poll_read_chunk(cx, need - received) {
Poll::Ready(Ok(next)) => next,
Poll::Ready(Err(err)) => return Poll::Ready(Err(err)),
Poll::Pending => return Poll::Pending,
};
let chunk = match next {
ShardChunkRead::Unsupported if received == 0 => return Poll::Ready(Ok(false)),
ShardChunkRead::Unsupported => {
return Poll::Ready(Err(std::io::Error::new(
std::io::ErrorKind::InvalidData,
"chunk handoff became unavailable after transferring data",
)));
}
ShardChunkRead::Eof => {
return Poll::Ready(Err(short_shard_read(received.saturating_sub(hash_size), want)));
}
ShardChunkRead::Chunk(chunk) => chunk,
};
if received == 0 {
tail_buf.clear();
}
if chunk.is_empty() {
return Poll::Ready(Err(std::io::Error::new(
std::io::ErrorKind::InvalidData,
"chunk handoff returned an empty chunk",
)));
}
let remaining = need - received;
if chunk.len() > remaining {
return Poll::Ready(Err(std::io::Error::new(
std::io::ErrorKind::InvalidData,
"chunk handoff exceeded its requested boundary",
)));
}
received += chunk.len();
if chunks.len() == MAX_RETAINED_CHUNKS_PER_BLOCK {
if tail_buf.is_empty() {
tail_buf.reserve_exact(need - (received - chunk.len()));
}
tail_buf.extend_from_slice(&chunk);
} else {
chunks.push(chunk);
}
if received == need {
return Poll::Ready(Ok(true));
}
}
cx.waker().wake_by_ref();
Poll::Pending
})
.await?
};
if handed_off {
if self.chunks.len() == 1 && self.buf.is_empty() {
let block = &self.chunks[0];
let (data, verify) = split_and_verify(&self.hash_algo, self.skip_verify, block)?;
out.extend_from_slice(data);
self.last_verify_duration = verify;
return Ok(want);
}
let block_chunks = || {
self.chunks
.iter()
.map(|chunk| chunk.as_ref())
.chain((!self.buf.is_empty()).then_some(self.buf.as_slice()))
};
if !self.skip_verify {
let verify_start = std::time::Instant::now();
let actual_hash = self
.hash_algo
.hash_encode_slices(block_chunks().scan(hash_size, |skip, chunk| {
let start = (*skip).min(chunk.len());
*skip -= start;
Some(&chunk[start..])
}));
let verify = verify_start.elapsed();
let mut hash_offset = 0;
let mut remaining = hash_size;
for chunk in block_chunks() {
let take = remaining.min(chunk.len());
if actual_hash.as_ref()[hash_offset..hash_offset + take] != chunk[..take] {
error!(
event = EVENT_BITROT_HASH_MISMATCH,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_ERASURE,
state = "failed",
data_len = want,
"bitrot hash mismatch"
);
return Err(std::io::Error::new(std::io::ErrorKind::InvalidData, "bitrot hash mismatch"));
}
hash_offset += take;
remaining -= take;
if remaining == 0 {
break;
}
}
self.last_verify_duration = verify;
}
let mut skip = hash_size;
for chunk in block_chunks() {
let start = skip.min(chunk.len());
skip -= start;
out.extend_from_slice(&chunk[start..]);
}
return Ok(want);
}
// Streaming path: same single pass and same verification as `read`; only
// the sink differs (`extend_from_slice` into `out` instead of
// `copy_from_slice` into a pre-zeroed buffer).
@@ -677,18 +822,167 @@ impl BitrotWriterWrapper {
#[cfg(test)]
mod tests {
use super::ShardSource;
use super::{
BitrotReader, BitrotWriter, BitrotWriterWrapper, CustomWriter, bitrot_shard_file_size, bitrot_verify, write_all_vectored,
};
use super::{MAX_RETAINED_CHUNKS_PER_BLOCK, ShardChunkRead, ShardSource};
use bytes::Bytes;
use rustfs_utils::HashAlgorithm;
use std::io::{Cursor, IoSlice};
use std::collections::VecDeque;
use std::io::{self, Cursor, IoSlice};
use std::pin::Pin;
use std::sync::{
Arc,
atomic::{AtomicUsize, Ordering},
};
use std::task::{Context, Poll};
use tokio::io::{AsyncWrite, AsyncWriteExt};
use std::time::Duration;
use tokio::io::{AsyncRead, AsyncWrite, AsyncWriteExt, ReadBuf};
struct FragmentedSource {
chunks: VecDeque<Bytes>,
}
impl FragmentedSource {
fn new(bytes: Vec<u8>, fragment_sizes: &[usize]) -> Self {
let mut chunks = VecDeque::new();
let mut offset = 0;
for &size in fragment_sizes {
let end = (offset + size).min(bytes.len());
if offset < end {
chunks.push_back(Bytes::copy_from_slice(&bytes[offset..end]));
}
offset = end;
}
if offset < bytes.len() {
chunks.push_back(Bytes::copy_from_slice(&bytes[offset..]));
}
Self { chunks }
}
}
impl AsyncRead for FragmentedSource {
fn poll_read(self: Pin<&mut Self>, _cx: &mut Context<'_>, _buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
Poll::Ready(Err(io::Error::other("fragmented source must use chunk handoff")))
}
}
impl ShardSource for FragmentedSource {
fn poll_read_chunk(mut self: Pin<&mut Self>, _cx: &mut Context<'_>, max: usize) -> Poll<io::Result<ShardChunkRead>> {
let Some(mut chunk) = self.chunks.pop_front() else {
return Poll::Ready(Ok(ShardChunkRead::Eof));
};
if chunk.len() > max {
self.chunks.push_front(chunk.split_off(max));
chunk.truncate(max);
}
Poll::Ready(Ok(ShardChunkRead::Chunk(chunk)))
}
}
struct GeneratedChunkSource {
bytes: Bytes,
offset: usize,
fragment_size: usize,
fail_at: Option<usize>,
}
impl GeneratedChunkSource {
fn new(bytes: Vec<u8>, fragment_size: usize) -> Self {
assert!(fragment_size > 0);
Self {
bytes: Bytes::from(bytes),
offset: 0,
fragment_size,
fail_at: None,
}
}
fn failing(bytes: Vec<u8>, fragment_size: usize, fail_at: usize) -> Self {
Self {
fail_at: Some(fail_at),
..Self::new(bytes, fragment_size)
}
}
}
impl AsyncRead for GeneratedChunkSource {
fn poll_read(self: Pin<&mut Self>, _cx: &mut Context<'_>, _buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
Poll::Ready(Err(io::Error::other("generated source must use chunk handoff")))
}
}
impl ShardSource for GeneratedChunkSource {
fn poll_read_chunk(mut self: Pin<&mut Self>, _cx: &mut Context<'_>, max: usize) -> Poll<io::Result<ShardChunkRead>> {
if self.fail_at == Some(self.offset) {
return Poll::Ready(Err(rustfs_rio::new_test_internode_http_io_error(
rustfs_rio::InternodeHttpErrorKind::BodyStreamAborted,
)));
}
if self.offset == self.bytes.len() {
return Poll::Ready(Ok(ShardChunkRead::Eof));
}
let error_limit = self.fail_at.unwrap_or(self.bytes.len());
let take = self
.fragment_size
.min(max)
.min(error_limit - self.offset)
.min(self.bytes.len() - self.offset);
let start = self.offset;
self.offset += take;
Poll::Ready(Ok(ShardChunkRead::Chunk(self.bytes.slice(start..start + take))))
}
}
struct InvalidChunkSource {
mode: InvalidChunkMode,
}
#[derive(Clone, Copy)]
enum InvalidChunkMode {
Empty,
Oversized,
UnsupportedAfterChunk,
Unsupported,
}
impl AsyncRead for InvalidChunkSource {
fn poll_read(self: Pin<&mut Self>, _cx: &mut Context<'_>, _buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
Poll::Ready(Err(io::Error::other("invalid source must use chunk handoff")))
}
}
impl ShardSource for InvalidChunkSource {
fn poll_read_chunk(mut self: Pin<&mut Self>, _cx: &mut Context<'_>, max: usize) -> Poll<io::Result<ShardChunkRead>> {
match self.mode {
InvalidChunkMode::Empty => Poll::Ready(Ok(ShardChunkRead::Chunk(Bytes::new()))),
InvalidChunkMode::Oversized => Poll::Ready(Ok(ShardChunkRead::Chunk(Bytes::from(vec![0; max + 1])))),
InvalidChunkMode::UnsupportedAfterChunk => {
self.mode = InvalidChunkMode::Unsupported;
Poll::Ready(Ok(ShardChunkRead::Chunk(Bytes::from_static(b"x"))))
}
InvalidChunkMode::Unsupported => Poll::Ready(Ok(ShardChunkRead::Unsupported)),
}
}
}
struct ScratchReuseSource {
block: Option<Bytes>,
saw_reused_scratch: bool,
}
impl AsyncRead for ScratchReuseSource {
fn poll_read(mut self: Pin<&mut Self>, _cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
let Some(block) = self.block.take() else {
return Poll::Ready(Ok(()));
};
self.saw_reused_scratch = buf.initialize_unfilled()[..block.len()].iter().all(|byte| *byte == 0xa5);
buf.put_slice(&block);
Poll::Ready(Ok(()))
}
}
impl ShardSource for ScratchReuseSource {}
#[derive(Default)]
struct VectoredCountingWriter {
@@ -1446,6 +1740,70 @@ mod tests {
assert!(out.is_empty(), "corrupt bytes must never reach the caller's buffer");
}
#[tokio::test]
async fn chunked_handoff_verifies_data_split_across_hash_boundaries() {
const SHARD: usize = 4096;
let algo = HashAlgorithm::HighwayHash256S;
let data: Vec<u8> = (0..SHARD).map(|index| (index % 251) as u8).collect();
let mut encoded = Vec::new();
BitrotWriter::new(&mut encoded, SHARD, algo.clone())
.write(&data)
.await
.expect("write shard");
let mut output = Vec::with_capacity(SHARD);
BitrotReader::new(FragmentedSource::new(encoded, &[3, 11, 19, 37, 128]), SHARD, algo, false)
.read_appending(&mut output, SHARD)
.await
.expect("fragmented shard must verify");
assert_eq!(output, data);
}
#[tokio::test]
async fn chunked_handoff_never_appends_a_corrupt_shard() {
const SHARD: usize = 4096;
let algo = HashAlgorithm::HighwayHash256S;
let mut encoded = Vec::new();
BitrotWriter::new(&mut encoded, SHARD, algo.clone())
.write(&vec![9u8; SHARD])
.await
.expect("write shard");
let last = encoded.len() - 1;
encoded[last] ^= 0xff;
let mut output = Vec::with_capacity(SHARD);
let err = BitrotReader::new(FragmentedSource::new(encoded, &[7, 17, 31]), SHARD, algo, false)
.read_appending(&mut output, SHARD)
.await
.expect_err("corrupt fragmented shard must fail");
assert_eq!(err.kind(), io::ErrorKind::InvalidData);
assert!(output.is_empty());
}
#[tokio::test]
async fn chunked_handoff_does_not_hash_when_verification_is_skipped() {
const SHARD: usize = 4096;
let algo = HashAlgorithm::HighwayHash256S;
let mut encoded = Vec::new();
BitrotWriter::new(&mut encoded, SHARD, algo.clone())
.write(&vec![9u8; SHARD])
.await
.expect("write shard");
encoded[0] ^= 0xff;
let mut output = Vec::with_capacity(SHARD);
let mut reader = BitrotReader::new(FragmentedSource::new(encoded, &[7, 17, 31]), SHARD, algo, true);
reader
.read_appending(&mut output, SHARD)
.await
.expect("skipped verification must accept fragmented shard bytes");
assert_eq!(reader.last_verify_duration(), Duration::ZERO);
assert_eq!(output, vec![9u8; SHARD]);
}
#[tokio::test]
async fn read_appending_rejects_a_want_larger_than_the_shard() {
let algo = HashAlgorithm::HighwayHash256;
@@ -1497,10 +1855,21 @@ mod tests {
// Equivalence: same bytes out of both paths.
let mut via_mem: Vec<u8> = Vec::with_capacity(SHARD);
BitrotReader::new(Cursor::new(Bytes::from(encoded.clone())), SHARD, algo.clone(), false)
let mut memory_reader = BitrotReader::new(Cursor::new(Bytes::from(encoded.clone())), SHARD, algo.clone(), false);
memory_reader
.read_appending(&mut via_mem, SHARD)
.await
.expect("in-memory read");
assert_eq!(
memory_reader.chunks.capacity(),
0,
"the synchronous fast path must not allocate chunk storage"
);
assert_eq!(
memory_reader.buf.capacity(),
0,
"the synchronous fast path must not allocate scratch storage"
);
let mut via_stream: Vec<u8> = Vec::with_capacity(SHARD);
BitrotReader::new(Cursor::new(encoded), SHARD, algo, false)
@@ -1537,4 +1906,152 @@ mod tests {
assert_eq!(err.kind(), std::io::ErrorKind::InvalidData);
assert!(out.is_empty(), "corrupt bytes must never reach the caller's buffer");
}
#[tokio::test]
async fn streaming_fallback_reuses_initialized_scratch() {
const SHARD: usize = 4096;
let algo = HashAlgorithm::HighwayHash256S;
let data = vec![7u8; SHARD];
let encoded = encode_one_block(&data, SHARD, algo.clone()).await;
let source = ScratchReuseSource {
block: Some(Bytes::copy_from_slice(&encoded)),
saw_reused_scratch: false,
};
let mut reader = BitrotReader::new(source, SHARD, algo, false);
reader.buf = vec![0xa5; encoded.len()];
let mut output = Vec::new();
reader
.read_appending(&mut output, SHARD)
.await
.expect("streaming fallback should verify");
assert!(reader.inner.saw_reused_scratch, "capability probing must not clear reusable scratch");
assert_eq!(output, data);
}
#[tokio::test]
async fn chunked_handoff_bounds_production_sized_one_byte_fragments() {
const SHARD: usize = 1024 * 1024 / 4;
let algo = HashAlgorithm::HighwayHash256S;
let data: Vec<u8> = (0..SHARD).map(|index| (index % 251) as u8).collect();
let encoded = encode_one_block(&data, SHARD, algo.clone()).await;
let encoded_len = encoded.len();
let mut reader = BitrotReader::new(GeneratedChunkSource::new(encoded, 1), SHARD, algo, false);
let mut output = Vec::with_capacity(SHARD);
reader
.read_appending(&mut output, SHARD)
.await
.expect("one-byte fragments should verify with bounded retained state");
assert_eq!(output, data);
assert_eq!(reader.chunks.len(), MAX_RETAINED_CHUNKS_PER_BLOCK);
assert!(reader.chunks.capacity() <= MAX_RETAINED_CHUNKS_PER_BLOCK);
assert_eq!(reader.buf.len(), encoded_len - MAX_RETAINED_CHUNKS_PER_BLOCK);
}
#[tokio::test]
async fn chunked_handoff_keeps_sixty_four_frames_zero_copy_and_respects_poll_budget() {
const SHARD: usize = 1024 * 1024;
const FRAME: usize = 16 * 1024;
let algo = HashAlgorithm::HighwayHash256S;
let small_data = vec![3u8; 4096];
let small_encoded = encode_one_block(&small_data, 4096, algo.clone()).await;
let mut exact_reader =
BitrotReader::new(FragmentedSource::new(small_encoded.clone(), &[1; 63]), 4096, algo.clone(), false);
let mut exact_output = Vec::new();
exact_reader
.read_appending(&mut exact_output, 4096)
.await
.expect("exactly sixty-four frames should verify");
assert_eq!(exact_output, small_data);
assert_eq!(exact_reader.chunks.len(), MAX_RETAINED_CHUNKS_PER_BLOCK);
assert!(exact_reader.buf.is_empty(), "the threshold itself must remain zero-copy");
let mut yielded_reader = BitrotReader::new(FragmentedSource::new(small_encoded, &[1; 65]), 4096, algo.clone(), false);
let mut yielded_output = Vec::new();
let mut yielded_read = Box::pin(yielded_reader.read_appending(&mut yielded_output, 4096));
let mut cx = Context::from_waker(std::task::Waker::noop());
assert!(std::future::Future::poll(yielded_read.as_mut(), &mut cx).is_pending());
assert!(matches!(std::future::Future::poll(yielded_read.as_mut(), &mut cx), Poll::Ready(Ok(4096))));
drop(yielded_read);
assert_eq!(yielded_output, small_data);
let data = vec![7u8; SHARD];
let encoded = encode_one_block(&data, SHARD, algo.clone()).await;
let mut reader = BitrotReader::new(FragmentedSource::new(encoded, &[FRAME; 64]), SHARD, algo, false);
let mut output = Vec::with_capacity(SHARD);
let mut read = Box::pin(reader.read_appending(&mut output, SHARD));
assert!(
matches!(std::future::Future::poll(read.as_mut(), &mut cx), Poll::Ready(Ok(SHARD))),
"sixty-five normal HTTP frames should complete without a cooperative yield"
);
drop(read);
assert_eq!(output, data);
assert_eq!(reader.chunks.len(), MAX_RETAINED_CHUNKS_PER_BLOCK);
assert_eq!(reader.buf.len(), HashAlgorithm::HighwayHash256S.size());
}
#[tokio::test]
async fn chunked_tail_failures_preserve_errors_and_output() {
const SHARD: usize = 4096;
let algo = HashAlgorithm::HighwayHash256S;
let data = vec![7u8; SHARD];
let encoded = encode_one_block(&data, SHARD, algo.clone()).await;
let sentinel = vec![1u8, 2, 3];
let mut short_output = sentinel.clone();
let short_err = BitrotReader::new(GeneratedChunkSource::new(encoded[..100].to_vec(), 1), SHARD, algo.clone(), false)
.read_appending(&mut short_output, SHARD)
.await
.expect_err("EOF after the retention threshold must stay a short read");
assert_eq!(short_err.kind(), io::ErrorKind::UnexpectedEof);
assert_eq!(short_output, sentinel);
let mut corrupt = encoded.clone();
let last = corrupt.len() - 1;
corrupt[last] ^= 0xff;
let mut corrupt_output = sentinel.clone();
let corrupt_err = BitrotReader::new(GeneratedChunkSource::new(corrupt, 1), SHARD, algo.clone(), false)
.read_appending(&mut corrupt_output, SHARD)
.await
.expect_err("corrupt coalesced tail must fail verification");
assert_eq!(corrupt_err.kind(), io::ErrorKind::InvalidData);
assert_eq!(corrupt_output, sentinel);
let mut failed_output = sentinel.clone();
let body_err = BitrotReader::new(GeneratedChunkSource::failing(encoded, 1, 65), SHARD, algo, false)
.read_appending(&mut failed_output, SHARD)
.await
.expect_err("a terminal body error must not become EOF");
let source = body_err
.get_ref()
.and_then(|source| source.downcast_ref::<rustfs_rio::InternodeHttpError>())
.expect("body error should retain internode classification");
assert_eq!(source.kind(), rustfs_rio::InternodeHttpErrorKind::BodyStreamAborted);
assert_eq!(failed_output, sentinel);
}
#[tokio::test]
async fn chunked_handoff_rejects_invalid_source_contracts() {
const SHARD: usize = 64;
for mode in [
InvalidChunkMode::Empty,
InvalidChunkMode::Oversized,
InvalidChunkMode::UnsupportedAfterChunk,
] {
let source = InvalidChunkSource { mode };
let mut output = vec![9u8];
let err = BitrotReader::new(source, SHARD, HashAlgorithm::HighwayHash256S, false)
.read_appending(&mut output, SHARD)
.await
.expect_err("invalid chunk contracts must fail closed");
assert_eq!(err.kind(), io::ErrorKind::InvalidData);
assert_eq!(output, vec![9u8]);
}
}
}
+119 -39
View File
@@ -25,7 +25,9 @@ use crate::disk::error_reduce::reduce_errs;
use crate::erasure::codec::workspace::ShardBufferPool;
use crate::erasure::coding::{BitrotReader, Erasure};
use crate::io_support::bitrot::DeferredReaderStripeHandle;
use crate::set_disk::shard_source::{ShardReadCost, ShardStripeSource, StripeReadState};
use crate::set_disk::shard_source::{
INLINE_SHARD_SLOTS, ShardBuffers, ShardErrors, ShardReadCost, ShardStripeSource, StripeReadState,
};
use futures::FutureExt;
use futures::stream::{FuturesUnordered, StreamExt};
use pin_project_lite::pin_project;
@@ -41,9 +43,6 @@ use tracing::{debug, error, warn};
type ShardReadFuture<'a> = Pin<Box<dyn Future<Output = (usize, ShardReadCost, Result<Vec<u8>, Error>, bool)> + Send + 'a>>;
const INLINE_SHARD_SLOTS: usize = 32;
type ShardBuffers = SmallVec<[Option<Vec<u8>>; INLINE_SHARD_SLOTS]>;
type ShardErrors = SmallVec<[Option<Error>; INLINE_SHARD_SLOTS]>;
type ShardIndexes = SmallVec<[usize; INLINE_SHARD_SLOTS]>;
type ActiveReaders = SmallVec<[bool; INLINE_SHARD_SLOTS]>;
@@ -214,6 +213,7 @@ fn shard_read_launch_rank(cost: ShardReadCost) -> u8 {
}
}
#[allow(dead_code, reason = "launch ordering asserted by this file's tests (backlog#1823)")]
fn shard_read_launch_order(read_costs: &[ShardReadCost], num_readers: usize, locality_preference_enabled: bool) -> Vec<usize> {
let mut order: Vec<usize> = (0..num_readers).collect();
if locality_preference_enabled {
@@ -392,6 +392,7 @@ pub(crate) struct ParallelReader<R> {
// Request-scoped shard buffers keyed by shard index. Keeping ownership in
// `ParallelReader` avoids dropping unused parity/backup slot buffers between stripes.
buffers: ShardBufferPool,
stripe_state: Option<Box<StripeReadState>>,
// Lockstep-path state (verify_reconstruction == true). `engaged[i]` marks
// readers that participate in each stripe read: all data slots from the
// start, parity slots only once a data shard is missing/dead. Unengaged
@@ -408,6 +409,10 @@ where
R: crate::erasure::coding::ShardSource,
{
// Readers should handle disk errors before being passed in, ensuring each reader reaches the available number of BitrotReaders
#[allow(
dead_code,
reason = "ParallelReader constructor used only by this file's tests (backlog#1823)"
)]
pub fn new(readers: Vec<Option<BitrotReader<R>>>, e: Erasure, offset: usize, total_length: usize) -> Self {
Self::new_with_metrics_path_read_timeout_and_reconstruction_verification(
readers,
@@ -420,6 +425,7 @@ where
)
}
#[allow(dead_code, reason = "constructor used only by this file's tests (backlog#1823)")]
pub fn new_with_metrics_path(
readers: Vec<Option<BitrotReader<R>>>,
e: Erasure,
@@ -438,6 +444,7 @@ where
)
}
#[allow(dead_code, reason = "constructor used only by this file's tests (backlog#1823)")]
pub fn new_with_metrics_path_and_read_costs(
readers: Vec<Option<BitrotReader<R>>>,
e: Erasure,
@@ -514,6 +521,7 @@ where
)
}
#[allow(dead_code, reason = "constructor used only by this file's tests (backlog#1823)")]
fn new_with_read_timeout(
readers: Vec<Option<BitrotReader<R>>>,
e: Erasure,
@@ -596,6 +604,7 @@ where
verify_reconstruction,
locality_preference_enabled: get_shard_locality_preference_enabled(),
buffers: ShardBufferPool::new(e.data_shards + e.parity_shards),
stripe_state: None,
engaged,
deferred_handles: Vec::new(),
stripe_index: 0,
@@ -700,6 +709,12 @@ where
{
#[hotpath::measure(impl_type = "ParallelReader")]
pub async fn read(&mut self) -> StripeReadOutput {
let mut state = StripeReadState::with_slot_count(self.readers.len(), self.data_shards);
self.read_into_state(&mut state).await;
state.into_parts()
}
async fn read_into_state(&mut self, state: &mut StripeReadState) {
// On the reconstruction-verifying GET path, read every live shard reader
// in lockstep so all readers advance one block per stripe and stay
// mutually aligned. The adaptive data-first path below only reads
@@ -709,12 +724,14 @@ where
// than the data shards, producing "inconsistent read source shards" and
// truncating large-object GETs under concurrency (backlog#832).
if self.verify_reconstruction {
return self.read_lockstep().await;
self.read_lockstep(state).await;
return;
}
// if self.readers.len() != self.total_shards {
// return Err(io::Error::new(ErrorKind::InvalidInput, "Invalid number of readers"));
// }
let num_readers = self.readers.len();
state.reset(num_readers, self.data_shards);
let shard_size = if self.offset + self.shard_size > self.shard_file_size {
self.shard_file_size - self.offset
@@ -723,7 +740,7 @@ where
};
if shard_size == 0 {
return (smallvec![None; num_readers], smallvec![None; num_readers]);
return;
}
// Advance to the next stripe so the following read() computes the correct
@@ -734,8 +751,7 @@ where
// is only read above to derive `shard_size`, so advancing here is safe.
self.offset += shard_size;
let mut shards: ShardBuffers = smallvec![None; num_readers];
let mut errs: ShardErrors = smallvec![None; num_readers];
let (shards, errs) = state.parts_mut();
let read_costs = self.read_costs.as_slice();
let locality_preference_enabled = self.locality_preference_enabled;
let low_cost_available = self
@@ -882,8 +898,8 @@ where
}
let result_is_err = record_shard_read_result(
&mut shards,
&mut errs,
shards,
errs,
&mut retire_readers,
&mut success,
&mut successful_costs,
@@ -944,8 +960,8 @@ where
active_readers[i] = false;
completed += 1;
if record_shard_read_result(
&mut shards,
&mut errs,
shards,
errs,
&mut retire_readers,
&mut success,
&mut successful_costs,
@@ -957,7 +973,7 @@ where
failed += 1;
}
}
retire_abandoned_readers(&mut errs, &mut retire_readers, &active_readers);
retire_abandoned_readers(errs, &mut retire_readers, &active_readers);
}
if let Some(path) = self.metrics_path {
@@ -1001,8 +1017,6 @@ where
for i in retire_readers {
self.readers[i] = None;
}
(shards, errs)
}
/// Lockstep stripe read for the reconstruction-verifying GET path.
@@ -1030,18 +1044,18 @@ where
/// stripe would reintroduce the desync. A parity reader that cannot be
/// realigned (no pending deferred handle) is likewise retired instead of
/// being read out of position.
async fn read_lockstep(&mut self) -> StripeReadOutput {
async fn read_lockstep(&mut self, state: &mut StripeReadState) {
let num_readers = self.readers.len();
state.reset(num_readers, self.data_shards);
let shard_size = if self.offset + self.shard_size > self.shard_file_size {
self.shard_file_size - self.offset
} else {
self.shard_size
};
let mut shards: ShardBuffers = smallvec![None; num_readers];
let mut errs: ShardErrors = smallvec![None; num_readers];
let (shards, errs) = state.parts_mut();
if shard_size == 0 {
return (shards, errs);
return;
}
// Advance to the next stripe (see the matching note in `read`); the
@@ -1279,8 +1293,6 @@ where
for i in retire_readers {
self.readers[i] = None;
}
(shards, errs)
}
/// Attempt to bring an as-yet-unread parity reader into the lockstep read
@@ -1326,10 +1338,6 @@ where
}
}
}
pub fn can_decode(&self, shards: &[Option<Vec<u8>>]) -> bool {
shards.iter().filter(|s| s.is_some()).count() >= self.data_shards
}
}
#[async_trait::async_trait]
@@ -1337,10 +1345,20 @@ impl<R> ShardStripeSource for ParallelReader<R>
where
R: crate::erasure::coding::ShardSource,
{
async fn read_next_stripe(&mut self) -> StripeReadState {
let read_quorum = self.data_shards;
let (shards, errors) = ParallelReader::read(self).await;
StripeReadState::from_parts_with_read_costs(shards, errors, &self.read_costs, read_quorum)
async fn read_next_stripe(&mut self) -> Box<StripeReadState> {
let mut state = self
.stripe_state
.take()
.unwrap_or_else(|| Box::new(StripeReadState::with_slot_count(self.readers.len(), self.data_shards)));
self.read_into_state(&mut state).await;
state
}
fn recycle_stripe(&mut self, mut state: Box<StripeReadState>) {
self.recycle_shards(state.shards_mut());
state.reset(0, self.data_shards);
debug_assert!(self.stripe_state.is_none(), "a stripe cannot be recycled twice");
self.stripe_state = Some(state);
}
}
@@ -1525,6 +1543,7 @@ impl Erasure {
.await
}
#[allow(dead_code, reason = "read-cost decode path asserted by this file's tests (backlog#1823)")]
pub(crate) async fn decode_with_read_costs<W, R>(
&self,
writer: &mut W,
@@ -1595,9 +1614,9 @@ impl Erasure {
*ret_err = Some(err.into());
}
// Equivalent to `ParallelReader::can_decode`; inlined so this helper does
// not need to borrow the reader, leaving the reader free for the
// concurrent next-stripe read under prefetch.
// Shard-availability check, written out here rather than called on the
// reader so this helper does not need to borrow it, leaving the reader
// free for the concurrent next-stripe read under prefetch.
let available_shards = shards.iter().filter(|shard| shard.is_some()).count();
if available_shards < self.data_shards {
let reason = GetObjectFailureReason::ReadQuorum;
@@ -1972,13 +1991,18 @@ mod tests {
type BoxedShardReader = crate::io_support::bitrot::ShardReader;
#[test]
fn shard_scratch_stays_inline_through_the_common_limit_and_spills_safely() {
let inline: ShardBuffers = smallvec![None; INLINE_SHARD_SLOTS];
assert!(!inline.spilled(), "the common shard-count boundary must not allocate");
let spilled: ShardBuffers = smallvec![None; INLINE_SHARD_SLOTS + 1];
assert!(spilled.spilled(), "larger supported shard counts must fall back to the heap");
assert_eq!(spilled.len(), INLINE_SHARD_SLOTS + 1);
fn parallel_reader_keeps_stripe_scratch_out_of_line() {
eprintln!(
"parallel_reader={} stripe_state={} cached_state={}",
std::mem::size_of::<ParallelReader<Cursor<Vec<u8>>>>(),
std::mem::size_of::<StripeReadState>(),
std::mem::size_of::<Option<Box<StripeReadState>>>()
);
assert_eq!(
std::mem::size_of::<Option<Box<StripeReadState>>>(),
std::mem::size_of::<usize>(),
"the request-scoped cache must remain pointer-sized",
);
}
#[tokio::test]
@@ -1997,6 +2021,62 @@ mod tests {
assert_eq!(errors.len(), TOTAL_SHARDS);
}
#[tokio::test]
async fn codec_reader_reuses_inline_and_spilled_stripe_scratch_between_reads() {
for total_shards in [INLINE_SHARD_SLOTS, INLINE_SHARD_SLOTS + 1] {
let data_shards = total_shards - 1;
let readers = std::iter::repeat_with(|| None).take(total_shards).collect();
let erasure = Erasure::new(data_shards, 1, data_shards * 2);
let mut reader: ParallelReader<Cursor<Vec<u8>>> = ParallelReader::new(readers, erasure, 0, data_shards * 2);
let first = ShardStripeSource::read_next_stripe(&mut reader).await;
let first_state = (&*first) as *const StripeReadState;
let first_storage = first.scratch_storage();
assert_eq!(first_storage.2, total_shards > INLINE_SHARD_SLOTS);
assert_eq!(first_storage.3, total_shards > INLINE_SHARD_SLOTS);
ShardStripeSource::recycle_stripe(&mut reader, first);
let second = ShardStripeSource::read_next_stripe(&mut reader).await;
let second_storage = second.scratch_storage();
assert_eq!(
(&*second) as *const StripeReadState,
first_state,
"the request-scoped state must be reused"
);
assert_eq!(second_storage.0, first_storage.0, "shard slots must reuse their allocation");
assert_eq!(second_storage.1, first_storage.1, "error slots must reuse their allocation");
assert_eq!(second.into_parts().0.len(), total_shards);
}
}
#[tokio::test]
async fn codec_reader_returns_shard_allocations_to_the_request_pool() {
const SHARD_SIZE: usize = 16;
let hash_algo = HashAlgorithm::None;
let readers = vec![Some(create_reader(SHARD_SIZE, 2, 0x5a, &hash_algo, false).await)];
let erasure = Erasure::new(1, 0, SHARD_SIZE);
let mut reader = ParallelReader::new(readers, erasure, 0, SHARD_SIZE * 2);
let first = ShardStripeSource::read_next_stripe(&mut reader).await;
let first_allocation = first
.shard_allocation(0)
.expect("the first stripe should own its shard allocation");
ShardStripeSource::recycle_stripe(&mut reader, first);
assert_eq!(
reader.buffers.stored_allocation(0),
Some(first_allocation),
"recycling a stripe must return its shard allocation to the request pool"
);
let second = ShardStripeSource::read_next_stripe(&mut reader).await;
assert_eq!(
second.shard_allocation(0),
Some(first_allocation),
"the next stripe must reuse the pooled shard allocation"
);
}
/// Counts the raw bytes pulled from a shard stream, to prove which shards
/// a decode path actually touches (backlog#923 call-count evidence).
struct CountingShardReader {
@@ -65,7 +65,7 @@ enum FillPolicy {
}
impl FillPolicy {
fn from_env() -> Self {
fn load() -> Self {
match rustfs_utils::get_env_usize(
ENV_RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT,
DEFAULT_RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT,
@@ -75,6 +75,22 @@ impl FillPolicy {
}
}
fn from_env() -> Self {
#[cfg(test)]
{
Self::load()
}
#[cfg(not(test))]
{
Self::cached_core(Self::load)
}
}
fn cached_core(load: impl FnOnce() -> Self) -> Self {
static CACHED: std::sync::OnceLock<FillPolicy> = std::sync::OnceLock::new();
*CACHED.get_or_init(load)
}
const fn max_inflight(self) -> usize {
match self {
Self::SingleInFlight => 1,
@@ -122,6 +138,10 @@ where
S: ShardStripeSource + Send + 'static,
E: ErasureDecodeEngine + Clone + Send + Sync + 'static,
{
#[allow(
dead_code,
reason = "default-metrics-path constructor used only by this file's tests (backlog#1823)"
)]
pub(crate) fn new(source: S, engine: E, total_length: usize) -> io::Result<Self> {
Self::new_with_metrics_path(source, engine, total_length, GET_OBJECT_PATH_CODEC_STREAMING)
}
@@ -479,22 +499,30 @@ where
let mut deferred_error = None;
let fill_stage_start = get_stage_timer_if_enabled(stage_metrics_enabled);
let stripe_read_stage_start = get_stage_timer_if_enabled(stage_metrics_enabled);
let state = source.read_next_stripe().await;
let mut state = source.read_next_stripe().await;
record_get_stage_duration_if_enabled(metrics_path, GET_STAGE_STRIPE_READ, stripe_read_stage_start);
let decode_stage_start = get_stage_timer_if_enabled(stage_metrics_enabled);
let mut output_buf = reusable_buffers.pop().unwrap_or_default();
let result =
match decode_stripe_into(metrics_path, stage_metrics_enabled, engine, workspace, state, remaining, &mut output_buf) {
Ok(true) => Ok(Some(output_buf)),
Ok(false) => {
reusable_buffers.push(output_buf);
Ok(None)
}
Err(err) => {
reusable_buffers.push(output_buf);
Err(err)
}
};
let result = match decode_stripe_into(
metrics_path,
stage_metrics_enabled,
engine,
workspace,
&mut state,
remaining,
&mut output_buf,
) {
Ok(true) => Ok(Some(output_buf)),
Ok(false) => {
reusable_buffers.push(output_buf);
Ok(None)
}
Err(err) => {
reusable_buffers.push(output_buf);
Err(err)
}
};
source.recycle_stripe(state);
record_get_stage_duration_if_enabled(metrics_path, GET_STAGE_DECODE, decode_stage_start);
if let Ok(Some(first_buf)) = result.as_ref() {
let mut remaining_after_first = remaining.saturating_sub(first_buf.len());
@@ -503,7 +531,7 @@ where
break;
}
let stripe_read_stage_start = get_stage_timer_if_enabled(stage_metrics_enabled);
let state = source.read_next_stripe().await;
let mut state = source.read_next_stripe().await;
record_get_stage_duration_if_enabled(metrics_path, GET_STAGE_STRIPE_READ, stripe_read_stage_start);
let decode_stage_start = get_stage_timer_if_enabled(stage_metrics_enabled);
let mut queued_buf = reusable_buffers.pop().unwrap_or_default();
@@ -512,10 +540,11 @@ where
stage_metrics_enabled,
engine,
workspace,
state,
&mut state,
remaining_after_first,
&mut queued_buf,
);
source.recycle_stripe(state);
record_get_stage_duration_if_enabled(metrics_path, GET_STAGE_DECODE, decode_stage_start);
match queued_result {
Ok(true) => {
@@ -654,6 +683,10 @@ pub(crate) struct SyncErasureDecodeReader<R> {
}
impl<R> SyncErasureDecodeReader<R> {
#[allow(
dead_code,
reason = "default-metrics-path constructor used only by this file's tests (backlog#1823)"
)]
pub(crate) fn new(inner: R) -> Self {
Self::new_with_metrics_path(inner, GET_OBJECT_PATH_CODEC_STREAMING)
}
@@ -717,7 +750,7 @@ fn decode_stripe_into<E>(
stage_metrics_enabled: bool,
engine: &E,
workspace: &mut E::Workspace,
state: StripeReadState,
state: &mut StripeReadState,
remaining: usize,
output: &mut Vec<u8>,
) -> io::Result<bool>
@@ -725,7 +758,7 @@ where
E: ErasureDecodeEngine,
{
output.clear();
if state.slots().is_empty() {
if state.is_empty() {
return Ok(false);
}
if !state.can_decode() {
@@ -741,13 +774,12 @@ where
);
record_get_stage_duration_if_enabled(metrics_path, GET_STAGE_RECONSTRUCT, reconstruct_stage_start);
let emit_stage_start = get_stage_timer_if_enabled(stage_metrics_enabled);
emit_data_shards_into(&state, engine.data_shards(), engine.block_size(), remaining, output)?;
emit_data_shards_into(state, engine.data_shards(), engine.block_size(), remaining, output)?;
record_get_stage_duration_if_enabled(metrics_path, GET_STAGE_EMIT, emit_stage_start);
return Ok(true);
}
let (mut shards, _errs) = state.into_parts();
let reconstruct_outcome = match engine.reconstruct_into(&mut shards, workspace) {
let reconstruct_outcome = match engine.reconstruct_into(state.shards_mut(), workspace) {
Ok(outcome) => outcome,
Err(err) => {
record_get_stage_duration_if_enabled(metrics_path, GET_STAGE_RECONSTRUCT, reconstruct_stage_start);
@@ -757,7 +789,7 @@ where
rustfs_io_metrics::record_get_object_reconstruct_outcome(metrics_path, engine.engine_name(), reconstruct_outcome);
record_get_stage_duration_if_enabled(metrics_path, GET_STAGE_RECONSTRUCT, reconstruct_stage_start);
if shards.len() < engine.data_shards() {
if state.shards_mut().len() < engine.data_shards() {
return Err(io::Error::new(
ErrorKind::UnexpectedEof,
"decoded stripe has fewer shards than data shard count",
@@ -766,7 +798,7 @@ where
let emit_stage_start = get_stage_timer_if_enabled(stage_metrics_enabled);
reserve_output_capacity(output, engine.block_size().min(remaining));
for shard in shards.iter().take(engine.data_shards()) {
for shard in state.shards_mut().iter().take(engine.data_shards()) {
if output.len() >= remaining {
break;
}
@@ -781,6 +813,7 @@ where
Ok(true)
}
#[allow(dead_code, reason = "shard emission asserted by this file's tests (backlog#1823)")]
fn emit_data_shards(state: &StripeReadState, data_shards: usize, block_size: usize, remaining: usize) -> io::Result<Vec<u8>> {
let mut output = Vec::new();
emit_data_shards_into(state, data_shards, block_size, remaining, &mut output)?;
@@ -806,10 +839,7 @@ fn emit_data_shards_into(
if output.len() >= remaining {
break;
}
let Some(slot) = state.slot_by_index(index) else {
return Err(io::Error::new(ErrorKind::UnexpectedEof, "decoded stripe is missing a data shard"));
};
let Some(shard) = slot.data_bytes() else {
let Some(shard) = state.data_bytes(index) else {
return Err(io::Error::new(ErrorKind::UnexpectedEof, "decoded stripe is missing a data shard"));
};
let copy_len = shard.len().min(remaining - output.len());
@@ -826,7 +856,7 @@ mod tests {
};
use crate::erasure::coding::decode::ParallelReader;
use crate::erasure::coding::{BitrotReader, BitrotWriter, Erasure};
use crate::set_disk::shard_source::{ShardSlot, StripeReadState};
use crate::set_disk::shard_source::StripeReadState;
use rustfs_utils::HashAlgorithm;
use std::collections::VecDeque;
use std::future::{pending, poll_fn};
@@ -845,6 +875,13 @@ mod tests {
read_count: Option<Arc<AtomicUsize>>,
}
struct RecordingStripeSource {
stripes: VecDeque<StripeReadState>,
read_quorum: usize,
reads: usize,
recycles: usize,
}
struct BlockingSource {
started: Arc<Notify>,
dropped: Arc<AtomicUsize>,
@@ -899,25 +936,43 @@ mod tests {
#[async_trait::async_trait]
impl ShardStripeSource for VecStripeSource {
async fn read_next_stripe(&mut self) -> StripeReadState {
async fn read_next_stripe(&mut self) -> Box<StripeReadState> {
if let Some(read_count) = &self.read_count {
read_count.fetch_add(1, Ordering::SeqCst);
}
self.stripes
.pop_front()
.unwrap_or_else(|| StripeReadState::new(Vec::new(), self.read_quorum))
Box::new(
self.stripes
.pop_front()
.unwrap_or_else(|| StripeReadState::from_parts(Vec::new(), Vec::new(), self.read_quorum)),
)
}
}
#[async_trait::async_trait]
impl ShardStripeSource for RecordingStripeSource {
async fn read_next_stripe(&mut self) -> Box<StripeReadState> {
self.reads += 1;
Box::new(
self.stripes
.pop_front()
.unwrap_or_else(|| StripeReadState::from_parts(Vec::new(), Vec::new(), self.read_quorum)),
)
}
fn recycle_stripe(&mut self, _state: Box<StripeReadState>) {
self.recycles += 1;
}
}
#[async_trait::async_trait]
impl ShardStripeSource for BlockingSource {
async fn read_next_stripe(&mut self) -> StripeReadState {
async fn read_next_stripe(&mut self) -> Box<StripeReadState> {
let _guard = BlockingSourceDropGuard {
dropped: Arc::clone(&self.dropped),
};
self.started.notify_one();
pending::<()>().await;
StripeReadState::new(Vec::new(), self.read_quorum)
Box::new(StripeReadState::from_parts(Vec::new(), Vec::new(), self.read_quorum))
}
}
@@ -1090,6 +1145,23 @@ mod tests {
});
}
#[test]
fn fill_policy_production_cache_loads_once() {
use std::cell::Cell;
let loads = Cell::new(0);
for _ in 0..3 {
assert_eq!(
FillPolicy::cached_core(|| {
loads.set(loads.get() + 1);
FillPolicy::DualInFlight
}),
FillPolicy::DualInFlight
);
}
assert_eq!(loads.get(), 1, "the production fill policy must not re-read the environment per reader");
}
#[test]
fn erasure_decode_reader_rejects_invalid_engine_shape() {
let source = VecStripeSource {
@@ -1689,7 +1761,10 @@ mod tests {
.pop_front()
.expect("first stripe should exist");
let mut source = VecStripeSource {
stripes: VecDeque::from([first_state, StripeReadState::new(Vec::new(), erasure.data_shards)]),
stripes: VecDeque::from([
first_state,
StripeReadState::from_parts(Vec::new(), Vec::new(), erasure.data_shards),
]),
read_quorum: erasure.data_shards,
read_count: None,
};
@@ -1724,13 +1799,14 @@ mod tests {
.stripes
.pop_front()
.expect("first stripe should exist");
let mut source = VecStripeSource {
let mut source = RecordingStripeSource {
stripes: VecDeque::from([
first_state,
StripeReadState::new(vec![ShardSlot::data(0, vec![1])], erasure.data_shards),
StripeReadState::from_parts(vec![Some(vec![1])], Vec::new(), erasure.data_shards),
]),
read_quorum: erasure.data_shards,
read_count: None,
reads: 0,
recycles: 0,
};
let engine = LegacyEcDecodeEngine::new(erasure);
let mut workspace = engine.prepare_workspace(4).expect("workspace should be prepared");
@@ -1756,6 +1832,8 @@ mod tests {
.kind(),
ErrorKind::Other
);
assert_eq!(source.reads, 2, "the fill must read the primary and queued stripe");
assert_eq!(source.recycles, source.reads, "every completed stripe read must be recycled");
}
#[tokio::test]
@@ -1768,7 +1846,7 @@ mod tests {
.stripes
.pop_front()
.expect("first stripe should exist"),
StripeReadState::new(Vec::new(), erasure.data_shards),
StripeReadState::from_parts(Vec::new(), Vec::new(), erasure.data_shards),
]),
read_quorum: erasure.data_shards,
read_count: None,
@@ -2028,17 +2106,11 @@ mod tests {
}
#[test]
fn emit_data_shards_preserves_output_order_for_out_of_order_slots() {
let state = StripeReadState::new(
vec![
ShardSlot::data(1, b"cd".to_vec()),
ShardSlot::data(0, b"ab".to_vec()),
ShardSlot::data(2, b"ef".to_vec()),
],
2,
);
fn emit_data_shards_preserves_output_order() {
let state =
StripeReadState::from_parts(vec![Some(b"ab".to_vec()), Some(b"cd".to_vec()), Some(b"ef".to_vec())], Vec::new(), 2);
let output = emit_data_shards(&state, 3, 6, 5).expect("out-of-order data slots should emit by shard index");
let output = emit_data_shards(&state, 3, 6, 5).expect("data slots should emit by shard index");
assert_eq!(output, b"abcde");
}
@@ -2051,27 +2123,27 @@ mod tests {
};
let mut workspace = engine.prepare_workspace(4).expect("workspace should be prepared");
let mut output = Vec::with_capacity(1);
let short_state = StripeReadState::new(vec![ShardSlot::data(0, vec![1, 2, 3, 4])], 1);
let mut short_state = StripeReadState::from_parts(vec![Some(vec![1, 2, 3, 4])], Vec::new(), 1);
let err = decode_stripe_into(
GET_OBJECT_PATH_CODEC_STREAMING,
false,
&engine,
&mut workspace,
short_state,
&mut short_state,
8,
&mut output,
)
.expect_err("decoded stripe shorter than data shard count must fail");
assert_eq!(err.kind(), ErrorKind::UnexpectedEof);
let missing_state = StripeReadState::from_parts(vec![None, Some(vec![5, 6, 7, 8])], Vec::new(), 1);
let mut missing_state = StripeReadState::from_parts(vec![None, Some(vec![5, 6, 7, 8])], Vec::new(), 1);
let err = decode_stripe_into(
GET_OBJECT_PATH_CODEC_STREAMING,
false,
&engine,
&mut workspace,
missing_state,
&mut missing_state,
8,
&mut output,
)
@@ -2082,6 +2154,35 @@ mod tests {
assert!(output.capacity() >= 32);
}
#[test]
fn decode_stripe_reconstructs_in_place_without_replacing_slot_storage() {
let erasure = Erasure::new(2, 1, 8);
let engine = LegacyEcDecodeEngine::new(erasure.clone());
let mut workspace = engine.prepare_workspace(4).expect("workspace should be prepared");
let encoded = erasure.encode_data(b"abcdefgh").expect("test stripe should encode");
let mut shards = encoded.into_iter().map(|shard| Some(shard.to_vec())).collect::<Vec<_>>();
shards[0] = None;
let mut state = StripeReadState::from_parts(shards, vec![Some(DiskError::FileCorrupt)], 2);
let before = state.scratch_storage();
let mut output = Vec::new();
let decoded = decode_stripe_into(
GET_OBJECT_PATH_CODEC_STREAMING,
false,
&engine,
&mut workspace,
&mut state,
8,
&mut output,
)
.expect("degraded stripe should reconstruct");
assert!(decoded);
assert_eq!(output, b"abcdefgh");
assert_eq!(state.scratch_storage().0, before.0, "reconstruction must retain shard slot storage");
assert_eq!(state.scratch_storage().1, before.1, "unused error storage must not be rebuilt");
}
#[tokio::test]
async fn erasure_decode_reader_reports_short_source() {
let erasure = Erasure::new(4, 2, 32);
+197 -42
View File
@@ -18,10 +18,12 @@ use crate::disk::error_reduce::{
};
use crate::erasure::coding::BitrotWriterWrapper;
use crate::erasure::coding::Erasure;
use crate::erasure::coding::erasure::EncodedBlock;
use crate::runtime::sources as runtime_sources;
use bytes::{Bytes, BytesMut};
use futures::StreamExt;
use futures::stream::FuturesUnordered;
use rustfs_utils::HashAlgorithm;
use std::sync::Arc;
use std::time::Instant;
use std::vec;
@@ -164,6 +166,7 @@ where
if total == 0 { Ok(None) } else { Ok(Some(total)) }
}
#[allow(dead_code, reason = "byte accounting asserted by this file's tests (backlog#1823)")]
fn queued_block_bytes(block: &[Bytes]) -> usize {
block.iter().map(Bytes::len).sum()
}
@@ -223,8 +226,8 @@ async fn send_queued<T>(
sender.send(InflightEntry::new(entry, bytes)).await
}
fn queued_batch_bytes(batch: &[Vec<Bytes>]) -> usize {
batch.iter().map(|block| queued_block_bytes(block)).sum()
fn queued_batch_bytes(batch: &[EncodedBlock]) -> usize {
batch.iter().map(EncodedBlock::queued_bytes).sum()
}
fn dominant_error_summary_label(summary: &WriteQuorumFailureSummary) -> &'static str {
@@ -336,7 +339,7 @@ impl<'a> MultiWriter<'a> {
}
}
async fn write_shard(writer_opt: &mut Option<BitrotWriterWrapper>, err: &mut Option<Error>, shard: &Bytes) {
async fn write_shard(writer_opt: &mut Option<BitrotWriterWrapper>, err: &mut Option<Error>, shard: &[u8]) {
match writer_opt {
Some(writer) => {
match writer.write(shard).await {
@@ -361,12 +364,20 @@ impl<'a> MultiWriter<'a> {
}
pub async fn write(&mut self, data: Vec<Bytes>) -> std::io::Result<()> {
assert_eq!(data.len(), self.writers.len());
self.write_shards(data.iter().map(Bytes::as_ref)).await
}
async fn write_block(&mut self, block: &EncodedBlock) -> std::io::Result<()> {
self.write_shards(block.shards()).await
}
async fn write_shards<'b>(&mut self, shards: impl ExactSizeIterator<Item = &'b [u8]>) -> std::io::Result<()> {
assert_eq!(shards.len(), self.writers.len());
let budget = self.next_progress_budget();
{
let mut futures = FuturesUnordered::new();
for ((writer_opt, err), shard) in self.writers.iter_mut().zip(self.errs.iter_mut()).zip(data.iter()) {
for ((writer_opt, err), shard) in self.writers.iter_mut().zip(self.errs.iter_mut()).zip(shards) {
if err.is_some() {
continue; // Skip if we already have an error for this writer
}
@@ -490,10 +501,10 @@ impl<'a> MultiWriter<'a> {
}
impl Erasure {
async fn encode_block(self: Arc<Self>, encode_buf: Vec<u8>, len: usize) -> std::io::Result<(Vec<Bytes>, Vec<u8>)> {
async fn encode_block(self: Arc<Self>, encode_buf: Vec<u8>, len: usize) -> std::io::Result<(EncodedBlock, Vec<u8>)> {
let encode_stage_start = stage_timer_if_enabled();
let encode_once = move || {
let res = self.encode_data(&encode_buf[..len]);
let res = self.encode_data_block(&encode_buf[..len]);
(res, encode_buf)
};
@@ -518,9 +529,9 @@ impl Erasure {
Ok((res?, returned_buf))
}
async fn encode_block_bytes_mut(self: Arc<Self>, encode_buf: BytesMut, len: usize) -> std::io::Result<Vec<Bytes>> {
async fn encode_block_bytes_mut(self: Arc<Self>, encode_buf: BytesMut, len: usize) -> std::io::Result<EncodedBlock> {
let encode_stage_start = stage_timer_if_enabled();
let encode_once = move || self.encode_data_bytes_mut(encode_buf, len);
let encode_once = move || self.encode_data_bytes_mut_block(encode_buf, len);
let res = match tokio::runtime::Handle::current().runtime_flavor() {
// Same rationale as encode_block: inline the short EC burst on the
@@ -576,13 +587,46 @@ impl Erasure {
));
}
let shards = self.encode_data_owned(buf)?;
let block = self.encode_data_owned_block(buf)?;
let mut mw = MultiWriter::new(writers, quorum);
mw.write(shards).await?;
mw.write_block(&block).await?;
mw.shutdown().await?;
Ok((reader, total))
}
/// Encode a small inline object directly into its per-disk bitrot payloads.
/// The returned bytes are the same `[hash][shard]` representation produced
/// by `BitrotWriter`, ready to be embedded in each disk's staged `xl.meta`.
#[hotpath::measure(impl_type = "Erasure")]
pub(crate) async fn encode_inline_shards_with_size_hint<R>(
self: Arc<Self>,
mut reader: R,
size_hint: usize,
) -> std::io::Result<(R, usize, Vec<Bytes>)>
where
R: AsyncRead + Send + Sync + Unpin,
{
use tokio::io::AsyncReadExt;
let mut buf = Vec::with_capacity(small_ingest_capacity(&self, size_hint));
let total = reader.read_to_end(&mut buf).await?;
if total == 0 {
return Ok((reader, 0, Vec::new()));
}
let block = self.encode_data_owned_block(buf)?;
let mut inline_shards = Vec::with_capacity(block.shards().len());
for shard in block.shards() {
let hash = HashAlgorithm::HighwayHash256S.hash_encode(shard);
let mut encoded = BytesMut::with_capacity(hash.as_ref().len() + shard.len());
encoded.extend_from_slice(hash.as_ref());
encoded.extend_from_slice(shard);
inline_shards.push(encoded.freeze());
}
Ok((reader, total, inline_shards))
}
#[hotpath::measure(impl_type = "Erasure")]
pub async fn encode<R>(
self: Arc<Self>,
@@ -624,7 +668,7 @@ impl Erasure {
let expanded_block_bytes = self.shard_size().saturating_mul(self.total_shard_count());
let max_inflight_bytes = erasure_encode_max_inflight_bytes();
let inflight_blocks = encode_channel_capacity(expanded_block_bytes, max_inflight_bytes);
let (tx, mut rx) = mpsc::channel::<InflightEntry<Vec<Bytes>>>(inflight_blocks);
let (tx, mut rx) = mpsc::channel::<InflightEntry<EncodedBlock>>(inflight_blocks);
let mut task = AbortOnDropTask::new(tokio::spawn(async move {
let block_size = self.block_size;
@@ -646,7 +690,7 @@ impl Erasure {
let encode_buf = buf;
let res = self.clone().encode_block_bytes_mut(encode_buf, n).await?;
buf = BytesMut::with_capacity(ingest_capacity);
let queued_bytes = queued_block_bytes(&res);
let queued_bytes = res.queued_bytes();
let _producer_stage = rustfs_io_metrics::track_ec_encode_producer_bytes(queued_bytes);
let send_wait_stage_start = stage_timer_if_enabled();
if let Err(err) = send_queued(&tx, res, queued_bytes).await {
@@ -676,7 +720,7 @@ impl Erasure {
let encode_buf = std::mem::take(&mut buf);
let (res, returned_buf) = self.clone().encode_block(encode_buf, n).await?;
buf = returned_buf;
let queued_bytes = queued_block_bytes(&res);
let queued_bytes = res.queued_bytes();
let _producer_stage = rustfs_io_metrics::track_ec_encode_producer_bytes(queued_bytes);
let send_wait_stage_start = stage_timer_if_enabled();
if let Err(err) = send_queued(&tx, res, queued_bytes).await {
@@ -720,9 +764,9 @@ impl Erasure {
if block.is_empty() {
break;
}
let _writer_stage = rustfs_io_metrics::track_ec_encode_writer_bytes(queued_block_bytes(&block));
let _writer_stage = rustfs_io_metrics::track_ec_encode_writer_bytes(block.queued_bytes());
let write_stage_start = stage_timer_if_enabled();
if let Err(err) = writers.write(block).await {
if let Err(err) = writers.write_block(&block).await {
write_err = Some(err);
break;
}
@@ -769,7 +813,7 @@ impl Erasure {
let inflight_blocks = encode_channel_capacity(expanded_block_bytes, max_inflight_bytes);
let batch_blocks = encode_batch_block_count().min(inflight_blocks);
let channel_capacity = inflight_blocks.div_ceil(batch_blocks).max(1);
let (tx, mut rx) = mpsc::channel::<InflightEntry<Vec<Vec<Bytes>>>>(channel_capacity);
let (tx, mut rx) = mpsc::channel::<InflightEntry<Vec<EncodedBlock>>>(channel_capacity);
let mut task = AbortOnDropTask::new(tokio::spawn(async move {
let block_size = self.block_size;
@@ -786,7 +830,7 @@ impl Erasure {
let encode_buf = std::mem::take(&mut buf);
let (res, returned_buf) = self.clone().encode_block(encode_buf, n).await?;
buf = returned_buf;
let queued_bytes = queued_block_bytes(&res);
let queued_bytes = res.queued_bytes();
pending_batch_bytes = pending_batch_bytes.saturating_add(queued_bytes);
pending_batch.push(res);
drop(pending_batch_stage.take());
@@ -845,7 +889,7 @@ impl Erasure {
let _writer_stage = rustfs_io_metrics::track_ec_encode_writer_bytes(queued_batch_bytes(&batch));
let write_stage_start = stage_timer_if_enabled();
for block in batch {
if let Err(err) = writers.write(block).await {
if let Err(err) = writers.write_block(&block).await {
write_err = Some(err);
break;
}
@@ -1895,7 +1939,11 @@ mod tests {
let baseline = rustfs_io_metrics::current_ec_encode_inflight_bytes();
let (tx, rx) = mpsc::channel(2);
let mut rx = rx;
let batch = vec![vec![Bytes::from_static(b"queued")], vec![Bytes::from_static(b"batch")]];
let erasure = Erasure::new(1, 0, 16);
let batch = vec![
erasure.encode_data_block(b"queued").expect("first block should encode"),
erasure.encode_data_block(b"batch").expect("second block should encode"),
];
let batch_bytes = queued_batch_bytes(&batch);
send_queued(&tx, batch, batch_bytes).await.expect("batch should be queued");
@@ -2117,6 +2165,39 @@ mod tests {
);
}
#[tokio::test]
async fn cancelling_inline_small_drops_stalled_write() {
const BLOCK_SIZE: usize = 16;
let (writer_entered_tx, writer_entered) = oneshot::channel();
let writes = Arc::new(std::sync::atomic::AtomicUsize::new(0));
let mut writers = vec![Some(bitrot_writer_plain(
StallOnWriteWithSignal {
entered: Some(writer_entered_tx),
writes: writes.clone(),
},
BLOCK_SIZE,
))];
let erasure = Arc::new(Erasure::new(1, 0, BLOCK_SIZE));
let reader = tokio::io::BufReader::new(Cursor::new(vec![0xA5; BLOCK_SIZE - 1]));
let encode = tokio::spawn(async move { erasure.encode_inline_small(reader, &mut writers, 1).await });
tokio::time::timeout(Duration::from_secs(1), writer_entered)
.await
.expect("inline writer should enter before cancellation")
.expect("stalling writer should signal entry");
encode.abort();
assert!(
matches!(encode.await, Err(err) if err.is_cancelled()),
"inline encode task should be cancelled"
);
assert_eq!(
writes.load(std::sync::atomic::Ordering::SeqCst),
1,
"cancellation must drop the stalled write instead of polling it again"
);
}
#[tokio::test]
async fn encode_returns_unexpected_eof_for_truncated_limited_reader() {
let committed = Arc::new(Mutex::new(Vec::new()));
@@ -2236,11 +2317,11 @@ mod tests {
.expect("bytesmut encode should succeed on current-thread runtime");
let expected_shard_size = payload.len().div_ceil(erasure.data_shards);
assert_eq!(shards.len(), erasure.total_shard_count());
assert!(shards.iter().all(|shard| shard.len() == expected_shard_size));
assert_eq!(shards.shards().len(), erasure.total_shard_count());
assert!(shards.shards().all(|shard| shard.len() == expected_shard_size));
let mut restored = Vec::new();
for shard in shards.iter().take(erasure.data_shards) {
for shard in shards.shards().take(erasure.data_shards) {
restored.extend_from_slice(shard);
}
restored.truncate(payload.len());
@@ -2343,6 +2424,41 @@ mod tests {
assert!(committed.lock().unwrap().is_empty());
}
#[tokio::test]
async fn encode_inline_shards_matches_writer_bitrot_layout() {
const DATA_SHARDS: usize = 2;
const PARITY_SHARDS: usize = 2;
const BLOCK_SIZE: usize = 64;
let checksum_algo = HashAlgorithm::HighwayHash256S;
for uses_legacy in [false, true] {
let erasure = Arc::new(Erasure::new_with_options(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE, uses_legacy));
for payload in [Vec::new(), vec![0xA5], vec![0x5A; BLOCK_SIZE - 1], vec![0xC3; BLOCK_SIZE]] {
let reader = tokio::io::BufReader::new(Cursor::new(payload.clone()));
let (_reader, total, inline_shards) = erasure
.clone()
.encode_inline_shards_with_size_hint(reader, payload.len())
.await
.expect("inline shards should encode");
assert_eq!(total, payload.len());
if payload.is_empty() {
assert!(inline_shards.is_empty());
continue;
}
let raw_shards = erasure.encode_data(&payload).expect("reference shards should encode");
assert_eq!(inline_shards.len(), DATA_SHARDS + PARITY_SHARDS);
for (inline, raw) in inline_shards.iter().zip(raw_shards) {
let mut writer =
BitrotWriterWrapper::new(CustomWriter::new_inline_buffer(), raw.len(), checksum_algo.clone());
writer.write(&raw).await.expect("reference writer should accept shard");
writer.shutdown().await.expect("reference writer should shutdown");
assert_eq!(inline.as_ref(), writer.into_inline_data().expect("reference writer should retain bytes"));
}
}
}
}
/// encode_inline_small: small payload is encoded into the correct number of shards
/// and each writer receives data after shutdown.
#[tokio::test]
@@ -2506,7 +2622,7 @@ mod tests {
assert_eq!(&next[..], &data[16..]);
}
async fn committed_shards_for_ingest_mode(use_bytesmut_ingest: bool, uses_legacy: bool, payload: &[u8]) -> Vec<Vec<u8>> {
async fn committed_shards_for_pipeline(pipeline: EncodePipeline, uses_legacy: bool, payload: &[u8]) -> Vec<Vec<u8>> {
const DATA_SHARDS: usize = 2;
const PARITY_SHARDS: usize = 2;
const TOTAL_SHARDS: usize = DATA_SHARDS + PARITY_SHARDS;
@@ -2520,10 +2636,16 @@ mod tests {
let erasure = Arc::new(Erasure::new_with_options(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE, uses_legacy));
let reader = tokio::io::BufReader::new(Cursor::new(payload.to_vec()));
let (_reader, total) = erasure
.encode_with_ingest_mode(reader, &mut writers, DATA_SHARDS, use_bytesmut_ingest)
.await
.expect("encode should succeed");
let (_reader, total) = match pipeline {
EncodePipeline::Vec => {
erasure
.encode_with_ingest_mode(reader, &mut writers, DATA_SHARDS, false)
.await
}
EncodePipeline::BytesMut => erasure.encode_with_ingest_mode(reader, &mut writers, DATA_SHARDS, true).await,
EncodePipeline::Batched => erasure.encode_batched(reader, &mut writers, DATA_SHARDS).await,
}
.expect("encode should succeed");
assert_eq!(total, payload.len());
committed
@@ -2532,31 +2654,64 @@ mod tests {
.collect()
}
/// HP-10 (rustfs/backlog#931) merge gate: the BytesMut ingest path must produce
/// byte-for-byte identical shard streams to the default Vec ingest path, for both
/// legacy-aware shard-size formulas, across empty, sub-block, exactly-full-block,
/// and multi-block-with-partial-tail payloads.
async fn expected_committed_shards(uses_legacy: bool, payload: &[u8]) -> Vec<Vec<u8>> {
const DATA_SHARDS: usize = 2;
const PARITY_SHARDS: usize = 2;
const TOTAL_SHARDS: usize = DATA_SHARDS + PARITY_SHARDS;
const BLOCK_SIZE: usize = 64;
let committed: Vec<Arc<Mutex<Vec<u8>>>> = (0..TOTAL_SHARDS).map(|_| Arc::new(Mutex::new(Vec::new()))).collect();
let mut writers: Vec<BitrotWriterWrapper> = committed
.iter()
.map(|c| bitrot_writer(DeferredCommitWriter::new(c.clone()), BLOCK_SIZE / DATA_SHARDS))
.collect();
let erasure = Erasure::new_with_options(DATA_SHARDS, PARITY_SHARDS, BLOCK_SIZE, uses_legacy);
for block in payload.chunks(BLOCK_SIZE) {
let shards = erasure.encode_data(block).expect("reference block should encode");
for (writer, shard) in writers.iter_mut().zip(shards) {
let written = writer.write(&shard).await.expect("reference shard should write");
assert_eq!(written, shard.len());
}
}
for writer in &mut writers {
writer.shutdown().await.expect("reference writer should commit");
}
committed
.iter()
.map(|c| c.lock().expect("committed buffer should be lockable").clone())
.collect()
}
/// The streaming and batched paths must produce the same bitrot-wrapped shard
/// bytes as the public block encoder for both shard-size formulas and all block
/// boundary shapes.
#[tokio::test]
async fn bytesmut_ingest_matches_vec_ingest_byte_for_byte() {
const BLOCK_SIZE: usize = 64;
let payloads: Vec<Vec<u8>> = vec![
Vec::new(),
b"tiny".to_vec(),
vec![1],
vec![2; BLOCK_SIZE - 1],
(0..BLOCK_SIZE as u32).map(|i| i as u8).collect(), // exactly one full block
vec![3u8; BLOCK_SIZE * 4], // whole number of blocks
vec![4; BLOCK_SIZE + 1],
vec![3u8; BLOCK_SIZE * 4], // whole number of blocks
(0..(BLOCK_SIZE * 3 + 7) as u32).map(|i| (i % 251) as u8).collect(), // partial tail
];
for uses_legacy in [false, true] {
for payload in &payloads {
let vec_path = committed_shards_for_ingest_mode(false, uses_legacy, payload).await;
let bytesmut_path = committed_shards_for_ingest_mode(true, uses_legacy, payload).await;
assert_eq!(
vec_path,
bytesmut_path,
"ingest paths must be byte-identical (legacy={uses_legacy}, payload_len={})",
payload.len()
);
let expected = expected_committed_shards(uses_legacy, payload).await;
for pipeline in [EncodePipeline::Vec, EncodePipeline::BytesMut, EncodePipeline::Batched] {
let actual = committed_shards_for_pipeline(pipeline, uses_legacy, payload).await;
assert_eq!(
actual,
expected,
"streaming shards must match the public block encoder (legacy={uses_legacy}, payload_len={})",
payload.len()
);
}
}
}
}
+338 -140
View File
@@ -29,12 +29,58 @@ use tokio::io::AsyncRead;
use tracing::warn;
use uuid::Uuid;
pub(crate) struct EncodedBlock {
data: Bytes,
shard_size: usize,
}
impl EncodedBlock {
fn empty() -> Self {
Self {
data: Bytes::new(),
shard_size: 0,
}
}
pub(crate) fn is_empty(&self) -> bool {
self.data.is_empty()
}
pub(crate) fn queued_bytes(&self) -> usize {
self.data.len()
}
pub(crate) fn shards(&self) -> impl ExactSizeIterator<Item = &[u8]> {
debug_assert!(self.shard_size > 0, "only non-empty encoded blocks reach shard writers");
debug_assert_eq!(self.data.len() % self.shard_size, 0);
self.data.chunks_exact(self.shard_size)
}
fn into_shards(mut self, shard_count: usize) -> Vec<Bytes> {
if self.shard_size == 0 {
return vec![Bytes::new(); shard_count];
}
let mut shards = Vec::with_capacity(shard_count);
for _ in 0..shard_count {
shards.push(self.data.split_to(self.shard_size));
}
shards
}
}
const MODERN_MAX_TOTAL_SHARDS: usize = <reed_solomon_erasure::galois_8::Field as reed_solomon_erasure::Field>::ORDER;
const MODERN_REED_SOLOMON_CACHE_MAX_ENTRIES: usize = 64;
const LEGACY_REED_SOLOMON_CACHE_MAX_ENTRIES: usize = 16;
// Vec growth may retain twice the requested logical length. Keeping the logical
// workspace at half the budget bounds each cached workspace's shard allocation to 1 MiB.
const LEGACY_REED_SOLOMON_CACHE_MAX_LOGICAL_SHARD_BYTES_PER_WORKSPACE: usize = 512 * 1024;
type ModernReedSolomonCache = RwLock<HashMap<(usize, usize), Arc<ReedSolomon>>>;
type LegacyReedSolomonCache = RwLock<HashMap<(usize, usize), Arc<LegacyReedSolomonEncoder>>>;
static MODERN_REED_SOLOMON_CACHE: OnceLock<ModernReedSolomonCache> = OnceLock::new();
static LEGACY_REED_SOLOMON_CACHE: OnceLock<LegacyReedSolomonCache> = OnceLock::new();
/// Errors returned when constructing an [`Erasure`] codec.
#[derive(Debug, thiserror::Error)]
@@ -101,43 +147,61 @@ pub fn calc_shard_size_legacy(block_size: usize, data_shards: usize) -> usize {
struct LegacyReedSolomonEncoder {
data_shards: usize,
parity_shards: usize,
encoder_cache: std::sync::RwLock<Option<reed_solomon_simd::ReedSolomonEncoder>>,
decoder_cache: std::sync::RwLock<Option<reed_solomon_simd::ReedSolomonDecoder>>,
}
impl Clone for LegacyReedSolomonEncoder {
fn clone(&self) -> Self {
Self {
data_shards: self.data_shards,
parity_shards: self.parity_shards,
encoder_cache: std::sync::RwLock::new(None),
decoder_cache: std::sync::RwLock::new(None),
}
}
cache_workspaces: bool,
encoder_cache: RwLock<Option<reed_solomon_simd::ReedSolomonEncoder>>,
decoder_cache: RwLock<Option<reed_solomon_simd::ReedSolomonDecoder>>,
}
impl LegacyReedSolomonEncoder {
fn new(_data_shards: usize, _parity_shards: usize) -> io::Result<Self> {
fn new(data_shards: usize, parity_shards: usize) -> io::Result<Self> {
Self::with_workspace_cache(data_shards, parity_shards, false)
}
fn with_workspace_cache(data_shards: usize, parity_shards: usize, cache_workspaces: bool) -> io::Result<Self> {
Ok(Self {
data_shards: _data_shards,
parity_shards: _parity_shards,
encoder_cache: std::sync::RwLock::new(None),
decoder_cache: std::sync::RwLock::new(None),
data_shards,
parity_shards,
cache_workspaces,
encoder_cache: RwLock::new(None),
decoder_cache: RwLock::new(None),
})
}
fn logical_shard_bytes_upper_bound(&self, shard_len: usize) -> Option<usize> {
let aligned_shard_len = shard_len.checked_add(63)?.checked_div(64)?.checked_mul(64)?;
let high_rate_decoder_work_count = self
.parity_shards
.checked_next_power_of_two()?
.checked_add(self.data_shards)?
.checked_next_power_of_two()?;
let low_rate_decoder_work_count = self
.data_shards
.checked_next_power_of_two()?
.checked_add(self.parity_shards)?
.checked_next_power_of_two()?;
aligned_shard_len.checked_mul(high_rate_decoder_work_count.max(low_rate_decoder_work_count))
}
fn should_cache_workspace(&self, shard_len: usize) -> bool {
self.cache_workspaces
&& self
.logical_shard_bytes_upper_bound(shard_len)
.is_some_and(|bytes| bytes <= LEGACY_REED_SOLOMON_CACHE_MAX_LOGICAL_SHARD_BYTES_PER_WORKSPACE)
}
fn encode(&self, shards: SmallVec<[&mut [u8]; 16]>) -> io::Result<()> {
let mut shards_vec: Vec<&mut [u8]> = shards.into_vec();
if shards_vec.is_empty() {
return Ok(());
}
let shard_len = shards_vec[0].len();
let cached_encoder = self
.encoder_cache
.write()
.map_err(|_| io::Error::other("Failed to acquire encoder cache lock"))?
.take();
let mut encoder = {
let mut cache_guard = self
.encoder_cache
.write()
.map_err(|_| io::Error::other("Failed to acquire encoder cache lock"))?;
match cache_guard.take() {
match cached_encoder {
Some(mut cached) => {
if cached.reset(self.data_shards, self.parity_shards, shard_len).is_err() {
reed_solomon_simd::ReedSolomonEncoder::new(self.data_shards, self.parity_shards, shard_len)
@@ -164,10 +228,15 @@ impl LegacyReedSolomonEncoder {
}
}
drop(result);
*self
.encoder_cache
.write()
.map_err(|_| io::Error::other("Failed to return encoder to cache"))? = Some(encoder);
if self.should_cache_workspace(shard_len) {
let mut cache = self
.encoder_cache
.write()
.map_err(|_| io::Error::other("Failed to return encoder to cache"))?;
if cache.is_none() {
*cache = Some(encoder);
}
}
Ok(())
}
@@ -181,13 +250,13 @@ impl LegacyReedSolomonEncoder {
.find_map(|s| s.as_ref().map(|v| v.len()))
.ok_or_else(|| io::Error::other("No valid shards found for reconstruction"))?;
let cached_decoder = self
.decoder_cache
.write()
.map_err(|_| io::Error::other("Failed to acquire decoder cache lock"))?
.take();
let mut decoder = {
let mut cache_guard = self
.decoder_cache
.write()
.map_err(|_| io::Error::other("Failed to acquire decoder cache lock"))?;
match cache_guard.take() {
match cached_decoder {
Some(mut cached_decoder) => {
if let Err(e) = cached_decoder.reset(self.data_shards, self.parity_shards, shard_len) {
warn!("Failed to reset SIMD decoder: {:?}, creating new one", e);
@@ -234,10 +303,15 @@ impl LegacyReedSolomonEncoder {
drop(result);
*self
.decoder_cache
.write()
.map_err(|_| io::Error::other("Failed to return decoder to cache"))? = Some(decoder);
if self.should_cache_workspace(shard_len) {
let mut cache = self
.decoder_cache
.write()
.map_err(|_| io::Error::other("Failed to return decoder to cache"))?;
if cache.is_none() {
*cache = Some(decoder);
}
}
Ok(())
}
@@ -395,6 +469,39 @@ fn cached_modern_reed_solomon(data_shards: usize, parity_shards: usize) -> Resul
Ok(encoder)
}
fn cached_legacy_reed_solomon(data_shards: usize, parity_shards: usize) -> io::Result<Arc<LegacyReedSolomonEncoder>> {
let cache = LEGACY_REED_SOLOMON_CACHE.get_or_init(|| RwLock::new(HashMap::new()));
cached_legacy_reed_solomon_in(cache, data_shards, parity_shards)
}
fn cached_legacy_reed_solomon_in(
cache: &LegacyReedSolomonCache,
data_shards: usize,
parity_shards: usize,
) -> io::Result<Arc<LegacyReedSolomonEncoder>> {
let key = (data_shards, parity_shards);
if let Some(encoder) = cache
.read()
.unwrap_or_else(|poisoned| poisoned.into_inner())
.get(&key)
.cloned()
{
return Ok(encoder);
}
let mut cache = cache.write().unwrap_or_else(|poisoned| poisoned.into_inner());
if let Some(existing) = cache.get(&key) {
return Ok(Arc::clone(existing));
}
if cache.len() < LEGACY_REED_SOLOMON_CACHE_MAX_ENTRIES {
let encoder = Arc::new(LegacyReedSolomonEncoder::with_workspace_cache(data_shards, parity_shards, true)?);
cache.insert(key, Arc::clone(&encoder));
return Ok(encoder);
}
drop(cache);
Ok(Arc::new(LegacyReedSolomonEncoder::new(data_shards, parity_shards)?))
}
fn encode_parity_shards<F>(shards: &mut [Option<Vec<u8>>], data_shards: usize, parity_shards: usize, encode: F) -> io::Result<()>
where
F: FnOnce(SmallVec<[&mut [u8]; 16]>) -> io::Result<()>,
@@ -511,7 +618,7 @@ pub struct Erasure {
pub data_shards: usize,
pub parity_shards: usize,
encoder: Option<ReedSolomonEncoder>,
legacy_encoder: Option<LegacyReedSolomonEncoder>,
legacy_encoder: Option<Arc<LegacyReedSolomonEncoder>>,
pub block_size: usize,
uses_legacy: bool,
_id: Uuid,
@@ -647,7 +754,7 @@ impl Erasure {
let legacy_encoder = if uses_legacy && parity_shards > 0 {
Some(
LegacyReedSolomonEncoder::new(data_shards, parity_shards)
cached_legacy_reed_solomon(data_shards, parity_shards)
.map_err(|source| ErasureConstructionError::LegacyEncoder { source })?,
)
} else {
@@ -675,106 +782,48 @@ impl Erasure {
#[tracing::instrument(level = "debug", skip_all, fields(data_len=data.len()))]
#[hotpath::measure(impl_type = "Erasure")]
pub fn encode_data(&self, data: &[u8]) -> io::Result<Vec<Bytes>> {
let shard_size_fn = if self.uses_legacy {
calc_shard_size_legacy
} else {
calc_shard_size
};
let per_shard_size = shard_size_fn(data.len(), self.data_shards);
if per_shard_size == 0 {
return Ok(vec![Bytes::new(); self.total_shard_count()]);
}
let need_total_size = per_shard_size * self.total_shard_count();
self.encode_data_block_inner(data)
.map(|block| block.into_shards(self.total_shard_count()))
}
let mut data_buffer = BytesMut::with_capacity(need_total_size);
#[tracing::instrument(level = "debug", skip_all, fields(data_len=data.len()))]
#[hotpath::measure(label = "Erasure::encode_data", impl_type = "Erasure")]
pub(crate) fn encode_data_block(&self, data: &[u8]) -> io::Result<EncodedBlock> {
self.encode_data_block_inner(data)
}
fn encode_data_block_inner(&self, data: &[u8]) -> io::Result<EncodedBlock> {
let mut data_buffer = BytesMut::with_capacity(self.encoded_capacity_for_data_len(data.len()));
data_buffer.extend_from_slice(data);
data_buffer.resize(need_total_size, 0u8);
{
let data_slices: SmallVec<[&mut [u8]; 16]> = data_buffer.chunks_exact_mut(per_shard_size).collect();
if self.parity_shards > 0 {
if self.uses_legacy {
if let Some(encoder) = self.legacy_encoder.as_ref() {
encoder.encode(data_slices)?;
} else {
warn!("parity_shards > 0, uses_legacy but legacy_encoder is None");
}
} else if let Some(encoder) = self.encoder.as_ref() {
encoder.encode(data_slices)?;
} else {
warn!("parity_shards > 0, but encoder is None");
}
}
}
// Zero-copy split, all shards reference data_buffer
let mut data_buffer = data_buffer.freeze();
let mut shards = Vec::with_capacity(self.total_shard_count());
for _ in 0..self.total_shard_count() {
let shard = data_buffer.split_to(per_shard_size);
shards.push(shard);
}
Ok(shards)
self.encode_buffer(data_buffer, data.len())
}
/// Encode owned data, avoiding a copy when the caller already has a heap buffer.
/// Falls back to copying into a new buffer if zero-copy conversion fails.
#[hotpath::measure(impl_type = "Erasure")]
pub fn encode_data_owned(&self, data: Vec<u8>) -> io::Result<Vec<Bytes>> {
let shard_size_fn = if self.uses_legacy {
calc_shard_size_legacy
} else {
calc_shard_size
};
let per_shard_size = shard_size_fn(data.len(), self.data_shards);
if per_shard_size == 0 {
return Ok(vec![Bytes::new(); self.total_shard_count()]);
}
let need_total_size = per_shard_size * self.total_shard_count();
self.encode_data_owned_block_inner(data)
.map(|block| block.into_shards(self.total_shard_count()))
}
#[hotpath::measure(label = "Erasure::encode_data_owned", impl_type = "Erasure")]
pub(crate) fn encode_data_owned_block(&self, data: Vec<u8>) -> io::Result<EncodedBlock> {
self.encode_data_owned_block_inner(data)
}
fn encode_data_owned_block_inner(&self, data: Vec<u8>) -> io::Result<EncodedBlock> {
let data_len = data.len();
// Try zero-copy: Vec<u8> -> Bytes -> BytesMut (succeeds when refcount == 1)
let mut data_buffer = match Bytes::from(data).try_into_mut() {
Ok(mut bm) => {
bm.resize(need_total_size, 0u8);
bm
}
let data_buffer = match Bytes::from(data).try_into_mut() {
Ok(data_buffer) => data_buffer,
Err(b) => {
// Rare path: refcount != 1, fall back to copy
let mut bm = BytesMut::with_capacity(need_total_size);
bm.extend_from_slice(&b);
bm.resize(need_total_size, 0u8);
bm
let mut data_buffer = BytesMut::with_capacity(self.encoded_capacity_for_data_len(data_len));
data_buffer.extend_from_slice(&b);
data_buffer
}
};
{
let data_slices: SmallVec<[&mut [u8]; 16]> = data_buffer.chunks_exact_mut(per_shard_size).collect();
if self.parity_shards > 0 {
if self.uses_legacy {
if let Some(encoder) = self.legacy_encoder.as_ref() {
encoder.encode(data_slices)?;
} else {
warn!("parity_shards > 0, uses_legacy but legacy_encoder is None");
}
} else if let Some(encoder) = self.encoder.as_ref() {
encoder.encode(data_slices)?;
} else {
warn!("parity_shards > 0, but encoder is None");
}
}
}
let mut data_buffer = data_buffer.freeze();
let mut shards = Vec::with_capacity(self.total_shard_count());
for _ in 0..self.total_shard_count() {
let shard = data_buffer.split_to(per_shard_size);
shards.push(shard);
}
Ok(shards)
self.encode_buffer(data_buffer, data_len)
}
/// Encode data from an owned `BytesMut` buffer, avoiding the initial copy
@@ -786,7 +835,17 @@ impl Erasure {
/// `data_len <= block_size` — both shard-size formulas are monotone in
/// `data_len` — so this function never reallocates the buffer.
#[hotpath::measure(impl_type = "Erasure")]
pub fn encode_data_bytes_mut(&self, mut data_buffer: BytesMut, data_len: usize) -> io::Result<Vec<Bytes>> {
pub fn encode_data_bytes_mut(&self, data_buffer: BytesMut, data_len: usize) -> io::Result<Vec<Bytes>> {
self.encode_buffer(data_buffer, data_len)
.map(|block| block.into_shards(self.total_shard_count()))
}
#[hotpath::measure(label = "Erasure::encode_data_bytes_mut", impl_type = "Erasure")]
pub(crate) fn encode_data_bytes_mut_block(&self, data_buffer: BytesMut, data_len: usize) -> io::Result<EncodedBlock> {
self.encode_buffer(data_buffer, data_len)
}
fn encode_buffer(&self, mut data_buffer: BytesMut, data_len: usize) -> io::Result<EncodedBlock> {
let shard_size_fn = if self.uses_legacy {
calc_shard_size_legacy
} else {
@@ -794,7 +853,7 @@ impl Erasure {
};
let per_shard_size = shard_size_fn(data_len, self.data_shards);
if per_shard_size == 0 {
return Ok(vec![Bytes::new(); self.total_shard_count()]);
return Ok(EncodedBlock::empty());
}
let need_total_size = per_shard_size * self.total_shard_count();
@@ -821,14 +880,10 @@ impl Erasure {
}
}
let mut data_buffer = data_buffer.freeze();
let mut shards = Vec::with_capacity(self.total_shard_count());
for _ in 0..self.total_shard_count() {
let shard = data_buffer.split_to(per_shard_size);
shards.push(shard);
}
Ok(shards)
Ok(EncodedBlock {
data: data_buffer.freeze(),
shard_size: per_shard_size,
})
}
/// Decode and reconstruct missing data shards in-place.
@@ -1055,6 +1110,10 @@ impl Erasure {
///
/// # Errors
/// Returns error if reading from reader fails or if callback returns error
#[allow(
dead_code,
reason = "callback encode path exercised only by this file's tests (backlog#1823)"
)]
pub(crate) async fn encode_stream_callback_async<F, Fut, E, R>(
self: std::sync::Arc<Self>,
reader: &mut R,
@@ -1417,7 +1476,7 @@ mod tests {
assert_eq!(cloned.block_size, legacy.block_size);
assert!(cloned.uses_legacy);
let data = b"legacy clone should keep independent SIMD caches";
let data = b"legacy clone should preserve SIMD codec behavior";
let encoded = cloned.encode_data(data).expect("legacy clone should encode");
let mut shards = optional_shards(&encoded);
shards[0] = None;
@@ -1425,6 +1484,93 @@ mod tests {
assert_eq!(recover_data(&shards, cloned.data_shards, data.len()), data);
}
#[test]
fn legacy_codecs_share_process_cache_across_erasure_instances() {
let first = Erasure::new_with_options(6, 3, 64, true)
.legacy_encoder
.expect("legacy codec should be initialized");
let second = Erasure::new_with_options(6, 3, 128, true)
.legacy_encoder
.expect("same legacy shard layout should be initialized");
assert!(Arc::ptr_eq(&first, &second));
}
#[test]
fn legacy_workspace_cache_rejects_oversize_buffers_and_isolates_layouts() {
let four_plus_two = Erasure::new_with_options(4, 2, 64, true)
.legacy_encoder
.expect("legacy codec should be initialized");
let four_plus_one = Erasure::new_with_options(4, 1, 64, true)
.legacy_encoder
.expect("distinct parity layout should be initialized");
let three_plus_two = Erasure::new_with_options(3, 2, 64, true)
.legacy_encoder
.expect("distinct data layout should be initialized");
assert!(!Arc::ptr_eq(&four_plus_two, &four_plus_one));
assert!(!Arc::ptr_eq(&four_plus_two, &three_plus_two));
assert_eq!(four_plus_two.logical_shard_bytes_upper_bound(64 * 1024), Some(512 * 1024));
assert!(four_plus_two.should_cache_workspace(64 * 1024));
assert!(!four_plus_two.should_cache_workspace(64 * 1024 + 1));
let nine_plus_seven =
LegacyReedSolomonEncoder::with_workspace_cache(9, 7, true).expect("9+7 legacy codec should construct");
assert_eq!(nine_plus_seven.logical_shard_bytes_upper_bound(16 * 1024), Some(512 * 1024));
assert!(nine_plus_seven.should_cache_workspace(16 * 1024));
assert!(!nine_plus_seven.should_cache_workspace(16 * 1024 + 1));
let uncached = LegacyReedSolomonEncoder::new(4, 2).expect("uncached legacy codec should construct");
assert!(!uncached.should_cache_workspace(64));
}
#[test]
fn saturated_legacy_codec_cache_does_not_retain_more_workspaces() {
let cache = RwLock::new(HashMap::new());
for parity_shards in 1..=LEGACY_REED_SOLOMON_CACHE_MAX_ENTRIES {
let cached =
cached_legacy_reed_solomon_in(&cache, 32, parity_shards).expect("cacheable legacy codec should construct");
assert!(cached.cache_workspaces);
}
let uncached =
cached_legacy_reed_solomon_in(&cache, 31, 1).expect("uncached legacy codec should construct after saturation");
assert!(!uncached.cache_workspaces);
assert_eq!(
cache.read().expect("cache lock should remain healthy").len(),
LEGACY_REED_SOLOMON_CACHE_MAX_ENTRIES
);
}
#[test]
fn concurrent_legacy_codecs_preserve_byte_exact_results() {
let barrier = Arc::new(std::sync::Barrier::new(2));
let payloads = [vec![0x35; 257], vec![0xca; 1025]];
std::thread::scope(|scope| {
let handles = payloads.each_ref().map(|payload| {
let barrier = Arc::clone(&barrier);
scope.spawn(move || {
let erasure = Erasure::new_with_options(6, 3, 2048, true);
barrier.wait();
let encoded = erasure.encode_data(payload).expect("concurrent legacy encode should succeed");
barrier.wait();
let mut shards = optional_shards(&encoded);
shards[0] = None;
erasure
.decode_data(&mut shards)
.expect("concurrent legacy decode should reconstruct the missing shard");
recover_data(&shards, erasure.data_shards, payload.len())
})
});
for (handle, payload) in handles.into_iter().zip(payloads.iter()) {
assert_eq!(handle.join().expect("concurrent legacy codec worker should not panic"), *payload);
}
});
}
#[test]
fn legacy_verify_reports_invalid_empty_valid_and_corrupt_parity_sets() {
let legacy = LegacyReedSolomonEncoder::new(2, 2).expect("legacy encoder should construct");
@@ -1498,10 +1644,16 @@ mod tests {
fn encode_data_owned_matches_borrowed_path() {
for uses_legacy in [false, true] {
let erasure = Erasure::new_with_options(4, 2, 64, uses_legacy);
assert_owned_encode_matches_borrowed(&erasure, Vec::new());
assert_owned_encode_matches_borrowed(&erasure, b"small payload".to_vec());
assert_owned_encode_matches_borrowed(&erasure, (0_u8..37).collect());
for data in [
Vec::new(),
vec![0xA5; 1],
b"small payload".to_vec(),
(0_u8..37).collect(),
vec![0xA5; erasure.block_size - 1],
vec![0x5A; erasure.block_size],
] {
assert_owned_encode_matches_borrowed(&erasure, data);
}
}
}
@@ -1547,6 +1699,52 @@ mod tests {
}
}
#[test]
fn streaming_encoded_block_uses_one_contiguous_backing_buffer() {
for uses_legacy in [false, true] {
let erasure = Erasure::new_with_options(8, 8, 64, uses_legacy);
for data_len in [0, 1, 63, 64] {
let data = (0..data_len).map(|i| i as u8).collect::<Vec<_>>();
let expected = erasure.encode_data(&data).expect("public encode should succeed");
let borrowed = erasure
.encode_data_block(&data)
.expect("borrowed streaming encode should succeed");
let owned = erasure
.encode_data_owned_block(data.clone())
.expect("owned streaming encode should succeed");
let bytes_mut = erasure
.encode_data_bytes_mut_block(BytesMut::from(&data[..]), data.len())
.expect("BytesMut streaming encode should succeed");
assert_eq!(borrowed.queued_bytes(), owned.queued_bytes());
assert_eq!(borrowed.queued_bytes(), bytes_mut.queued_bytes());
if data_len == 0 {
assert!(expected.iter().all(Bytes::is_empty));
assert!(borrowed.is_empty());
assert!(owned.is_empty());
assert!(bytes_mut.is_empty());
continue;
}
assert!(borrowed.shards().eq(expected.iter().map(Bytes::as_ref)));
assert!(owned.shards().eq(expected.iter().map(Bytes::as_ref)));
assert!(bytes_mut.shards().eq(expected.iter().map(Bytes::as_ref)));
assert_eq!(borrowed.shards().len(), 16);
let first = borrowed.shards().next().expect("encoded block should have shards").as_ptr();
for (index, shard) in borrowed.shards().enumerate() {
assert_eq!(shard.as_ptr(), first.wrapping_add(index * shard.len()));
}
}
}
assert_eq!(
std::mem::size_of::<EncodedBlock>(),
std::mem::size_of::<Bytes>() + std::mem::size_of::<usize>(),
"queue entries must contain one backing buffer handle, not per-shard handles"
);
}
/// HP-10 capacity invariant: both shard-size formulas are monotone in `data_len`,
/// so pre-reserving `shard_size(block_size) * total_shard_count` covers the
/// `need_total_size` of every block-or-smaller payload and the ingest buffer
-1
View File
@@ -13,7 +13,6 @@
// limitations under the License.
// #730: erasure codec migration keeps staged streaming decode paths in this module.
#![allow(dead_code)]
pub(crate) mod codec;
pub(crate) mod coding;
+3 -42
View File
@@ -13,13 +13,12 @@
// limitations under the License.
// #730: error taxonomy still exposes compatibility variants while callers move to contracts.
#![allow(dead_code)]
use crate::bucket::error::BucketMetadataError;
use crate::disk::error::DiskError;
use crate::storage_api_contracts::{error::StorageErrorCode, range::HTTPRangeError};
use rustfs_utils::path::decode_dir_object;
use s3s::{S3Error, S3ErrorCode};
use s3s::S3ErrorCode;
pub type Error = StorageError;
pub type Result<T> = core::result::Result<T, Error>;
@@ -902,6 +901,7 @@ pub fn is_err_decommission_running(err: &Error) -> bool {
matches!(err, &StorageError::DecommissionAlreadyRunning)
}
#[allow(dead_code, reason = "predicate asserted by this file's tests (backlog#1823)")]
pub fn is_err_rebalance_running(err: &Error) -> bool {
matches!(err, &StorageError::RebalanceAlreadyRunning)
}
@@ -910,14 +910,11 @@ pub fn is_err_operation_canceled(err: &Error) -> bool {
matches!(err, &StorageError::OperationCanceled)
}
#[allow(dead_code, reason = "predicate asserted by this file's tests (backlog#1823)")]
pub fn is_err_not_initialized(err: &Error) -> bool {
err.to_string().contains("errServerNotInitialized") || err.to_string().contains("ServerNotInitialized")
}
pub fn is_err_io(err: &Error) -> bool {
matches!(err, &StorageError::Io(_))
}
/// Strict "not found" predicate that only matches genuine object/version/volume
/// absence errors: `FileNotFound`/`VolumeNotFound`/`FileVersionNotFound`/
/// `ObjectNotFound`/`VersionNotFound`.
@@ -1078,21 +1075,9 @@ pub struct GenericError {
#[derive(Debug, thiserror::Error, PartialEq, Eq)]
pub enum ObjectApiError {
#[error("Operation timed out")]
OperationTimedOut,
#[error("etag of the object has changed")]
InvalidETag,
#[error("BackendDown")]
BackendDown(String),
#[error("Unsupported headers in Metadata")]
UnsupportedMetadata,
#[error("Method not allowed: {}/{}", .0.bucket, .0.object)]
MethodNotAllowed(GenericError),
#[error("The operation is not valid for the current state of the object {}/{}({})", .0.bucket, .0.object, .0.version_id)]
InvalidObjectState(GenericError),
}
@@ -1175,30 +1160,6 @@ pub fn error_resp_to_object_err(err: ErrorResponse, params: Vec<&str>) -> std::i
err
}
pub fn storage_to_object_err(err: Error, params: Vec<&str>) -> S3Error {
let storage_err = &err;
let mut bucket: String = "".to_string();
let mut object: String = "".to_string();
if !params.is_empty() {
bucket = params[0].to_string();
}
if params.len() >= 2 {
object = decode_dir_object(params[1]);
}
match storage_err {
StorageError::MethodNotAllowed => S3Error::with_message(
S3ErrorCode::MethodNotAllowed,
ObjectApiError::MethodNotAllowed(GenericError {
bucket,
object,
..Default::default()
})
.to_string(),
),
_ => s3s::S3Error::with_message(S3ErrorCode::Custom("err".into()), err.to_string()),
}
}
#[cfg(test)]
mod tests {
use super::*;
-2
View File
@@ -13,8 +13,6 @@
// limitations under the License.
// #730: event target types are retained for notification owner migration.
#![allow(dead_code)]
pub mod name;
pub mod targetid;
pub mod targetlist;
-25
View File
@@ -1,25 +0,0 @@
#![allow(clippy::all)]
// Copyright 2024 RustFS Team
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at
//
// http://www.apache.org/licenses/LICENSE-2.0
//
// Unless required by applicable law or agreed to in writing, software
// distributed under the License is distributed on an "AS IS" BASIS,
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
// See the License for the specific language governing permissions and
// limitations under the License.
pub struct TargetID {
id: String,
name: String,
}
impl TargetID {
fn to_string(&self) -> String {
format!("{}:{}", self.id, self.name)
}
}
+5 -18
View File
@@ -12,18 +12,16 @@
// See the License for the specific language governing permissions and
// limitations under the License.
use crate::event::targetid::TargetID;
use std::sync::atomic::AtomicI64;
/// Placeholder notification target list held by `EventNotifier`.
///
/// The working notification stack lives in `rustfs-notify` / `rustfs-targets`;
/// this type never grew past its counter. `total_events` is read by the
/// notifier's log line but nothing increments it, so that field reports zero.
#[derive(Default)]
pub struct TargetList {
pub current_send_calls: AtomicI64,
pub total_events: AtomicI64,
pub events_skipped: AtomicI64,
pub events_errors_total: AtomicI64,
//pub targets: HashMap<TargetID, Target>,
//pub queue: AsyncEvent,
//pub targetStats: HashMap<TargetID, TargetStat>,
}
impl TargetList {
@@ -31,14 +29,3 @@ impl TargetList {
TargetList::default()
}
}
struct TargetStat {
current_send_calls: i64,
total_events: i64,
failed_events: i64,
}
struct TargetIDResult {
id: TargetID,
err: std::io::Error,
}
+128 -3
View File
@@ -22,12 +22,13 @@ use crate::diagnostics::get::{
#[cfg(feature = "hotpath")]
use crate::disk::FileWriter;
use crate::disk::{self, DiskAPI as _, DiskStore, FileReader, MmapCopyStageMetrics, error::DiskError};
use crate::erasure::coding::{BitrotReader, BitrotWriterWrapper, CustomWriter};
use crate::erasure::coding::{BitrotReader, BitrotWriterWrapper, CustomWriter, ShardChunkRead};
use bytes::Bytes;
use rustfs_config::{
DEFAULT_OBJECT_MMAP_READ_ENABLE, DEFAULT_OBJECT_MMAP_READ_MAX_LENGTH, ENV_OBJECT_MMAP_READ_ENABLE,
ENV_OBJECT_MMAP_READ_MAX_LENGTH, ENV_OBJECT_ZERO_COPY_ENABLE,
};
use rustfs_rio::ChunkReaderBox;
use rustfs_utils::HashAlgorithm;
use std::future::Future;
use std::io::{self, Cursor};
@@ -51,13 +52,25 @@ tokio::task_local! {
/// (rustfs/backlog#1159). Everything else is a stream and keeps the old path.
pub enum ShardReader {
InMemory(Cursor<Bytes>),
Chunked(ChunkReaderBox),
Stream(Box<dyn AsyncRead + Send + Sync + Unpin>),
}
#[cfg(test)]
impl ShardReader {
pub(crate) fn inline_bytes(&self) -> Option<&Bytes> {
match self {
Self::InMemory(cursor) => Some(cursor.get_ref()),
Self::Chunked(_) | Self::Stream(_) => None,
}
}
}
impl AsyncRead for ShardReader {
fn poll_read(self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut tokio::io::ReadBuf<'_>) -> Poll<std::io::Result<()>> {
match self.get_mut() {
Self::InMemory(cursor) => Pin::new(cursor).poll_read(cx, buf),
Self::Chunked(reader) => Pin::new(&mut **reader).poll_read(cx, buf),
Self::Stream(reader) => Pin::new(reader).poll_read(cx, buf),
}
}
@@ -67,7 +80,19 @@ impl crate::erasure::coding::ShardSource for ShardReader {
fn try_take_block(&mut self, n: usize) -> Option<Bytes> {
match self {
Self::InMemory(cursor) => cursor.try_take_block(n),
Self::Stream(_) => None,
Self::Chunked(_) | Self::Stream(_) => None,
}
}
fn poll_read_chunk(self: Pin<&mut Self>, cx: &mut Context<'_>, max: usize) -> Poll<io::Result<ShardChunkRead>> {
let Self::Chunked(reader) = self.get_mut() else {
return Poll::Ready(Ok(ShardChunkRead::Unsupported));
};
match Pin::new(&mut **reader).poll_read_chunk(cx, max) {
Poll::Ready(Ok(Some(chunk))) => Poll::Ready(Ok(ShardChunkRead::Chunk(chunk))),
Poll::Ready(Ok(None)) => Poll::Ready(Ok(ShardChunkRead::Eof)),
Poll::Ready(Err(err)) => Poll::Ready(Err(err)),
Poll::Pending => Poll::Pending,
}
}
}
@@ -345,6 +370,17 @@ async fn open_disk_reader(
let metrics_path = metrics_path.filter(|_| rustfs_io_metrics::get_stage_metrics_enabled());
let stage_metrics_enabled = metrics_path.is_some();
// Preserve HTTP body ownership only on healthy remote reads. Instrumented
// and local paths retain their existing AsyncRead wrappers.
if use_mmap_read
&& !disk.is_local()
&& !stage_metrics_enabled
&& !cfg!(feature = "hotpath")
&& let Some(reader) = disk.read_file_stream_chunks(bucket, path, offset, length).await?
{
return Ok(ShardReader::Chunked(reader));
}
// Mmap-copy materializes the whole `offset..offset+length` range as one
// owned allocation before any byte is served, and GET/heal shard reads
// request the entire part span in one call. Over-cap reads (e.g. a huge
@@ -620,7 +656,7 @@ pub async fn create_bitrot_reader_from_bytes(
}
#[allow(clippy::too_many_arguments)]
async fn create_bitrot_reader_from_bytes_with_stage_metrics(
pub(crate) async fn create_bitrot_reader_from_bytes_with_stage_metrics(
inline_data: Option<Bytes>,
disk: Option<&DiskStore>,
bucket: &str,
@@ -780,6 +816,50 @@ pub async fn create_bitrot_writer(
#[cfg(test)]
mod tests {
use super::*;
use rustfs_rio::ChunkReader;
use std::collections::VecDeque;
struct TestChunkReader {
chunks: VecDeque<Bytes>,
}
impl TestChunkReader {
fn new(bytes: Bytes, fragment_sizes: &[usize]) -> Self {
let mut chunks = VecDeque::new();
let mut offset = 0;
for &size in fragment_sizes {
let end = (offset + size).min(bytes.len());
if offset < end {
chunks.push_back(bytes.slice(offset..end));
}
offset = end;
}
if offset < bytes.len() {
chunks.push_back(bytes.slice(offset..));
}
Self { chunks }
}
}
impl AsyncRead for TestChunkReader {
fn poll_read(self: Pin<&mut Self>, _cx: &mut Context<'_>, _buf: &mut ReadBuf<'_>) -> Poll<io::Result<()>> {
Poll::Ready(Err(io::Error::other("test chunk reader must use chunk handoff")))
}
}
impl ChunkReader for TestChunkReader {
fn poll_read_chunk(mut self: Pin<&mut Self>, _cx: &mut Context<'_>, max: usize) -> Poll<io::Result<Option<Bytes>>> {
let Some(mut chunk) = self.chunks.pop_front() else {
return Poll::Ready(Ok(None));
};
let take = chunk.len().min(max);
if take < chunk.len() {
self.chunks.push_front(chunk.split_off(take));
}
chunk.truncate(take);
Poll::Ready(Ok(Some(chunk)))
}
}
#[cfg(feature = "hotpath")]
use crate::cluster::rpc::RemoteDisk;
@@ -1669,4 +1749,49 @@ mod tests {
println!("error: {error:?}");
assert_eq!(error, DiskError::DiskNotFound);
}
#[tokio::test]
async fn shard_reader_chunked_path_verifies_fragmented_remote_block() {
const SHARD_SIZE: usize = 1024;
let algo = HashAlgorithm::HighwayHash256S;
let data = vec![42u8; SHARD_SIZE];
let mut encoded = Vec::new();
crate::erasure::coding::BitrotWriter::new(&mut encoded, SHARD_SIZE, algo.clone())
.write(&data)
.await
.expect("test shard should encode");
let source = TestChunkReader::new(Bytes::from(encoded), &[3, 7, 17, 31]);
let mut reader = BitrotReader::new(ShardReader::Chunked(Box::new(source)), SHARD_SIZE, algo, false);
let mut output = Vec::with_capacity(SHARD_SIZE);
reader
.read_appending(&mut output, SHARD_SIZE)
.await
.expect("fragmented remote shard should verify");
assert_eq!(output, data);
}
#[tokio::test]
async fn shard_reader_chunked_path_handles_more_than_one_poll_budget() {
const SHARD_SIZE: usize = 1024;
let algo = HashAlgorithm::HighwayHash256S;
let data = vec![42u8; SHARD_SIZE];
let mut encoded = Vec::new();
crate::erasure::coding::BitrotWriter::new(&mut encoded, SHARD_SIZE, algo.clone())
.write(&data)
.await
.expect("test shard should encode");
let fragment_sizes = vec![1; encoded.len()];
let source = TestChunkReader::new(Bytes::from(encoded), &fragment_sizes);
let mut reader = BitrotReader::new(ShardReader::Chunked(Box::new(source)), SHARD_SIZE, algo, false);
let mut output = Vec::with_capacity(SHARD_SIZE);
reader
.read_appending(&mut output, SHARD_SIZE)
.await
.expect("fragmented remote shard should verify after multiple polls");
assert_eq!(output, data);
}
}
-1
View File
@@ -13,7 +13,6 @@
// limitations under the License.
// #730: I/O backend selection keeps test-only and staged rio helpers scoped here.
#![allow(dead_code)]
pub(crate) mod bitrot;
pub(crate) mod compress;
+16 -11
View File
@@ -25,9 +25,20 @@ use tokio::io::AsyncRead;
#[cfg(feature = "rio-v2")]
const MINIO_S2_COMPRESSION_SCHEME: &str = "klauspost/compress/s2";
// The S2 padding multiple rio-v2 pads compressed streams to before
// encryption. Only the padding test asserts it today, so the lib target sees
// it as unused (backlog#1823).
#[cfg(feature = "rio-v2")]
#[allow(dead_code, reason = "on-disk contract asserted by the rio-v2 padding test (backlog#1823)")]
const ENCRYPTED_S2_PADDING_MULTIPLE: usize = 256;
/// Which rio implementation this build compiled in. Only the feature-seam
/// guard test in lib.rs reads it, so the lib target sees it as unused
/// (backlog#1823).
#[allow(
dead_code,
reason = "asserted by the rio backend feature-seam test in lib.rs (backlog#1823)"
)]
pub const fn backend_name() -> &'static str {
#[cfg(feature = "rio-v2")]
{
@@ -53,17 +64,6 @@ pub fn compression_metadata_value(algorithm: CompressionAlgorithm) -> String {
}
}
pub fn compression_scheme_to_algorithm(scheme: &str) -> std::io::Result<CompressionAlgorithm> {
#[cfg(feature = "rio-v2")]
if scheme.eq_ignore_ascii_case(MINIO_S2_COMPRESSION_SCHEME) {
// rio_v2 currently routes all compressed-object handling through the S2
// reader implementation, so the enum is only a placeholder token here.
return Ok(CompressionAlgorithm::default());
}
CompressionAlgorithm::from_str(scheme)
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ReadCompressionBackend {
Legacy,
@@ -82,6 +82,11 @@ pub fn compression_scheme_to_read_plan(scheme: &str) -> std::io::Result<(Compres
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum ReadEncryptionBackend {
Legacy,
// Never constructed today — every read still selects Legacy — but the
// decrypt paths below carry live match arms for it. This is the rio-v2
// read seam (backlog#1638 / #1835), not dead code: deleting the variant
// would delete those arms with it.
#[allow(dead_code, reason = "rio-v2 read seam; match arms below are live (backlog#1823)")]
V2,
}
+3 -2
View File
@@ -21,7 +21,8 @@ use tracing::debug;
/// Supported set sizes this is used to find the optimal
/// single set size.
const SET_SIZES: [usize; 15] = [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16];
pub(crate) const MAX_ERASURE_SET_DRIVE_COUNT: usize = 16;
const SET_SIZES: [usize; 15] = [2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, MAX_ERASURE_SET_DRIVE_COUNT];
const ENV_RUSTFS_ERASURE_SET_DRIVE_COUNT: &str = "RUSTFS_ERASURE_SET_DRIVE_COUNT";
#[derive(Deserialize, Debug, Default)]
@@ -327,7 +328,7 @@ fn possible_set_counts(set_size: usize) -> Vec<usize> {
/// checks whether given count is a valid set size for erasure coding.
fn is_valid_set_size(count: usize) -> bool {
count >= SET_SIZES[0] && count <= SET_SIZES[SET_SIZES.len() - 1]
count >= SET_SIZES[0] && count <= MAX_ERASURE_SET_DRIVE_COUNT
}
/// Final set size with all the symmetry accounted for.
+10 -9
View File
@@ -209,15 +209,12 @@ impl AsMut<Vec<Endpoints>> for PoolEndpointList {
}
impl PoolEndpointList {
/// creates a list of endpoints per pool, resolves their relevant
/// hostnames and discovers those are local or remote.
async fn create_pool_endpoints(server_addr: &str, disks_layout: &DisksLayout) -> Result<Self> {
Self::create_pool_endpoints_with(server_addr, disks_layout, None, None).await
}
/// Same as [`create_pool_endpoints`] but lets tests inject an explicit
/// startup topology convergence policy and local endpoint host instead of
/// resolving them from the environment.
/// Creates a list of endpoints per pool, resolves their relevant hostnames
/// and discovers whether those are local or remote.
///
/// The policy and host overrides let tests inject an explicit startup
/// topology convergence policy and local endpoint host instead of
/// resolving them from the environment; production passes `None` for both.
async fn create_pool_endpoints_with(
server_addr: &str,
disks_layout: &DisksLayout,
@@ -594,6 +591,10 @@ impl PoolEndpointList {
}
const DNS_RETRY_BASE_DELAY: Duration = Duration::from_millis(500);
#[allow(
dead_code,
reason = "retry-cap bound asserted by this file's dns_retry_delay tests (backlog#1823)"
)]
const DNS_RETRY_MAX_DELAY: Duration = Duration::from_secs(8);
const DNS_RETRY_JITTER_PERCENT: u64 = 20;
/// Minimum spacing between "still retrying" warnings so a long orchestrated
-1
View File
@@ -13,7 +13,6 @@
// limitations under the License.
// #730: set-layout contracts are staged while ECStore ownership boundaries shrink.
#![allow(dead_code)]
//! Static ECStore layout boundaries.
//!
+6
View File
@@ -4,6 +4,7 @@ use std::io::{Error, Result};
use uuid::Uuid;
#[derive(Debug, Clone, PartialEq, Eq)]
#[allow(dead_code, reason = "ESET-001 layout model; exercised by this file's tests (backlog#1823)")]
pub(crate) struct StaticSetLayoutSnapshot {
pub(crate) deployment_id: Uuid,
pub(crate) set_count: usize,
@@ -12,6 +13,7 @@ pub(crate) struct StaticSetLayoutSnapshot {
pub(crate) distribution_algo: DistributionAlgoVersion,
}
#[allow(dead_code, reason = "ESET-001 layout model; exercised by this file's tests (backlog#1823)")]
impl StaticSetLayoutSnapshot {
pub(crate) fn from_format(format: &FormatV3) -> Self {
let disk_ids = format.erasure.sets.clone();
@@ -39,17 +41,20 @@ impl StaticSetLayoutSnapshot {
}
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
#[allow(dead_code, reason = "ESET-001 layout model; exercised by this file's tests (backlog#1823)")]
pub(crate) struct SetDiskPosition {
pub(crate) set_index: usize,
pub(crate) disk_index: usize,
}
#[derive(Debug, Clone, PartialEq, Eq)]
#[allow(dead_code, reason = "ESET-001 layout model; exercised by this file's tests (backlog#1823)")]
pub(crate) struct RuntimeSetLayoutPlan {
pub(crate) sets: Vec<Vec<RuntimeSetDrivePlan>>,
lock_hosts_by_set: Vec<Vec<String>>,
}
#[allow(dead_code, reason = "ESET-001 layout model; exercised by this file's tests (backlog#1823)")]
impl RuntimeSetLayoutPlan {
pub(crate) fn from_endpoint_hosts<S>(set_count: usize, drives_per_set: usize, endpoint_hosts: &[S]) -> Result<Self>
where
@@ -108,6 +113,7 @@ impl RuntimeSetLayoutPlan {
}
#[derive(Debug, Clone, PartialEq, Eq)]
#[allow(dead_code, reason = "ESET-001 layout model; exercised by this file's tests (backlog#1823)")]
pub(crate) struct RuntimeSetDrivePlan {
pub(crate) set_index: usize,
pub(crate) disk_index: usize,
-1
View File
@@ -13,7 +13,6 @@
// limitations under the License.
// #730: object API readers keep staged compatibility paths during facade migration.
#![allow(dead_code)]
use crate::bucket::metadata_sys::get_versioning_config;
use crate::bucket::replication::{
+88 -26
View File
@@ -15,6 +15,7 @@
use super::*;
use crate::io_support::rio::Index;
use std::mem::MaybeUninit;
#[cfg(feature = "rio-v2")]
const DARE_PAYLOAD_SIZE: i64 = 64 * 1024;
@@ -448,10 +449,16 @@ impl GetObjectReader {
}
enum ReadTransform {
Plain {
visible_offset: usize,
visible_length: i64,
},
// Written but never read by production code: the enclosing struct already
// carries the same pair as `storage_offset`/`storage_length`. They survive
// as the read plan's test-visible record — four tests assert them by
// literal pattern (`Plain { visible_offset: 6, visible_length: 4 }`), which
// rustc does not count as a read.
#[allow(
dead_code,
reason = "asserted by literal pattern in this file's read-plan tests (backlog#1823)"
)]
Plain { visible_offset: usize, visible_length: i64 },
Compressed {
algorithm: CompressionAlgorithm,
backend: crate::io_support::rio::ReadCompressionBackend,
@@ -922,7 +929,7 @@ struct SkipReader<R> {
inner: R,
bytes_to_skip: usize,
bytes_skipped: usize,
scratch: Vec<u8>,
scratch: Box<[MaybeUninit<u8>]>,
}
impl<R: AsyncRead + Unpin + Send + Sync> SkipReader<R> {
@@ -931,7 +938,7 @@ impl<R: AsyncRead + Unpin + Send + Sync> SkipReader<R> {
inner,
bytes_to_skip,
bytes_skipped: 0,
scratch: vec![0u8; 8192],
scratch: Box::<[u8]>::new_uninit_slice(8192),
}
}
}
@@ -943,7 +950,7 @@ impl<R: AsyncRead + Unpin + Send + Sync> AsyncRead for SkipReader<R> {
while this.bytes_skipped < this.bytes_to_skip {
let remaining = this.bytes_to_skip - this.bytes_skipped;
let scratch_len = remaining.min(this.scratch.len());
let mut scratch_buf = ReadBuf::new(&mut this.scratch[..scratch_len]);
let mut scratch_buf = ReadBuf::uninit(&mut this.scratch[..scratch_len]);
match Pin::new(&mut this.inner).poll_read(cx, &mut scratch_buf) {
Poll::Pending => return Poll::Pending,
Poll::Ready(Err(err)) => return Poll::Ready(Err(err)),
@@ -974,7 +981,7 @@ pub struct RangedDecompressReader<R: AsyncRead + Unpin + Send + Sync + 'static>
target_length: usize,
current_offset: usize,
bytes_returned: usize,
scratch: Vec<u8>,
scratch: Box<[MaybeUninit<u8>]>,
drain_on_done: bool,
drain_task: Option<tokio::task::JoinHandle<()>>,
}
@@ -1012,7 +1019,7 @@ impl<R: AsyncRead + Unpin + Send + Sync + 'static> RangedDecompressReader<R> {
target_length: actual_length,
current_offset: 0,
bytes_returned: 0,
scratch: vec![0u8; 8192],
scratch: Box::<[u8]>::new_uninit_slice(8192),
drain_on_done,
drain_task: None,
})
@@ -1062,7 +1069,7 @@ impl<R: AsyncRead + Unpin + Send + Sync + 'static> AsyncRead for RangedDecompres
}
let scratch_len = std::cmp::min(this.scratch.len(), std::cmp::max(buf_capacity, 1));
let mut temp_read_buf = ReadBuf::new(&mut this.scratch[..scratch_len]);
let mut temp_read_buf = ReadBuf::uninit(&mut this.scratch[..scratch_len]);
let Some(inner) = this.inner.as_mut() else {
return Poll::Ready(Ok(()));
@@ -1114,7 +1121,8 @@ impl<R: AsyncRead + Unpin + Send + Sync + 'static> AsyncRead for RangedDecompres
);
if bytes_to_return > 0 {
let data_slice = &this.scratch[data_start_in_buffer..data_start_in_buffer + bytes_to_return];
let data_slice =
&temp_read_buf.filled()[data_start_in_buffer..data_start_in_buffer + bytes_to_return];
buf.put_slice(data_slice);
this.bytes_returned += bytes_to_return;
@@ -1133,7 +1141,7 @@ impl<R: AsyncRead + Unpin + Send + Sync + 'static> AsyncRead for RangedDecompres
std::cmp::min(n, std::cmp::min(buf.remaining(), this.target_length - this.bytes_returned));
if bytes_to_return > 0 {
buf.put_slice(&this.scratch[..bytes_to_return]);
buf.put_slice(&temp_read_buf.filled()[..bytes_to_return]);
this.bytes_returned += bytes_to_return;
tracing::trace!("Returned {} bytes at offset {}", bytes_to_return, old_offset);
@@ -1203,20 +1211,7 @@ impl<R: AsyncRead + Unpin + Send + 'static> AsyncRead for StreamConsumer<R> {
impl<R: AsyncRead + Unpin + Send + 'static> Drop for StreamConsumer<R> {
fn drop(&mut self) {
if self.consumer_task.is_none() && self.inner.is_some() {
let mut inner = self.inner.take().unwrap();
let task = tokio::spawn(async move {
let mut buf = [0u8; 8192];
loop {
match inner.read(&mut buf).await {
Ok(0) => break, // EOF
Ok(_) => continue, // Keep consuming
Err(_) => break, // Error, stop consuming
}
}
});
self.consumer_task = Some(task);
}
self.ensure_consumer_started();
}
}
@@ -1263,6 +1258,43 @@ mod tests {
use temp_env::async_with_vars;
use tokio::io::AsyncReadExt;
#[derive(Debug)]
struct PendingPartialReader {
data: &'static [u8],
position: usize,
pending: bool,
}
impl PendingPartialReader {
fn new(data: &'static [u8]) -> Self {
Self {
data,
position: 0,
pending: true,
}
}
}
impl AsyncRead for PendingPartialReader {
fn poll_read(mut self: Pin<&mut Self>, cx: &mut Context<'_>, buf: &mut ReadBuf<'_>) -> Poll<std::io::Result<()>> {
if self.pending {
self.pending = false;
cx.waker().wake_by_ref();
return Poll::Pending;
}
if self.position == self.data.len() {
return Poll::Ready(Ok(()));
}
let length = buf.remaining().min(3).min(self.data.len() - self.position);
let end = self.position + length;
buf.put_slice(&self.data[self.position..end]);
self.position = end;
self.pending = true;
Poll::Ready(Ok(()))
}
}
const TEST_DIRECT_KEY_HEADER: &str = "x-rustfs-test-direct-key";
const TEST_OBJECT_KEY_HEADER: &str = "x-rustfs-test-object-key";
const TEST_NONCE_HEADER: &str = "x-rustfs-test-nonce";
@@ -1400,6 +1432,36 @@ mod tests {
assert_eq!(result, b"World");
}
#[tokio::test]
async fn uninitialized_scratch_preserves_partial_pending_and_eof_reads() {
let mut skipped = SkipReader::new(PendingPartialReader::new(b"0123456789abcdef"), 5);
let mut skipped_output = Vec::new();
skipped
.read_to_end(&mut skipped_output)
.await
.expect("skip reader should survive partial pending reads through EOF");
assert_eq!(skipped_output, b"56789abcdef");
let mut ranged = RangedDecompressReader::new(PendingPartialReader::new(b"0123456789abcdef"), 5, 7, 16)
.expect("valid range should construct");
let mut ranged_output = Vec::new();
ranged
.read_to_end(&mut ranged_output)
.await
.expect("range reader should survive partial pending reads through EOF");
assert_eq!(ranged_output, b"56789ab");
}
#[tokio::test]
async fn uninitialized_skip_scratch_reports_early_eof() {
let mut reader = SkipReader::new(PendingPartialReader::new(b"short"), 6);
let error = reader
.read_to_end(&mut Vec::new())
.await
.expect_err("EOF before the skip boundary must remain visible");
assert_eq!(error.kind(), std::io::ErrorKind::UnexpectedEof);
}
#[tokio::test]
async fn test_ranged_decompress_reader_from_start() {
let original_data = b"Hello, World! This is a test.";
+4
View File
@@ -172,6 +172,7 @@ impl ObjectLockConfigSnapshot {
}
}
#[allow(dead_code, reason = "snapshot-scope predicate asserted by this file's tests (backlog#1823)")]
pub(crate) fn is_for_store_bucket(
&self,
store_id: Uuid,
@@ -260,6 +261,9 @@ pub struct ObjectOptions {
pub data_movement: bool,
pub raw_data_movement_read: bool,
/// Materialize the data-movement per-part checksum sidecar for APIs that
/// return part checksums. Ordinary object reads leave it encoded.
pub include_part_checksums: bool,
pub src_pool_idx: usize,
pub user_defined: HashMap<String, String>,
pub preserve_etag: Option<String>,
-38
View File
@@ -31,7 +31,6 @@ use std::{
use tokio::sync::{OnceCell, RwLock};
use tokio_util::sync::CancellationToken;
use tracing::warn;
use uuid::Uuid;
pub const DISK_ASSUME_UNKNOWN_SIZE: u64 = 1 << 30;
pub const DISK_MIN_INODES: u64 = 1000;
@@ -109,18 +108,6 @@ pub fn set_global_rustfs_port(value: u16) {
}
}
/// Set the global deployment id
///
/// # Arguments
/// * `id` - The Uuid to set as the global deployment id
///
/// # Returns
/// * None
///
pub fn set_global_deployment_id(id: Uuid) {
current_ctx().set_deployment_id(id);
}
/// Get the global deployment id
///
/// # Returns
@@ -288,19 +275,6 @@ pub fn get_global_region() -> Option<s3s::region::Region> {
current_ctx().region()
}
/// Initialize the global background services cancellation token
///
/// # Arguments
/// * `cancel_token` - The CancellationToken instance to set globally
///
/// # Returns
/// * `Ok(())` if successful
/// * `Err(CancellationToken)` if setting fails
///
pub fn init_background_services_cancel_token(cancel_token: CancellationToken) -> Result<(), CancellationToken> {
current_ctx().init_background_cancel_token(cancel_token)
}
/// Get the global background services cancellation token
///
/// # Returns
@@ -310,18 +284,6 @@ pub fn get_background_services_cancel_token() -> Option<CancellationToken> {
current_ctx().background_cancel_token()
}
/// Create and initialize the global background services cancellation token
///
/// # Returns
/// * `CancellationToken` - The newly created global cancellation token
///
pub fn create_background_services_cancel_token() -> CancellationToken {
let cancel_token = CancellationToken::new();
init_background_services_cancel_token(cancel_token.clone())
.expect("background services cancel token should be initialized once during startup");
cancel_token
}
/// Shutdown all background services gracefully
///
/// # Returns
+4
View File
@@ -402,6 +402,10 @@ impl InstanceContext {
}
#[cfg(test)]
#[allow(
dead_code,
reason = "driven by the tier-delete-journal recovery test behind `--features test-util` (backlog#1823)"
)]
pub(crate) fn wake_tier_delete_journal_recovery(&self) {
self.tier_delete_journal_recovery_wakeup.notify_one();
}
-1
View File
@@ -13,7 +13,6 @@
// limitations under the License.
// #730: runtime source migration keeps fallback handles until all owners inject state.
#![allow(dead_code)]
pub(crate) mod global;
pub(crate) mod instance;
+11 -52
View File
@@ -38,7 +38,6 @@ use crate::{
set_object_layer, update_erasure_type,
},
services::batch_processor::{GlobalBatchProcessors, get_global_processors},
services::event_notification::EventNotifier,
services::notification_sys::{NotificationSys, get_global_notification_sys},
services::tier::tier::TierConfigMgr,
store::ECStore,
@@ -143,6 +142,10 @@ pub async fn setup_is_erasure_sd() -> bool {
is_erasure_sd().await
}
#[allow(
dead_code,
reason = "setup-type override used only by tests across this crate (backlog#1823)"
)]
pub(crate) async fn current_setup_type() -> SetupType {
if setup_is_dist_erasure().await {
SetupType::DistErasure
@@ -155,6 +158,10 @@ pub(crate) async fn current_setup_type() -> SetupType {
}
}
#[allow(
dead_code,
reason = "setup-type override used only by tests across this crate (backlog#1823)"
)]
pub(crate) async fn set_setup_type(setup_type: SetupType) {
update_erasure_type(setup_type).await;
}
@@ -164,6 +171,9 @@ pub(crate) async fn local_node_name() -> String {
}
pub(crate) async fn set_local_node_name(node_name: String) {
// Also stamp the internode-metrics server label: io-metrics is a leaf
// crate and no longer resolves node identity itself (backlog#1834).
rustfs_io_metrics::internode_metrics::set_internode_server_label(node_name.as_str());
rustfs_common::set_global_local_node_name(&node_name).await;
}
@@ -229,14 +239,6 @@ pub(crate) fn ensure_test_rpc_secret() {
let _ = rustfs_credentials::set_global_rpc_secret(TEST_RPC_SECRET.to_owned());
}
pub(crate) fn storage_class_parity(storage_class: Option<&str>) -> Option<usize> {
get_global_storage_class_snapshot().get_parity_for_sc(storage_class.unwrap_or_default())
}
pub(crate) fn storage_class_should_inline(shard_size: i64, versioned: bool) -> bool {
get_global_storage_class_snapshot().should_inline(shard_size, versioned)
}
pub(crate) fn deployment_upload_id(upload_id: &str) -> String {
base64_simd::URL_SAFE_NO_PAD
.encode_to_string(format!("{}.{}", get_global_deployment_id().unwrap_or_default(), upload_id).as_bytes())
@@ -329,21 +331,6 @@ pub(crate) fn storage_class_config_snapshot() -> Arc<storageclass::Config> {
get_global_storage_class_snapshot()
}
/// Scalar STANDARD / RRS parity for backend-info reporting.
///
/// Retained for the rebalance/backend-info path. `get_parity_for_sc` returns
/// `None` when the runtime config is uninitialized or (post per-pool support)
/// when pools disagree, so STANDARD falls back to the caller's default and RRS
/// stays `None` — matching the pre-per-pool scalar reporting.
pub(crate) fn backend_storage_class_parities(default_standard_parity: usize) -> (Option<usize>, Option<usize>) {
let sc = get_global_storage_class_snapshot();
let standard = sc
.get_parity_for_sc(storageclass::CLASS_STANDARD)
.or(Some(default_standard_parity));
let reduced_redundancy = sc.get_parity_for_sc(storageclass::RRS);
(standard, reduced_redundancy)
}
pub(crate) fn set_storage_class_config(config: storageclass::Config) {
set_global_storage_class(config);
}
@@ -411,10 +398,6 @@ pub fn transition_state_handle() -> Arc<TransitionState> {
crate::runtime::global::current_ctx().transition_state()
}
pub(crate) fn event_notifier_handle() -> Arc<RwLock<EventNotifier>> {
crate::runtime::global::current_ctx().event_notifier()
}
pub(crate) async fn local_disk_by_path(path: &str) -> Option<DiskStore> {
local_disk_map_handle().read().await.get(path).cloned().flatten()
}
@@ -508,30 +491,6 @@ pub(crate) async fn local_disk_set_drive(
instance_ctx.local_disk_set_drives().read().await[pool_idx][set_idx][disk_idx].clone()
}
pub(crate) async fn local_disk_for_endpoint(endpoint: &Endpoint) -> Option<DiskStore> {
let set_drives = local_disk_set_drives_handle();
let global_set_drives = set_drives.read().await;
if global_set_drives.is_empty() {
return local_disk_map_handle()
.read()
.await
.get(&endpoint.to_string())
.cloned()
.unwrap_or(None);
}
let pool_idx = usize::try_from(endpoint.pool_idx).ok()?;
let set_idx = usize::try_from(endpoint.set_idx).ok()?;
let disk_idx = usize::try_from(endpoint.disk_idx).ok()?;
global_set_drives
.get(pool_idx)
.and_then(|sets| sets.get(set_idx))
.and_then(|disks| disks.get(disk_idx))
.cloned()
.unwrap_or(None)
}
pub(crate) async fn local_disk_paths() -> Vec<String> {
local_disk_map_handle().read().await.keys().cloned().collect()
}
@@ -206,6 +206,38 @@ pub(crate) fn remote_version_state_fleet_proof_matches(proof: &RemoteVersionStat
})
}
#[cfg(test)]
pub(crate) struct RemoteVersionStateFleetProofGuard;
#[cfg(test)]
impl Drop for RemoteVersionStateFleetProofGuard {
fn drop(&mut self) {
replace_remote_version_state_fleet_proof(None);
}
}
#[cfg(test)]
pub(crate) fn install_remote_version_state_fleet_proof_for_test(topology_fingerprint: &str) -> RemoteVersionStateFleetProofGuard {
match REMOTE_VERSION_STATE_PROBE_TOPOLOGY.set(topology_fingerprint.to_string()) {
Ok(()) => {}
Err(_)
if REMOTE_VERSION_STATE_PROBE_TOPOLOGY
.get()
.is_some_and(|current| current == topology_fingerprint) => {}
Err(_) => panic!("remote version state test topology is already bound to another fingerprint"),
}
let peer_epochs = BTreeMap::new();
if let Some(err) = publish_remote_version_state_probe_result(
remote_version_state_fleet_proof_slot(),
topology_fingerprint,
Ok(peer_epochs),
Instant::now(),
) {
panic!("test proof installation must not fail: {err}");
}
RemoteVersionStateFleetProofGuard
}
fn remote_version_state_fleet_proof_valid_at(
proof: Option<&RemoteVersionStateFleetProof>,
expected_topology: &str,
+31 -5
View File
@@ -16,7 +16,7 @@ use super::meta::{
clone_arc_by_index, ensure_valid_rebalance_pool_index, invalid_rebalance_pool_index_error,
rebalance_metadata_not_initialized_error, should_ignore_rebalance_data_usage_cache,
};
use super::migration::migrate_entry_version;
use super::migration::{RebalanceMigrationBackend, migrate_entry_version};
use super::worker::{
RebalanceEntryCleanupResult, RebalanceEntryTask, load_rebalance_bucket_configs, rebalance_max_attempts,
resolve_rebalance_bucket_error, resolve_rebalance_entry_cleanup_delete_result, resolve_rebalance_file_info_versions_result,
@@ -144,6 +144,11 @@ impl ECStore {
return Ok(RebalanceEntryOutcome::Completed);
}
let bucket_incarnation_fence = match bucket_configs.bucket_incarnation_id {
Some(expected) => Some(self.acquire_bucket_incarnation_fence(&bucket, expected).await?),
None => None,
};
let mut fivs =
resolve_rebalance_file_info_versions_result(entry.file_info_versions(&bucket), bucket.as_str(), entry.name.as_str())?;
@@ -203,9 +208,14 @@ impl ECStore {
}
let version_id = version.version_id.map(|v| v.to_string());
let expected_bucket_incarnation_id = bucket_configs.bucket_incarnation_id;
let mut transfer = |src_pool_idx: usize, bucket: String, rd: GetObjectReader| {
let store = self.clone();
async move { store.rebalance_object(src_pool_idx, bucket, rd).await }
async move {
store
.rebalance_object(src_pool_idx, bucket, rd, expected_bucket_incarnation_id)
.await
}
};
// Route delete-marker migration through the store layer so it lands on the
// cross-pool target (excluding the source pool), not back onto the source set.
@@ -214,11 +224,12 @@ impl ECStore {
async move { store.delete_object(&bucket, &object, opts).await }
};
let result = migrate_entry_version(
set.as_ref(),
&RebalanceMigrationBackend::new(set.as_ref(), self.as_ref()),
bucket.clone(),
pool_index,
version,
version_id.clone(),
expected_bucket_incarnation_id,
rebalance_max_attempts(),
should_ignore_rebalance_data_usage_cache(bucket.as_str()),
&mut transfer,
@@ -303,6 +314,9 @@ impl ECStore {
}
if should_cleanup_rebalance_source_entry(rebalanced, fivs.versions.len(), expired) {
if bucket_incarnation_fence.as_ref().is_some_and(|guard| guard.is_lock_lost()) {
return Err(Error::other("rebalance bucket incarnation fence was lost before source cleanup"));
}
let cleanup_result = self
.finish_rebalance_entry_after_cleanup(
pool_index,
@@ -315,6 +329,12 @@ impl ECStore {
entry.name.as_str(),
&fivs,
&cleanup_preflight_allowed_missing,
data_movement::SourceCleanupBucketFence {
expected_incarnation_id: bucket_configs.bucket_incarnation_id,
lifecycle_guard: bucket_incarnation_fence
.as_ref()
.and_then(|guard| guard.namespace_lock_guard()),
},
"rebalance",
),
)
@@ -389,8 +409,14 @@ impl ECStore {
}
#[tracing::instrument(skip(self, rd))]
async fn rebalance_object(self: Arc<Self>, pool_idx: usize, bucket: String, rd: GetObjectReader) -> Result<()> {
data_movement::migrate_object(self, pool_idx, bucket, rd, "rebalance_object").await
async fn rebalance_object(
self: Arc<Self>,
pool_idx: usize,
bucket: String,
rd: GetObjectReader,
expected_bucket_incarnation_id: Option<uuid::Uuid>,
) -> Result<()> {
data_movement::migrate_object(self, pool_idx, bucket, rd, expected_bucket_incarnation_id, "rebalance_object").await
}
async fn update_rebalance_last_error(&self, pool_idx: usize, message: String) -> Result<()> {
@@ -5,6 +5,7 @@ use crate::error::{Error, Result, is_err_object_not_found, is_err_version_not_fo
use crate::object_api::{GetObjectReader, ObjectInfo, ObjectOptions};
use crate::set_disk::SetDisks;
use crate::storage_api_contracts::{object::ObjectIO, range::HTTPRangeSpec};
use crate::store::ECStore;
use http::HeaderMap;
use rustfs_filemeta::FileInfo;
use rustfs_utils::path::encode_dir_object;
@@ -21,15 +22,23 @@ pub(crate) struct MigrationVersionResult {
pub error: Option<Error>,
}
pub(super) fn rebalance_delete_marker_opts(version: &FileInfo, version_id: Option<String>, src_pool_idx: usize) -> ObjectOptions {
pub(super) fn rebalance_delete_marker_opts(
version: &FileInfo,
version_id: Option<String>,
src_pool_idx: usize,
expected_bucket_incarnation_id: Option<uuid::Uuid>,
) -> ObjectOptions {
let version_suspended = version.version_id.is_none() && version_id.is_none();
ObjectOptions {
versioned: true,
version_id,
versioned: !version_suspended,
version_suspended,
version_id: version_id.or_else(|| version_suspended.then(|| uuid::Uuid::nil().to_string())),
mod_time: version.mod_time,
src_pool_idx,
data_movement: true,
delete_marker: true,
skip_decommissioned: true,
expected_bucket_incarnation_id,
delete_replication: version
.replication_state_internal
.as_ref()
@@ -38,7 +47,12 @@ pub(super) fn rebalance_delete_marker_opts(version: &FileInfo, version_id: Optio
}
}
fn rebalance_remote_tiered_opts(version: &FileInfo, version_id: Option<String>, src_pool_idx: usize) -> ObjectOptions {
fn rebalance_remote_tiered_opts(
version: &FileInfo,
version_id: Option<String>,
src_pool_idx: usize,
expected_bucket_incarnation_id: Option<uuid::Uuid>,
) -> ObjectOptions {
ObjectOptions {
versioned: version_id.is_some(),
version_id,
@@ -46,6 +60,21 @@ fn rebalance_remote_tiered_opts(version: &FileInfo, version_id: Option<String>,
user_defined: version.metadata.clone(),
src_pool_idx,
data_movement: true,
include_part_checksums: true,
http_preconditions: Some(crate::data_movement::data_movement_target_precondition()),
expected_bucket_incarnation_id,
..Default::default()
}
}
pub(super) fn rebalance_object_migration_read_opts(version_id: Option<String>) -> ObjectOptions {
ObjectOptions {
version_id,
no_lock: true,
data_movement: true,
raw_data_movement_read: true,
skip_decommissioned: true,
skip_rebalancing: true,
..Default::default()
}
}
@@ -70,8 +99,19 @@ pub(crate) trait MigrationBackend: Send + Sync {
) -> Result<()>;
}
pub(crate) struct RebalanceMigrationBackend<'a> {
source: &'a SetDisks,
store: &'a ECStore,
}
impl<'a> RebalanceMigrationBackend<'a> {
pub(crate) fn new(source: &'a SetDisks, store: &'a ECStore) -> Self {
Self { source, store }
}
}
#[async_trait::async_trait]
impl MigrationBackend for SetDisks {
impl MigrationBackend for RebalanceMigrationBackend<'_> {
async fn get_object_reader_for_migration(
&self,
bucket: &str,
@@ -80,7 +120,7 @@ impl MigrationBackend for SetDisks {
h: HeaderMap,
opts: &ObjectOptions,
) -> Result<GetObjectReader> {
self.get_object_reader(bucket, object, range, h, opts).await
self.source.get_object_reader(bucket, object, range, h, opts).await
}
async fn move_remote_version_for_migration(
@@ -90,7 +130,7 @@ impl MigrationBackend for SetDisks {
fi: &FileInfo,
opts: &ObjectOptions,
) -> Result<()> {
self.decommission_tiered_object(bucket, object, fi, opts).await
self.store.decommission_tiered_object(bucket, object, fi, opts).await
}
}
@@ -101,6 +141,7 @@ pub(crate) async fn migrate_entry_version<Backend, F, Fut, D, DFut>(
pool_index: usize,
version: &FileInfo,
version_id: Option<String>,
expected_bucket_incarnation_id: Option<uuid::Uuid>,
max_attempts: usize,
ignore_data_usage_cache: bool,
transfer: F,
@@ -113,12 +154,13 @@ where
D: FnMut(String, String, ObjectOptions) -> DFut + Send,
DFut: Future<Output = Result<ObjectInfo>> + Send,
{
migrate_entry_version_with_retry_wait(
migrate_entry_version_with_retry_wait_and_incarnation(
set,
bucket,
pool_index,
version,
version_id,
expected_bucket_incarnation_id,
max_attempts,
ignore_data_usage_cache,
transfer,
@@ -137,6 +179,45 @@ pub(super) async fn migrate_entry_version_with_retry_wait<Backend, F, Fut, D, DF
version_id: Option<String>,
max_attempts: usize,
ignore_data_usage_cache: bool,
transfer: F,
delete_marker: D,
wait_retry: W,
) -> MigrationVersionResult
where
Backend: MigrationBackend + ?Sized,
F: FnMut(usize, String, GetObjectReader) -> Fut + Send,
Fut: Future<Output = Result<()>> + Send,
D: FnMut(String, String, ObjectOptions) -> DFut + Send,
DFut: Future<Output = Result<ObjectInfo>> + Send,
W: FnMut(Duration) -> WFut + Send,
WFut: Future<Output = ()> + Send,
{
migrate_entry_version_with_retry_wait_and_incarnation(
set,
bucket,
pool_index,
version,
version_id,
None,
max_attempts,
ignore_data_usage_cache,
transfer,
delete_marker,
wait_retry,
)
.await
}
#[allow(clippy::too_many_arguments)]
async fn migrate_entry_version_with_retry_wait_and_incarnation<Backend, F, Fut, D, DFut, W, WFut>(
set: &Backend,
bucket: String,
pool_index: usize,
version: &FileInfo,
version_id: Option<String>,
expected_bucket_incarnation_id: Option<uuid::Uuid>,
max_attempts: usize,
ignore_data_usage_cache: bool,
mut transfer: F,
mut delete_marker: D,
mut wait_retry: W,
@@ -169,7 +250,7 @@ where
&bucket,
&version.name,
version,
&rebalance_remote_tiered_opts(version, version_id, pool_index),
&rebalance_remote_tiered_opts(version, version_id, pool_index, expected_bucket_incarnation_id),
)
.await
{
@@ -212,7 +293,7 @@ where
if let Err(err) = delete_marker(
bucket.clone(),
version.name.clone(),
rebalance_delete_marker_opts(version, version_id, pool_index),
rebalance_delete_marker_opts(version, version_id, pool_index, expected_bucket_incarnation_id),
)
.await
{
@@ -255,11 +336,7 @@ where
&encode_dir_object(&version.name),
None,
HeaderMap::new(),
&ObjectOptions {
version_id: version_id.clone(),
no_lock: true,
..Default::default()
},
&rebalance_object_migration_read_opts(version_id.clone()),
)
.await
{
@@ -113,6 +113,8 @@ struct LegacyRebalanceMeta {
struct MigrationBackendSpy {
get_object_reader: Mutex<Option<core::result::Result<GetObjectReader, Error>>>,
move_remote: Mutex<Option<core::result::Result<(), Error>>>,
get_opts: Mutex<Vec<ObjectOptions>>,
move_remote_opts: Mutex<Vec<ObjectOptions>>,
get_calls: AtomicUsize,
move_remote_calls: AtomicUsize,
}
@@ -125,6 +127,8 @@ impl MigrationBackendSpy {
Self {
get_object_reader: Mutex::new(get_object_reader),
move_remote: Mutex::new(move_remote),
get_opts: Mutex::new(Vec::new()),
move_remote_opts: Mutex::new(Vec::new()),
get_calls: AtomicUsize::new(0),
move_remote_calls: AtomicUsize::new(0),
}
@@ -138,6 +142,24 @@ impl MigrationBackendSpy {
self.move_remote_calls.load(Ordering::SeqCst)
}
fn last_get_opts(&self) -> ObjectOptions {
self.get_opts
.lock()
.unwrap()
.last()
.cloned()
.expect("reader opts should be captured")
}
fn last_move_remote_opts(&self) -> ObjectOptions {
self.move_remote_opts
.lock()
.unwrap()
.last()
.cloned()
.expect("remote opts should be captured")
}
fn make_reader() -> GetObjectReader {
GetObjectReader {
stream: Box::new(Cursor::new(vec![0_u8; 3])),
@@ -156,9 +178,10 @@ impl MigrationBackend for MigrationBackendSpy {
_object: &str,
_range: Option<HTTPRangeSpec>,
_h: http::HeaderMap,
_opts: &ObjectOptions,
opts: &ObjectOptions,
) -> Result<GetObjectReader> {
self.get_calls.fetch_add(1, Ordering::SeqCst);
self.get_opts.lock().unwrap().push(opts.clone());
if let Some(result) = self.get_object_reader.lock().unwrap().take() {
return result;
}
@@ -171,9 +194,10 @@ impl MigrationBackend for MigrationBackendSpy {
_bucket: &str,
_object: &str,
_fi: &FileInfo,
_opts: &ObjectOptions,
opts: &ObjectOptions,
) -> Result<()> {
self.move_remote_calls.fetch_add(1, Ordering::SeqCst);
self.move_remote_opts.lock().unwrap().push(opts.clone());
if let Some(result) = self.move_remote.lock().unwrap().take() {
return result;
}
@@ -217,7 +241,8 @@ fn test_rebalance_delete_marker_opts_preserves_replication_state() {
..version_deleted()
};
let opts = rebalance_delete_marker_opts(&version, Some("version-id".to_string()), 7);
let incarnation = uuid::Uuid::new_v4();
let opts = rebalance_delete_marker_opts(&version, Some("version-id".to_string()), 7, Some(incarnation));
let replication = opts.delete_replication.expect("replication state should be preserved");
assert!(opts.versioned);
@@ -227,11 +252,22 @@ fn test_rebalance_delete_marker_opts_preserves_replication_state() {
assert_eq!(opts.src_pool_idx, 7);
assert_eq!(opts.version_id.as_deref(), Some("version-id"));
assert_eq!(opts.mod_time, Some(mod_time));
assert_eq!(opts.expected_bucket_incarnation_id, Some(incarnation));
assert_eq!(replication.replica_status, ReplicationStatusType::Replica);
assert!(replication.delete_marker);
assert_eq!(replication.replicate_decision_str, "existing");
}
#[test]
fn test_rebalance_delete_marker_opts_preserves_suspended_null_version() {
let version = version_deleted();
let opts = rebalance_delete_marker_opts(&version, None, 7, None);
assert!(!opts.versioned);
assert!(opts.version_suspended);
assert_eq!(opts.version_id.as_deref(), Some(uuid::Uuid::nil().to_string().as_str()));
}
#[tokio::test]
async fn test_migrate_entry_version_remote_version_is_moved_without_transfer() {
let backend = MigrationBackendSpy::new(None, Some(Ok(())));
@@ -248,12 +284,14 @@ async fn test_migrate_entry_version_remote_version_is_moved_without_transfer() {
}
};
let incarnation = uuid::Uuid::new_v4();
let result = migrate_entry_version(
&backend,
"bucket".to_string(),
0,
&version,
version.version_id.map(|v| v.to_string()),
Some(incarnation),
3,
false,
&mut transfer,
@@ -269,6 +307,10 @@ async fn test_migrate_entry_version_remote_version_is_moved_without_transfer() {
assert_eq!(transfer_count.load(Ordering::SeqCst), 0);
assert_eq!(backend.move_remote_calls(), 1);
assert_eq!(backend.get_calls(), 0);
let remote_opts = backend.last_move_remote_opts();
assert!(remote_opts.include_part_checksums);
assert!(remote_opts.http_preconditions.is_some());
assert_eq!(remote_opts.expected_bucket_incarnation_id, Some(incarnation));
}
#[tokio::test]
@@ -294,6 +336,7 @@ async fn test_migrate_entry_version_remote_not_found_is_cleanup_ignored() {
0,
&version,
version.version_id.map(|v| v.to_string()),
None,
3,
false,
&mut transfer,
@@ -330,6 +373,7 @@ async fn test_migrate_entry_version_remote_overwrite_is_not_ignored() {
0,
&version,
Some("vid-1".to_string()),
None,
3,
false,
&mut transfer,
@@ -368,6 +412,7 @@ async fn test_migrate_entry_version_remote_failure_is_reported() {
0,
&version,
version.version_id.map(|v| v.to_string()),
None,
3,
false,
&mut transfer,
@@ -410,6 +455,7 @@ async fn test_migrate_entry_version_deleted_version_routes_delete_through_store_
1,
&version,
version.version_id.map(|v| v.to_string()),
None,
3,
false,
&mut transfer,
@@ -449,6 +495,7 @@ async fn test_migrate_entry_version_deleted_version_not_found_is_ignored() {
1,
&version,
version.version_id.map(|v| v.to_string()),
None,
3,
false,
&mut transfer,
@@ -491,6 +538,7 @@ async fn test_migrate_entry_version_deleted_version_overwrite_is_not_ignored() {
1,
&version,
Some("vid-1".to_string()),
None,
3,
false,
&mut transfer,
@@ -520,6 +568,7 @@ async fn test_migrate_entry_version_reader_not_found_is_ignored() {
1,
&version,
version.version_id.map(|v| v.to_string()),
None,
3,
false,
&mut transfer,
@@ -647,6 +696,7 @@ async fn test_migrate_entry_version_reader_fails_after_retries() {
1,
&version,
version.version_id.map(|v| v.to_string()),
None,
3,
false,
&mut transfer,
@@ -685,6 +735,7 @@ async fn test_migrate_entry_version_zero_max_attempts_still_attempts_once() {
1,
&version,
version.version_id.map(|v| v.to_string()),
None,
0,
false,
&mut transfer,
@@ -750,6 +801,13 @@ async fn test_migrate_entry_version_transfer_retries_before_success() {
assert_eq!(backend.get_calls(), 2);
assert_eq!(transfer_count.load(Ordering::SeqCst), 2);
assert_eq!(wait_count.load(Ordering::SeqCst), 1);
let read_opts = backend.last_get_opts();
assert_eq!(read_opts.version_id.as_deref(), version.version_id.map(|id| id.to_string()).as_deref());
assert!(read_opts.no_lock);
assert!(read_opts.data_movement);
assert!(read_opts.raw_data_movement_read);
assert!(read_opts.skip_decommissioned);
assert!(read_opts.skip_rebalancing);
}
#[tokio::test]
@@ -822,6 +880,7 @@ async fn test_migrate_entry_version_transfer_fails_after_retries() {
1,
&version,
version.version_id.map(|v| v.to_string()),
None,
2,
false,
&mut transfer,
@@ -860,6 +919,7 @@ async fn test_migrate_entry_version_transfer_not_found_is_ignored() {
1,
&version,
version.version_id.map(|v| v.to_string()),
None,
3,
false,
&mut transfer,
@@ -901,6 +961,7 @@ async fn test_migrate_entry_version_transfer_overwrite_is_not_ignored() {
1,
&version,
Some("vid-1".to_string()),
None,
3,
false,
&mut transfer,
@@ -943,6 +1004,7 @@ async fn test_migrate_entry_version_ignores_data_usage_cache_when_enabled() {
1,
&version,
version.version_id.map(|v| v.to_string()),
None,
2,
true,
&mut transfer,
@@ -985,6 +1047,7 @@ async fn test_migrate_entry_version_data_usage_cache_moves_when_ignore_disabled(
1,
&version,
version.version_id.map(|v| v.to_string()),
None,
2,
false,
&mut transfer,
@@ -2026,6 +2089,7 @@ async fn test_migrate_entry_version_transfer_failure_reports_write_target_stage(
1,
&version,
version.version_id.map(|v| v.to_string()),
None,
1,
false,
&mut transfer,
@@ -2050,6 +2114,7 @@ async fn test_migrate_entry_version_reader_failure_reports_read_source_stage() {
1,
&version,
version.version_id.map(|v| v.to_string()),
None,
1,
false,
&mut transfer,
@@ -36,6 +36,7 @@ pub type RStats = Vec<Arc<RebalanceStats>>;
#[derive(Debug, Default)]
pub(super) struct RebalanceBucketConfigs {
pub(super) bucket_incarnation_id: Option<uuid::Uuid>,
pub(super) lifecycle_config: Option<s3s::dto::BucketLifecycleConfiguration>,
pub(super) object_lock_config: Option<s3s::dto::ObjectLockConfiguration>,
pub(super) replication_config: Option<(s3s::dto::ReplicationConfiguration, OffsetDateTime)>,
@@ -406,6 +406,7 @@ pub(super) async fn load_rebalance_bucket_configs(api: &ECStore, bucket: &str) -
let expiry_configs = crate::bucket::lifecycle::get_expiry_configs(api, bucket).await?;
Ok(RebalanceBucketConfigs {
bucket_incarnation_id: Some(api.bucket_incarnation_id_from_disk(bucket).await?),
lifecycle_config: expiry_configs.lifecycle.map(|config| (*config).clone()),
object_lock_config: expiry_configs.object_lock.map(|config| (*config).clone()),
replication_config: resolve_rebalance_optional_bucket_config_result(
File diff suppressed because it is too large Load Diff
+83
View File
@@ -85,6 +85,15 @@ impl SetDisks {
format!("{}/{}", Self::get_multipart_sha_dir(bucket, object), upload_uuid)
}
pub(super) fn get_multipart_upload_dir(bucket: &str, object: &str, upload_id: &str, data_movement: bool) -> String {
let upload_dir = Self::get_upload_id_dir(bucket, object, upload_id);
if data_movement {
format!("{DATA_MOVEMENT_MULTIPART_PREFIX}/{upload_dir}")
} else {
upload_dir
}
}
pub(super) fn get_multipart_sha_dir(bucket: &str, object: &str) -> String {
let path = format!("{bucket}/{object}");
let mut hasher = Sha256::new();
@@ -466,6 +475,28 @@ impl SetDisks {
Self::find_file_info_in_quorum(metas, &mod_time, &etag, quorum)
}
pub(crate) fn hydrate_selected_fileinfo_part_checksums(fi: &mut FileInfo) -> disk::error::Result<()> {
fi.hydrate_data_movement_part_checksums().map_err(DiskError::from)?;
for part in &fi.parts {
let Some(checksums) = part.checksums.as_ref() else {
continue;
};
let mut algorithms = HashSet::with_capacity(checksums.len());
for (name, value) in checksums {
let Some(checksum) = rustfs_rio::Checksum::new_from_string(name, value) else {
return Err(DiskError::FileCorrupt);
};
if checksum.checksum_type.is(rustfs_rio::ChecksumType::MULTIPART) {
return Err(DiskError::FileCorrupt);
}
if !algorithms.insert(checksum.checksum_type.base().0) {
return Err(DiskError::FileCorrupt);
}
}
}
Ok(())
}
fn update_hash_bytes(hasher: &mut Sha256, value: &[u8]) {
hasher.update(value.len().to_le_bytes());
hasher.update(value);
@@ -1079,6 +1110,25 @@ impl SetDisks {
shuffled_disks
}
pub(super) fn shuffle_disks_owned(mut disks: Vec<Option<DiskStore>>, distribution: &[usize]) -> Vec<Option<DiskStore>> {
if distribution.is_empty() {
return disks;
}
let mut shuffled_disks = vec![None; disks.len()];
for (index, disk) in disks.iter_mut().enumerate() {
let Some(slot) = distribution
.get(index)
.and_then(|block_index| block_index.checked_sub(1))
.filter(|slot| *slot < shuffled_disks.len())
else {
continue;
};
shuffled_disks[slot] = disk.take();
}
shuffled_disks
}
pub(super) fn shuffle_check_parts(parts_errs: &[usize], distribution: &[usize]) -> Vec<usize> {
if distribution.is_empty() {
return parts_errs.to_vec();
@@ -1390,6 +1440,23 @@ mod tests {
assert_eq!(owned_slots, expected_slots, "fallback disk slots must match the borrowing variant");
}
#[tokio::test]
async fn owned_shuffle_preserves_fresh_put_metadata() {
let tempdir = tempfile::tempdir().expect("tempdir should be created");
let fi = FileInfo::new("bucket/object", 2, 1);
let parts = vec![fi.clone(); fi.erasure.distribution.len()];
let disks = shuffle_test_disks(&tempdir, parts.len()).await;
let (owned_disks, owned_parts) = SetDisks::shuffle_disks_and_parts_metadata_by_index_owned(disks, parts, &fi);
assert!(owned_disks.iter().all(Option::is_some), "fresh PUT must retain every online disk");
assert_eq!(
owned_parts,
vec![fi; owned_disks.len()],
"fresh PUT metadata with pending shard indexes must survive init fallback"
);
}
// backlog#949: corrupt/adversarial distribution values (0 or > N) must not
// trigger a `usize` underflow / out-of-bounds panic in the shuffle helpers.
#[test]
@@ -1419,6 +1486,22 @@ mod tests {
assert_eq!(result.len(), disks.len(), "output length must be preserved");
}
#[tokio::test]
async fn owned_disk_shuffle_matches_borrowing_variant() {
let tempdir = tempfile::tempdir().expect("tempdir should be created");
let mut disks = shuffle_test_disks(&tempdir, 4).await;
disks[1] = None;
disks[3] = None;
let distribution = [3, 1, 4, 2];
let expected = SetDisks::shuffle_disks(&disks, &distribution);
let actual = SetDisks::shuffle_disks_owned(disks, &distribution);
let expected_slots = expected.iter().map(Option::is_some).collect::<Vec<_>>();
let actual_slots = actual.iter().map(Option::is_some).collect::<Vec<_>>();
assert_eq!(actual_slots, expected_slots, "owned shuffle must preserve disk placement");
}
#[tokio::test]
async fn shuffle_disks_and_parts_metadata_survives_corrupt_distribution() {
let tempdir = tempfile::tempdir().expect("tempdir should be created");
File diff suppressed because it is too large Load Diff
+31 -2
View File
@@ -542,7 +542,8 @@ impl SetDisks {
let filter_by_etag = quorum_etag.is_some();
match Self::pick_valid_fileinfo(&parts_metadata, quorum_mod_time, quorum_etag.clone(), read_quorum as usize) {
Ok(latest_meta) => {
Ok(mut latest_meta) => {
Self::hydrate_selected_fileinfo_part_checksums(&mut latest_meta)?;
trace!(
event = EVENT_SET_DISK_HEAL,
component = LOG_COMPONENT_ECSTORE,
@@ -1424,6 +1425,33 @@ impl SetDisks {
/// post-heal tail — reclaim identically. Never fails the heal: delete errors
/// are logged and swallowed. Callers must gate this on `!opts.dry_run`.
async fn reclaim_orphan_data_dirs_best_effort(&self, bucket: &str, object: &str) {
match self.reconcile_old_data_cleanup_receipts(bucket, object).await {
Ok(removed) if removed > 0 => {
debug!(
event = EVENT_SET_DISK_HEAL,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_SET_DISK,
bucket,
object,
removed,
state = "old_data_cleanup_receipt_reconciled",
"Set disk old-data cleanup receipts reconciled"
);
}
Ok(_) => {}
Err(e) => {
warn!(
event = EVENT_SET_DISK_HEAL,
component = LOG_COMPONENT_ECSTORE,
subsystem = LOG_SUBSYSTEM_SET_DISK,
bucket,
object,
error = %e,
state = "old_data_cleanup_receipt_reconcile_failed",
"Set disk old-data cleanup receipt reconcile failed"
);
}
}
match self.reclaim_orphan_data_dirs(bucket, object).await {
Ok(removed) if removed > 0 => {
debug!(
@@ -3170,10 +3198,11 @@ mod heal_result_report_tests {
.await
.expect("object should be written");
let (fi, _, _) = set
let snapshot = set
.get_object_fileinfo(bucket, object, &opts, true, false)
.await
.expect("object metadata should resolve");
let fi = snapshot.fi();
assert_eq!(fi.erasure.parity_blocks, 0);
let data_dir = fi.data_dir.expect("non-inline object should have a data directory");
let part_path = dir.path().join(bucket).join(object).join(data_dir.to_string()).join("part.1");
+14 -3
View File
@@ -36,10 +36,21 @@ impl crate::storage_api_contracts::namespace::NamespaceLocking for SetDisks {
// test's transient DistErasure window) would push this set's namespace
// locking onto its own — possibly empty — dist locker list.
let set_lock = if self.ctx.is_dist_erasure().await {
// Calculate quorum based on lockers count (majority)
let lockers_count = self.lockers.len();
let lockers = if self.lockers.len() == self.shared_lockers.len()
&& self
.lockers
.iter()
.zip(self.shared_lockers.iter())
.all(|(current, shared)| Arc::ptr_eq(current, shared))
{
self.shared_lockers.clone()
} else {
Arc::from(self.lockers.clone())
};
// Calculate quorum from the exact client domain used by this lock.
let lockers_count = lockers.len();
let write_quorum = if lockers_count > 1 { (lockers_count / 2) + 1 } else { 1 };
NamespaceLock::with_clients_and_quorum_shared(self.set_lock_namespace.clone(), self.lockers.clone(), write_quorum)
NamespaceLock::with_clients_and_quorum_shared(self.set_lock_namespace.clone(), lockers, write_quorum)
} else {
NamespaceLock::with_local_manager_shared(self.set_lock_namespace.clone(), self.local_lock_manager.clone())
};
File diff suppressed because it is too large Load Diff
File diff suppressed because it is too large Load Diff
+238 -68
View File
@@ -19,7 +19,7 @@ use crate::diagnostics::get::{
GET_METADATA_CACHE_REASON_DATA_MOVEMENT, GET_METADATA_CACHE_REASON_DELETE_MARKER, GET_METADATA_CACHE_REASON_DIST_ERASURE,
GET_METADATA_CACHE_REASON_INCL_FREE_VERSIONS, GET_METADATA_CACHE_REASON_INSUFFICIENT_CACHED_QUORUM,
GET_METADATA_CACHE_REASON_META_BUCKET, GET_METADATA_CACHE_REASON_NO_LOCK, GET_METADATA_CACHE_REASON_NOT_FOUND_OR_EXPIRED,
GET_METADATA_CACHE_REASON_NOT_READ_DATA, GET_METADATA_CACHE_REASON_PART_NUMBER,
GET_METADATA_CACHE_REASON_NOT_READ_DATA, GET_METADATA_CACHE_REASON_PART_CHECKSUMS, GET_METADATA_CACHE_REASON_PART_NUMBER,
GET_METADATA_CACHE_REASON_RAW_DATA_MOVEMENT_READ, GET_METADATA_CACHE_REASON_STALE_PUBLICATION,
GET_METADATA_CACHE_REASON_USABLE, GET_METADATA_CACHE_REASON_VERSION_ID, GET_METADATA_CACHE_REASON_VERSION_SUSPENDED,
GET_METADATA_CACHE_REASON_VERSIONED, GET_METADATA_EARLY_STOP_REASON_CONFLICTING_METADATA,
@@ -180,9 +180,9 @@ impl SetDisks {
let key = GetObjectMetadataCacheKey::new(bucket, object, generation);
let entry = Arc::new(GetObjectMetadataCacheEntry {
created_at: Instant::now(),
fi: Arc::new(fi.clone()),
parts_metadata: Arc::new(parts_metadata.to_vec()),
online_disks: Arc::new(online_disks.to_vec()),
fi: fi.clone(),
parts_metadata: parts_metadata.to_vec(),
online_disks: online_disks.to_vec(),
read_quorum,
});
self.insert_get_object_metadata_cache_entry_after_insert(key, generation, entry, || {})
@@ -300,11 +300,7 @@ impl SetDisks {
GET_STAGE_METADATA_CACHE_LOOKUP,
metadata_cache_lookup_start,
);
return Ok((
GetObjectMetadata::Shared(Arc::clone(&cached.fi)),
GetObjectMetadata::Shared(Arc::clone(&cached.parts_metadata)),
GetObjectMetadata::Shared(Arc::clone(&cached.online_disks)),
));
return Ok(GetObjectFileInfo::shared(cached));
}
MetadataCacheLookup::Miss => {
rustfs_io_metrics::record_get_object_metadata_cache_decision(
@@ -340,7 +336,7 @@ impl SetDisks {
// read_all_fileinfo_observed (see read_all_fileinfo_early_stop in
// core/io_primitives.rs); unsafe requests and callers that opt out
// (allow_early_stop=false) fall back to full-wait.
let (parts_metadata, errs, metadata_fanout_diagnostics) = Self::read_all_fileinfo_observed(
let (mut parts_metadata, errs, metadata_fanout_diagnostics) = Self::read_all_fileinfo_observed(
&disks,
"",
bucket,
@@ -394,8 +390,17 @@ impl SetDisks {
return Err(to_object_err(err.into(), vec![bucket, object]));
}
let (op_online_disks, fi, fileinfo_selection_quorum) =
let (op_online_disks, mut fi, fileinfo_selection_quorum) =
Self::select_valid_fileinfo(&disks, &parts_metadata, &errs, vid.as_str(), read_quorum, write_quorum)?;
let include_part_checksums =
opts.include_part_checksums || opts.part_number.is_some() || opts.data_movement || opts.raw_data_movement_read;
if include_part_checksums {
Self::hydrate_selected_fileinfo_part_checksums(&mut fi)?;
} else {
for metadata in std::iter::once(&mut fi).chain(parts_metadata.iter_mut()) {
rustfs_utils::http::remove_str(&mut metadata.metadata, rustfs_utils::http::SUFFIX_PART_CHECKSUMS);
}
}
metadata_fanout_diagnostics.record_quorum_candidate_latency(metadata_metrics_path, fileinfo_selection_quorum);
if errs.iter().any(|err| err.is_some()) {
let version_id = resolved_read_repair_version_id(&fi, opts.version_id.as_deref());
@@ -427,11 +432,7 @@ impl SetDisks {
// let online_disks: Vec<Option<DiskStore>> = op_online_disks.iter().filter(|v| v.is_some()).cloned().collect();
Ok((
GetObjectMetadata::Owned(fi),
GetObjectMetadata::Owned(parts_metadata),
GetObjectMetadata::Owned(op_online_disks),
))
Ok(GetObjectFileInfo::owned(fi, parts_metadata, op_online_disks))
}
#[hotpath::measure(impl_type = "SetDisks")]
@@ -441,14 +442,15 @@ impl SetDisks {
object: &str,
opts: &ObjectOptions,
) -> (ObjectInfo, usize, Option<StorageError>) {
let fi = match self.get_object_fileinfo(bucket, object, opts, false, false).await {
Ok((fi, _, _)) => fi,
let snapshot = match self.get_object_fileinfo(bucket, object, opts, false, false).await {
Ok(snapshot) => snapshot,
Err(e) => return (ObjectInfo::default(), 0, Some(e)),
};
let fi = snapshot.fi();
let write_quorum = fi.write_quorum(self.default_write_quorum());
let oi = ObjectInfo::from_file_info(&fi, bucket, object, opts.versioned || opts.version_suspended);
let oi = ObjectInfo::from_file_info(fi, bucket, object, opts.versioned || opts.version_suspended);
if !fi.version_purge_status().is_empty() && opts.version_id.is_some() {
return (
@@ -480,6 +482,7 @@ impl SetDisks {
pub(super) async fn try_get_object_direct_data_shards_with_fileinfo(
bucket: &str,
object: &str,
erasure_cache: Arc<ErasureCache>,
fi: &FileInfo,
files: &[FileInfo],
disks: &[Option<DiskStore>],
@@ -500,13 +503,7 @@ impl SetDisks {
return Ok(None);
}
let erasure = coding::Erasure::try_new_with_options(
fi.erasure.data_blocks,
fi.erasure.parity_blocks,
fi.erasure.block_size,
fi.uses_legacy_checksum,
)
.map_err(Error::from)?;
let erasure = erasure_cache.get_for_file_info(fi)?;
let checksum_info = fi.erasure.get_checksum_info(part.number);
let checksum_algo = if fi.uses_legacy_checksum && checksum_info.algorithm == HashAlgorithm::HighwayHash256S {
@@ -634,6 +631,7 @@ impl SetDisks {
// &self,
bucket: &str,
object: &str,
erasure_cache: Arc<ErasureCache>,
offset: usize,
length: i64,
writer: &mut W,
@@ -728,13 +726,7 @@ impl SetDisks {
object, offset, length, end_offset, part_index, last_part_index, last_part_relative_offset, "Multipart read bounds"
);
let erasure = coding::Erasure::try_new_with_options(
fi.erasure.data_blocks,
fi.erasure.parity_blocks,
fi.erasure.block_size,
fi.uses_legacy_checksum,
)
.map_err(Error::from)?;
let erasure = erasure_cache.get_for_file_info(&fi)?;
let part_indices: Vec<usize> = (part_index..=last_part_index).collect();
debug!(bucket, object, ?part_indices, "Multipart part indices to stream");
@@ -1168,6 +1160,7 @@ impl SetDisks {
pub(super) async fn get_object_decode_reader_with_fileinfo(
bucket: &str,
object: &str,
erasure_cache: Arc<ErasureCache>,
fi: &FileInfo,
files: &[FileInfo],
disks: &[Option<DiskStore>],
@@ -1178,14 +1171,7 @@ impl SetDisks {
metrics_size_bucket: &'static str,
prefer_data_blocks_first_reader_setup: bool,
) -> Result<GetCodecStreamingReaderBuildOutcome> {
let erasure = coding::Erasure::try_new_with_options(
fi.erasure.data_blocks,
fi.erasure.parity_blocks,
fi.erasure.block_size,
fi.uses_legacy_checksum,
)
.map_err(Error::from)?;
let erasure = erasure_cache.get_for_file_info(fi)?;
let (disks, files) = Self::shuffle_disks_and_parts_metadata_by_index(disks, files, fi);
if fi.parts.len() == 1 {
@@ -1572,7 +1558,7 @@ struct LazyCodecPartContext {
fi: FileInfo,
files: Vec<FileInfo>,
disks: Vec<Option<DiskStore>>,
erasure: coding::Erasure,
erasure: Arc<coding::Erasure>,
skip_verify_bitrot: bool,
metrics_object_class: &'static str,
metrics_size_bucket: &'static str,
@@ -1826,6 +1812,9 @@ fn get_object_metadata_cache_request_bypass_reason(bucket: &str, opts: &ObjectOp
if opts.part_number.is_some() {
return Some(GET_METADATA_CACHE_REASON_PART_NUMBER);
}
if opts.include_part_checksums {
return Some(GET_METADATA_CACHE_REASON_PART_CHECKSUMS);
}
if opts.data_movement {
return Some(GET_METADATA_CACHE_REASON_DATA_MOVEMENT);
}
@@ -2053,6 +2042,7 @@ mod metadata_cache_tests {
let err = SetDisks::get_object_with_fileinfo(
"bucket",
"object",
Arc::new(ErasureCache::new()),
0,
1,
&mut output,
@@ -2083,6 +2073,7 @@ mod metadata_cache_tests {
let err = SetDisks::get_object_with_fileinfo(
bucket,
object,
Arc::new(ErasureCache::new()),
2,
1,
&mut output,
@@ -2106,6 +2097,7 @@ mod metadata_cache_tests {
let err = SetDisks::get_object_with_fileinfo(
bucket,
object,
Arc::new(ErasureCache::new()),
usize::MAX,
1,
&mut output,
@@ -2127,6 +2119,7 @@ mod metadata_cache_tests {
let err = SetDisks::get_object_with_fileinfo(
bucket,
object,
Arc::new(ErasureCache::new()),
1,
1,
&mut output,
@@ -2150,6 +2143,7 @@ mod metadata_cache_tests {
let err = SetDisks::get_object_with_fileinfo(
bucket,
object,
Arc::new(ErasureCache::new()),
0,
1,
&mut output,
@@ -2187,6 +2181,7 @@ mod metadata_cache_tests {
SetDisks::get_object_with_fileinfo(
bucket,
object,
Arc::new(ErasureCache::new()),
0,
0,
&mut output,
@@ -2219,6 +2214,7 @@ mod metadata_cache_tests {
let err = SetDisks::get_object_with_fileinfo(
bucket,
object,
Arc::new(ErasureCache::new()),
0,
1,
&mut output,
@@ -2497,6 +2493,16 @@ mod metadata_cache_tests {
Some(GET_METADATA_CACHE_REASON_PART_NUMBER)
);
opts = ObjectOptions {
include_part_checksums: true,
..Default::default()
};
assert!(!is_get_object_metadata_cache_request_eligible("bucket", &opts, true));
assert_eq!(
get_object_metadata_cache_request_bypass_reason("bucket", &opts, true),
Some(GET_METADATA_CACHE_REASON_PART_CHECKSUMS)
);
opts = ObjectOptions {
data_movement: true,
..Default::default()
@@ -2701,22 +2707,14 @@ mod metadata_cache_tests {
.await
.expect("fresh cache entry should be returned");
let (returned_fi, returned_parts_metadata, returned_online_disks) = set
let returned = set
.get_object_fileinfo("bucket", "object", &ObjectOptions::default(), true, false)
.await
.expect("cache-backed metadata lookup should succeed");
assert!(
matches!(returned_fi, GetObjectMetadata::Shared(ref value) if Arc::ptr_eq(value, &cached.fi)),
"cache hits must share FileInfo ownership"
);
assert!(
matches!(returned_parts_metadata, GetObjectMetadata::Shared(ref value) if Arc::ptr_eq(value, &cached.parts_metadata)),
"cache hits must share the metadata vector"
);
assert!(
matches!(returned_online_disks, GetObjectMetadata::Shared(ref value) if Arc::ptr_eq(value, &cached.online_disks)),
"cache hits must share the online-disk vector"
returned.shared_entry().is_some_and(|value| Arc::ptr_eq(value, &cached)),
"cache hits must share the complete metadata snapshot"
);
}
@@ -2759,9 +2757,9 @@ mod metadata_cache_tests {
),
Arc::new(GetObjectMetadataCacheEntry {
created_at: Instant::now(),
fi: Arc::new(fi.clone()),
parts_metadata: Arc::new(vec![fi]),
online_disks: Arc::new(vec![None]),
fi: fi.clone(),
parts_metadata: vec![fi],
online_disks: vec![None],
read_quorum: 1,
}),
)
@@ -2855,13 +2853,12 @@ mod metadata_cache_tests {
barrier.wait_until_paused().await;
set.invalidate_get_object_metadata_cache(bucket, object).await;
barrier.release();
let (fi, parts_metadata, online_disks) = read
let snapshot = read
.await
.expect("metadata read task should not panic")
.expect("metadata fanout should still return its selected FileInfo");
assert!(matches!(fi, GetObjectMetadata::Owned(_)));
assert!(matches!(parts_metadata, GetObjectMetadata::Owned(_)));
assert!(matches!(online_disks, GetObjectMetadata::Owned(_)));
assert!(snapshot.owned.is_some());
assert!(snapshot.has_valid_representation());
assert!(
set.get_object_metadata_cache
@@ -2908,9 +2905,9 @@ mod metadata_cache_tests {
let key = GetObjectMetadataCacheKey::new("bucket", "object", generation);
let entry = Arc::new(GetObjectMetadataCacheEntry {
created_at: Instant::now(),
fi: Arc::new(fi.clone()),
parts_metadata: Arc::new(vec![fi]),
online_disks: Arc::new(Vec::new()),
fi: fi.clone(),
parts_metadata: vec![fi],
online_disks: Vec::new(),
read_quorum: 0,
});
@@ -3018,9 +3015,9 @@ mod metadata_cache_tests {
let entry = |fi: FileInfo| {
Arc::new(GetObjectMetadataCacheEntry {
created_at: Instant::now(),
parts_metadata: Arc::new(vec![fi.clone()]),
fi: Arc::new(fi),
online_disks: Arc::new(Vec::new()),
parts_metadata: vec![fi.clone()],
fi,
online_disks: Vec::new(),
read_quorum: 0,
})
};
@@ -4083,8 +4080,8 @@ mod tests {
get_codec_streaming_reader_gate(
CODEC_STREAMING_TEST_BUCKET,
CODEC_STREAMING_TEST_OBJECT,
range,
None,
classify_get_codec_streaming_object_class(range, object_info, fi),
object_info,
fi,
lock_optimization_enabled,
@@ -4101,8 +4098,8 @@ mod tests {
get_codec_streaming_reader_gate(
CODEC_STREAMING_TEST_BUCKET,
CODEC_STREAMING_TEST_OBJECT,
range,
part_number,
classify_get_codec_streaming_object_class(range, object_info, fi),
object_info,
fi,
lock_optimization_enabled,
@@ -4122,6 +4119,7 @@ mod tests {
let result = SetDisks::get_object_decode_reader_with_fileinfo(
CODEC_STREAMING_TEST_BUCKET,
CODEC_STREAMING_TEST_OBJECT,
Arc::new(ErasureCache::new()),
&fi,
&[],
&[],
@@ -4144,6 +4142,7 @@ mod tests {
let invalid_size = SetDisks::get_object_decode_reader_with_fileinfo(
CODEC_STREAMING_TEST_BUCKET,
CODEC_STREAMING_TEST_OBJECT,
Arc::new(ErasureCache::new()),
&single_part,
&[],
&[],
@@ -4164,6 +4163,7 @@ mod tests {
SetDisks::get_object_decode_reader_with_fileinfo(
CODEC_STREAMING_TEST_BUCKET,
CODEC_STREAMING_TEST_OBJECT,
Arc::new(ErasureCache::new()),
&multipart,
&[],
&[],
@@ -4188,6 +4188,7 @@ mod tests {
SetDisks::get_object_decode_reader_with_fileinfo(
CODEC_STREAMING_TEST_BUCKET,
CODEC_STREAMING_TEST_OBJECT,
Arc::new(ErasureCache::new()),
&multipart,
&[],
&[],
@@ -4216,6 +4217,7 @@ mod tests {
SetDisks::get_object_decode_reader_with_fileinfo(
CODEC_STREAMING_TEST_BUCKET,
CODEC_STREAMING_TEST_OBJECT,
Arc::new(ErasureCache::new()),
&multipart,
&[],
&[],
@@ -4269,6 +4271,7 @@ mod tests {
SetDisks::get_object_decode_reader_with_fileinfo(
CODEC_STREAMING_TEST_BUCKET,
CODEC_STREAMING_TEST_OBJECT,
Arc::new(ErasureCache::new()),
&fi,
&files,
&disks,
@@ -4322,6 +4325,7 @@ mod tests {
SetDisks::get_object_decode_reader_with_fileinfo(
CODEC_STREAMING_TEST_BUCKET,
CODEC_STREAMING_TEST_OBJECT,
Arc::new(ErasureCache::new()),
&fi,
&files,
&disks,
@@ -4366,6 +4370,7 @@ mod tests {
SetDisks::get_object_with_fileinfo(
CODEC_STREAMING_TEST_BUCKET,
CODEC_STREAMING_TEST_OBJECT,
Arc::new(ErasureCache::new()),
0,
part_data.len() as i64,
&mut output,
@@ -4828,6 +4833,114 @@ mod tests {
.await
}
async fn encoded_inline_blocks(blocks: &[&[u8]], shard_size: usize, hash_algo: HashAlgorithm) -> Bytes {
let mut writer = BitrotWriter::new(Cursor::new(Vec::new()), shard_size, hash_algo);
for block in blocks {
writer.write(block).await.expect("test block should be encoded");
}
Bytes::from(writer.into_inner().into_inner())
}
fn assert_reader_shares_inline_allocation(reader: &ObjectBitrotReader, source: &Bytes) {
let reader_bytes = reader
.inner_ref()
.inline_bytes()
.expect("inline scheduler should retain an in-memory Bytes source");
assert_eq!(
reader_bytes.as_ptr(),
source.as_ptr(),
"the scheduler must clone Bytes ownership instead of copying the inline shard payload"
);
}
#[tokio::test]
async fn inline_range_scheduler_shares_bytes_and_rejects_bitrot_mismatch() {
const SHARD_SIZE: usize = 16;
let hash_algo = HashAlgorithm::HighwayHash256S;
let first = [b'a'; SHARD_SIZE];
let second = [b'b'; SHARD_SIZE];
let mut source = encoded_inline_blocks(&[&first, &second], SHARD_SIZE, hash_algo.clone()).await;
let second_payload = hash_algo.size() * 2 + SHARD_SIZE;
source = {
let mut corrupt = source.to_vec();
corrupt[second_payload] ^= 0xff;
Bytes::from(corrupt)
};
let files = vec![encoded_reader_setup_fileinfo(Some(source.to_vec()))];
let source = files[0].data.clone().expect("inline shard should exist");
let disks = vec![None];
let mut setup = create_bitrot_readers_until_quorum_with_preference(
&files,
&disks,
"bucket",
"object",
1,
SHARD_SIZE,
SHARD_SIZE,
SHARD_SIZE,
hash_algo,
false,
false,
1,
0,
BitrotReaderSetupMode::ReadQuorum,
true,
None,
None,
)
.await;
let mut reader = setup.readers[0].take().expect("range reader should be ready");
assert_reader_shares_inline_allocation(&reader, &source);
let err = reader
.read(&mut [0; SHARD_SIZE])
.await
.expect_err("corrupt ranged inline block must fail bitrot verification");
assert_eq!(err.kind(), ErrorKind::InvalidData);
}
#[tokio::test]
async fn inline_part_scheduler_shares_bytes_and_rejects_bitrot_mismatch() {
const SHARD_SIZE: usize = 16;
let hash_algo = HashAlgorithm::HighwayHash256S;
let block = [b'p'; SHARD_SIZE];
let encoded = encoded_inline_blocks(&[&block], SHARD_SIZE, hash_algo.clone()).await;
let mut corrupt = encoded.to_vec();
corrupt[hash_algo.size()] ^= 0xff;
let files = vec![encoded_reader_setup_fileinfo(Some(corrupt))];
let source = files[0].data.clone().expect("inline shard should exist");
let disks = vec![None];
let mut setup = create_bitrot_readers_until_quorum_all_shards(
&files,
&disks,
"bucket",
"object",
7,
0,
SHARD_SIZE,
SHARD_SIZE,
hash_algo,
false,
false,
1,
0,
BitrotReaderSetupMode::VerifyReconstruction,
None,
None,
)
.await;
let mut reader = setup.readers[0].take().expect("part reader should be ready");
assert_reader_shares_inline_allocation(&reader, &source);
let err = reader
.read(&mut [0; SHARD_SIZE])
.await
.expect_err("corrupt inline part must fail bitrot verification");
assert_eq!(err.kind(), ErrorKind::InvalidData);
}
async fn decode_codec_data_blocks_first_setup(
erasure: coding::Erasure,
data: &[u8],
@@ -5528,6 +5641,63 @@ mod tests {
});
}
#[test]
fn codec_streaming_config_cache_loads_once() {
use std::cell::Cell;
let loads = Cell::new(0);
let expected = GetCodecStreamingConfig {
enabled: true,
rollout: GetCodecStreamingRollout::Off,
rollout_pct: 100,
body_compat_confirmed: true,
header_compat_confirmed: true,
engine: GetCodecStreamingEngine::Legacy,
min_size: DEFAULT_RUSTFS_GET_CODEC_STREAMING_MIN_SIZE,
};
for _ in 0..3 {
assert_eq!(
get_codec_streaming_config_cached_core(|| {
loads.set(loads.get() + 1);
expected
}),
expected
);
}
assert_eq!(loads.get(), 1, "production config cache must not reload env per GET");
}
#[test]
fn codec_streaming_config_loader_preserves_all_gate_env_overrides() {
temp_env::with_vars(
[
(ENV_RUSTFS_GET_CODEC_STREAMING_ENABLE, Some("false")),
(ENV_RUSTFS_GET_CODEC_STREAMING_ENGINE, Some(GET_CODEC_STREAMING_ENGINE_RUSTFS)),
(ENV_RUSTFS_GET_CODEC_STREAMING_ROLLOUT, Some("production")),
(ENV_RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT, Some("37")),
(ENV_RUSTFS_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED, Some("false")),
(ENV_RUSTFS_GET_CODEC_STREAMING_HEADER_COMPAT_CONFIRMED, Some("false")),
(ENV_RUSTFS_GET_CODEC_STREAMING_MIN_SIZE, None::<&str>),
(ENV_RUSTFS_GET_CODEC_STREAMING_RUSTFS_MIN_SIZE, Some("262144")),
],
|| {
assert_eq!(
load_get_codec_streaming_config(),
GetCodecStreamingConfig {
enabled: false,
rollout: GetCodecStreamingRollout::On,
rollout_pct: 37,
body_compat_confirmed: false,
header_compat_confirmed: false,
engine: GetCodecStreamingEngine::Rustfs,
min_size: 262144,
}
);
},
);
}
#[test]
fn codec_streaming_default_min_size_meets_direct_memory_ceiling() {
for engine in [None, Some(GET_CODEC_STREAMING_ENGINE_RUSTFS)] {
+27 -6
View File
@@ -48,6 +48,23 @@ impl RestoreCleanupIdentity {
}
}
fn ensure_restore_metadata_lock_held(bucket: &str, object: &str, opts: &ObjectOptions, mode: &'static str) -> Result<()> {
if opts
.namespace_lock_fence
.as_ref()
.is_some_and(NamespaceLockFence::is_lock_lost)
{
return Err(StorageError::NamespaceLockQuorumUnavailable {
mode,
bucket: bucket.to_string(),
object: object.to_string(),
required: 1,
achieved: 0,
});
}
Ok(())
}
impl SetDisks {
pub(super) async fn finalize_restore_metadata(
&self,
@@ -75,18 +92,20 @@ impl SetDisks {
version_id,
versioned: opts.versioned,
version_suspended: opts.version_suspended,
include_part_checksums: true,
..Default::default()
};
let (fi, _, disks) = self
let (mut fi, _, disks) = self
.get_object_fileinfo_gated(bucket, object, &read_opts, false, false)
.await?;
let mut fi = fi.into_owned();
.await?
.into_owned();
if let Some(expected_operation_id) = expected_operation_id {
require_restore_operation_id(&fi.metadata, expected_operation_id)?;
}
if !expected.matches_file_info(&fi, &expected_etag) {
return Err(Error::other("restored object changed before restore metadata finalization"));
}
ensure_restore_metadata_lock_held(bucket, object, opts, "restore_finalize_metadata")?;
let restore_expiry =
lifecycle::expected_expiry_time(OffsetDateTime::now_utc(), opts.transition.restore_request.days.unwrap_or(1));
fi.metadata.insert(
@@ -142,12 +161,13 @@ impl SetDisks {
version_id,
versioned: opts.versioned,
version_suspended: opts.version_suspended,
include_part_checksums: true,
..Default::default()
};
let (fi, _, disks) = self
let (mut fi, _, disks) = self
.get_object_fileinfo_gated(bucket, object, &read_opts, false, false)
.await?;
let mut fi = fi.into_owned();
.await?
.into_owned();
if let Some(expected_operation_id) = expected_operation_id {
match restore_operation_id_from_metadata(&fi.metadata)? {
Some(actual_operation_id) if actual_operation_id == expected_operation_id => {}
@@ -157,6 +177,7 @@ impl SetDisks {
if !expected.matches_file_info(&fi, &expected_etag) {
return Ok(());
}
ensure_restore_metadata_lock_held(bucket, object, opts, "restore_cleanup_metadata")?;
fi.metadata.remove(X_AMZ_RESTORE.as_str());
fi.metadata.remove(AMZ_RESTORE_EXPIRY_DAYS);
fi.metadata.remove(AMZ_RESTORE_REQUEST_DATE);
+94 -159
View File
@@ -16,6 +16,14 @@ use crate::diagnostics::get::{
GET_SHARD_READ_COST_LOCAL, GET_SHARD_READ_COST_REMOTE, GET_SHARD_READ_COST_SAME_NODE, GET_SHARD_READ_COST_UNKNOWN,
};
use crate::disk::error::Error;
use crate::layout::disks_layout::MAX_ERASURE_SET_DRIVE_COUNT;
use smallvec::SmallVec;
/// Generic codec callers may exceed the production set limit; `SmallVec` then
/// spills without changing slot semantics.
pub(crate) const INLINE_SHARD_SLOTS: usize = MAX_ERASURE_SET_DRIVE_COUNT;
pub(crate) type ShardBuffers = SmallVec<[Option<Vec<u8>>; INLINE_SHARD_SLOTS]>;
pub(crate) type ShardErrors = SmallVec<[Option<Error>; INLINE_SHARD_SLOTS]>;
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub(crate) enum ShardReadCost {
@@ -43,202 +51,137 @@ impl ShardReadCost {
}
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub(crate) struct ShardSlot {
index: usize,
read_cost: ShardReadCost,
data: Option<Vec<u8>>,
error: Option<Error>,
}
impl ShardSlot {
pub(crate) fn new(index: usize, data: Option<Vec<u8>>, error: Option<Error>) -> Self {
Self::with_read_cost(index, ShardReadCost::Unknown, data, error)
}
pub(crate) fn with_read_cost(index: usize, read_cost: ShardReadCost, data: Option<Vec<u8>>, error: Option<Error>) -> Self {
Self {
index,
read_cost,
data,
error,
}
}
pub(crate) fn data(index: usize, data: Vec<u8>) -> Self {
Self::new(index, Some(data), None)
}
pub(crate) fn data_with_read_cost(index: usize, read_cost: ShardReadCost, data: Vec<u8>) -> Self {
Self::with_read_cost(index, read_cost, Some(data), None)
}
pub(crate) fn missing(index: usize, error: Error) -> Self {
Self::new(index, None, Some(error))
}
pub(crate) fn missing_with_read_cost(index: usize, read_cost: ShardReadCost, error: Error) -> Self {
Self::with_read_cost(index, read_cost, None, Some(error))
}
pub(crate) fn index(&self) -> usize {
self.index
}
pub(crate) fn read_cost(&self) -> ShardReadCost {
self.read_cost
}
pub(crate) fn has_data(&self) -> bool {
self.data.is_some()
}
pub(crate) fn data_bytes(&self) -> Option<&[u8]> {
self.data.as_deref()
}
pub(crate) fn error(&self) -> Option<&Error> {
self.error.as_ref()
}
}
#[derive(Debug, Clone, PartialEq, Eq)]
pub(crate) struct StripeReadState {
slots: Vec<ShardSlot>,
shards: ShardBuffers,
errors: ShardErrors,
read_quorum: usize,
}
impl StripeReadState {
pub(crate) fn new(slots: Vec<ShardSlot>, read_quorum: usize) -> Self {
Self { slots, read_quorum }
}
#[cfg(test)]
pub(crate) fn from_parts(shards: Vec<Option<Vec<u8>>>, errors: Vec<Option<Error>>, read_quorum: usize) -> Self {
Self::from_parts_with_read_costs(shards, errors, &[], read_quorum)
let mut shards = SmallVec::from_vec(shards);
let mut errors = SmallVec::from_vec(errors);
let slot_count = shards.len().max(errors.len());
shards.resize_with(slot_count, || None);
errors.resize_with(slot_count, || None);
Self {
shards,
errors,
read_quorum,
}
}
pub(crate) fn from_parts_with_read_costs<S, E>(shards: S, errors: E, read_costs: &[ShardReadCost], read_quorum: usize) -> Self
where
S: IntoIterator<Item = Option<Vec<u8>>>,
S::IntoIter: ExactSizeIterator,
E: IntoIterator<Item = Option<Error>>,
E::IntoIter: ExactSizeIterator,
{
let mut shards = shards.into_iter();
let mut errors = errors.into_iter();
let slot_count = shards.len().max(errors.len());
let mut slots = Vec::with_capacity(slot_count);
for index in 0..slot_count {
let read_cost = read_costs.get(index).copied().unwrap_or(ShardReadCost::Unknown);
slots.push(ShardSlot::with_read_cost(
index,
read_cost,
shards.next().flatten(),
errors.next().flatten(),
));
}
Self::new(slots, read_quorum)
pub(crate) fn with_slot_count(slot_count: usize, read_quorum: usize) -> Self {
let mut state = Self {
shards: SmallVec::new(),
errors: SmallVec::new(),
read_quorum,
};
state.reset(slot_count, read_quorum);
state
}
pub(crate) fn reset(&mut self, slot_count: usize, read_quorum: usize) {
self.shards.clear();
self.shards.resize_with(slot_count, || None);
self.errors.clear();
self.errors.resize_with(slot_count, || None);
self.read_quorum = read_quorum;
}
pub(crate) fn available_shards(&self) -> usize {
self.slots.iter().filter(|slot| slot.has_data()).count()
self.shards.iter().filter(|shard| shard.is_some()).count()
}
pub(crate) fn can_decode(&self) -> bool {
self.available_shards() >= self.read_quorum
}
pub(crate) fn slots(&self) -> &[ShardSlot] {
&self.slots
pub(crate) fn is_empty(&self) -> bool {
self.shards.is_empty()
}
pub(crate) fn slot_by_index(&self, index: usize) -> Option<&ShardSlot> {
if let Some(slot) = self.slots.get(index)
&& slot.index == index
{
return Some(slot);
}
self.slots.iter().find(|slot| slot.index == index)
pub(crate) fn data_bytes(&self, index: usize) -> Option<&[u8]> {
self.shards.get(index).and_then(Option::as_deref)
}
#[cfg(test)]
pub(crate) fn error(&self, index: usize) -> Option<&Error> {
self.errors.get(index).and_then(Option::as_ref)
}
pub(crate) fn data_shards_complete(&self, data_shards: usize) -> bool {
(0..data_shards).all(|index| self.slot_by_index(index).is_some_and(ShardSlot::has_data))
self.shards.len() >= data_shards && self.shards.iter().take(data_shards).all(Option::is_some)
}
pub(crate) fn into_parts(self) -> (Vec<Option<Vec<u8>>>, Vec<Option<Error>>) {
let part_count = self.slots.iter().map(|slot| slot.index).max().map_or(0, |index| index + 1);
let mut shards = Vec::with_capacity(part_count);
shards.resize_with(part_count, || None);
let mut errors = Vec::with_capacity(part_count);
errors.resize_with(part_count, || None);
for slot in self.slots {
shards[slot.index] = slot.data;
errors[slot.index] = slot.error;
}
(shards, errors)
pub(crate) fn parts_mut(&mut self) -> (&mut ShardBuffers, &mut ShardErrors) {
(&mut self.shards, &mut self.errors)
}
pub(crate) fn shards_mut(&mut self) -> &mut ShardBuffers {
&mut self.shards
}
pub(crate) fn into_parts(self) -> (ShardBuffers, ShardErrors) {
(self.shards, self.errors)
}
#[cfg(test)]
pub(crate) fn scratch_storage(&self) -> (*const Option<Vec<u8>>, *const Option<Error>, bool, bool) {
(self.shards.as_ptr(), self.errors.as_ptr(), self.shards.spilled(), self.errors.spilled())
}
#[cfg(test)]
pub(crate) fn shard_allocation(&self, index: usize) -> Option<(*const u8, usize)> {
self.shards
.get(index)
.and_then(|shard| shard.as_ref().map(|shard| (shard.as_ptr(), shard.capacity())))
}
}
#[async_trait::async_trait]
pub(crate) trait ShardStripeSource: Send {
async fn read_next_stripe(&mut self) -> StripeReadState;
async fn read_next_stripe(&mut self) -> Box<StripeReadState>;
fn recycle_stripe(&mut self, _state: Box<StripeReadState>) {}
}
#[cfg(test)]
mod tests {
use super::*;
use std::mem::size_of;
#[test]
fn stripe_read_state_tracks_decode_quorum() {
let state = StripeReadState::new(
vec![
ShardSlot::data_with_read_cost(0, ShardReadCost::Local, vec![1]),
ShardSlot::missing_with_read_cost(1, ShardReadCost::Remote, Error::FileNotFound),
ShardSlot::data_with_read_cost(2, ShardReadCost::SameNode, vec![2]),
],
2,
);
fn stripe_scratch_capacity_matches_the_production_set_limit() {
type OversizedShardBuffers = SmallVec<[Option<Vec<u8>>; 32]>;
type OversizedShardErrors = SmallVec<[Option<Error>; 32]>;
assert_eq!(INLINE_SHARD_SLOTS, MAX_ERASURE_SET_DRIVE_COUNT);
assert!(size_of::<ShardBuffers>() < size_of::<OversizedShardBuffers>());
assert!(size_of::<ShardErrors>() < size_of::<OversizedShardErrors>());
}
#[test]
fn stripe_read_state_tracks_decode_quorum_and_slot_access() {
let state =
StripeReadState::from_parts(vec![Some(vec![1]), None, Some(vec![2])], vec![None, Some(Error::FileNotFound), None], 2);
assert_eq!(state.available_shards(), 2);
assert!(state.can_decode());
assert_eq!(state.slots()[1].index(), 1);
assert_eq!(state.slots()[0].read_cost(), ShardReadCost::Local);
assert!(state.slots()[2].read_cost().is_low_cost());
assert_eq!(state.data_bytes(0), Some(&[1][..]));
assert_eq!(state.error(1), Some(&Error::FileNotFound));
}
#[test]
fn stripe_read_state_preserves_shards_and_errors() {
let state = StripeReadState::new(vec![ShardSlot::missing(1, Error::FileCorrupt), ShardSlot::data(0, vec![1, 2, 3])], 2);
let state = StripeReadState::from_parts(vec![Some(vec![1, 2, 3]), None], vec![None, Some(Error::FileCorrupt)], 2);
assert!(!state.can_decode());
let (shards, errors) = state.into_parts();
assert_eq!(shards, vec![Some(vec![1, 2, 3]), None]);
assert_eq!(errors, vec![None, Some(Error::FileCorrupt)]);
}
#[test]
fn stripe_read_state_builds_slots_from_parallel_reader_parts() {
let state =
StripeReadState::from_parts(vec![Some(vec![1]), None, Some(vec![3])], vec![None, Some(Error::FileNotFound)], 2);
assert!(state.can_decode());
assert_eq!(state.slots()[1].index(), 1);
assert_eq!(state.slots()[1].error(), Some(&Error::FileNotFound));
}
#[test]
fn stripe_read_state_preserves_read_cost_hints() {
let state = StripeReadState::from_parts_with_read_costs(
vec![Some(vec![1]), None, Some(vec![3])],
vec![None, Some(Error::FileNotFound)],
&[ShardReadCost::Local, ShardReadCost::Remote, ShardReadCost::Unknown],
2,
);
assert_eq!(state.slots()[0].read_cost(), ShardReadCost::Local);
assert_eq!(state.slots()[1].read_cost(), ShardReadCost::Remote);
assert_eq!(state.slots()[2].read_cost(), ShardReadCost::Unknown);
assert_eq!(ShardReadCost::SameNode.as_str(), GET_SHARD_READ_COST_SAME_NODE);
assert_eq!(shards.as_slice(), &[Some(vec![1, 2, 3]), None]);
assert_eq!(errors.as_slice(), &[None, Some(Error::FileCorrupt)]);
}
#[test]
@@ -258,8 +201,8 @@ mod tests {
let state = StripeReadState::from_parts(vec![Some(vec![1]), Some(vec![2]), None], Vec::new(), 2);
assert!(state.data_shards_complete(2));
assert_eq!(state.slots()[0].data_bytes(), Some(&[1][..]));
assert_eq!(state.slot_by_index(1).and_then(ShardSlot::data_bytes), Some(&[2][..]));
assert_eq!(state.data_bytes(0), Some(&[1][..]));
assert_eq!(state.data_bytes(1), Some(&[2][..]));
}
#[test]
@@ -268,12 +211,4 @@ mod tests {
assert!(!state.data_shards_complete(2));
}
#[test]
fn stripe_read_state_finds_out_of_order_slots_by_index() {
let state = StripeReadState::new(vec![ShardSlot::data(2, vec![3]), ShardSlot::data(0, vec![1])], 2);
assert_eq!(state.slot_by_index(0).and_then(ShardSlot::data_bytes), Some(&[1][..]));
assert!(state.slot_by_index(1).is_none());
}
}
+4
View File
@@ -161,6 +161,10 @@ impl BucketIncarnationFenceGuard {
pub(crate) fn is_lock_lost(&self) -> bool {
self.inner.as_ref().is_some_and(NamespaceLockGuard::is_lock_lost)
}
pub(crate) fn namespace_lock_guard(&self) -> Option<&NamespaceLockGuard> {
self.inner.as_ref()
}
}
impl Drop for BucketIncarnationFenceGuard {
File diff suppressed because it is too large Load Diff
+28
View File
@@ -309,9 +309,17 @@ const ENV_API_LIST_OBJECTS_INDEX_PROVIDER: &str = "RUSTFS_LIST_OBJECTS_INDEX_PRO
const ENV_API_LIST_OBJECTS_INDEX_PROVIDER_PATH: &str = "RUSTFS_LIST_OBJECTS_INDEX_PROVIDER_PATH";
const ENV_API_LIST_OBJECTS_INDEX_PROVIDER_GENERATION: &str = "RUSTFS_LIST_OBJECTS_INDEX_PROVIDER_GENERATION";
const ENV_API_LIST_OBJECTS_NAMESPACE_JOURNAL_PATH: &str = "RUSTFS_LIST_OBJECTS_NAMESPACE_JOURNAL_PATH";
// The chaos machinery below is compiled only for tests and the opt-in
// `list-chaos` feature (backlog#1832): a production binary without the
// feature carries no chaos symbols, so the two env vars cannot silently
// rewrite a bucket's namespace-journal state.
#[cfg(any(test, feature = "list-chaos"))]
const ENV_API_LIST_OBJECTS_NAMESPACE_JOURNAL_CHAOS_ENABLED: &str = "RUSTFS_LIST_OBJECTS_NAMESPACE_JOURNAL_CHAOS_ENABLED";
#[cfg(any(test, feature = "list-chaos"))]
const ENV_API_LIST_OBJECTS_NAMESPACE_JOURNAL_CHAOS_BUCKET: &str = "RUSTFS_LIST_OBJECTS_NAMESPACE_JOURNAL_CHAOS_BUCKET";
#[cfg(any(test, feature = "list-chaos"))]
const ENV_API_LIST_OBJECTS_NAMESPACE_JOURNAL_CHAOS_SEQUENCE: &str = "RUSTFS_LIST_OBJECTS_NAMESPACE_JOURNAL_CHAOS_SEQUENCE";
#[cfg(any(test, feature = "list-chaos"))]
const ENV_API_LIST_OBJECTS_NAMESPACE_JOURNAL_CHAOS_STATUS: &str = "RUSTFS_LIST_OBJECTS_NAMESPACE_JOURNAL_CHAOS_STATUS";
const ENV_API_LIST_OBJECTS_METADATA_FAST_ENABLED: &str = "RUSTFS_LIST_OBJECTS_METADATA_FAST_ENABLED";
const ENV_API_LIST_OBJECTS_METADATA_FAST_STALENESS_MS: &str = "RUSTFS_LIST_OBJECTS_METADATA_FAST_STALENESS_MS";
@@ -552,7 +560,9 @@ static LIST_OBJECTS_MUTATION_SEQUENCE: AtomicU64 = AtomicU64::new(0);
static SCANNER_NAMESPACE_MUTATION_GENERATION: AtomicU64 = AtomicU64::new(0);
static LIST_OBJECTS_BUCKET_MUTATION_SEQUENCE: OnceCell<RwLock<HashMap<String, u64>>> = OnceCell::const_new();
static LIST_OBJECTS_NAMESPACE_JOURNAL_DEGRADED_BUCKETS: OnceCell<RwLock<HashSet<String>>> = OnceCell::const_new();
#[cfg(any(test, feature = "list-chaos"))]
static LIST_OBJECTS_NAMESPACE_JOURNAL_CHAOS_CONFIG: OnceCell<Option<NamespaceMutationJournalChaosConfig>> = OnceCell::const_new();
#[cfg(any(test, feature = "list-chaos"))]
static LIST_OBJECTS_NAMESPACE_JOURNAL_CHAOS_APPLIED: OnceCell<RwLock<HashSet<String>>> = OnceCell::const_new();
async fn persistent_key_only_index_cache() -> &'static RwLock<Option<PersistentKeyOnlyIndexCache>> {
@@ -579,6 +589,7 @@ async fn list_objects_namespace_journal_degraded_buckets() -> &'static RwLock<Ha
.await
}
#[cfg(any(test, feature = "list-chaos"))]
async fn list_objects_namespace_journal_chaos_config() -> Option<&'static NamespaceMutationJournalChaosConfig> {
LIST_OBJECTS_NAMESPACE_JOURNAL_CHAOS_CONFIG
.get_or_init(|| async { namespace_mutation_journal_chaos_config_from_env() })
@@ -586,6 +597,7 @@ async fn list_objects_namespace_journal_chaos_config() -> Option<&'static Namesp
.as_ref()
}
#[cfg(any(test, feature = "list-chaos"))]
async fn list_objects_namespace_journal_chaos_applied() -> &'static RwLock<HashSet<String>> {
LIST_OBJECTS_NAMESPACE_JOURNAL_CHAOS_APPLIED
.get_or_init(|| async { RwLock::new(HashSet::new()) })
@@ -681,6 +693,7 @@ enum NamespaceMutationJournalStatus {
}
impl NamespaceMutationJournalStatus {
#[cfg(any(test, feature = "list-chaos"))]
fn from_env_value(value: &str) -> Option<Self> {
if value.eq_ignore_ascii_case(LIST_OBJECTS_NAMESPACE_JOURNAL_STATUS_HEALTHY) {
Some(Self::Healthy)
@@ -691,6 +704,7 @@ impl NamespaceMutationJournalStatus {
}
}
#[cfg(any(test, feature = "list-chaos"))]
fn env_value(self) -> &'static str {
match self {
Self::Healthy => LIST_OBJECTS_NAMESPACE_JOURNAL_STATUS_HEALTHY,
@@ -712,6 +726,7 @@ struct NamespaceMutationJournalSnapshot {
degraded: bool,
}
#[cfg(any(test, feature = "list-chaos"))]
#[derive(Debug, Clone, PartialEq, Eq)]
struct NamespaceMutationJournalChaosConfig {
bucket: String,
@@ -795,30 +810,35 @@ fn list_objects_namespace_journal_root_from_env() -> Option<PathBuf> {
.filter(|path| !path.as_os_str().is_empty())
}
#[cfg(any(test, feature = "list-chaos"))]
fn namespace_mutation_journal_chaos_enabled_from_env() -> bool {
std::env::var(ENV_API_LIST_OBJECTS_NAMESPACE_JOURNAL_CHAOS_ENABLED)
.ok()
.is_some_and(|value| value == "1" || value.eq_ignore_ascii_case("on") || value.eq_ignore_ascii_case("true"))
}
#[cfg(any(test, feature = "list-chaos"))]
fn namespace_mutation_journal_chaos_bucket_from_env() -> Option<String> {
std::env::var(ENV_API_LIST_OBJECTS_NAMESPACE_JOURNAL_CHAOS_BUCKET)
.ok()
.filter(|bucket| !bucket.is_empty())
}
#[cfg(any(test, feature = "list-chaos"))]
fn namespace_mutation_journal_chaos_sequence_from_env() -> Option<u64> {
std::env::var(ENV_API_LIST_OBJECTS_NAMESPACE_JOURNAL_CHAOS_SEQUENCE)
.ok()
.and_then(|value| value.parse::<u64>().ok())
}
#[cfg(any(test, feature = "list-chaos"))]
fn namespace_mutation_journal_chaos_status_from_env() -> Option<NamespaceMutationJournalStatus> {
std::env::var(ENV_API_LIST_OBJECTS_NAMESPACE_JOURNAL_CHAOS_STATUS)
.ok()
.and_then(|value| NamespaceMutationJournalStatus::from_env_value(&value))
}
#[cfg(any(test, feature = "list-chaos"))]
fn namespace_mutation_journal_chaos_config_from_env() -> Option<NamespaceMutationJournalChaosConfig> {
if !namespace_mutation_journal_chaos_enabled_from_env() {
return None;
@@ -846,6 +866,7 @@ fn namespace_mutation_journal_chaos_config_from_env() -> Option<NamespaceMutatio
})
}
#[cfg(any(test, feature = "list-chaos"))]
fn namespace_mutation_journal_chaos_applied_key(bucket: &str, status: NamespaceMutationJournalStatus) -> String {
let mut key = String::with_capacity(bucket.len() + 1 + status.env_value().len());
key.push_str(bucket);
@@ -854,6 +875,13 @@ fn namespace_mutation_journal_chaos_applied_key(bucket: &str, status: NamespaceM
key
}
/// Production no-op twin of the chaos injector: without `list-chaos` the
/// injection point compiles to nothing (backlog#1832).
#[cfg(not(any(test, feature = "list-chaos")))]
#[inline]
async fn maybe_apply_system_namespace_mutation_journal_chaos(_store: &ECStore, _bucket: &str, _default_sequence: u64) {}
#[cfg(any(test, feature = "list-chaos"))]
async fn maybe_apply_system_namespace_mutation_journal_chaos(store: &ECStore, bucket: &str, default_sequence: u64) {
let Some(config) = list_objects_namespace_journal_chaos_config().await else {
return;
+1 -1
View File
@@ -389,7 +389,7 @@ impl crate::storage_api_contracts::object::ObjectIO for ECStore {
type GetObjectReader = GetObjectReader;
type PutObjectReader = PutObjReader;
#[instrument(level = "debug", skip(self))]
#[instrument(level = "debug", skip(self, h))]
async fn get_object_reader(
&self,
bucket: &str,
+188 -5
View File
@@ -66,6 +66,76 @@ fn ensure_multipart_bucket_lifecycle_guard_held(
Ok(())
}
#[cfg(test)]
struct DataMovementMultipartCompletionBarrierState {
bucket: String,
arrived: tokio::sync::Notify,
release: tokio::sync::Notify,
}
#[cfg(test)]
pub(crate) struct DataMovementMultipartCompletionBarrier {
state: Arc<DataMovementMultipartCompletionBarrierState>,
}
#[cfg(test)]
static DATA_MOVEMENT_MULTIPART_COMPLETION_BARRIER: std::sync::OnceLock<
std::sync::Mutex<Option<Arc<DataMovementMultipartCompletionBarrierState>>>,
> = std::sync::OnceLock::new();
#[cfg(test)]
impl DataMovementMultipartCompletionBarrier {
pub(crate) fn install(bucket: &str) -> Self {
let state = Arc::new(DataMovementMultipartCompletionBarrierState {
bucket: bucket.to_string(),
arrived: tokio::sync::Notify::new(),
release: tokio::sync::Notify::new(),
});
let mut slot = DATA_MOVEMENT_MULTIPART_COMPLETION_BARRIER
.get_or_init(|| std::sync::Mutex::new(None))
.lock()
.expect("data movement multipart completion barrier mutex should not poison");
assert!(slot.is_none(), "data movement multipart completion barrier must be unique");
*slot = Some(Arc::clone(&state));
Self { state }
}
pub(crate) async fn wait_until_paused(&self) {
tokio::time::timeout(std::time::Duration::from_secs(30), self.state.arrived.notified())
.await
.expect("data movement multipart operation should reach selected completion");
}
}
#[cfg(test)]
impl Drop for DataMovementMultipartCompletionBarrier {
fn drop(&mut self) {
self.state.release.notify_one();
let mut slot = DATA_MOVEMENT_MULTIPART_COMPLETION_BARRIER
.get_or_init(|| std::sync::Mutex::new(None))
.lock()
.expect("data movement multipart completion barrier mutex should not poison");
if slot.as_ref().is_some_and(|state| Arc::ptr_eq(state, &self.state)) {
*slot = None;
}
}
}
#[cfg(test)]
async fn pause_data_movement_multipart_before_selected_completion(bucket: &str) {
let barrier = DATA_MOVEMENT_MULTIPART_COMPLETION_BARRIER
.get_or_init(|| std::sync::Mutex::new(None))
.lock()
.expect("data movement multipart completion barrier mutex should not poison")
.as_ref()
.filter(|barrier| barrier.bucket == bucket)
.cloned();
if let Some(barrier) = barrier {
barrier.arrived.notify_one();
barrier.release.notified().await;
}
}
async fn list_pool_multipart_uploads_for_incarnation(
pool: &crate::core::sets::Sets,
bucket: &str,
@@ -332,7 +402,7 @@ impl ECStore {
) -> Result<MultipartUploadResult> {
self.handle_new_multipart_upload_with_pool_idx(bucket, object, opts)
.await
.map(|(res, _)| res)
.map(|(res, _, _)| res)
}
pub(crate) async fn handle_new_multipart_upload_with_pool_idx(
@@ -340,7 +410,7 @@ impl ECStore {
bucket: &str,
object: &str,
opts: &ObjectOptions,
) -> Result<(MultipartUploadResult, usize)> {
) -> Result<(MultipartUploadResult, usize, Option<Uuid>)> {
check_new_multipart_args(bucket, object)?;
let (opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?;
let opts = &opts;
@@ -349,7 +419,20 @@ impl ECStore {
return self.pools[0]
.new_multipart_upload(bucket, object, opts)
.await
.map(|res| (res, 0));
.map(|res| (res, 0, opts.expected_bucket_incarnation_id));
}
if opts.data_movement && opts.version_id.is_some() {
let idx = self.select_data_movement_pool_idx(bucket, object, -1, opts, false).await?;
if idx == opts.src_pool_idx {
return Err(StorageError::DataMovementOverwriteErr(
bucket.to_owned(),
object.to_owned(),
opts.version_id.clone().unwrap_or_default(),
));
}
let res = self.pools[idx].new_multipart_upload(bucket, object, opts).await?;
return Ok((res, idx, opts.expected_bucket_incarnation_id));
}
for (idx, pool) in self.pools.iter().enumerate() {
@@ -372,7 +455,7 @@ impl ECStore {
if !res.uploads.is_empty() {
let res = self.pools[idx].new_multipart_upload(bucket, object, opts).await?;
return Ok((res, idx));
return Ok((res, idx, opts.expected_bucket_incarnation_id));
}
}
let idx = self.get_pool_idx(bucket, object, -1).await?;
@@ -385,7 +468,7 @@ impl ECStore {
}
let res = self.pools[idx].new_multipart_upload(bucket, object, opts).await?;
Ok((res, idx))
Ok((res, idx, opts.expected_bucket_incarnation_id))
}
#[instrument(skip(self))]
@@ -456,6 +539,30 @@ impl ECStore {
Err(StorageError::InvalidUploadID(bucket.to_owned(), object.to_owned(), upload_id.to_owned()))
}
pub(crate) async fn put_object_part_for_data_movement(
&self,
target_pool_idx: usize,
bucket: &str,
object: &str,
upload_id: &str,
data: &mut PutObjReader,
opts: &ObjectOptions,
) -> Result<PartInfo> {
let part_id = opts
.part_number
.ok_or_else(|| Error::other("targeted multipart upload requires a part number"))?;
check_put_object_part_args(bucket, object, upload_id)?;
if !opts.data_movement {
return Err(Error::other("targeted multipart upload requires data_movement options"));
}
let (opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?;
let pool = self
.pools
.get(target_pool_idx)
.ok_or_else(|| Error::other(format!("data movement target pool {target_pool_idx} is out of range")))?;
pool.put_object_part(bucket, object, upload_id, part_id, data, &opts).await
}
#[instrument(skip(self))]
pub(super) async fn handle_get_multipart_info(
&self,
@@ -530,6 +637,26 @@ impl ECStore {
Err(StorageError::InvalidUploadID(bucket.to_owned(), object.to_owned(), upload_id.to_owned()))
}
pub(crate) async fn abort_multipart_upload_for_data_movement(
&self,
target_pool_idx: usize,
bucket: &str,
object: &str,
upload_id: &str,
opts: &ObjectOptions,
) -> Result<()> {
check_abort_multipart_args(bucket, object, upload_id)?;
if !opts.data_movement {
return Err(Error::other("targeted multipart abort requires data_movement options"));
}
let (opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?;
let pool = self
.pools
.get(target_pool_idx)
.ok_or_else(|| Error::other(format!("data movement target pool {target_pool_idx} is out of range")))?;
pool.abort_multipart_upload(bucket, object, upload_id, &opts).await
}
#[instrument(skip(self))]
pub(super) async fn handle_complete_multipart_upload(
self: Arc<Self>,
@@ -574,6 +701,62 @@ impl ECStore {
Err(StorageError::InvalidUploadID(bucket.to_owned(), object.to_owned(), upload_id.to_owned()))
}
pub(crate) async fn complete_multipart_upload_for_data_movement(
self: Arc<Self>,
target_pool_idx: usize,
bucket: &str,
object: &str,
upload_id: &str,
uploaded_parts: Vec<CompletePart>,
opts: &ObjectOptions,
) -> Result<ObjectInfo> {
check_complete_multipart_args(bucket, object, upload_id)?;
if !opts.data_movement {
return Err(Error::other("targeted multipart completion requires data_movement options"));
}
let (mut opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?;
if opts.overwrites_existing_version() && !is_meta_bucketname(bucket) {
let expected_incarnation_id = opts
.expected_bucket_incarnation_id
.ok_or_else(|| Error::other("data movement completion is missing its bucket incarnation"))?;
let lifecycle_fence = opts
.bucket_lifecycle_lock_fence
.as_ref()
.ok_or_else(|| Error::other("data movement completion is missing its bucket lifecycle fence"))?;
let snapshot = match opts.object_lock_config_snapshot.as_ref() {
Some(snapshot) => Arc::clone(snapshot),
None => {
self.object_lock_config_snapshot_under_lifecycle_fence(bucket, lifecycle_fence)
.await?
}
};
if !snapshot.is_valid_for_destructive_put(self.id, bucket, expected_incarnation_id) {
return Err(Error::other(
"data movement Object Lock snapshot does not match the target bucket generation",
));
}
snapshot.add_lock_fences(&mut opts);
opts.object_lock_config_snapshot = Some(snapshot);
}
#[cfg(test)]
pause_data_movement_multipart_before_selected_completion(bucket).await;
let pool = self
.pools
.get(target_pool_idx)
.ok_or_else(|| Error::other(format!("data movement target pool {target_pool_idx} is out of range")))?
.clone();
let result = enqueue_transition_after_write(
pool.complete_multipart_upload(bucket, object, upload_id, uploaded_parts, &opts)
.await,
LcEventSrc::S3CompleteMultipartUpload,
)
.await;
if result.is_ok() {
list_objects::observe_list_objects_mutation(self.as_ref(), bucket).await;
}
result
}
}
/// Merges per-pool `ListMultipartUploads` pages into a single globally paginated
+627 -70
View File
@@ -667,18 +667,6 @@ impl SelectObjectSnapshotLockLossWake {
}
}
fn select_object_ssec_headers(headers: &HeaderMap) -> HeaderMap {
use rustfs_utils::http::headers::{SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER};
let mut selected = HeaderMap::new();
for name in [SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER] {
if let Some(value) = headers.get(name) {
selected.insert(name, value.clone());
}
}
selected
}
// LockRegistry clones its canonical client Arc for each endpoint host, so an
// exact Arc set identifies one distributed namespace-lock quorum domain.
fn same_distributed_lock_domain(left: &[Arc<dyn rustfs_lock::LockClient>], right: &[Arc<dyn rustfs_lock::LockClient>]) -> bool {
@@ -918,7 +906,7 @@ fn is_equivalent_data_movement_delete_marker(source: &ObjectInfo, target: &Objec
&& is_data_movement_delete_marker(target)
&& source.version_id == target.version_id
&& source.mod_time == target.mod_time
&& source.user_defined == target.user_defined
&& is_equivalent_data_movement_delete_marker_metadata(&source.user_defined, &target.user_defined)
&& source.user_tags == target.user_tags
&& source.replication_status_internal == target.replication_status_internal
&& source.replication_status == target.replication_status
@@ -926,24 +914,185 @@ fn is_equivalent_data_movement_delete_marker(source: &ObjectInfo, target: &Objec
&& source.version_purge_status == target.version_purge_status
}
fn is_equivalent_data_movement_delete_marker_metadata(
source: &HashMap<String, String>,
target: &HashMap<String, String>,
) -> bool {
matches!(
(
data_movement_delete_marker_metadata_identity(source),
data_movement_delete_marker_metadata_identity(target)
),
(Some(source), Some(target)) if source == target
)
}
fn data_movement_delete_marker_metadata_identity(metadata: &HashMap<String, String>) -> Option<HashMap<String, String>> {
let mut identity = HashMap::with_capacity(metadata.len());
let mut local_tier_free_version_id = None;
for (key, value) in metadata {
let Some(suffix) = rustfs_utils::http::strip_internal_prefix_preserving_case(key) else {
identity.insert(key.clone(), value.clone());
continue;
};
if suffix.eq_ignore_ascii_case(rustfs_utils::http::SUFFIX_TIER_FV_ID) {
let version_id = Uuid::parse_str(value).ok().filter(|version_id| !version_id.is_nil())?;
if local_tier_free_version_id.is_some_and(|expected| expected != version_id) {
return None;
}
local_tier_free_version_id = Some(version_id);
continue;
}
let canonical_suffix = [
rustfs_utils::http::SUFFIX_REPLICA_TIMESTAMP,
rustfs_utils::http::SUFFIX_REPLICA_STATUS,
rustfs_utils::http::SUFFIX_REPLICATION_TIMESTAMP,
rustfs_utils::http::SUFFIX_REPLICATION_STATUS,
rustfs_utils::http::SUFFIX_PURGESTATUS,
]
.into_iter()
.find(|candidate| suffix.eq_ignore_ascii_case(candidate))
.map(str::to_string)
.or_else(|| {
[
rustfs_utils::http::SUFFIX_REPLICATION_RESET_ARN_PREFIX,
rustfs_utils::http::SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX,
]
.into_iter()
.find_map(|prefix| {
suffix
.get(..prefix.len())
.is_some_and(|candidate| candidate.eq_ignore_ascii_case(prefix))
.then(|| format!("{prefix}{}", &suffix[prefix.len()..]))
})
})
.unwrap_or_else(|| suffix.to_string());
let canonical_value = if canonical_suffix.eq_ignore_ascii_case(rustfs_utils::http::SUFFIX_REPLICA_TIMESTAMP)
|| canonical_suffix.eq_ignore_ascii_case(rustfs_utils::http::SUFFIX_REPLICATION_TIMESTAMP)
{
rustfs_filemeta::parse_replication_timestamp(value)?
.unix_timestamp_nanos()
.to_string()
} else {
value.clone()
};
let canonical_key = format!("{}{canonical_suffix}", rustfs_utils::http::RUSTFS_INTERNAL_PREFIX);
if identity
.insert(canonical_key, canonical_value.clone())
.is_some_and(|existing| existing != canonical_value)
{
return None;
}
}
for (status_suffix, timestamp_suffix) in [
(rustfs_utils::http::SUFFIX_REPLICA_STATUS, rustfs_utils::http::SUFFIX_REPLICA_TIMESTAMP),
(
rustfs_utils::http::SUFFIX_REPLICATION_STATUS,
rustfs_utils::http::SUFFIX_REPLICATION_TIMESTAMP,
),
] {
let status_key = format!("{}{status_suffix}", rustfs_utils::http::RUSTFS_INTERNAL_PREFIX);
let timestamp_key = format!("{}{timestamp_suffix}", rustfs_utils::http::RUSTFS_INTERNAL_PREFIX);
match (identity.contains_key(&status_key), identity.contains_key(&timestamp_key)) {
(true, false) => {
identity.insert(timestamp_key, OffsetDateTime::UNIX_EPOCH.unix_timestamp_nanos().to_string());
}
(false, true) => return None,
_ => {}
}
}
Some(identity)
}
fn is_data_movement_delete_marker(info: &ObjectInfo) -> bool {
info.delete_marker
}
fn is_expected_data_movement_delete_marker_source(source: &ObjectInfo, expected_mod_time: Option<OffsetDateTime>) -> bool {
is_data_movement_delete_marker(source)
&& source.mod_time.is_some()
&& source.mod_time == expected_mod_time
&& data_movement_delete_marker_metadata_identity(&source.user_defined).is_some()
}
fn current_data_movement_delete_marker_opts(source: &ObjectInfo, opts: &ObjectOptions) -> Option<ObjectOptions> {
let replica_status = rustfs_utils::http::get_str(&source.user_defined, rustfs_utils::http::SUFFIX_REPLICA_STATUS);
let replica_timestamp = rustfs_utils::http::get_str(&source.user_defined, rustfs_utils::http::SUFFIX_REPLICA_TIMESTAMP);
let (replica_status, replica_timestamp) = match (replica_status, replica_timestamp) {
(None, None) => Default::default(),
(Some(status), timestamp) => {
let status = crate::bucket::replication::ReplicationStatusType::from(status.as_str());
if status.is_empty() {
return None;
}
let timestamp = match timestamp {
Some(timestamp) => rustfs_filemeta::parse_replication_timestamp(&timestamp)?,
None => OffsetDateTime::UNIX_EPOCH,
};
(status, Some(timestamp))
}
(None, Some(_)) => return None,
};
let replication_status = rustfs_utils::http::get_str(&source.user_defined, rustfs_utils::http::SUFFIX_REPLICATION_STATUS);
let replication_timestamp =
rustfs_utils::http::get_str(&source.user_defined, rustfs_utils::http::SUFFIX_REPLICATION_TIMESTAMP);
let (replication_status, replication_timestamp, replication_targets) = match (replication_status, replication_timestamp) {
(None, None) => Default::default(),
(Some(status), timestamp) => {
let direct_status = crate::bucket::replication::ReplicationStatusType::from(status.as_str());
let targets = crate::bucket::replication::replication_statuses_map(status.as_str());
if direct_status.is_empty() && targets.is_empty() {
return None;
}
let timestamp = match timestamp {
Some(timestamp) => rustfs_filemeta::parse_replication_timestamp(&timestamp)?,
None => OffsetDateTime::UNIX_EPOCH,
};
(Some(status), Some(timestamp), targets)
}
(None, Some(_)) => return None,
};
let mut state = source.replication_state();
if state.target_delete_marker_version_ids_corrupt {
return None;
}
state.replica_status = replica_status;
state.replica_timestamp = replica_timestamp;
state.replication_status_internal = replication_status;
state.replication_timestamp = replication_timestamp;
state.targets = replication_targets;
state.replicate_decision_str = source.replication_decision.clone();
state.delete_marker = true;
let mut target_opts = opts.clone();
target_opts.mod_time = source.mod_time;
target_opts.delete_replication = Some(state);
Some(target_opts)
}
fn expected_data_movement_tiered_object(source: &rustfs_filemeta::FileInfo) -> ObjectInfo {
ObjectInfo::from_file_info(source, "", &source.name, source.version_id.is_some())
}
fn is_equivalent_data_movement_tiered_object(source: &rustfs_filemeta::FileInfo, target: &ObjectInfo) -> bool {
let expected = expected_data_movement_tiered_object(source);
let Some(source_actual_size) = effective_object_actual_size(&expected) else {
return false;
};
let Some(target_actual_size) = effective_object_actual_size(target) else {
return false;
};
source.version_id == target.version_id
&& !target.delete_marker
&& source.size == target.size
&& source.get_etag() == target.etag
&& source.checksum == target.checksum
&& crate::data_movement::are_equivalent_data_movement_parts(&source.parts, &target.parts)
&& source.mod_time == target.mod_time
&& expected.user_defined == target.user_defined
&& crate::data_movement::is_equivalent_data_movement_metadata(&expected, target, source_actual_size, target_actual_size)
&& expected.user_tags == target.user_tags
&& expected.expires == target.expires
&& expected.storage_class == target.storage_class
@@ -952,11 +1101,12 @@ fn is_equivalent_data_movement_tiered_object(source: &rustfs_filemeta::FileInfo,
&& expected.version_purge_status_internal == target.version_purge_status_internal
&& expected.version_purge_status == target.version_purge_status
&& expected.transitioned_object.status == target.transitioned_object.status
&& expected.transition_version_state == target.transition_version_state
&& expected.transitioned_object.name == target.transitioned_object.name
&& expected.transitioned_object.tier == target.transitioned_object.tier
&& expected.transitioned_object.version_id == target.transitioned_object.version_id
&& expected.transitioned_object.free_version == target.transitioned_object.free_version
&& effective_object_actual_size(target) == Some(source.size)
&& source_actual_size == target_actual_size
}
fn should_check_data_movement_resume_target(src_pool_idx: usize, target_pool_idx: usize) -> bool {
@@ -1054,7 +1204,7 @@ impl ECStore {
)))
}
async fn object_lock_config_snapshot_under_lifecycle_fence(
pub(super) async fn object_lock_config_snapshot_under_lifecycle_fence(
&self,
bucket: &str,
lifecycle_fence: &NamespaceLockFence,
@@ -1134,7 +1284,7 @@ impl ECStore {
pool,
bucket: bucket.to_owned(),
object,
headers: select_object_ssec_headers(headers),
headers: rustfs_utils::http::project_ssec_transport_headers(headers),
opts,
object_info,
logical_size,
@@ -1454,7 +1604,8 @@ impl ECStore {
target_pool_idx: usize,
opts: &ObjectOptions,
) -> Result<Option<ObjectInfo>> {
let lookup_opts = version_aware_lookup_opts(opts, true);
let mut lookup_opts = version_aware_lookup_opts(opts, true);
lookup_opts.include_part_checksums = true;
let Some(pool) = self.pools.get(target_pool_idx) else {
return Err(Error::other(format!(
@@ -1521,6 +1672,25 @@ impl ECStore {
) -> Result<()> {
check_put_object_args(bucket, object)?;
let mut opts = opts.clone();
let bucket_incarnation_fence = if is_meta_bucketname(bucket) {
None
} else {
let expected = opts
.expected_bucket_incarnation_id
.ok_or_else(|| Error::other("tiered data movement is missing its bucket incarnation snapshot"))?;
let guard = self.acquire_bucket_incarnation_fence(bucket, expected).await?;
if let Some(namespace_guard) = guard.namespace_lock_guard() {
opts.add_bucket_lifecycle_lock_guard(namespace_guard);
}
Some(guard)
};
let mut fi = fi.clone();
if opts.data_movement {
crate::data_movement::prepare_tiered_data_movement_file_info(&mut fi)?;
}
let object = encode_dir_object(object);
if self.single_pool() {
@@ -1533,7 +1703,8 @@ impl ECStore {
let idx = if opts.data_movement && opts.version_id.is_some() {
Self::resolve_decommission_target_pool_idx_result(
self.select_data_movement_pool_idx(bucket, &object, fi.size, opts, true).await,
self.select_data_movement_pool_idx(bucket, &object, fi.size, &opts, true)
.await,
bucket,
&object,
)?
@@ -1550,7 +1721,7 @@ impl ECStore {
.await;
let target_pool_idx = resolve_data_movement_resume_target_pool(idx, resume_target_pool_idx, opts.src_pool_idx);
if self
.has_equivalent_data_movement_tiered_object(bucket, &object, fi, opts, target_pool_idx)
.has_equivalent_data_movement_tiered_object(bucket, &object, &fi, &opts, target_pool_idx)
.await?
{
return Ok(());
@@ -1563,17 +1734,30 @@ impl ECStore {
));
}
Self::resolve_decommission_tiered_object_result(
self.pools[idx]
.get_disks_by_key(&object)
.decommission_tiered_object(bucket, &object, fi, opts)
.await,
bucket,
&object,
)
let result = self.pools[idx]
.get_disks_by_key(&object)
.decommission_tiered_object(bucket, &object, &fi, &opts)
.await;
if matches!(result, Err(Error::PreconditionFailed)) {
if self
.has_equivalent_data_movement_tiered_object(bucket, &object, &fi, &opts, idx)
.await?
{
return Ok(());
}
return Err(StorageError::DataMovementOverwriteErr(
bucket.to_owned(),
object,
opts.version_id.clone().unwrap_or_default(),
));
}
if bucket_incarnation_fence.as_ref().is_some_and(|guard| guard.is_lock_lost()) {
return Err(Error::other("tiered data movement bucket incarnation fence was lost during target write"));
}
Self::resolve_decommission_tiered_object_result(result, bucket, &object)
}
#[instrument(level = "debug", skip(self))]
#[instrument(level = "debug", skip(self, h))]
#[hotpath::measure(impl_type = "ECStore")]
pub(super) async fn handle_get_object_reader(
&self,
@@ -1585,7 +1769,7 @@ impl ECStore {
) -> Result<GetObjectReader> {
check_get_obj_args(bucket, object)?;
let object = encode_dir_object(object);
let object = rustfs_utils::path::encode_dir_object_ref(object);
let mut opts = opts.clone();
let read_lock_guard = self
.acquire_object_read_lock_if_needed("get_object", bucket, &object, &mut opts)
@@ -1593,29 +1777,21 @@ impl ECStore {
let reader = if self.single_pool() {
self.pools[0]
.get_object_reader(bucket, object.as_str(), range, h, &opts)
.get_object_reader(bucket, object.as_ref(), range, h, &opts)
.await?
} else {
let (_, idx) = self
.get_latest_accessible_object_info_with_idx(bucket, &object, &opts)
.await?;
self.pools[idx]
.get_object_reader(bucket, object.as_str(), range, h, &opts)
.get_object_reader(bucket, object.as_ref(), range, h, &opts)
.await?
};
Ok(Self::attach_read_lock_guard(reader, read_lock_guard))
}
#[instrument(level = "debug", skip(self, data))]
#[hotpath::measure(impl_type = "ECStore")]
pub(super) async fn handle_put_object(
&self,
bucket: &str,
object: &str,
data: &mut PutObjReader,
opts: &ObjectOptions,
) -> Result<(ObjectInfo, Option<OldCurrentSize>)> {
async fn prepare_put_object(&self, bucket: &str, object: &str, opts: &ObjectOptions) -> Result<(String, ObjectOptions)> {
check_put_object_args(bucket, object)?;
let object = encode_dir_object(object);
@@ -1642,22 +1818,20 @@ impl ECStore {
};
snapshot.add_lock_fences(&mut opts);
}
Ok((object, opts))
}
// Keep PUT atomic-read friendly: SetDisks takes the object write lock only
// around precondition checks and the final rename/commit.
async fn select_put_object_pool_idx(&self, bucket: &str, object: &str, size: i64, opts: &ObjectOptions) -> Result<usize> {
if self.single_pool() {
return self.pools[0]
.put_object_with_old_current_size(bucket, object.as_str(), data, &opts)
.await;
return Ok(0);
}
let idx = if opts.data_movement && opts.version_id.is_some() {
self.select_data_movement_pool_idx(bucket, &object, data.size(), &opts, false)
.await?
self.select_data_movement_pool_idx(bucket, object, size, opts, false).await?
} else if opts.no_lock {
self.get_pool_idx_no_lock(bucket, &object, data.size()).await?
self.get_pool_idx_no_lock(bucket, object, size).await?
} else {
self.get_pool_idx(bucket, &object, data.size()).await?
self.get_pool_idx(bucket, object, size).await?
};
if opts.data_movement && idx == opts.src_pool_idx {
@@ -1667,7 +1841,50 @@ impl ECStore {
opts.version_id.clone().unwrap_or_default(),
));
}
Ok(idx)
}
pub(crate) async fn put_object_for_data_movement(
&self,
bucket: &str,
object: &str,
data: &mut PutObjReader,
opts: &ObjectOptions,
) -> Result<(usize, Result<ObjectInfo>)> {
if !opts.data_movement {
return Err(Error::other("data movement PUT requires data_movement options"));
}
let (object, opts) = self.prepare_put_object(bucket, object, opts).await?;
let idx = self
.select_put_object_pool_idx(bucket, object.as_str(), data.size(), &opts)
.await?;
let result = self.pools[idx]
.put_object_with_old_current_size(bucket, &object, data, &opts)
.await
.map(|(object_info, _)| object_info);
let result = enqueue_transition_after_write(result, LcEventSrc::S3PutObject).await;
if result.is_ok() {
list_objects::observe_list_objects_mutation(self, bucket).await;
}
Ok((idx, result))
}
#[instrument(level = "debug", skip(self, data))]
#[hotpath::measure(impl_type = "ECStore")]
pub(super) async fn handle_put_object(
&self,
bucket: &str,
object: &str,
data: &mut PutObjReader,
opts: &ObjectOptions,
) -> Result<(ObjectInfo, Option<OldCurrentSize>)> {
let (object, opts) = self.prepare_put_object(bucket, object, opts).await?;
let idx = self
.select_put_object_pool_idx(bucket, object.as_str(), data.size(), &opts)
.await?;
// Keep PUT atomic-read friendly: SetDisks takes the object write lock only
// around precondition checks and the final rename/commit.
self.pools[idx]
.put_object_with_old_current_size(bucket, &object, data, &opts)
.await
@@ -2110,6 +2327,50 @@ impl ECStore {
};
let target_pool_idx =
resolve_data_movement_resume_target_pool(selected_target_pool_idx, resume_target_pool_idx, opts.src_pool_idx);
let mut delete_marker_target_opts = None;
if opts.delete_marker && should_check_data_movement_resume_target(opts.src_pool_idx, target_pool_idx) {
let source = self
.find_data_movement_target_info(bucket, object, opts.src_pool_idx, &opts)
.await?;
let Some(source) = source else {
return Err(StorageError::DataMovementOverwriteErr(
bucket.to_owned(),
object.to_owned(),
opts.version_id.unwrap_or_default(),
));
};
if !is_expected_data_movement_delete_marker_source(&source, opts.mod_time) {
return Err(StorageError::DataMovementOverwriteErr(
bucket.to_owned(),
object.to_owned(),
opts.version_id.unwrap_or_default(),
));
}
let Some(target_opts) = current_data_movement_delete_marker_opts(&source, &opts) else {
return Err(StorageError::DataMovementOverwriteErr(
bucket.to_owned(),
object.to_owned(),
opts.version_id.unwrap_or_default(),
));
};
let target = self
.find_data_movement_target_info(bucket, object, target_pool_idx, &target_opts)
.await?;
if let Some(target) = target {
if is_equivalent_data_movement_delete_marker(&source, &target) {
let mut target = target;
target.name = decode_dir_object(object);
return Ok(target);
}
return Err(StorageError::DataMovementOverwriteErr(
bucket.to_owned(),
object.to_owned(),
opts.version_id.unwrap_or_default(),
));
}
delete_marker_target_opts = Some(target_opts);
}
if !should_check_data_movement_resume_target(opts.src_pool_idx, target_pool_idx) {
if let Ok((source_pool_info, _)) = existing_pool_info
@@ -2137,7 +2398,8 @@ impl ECStore {
));
}
let mut obj = self.pools[target_pool_idx].delete_object(bucket, object, opts).await?;
let target_opts = delete_marker_target_opts.unwrap_or(opts);
let mut obj = self.pools[target_pool_idx].delete_object(bucket, object, target_opts).await?;
obj.name = decode_dir_object(obj.name.as_str());
return Ok(obj);
}
@@ -3086,25 +3348,6 @@ mod tests {
assert!(second_signal.is_lost());
}
#[test]
fn select_snapshot_retains_only_ssec_headers() {
use rustfs_utils::http::headers::{SSEC_ALGORITHM_HEADER, SSEC_KEY_HEADER, SSEC_KEY_MD5_HEADER};
let mut headers = HeaderMap::new();
headers.insert(SSEC_ALGORITHM_HEADER, "AES256".parse().expect("valid SSE-C algorithm header"));
headers.insert(SSEC_KEY_HEADER, "secret-key".parse().expect("valid SSE-C key header"));
headers.insert(SSEC_KEY_MD5_HEADER, "key-md5".parse().expect("valid SSE-C key digest header"));
headers.insert("authorization", "credential".parse().expect("valid authorization header"));
let selected = select_object_ssec_headers(&headers);
assert_eq!(selected.len(), 3);
assert_eq!(selected.get(SSEC_ALGORITHM_HEADER), headers.get(SSEC_ALGORITHM_HEADER));
assert_eq!(selected.get(SSEC_KEY_HEADER), headers.get(SSEC_KEY_HEADER));
assert_eq!(selected.get(SSEC_KEY_MD5_HEADER), headers.get(SSEC_KEY_MD5_HEADER));
assert!(selected.get("authorization").is_none());
}
#[test]
fn tier_delete_entry_is_prepared_and_bound_to_source_generation() {
let identity = [9_u8; 32];
@@ -3200,6 +3443,247 @@ mod tests {
assert!(!is_equivalent_data_movement_delete_marker(&source, &mismatched));
}
#[test]
fn equivalent_data_movement_delete_marker_accepts_distinct_local_free_version_ids() {
let mut source = ObjectInfo {
version_id: Some(Uuid::from_u128(1)),
delete_marker: true,
mod_time: Some(OffsetDateTime::UNIX_EPOCH),
..Default::default()
};
rustfs_utils::http::insert_str(
Arc::make_mut(&mut source.user_defined),
rustfs_utils::http::SUFFIX_TIER_FV_ID,
Uuid::from_u128(2).to_string(),
);
let mut target = source.clone();
rustfs_utils::http::insert_str(
Arc::make_mut(&mut target.user_defined),
rustfs_utils::http::SUFFIX_TIER_FV_ID,
Uuid::from_u128(3).to_string(),
);
assert!(is_equivalent_data_movement_delete_marker(&source, &target));
Arc::make_mut(&mut target.user_defined).insert(
format!("{}{}", rustfs_utils::http::MINIO_INTERNAL_PREFIX, rustfs_utils::http::SUFFIX_TIER_FV_ID),
Uuid::from_u128(4).to_string(),
);
assert!(!is_equivalent_data_movement_delete_marker(&source, &target));
}
#[test]
fn equivalent_data_movement_delete_marker_accepts_replication_alias_expansion() {
let key = format!(
"{}{}",
rustfs_utils::http::MINIO_INTERNAL_PREFIX,
rustfs_utils::http::SUFFIX_REPLICATION_STATUS
);
let timestamp_key = format!(
"{}{}",
rustfs_utils::http::MINIO_INTERNAL_PREFIX,
rustfs_utils::http::SUFFIX_REPLICATION_TIMESTAMP
);
let source = ObjectInfo {
version_id: Some(Uuid::from_u128(1)),
delete_marker: true,
mod_time: Some(OffsetDateTime::UNIX_EPOCH),
user_defined: Arc::new(HashMap::from([
(key.clone(), "arn=COMPLETED;".to_string()),
(timestamp_key, "1970-01-01T00:00:01Z".to_string()),
])),
..Default::default()
};
let mut target = source.clone();
rustfs_utils::http::insert_str(
Arc::make_mut(&mut target.user_defined),
rustfs_utils::http::SUFFIX_REPLICATION_STATUS,
"arn=COMPLETED;".to_string(),
);
rustfs_utils::http::insert_str(
Arc::make_mut(&mut target.user_defined),
rustfs_utils::http::SUFFIX_REPLICATION_TIMESTAMP,
(OffsetDateTime::UNIX_EPOCH + time::Duration::SECOND).to_string(),
);
assert!(is_equivalent_data_movement_delete_marker(&source, &target));
Arc::make_mut(&mut target.user_defined).insert(key, "arn=FAILED;".to_string());
assert!(!is_equivalent_data_movement_delete_marker(&source, &target));
}
#[test]
fn data_movement_delete_marker_source_requires_persisted_mod_time() {
let source = ObjectInfo {
delete_marker: true,
..Default::default()
};
assert!(!is_expected_data_movement_delete_marker_source(&source, None));
let source = ObjectInfo {
mod_time: Some(OffsetDateTime::UNIX_EPOCH),
..source
};
assert!(is_expected_data_movement_delete_marker_source(&source, Some(OffsetDateTime::UNIX_EPOCH)));
assert!(!is_expected_data_movement_delete_marker_source(&source, None));
}
#[test]
fn data_movement_delete_marker_uses_current_source_replication_state() {
let expected_timestamp = OffsetDateTime::UNIX_EPOCH + time::Duration::SECOND;
let timestamp = expected_timestamp.to_string();
let mut metadata = HashMap::new();
rustfs_utils::http::insert_str(
&mut metadata,
rustfs_utils::http::SUFFIX_REPLICA_STATUS,
ReplicationStatusType::Replica.to_string(),
);
rustfs_utils::http::insert_str(&mut metadata, rustfs_utils::http::SUFFIX_REPLICA_TIMESTAMP, timestamp.clone());
rustfs_utils::http::insert_str(&mut metadata, rustfs_utils::http::SUFFIX_REPLICATION_TIMESTAMP, timestamp);
rustfs_utils::http::insert_str(
&mut metadata,
rustfs_utils::http::SUFFIX_REPLICATION_STATUS,
"arn=COMPLETED;".to_string(),
);
rustfs_utils::http::insert_str(
&mut metadata,
&format!(
"{}{}",
rustfs_utils::http::SUFFIX_REPLICATION_RESET_ARN_PREFIX,
"arn:minio:replication::TenantA:bucket"
),
"reset-id".to_string(),
);
rustfs_utils::http::insert_str(
&mut metadata,
&format!(
"{}{}",
rustfs_utils::http::SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX,
"arn:minio:replication::TenantA:bucket"
),
"target-version".to_string(),
);
let source = ObjectInfo {
delete_marker: true,
mod_time: Some(OffsetDateTime::UNIX_EPOCH),
replication_status_internal: Some("arn=COMPLETED;".to_string()),
replication_decision: "arn=replicate;".to_string(),
user_defined: Arc::new(metadata),
..Default::default()
};
let opts = ObjectOptions {
mod_time: source.mod_time,
delete_replication: Some(ReplicationState {
replication_status_internal: Some("arn=PENDING;".to_string()),
..Default::default()
}),
..Default::default()
};
let target_opts = current_data_movement_delete_marker_opts(&source, &opts).expect("valid current source state");
let state = target_opts.delete_replication.as_ref().expect("current replication state");
assert_eq!(state.replication_status_internal.as_deref(), Some("arn=COMPLETED;"));
assert_eq!(state.replica_status, crate::bucket::replication::ReplicationStatusType::Replica);
assert_eq!(state.replica_timestamp, Some(expected_timestamp));
assert_eq!(state.replication_timestamp, state.replica_timestamp);
assert_eq!(state.replicate_decision_str, "arn=replicate;");
assert_eq!(
state
.reset_statuses_map
.get("arn:minio:replication::TenantA:bucket")
.map(String::as_str),
Some("reset-id")
);
assert_eq!(
state
.target_delete_marker_version_ids
.get("arn:minio:replication::TenantA:bucket")
.map(String::as_str),
Some("target-version")
);
}
#[test]
fn data_movement_delete_marker_rejects_corrupt_target_version_maps() {
let suffix = format!("{}not-an-arn", rustfs_utils::http::SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX);
let mut malformed = HashMap::new();
rustfs_utils::http::insert_str(&mut malformed, &suffix, "target-version".to_string());
let malformed_source = ObjectInfo {
delete_marker: true,
mod_time: Some(OffsetDateTime::UNIX_EPOCH),
user_defined: Arc::new(malformed),
..Default::default()
};
assert!(current_data_movement_delete_marker_opts(&malformed_source, &ObjectOptions::default()).is_none());
let mut conflicted = HashMap::new();
let suffix = format!(
"{}arn:minio:replication::target:bucket",
rustfs_utils::http::SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX
);
rustfs_utils::http::insert_str(&mut conflicted, &suffix, "target-version-a".to_string());
conflicted.insert(
format!("{}{suffix}", rustfs_utils::http::MINIO_INTERNAL_PREFIX),
"target-version-b".to_string(),
);
let conflicted_source = ObjectInfo {
user_defined: Arc::new(conflicted),
..malformed_source.clone()
};
assert!(current_data_movement_delete_marker_opts(&conflicted_source, &ObjectOptions::default()).is_none());
let mut over_cap = HashMap::new();
for index in 0..=1_000 {
let suffix = format!(
"{}arn:minio:replication::target:bucket-{index}",
rustfs_utils::http::SUFFIX_REPLICATION_DELETE_MARKER_VERSION_ARN_PREFIX
);
rustfs_utils::http::insert_str(&mut over_cap, &suffix, format!("target-version-{index}"));
}
let over_cap_source = ObjectInfo {
user_defined: Arc::new(over_cap),
..malformed_source
};
assert!(current_data_movement_delete_marker_opts(&over_cap_source, &ObjectOptions::default()).is_none());
}
#[test]
fn data_movement_delete_marker_normalizes_legacy_missing_replication_timestamps() {
let mut source_metadata = HashMap::new();
rustfs_utils::http::insert_str(
&mut source_metadata,
rustfs_utils::http::SUFFIX_REPLICA_STATUS,
ReplicationStatusType::Replica.to_string(),
);
rustfs_utils::http::insert_str(
&mut source_metadata,
rustfs_utils::http::SUFFIX_REPLICATION_STATUS,
"arn=COMPLETED;".to_string(),
);
let source = ObjectInfo {
delete_marker: true,
mod_time: Some(OffsetDateTime::UNIX_EPOCH),
replication_status_internal: Some("arn=COMPLETED;".to_string()),
user_defined: Arc::new(source_metadata),
..Default::default()
};
let target_opts = current_data_movement_delete_marker_opts(&source, &ObjectOptions::default())
.expect("legacy status-only metadata should remain migratable");
let state = target_opts
.delete_replication
.expect("replication state should be reconstructed");
assert_eq!(state.replica_timestamp, Some(OffsetDateTime::UNIX_EPOCH));
assert_eq!(state.replication_timestamp, Some(OffsetDateTime::UNIX_EPOCH));
let mut target_metadata = (*source.user_defined).clone();
let epoch = OffsetDateTime::UNIX_EPOCH
.format(&time::format_description::well_known::Rfc3339)
.unwrap();
rustfs_utils::http::insert_str(&mut target_metadata, rustfs_utils::http::SUFFIX_REPLICA_TIMESTAMP, epoch.clone());
rustfs_utils::http::insert_str(&mut target_metadata, rustfs_utils::http::SUFFIX_REPLICATION_TIMESTAMP, epoch);
assert!(is_equivalent_data_movement_delete_marker_metadata(&source.user_defined, &target_metadata));
}
#[test]
fn equivalent_data_movement_delete_marker_rejects_metadata_and_replication_mismatch() {
let version_id = Uuid::nil();
@@ -3344,6 +3828,67 @@ mod tests {
assert!(is_equivalent_data_movement_tiered_object(&source, &target));
}
#[test]
fn equivalent_data_movement_tiered_object_uses_logical_compressed_and_encrypted_sizes() {
let mut compressed = tiered_equivalence_source();
compressed.size = 600;
rustfs_utils::http::insert_str(&mut compressed.metadata, rustfs_utils::http::SUFFIX_COMPRESSION, "S2".to_string());
rustfs_utils::http::insert_str(&mut compressed.metadata, rustfs_utils::http::SUFFIX_ACTUAL_SIZE, "1024".to_string());
let compressed_target = tiered_equivalence_target(&compressed);
assert!(is_equivalent_data_movement_tiered_object(&compressed, &compressed_target));
let mut encrypted = tiered_equivalence_source();
encrypted.size = 640;
encrypted.metadata.insert(
rustfs_utils::http::object_encryption_keys::INTERNAL_ENCRYPTION_KEY_ID_HEADER.to_string(),
"key-id".to_string(),
);
encrypted.metadata.insert(
rustfs_utils::http::object_encryption_keys::INTERNAL_ENCRYPTION_ORIGINAL_SIZE_HEADER.to_string(),
"1024".to_string(),
);
let encrypted_target = tiered_equivalence_target(&encrypted);
assert!(is_equivalent_data_movement_tiered_object(&encrypted, &encrypted_target));
}
#[test]
fn equivalent_data_movement_tiered_object_accepts_transition_alias_expansion() {
let mut source = tiered_equivalence_source();
let suffix = rustfs_utils::http::SUFFIX_TRANSITION_TIER;
source.metadata.insert(
format!("{}{suffix}", rustfs_utils::http::MINIO_INTERNAL_PREFIX),
source.transition_tier.clone(),
);
let mut target = tiered_equivalence_target(&source);
Arc::make_mut(&mut target.user_defined)
.insert(rustfs_utils::http::internal_key_rustfs(suffix), source.transition_tier.clone());
assert!(is_equivalent_data_movement_tiered_object(&source, &target));
}
#[test]
fn equivalent_data_movement_tiered_object_requires_hydrated_part_checksums() {
let mut source = tiered_equivalence_source();
source.parts = vec![rustfs_filemeta::ObjectPartInfo {
number: 1,
mod_time: Some(OffsetDateTime::UNIX_EPOCH + time::Duration::SECOND),
checksums: Some(HashMap::from([("CRC32C".to_string(), "AAAAAA==".to_string())])),
..Default::default()
}];
rustfs_utils::http::insert_str(
&mut source.metadata,
rustfs_utils::http::SUFFIX_PART_CHECKSUMS,
r#"[[1,[["CRC32C","AAAAAA=="]]]]"#.to_string(),
);
let mut target = tiered_equivalence_target(&source);
Arc::make_mut(&mut target.parts)[0].mod_time = None;
assert!(is_equivalent_data_movement_tiered_object(&source, &target));
let mut missing = target;
Arc::make_mut(&mut missing.parts)[0].checksums = None;
assert!(!is_equivalent_data_movement_tiered_object(&source, &missing));
}
#[test]
fn equivalent_data_movement_tiered_object_rejects_transition_mismatch() {
let source = tiered_equivalence_source();
@@ -3353,6 +3898,16 @@ mod tests {
assert!(!is_equivalent_data_movement_tiered_object(&source, &target));
}
#[test]
fn equivalent_data_movement_tiered_object_rejects_transition_version_state_mismatch() {
let mut source = tiered_equivalence_source();
source.transition_version_state = rustfs_filemeta::TransitionVersionState::Exact;
let mut target = tiered_equivalence_target(&source);
target.transition_version_state = rustfs_filemeta::TransitionVersionState::Unknown;
assert!(!is_equivalent_data_movement_tiered_object(&source, &target));
}
#[test]
fn equivalent_data_movement_tiered_object_rejects_user_metadata_mismatch() {
let source = tiered_equivalence_source();
@@ -4022,6 +4577,8 @@ mod tests {
assert_eq!(snapshot.headers.get(SSEC_KEY_HEADER), request_headers.get(SSEC_KEY_HEADER));
assert_eq!(snapshot.headers.get(SSEC_KEY_MD5_HEADER), request_headers.get(SSEC_KEY_MD5_HEADER));
assert!(snapshot.headers.get("authorization").is_none());
assert!(snapshot.headers.values().all(http::HeaderValue::is_sensitive));
assert!(!format!("{:?}", snapshot.headers).contains("secret-key"));
assert_eq!(
snapshot.logical_size(),
u64::try_from(payload.len()).expect("test payload length should fit in u64")
@@ -109,6 +109,7 @@ async fn run_legacy_bitrot_test_for_object(root: &std::path::Path, disk_name: &s
FileInfoOpts {
data: true, // need inline data for inline objects
include_free_versions: false,
include_part_checksums: true,
},
) {
Ok(f) => f,
+1 -1
View File
@@ -37,6 +37,7 @@ crc-fast = { workspace = true }
rmp.workspace = true
rmp-serde.workspace = true
serde = { workspace = true, features = ["derive"] }
serde_json.workspace = true
time = { workspace = true, features = ["parsing", "formatting", "macros", "serde"] }
uuid = { workspace = true, features = ["v4", "fast-rng", "serde", "macro-diagnostics"] }
tokio = { workspace = true, features = ["io-util", "macros", "sync", "fs", "rt-multi-thread"] }
@@ -54,7 +55,6 @@ arc-swap.workspace = true
criterion = { workspace = true, features = ["html_reports"] }
tempfile = { workspace = true }
proptest = "1"
serde_json.workspace = true
[[bench]]
name = "xl_meta_bench"
@@ -52,6 +52,7 @@ fn main() {
FileInfoOpts {
data: false,
include_free_versions: true,
include_part_checksums: true,
},
)
.expect("decode file info");
+78 -3
View File
@@ -17,9 +17,10 @@ use bytes::Bytes;
use rmp_serde::Serializer;
use rustfs_utils::HashAlgorithm;
use rustfs_utils::http::{
SUFFIX_COMPRESSION, SUFFIX_DATA_MOVED, SUFFIX_FREE_VERSION, SUFFIX_HEALING, SUFFIX_INLINE_DATA, SUFFIX_TIER_FV_ID,
SUFFIX_TIER_FV_MARKER, SUFFIX_TIER_SKIP_FV_ID, contains_key_str, get_str, has_internal_suffix, insert_str,
is_encryption_metadata_key, starts_with_ignore_ascii_case,
AMZ_OBJECT_TAGGING, SUFFIX_COMPRESSION, SUFFIX_DATA_MOVED, SUFFIX_DATA_MOVED_TAGS, SUFFIX_FREE_VERSION, SUFFIX_HEALING,
SUFFIX_INLINE_DATA, SUFFIX_OBJECT_TRANSACTION_EPOCH, SUFFIX_TIER_FV_ID, SUFFIX_TIER_FV_MARKER, SUFFIX_TIER_SKIP_FV_ID,
contains_key_str, get_consistent_str, get_str, has_internal_suffix, insert_str, is_encryption_metadata_key,
starts_with_ignore_ascii_case,
};
use s3s::dto::{RestoreStatus, Timestamp};
use s3s::header::X_AMZ_RESTORE;
@@ -232,6 +233,17 @@ pub enum TransitionVersionState {
Exact,
}
impl TransitionVersionState {
pub const fn as_str(self) -> &'static str {
match self {
Self::Unknown => "unknown",
Self::KnownDisabled => "known-disabled",
Self::SuspendedNull => "suspended-null",
Self::Exact => "exact",
}
}
}
#[derive(PartialEq, Clone, Default)]
pub struct FileInfo {
pub volume: String,
@@ -1151,9 +1163,32 @@ impl FileInfo {
}
pub fn set_data_moved(&mut self) {
let tags_proof = format!("v1:{}", self.metadata.get(AMZ_OBJECT_TAGGING).map(String::as_str).unwrap_or_default());
insert_str(&mut self.metadata, SUFFIX_DATA_MOVED_TAGS, tags_proof);
insert_str(&mut self.metadata, SUFFIX_DATA_MOVED, "true".to_string());
}
pub fn acknowledge_data_movement(&mut self) {
// Keep both empty aliases so mixed-version disks retain one metadata identity.
insert_str(&mut self.metadata, SUFFIX_DATA_MOVED, String::new());
}
pub fn set_object_transaction_epoch(&mut self, epoch: Uuid) {
insert_str(&mut self.metadata, SUFFIX_OBJECT_TRANSACTION_EPOCH, epoch.to_string());
}
pub fn object_transaction_epoch(&self) -> Result<Option<Uuid>> {
if !contains_key_str(&self.metadata, SUFFIX_OBJECT_TRANSACTION_EPOCH) {
return Ok(None);
}
let value = get_consistent_str(&self.metadata, SUFFIX_OBJECT_TRANSACTION_EPOCH).ok_or(Error::FileCorrupt)?;
let epoch = Uuid::parse_str(value).map_err(|_| Error::FileCorrupt)?;
if epoch.is_nil() {
return Err(Error::FileCorrupt);
}
Ok(Some(epoch))
}
pub fn inline_data(&self) -> bool {
contains_key_str(&self.metadata, SUFFIX_INLINE_DATA) && !self.is_remote()
}
@@ -1466,6 +1501,46 @@ mod tests {
assert_eq!(ei.get_checksum_info(99).algorithm, HashAlgorithm::HighwayHash256S);
}
#[test]
fn object_transaction_epoch_uses_consistent_dual_internal_metadata() {
let mut fi = validation_test_fileinfo();
assert_eq!(fi.object_transaction_epoch().expect("absent epoch should decode"), None);
let epoch = Uuid::new_v4();
let epoch_text = epoch.to_string();
fi.set_object_transaction_epoch(epoch);
assert_eq!(fi.object_transaction_epoch().expect("written epoch should decode"), Some(epoch));
assert_eq!(fi.metadata.get("x-rustfs-internal-object-transaction-epoch"), Some(&epoch_text));
assert_eq!(fi.metadata.get("x-minio-internal-object-transaction-epoch"), Some(&epoch_text));
let mut rustfs_only = validation_test_fileinfo();
rustfs_only
.metadata
.insert("x-rustfs-internal-object-transaction-epoch".to_string(), epoch_text);
assert_eq!(
rustfs_only
.object_transaction_epoch()
.expect("single compatibility key should decode"),
Some(epoch)
);
let mut conflicting = fi.clone();
conflicting
.metadata
.insert("x-minio-internal-object-transaction-epoch".to_string(), Uuid::new_v4().to_string());
assert_eq!(conflicting.object_transaction_epoch(), Err(Error::FileCorrupt));
let mut malformed = validation_test_fileinfo();
malformed
.metadata
.insert("x-rustfs-internal-object-transaction-epoch".to_string(), "not-a-uuid".to_string());
assert_eq!(malformed.object_transaction_epoch(), Err(Error::FileCorrupt));
let mut nil = validation_test_fileinfo();
nil.set_object_transaction_epoch(Uuid::nil());
assert_eq!(nil.object_transaction_epoch(), Err(Error::FileCorrupt));
}
// backlog#949: distribution range/permutation validation.
#[test]
fn is_valid_distribution_accepts_permutation() {

Some files were not shown because too many files have changed in this diff Show More