Keep fresh metadata fanout results owned while sharing cache-backed metadata through Arc, avoiding deep clones on eligible local cache hits without enabling unsafe distributed caching.
Co-authored-by: heihutu <heihutu@gmail.com>
Keep common shard-indexed decode scratch vectors inline while preserving heap fallback for larger supported erasure layouts. Consume scratch iterators directly at the stripe-state boundary to avoid reallocating.
Co-authored-by: heihutu <heihutu@gmail.com>
put_object_part's commit phase held an exclusive write lock on the whole
upload_id_path namespace, so concurrent UploadPart commits for different
part numbers of one upload serialized behind a single lock and returned
503 once the 5s lock-acquire timeout elapsed.
Adopt MinIO's PutObjectPart lock scope: a shared read lock on the
uploadId namespace plus an exclusive write lock on
{upload_id_path}/part.{N}. Different part numbers now commit
concurrently; same-part retries still serialize (backlog#853);
complete/abort keep the uploadId write lock and still exclude every
in-flight part commit. The lock-loss fence covers both guards.
Fixes#5961
Every GET fans out a `read_version` across all disks to resolve xl.meta. Each
fanout allocated an `Arc<ReadOptions>` (3 bools) plus four `Arc<String>`
(`Arc::new(x.to_string())` = two allocations each) and cloned them into every
spawned task. This trims the per-fanout allocation footprint.
- `ReadOptions` is three bools, so it is now `Copy`. The fanout drops the
`Arc<ReadOptions>` and hands each spawned task a copy; the two pre-existing
`ReadOptions::clone()` sites (set_disk/read.rs, set_disk/ops/heal.rs) stop
cloning a `Copy` type.
- The four request strings use `Arc::<str>::from(&str)` (one allocation each)
instead of `Arc::new(..to_string())` (string buffer + Arc = two each) — four
fewer allocations per fanout, transparent to the `read_version(&str)` call.
Behavior is unchanged: the fanout still spawns one task per disk (the spawn is
deliberate — `read_version_call_counter_observes_spawned_fanout` verifies the
process-global counter observes every per-disk increment across workers), quorum
/ early-stop / full-wait semantics are untouched, and no result ordering or
error handling changed.
Two larger items from the audit are intentionally NOT in this PR:
- `tokio::spawn` -> `FuturesUnordered`: the spawn is a tested, deliberate
design (cross-worker counter observation for #1309/#1314), and converting
would also change panic isolation. Left as-is.
- `vec![FileInfo::default(); N]`: `FileInfo`'s empty containers (String /
HashMap / Vec) do not allocate, so this is one `Vec` allocation, not the
per-element allocation the audit implied — not a real hot spot.
`cargo fmt`, `cargo clippy -p rustfs-ecstore --lib` (0 warnings),
`cargo check --lib --tests`, and the 26 fanout / call-counter unit tests pass
on macOS (the change is fully cross-platform).
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
A small object whose data shards are inlined in xl.meta can be reassembled
straight from the already-resolved metadata, skipping the Erasure reconstruct
pipeline. The fast path existed but was opt-in
(`RUSTFS_GET_SMALL_OBJECT_DIRECT_MEMORY`, default off), so every deployment
paid the full shard-read fan-out for eligible small GETs by default.
- Flip `DEFAULT_RUSTFS_GET_SMALL_OBJECT_DIRECT_MEMORY` to `true`. The path is
correctness-neutral on a miss: `try_get_object_direct_data_shards_*` returns
None when the inline reassembly cannot satisfy the read, and the GET then
proceeds through the normal shard-read pipeline. The env stays as a kill
switch (`=false` restores the legacy path).
- Drop the bucket-level `versioned` / `version_suspended` exclusions. The
decision is made on `fi` — the already-resolved target version — so
reassembling its inlined data is correct on a versioned bucket too. An
explicit versionId GET still falls back (`opts.version_id`), and a
delete-marker latest is still rejected. The two now-unused fallback reasons
and their metric labels are removed.
- Tests updated: a versioned latest-version object is now eligible / `Use`
(covers the newly allowed path); the removed reasons' label assertions are
dropped.
cargo check --lib --tests and the direct_memory unit tests pass on macOS
(the change is fully cross-platform).
Co-authored-by: heihutu <heihutu@gmail.com>
Keep opt-in bounded GET data-read fanout from waiting on a single pending ReadVersion response when an unscheduled spare disk can satisfy quorum. Add a deterministic 2+2 regression that pauses the third scheduled metadata read and verifies the spare is started before returning.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Assert the prepared GET metadata path keeps the default full data-read fanout after PR #5929 made bounded data-read fanout opt-in.
Co-authored-by: heihutu <heihutu@gmail.com>
Keep GET data-read metadata early-stop and bounded fanout behind explicit environment switches so the default path preserves full fanout read-failure tolerance.
Retain the focused opt-in A/B coverage and the invalid parity full-fanout guard for heterogeneous set layouts.
Co-authored-by: heihutu <heihutu@gmail.com>
Complete the encrypted-object replication series (backlog#1783, PR-C of
3, after #5872 and #5885): SSE-C objects replicate as ciphertext
passthrough — the source holds no customer key, so the stored bytes and
their encryption metadata travel verbatim and the replica decrypts only
with the original customer key, single-part and multipart.
- Sender: SSE-C objects read raw (raw_data_movement_read), transfer at
ciphertext size, and range multipart parts over stored part sizes.
- Receiver: authorized replication PUTs restore the stored SSE-C keys
from the transport headers (exact lowercase forms - the read-path
check is case-sensitive), set ObjectOptions.preserve_ciphertext, and
skip compression, bucket-default SSE, and sse_encryption behind one
restore-derived gate. Multipart uses an internal session marker to
store parts verbatim and strips it on complete.
- Convergence: the replication HEAD sends
x-rustfs-source-replication-check; the target authorizes it as
ReplicateObjectAction and skips SSE-C read validation for that
request only, so keyless convergence HEADs see etag/size/mtime
instead of 400 and SSE-C replicas stop re-driving forever.
- e2e: SSE-C contract flips to a key-gated readable replica (no-key and
wrong-key GETs fail - the direct silent-plaintext detector); new
multipart passthrough contract with ETag/marker/stability assertions.
Open the managed-SSE replication gate (backlog#1783, PR-B of 3, after
#5872): the replication reader already decrypts through the injected
object-encryption resolver, so the source sends plaintext plus an
encryption intent header (AES256 / aws:kms, never the source key id) and
the target re-encrypts on its normal PUT path with its own KMS. No DEK
crosses sites.
- replication_put_object_options: fail closed only on Unsupported;
insert the SSE intent after the strip loop.
- TargetClient::create_multipart_upload sends the full opts.header()
set, fixing multipart replicas losing content-type/user metadata
(plaintext included).
- Preserve source ETag and mtime on replicas (authorized replication
only): receiver wires x-rustfs-source-etag into preserve_etag for PUT
and CompleteMultipartUpload, resolve_complete_etag consumes it, and
complete options carry source_etag/source_mtime (absent mtime
degrades to epoch, not now_utc). Without this every replication HEAD
comparison re-drives re-encrypted objects forever.
- e2e: managed SSE contracts flip to success on an independent-KMS
dual-process pair (byte-identical plain GET proves target-owned
envelopes; ETag/mtime preserved; version stable across scanner
cycles; resync converges; multipart keeps structure and metadata);
new target-without-KMS fail-closed contract; SSE-C stays FAILED.
Co-authored-by: houseme <housemecn@gmail.com>
* fix(filemeta): classify xl.meta CRC mismatch as FileCorrupt so heal repairs it
A failed CRC means the metadata bytes on disk are not the bytes that were
written — bitrot. Raising it as Error::other() surfaces a generic Io error,
which should_heal_object_on_disk does not recognise as heal-worthy: the drive
is skipped, disks_to_heal_count stays 0, heal_object returns ok, and the
corrupted xl.meta is never rewritten — while the scanner re-submits the same
no-op heal every deep-scan cycle. An explicit admin deep heal fails the same
way, so no heal path repairs metadata bitrot, and every one of them reports
success.
check_xl2_v1 already classifies a short or wrong-magic header as FileCorrupt
for exactly this reason (#5716); this completes the pattern for the two CRC
sites. The existing From<rustfs_filemeta::Error> for DiskError conversion maps
the variant to DiskError::FileCorrupt, which the heal path already handles.
The previously silent is_indexed_meta site now logs the mismatch (structured
event shape) like unmarshal_msg does.
Regression test: corrupt one byte of a marshalled FileMeta and assert
unmarshal_msg reports FileCorrupt; fails on the previous code, which returned
Io(Other).
Verified end-to-end on a 3-node / 12-drive EC:4 cluster: xl.meta corrupted on
2 of 12 drives via dd, admin deep heal — before this change the heal returns
ok with the corruption intact and the scanner loops forever; with it, both
copies are rewritten (decode-identical to the healthy quorum), the object
reads back byte-correct, and a follow-up heal reports all twelve drives
clean.
* test(filemeta): cover crc heal classification
Add regression coverage for the indexed xl.meta CRC path and the metadata-heal decision that consumes FileCorrupt.
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: terem42 <9478806+terem42@users.noreply.github.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): memoize bucket-incarnation fence validation under lifecycle read-lock coverage
The PUT commit fence from #5648 validated the bucket incarnation with an uncached read (a distributed metadata-transaction read lock plus an EC quorum read of bucket metadata) on every PUT commit. Under 64-concurrency 4KiB PUT load this adds two quorum round-trips per PUT and the resulting lock-manager pressure produced ~1,000 client-visible 'Lock acquisition timeout' failures per 5-minute window (see rustfs/backlog#1776).
Memoize the validation per node while lifecycle read-lock coverage is continuous: bucket deletion/recreation requires the lifecycle WRITE lock, so while at least one read guard on this node has been held continuously the incarnation cannot have changed. The first fenced PUT in a coverage window performs the exact authoritative disk validation as before; overlapping PUTs reuse its result. The memo clears when the node's last guard drops or any guard observes a lost lock, so the next PUT revalidates from disk. Fence semantics are unchanged; only the redundant re-validations under continuous coverage are elided.
Also right-size the s3s footprint ratchet baselines: -1 s3_error! line from this change's error-path consolidation, and +1 s3s-importing file inherited from #5763 (crates/obs/src/telemetry/filter.rs) which landed on main without the baseline bump.
* fix(ecstore): carry the bucket fence registry through the rebalance test store
The rebalance entry test constructor landed on main after this branch was cut and needs the new field.
fix(ecstore): reject over-NAME_MAX object key segments up front and classify irreconcilable parity as corrupt metadata
Two defects found during release acceptance and the backlog#1776 investigation:
Object keys with any path segment longer than 255 bytes could never be stored (each segment maps to one on-disk directory entry), but the failure surfaced only when the disk layer hit ENAMETOOLONG, which leaked to clients as InternalError 500 (rustfs#5785). Validate the on-disk segment budget in check_bucket_and_object_names so such keys fail deterministically as ObjectNameInvalid (4xx) before any I/O. Directory-object keys (trailing '/') account for the __XLDIR__ suffix their final segment carries on disk.
object_quorum_from_meta conflated two very different no-quorum situations (rustfs#5801): stray or foreign metadata whose parity values are garbage produced the same retryable-looking ErasureReadQuorum (503) as a genuine partial outage, so clients retried unrecoverable reads and monitoring could not tell corruption from capacity loss. Now (a) parity counts outside [0, total_shards] are treated as invalid entries instead of being clamped to i32::MAX, which could poison common_parity's occurrence counting, and (b) when a full read quorum of disks answers but their parity values cannot be reconciled, the error is FileCorrupt — heal-actionable and non-retryable — while too-few-healthy-replies keeps returning ErasureReadQuorum.
Verification: 4 new unit tests (segment budget boundaries incl. byte-vs-char and __XLDIR__ budget; garbage parity sanitization; corrupt-vs-quorum classification), metadata::tests + utils::tests 62/62, set_disk+bucket suites 1214 passed with the single pre-existing heal_queue_marks_missing_versioning_state_as_missed cross-test flake also failing on a clean tree (not introduced here), clippy clean, make pre-commit green.
PR #5719 fixed the issue #5716 per-object heal log amplification (per-object statements demoted, heal spans forced to TRACE, raw metadata dumps banned by guardrail) and superseded the demotion originally proposed here. This PR now carries only the residual cleanup on top of it:
- Convert the remaining bare-field and format-arg heal logs in crates/ecstore/src/set_disk/ops/heal.rs to the file's structured convention (event/component/subsystem + context fields): missing-object skip, disk-marked-for-healing, cannot-reconstruct errors, dangling-cleanup error, missing data_dir error, xl.meta regeneration warn, and orphan-reclaim failure warn.
- Demote the last remaining info! in the file — the per-set heal_format "set disk formats success, NoHealRequired" no-op message — to a structured debug! (error_count instead of a raw errs dump), and drop its whitelist exclusion in scripts/check_logging_guardrails.sh so the no-INFO check for set-disk heal files is strict.
No control flow or behavior changes.
Multipart upload ids embed the process-global deployment id at both create time and list time. Under plain cargo test (thread-parallel, shared process globals) a concurrently running test that re-initializes a store can swap the global between the two reads, making full-upload-id equality assertions fail spuriously (observed: core::sets::tests::list_multipart_uploads_merges_all_sets_without_pagination_loss failing when run concurrently with bucket::quota tests, passing in isolation).
Add a test-only upload_uuid_suffix helper next to deployment_upload_id and make the affected assertions compare only the decoded <uuid>x<timestamp> suffix. Where suffix normalization changes within-key ordering (base64 alphabet order is not byte order), both sides are sorted before comparison. nextest/CI is unaffected (process-per-test); this only hardens local plain cargo test runs.
The multipart staging namespace is one flat set of sha256(bucket/object) directories shared by every bucket, and the cross-set listing rewrite reads every upload's metadata. Two shapes poisoned the whole ListMultipartUploads response with InternalError: Corrupted format: a healthy in-flight upload belonging to another bucket (its stored owner bucket fails the guard and fell into the corrupted-format arm), and a single upload directory whose xl.meta was torn by an unclean shutdown. Docker Distribution calls ListMultipartUploads on every PATCH/commit, so either shape broke OCI registry pushes entirely (issue #5716).
Foreign-bucket uploads are now skipped silently, and directories whose metadata is affirmatively corrupt at quorum are skipped with a debug log, while every other decode failure (quorum loss from offline disks, timeouts, transport errors) keeps failing the listing so clients retry instead of silently losing entries. The degrade-vs-propagate decision is a named corrupt-family classifier with a unit test pinning both sides. FileMeta::check_xl2_v1 now classifies a missing or wrong XL2 magic as FileCorrupt instead of an anonymous io error so damage is distinguishable from transient IO faults.
Refs #5716
#5724 reclaimed the synthetic inline-rollback dir after a committed rename with delete_data_dir(recursive: true), which has no notion of object metadata: for unversioned objects the synthetic UUID is a fixed, publicly-known constant, so object/<rollback-dir> can simultaneously be a legitimate child key's directory, and recursively deleting it reopens the authorization bypass #5703 closed (PutObject on K destroying K/<uuid> without DeleteObject permission).
Replace the recursive pass with a file-precise one: after quorum commit, delete exactly object/<rollback>/xl.meta.bkp with a non-recursive delete on every disk whose rollback dir is not also the cleanup dir. The parent-rmdir walk removes the dir only when the backup was its sole content, so the BucketNotEmpty leak fix is preserved (#5724's regression test passes unchanged) while a child key at the same path keeps its metadata. The undo path's restore_metadata_backup now also reclaims the emptied synthetic dir, mirroring restore_delete_rollback.
#5703 split rollback state from old-data-dir cleanup so the synthetic inline-rollback dir is reported only as rollback_data_dir and never reclaimed as if it were a real data dir. But nothing reclaims it after a successful commit either: every overwrite of an inline version by a non-inline one leaves <object>/<rollback-dir>/xl.meta.bkp behind. The residue is not referenced by any version, so it survives object deletion and DeleteBucket fails with BucketNotEmpty forever — the mass teardown cascade currently failing the S3 Implemented Tests CI lane.
Reclaim the synthetic dirs in SetDisks::rename_data once the commit holds write quorum. The quorum-failure undo inside the same function is the only consumer of the backup, so its window is closed at that point. Best-effort with the same anti-misdelete posture as commit_rename_data_dir: never touch the just-committed data dir, and residue must not fail a durable write (backlog#898).
* refactor(time): migrate audit and notify timestamps to jiff
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): initialize heal walk decode error
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(targets): parse MySQL event time with jiff
Preserve MySQL DATETIME(6) wall-time formatting for RFC3339 eventTime values while removing the direct chrono dependency from rustfs-targets.
Co-Authored-By: heihutu <heihutu@gmail.com>
* chore(deps): prune unused workspace dependencies
Apply cargo shear --fix to remove unused path-clean and s3select-api tempfile entries after the scoped jiff migration.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(ecstore): remove duplicate heal walk decode error init
Remove the duplicate decode_error field from the heal walk test collector initializer so lib-test clippy compiles on CI.
Co-Authored-By: heihutu <heihutu@gmail.com>
* refactor(policy): emit OPA timestamps with jiff
Co-Authored-By: heihutu <heihutu@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
When a returning node carries a stale object version that was deleted on
the quorum, the heal disk-walk partial callback used `resolve_union`
which picks only one entry from divergent disk entries. The minority
version was never enumerated and therefore never cleaned up.
Replace `resolve_union` + `ingest` with a new `ingest_merged` that
collects all unique versions from every partial entry across disks,
deduplicating by (name, version_id). This ensures stale data on a
returning node is surfaced for healing and can be deleted as dangling.
Fixes#5029
Classify expected metadata-missing errors separately from unknown get pipeline failures and attribute internal meta-bucket reader failures to an internal_meta path instead of legacy_duplex.
This keeps scanner/data-usage metadata probes from polluting user GET/mixed failure attribution while preserving the existing read error behavior.
Co-authored-by: heihutu <heihutu@gmail.com>
Collapse per-shard blocking-pool round-trips into one spawn_blocking
pread batch when all online shards are local and mmap-read is enabled.
Co-authored-by: ba <ba@ubuntu-server.alpha30.bos16>
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
houseme