mirror of
https://github.com/rustfs/rustfs.git
synced 2026-08-10 23:26:53 +00:00
8fc637fb14d33f1907f29b2c8c85781851cf0d82
12 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 houseme
|
3f13d098b4 |
feat(observability): feature-gated hotpath instrumentation for the data path (#4394)
Merge the hotpath-rs wall-time instrumentation from the backlog#936 analysis worktree behind an opt-in 'hotpath' cargo feature, keeping the default build at zero overhead and zero dependency. - hotpath is an optional dependency everywhere (dep:hotpath feature syntax); the default dependency tree contains no hotpath crate at all - 40+ measurement points across S3 handlers, ECStore/SetDisks object and multipart ops, erasure encode/decode, bitrot, LocalDisk I/O, FileMeta codec, and HashReader - attribute sites use #[cfg_attr(feature = "hotpath", hotpath::measure)]; async_trait bodies use per-crate hp_guard! macros (ecstore + rustfs bin); rio gates measure_block! behind hp_measure_block! - feature chain: rustfs -> rustfs-ecstore -> rustfs-rio / rustfs-filemeta, each crate owning its own gate - hotpath-alloc is intentionally not wired up (hotpath 0.21.x TLS panic on cross-thread guard drop under tokio, see backlog#935); mimalloc stays the unconditional global allocator - docs/development/hotpath-profiling.md documents building, HOTPATH_* env vars, SIGTERM report flow, and how to reproduce the backlog#936 timing reports Refs: https://github.com/rustfs/backlog/issues/935 (HP-14, item 2), https://github.com/rustfs/backlog/issues/936 Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
bd5d3c5d92 |
perf(ecstore): data-shards-only lockstep GET reads with stripe-aligned deferred parity engagement (opt-in) (#4392)
* feat(ecstore): add stripe-advance handles for deferred bitrot readers Give DeferredObjectReader a shared pending state and expose a DeferredReaderStripeHandle that advances the still-unopened source by whole bitrot blocks using the same bitrot_encoded_range geometry the reader was created with (identity mapping when hash_size == 0). This lets the GET decode path open a parity shard aligned to the stripe where a data shard failed instead of reading every parity shard on every stripe (backlog#923). An already-opened (or failed) reader rejects the advance so callers retire it rather than engage it out of alignment; bitrot verification after an advance checks the advanced stripe's block against that stripe's stored hash. Co-Authored-By: heihutu <heihutu@gmail.com> * perf(ecstore): read only data shards on healthy lockstep GET behind opt-in gate PR #4289's lockstep fix made every reconstruction-verifying GET read all data+parity shards per stripe; the parity blocks are read, bitrot-hashed and then discarded, a deterministic 2x read-bytes/IOPS/hash-CPU amplification on healthy 2+2 objects (backlog#923). With the new opt-in gate RUSTFS_GET_LOCKSTEP_DATA_SHARDS_ONLY_ENABLE=true (default: false, behavior identical to main): - read_lockstep keeps only the data slots engaged while the object is healthy; parity slots stay unopened deferred readers. - When a data shard is missing or dies at stripe k, parity readers are engaged mid-object by advancing their deferred stripe handle to stripe k, preserving the lockstep alignment invariant from backlog#832. - Degraded stripes engage one parity beyond the decode quorum so reconstruction verification keeps an extra source to check against (erasure.rs only verifies when available > data shards); an engaged parity reader that errors is retired for the rest of the object like any other, and a parity reader that cannot be realigned is retired instead of being read out of position. - fill_deferred_bitrot_readers records stripe handles for deferred slots and, gate-on only, swaps eagerly opened parity readers for unopened deferred ones so they remain engageable mid-object; ready/error bookkeeping used by quorum decisions is untouched. - Both GET paths (legacy duplex via Erasure::decode_with_stripe_handles, codec streaming via ParallelReader::with_deferred_parity_handles) carry the handles from reader setup. Short-read -> UnexpectedEof -> whole-object retirement and the inconsistent-source rejection are unchanged in both gate modes; tests lock the healthy-path data-shards-only call counts, the default read-all-shards behavior, mid-object parity engagement for streaming and hash_size==0 formats, and mid-stream inconsistent-parity rejection. Co-Authored-By: heihutu <heihutu@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
Zhengchao An
|
655c5cb403 |
fix(ecstore): reject short-read shards and advance ParallelReader per stripe (backlog#799 B2) (#4327)
Two compounding defects let a truncated shard corrupt a GET (ranged GET could return HTTP 200 with wrong bytes): - `BitrotReader::read` returned `Ok(short_len)` when the shard stream hit EOF before filling the caller's buffer. With `skip_verify` / `HashAlgorithm::None` / parity=0 there is no hash to catch it, so the short shard was accepted and every downstream byte shifted. - `ParallelReader.offset` was set once and never advanced per stripe, so every stripe after the first reused the first stripe's geometry and the last-stripe length clamp was wrong. Fix both (they are mutually required): - `BitrotReader::read` now errors (`UnexpectedEof`) on a short read, before and independent of the bitrot hash check, so it fires under skip-verify/no-hash too. The caller sizes the buffer to the expected per-stripe shard length, so "buffer not filled" == "shard truncated". A short read routes through the existing `errs[i]` path, dropping that reader from the stripe so parity reconstruction engages; with parity=0 the stripe fails read quorum and the GET errors loudly instead of streaming shifted bytes. Mirrors MinIO's `parallelReader.Read` (`n != shardSize` -> reader failed). - `ParallelReader::read` / `read_lockstep` advance `self.offset += shard_size` per stripe so the per-stripe expected length (incl. the shorter final stripe) is exact, matching the correct pattern already used by heal. Updates three bitrot tests that used an obsolete oversized-buffer pattern to size per-stripe (as the real decode/heal paths do), and adds a regression test that a truncated shard errors under None/HighwayHash + skip_verify. Refs backlog#799 (B2), issue rustfs/backlog#851. Design converged by two independent expert reviews referencing MinIO cmd/erasure-decode.go. |
||
|
Zhengchao An
|
e68a52ff59 | fix: replace while-let loop with for-loop to fix clippy lint on main (#4292) | ||
|
Zhengchao An
|
3e4a7c1f6a |
fix(ecstore): lockstep EC stripe read to fix large-object GET EOF (#4289)
fix(ecstore): lockstep stripe read to stop EC GET desync truncation On the reconstruction-verifying GET path, ParallelReader::read used a data-first schedule: it read only `data_shards` readers per stripe and pulled in a parity reader as a substitute on demand. Because the shard readers are streaming (advanced only by being read, no seek), a parity reader first used mid-object was still positioned at its stream start (block 0) and returned an earlier stripe than the surviving data shards. Every shard passed its own bitrot hash, yet the set was mutually misaligned, so decode_data_with_reconstruction_verification correctly rejected it with "inconsistent read source shards" and the large-object GET truncated mid-stream (client "unexpected EOF"). Add read_lockstep(): on the verify_reconstruction path, read every live shard reader once per stripe and wait for all of them, so all readers advance one block per stripe and stay mutually aligned; any reader that errors is retired for the rest of the object (a stream that failed mid-block can no longer be trusted to be aligned). The adaptive data-first path is unchanged for non-verifying callers (e.g. heal). Adds a regression test reproducing a data shard that dies partway through a multi-stripe object; it must still reconstruct byte-exact output. Refs backlog#832. |
||
|
houseme
|
25d80d7c60 |
feat(storage): harden internode data-path controls (#4224)
* fix(rio): propagate http writer shutdown errors * fix(ecstore): unify remote lock rpc deadlines * fix(storage): reject corrupt read multiple payloads * feat(rio): add internode http tuning profiles * feat(metrics): add internode baseline signals * feat(ecstore): observe shard locality topology * feat(ecstore): gate shard locality scheduling * feat(ecstore): gate batch read version rpc * feat(ecstore): observe batch processor adaptation * feat(ecstore): gate batch processor observation * docs: add get benchmark regression analysis * docs: add issue 797 execution plan status * fix(ecstore): require explicit batch rpc support * fix(ecstore): honor documented batch read gate * fix(ecstore): keep batch read gate stable per call * chore: update workspace dependencies * feat(ecstore): log batch read gate decisions * feat(ecstore): count batch read gate decisions * test(issue-797): add local internode A/B runner * test(rio): fix tuning profile spelling fixture * fix(protocols): adapt sftp channel open callbacks * fix(metrics): wrap batch processor observation args * chore(docs): keep issue notes local only * fix(storage): address internode review feedback * fix(storage): address internode data-path review findings - Run the BatchReadVersion auto-mode unary fallback outside the batch RPC deadline so each read_version keeps its own per-op timeout and health accounting instead of racing the whole batch against one drive timeout. - Cap adaptive batch-processor concurrency growth at a hard multiple of the configured baseline so sustained fast batches cannot ratchet past the configured limit. - Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE, and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading the environment on hot paths. - Skip shard read-cost collection in observe mode when stage metrics are disabled, and cache the local endpoint host list instead of rebuilding it on every read. - Allow --warp-extra-args values starting with -- and drop the unused warp_hosts_csv helper in the issue-797 A/B runner. * fix(storage): address internode data-path review findings - Run the BatchReadVersion auto-mode unary fallback outside the batch RPC deadline so each read_version keeps its own per-op timeout and health accounting instead of racing the whole batch against one drive timeout. - Cap adaptive batch-processor concurrency growth at a hard multiple of the configured baseline so sustained fast batches cannot ratchet past the configured limit. - Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE, and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading the environment on hot paths. - Skip shard read-cost collection in observe mode when stage metrics are disabled, and cache the local endpoint host list instead of rebuilding it on every read. - Allow --warp-extra-args values starting with -- and drop the unused warp_hosts_csv helper in the issue-797 A/B runner. Co-Authored-By: heihutu<heihutu@gmail.com> * fix(storage): align buffer clamp test with media cap * fix(ecstore): release optimized read locks before streaming --------- Co-authored-by: Zhengchao An <anzhengchao@gmail.com> |
||
|
Zhengchao An
|
cf056b39e3 |
fix(core-storage): fix critical correctness defects from core-storage reliability audit (#4222)
* fix(core-storage): fix critical correctness defects from core-storage audit Fixes verified defects found in a deep audit of the core storage path (erasure coding, disk persistence, quorum, heal, replication resync): - ecstore/disk: rewrite live xl.meta atomically (temp+rename) in delete_versions_internal and write_metadata instead of in-place truncate, which exposed torn metadata to concurrent readers and crashes on the DeleteObjects hot path - ecstore/erasure: allow heal to reconstruct from exactly data_shards bitrot-verified sources; requiring data_shards+1 made objects permanently unhealable after losing parity_shards disks - ecstore/set_disk: direct-memory inline GET applied the erasure distribution permutation twice (shuffled inputs re-indexed through distribution), concatenating wrong shards into the response body in degraded reads; collect from canonical disk-ordered inputs - ecstore/set_disk: heal now preserves the committed inline layout instead of recomputing it with a hardcoded unversioned threshold, which split quorum identity of healed replicas and caused endless re-heal churn - ecstore/replication: resync results channel switched from broadcast(1) to mpsc; a lagged broadcast receiver ended the stats collector and every subsequent failure went uncounted, letting failed resyncs be marked completed - ecstore/replication: ignore an empty persisted resync checkpoint; resuming with one skipped every object and marked the resync completed without replicating anything - ecstore/replication: fix inverted not-found error classification in replicate_object/replicate_delete logging paths - ecstore/erasure: guard decode paths against zero block_size or data_shards from corrupt on-disk metadata (divide-by-zero panic) - ecstore/disk: os::read_dir no longer consumes the entry limit on entries it does not return (is_empty_dir misjudgment); create_file opens with O_TRUNC to avoid stale trailing bytes - filemeta: treat Some(nil) version id as a null version in matches_not_strict; disk-loaded headers never store None, so the mod_time quorum guard for unversioned overwrites never fired and an interrupted overwrite could displace the committed version in merge - filemeta: fix msgpack skip lengths for fixext (missed the ext type byte) and ext16/32 (over-skipped) unknown fields - filemeta: return FileCorrupt instead of usize underflow when xl.meta is truncated inside the CRC trailer - filemeta: surface delete-marker insertion failure in delete_version instead of reporting success when the data dir is shared Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> * fix(replication): drop duplicate cfg(test) etag import from boundary module The test module already imports content_matches_by_etag locally, so the top-level cfg(test) import is unused under -D warnings and fails clippy. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com> --------- Co-authored-by: Claude Fable 5 <noreply@anthropic.com> |
||
|
houseme
|
7a075c91da | perf: avoid eager parity reader setup (#4133) | ||
|
houseme
|
46d7f9e1f2 |
feat(get): harden codec streaming rollout (#3981)
* feat(get): consolidate GET performance optimization Consolidated implementation of all GET performance optimizations into a single, well-organized commit replacing the previous patch-on-patch approach. ## Changes ### Configuration (set_disk/mod.rs) - Consolidated all GET optimization flags into a single organized section - Enabled by default: codec streaming, metadata early-stop, page cache reclaim - Added codec streaming multipart flag (default: disabled) - Added version-aware early-stop flag (default: disabled) - Added adaptive duplex buffer sizing based on object size - All flags use OnceLock caching with rollout percentage support ### Metadata Early-Stop (set_disk/read.rs) - Delete marker early-stop when quorum agrees - Version-aware early-stop for versioned GET requests - MetadataQuorumAccumulator enhanced with: - delete_marker_votes tracking - requested_version_id and matching_version_votes tracking - version_early_stop_decision() method - 6 new tests for version early-stop scenarios ### Codec Streaming (erasure/coding/decode_reader.rs) - DualInFlight (2-stripe lookahead) enabled by default ### Decode Pipeline (erasure/coding/decode.rs) - Stripe prefetch count configuration - Bitrot-decode overlap configuration ### Disk Layer (disk/local.rs) - O_DIRECT read configuration constants (preparation) ### Metrics (io-metrics/lib.rs) - BytesPool acquisition/return metrics - Metadata phase duration with early-stop label - Total duration with reader_path label ### Diagnostics (diagnostics/) - Early-stop reason constants - Pool tier/outcome label constants ### Observability (.docker/observability/) - 3 Grafana dashboards for GET optimization monitoring - Prometheus alert rules (6 alerts: 3 critical, 3 warning) - Updated README.md and README_ZH.md with usage docs ### Config (config/src/constants/runtime.rs) - Page cache reclaim read enabled by default ## Environment Variables | Variable | Default | Description | |----------|---------|-------------| | RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag | | RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % | | RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming | | RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag | | RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % | | RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop | | RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim | | RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) | | RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch | | RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap | | RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes | ## Rollback All optimizations can be disabled via environment variables: RUSTFS_GET_CODEC_STREAMING_ENABLE=false RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false Co-Authored-By: heihutu <heihutu@gmail.com> * test(get): add stress test scripts for GET optimization validation - quick-validate-get-optimization.sh: Quick 5-minute validation - stress-test-get-optimization.sh: Full 30+ minute stress test - README-stress-test.md: Usage documentation Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): align file cache reclaim defaults * chore(deps): update redis and erasure codec * test(ecstore): align decode fill policy default * fix(get): wire codec streaming rollout gate * perf(get): skip metrics-off codec timers * test(get): capture codec streaming diagnostics * test(get): add multipart fallback probe * test(get): add encrypted fallback probe * test(get): add compressed fallback probe * test(get): add degraded read fallback probe * test(get): cover remote fallback probe * test(get): report warp request p99 * test(get): capture OTLP metric deltas * perf(get): align codec streaming inflight default * perf(get): reuse codec reader output buffers * test(get): count codec reader fill starts * perf(get): reuse codec reader fill worker * perf(get): lazy init rustfs codec reconstruct * test(get): cover rustfs codec source faults * docs(get): record rustfs codec fallback scope * feat(get): add multipart codec reader opt-in * test(get): add multipart codec smoke option * test(get): cover multipart codec degraded fallback * perf(get): bound multipart codec eager setup * test(get): satisfy codec hardening PR gate --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
houseme
|
27468ebfa9 |
feat(get): consolidate GET performance optimization (#3972)
* feat(get): consolidate GET performance optimization Consolidated implementation of all GET performance optimizations into a single, well-organized commit replacing the previous patch-on-patch approach. ## Changes ### Configuration (set_disk/mod.rs) - Consolidated all GET optimization flags into a single organized section - Enabled by default: codec streaming, metadata early-stop, page cache reclaim - Added codec streaming multipart flag (default: disabled) - Added version-aware early-stop flag (default: disabled) - Added adaptive duplex buffer sizing based on object size - All flags use OnceLock caching with rollout percentage support ### Metadata Early-Stop (set_disk/read.rs) - Delete marker early-stop when quorum agrees - Version-aware early-stop for versioned GET requests - MetadataQuorumAccumulator enhanced with: - delete_marker_votes tracking - requested_version_id and matching_version_votes tracking - version_early_stop_decision() method - 6 new tests for version early-stop scenarios ### Codec Streaming (erasure/coding/decode_reader.rs) - DualInFlight (2-stripe lookahead) enabled by default ### Decode Pipeline (erasure/coding/decode.rs) - Stripe prefetch count configuration - Bitrot-decode overlap configuration ### Disk Layer (disk/local.rs) - O_DIRECT read configuration constants (preparation) ### Metrics (io-metrics/lib.rs) - BytesPool acquisition/return metrics - Metadata phase duration with early-stop label - Total duration with reader_path label ### Diagnostics (diagnostics/) - Early-stop reason constants - Pool tier/outcome label constants ### Observability (.docker/observability/) - 3 Grafana dashboards for GET optimization monitoring - Prometheus alert rules (6 alerts: 3 critical, 3 warning) - Updated README.md and README_ZH.md with usage docs ### Config (config/src/constants/runtime.rs) - Page cache reclaim read enabled by default ## Environment Variables | Variable | Default | Description | |----------|---------|-------------| | RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag | | RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % | | RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming | | RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag | | RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % | | RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop | | RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim | | RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) | | RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch | | RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap | | RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes | ## Rollback All optimizations can be disabled via environment variables: RUSTFS_GET_CODEC_STREAMING_ENABLE=false RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false Co-Authored-By: heihutu <heihutu@gmail.com> * test(get): add stress test scripts for GET optimization validation - quick-validate-get-optimization.sh: Quick 5-minute validation - stress-test-get-optimization.sh: Full 30+ minute stress test - README-stress-test.md: Usage documentation Co-Authored-By: heihutu <heihutu@gmail.com> * test(ecstore): align file cache reclaim defaults * chore(deps): update redis and erasure codec * test(ecstore): align decode fill policy default * test(ecstore): align metadata early-stop default * fix(ecstore): keep metadata early stop opt-in --------- Co-authored-by: heihutu <heihutu@gmail.com> |
||
|
GatewayJ
|
675597ec16 |
fix(ecstore): handle stalled recovery reads and listings (#3790)
* fix(ecstore): handle stalled recovery reads and listings * fix(rio): start HTTP stall timeout on read * fix(ecstore): handle stalled reads and partial lists * fix(ecstore): retire stalled shards and list errors * fix(ecstore): preserve list merge lookahead entries * fix(ecstore): bound zero-copy shard reads * fix(ecstore): hedge stalled shard reads * fix(ecstore): retire abandoned shard reads * fix(ecstore): include part identity in metadata quorum * fix(ecstore): validate heal shard sources * fix(ecstore): verify reconstructed read shards * chore(ecstore): log slow object read stages * fix(heal): throttle auto heal during recovery * fix(scanner): yield to foreground reads * fix(scanner): track streaming object reads * fix(ecstore): avoid false read heal fanout * fix(ecstore): verify codec streaming reconstruction sources * fix(ecstore): preserve quorum progress on slow shards * fix(storage): restore read timeout facade * fix(ecstore): retain fallback readers after quorum * chore: allow decode helper argument lists --------- Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Zhengchao An
|
0a5b1b1b3a |
refactor: consolidate ecstore owner module layout (#3934)
* refactor: shrink ecstore root owner facades * refactor: remove ecstore core store root shims * refactor: move ecstore erasure owner modules * refactor: remove ecstore root rpc facade * refactor: move ecstore services domain modules |