63b4568f85
fix(ecstore): reclaim orphan data dirs on the healthy heal path ( #4781 )
...
PR #4356 wired `reclaim_orphan_data_dirs` only into `heal_object`'s
post-heal tail, which runs after the `disks_to_heal_count == 0` early
return. That early return is exactly the state of the objects the sweep
targets: a valid `xl.meta` with all shards present plus a leaked
pre-#3510 data dir needs no shard healing, so a healthy heal returned
before reclaim and swept nothing. On a healthy deployment (single node,
no degraded disks) the reclaim was therefore dead code — an admin heal
walked the objects, "healed" them, and reclaimed no leaked space.
Run the best-effort reclaim on the `disks_to_heal_count == 0` path as
well, gated on `!opts.dry_run`. The shared match+log block is factored
into `reclaim_orphan_data_dirs_best_effort` so both exits behave
identically. A reclaim failure still never fails the heal.
Adds an end-to-end regression: put a healthy non-inline object, plant an
unreferenced UUID data dir under it on every disk that holds the object,
then drive `heal_object`. A dry-run heal must leave the stray in place; a
real heal must reclaim it while preserving the live data dirs, `xl.meta`,
and object contents. The test fails against the pre-fix control flow.
Refs #3231 , #3191 , #4356 .
Co-authored-by: heihutu <heihutu@gmail.com >
2026-07-12 14:19:47 +00:00
a6b3e4f5d6
refactor: use canonical Rust module paths ( #4269 )
2026-07-05 03:01:14 +08:00
27468ebfa9
feat(get): consolidate GET performance optimization ( #3972 )
...
* feat(get): consolidate GET performance optimization
Consolidated implementation of all GET performance optimizations into
a single, well-organized commit replacing the previous patch-on-patch
approach.
## Changes
### Configuration (set_disk/mod.rs)
- Consolidated all GET optimization flags into a single organized section
- Enabled by default: codec streaming, metadata early-stop, page cache reclaim
- Added codec streaming multipart flag (default: disabled)
- Added version-aware early-stop flag (default: disabled)
- Added adaptive duplex buffer sizing based on object size
- All flags use OnceLock caching with rollout percentage support
### Metadata Early-Stop (set_disk/read.rs)
- Delete marker early-stop when quorum agrees
- Version-aware early-stop for versioned GET requests
- MetadataQuorumAccumulator enhanced with:
- delete_marker_votes tracking
- requested_version_id and matching_version_votes tracking
- version_early_stop_decision() method
- 6 new tests for version early-stop scenarios
### Codec Streaming (erasure/coding/decode_reader.rs)
- DualInFlight (2-stripe lookahead) enabled by default
### Decode Pipeline (erasure/coding/decode.rs)
- Stripe prefetch count configuration
- Bitrot-decode overlap configuration
### Disk Layer (disk/local.rs)
- O_DIRECT read configuration constants (preparation)
### Metrics (io-metrics/lib.rs)
- BytesPool acquisition/return metrics
- Metadata phase duration with early-stop label
- Total duration with reader_path label
### Diagnostics (diagnostics/)
- Early-stop reason constants
- Pool tier/outcome label constants
### Observability (.docker/observability/)
- 3 Grafana dashboards for GET optimization monitoring
- Prometheus alert rules (6 alerts: 3 critical, 3 warning)
- Updated README.md and README_ZH.md with usage docs
### Config (config/src/constants/runtime.rs)
- Page cache reclaim read enabled by default
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag |
| RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % |
| RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming |
| RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag |
| RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % |
| RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop |
| RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim |
| RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) |
| RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch |
| RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap |
| RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes |
## Rollback
All optimizations can be disabled via environment variables:
RUSTFS_GET_CODEC_STREAMING_ENABLE=false
RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false
RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false
Co-Authored-By: heihutu <heihutu@gmail.com >
* test(get): add stress test scripts for GET optimization validation
- quick-validate-get-optimization.sh: Quick 5-minute validation
- stress-test-get-optimization.sh: Full 30+ minute stress test
- README-stress-test.md: Usage documentation
Co-Authored-By: heihutu <heihutu@gmail.com >
* test(ecstore): align file cache reclaim defaults
* chore(deps): update redis and erasure codec
* test(ecstore): align decode fill policy default
* test(ecstore): align metadata early-stop default
* fix(ecstore): keep metadata early stop opt-in
---------
Co-authored-by: heihutu <heihutu@gmail.com >
2026-06-28 07:14:07 +08:00
1f7e159388
refactor: segment external storage api boundaries ( #3903 )
2026-06-26 15:10:49 +08:00
4d6ea453a7
refactor: route scanner heal storage boundaries ( #3888 )
2026-06-26 05:24:41 +08:00
30559f7e1b
refactor: collapse test fuzz ecstore thin bridges ( #3765 )
2026-06-23 07:04:51 +08:00
198fd4f150
refactor(runtime): route RustFS runtime consumers through storage owner ( #3756 )
2026-06-23 05:17:56 +08:00
e0b79aa00c
refactor: expose test fuzz owner symbols ( #3752 )
2026-06-23 01:11:46 +08:00
88a87b3e8f
refactor: centralize external ECStore facade aliases ( #3746 )
2026-06-22 20:37:13 +08:00
b63b076275
refactor: remove remaining compatibility bridges ( #3738 )
2026-06-22 16:53:14 +08:00
7d81183cf1
refactor: use relative heal test compat consumers ( #3733 )
2026-06-22 14:55:12 +08:00
ada6f7587e
refactor: flatten test harness storage compat aliases ( #3596 )
...
* refactor: flatten test harness storage compat aliases
* refactor: flatten rustfs storage compat aliases (#3597 )
* refactor: prune runtime storage compat surface (#3598 )
* refactor: flatten runtime secondary storage compat (#3599 )
* docs: add scheduler placement profiling baselines (#3600 )
* feat: add observability topology capability contracts (#3601 )
2026-06-19 08:30:47 +08:00
c098184c49
refactor: clean runtime and test storage boundaries ( #3573 )
...
* refactor: clean runtime observability select boundaries
* refactor: clean test harness fuzz storage boundaries
2026-06-18 18:09:25 +08:00
0e7e2660bb
refactor: clean remaining storage DTO imports ( #3568 )
2026-06-18 14:44:46 +08:00
e5cad7ed20
refactor: move object operation contracts ( #3559 )
2026-06-18 09:48:13 +08:00
ed55857bca
refactor: move bucket operations contract ( #3507 )
2026-06-17 09:10:38 +08:00
7146c893cb
refactor(storage-api): remove legacy bucket DTO re-export ( #3364 )
2026-06-11 21:24:56 +08:00
28edfd6190
fix(storage): harden offline drive fail-fast paths ( #2564 )
...
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com >
2026-04-16 09:21:45 +00:00
2a3d127a86
revert: restore PutObjReader and remove PUT zero-copy chunk path ( #2351 phase 2) ( #2532 )
2026-04-14 16:35:03 +08:00
32bf8f5bf3
feat(storage): add direct chunk GET fast path ( #2351 )
...
Signed-off-by: houseme <housemecn@gmail.com >
Co-authored-by: heihutu <heihutu@gmail.com >
Co-authored-by: cxymds <Cxymds@qq.com >
2026-04-07 08:33:46 +08:00
b9b7d86ae4
feat: improve legacy metadata and admin compatibility ( #2202 )
2026-03-18 21:05:09 +08:00
af6c32efac
refactor: improve code quality with safer error handling, trait decomposition, and dead code cleanup ( #1997 )
2026-02-28 01:19:47 +08:00
9d2b8822cf
refactor: stabilize heal format recovery integration tests ( #1984 )
...
Co-authored-by: houseme <housemecn@gmail.com >
2026-02-27 15:26:19 +08:00
68cc0a5df7
chore(heal): remove unused global response broadcast ( #1991 )
2026-02-27 14:12:03 +08:00
1184806c3f
Fix/resolve pr 1710 ( #1743 )
2026-02-11 08:24:55 +08:00
ccf3b29df5
fix: stabilize head metadata responses and heal tests ( #1732 )
...
Signed-off-by: LoganZ2 <103290230+LoganZ2@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
2026-02-10 09:44:14 +08:00
dce117840c
refactor: NamespaceLock (nslock), AHM→Heal Crate, and Lock/Clippy Fixes ( #1664 )
...
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com >
Co-authored-by: weisd <2057561+weisd@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
2026-01-30 13:13:41 +08:00
houseme