aaffd3ede8
fix(ecstore): align Deep verifier shard geometry ( #4656 )
2026-07-10 08:06:38 +00:00
3531abb34a
feat(heal): disk-walk UNION enumeration to heal sub-quorum versions (backlog#920) ( #4527 )
...
B5 switched heal enumeration to list_object_versions, which only reflects the
read-quorum metadata view: a version present on fewer than read-quorum disks was
never enumerated, so it was never healed. Add a per-erasure-set disk-walk UNION
enumerator (mirrors MinIO global-heal.go objQuorum=1 listPathRaw +
mergeXLV2Versions) that surfaces every (object, version) present on ANY disk and
feeds each to the existing per-version heal_object.
- filemeta: MetaCacheEntries::resolve_union (dir_quorum=1/obj_quorum=1) yields the
cross-disk version union at one tested seam.
- ecstore: SetDisks::heal_walk_versions_page (list_path_raw fan-out, min_disks=1,
dual object/version page bound, inclusive-forward de-overlap) + ECStore delegator
+ HealWalkVersion.
- ecstore data-safety guard: before dangling-delete, try_regenerate_recoverable_meta
physically probes part files via check_parts; when >= data_blocks data shards
survive (meta lost but data recoverable) it regenerates xl.meta from a surviving
FileInfo with the correct per-disk shard index instead of dangling-deleting.
Genuine torn writes (< data_blocks) keep the current behavior — no resurrection.
- heal: dw1: forward-marker cursor codec (reuses ResumeState.resume_cursor,
idempotent restart on foreign tokens); list_versions_for_heal_page_disk_walk
trait method (default falls back to the B5 read-quorum path); heal_bucket_with_resume
selects the disk-walk enumerator when scan_mode==Deep || source==AutoHeal, else
the unchanged B5 path; anti-loop guard aborts on (empty && truncated).
Closes rustfs/backlog#920
2026-07-09 01:07:54 +08:00
2b063b0c4a
fix(heal): repair all object versions during disk-replacement heal ( #4400 )
...
Disk-replacement heal previously repaired only the latest version of each
object and never enumerated objects whose latest version is a delete marker,
so old versions were left unrepaired on a replaced drive.
Switch heal enumeration from list_objects_v2 (latest-only) to
list_object_versions (every version incl. delete markers), thread the concrete
version_id into the existing per-version heal_object, and make resume
cursor-based instead of positional: an opaque (marker, version_marker) paging
token persisted in ResumeState, a length-prefixed injective per-version dedup
key, schema_version bumps (v2) migrated independently in each of the two
persisted files, and a retry that resets both managers together (fixing a
latent rescan-skips-everything defect). Adds a real-disk-wipe e2e regression
suite proving old versions and delete-marker-latest objects are physically
restored.
Fixes rustfs/backlog#918
Fixes rustfs/backlog#919
Closes rustfs/backlog#854
2026-07-08 08:58:52 +08:00
a6b3e4f5d6
refactor: use canonical Rust module paths ( #4269 )
2026-07-05 03:01:14 +08:00
6dabbaab4d
refactor(deps): replace snafu and heal anyhow ( #4266 )
2026-07-05 00:31:37 +08:00
27468ebfa9
feat(get): consolidate GET performance optimization ( #3972 )
...
* feat(get): consolidate GET performance optimization
Consolidated implementation of all GET performance optimizations into
a single, well-organized commit replacing the previous patch-on-patch
approach.
## Changes
### Configuration (set_disk/mod.rs)
- Consolidated all GET optimization flags into a single organized section
- Enabled by default: codec streaming, metadata early-stop, page cache reclaim
- Added codec streaming multipart flag (default: disabled)
- Added version-aware early-stop flag (default: disabled)
- Added adaptive duplex buffer sizing based on object size
- All flags use OnceLock caching with rollout percentage support
### Metadata Early-Stop (set_disk/read.rs)
- Delete marker early-stop when quorum agrees
- Version-aware early-stop for versioned GET requests
- MetadataQuorumAccumulator enhanced with:
- delete_marker_votes tracking
- requested_version_id and matching_version_votes tracking
- version_early_stop_decision() method
- 6 new tests for version early-stop scenarios
### Codec Streaming (erasure/coding/decode_reader.rs)
- DualInFlight (2-stripe lookahead) enabled by default
### Decode Pipeline (erasure/coding/decode.rs)
- Stripe prefetch count configuration
- Bitrot-decode overlap configuration
### Disk Layer (disk/local.rs)
- O_DIRECT read configuration constants (preparation)
### Metrics (io-metrics/lib.rs)
- BytesPool acquisition/return metrics
- Metadata phase duration with early-stop label
- Total duration with reader_path label
### Diagnostics (diagnostics/)
- Early-stop reason constants
- Pool tier/outcome label constants
### Observability (.docker/observability/)
- 3 Grafana dashboards for GET optimization monitoring
- Prometheus alert rules (6 alerts: 3 critical, 3 warning)
- Updated README.md and README_ZH.md with usage docs
### Config (config/src/constants/runtime.rs)
- Page cache reclaim read enabled by default
## Environment Variables
| Variable | Default | Description |
|----------|---------|-------------|
| RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag |
| RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % |
| RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming |
| RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag |
| RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % |
| RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop |
| RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim |
| RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) |
| RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch |
| RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap |
| RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes |
## Rollback
All optimizations can be disabled via environment variables:
RUSTFS_GET_CODEC_STREAMING_ENABLE=false
RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false
RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false
Co-Authored-By: heihutu <heihutu@gmail.com >
* test(get): add stress test scripts for GET optimization validation
- quick-validate-get-optimization.sh: Quick 5-minute validation
- stress-test-get-optimization.sh: Full 30+ minute stress test
- README-stress-test.md: Usage documentation
Co-Authored-By: heihutu <heihutu@gmail.com >
* test(ecstore): align file cache reclaim defaults
* chore(deps): update redis and erasure codec
* test(ecstore): align decode fill policy default
* test(ecstore): align metadata early-stop default
* fix(ecstore): keep metadata early stop opt-in
---------
Co-authored-by: heihutu <heihutu@gmail.com >
2026-06-28 07:14:07 +08:00
72ae43cb90
refactor: segment external storage contract imports ( #3908 )
2026-06-26 16:44:59 +08:00
1f7e159388
refactor: segment external storage api boundaries ( #3903 )
2026-06-26 15:10:49 +08:00
4d6ea453a7
refactor: route scanner heal storage boundaries ( #3888 )
2026-06-26 05:24:41 +08:00
30559f7e1b
refactor: collapse test fuzz ecstore thin bridges ( #3765 )
2026-06-23 07:04:51 +08:00
198fd4f150
refactor(runtime): route RustFS runtime consumers through storage owner ( #3756 )
2026-06-23 05:17:56 +08:00
e0b79aa00c
refactor: expose test fuzz owner symbols ( #3752 )
2026-06-23 01:11:46 +08:00
88a87b3e8f
refactor: centralize external ECStore facade aliases ( #3746 )
2026-06-22 20:37:13 +08:00
b63b076275
refactor: remove remaining compatibility bridges ( #3738 )
2026-06-22 16:53:14 +08:00
7d81183cf1
refactor: use relative heal test compat consumers ( #3733 )
2026-06-22 14:55:12 +08:00
de158743c3
refactor: split compat facade imports ( #3716 )
2026-06-22 09:53:22 +08:00
d39815470e
refactor: prune consumer storage compat paths ( #3704 )
2026-06-22 02:31:30 +08:00
6b3d96fde3
refactor: prune trait import compat re-exports ( #3699 )
2026-06-21 23:17:58 +08:00
0cf9d07e40
refactor: prune test protocol compat aliases ( #3693 )
2026-06-21 18:09:39 +08:00
0985225448
refactor: consolidate ecstore public facade ( #3678 )
...
* refactor: expand ecstore compatibility facade
* test: enforce ecstore api facade imports
* refactor: hide legacy ecstore layout modules
* refactor: hide ecstore facade root modules
2026-06-21 09:09:11 +08:00
9b3fda9e30
refactor: route ecstore public surfaces through api ( #3673 )
2026-06-21 05:23:23 +08:00
ada6f7587e
refactor: flatten test harness storage compat aliases ( #3596 )
...
* refactor: flatten test harness storage compat aliases
* refactor: flatten rustfs storage compat aliases (#3597 )
* refactor: prune runtime storage compat surface (#3598 )
* refactor: flatten runtime secondary storage compat (#3599 )
* docs: add scheduler placement profiling baselines (#3600 )
* feat: add observability topology capability contracts (#3601 )
2026-06-19 08:30:47 +08:00
b1c6578df1
refactor: narrow test harness compatibility surfaces ( #3592 )
2026-06-19 07:10:52 +08:00
08371a6e09
refactor: narrow storage compatibility exports ( #3576 )
2026-06-18 18:31:14 +08:00
c098184c49
refactor: clean runtime and test storage boundaries ( #3573 )
...
* refactor: clean runtime observability select boundaries
* refactor: clean test harness fuzz storage boundaries
2026-06-18 18:09:25 +08:00
0e7e2660bb
refactor: clean remaining storage DTO imports ( #3568 )
2026-06-18 14:44:46 +08:00
acdf439371
refactor: clean external DTO consumers ( #3566 )
...
* refactor: clean external DTO consumers
* fix: handle empty erasure shard recovery
2026-06-18 14:22:44 +08:00
e5cad7ed20
refactor: move object operation contracts ( #3559 )
2026-06-18 09:48:13 +08:00
ed55857bca
refactor: move bucket operations contract ( #3507 )
2026-06-17 09:10:38 +08:00
7146c893cb
refactor(storage-api): remove legacy bucket DTO re-export ( #3364 )
2026-06-11 21:24:56 +08:00
28edfd6190
fix(storage): harden offline drive fail-fast paths ( #2564 )
...
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com >
2026-04-16 09:21:45 +00:00
2a3d127a86
revert: restore PutObjReader and remove PUT zero-copy chunk path ( #2351 phase 2) ( #2532 )
2026-04-14 16:35:03 +08:00
32bf8f5bf3
feat(storage): add direct chunk GET fast path ( #2351 )
...
Signed-off-by: houseme <housemecn@gmail.com >
Co-authored-by: heihutu <heihutu@gmail.com >
Co-authored-by: cxymds <Cxymds@qq.com >
2026-04-07 08:33:46 +08:00
b9b7d86ae4
feat: improve legacy metadata and admin compatibility ( #2202 )
2026-03-18 21:05:09 +08:00
af6c32efac
refactor: improve code quality with safer error handling, trait decomposition, and dead code cleanup ( #1997 )
2026-02-28 01:19:47 +08:00
9d2b8822cf
refactor: stabilize heal format recovery integration tests ( #1984 )
...
Co-authored-by: houseme <housemecn@gmail.com >
2026-02-27 15:26:19 +08:00
68cc0a5df7
chore(heal): remove unused global response broadcast ( #1991 )
2026-02-27 14:12:03 +08:00
1184806c3f
Fix/resolve pr 1710 ( #1743 )
2026-02-11 08:24:55 +08:00
ccf3b29df5
fix: stabilize head metadata responses and heal tests ( #1732 )
...
Signed-off-by: LoganZ2 <103290230+LoganZ2@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
2026-02-10 09:44:14 +08:00
dce117840c
refactor: NamespaceLock (nslock), AHM→Heal Crate, and Lock/Clippy Fixes ( #1664 )
...
Co-authored-by: Copilot <198982749+Copilot@users.noreply.github.com >
Co-authored-by: weisd <2057561+weisd@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
2026-01-30 13:13:41 +08:00
Zhengchao An