gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-28 00:58:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,788 Commits 12 Branches 117 Tags
dfcff2db28e18a8d14d70d00529945ae9bc1ed4f
Commit Graph

642 Commits

Author SHA1 Message Date
houseme dfcff2db28 feat: isolate list objects quorum config 2026-06-29 23:52:43 +08:00
Zhengchao An c475d03b48 fix: replace unwrap() with expect() in more files (#729 batch 14) (#3994) 2026-06-28 11:45:23 +08:00
Zhengchao An 05d201679c fix: replace unwrap() with expect() in more files (#729 batch 13) (#3993) 2026-06-28 11:45:13 +08:00
Zhengchao An f0ab812213 fix: replace unwrap() with expect() in remaining files (#729 batch 12) (#3992) 2026-06-28 11:45:03 +08:00
Zhengchao An 1f8a5bc095 fix(ecstore): replace unwrap() with proper error handling in api_get_object_attributes (#729 batch 11) (#3991) 
fix(ecstore): replace unwrap() with proper error handling in api_get_object_attributes

Replace unsafe unwrap() calls with proper error handling in
api_get_object_attributes.rs:
- HTTP header access now uses ok_or_else with descriptive messages
- String parsing now uses map_err with descriptive messages
- HeaderValue creation now uses expect with descriptive messages

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 11:23:41 +08:00
Zhengchao An 25170943ce fix(ecstore): improve expect() messages in admin_server_info (#729 batch 9) (#3990) 
fix(ecstore): improve expect() messages in admin_server_info

Replace unwrap() with expect() for better error diagnostics in
admin_server_info.rs:
- URL host/port access now has descriptive messages
- HashMap get_mut calls now have descriptive messages

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 11:23:04 +08:00
Zhengchao An 5c94fe7dd7 fix(ecstore): improve expect() messages in replication_resyncer (#729 batch 7) (#3988) 
fix(ecstore): improve expect() messages in replication_resyncer

Replace unwrap() with expect() for better error diagnostics in
replication_resyncer.rs:
- HashMap get_mut calls now have descriptive expect messages
- format() calls now use unwrap_or_else for error handling

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 11:22:26 +08:00
houseme 46d7f9e1f2 feat(get): harden codec streaming rollout (#3981) 
* feat(get): consolidate GET performance optimization

Consolidated implementation of all GET performance optimizations into
a single, well-organized commit replacing the previous patch-on-patch
approach.

## Changes

### Configuration (set_disk/mod.rs)
- Consolidated all GET optimization flags into a single organized section
- Enabled by default: codec streaming, metadata early-stop, page cache reclaim
- Added codec streaming multipart flag (default: disabled)
- Added version-aware early-stop flag (default: disabled)
- Added adaptive duplex buffer sizing based on object size
- All flags use OnceLock caching with rollout percentage support

### Metadata Early-Stop (set_disk/read.rs)
- Delete marker early-stop when quorum agrees
- Version-aware early-stop for versioned GET requests
- MetadataQuorumAccumulator enhanced with:
  - delete_marker_votes tracking
  - requested_version_id and matching_version_votes tracking
  - version_early_stop_decision() method
- 6 new tests for version early-stop scenarios

### Codec Streaming (erasure/coding/decode_reader.rs)
- DualInFlight (2-stripe lookahead) enabled by default

### Decode Pipeline (erasure/coding/decode.rs)
- Stripe prefetch count configuration
- Bitrot-decode overlap configuration

### Disk Layer (disk/local.rs)
- O_DIRECT read configuration constants (preparation)

### Metrics (io-metrics/lib.rs)
- BytesPool acquisition/return metrics
- Metadata phase duration with early-stop label
- Total duration with reader_path label

### Diagnostics (diagnostics/)
- Early-stop reason constants
- Pool tier/outcome label constants

### Observability (.docker/observability/)
- 3 Grafana dashboards for GET optimization monitoring
- Prometheus alert rules (6 alerts: 3 critical, 3 warning)
- Updated README.md and README_ZH.md with usage docs

### Config (config/src/constants/runtime.rs)
- Page cache reclaim read enabled by default

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag |
| RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % |
| RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming |
| RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag |
| RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % |
| RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop |
| RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim |
| RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) |
| RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch |
| RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap |
| RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes |

## Rollback

All optimizations can be disabled via environment variables:
RUSTFS_GET_CODEC_STREAMING_ENABLE=false
RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false
RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(get): add stress test scripts for GET optimization validation

- quick-validate-get-optimization.sh: Quick 5-minute validation
- stress-test-get-optimization.sh: Full 30+ minute stress test
- README-stress-test.md: Usage documentation

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): align file cache reclaim defaults

* chore(deps): update redis and erasure codec

* test(ecstore): align decode fill policy default

* fix(get): wire codec streaming rollout gate

* perf(get): skip metrics-off codec timers

* test(get): capture codec streaming diagnostics

* test(get): add multipart fallback probe

* test(get): add encrypted fallback probe

* test(get): add compressed fallback probe

* test(get): add degraded read fallback probe

* test(get): cover remote fallback probe

* test(get): report warp request p99

* test(get): capture OTLP metric deltas

* perf(get): align codec streaming inflight default

* perf(get): reuse codec reader output buffers

* test(get): count codec reader fill starts

* perf(get): reuse codec reader fill worker

* perf(get): lazy init rustfs codec reconstruct

* test(get): cover rustfs codec source faults

* docs(get): record rustfs codec fallback scope

* feat(get): add multipart codec reader opt-in

* test(get): add multipart codec smoke option

* test(get): cover multipart codec degraded fallback

* perf(get): bound multipart codec eager setup

* test(get): satisfy codec hardening PR gate

---------

Co-authored-by: heihutu <heihutu@gmail.com>
 2026-06-28 11:20:21 +08:00
Zhengchao An 6e72dc3076 fix(ecstore): replace unsafe k.unwrap() in bucket_target_sys (#729) (#3982) 
fix(ecstore): replace k.unwrap() with safe pattern in bucket_target_sys

Replace unsafe k.unwrap().as_str() with if let Some(key_str) pattern
in 5 locations where HeaderMap iterator yields (Option<HeaderName>, Value).

This prevents potential panics if header names are invalid.

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 10:42:56 +08:00
Zhengchao An 7238a937a9 fix: correct misleading zero-copy/direct-io docs and internal naming (#733 Phase 1) (#3976) 2026-06-28 09:12:49 +08:00
Zhengchao An e1272f2aba revert: restore #![allow(dead_code)] - CI clippy -D warnings conflict (#3979) 
revert: restore #![allow(dead_code)] - clippy -D warnings treats warn as error

The #742 PR changed #![allow(dead_code)] to #![warn(dead_code)], but
CI runs clippy with -D warnings which turns warnings into errors.
This caused CI failures across multiple PRs.

Reverting to #![allow(dead_code)] until the dead code is actually
cleaned up. The 189 warnings in ecstore should be fixed incrementally
by deleting dead code and adding item-level allows, not by changing
the crate-level policy.
 2026-06-28 08:32:34 +08:00
Henry Guo 1e303e5be0 fix(data-usage): refresh versioned usage state (#3969) 
fix(data-usage): refresh versioned usage from authoritative state

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-28 07:58:07 +08:00
Zhengchao An 113058af54 chore: replace blanket #![allow(dead_code)] with #![warn(dead_code)] (#742) (#3974) 2026-06-28 07:50:51 +08:00
houseme 27468ebfa9 feat(get): consolidate GET performance optimization (#3972) 
* feat(get): consolidate GET performance optimization

Consolidated implementation of all GET performance optimizations into
a single, well-organized commit replacing the previous patch-on-patch
approach.

## Changes

### Configuration (set_disk/mod.rs)
- Consolidated all GET optimization flags into a single organized section
- Enabled by default: codec streaming, metadata early-stop, page cache reclaim
- Added codec streaming multipart flag (default: disabled)
- Added version-aware early-stop flag (default: disabled)
- Added adaptive duplex buffer sizing based on object size
- All flags use OnceLock caching with rollout percentage support

### Metadata Early-Stop (set_disk/read.rs)
- Delete marker early-stop when quorum agrees
- Version-aware early-stop for versioned GET requests
- MetadataQuorumAccumulator enhanced with:
  - delete_marker_votes tracking
  - requested_version_id and matching_version_votes tracking
  - version_early_stop_decision() method
- 6 new tests for version early-stop scenarios

### Codec Streaming (erasure/coding/decode_reader.rs)
- DualInFlight (2-stripe lookahead) enabled by default

### Decode Pipeline (erasure/coding/decode.rs)
- Stripe prefetch count configuration
- Bitrot-decode overlap configuration

### Disk Layer (disk/local.rs)
- O_DIRECT read configuration constants (preparation)

### Metrics (io-metrics/lib.rs)
- BytesPool acquisition/return metrics
- Metadata phase duration with early-stop label
- Total duration with reader_path label

### Diagnostics (diagnostics/)
- Early-stop reason constants
- Pool tier/outcome label constants

### Observability (.docker/observability/)
- 3 Grafana dashboards for GET optimization monitoring
- Prometheus alert rules (6 alerts: 3 critical, 3 warning)
- Updated README.md and README_ZH.md with usage docs

### Config (config/src/constants/runtime.rs)
- Page cache reclaim read enabled by default

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag |
| RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % |
| RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming |
| RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag |
| RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % |
| RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop |
| RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim |
| RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) |
| RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch |
| RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap |
| RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes |

## Rollback

All optimizations can be disabled via environment variables:
RUSTFS_GET_CODEC_STREAMING_ENABLE=false
RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false
RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(get): add stress test scripts for GET optimization validation

- quick-validate-get-optimization.sh: Quick 5-minute validation
- stress-test-get-optimization.sh: Full 30+ minute stress test
- README-stress-test.md: Usage documentation

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): align file cache reclaim defaults

* chore(deps): update redis and erasure codec

* test(ecstore): align decode fill policy default

* test(ecstore): align metadata early-stop default

* fix(ecstore): keep metadata early stop opt-in

---------

Co-authored-by: heihutu <heihutu@gmail.com>
 2026-06-28 07:14:07 +08:00
Zhengchao An 512418cda9 fix(ecstore): replace unbounded metadata cache with moka (#743) (#3970) 
fix(ecstore): replace unbounded metadata cache with moka

Replace the manual Arc<RwLock<HashMap>> metadata cache with
moka::future::Cache, which provides:
- Built-in LRU eviction when max_capacity is reached
- Automatic TTL expiry via time_to_live (250ms)
- Lock-free concurrent reads
- Non-blocking invalidation

Fixes the memory leak risk from unbounded HashMap and the
all-or-nothing eviction logic that cleared all entries at once.

Closes #743

Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-28 03:01:32 +08:00
Zhengchao An 82bbef0b60 test: cover runtime and repair preservation (#3964) 2026-06-27 23:34:59 +08:00
houseme bf03ff2869 feat(get): add limited opt-in rollout gates (#3963) 
* feat(bench): harden cooled get ab harness

* feat(get): add limited opt-in rollout gates

* fix(get): tighten rollout gate fallbacks
 2026-06-27 23:13:02 +08:00
houseme 20f56af09c feat(get): tune output response handoff (#3956) 2026-06-27 21:46:38 +08:00
houseme de86025f2c feat(get): overlap read verify decode (#3945) 2026-06-27 15:14:44 +08:00
houseme 58b76a3d45 feat(get): add codec engine ab matrix (#3940) 
* upgrade version

* feat(get): add codec engine ab matrix

* chore(get): drop unrelated dependency drift

* upgrade version

* upgrade version

* fix cargo deny
 2026-06-27 13:27:16 +08:00
Zhengchao An 1b3dea012e refactor: route ecstore runtime globals through facade (#3941) 2026-06-27 12:27:03 +08:00
Zhengchao An e1a4b9e0b6 refactor: batch cluster lock and health readiness (#3936) 2026-06-27 10:51:06 +08:00
GatewayJ 675597ec16 fix(ecstore): handle stalled recovery reads and listings (#3790) 
* fix(ecstore): handle stalled recovery reads and listings

* fix(rio): start HTTP stall timeout on read

* fix(ecstore): handle stalled reads and partial lists

* fix(ecstore): retire stalled shards and list errors

* fix(ecstore): preserve list merge lookahead entries

* fix(ecstore): bound zero-copy shard reads

* fix(ecstore): hedge stalled shard reads

* fix(ecstore): retire abandoned shard reads

* fix(ecstore): include part identity in metadata quorum

* fix(ecstore): validate heal shard sources

* fix(ecstore): verify reconstructed read shards

* chore(ecstore): log slow object read stages

* fix(heal): throttle auto heal during recovery

* fix(scanner): yield to foreground reads

* fix(scanner): track streaming object reads

* fix(ecstore): avoid false read heal fanout

* fix(ecstore): verify codec streaming reconstruction sources

* fix(ecstore): preserve quorum progress on slow shards

* fix(storage): restore read timeout facade

* fix(ecstore): retain fallback readers after quorum

* chore: allow decode helper argument lists

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
 2026-06-27 10:21:09 +08:00
Zhengchao An 3fb4dcd52e refactor: route cluster control plane readiness (#3935) 2026-06-27 09:47:30 +08:00
Zhengchao An 0a5b1b1b3a refactor: consolidate ecstore owner module layout (#3934) 
* refactor: shrink ecstore root owner facades

* refactor: remove ecstore core store root shims

* refactor: move ecstore erasure owner modules

* refactor: remove ecstore root rpc facade

* refactor: move ecstore services domain modules
 2026-06-27 09:03:20 +08:00
cxymds 080363f10f feat: harden site replication control plane (#3842) 2026-06-27 08:35:49 +08:00
houseme ae4aa4f5d4 feat(get): add rustfs codec decode engine (#3928) 2026-06-27 08:35:38 +08:00
cxymds 688b14b572 fix(ecstore): preserve multipart part quorum errors (#3920) 
* fix(ecstore): preserve multipart part quorum errors

* fix(ecstore): honor confirmed missing part quorum

* fix(ecstore): add multipart quorum diagnostics
 2026-06-27 08:33:43 +08:00
Zhengchao An 27bb9c75dc refactor: move ecstore rpc metadata modules (#3933) 2026-06-27 07:19:45 +08:00
Zhengchao An c6ecfae39e refactor: move ecstore owner layout modules (#3932) 2026-06-27 05:54:25 +08:00
Zhengchao An 61b1296972 refactor: move ecstore store init support (#3931) 2026-06-27 05:05:01 +08:00
Zhengchao An bdfcde1866 refactor: move ecstore store support modules (#3930) 2026-06-27 03:51:11 +08:00
Zhengchao An 07428c8a34 refactor: move ecstore store owner roots (#3929) 2026-06-27 02:25:04 +08:00
houseme 8994a8ff00 feat(get): add local-first shard reads (#3924) 
* feat(get): observe shard locality cost

* feat(get): add local-first shard reads

----

Co-Authored-By: Heihutu <heihutu@gmail.com>
 2026-06-27 01:08:36 +08:00
houseme c0ddc14bb8 feat(get): observe shard locality cost (#3922) 
* feat(get): observe shard locality cost

* fix(storage): avoid request counter underflow panic

* fix(storage): import put guard in concurrency tests
 2026-06-26 23:50:37 +08:00
houseme 2c04807f05 feat(get): add guarded metadata early stop (#3916) 
* feat(get): add v2 metrics compatibility harness

* feat(get): observe metadata fanout quorum (#3915)

* feat(get): observe metadata fanout quorum

* fix(app): use storage ECStore type in app boundary

* feat(get): add guarded metadata early stop
 2026-06-26 21:56:23 +08:00
cxymds 7820a55fdc fix: decouple readiness from cluster health (#3912) 
* fix: include foreground write pressure

* fix: split readiness and cluster health probes

* fix: publish ready on node readiness

* fix: bound cluster health collection

* test: pin health probe collector routing

* fix: qualify app storage ECStore type

* test(admin): narrow runtime capabilities topology assertion

---------

Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-26 21:55:18 +08:00
houseme 5a78a9c416 feat(get): add v2 metrics compatibility harness (#3913) 
* feat(get): add v2 metrics compatibility harness

* feat(get): observe metadata fanout quorum (#3915)

* feat(get): observe metadata fanout quorum

* fix(app): use storage ECStore type in app boundary

----
Co-Authored-By: heihutu <heihutu@gmail.com>
 2026-06-26 20:09:09 +08:00
Henry Guo a8327f8901 fix(scanner): account versioned delete markers in usage (#3904) 
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
 2026-06-26 18:34:21 +08:00
houseme 0321e4c6ca perf(get): reduce metrics and streaming handoff overhead (#3907) 2026-06-26 18:12:51 +08:00
cxymds 30957e2b51 fix(ecstore): throttle data movement under read pressure (#3906) 2026-06-26 17:48:09 +08:00
Zhengchao An b38976d5ee refactor: segment ECStore storage contracts by domain (#3910) 2026-06-26 17:47:36 +08:00
Zhengchao An 72ae43cb90 refactor: segment external storage contract imports (#3908) 2026-06-26 16:44:59 +08:00
Zhengchao An 6efbfff2c5 refactor: segment residual storage api boundaries (#3905) 2026-06-26 15:56:38 +08:00
cxymds 90638bfc19 fix(heal): add backpressure to repair admission (#3900) 2026-06-26 15:19:37 +08:00
Zhengchao An c9614eb7cb refactor: route ecstore storage api boundaries (#3892) 2026-06-26 09:35:18 +08:00
cxymds 0b9c8a7731 fix(heal): align scanner repair safety with quorum errors (#3887) 
* fix(heal): avoid task-level delete on repair failure

* fix(scanner): avoid recreate for scanner heal checks

* fix(heal): preserve typed quorum failures

* test(heal): satisfy clippy bool assertion
 2026-06-26 08:58:27 +08:00
houseme 7bf411f465 obs(get): add GET read pipeline metrics (#3868) 2026-06-26 04:02:02 +08:00
cxymds 86e9362459 fix(heal): canonicalize scanner object-dir repairs (#3864) 2026-06-25 22:07:38 +08:00
Henry Guo 30f781871d fix(scanner): preserve dirty usage convergence (#3844) 
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
 2026-06-25 21:09:12 +08:00