gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-26 16:28:15 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,768 Commits 21 Branches 117 Tags
1e303e5be0519d783ee5099fe44deaa288b7dfbc
Commit Graph

3768 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Henry Guo 1e303e5be0 fix(data-usage): refresh versioned usage state (#3969) 
fix(data-usage): refresh versioned usage from authoritative state

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-28 07:58:07 +08:00
Zhengchao An 129ad2296b ci: add protocol feature test matrix (swift, sftp, no-default) (#736) (#3975) 2026-06-28 07:51:15 +08:00
Zhengchao An 113058af54 chore: replace blanket #![allow(dead_code)] with #![warn(dead_code)] (#742) (#3974) 2026-06-28 07:50:51 +08:00
houseme 27468ebfa9 feat(get): consolidate GET performance optimization (#3972) 
* feat(get): consolidate GET performance optimization

Consolidated implementation of all GET performance optimizations into
a single, well-organized commit replacing the previous patch-on-patch
approach.

## Changes

### Configuration (set_disk/mod.rs)
- Consolidated all GET optimization flags into a single organized section
- Enabled by default: codec streaming, metadata early-stop, page cache reclaim
- Added codec streaming multipart flag (default: disabled)
- Added version-aware early-stop flag (default: disabled)
- Added adaptive duplex buffer sizing based on object size
- All flags use OnceLock caching with rollout percentage support

### Metadata Early-Stop (set_disk/read.rs)
- Delete marker early-stop when quorum agrees
- Version-aware early-stop for versioned GET requests
- MetadataQuorumAccumulator enhanced with:
  - delete_marker_votes tracking
  - requested_version_id and matching_version_votes tracking
  - version_early_stop_decision() method
- 6 new tests for version early-stop scenarios

### Codec Streaming (erasure/coding/decode_reader.rs)
- DualInFlight (2-stripe lookahead) enabled by default

### Decode Pipeline (erasure/coding/decode.rs)
- Stripe prefetch count configuration
- Bitrot-decode overlap configuration

### Disk Layer (disk/local.rs)
- O_DIRECT read configuration constants (preparation)

### Metrics (io-metrics/lib.rs)
- BytesPool acquisition/return metrics
- Metadata phase duration with early-stop label
- Total duration with reader_path label

### Diagnostics (diagnostics/)
- Early-stop reason constants
- Pool tier/outcome label constants

### Observability (.docker/observability/)
- 3 Grafana dashboards for GET optimization monitoring
- Prometheus alert rules (6 alerts: 3 critical, 3 warning)
- Updated README.md and README_ZH.md with usage docs

### Config (config/src/constants/runtime.rs)
- Page cache reclaim read enabled by default

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag |
| RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % |
| RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming |
| RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag |
| RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % |
| RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop |
| RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim |
| RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) |
| RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch |
| RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap |
| RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes |

## Rollback

All optimizations can be disabled via environment variables:
RUSTFS_GET_CODEC_STREAMING_ENABLE=false
RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false
RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(get): add stress test scripts for GET optimization validation

- quick-validate-get-optimization.sh: Quick 5-minute validation
- stress-test-get-optimization.sh: Full 30+ minute stress test
- README-stress-test.md: Usage documentation

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): align file cache reclaim defaults

* chore(deps): update redis and erasure codec

* test(ecstore): align decode fill policy default

* test(ecstore): align metadata early-stop default

* fix(ecstore): keep metadata early stop opt-in

---------

Co-authored-by: heihutu <heihutu@gmail.com>
 2026-06-28 07:14:07 +08:00
Zhengchao An 512418cda9 fix(ecstore): replace unbounded metadata cache with moka (#743) (#3970) 
fix(ecstore): replace unbounded metadata cache with moka

Replace the manual Arc<RwLock<HashMap>> metadata cache with
moka::future::Cache, which provides:
- Built-in LRU eviction when max_capacity is reached
- Automatic TTL expiry via time_to_live (250ms)
- Lock-free concurrent reads
- Non-blocking invalidation

Fixes the memory leak risk from unbounded HashMap and the
all-or-nothing eviction logic that cleared all entries at once.

Closes #743

Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-28 03:01:32 +08:00
houseme 175566f037 docs(get): record default switch readiness (#3967) 
* docs(get): record default switch readiness

* chore(docs): keep pr35 evaluation note local

* refactor(bench): trim pr35 readiness python
 2026-06-28 01:12:44 +08:00
Zhengchao An 7ae9697d42 docs: close final architecture audit gaps (#3966) 2026-06-28 00:07:42 +08:00
Zhengchao An 79234c030d docs: close architecture migration validation (#3965) 2026-06-27 23:42:51 +08:00
Zhengchao An 82bbef0b60 test: cover runtime and repair preservation (#3964) 2026-06-27 23:34:59 +08:00
houseme bf03ff2869 feat(get): add limited opt-in rollout gates (#3963) 
* feat(bench): harden cooled get ab harness

* feat(get): add limited opt-in rollout gates

* fix(get): tighten rollout gate fallbacks
 2026-06-27 23:13:02 +08:00
Zhengchao An 66ad138505 docs: close phase 7 global split plan (#3962) 2026-06-27 22:25:16 +08:00
houseme 47d05a1d6a feat(bench): harden cooled get ab harness (#3960) 2026-06-27 22:23:09 +08:00
Zhengchao An c7dc3d7974 refactor: move runtime fallback defaults to owners (#3961) 2026-06-27 22:21:09 +08:00
houseme 20f56af09c feat(get): tune output response handoff (#3956) 2026-06-27 21:46:38 +08:00
Zhengchao An 68d5d1d41d refactor: isolate specialized runtime fallbacks (#3958) 2026-06-27 21:35:47 +08:00
Zhengchao An 672f6e9ea9 refactor: remove scalar runtime fallbacks (#3957) 2026-06-27 20:45:13 +08:00
Zhengchao An 243a20b14e refactor: remove service resolver fallbacks (#3955) 2026-06-27 19:35:43 +08:00
Zhengchao An 4732f4ee2d refactor: expose runtime fallback boundaries (#3954) 2026-06-27 18:57:51 +08:00
Zhengchao An cdebdeca61 refactor: clean app context fallback signatures (#3953) 2026-06-27 18:30:06 +08:00
Zhengchao An d66391636b refactor: remove optional runtime handle fallbacks (#3952) 2026-06-27 18:07:09 +08:00
Zhengchao An 430aa8e8cb refactor: remove notification config fallbacks (#3951) 2026-06-27 17:43:05 +08:00
Zhengchao An ab9e98a9ca refactor: align core runtime facade helpers (#3948) 2026-06-27 17:14:25 +08:00
houseme de86025f2c feat(get): overlap read verify decode (#3945) 2026-06-27 15:14:44 +08:00
Zhengchao An dff0e467f4 refactor: align app runtime facade helpers (#3947) 2026-06-27 15:14:29 +08:00
Zhengchao An 7979ecb545 refactor: align storage runtime facade helpers (#3946) 2026-06-27 14:55:37 +08:00
Zhengchao An 261e16a446 refactor: route admin runtime sources through current helpers (#3944) 2026-06-27 14:19:18 +08:00
Zhengchao An cc41a50efe refactor: route admin runtime consumers through app context (#3943) 2026-06-27 13:38:21 +08:00
houseme 58b76a3d45 feat(get): add codec engine ab matrix (#3940) 
* upgrade version

* feat(get): add codec engine ab matrix

* chore(get): drop unrelated dependency drift

* upgrade version

* upgrade version

* fix cargo deny
 2026-06-27 13:27:16 +08:00
Zhengchao An 996e58fd9a refactor: route admin runtime config through app context (#3942) 2026-06-27 13:10:00 +08:00
Zhengchao An 1b3dea012e refactor: route ecstore runtime globals through facade (#3941) 2026-06-27 12:27:03 +08:00
Zhengchao An d3f1ff36af docs: record global state cleanup plan (#3939) 2026-06-27 12:04:59 +08:00
Zhengchao An fe9c0daf0f docs: add architecture readiness matrix (#3938) 2026-06-27 11:23:07 +08:00
Zhengchao An e1a4b9e0b6 refactor: batch cluster lock and health readiness (#3936) 2026-06-27 10:51:06 +08:00
GatewayJ 675597ec16 fix(ecstore): handle stalled recovery reads and listings (#3790) 
* fix(ecstore): handle stalled recovery reads and listings

* fix(rio): start HTTP stall timeout on read

* fix(ecstore): handle stalled reads and partial lists

* fix(ecstore): retire stalled shards and list errors

* fix(ecstore): preserve list merge lookahead entries

* fix(ecstore): bound zero-copy shard reads

* fix(ecstore): hedge stalled shard reads

* fix(ecstore): retire abandoned shard reads

* fix(ecstore): include part identity in metadata quorum

* fix(ecstore): validate heal shard sources

* fix(ecstore): verify reconstructed read shards

* chore(ecstore): log slow object read stages

* fix(heal): throttle auto heal during recovery

* fix(scanner): yield to foreground reads

* fix(scanner): track streaming object reads

* fix(ecstore): avoid false read heal fanout

* fix(ecstore): verify codec streaming reconstruction sources

* fix(ecstore): preserve quorum progress on slow shards

* fix(storage): restore read timeout facade

* fix(ecstore): retain fallback readers after quorum

* chore: allow decode helper argument lists

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
 2026-06-27 10:21:09 +08:00
Zhengchao An 3fb4dcd52e refactor: route cluster control plane readiness (#3935) 2026-06-27 09:47:30 +08:00
Zhengchao An 0a5b1b1b3a refactor: consolidate ecstore owner module layout (#3934) 
* refactor: shrink ecstore root owner facades

* refactor: remove ecstore core store root shims

* refactor: move ecstore erasure owner modules

* refactor: remove ecstore root rpc facade

* refactor: move ecstore services domain modules
 2026-06-27 09:03:20 +08:00
cxymds 080363f10f feat: harden site replication control plane (#3842) 2026-06-27 08:35:49 +08:00
houseme ae4aa4f5d4 feat(get): add rustfs codec decode engine (#3928) 2026-06-27 08:35:38 +08:00
cxymds 688b14b572 fix(ecstore): preserve multipart part quorum errors (#3920) 
* fix(ecstore): preserve multipart part quorum errors

* fix(ecstore): honor confirmed missing part quorum

* fix(ecstore): add multipart quorum diagnostics
 2026-06-27 08:33:43 +08:00
Zhengchao An 27bb9c75dc refactor: move ecstore rpc metadata modules (#3933) 2026-06-27 07:19:45 +08:00
Zhengchao An c6ecfae39e refactor: move ecstore owner layout modules (#3932) 2026-06-27 05:54:25 +08:00
Zhengchao An 61b1296972 refactor: move ecstore store init support (#3931) 2026-06-27 05:05:01 +08:00
Zhengchao An bdfcde1866 refactor: move ecstore store support modules (#3930) 2026-06-27 03:51:11 +08:00
Zhengchao An 07428c8a34 refactor: move ecstore store owner roots (#3929) 2026-06-27 02:25:04 +08:00
houseme 8994a8ff00 feat(get): add local-first shard reads (#3924) 
* feat(get): observe shard locality cost

* feat(get): add local-first shard reads

----

Co-Authored-By: Heihutu <heihutu@gmail.com>
 2026-06-27 01:08:36 +08:00
Zhengchao An e5706ead0e ci: close config model ownership guard (#3927) 2026-06-27 00:25:36 +08:00
Zhengchao An 840d21d201 fix(deps): remove vulnerable thrift dependency (#3926) 2026-06-27 00:16:51 +08:00
houseme c0ddc14bb8 feat(get): observe shard locality cost (#3922) 
* feat(get): observe shard locality cost

* fix(storage): avoid request counter underflow panic

* fix(storage): import put guard in concurrency tests
 2026-06-26 23:50:37 +08:00
Zhengchao An 741b4dab7f refactor: close external storage API boundary guard (#3925) 2026-06-26 23:20:25 +08:00
Zhengchao An a22384708e refactor: decouple server health helpers (#3923) 2026-06-26 22:40:22 +08:00