gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-26 16:28:15 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,777 Commits 21 Branches 117 Tags
e8470bbc6690af7b773d830a5f019a23647cebd8
Commit Graph

450 Commits

Author SHA1 Message Date
houseme 46d7f9e1f2 feat(get): harden codec streaming rollout (#3981) 
* feat(get): consolidate GET performance optimization

Consolidated implementation of all GET performance optimizations into
a single, well-organized commit replacing the previous patch-on-patch
approach.

## Changes

### Configuration (set_disk/mod.rs)
- Consolidated all GET optimization flags into a single organized section
- Enabled by default: codec streaming, metadata early-stop, page cache reclaim
- Added codec streaming multipart flag (default: disabled)
- Added version-aware early-stop flag (default: disabled)
- Added adaptive duplex buffer sizing based on object size
- All flags use OnceLock caching with rollout percentage support

### Metadata Early-Stop (set_disk/read.rs)
- Delete marker early-stop when quorum agrees
- Version-aware early-stop for versioned GET requests
- MetadataQuorumAccumulator enhanced with:
  - delete_marker_votes tracking
  - requested_version_id and matching_version_votes tracking
  - version_early_stop_decision() method
- 6 new tests for version early-stop scenarios

### Codec Streaming (erasure/coding/decode_reader.rs)
- DualInFlight (2-stripe lookahead) enabled by default

### Decode Pipeline (erasure/coding/decode.rs)
- Stripe prefetch count configuration
- Bitrot-decode overlap configuration

### Disk Layer (disk/local.rs)
- O_DIRECT read configuration constants (preparation)

### Metrics (io-metrics/lib.rs)
- BytesPool acquisition/return metrics
- Metadata phase duration with early-stop label
- Total duration with reader_path label

### Diagnostics (diagnostics/)
- Early-stop reason constants
- Pool tier/outcome label constants

### Observability (.docker/observability/)
- 3 Grafana dashboards for GET optimization monitoring
- Prometheus alert rules (6 alerts: 3 critical, 3 warning)
- Updated README.md and README_ZH.md with usage docs

### Config (config/src/constants/runtime.rs)
- Page cache reclaim read enabled by default

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag |
| RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % |
| RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming |
| RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag |
| RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % |
| RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop |
| RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim |
| RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) |
| RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch |
| RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap |
| RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes |

## Rollback

All optimizations can be disabled via environment variables:
RUSTFS_GET_CODEC_STREAMING_ENABLE=false
RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false
RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(get): add stress test scripts for GET optimization validation

- quick-validate-get-optimization.sh: Quick 5-minute validation
- stress-test-get-optimization.sh: Full 30+ minute stress test
- README-stress-test.md: Usage documentation

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): align file cache reclaim defaults

* chore(deps): update redis and erasure codec

* test(ecstore): align decode fill policy default

* fix(get): wire codec streaming rollout gate

* perf(get): skip metrics-off codec timers

* test(get): capture codec streaming diagnostics

* test(get): add multipart fallback probe

* test(get): add encrypted fallback probe

* test(get): add compressed fallback probe

* test(get): add degraded read fallback probe

* test(get): cover remote fallback probe

* test(get): report warp request p99

* test(get): capture OTLP metric deltas

* perf(get): align codec streaming inflight default

* perf(get): reuse codec reader output buffers

* test(get): count codec reader fill starts

* perf(get): reuse codec reader fill worker

* perf(get): lazy init rustfs codec reconstruct

* test(get): cover rustfs codec source faults

* docs(get): record rustfs codec fallback scope

* feat(get): add multipart codec reader opt-in

* test(get): add multipart codec smoke option

* test(get): cover multipart codec degraded fallback

* perf(get): bound multipart codec eager setup

* test(get): satisfy codec hardening PR gate

---------

Co-authored-by: heihutu <heihutu@gmail.com>
 2026-06-28 11:20:21 +08:00
houseme 27468ebfa9 feat(get): consolidate GET performance optimization (#3972) 
* feat(get): consolidate GET performance optimization

Consolidated implementation of all GET performance optimizations into
a single, well-organized commit replacing the previous patch-on-patch
approach.

## Changes

### Configuration (set_disk/mod.rs)
- Consolidated all GET optimization flags into a single organized section
- Enabled by default: codec streaming, metadata early-stop, page cache reclaim
- Added codec streaming multipart flag (default: disabled)
- Added version-aware early-stop flag (default: disabled)
- Added adaptive duplex buffer sizing based on object size
- All flags use OnceLock caching with rollout percentage support

### Metadata Early-Stop (set_disk/read.rs)
- Delete marker early-stop when quorum agrees
- Version-aware early-stop for versioned GET requests
- MetadataQuorumAccumulator enhanced with:
  - delete_marker_votes tracking
  - requested_version_id and matching_version_votes tracking
  - version_early_stop_decision() method
- 6 new tests for version early-stop scenarios

### Codec Streaming (erasure/coding/decode_reader.rs)
- DualInFlight (2-stripe lookahead) enabled by default

### Decode Pipeline (erasure/coding/decode.rs)
- Stripe prefetch count configuration
- Bitrot-decode overlap configuration

### Disk Layer (disk/local.rs)
- O_DIRECT read configuration constants (preparation)

### Metrics (io-metrics/lib.rs)
- BytesPool acquisition/return metrics
- Metadata phase duration with early-stop label
- Total duration with reader_path label

### Diagnostics (diagnostics/)
- Early-stop reason constants
- Pool tier/outcome label constants

### Observability (.docker/observability/)
- 3 Grafana dashboards for GET optimization monitoring
- Prometheus alert rules (6 alerts: 3 critical, 3 warning)
- Updated README.md and README_ZH.md with usage docs

### Config (config/src/constants/runtime.rs)
- Page cache reclaim read enabled by default

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag |
| RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % |
| RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming |
| RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag |
| RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % |
| RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop |
| RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim |
| RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) |
| RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch |
| RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap |
| RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes |

## Rollback

All optimizations can be disabled via environment variables:
RUSTFS_GET_CODEC_STREAMING_ENABLE=false
RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false
RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(get): add stress test scripts for GET optimization validation

- quick-validate-get-optimization.sh: Quick 5-minute validation
- stress-test-get-optimization.sh: Full 30+ minute stress test
- README-stress-test.md: Usage documentation

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): align file cache reclaim defaults

* chore(deps): update redis and erasure codec

* test(ecstore): align decode fill policy default

* test(ecstore): align metadata early-stop default

* fix(ecstore): keep metadata early stop opt-in

---------

Co-authored-by: heihutu <heihutu@gmail.com>
 2026-06-28 07:14:07 +08:00
houseme 175566f037 docs(get): record default switch readiness (#3967) 
* docs(get): record default switch readiness

* chore(docs): keep pr35 evaluation note local

* refactor(bench): trim pr35 readiness python
 2026-06-28 01:12:44 +08:00
houseme bf03ff2869 feat(get): add limited opt-in rollout gates (#3963) 
* feat(bench): harden cooled get ab harness

* feat(get): add limited opt-in rollout gates

* fix(get): tighten rollout gate fallbacks
 2026-06-27 23:13:02 +08:00
houseme 47d05a1d6a feat(bench): harden cooled get ab harness (#3960) 2026-06-27 22:23:09 +08:00
houseme 20f56af09c feat(get): tune output response handoff (#3956) 2026-06-27 21:46:38 +08:00
houseme de86025f2c feat(get): overlap read verify decode (#3945) 2026-06-27 15:14:44 +08:00
houseme 58b76a3d45 feat(get): add codec engine ab matrix (#3940) 
* upgrade version

* feat(get): add codec engine ab matrix

* chore(get): drop unrelated dependency drift

* upgrade version

* upgrade version

* fix cargo deny
 2026-06-27 13:27:16 +08:00
Zhengchao An 1b3dea012e refactor: route ecstore runtime globals through facade (#3941) 2026-06-27 12:27:03 +08:00
Zhengchao An d3f1ff36af docs: record global state cleanup plan (#3939) 2026-06-27 12:04:59 +08:00
Zhengchao An fe9c0daf0f docs: add architecture readiness matrix (#3938) 2026-06-27 11:23:07 +08:00
Zhengchao An 3fb4dcd52e refactor: route cluster control plane readiness (#3935) 2026-06-27 09:47:30 +08:00
Zhengchao An 0a5b1b1b3a refactor: consolidate ecstore owner module layout (#3934) 
* refactor: shrink ecstore root owner facades

* refactor: remove ecstore core store root shims

* refactor: move ecstore erasure owner modules

* refactor: remove ecstore root rpc facade

* refactor: move ecstore services domain modules
 2026-06-27 09:03:20 +08:00
Zhengchao An 27bb9c75dc refactor: move ecstore rpc metadata modules (#3933) 2026-06-27 07:19:45 +08:00
Zhengchao An c6ecfae39e refactor: move ecstore owner layout modules (#3932) 2026-06-27 05:54:25 +08:00
Zhengchao An 61b1296972 refactor: move ecstore store init support (#3931) 2026-06-27 05:05:01 +08:00
Zhengchao An bdfcde1866 refactor: move ecstore store support modules (#3930) 2026-06-27 03:51:11 +08:00
Zhengchao An 07428c8a34 refactor: move ecstore store owner roots (#3929) 2026-06-27 02:25:04 +08:00
Zhengchao An e5706ead0e ci: close config model ownership guard (#3927) 2026-06-27 00:25:36 +08:00
Zhengchao An 741b4dab7f refactor: close external storage API boundary guard (#3925) 2026-06-26 23:20:25 +08:00
Zhengchao An a22384708e refactor: decouple server health helpers (#3923) 2026-06-26 22:40:22 +08:00
Zhengchao An 15c39db42f refactor: consolidate storage owner boundary cleanup (#3921) 
* refactor: segment app usecase contracts by domain

* refactor: segment root storage contracts by domain

* refactor: segment root runtime facades by domain

* refactor: segment storage owner consumers by domain

* refactor: route storage owner runtime consumers by domain

* refactor: make storage owner rpc imports explicit

* refactor: remove storage owner wildcard imports (#3917)

* refactor: make storage owner root exports explicit (#3918)

* refactor: route storage facades through owner api (#3919)

* refactor: finalize storage owner phase branch
 2026-06-26 22:18:00 +08:00
Zhengchao An 6469d6ada8 refactor: segment admin storage contracts by domain (#3914) 2026-06-26 20:16:51 +08:00
houseme 5a78a9c416 feat(get): add v2 metrics compatibility harness (#3913) 
* feat(get): add v2 metrics compatibility harness

* feat(get): observe metadata fanout quorum (#3915)

* feat(get): observe metadata fanout quorum

* fix(app): use storage ECStore type in app boundary

----
Co-Authored-By: heihutu <heihutu@gmail.com>
 2026-06-26 20:09:09 +08:00
Zhengchao An a010dce93c refactor: segment storage owner contracts by domain (#3911) 2026-06-26 18:34:32 +08:00
houseme 0321e4c6ca perf(get): reduce metrics and streaming handoff overhead (#3907) 2026-06-26 18:12:51 +08:00
Zhengchao An b38976d5ee refactor: segment ECStore storage contracts by domain (#3910) 2026-06-26 17:47:36 +08:00
Zhengchao An 72a589f0c0 refactor: segment RustFS storage API root exports (#3909) 2026-06-26 17:15:33 +08:00
Zhengchao An 72ae43cb90 refactor: segment external storage contract imports (#3908) 2026-06-26 16:44:59 +08:00
Zhengchao An 6efbfff2c5 refactor: segment residual storage api boundaries (#3905) 2026-06-26 15:56:38 +08:00
Zhengchao An 1f7e159388 refactor: segment external storage api boundaries (#3903) 2026-06-26 15:10:49 +08:00
Zhengchao An 738b805ec0 refactor: segment app storage api boundary (#3902) 2026-06-26 14:45:05 +08:00
Zhengchao An 2a1bddfcca refactor: segment storage api domain boundaries (#3901) 2026-06-26 14:04:28 +08:00
Zhengchao An 92c50156a3 refactor: centralize runtime source boundaries (#3899) 2026-06-26 13:22:00 +08:00
Zhengchao An c9614eb7cb refactor: route ecstore storage api boundaries (#3892) 2026-06-26 09:35:18 +08:00
Zhengchao An 97dce107f0 refactor: route storage owner contract imports (#3891) 2026-06-26 08:58:07 +08:00
Zhengchao An 2aca607119 refactor: move storage owner boundary aggregation (#3890) 2026-06-26 07:52:49 +08:00
Zhengchao An e37e367390 refactor: route remaining external storage boundaries (#3889) 2026-06-26 06:22:51 +08:00
Zhengchao An 4d6ea453a7 refactor: route scanner heal storage boundaries (#3888) 2026-06-26 05:24:41 +08:00
Zhengchao An 19c925c480 refactor: expand storage api boundaries (#3886) 
* refactor: route root storage contracts

* refactor: expand storage api boundaries
 2026-06-26 04:19:02 +08:00
houseme 7bf411f465 obs(get): add GET read pipeline metrics (#3868) 2026-06-26 04:02:02 +08:00
Zhengchao An 33b7f48f37 refactor: route root storage facades (#3867) 2026-06-25 23:23:45 +08:00
Zhengchao An 987ea4919f refactor: route admin storage facades (#3866) 2026-06-25 22:37:54 +08:00
Zhengchao An 1c973bc1d9 refactor: route app facade boundaries (#3865) 
* refactor: route app storage error helpers

* refactor: route app bucket owner facades

* refactor: route app runtime facades
 2026-06-25 21:57:42 +08:00
Zhengchao An 60c00f7622 refactor: route app and admin storage helpers (#3863) 
* refactor: route admin storage helper imports

* refactor: route app storage io helper imports
 2026-06-25 21:12:51 +08:00
houseme ef0dcec479 feat(ecstore): add deeper zero-copy ingest experiment (#3847) 
* feat(storage): add multipart put stage metrics

* feat(scripts): add multipart put focus runner

* docs(operations): add multipart put server-path guides

* chore(scripts): add local rustfs restart helper

* docs(observability): add local metrics backend guide

* docs(observability): add localized multipart guides

* fix(ecstore): validate multipart batching path

* feat(obs): add erasure encode overlap metrics

* docs(ops): update overlap retest summary

* docs(ops): add batchblocks retest matrix

* docs(ops): extend overlap candidate summary

* docs(ops): capture 8-run overlap summary

* feat(storage): switch rename_data to msgpack map

* test(storage): add rename_data payload checks

* feat(object): add zero_copy_eager put path

* docs(ops): add zero_copy_eager put guide

* docs(ops): add deeper zero-copy next steps

* feat(ecstore): add bytesmut erasure ingest gate

* docs(ops): add bytesmut ingest summary

* docs(ops): extend bytesmut ingest matrix summary

* docs(ops): extend bytesmut larger-object summary

* docs(ops): capture bytesmut variability summary

* chore(scripts): add deeper zero-copy capture flow

* chore(scripts): add deeper zero-copy capture support

* docs(ops): add capture-backed bytesmut retest

* docs(ops): update deeper zero-copy retests

* chore(docs): keep issue-712 notes local only

Co-Authored-By: heihutu <heihutu@gmail.com>
 2026-06-25 21:05:00 +08:00
houseme 96b1f5c373 feat(storage): optimize gt1g get read path (#3860) 
* feat(storage): tune gt1g get read path

* docs(ops): add gt1g get benchmark guide

* feat(scripts): add gt1g get fallback runner

* test(storage): trim gt1g get helper warnings

* test(storage): reduce gt1g get helper type complexity
 2026-06-25 20:12:08 +08:00
houseme a30357d21e feat(storage): extend PUT path tuning and observability (#3829) 
* feat(storage): add multipart put stage metrics

* feat(scripts): add multipart put focus runner

* docs(operations): add multipart put server-path guides

* chore(scripts): add local rustfs restart helper

* docs(observability): add local metrics backend guide

* docs(observability): add localized multipart guides

* fix(ecstore): validate multipart batching path

* feat(obs): add erasure encode overlap metrics

* docs(ops): update overlap retest summary

* docs(ops): add batchblocks retest matrix

* docs(ops): extend overlap candidate summary

* docs(ops): capture 8-run overlap summary

* feat(storage): switch rename_data to msgpack map

* test(storage): add rename_data payload checks

* feat(object): add zero_copy_eager put path

* docs(ops): add zero_copy_eager put guide

* docs(ops): add deeper zero-copy next steps
 2026-06-25 19:24:35 +08:00
Zhengchao An 3942186f3d refactor: route app storage helper imports (#3856) 2026-06-25 18:10:49 +08:00
Zhengchao An a21f3c1a3f refactor: route app s3 api helpers (#3854) 2026-06-25 16:50:17 +08:00