gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-26 16:28:15 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,792 Commits 21 Branches 117 Tags
710ae74cde795bd57c5e301a1de0dd7441ecb6d7
Commit Graph

3792 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Zhengchao An 710ae74cde perf: add S3 operations benchmark framework (#738) (#4005) 2026-06-28 18:02:41 +08:00
Zhengchao An 51acf2a99c feat: add rate limiting middleware framework (#737) (#4002) 
* feat: add rate limiting middleware framework

Add rate limiting middleware with token bucket algorithm for
per-client request rate limiting. This provides the foundation
for DoS protection.

Refs #737

* fix: address clippy lints in rate_limit.rs

- Collapse nested if statements into single if-let chains
- Use .is_multiple_of() instead of manual modulo check
 2026-06-28 17:48:40 +08:00
Zhengchao An ee82d6c026 test: add insta snapshot test for storage error display format (#740) (#4001) 
test: add insta snapshot test for storage error display format

Add snapshot test to detect unexpected changes in StorageError
display format. This catches output format regressions that
traditional assert tests might miss.

Refs #740
 2026-06-28 16:10:07 +08:00
Zhengchao An 84cdf12083 test(security): add security boundary tests (#748) (#3998) 
test(security): add security boundary tests

Add e2e tests for security-sensitive scenarios:
- Large XML body handling (DoS protection)
- Excessive multipart parts (DoS protection)
- Concurrent object operations (race condition handling)
- Internal URL validation (SSRF prevention)

Refs #748
 2026-06-28 15:17:30 +08:00
Zhengchao An c768a9c382 docs(storage-api): document filemeta dependency as known limitation (#731) (#3997) 
* docs(storage-api): document filemeta dependency as known limitation

Add comment explaining why storage-api depends on filemeta and
the scope of work required to break this dependency (300+ files).

Refs https://github.com/rustfs/backlog/issues/731

* docs(storage-api): remove backlog link from comment
 2026-06-28 14:59:24 +08:00
Zhengchao An c475d03b48 fix: replace unwrap() with expect() in more files (#729 batch 14) (#3994) 2026-06-28 11:45:23 +08:00
Zhengchao An 05d201679c fix: replace unwrap() with expect() in more files (#729 batch 13) (#3993) 2026-06-28 11:45:13 +08:00
Zhengchao An f0ab812213 fix: replace unwrap() with expect() in remaining files (#729 batch 12) (#3992) 2026-06-28 11:45:03 +08:00
Zhengchao An 1f8a5bc095 fix(ecstore): replace unwrap() with proper error handling in api_get_object_attributes (#729 batch 11) (#3991) 
fix(ecstore): replace unwrap() with proper error handling in api_get_object_attributes

Replace unsafe unwrap() calls with proper error handling in
api_get_object_attributes.rs:
- HTTP header access now uses ok_or_else with descriptive messages
- String parsing now uses map_err with descriptive messages
- HeaderValue creation now uses expect with descriptive messages

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 11:23:41 +08:00
Zhengchao An 25170943ce fix(ecstore): improve expect() messages in admin_server_info (#729 batch 9) (#3990) 
fix(ecstore): improve expect() messages in admin_server_info

Replace unwrap() with expect() for better error diagnostics in
admin_server_info.rs:
- URL host/port access now has descriptive messages
- HashMap get_mut calls now have descriptive messages

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 11:23:04 +08:00
Zhengchao An 20c7cb1074 fix(server): improve expect() messages in layer.rs (#729 batch 8) (#3989) 
fix(server): improve expect() messages in layer.rs

Replace unwrap() with expect(valid response body) for Response
builder calls in layer.rs.

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 11:22:46 +08:00
Zhengchao An 5c94fe7dd7 fix(ecstore): improve expect() messages in replication_resyncer (#729 batch 7) (#3988) 
fix(ecstore): improve expect() messages in replication_resyncer

Replace unwrap() with expect() for better error diagnostics in
replication_resyncer.rs:
- HashMap get_mut calls now have descriptive expect messages
- format() calls now use unwrap_or_else for error handling

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 11:22:26 +08:00
Zhengchao An d668ef837a fix(admin): improve expect() messages in remaining admin handlers (#729 batch 6) (#3987) 
fix(admin): improve expect() messages in remaining admin handlers

Replace generic parse().unwrap() with parse().expect(valid header value)
for better error diagnostics in service_account.rs, group.rs, and console.rs.

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 11:22:08 +08:00
Zhengchao An e5a355efe9 fix(admin): improve expect() messages in policies handler (#729 batch 5) (#3986) 
fix(admin): improve expect() messages in policies handler

Replace generic parse().unwrap() with parse().expect(valid header value)
for better error diagnostics in policies.rs.

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 11:21:51 +08:00
Zhengchao An dbca96faf6 fix(admin): improve expect() messages in user handler (#729 batch 4) (#3985) 
fix(admin): improve expect() messages in user handler

Replace generic parse().unwrap() with parse().expect(valid header value)
for better error diagnostics in user.rs.

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 11:21:24 +08:00
Zhengchao An e8470bbc66 fix(admin): replace unwrap() with safe pattern in bucket_meta handler (#729 batch 3) (#3984) 
fix(admin): replace unwrap() with safe pattern in bucket_meta handler

Replace 11 instances of HashMap.get_mut().unwrap() with
match pattern that continues to next iteration if key is missing.

Also improve expect() messages for header value parsing.

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 11:20:51 +08:00
houseme 46d7f9e1f2 feat(get): harden codec streaming rollout (#3981) 
* feat(get): consolidate GET performance optimization

Consolidated implementation of all GET performance optimizations into
a single, well-organized commit replacing the previous patch-on-patch
approach.

## Changes

### Configuration (set_disk/mod.rs)
- Consolidated all GET optimization flags into a single organized section
- Enabled by default: codec streaming, metadata early-stop, page cache reclaim
- Added codec streaming multipart flag (default: disabled)
- Added version-aware early-stop flag (default: disabled)
- Added adaptive duplex buffer sizing based on object size
- All flags use OnceLock caching with rollout percentage support

### Metadata Early-Stop (set_disk/read.rs)
- Delete marker early-stop when quorum agrees
- Version-aware early-stop for versioned GET requests
- MetadataQuorumAccumulator enhanced with:
  - delete_marker_votes tracking
  - requested_version_id and matching_version_votes tracking
  - version_early_stop_decision() method
- 6 new tests for version early-stop scenarios

### Codec Streaming (erasure/coding/decode_reader.rs)
- DualInFlight (2-stripe lookahead) enabled by default

### Decode Pipeline (erasure/coding/decode.rs)
- Stripe prefetch count configuration
- Bitrot-decode overlap configuration

### Disk Layer (disk/local.rs)
- O_DIRECT read configuration constants (preparation)

### Metrics (io-metrics/lib.rs)
- BytesPool acquisition/return metrics
- Metadata phase duration with early-stop label
- Total duration with reader_path label

### Diagnostics (diagnostics/)
- Early-stop reason constants
- Pool tier/outcome label constants

### Observability (.docker/observability/)
- 3 Grafana dashboards for GET optimization monitoring
- Prometheus alert rules (6 alerts: 3 critical, 3 warning)
- Updated README.md and README_ZH.md with usage docs

### Config (config/src/constants/runtime.rs)
- Page cache reclaim read enabled by default

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag |
| RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % |
| RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming |
| RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag |
| RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % |
| RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop |
| RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim |
| RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) |
| RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch |
| RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap |
| RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes |

## Rollback

All optimizations can be disabled via environment variables:
RUSTFS_GET_CODEC_STREAMING_ENABLE=false
RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false
RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(get): add stress test scripts for GET optimization validation

- quick-validate-get-optimization.sh: Quick 5-minute validation
- stress-test-get-optimization.sh: Full 30+ minute stress test
- README-stress-test.md: Usage documentation

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): align file cache reclaim defaults

* chore(deps): update redis and erasure codec

* test(ecstore): align decode fill policy default

* fix(get): wire codec streaming rollout gate

* perf(get): skip metrics-off codec timers

* test(get): capture codec streaming diagnostics

* test(get): add multipart fallback probe

* test(get): add encrypted fallback probe

* test(get): add compressed fallback probe

* test(get): add degraded read fallback probe

* test(get): cover remote fallback probe

* test(get): report warp request p99

* test(get): capture OTLP metric deltas

* perf(get): align codec streaming inflight default

* perf(get): reuse codec reader output buffers

* test(get): count codec reader fill starts

* perf(get): reuse codec reader fill worker

* perf(get): lazy init rustfs codec reconstruct

* test(get): cover rustfs codec source faults

* docs(get): record rustfs codec fallback scope

* feat(get): add multipart codec reader opt-in

* test(get): add multipart codec smoke option

* test(get): cover multipart codec degraded fallback

* perf(get): bound multipart codec eager setup

* test(get): satisfy codec hardening PR gate

---------

Co-authored-by: heihutu <heihutu@gmail.com>
 2026-06-28 11:20:21 +08:00
Zhengchao An d99056902e fix(admin): replace unwrap() with proper error handling in tier handler (#729 batch 2) (#3983) 
fix(admin): replace unwrap() with proper error handling in tier handler

Replace 9 instances of args.{type}.clone().unwrap() with
ok_or_else() that returns a descriptive S3Error when the
tier configuration is missing.

Also improve expect() messages for header value parsing.

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 11:00:50 +08:00
Zhengchao An 6e72dc3076 fix(ecstore): replace unsafe k.unwrap() in bucket_target_sys (#729) (#3982) 
fix(ecstore): replace k.unwrap() with safe pattern in bucket_target_sys

Replace unsafe k.unwrap().as_str() with if let Some(key_str) pattern
in 5 locations where HeaderMap iterator yields (Option<HeaderName>, Value).

This prevents potential panics if header names are invalid.

Refs https://github.com/rustfs/backlog/issues/729
 2026-06-28 10:42:56 +08:00
Zhengchao An 259a99a501 fix(ci): pin protocol matrix checkout action (#3977) 2026-06-28 10:04:36 +08:00
houseme c928fd1c7a chore(deps): update flake.lock (#3978) 2026-06-28 09:54:22 +08:00
Zhengchao An 7238a937a9 fix: correct misleading zero-copy/direct-io docs and internal naming (#733 Phase 1) (#3976) 2026-06-28 09:12:49 +08:00
Zhengchao An 3c6dc2a633 ci: remove --no-default-features from protocol test matrix (#3980) 2026-06-28 09:12:23 +08:00
Zhengchao An e1272f2aba revert: restore #![allow(dead_code)] - CI clippy -D warnings conflict (#3979) 
revert: restore #![allow(dead_code)] - clippy -D warnings treats warn as error

The #742 PR changed #![allow(dead_code)] to #![warn(dead_code)], but
CI runs clippy with -D warnings which turns warnings into errors.
This caused CI failures across multiple PRs.

Reverting to #![allow(dead_code)] until the dead code is actually
cleaned up. The 189 warnings in ecstore should be fixed incrementally
by deleting dead code and adding item-level allows, not by changing
the crate-level policy.
 2026-06-28 08:32:34 +08:00
Henry Guo 1e303e5be0 fix(data-usage): refresh versioned usage state (#3969) 
fix(data-usage): refresh versioned usage from authoritative state

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-28 07:58:07 +08:00
Zhengchao An 129ad2296b ci: add protocol feature test matrix (swift, sftp, no-default) (#736) (#3975) 2026-06-28 07:51:15 +08:00
Zhengchao An 113058af54 chore: replace blanket #![allow(dead_code)] with #![warn(dead_code)] (#742) (#3974) 2026-06-28 07:50:51 +08:00
houseme 27468ebfa9 feat(get): consolidate GET performance optimization (#3972) 
* feat(get): consolidate GET performance optimization

Consolidated implementation of all GET performance optimizations into
a single, well-organized commit replacing the previous patch-on-patch
approach.

## Changes

### Configuration (set_disk/mod.rs)
- Consolidated all GET optimization flags into a single organized section
- Enabled by default: codec streaming, metadata early-stop, page cache reclaim
- Added codec streaming multipart flag (default: disabled)
- Added version-aware early-stop flag (default: disabled)
- Added adaptive duplex buffer sizing based on object size
- All flags use OnceLock caching with rollout percentage support

### Metadata Early-Stop (set_disk/read.rs)
- Delete marker early-stop when quorum agrees
- Version-aware early-stop for versioned GET requests
- MetadataQuorumAccumulator enhanced with:
  - delete_marker_votes tracking
  - requested_version_id and matching_version_votes tracking
  - version_early_stop_decision() method
- 6 new tests for version early-stop scenarios

### Codec Streaming (erasure/coding/decode_reader.rs)
- DualInFlight (2-stripe lookahead) enabled by default

### Decode Pipeline (erasure/coding/decode.rs)
- Stripe prefetch count configuration
- Bitrot-decode overlap configuration

### Disk Layer (disk/local.rs)
- O_DIRECT read configuration constants (preparation)

### Metrics (io-metrics/lib.rs)
- BytesPool acquisition/return metrics
- Metadata phase duration with early-stop label
- Total duration with reader_path label

### Diagnostics (diagnostics/)
- Early-stop reason constants
- Pool tier/outcome label constants

### Observability (.docker/observability/)
- 3 Grafana dashboards for GET optimization monitoring
- Prometheus alert rules (6 alerts: 3 critical, 3 warning)
- Updated README.md and README_ZH.md with usage docs

### Config (config/src/constants/runtime.rs)
- Page cache reclaim read enabled by default

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag |
| RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % |
| RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming |
| RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag |
| RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % |
| RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop |
| RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim |
| RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) |
| RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch |
| RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap |
| RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes |

## Rollback

All optimizations can be disabled via environment variables:
RUSTFS_GET_CODEC_STREAMING_ENABLE=false
RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false
RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(get): add stress test scripts for GET optimization validation

- quick-validate-get-optimization.sh: Quick 5-minute validation
- stress-test-get-optimization.sh: Full 30+ minute stress test
- README-stress-test.md: Usage documentation

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): align file cache reclaim defaults

* chore(deps): update redis and erasure codec

* test(ecstore): align decode fill policy default

* test(ecstore): align metadata early-stop default

* fix(ecstore): keep metadata early stop opt-in

---------

Co-authored-by: heihutu <heihutu@gmail.com>
 2026-06-28 07:14:07 +08:00
Zhengchao An 512418cda9 fix(ecstore): replace unbounded metadata cache with moka (#743) (#3970) 
fix(ecstore): replace unbounded metadata cache with moka

Replace the manual Arc<RwLock<HashMap>> metadata cache with
moka::future::Cache, which provides:
- Built-in LRU eviction when max_capacity is reached
- Automatic TTL expiry via time_to_live (250ms)
- Lock-free concurrent reads
- Non-blocking invalidation

Fixes the memory leak risk from unbounded HashMap and the
all-or-nothing eviction logic that cleared all entries at once.

Closes #743

Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-28 03:01:32 +08:00
houseme 175566f037 docs(get): record default switch readiness (#3967) 
* docs(get): record default switch readiness

* chore(docs): keep pr35 evaluation note local

* refactor(bench): trim pr35 readiness python
 2026-06-28 01:12:44 +08:00
Zhengchao An 7ae9697d42 docs: close final architecture audit gaps (#3966) 2026-06-28 00:07:42 +08:00
Zhengchao An 79234c030d docs: close architecture migration validation (#3965) 2026-06-27 23:42:51 +08:00
Zhengchao An 82bbef0b60 test: cover runtime and repair preservation (#3964) 2026-06-27 23:34:59 +08:00
houseme bf03ff2869 feat(get): add limited opt-in rollout gates (#3963) 
* feat(bench): harden cooled get ab harness

* feat(get): add limited opt-in rollout gates

* fix(get): tighten rollout gate fallbacks
 2026-06-27 23:13:02 +08:00
Zhengchao An 66ad138505 docs: close phase 7 global split plan (#3962) 2026-06-27 22:25:16 +08:00
houseme 47d05a1d6a feat(bench): harden cooled get ab harness (#3960) 2026-06-27 22:23:09 +08:00
Zhengchao An c7dc3d7974 refactor: move runtime fallback defaults to owners (#3961) 2026-06-27 22:21:09 +08:00
houseme 20f56af09c feat(get): tune output response handoff (#3956) 2026-06-27 21:46:38 +08:00
Zhengchao An 68d5d1d41d refactor: isolate specialized runtime fallbacks (#3958) 2026-06-27 21:35:47 +08:00
Zhengchao An 672f6e9ea9 refactor: remove scalar runtime fallbacks (#3957) 2026-06-27 20:45:13 +08:00
Zhengchao An 243a20b14e refactor: remove service resolver fallbacks (#3955) 2026-06-27 19:35:43 +08:00
Zhengchao An 4732f4ee2d refactor: expose runtime fallback boundaries (#3954) 2026-06-27 18:57:51 +08:00
Zhengchao An cdebdeca61 refactor: clean app context fallback signatures (#3953) 2026-06-27 18:30:06 +08:00
Zhengchao An d66391636b refactor: remove optional runtime handle fallbacks (#3952) 2026-06-27 18:07:09 +08:00
Zhengchao An 430aa8e8cb refactor: remove notification config fallbacks (#3951) 2026-06-27 17:43:05 +08:00
Zhengchao An ab9e98a9ca refactor: align core runtime facade helpers (#3948) 2026-06-27 17:14:25 +08:00
houseme de86025f2c feat(get): overlap read verify decode (#3945) 2026-06-27 15:14:44 +08:00
Zhengchao An dff0e467f4 refactor: align app runtime facade helpers (#3947) 2026-06-27 15:14:29 +08:00
Zhengchao An 7979ecb545 refactor: align storage runtime facade helpers (#3946) 2026-06-27 14:55:37 +08:00
Zhengchao An 261e16a446 refactor: route admin runtime sources through current helpers (#3944) 2026-06-27 14:19:18 +08:00