gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-28 09:08:58 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,782 Commits 33 Branches 117 Tags
e2aa4f703e1f2edd74e14893804a77c843fed43f
Commit Graph

3782 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
houseme e2aa4f703e fix 2026-06-28 10:23:22 +08:00
houseme 9d5a2103c3 Merge branch 'main' of github.com:rustfs/rustfs into houseme/get-small-file-optimization 
* 'main' of github.com:rustfs/rustfs:
  fix: correct misleading zero-copy/direct-io docs and internal naming (#733 Phase 1) (#3976)
  ci: remove --no-default-features from protocol test matrix (#3980)
  revert: restore #![allow(dead_code)] - CI clippy -D warnings conflict (#3979)
 2026-06-28 09:46:53 +08:00
Zhengchao An 7238a937a9 fix: correct misleading zero-copy/direct-io docs and internal naming (#733 Phase 1) (#3976) 2026-06-28 09:12:49 +08:00
Zhengchao An 3c6dc2a633 ci: remove --no-default-features from protocol test matrix (#3980) 2026-06-28 09:12:23 +08:00
Zhengchao An e1272f2aba revert: restore #![allow(dead_code)] - CI clippy -D warnings conflict (#3979) 
revert: restore #![allow(dead_code)] - clippy -D warnings treats warn as error

The #742 PR changed #![allow(dead_code)] to #![warn(dead_code)], but
CI runs clippy with -D warnings which turns warnings into errors.
This caused CI failures across multiple PRs.

Reverting to #![allow(dead_code)] until the dead code is actually
cleaned up. The 189 warnings in ecstore should be fixed incrementally
by deleting dead code and adding item-level allows, not by changing
the crate-level policy.
 2026-06-28 08:32:34 +08:00
houseme 67d5620c22 Merge branch 'main' of github.com:rustfs/rustfs into houseme/get-small-file-optimization 
* 'main' of github.com:rustfs/rustfs:
  fix(data-usage): refresh versioned usage state (#3969)
  ci: add protocol feature test matrix (swift, sftp, no-default) (#736) (#3975)
  chore: replace blanket #![allow(dead_code)] with #![warn(dead_code)] (#742) (#3974)
 2026-06-28 08:18:46 +08:00
Henry Guo 1e303e5be0 fix(data-usage): refresh versioned usage state (#3969) 
fix(data-usage): refresh versioned usage from authoritative state

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-28 07:58:07 +08:00
houseme 2c493fb5fd refactor: translate Chinese comments to English 
Translate all Chinese comments to English in modified files:
- rustfs/src/storage/ecfs_extend.rs
- rustfs/src/app/bucket_usecase.rs

Co-Authored-By: heihutu <heihutu@gmail.com>
 2026-06-28 07:53:34 +08:00
Zhengchao An 129ad2296b ci: add protocol feature test matrix (swift, sftp, no-default) (#736) (#3975) 2026-06-28 07:51:15 +08:00
Zhengchao An 113058af54 chore: replace blanket #![allow(dead_code)] with #![warn(dead_code)] (#742) (#3974) 2026-06-28 07:50:51 +08:00
houseme 096fadcb31 feat(get): SF02 - inline data fast path 
Add fast path for small inline objects that bypasses duplex pipe,
tokio::spawn, and bitrot reader creation when data is already in memory.

Changes:
- Add inline data detection before codec streaming gate
- Direct in-memory erasure decode for inline objects <= 128KB
- Add GET_OBJECT_PATH_INLINE_DIRECT metric path
- Skip duplex pipe and background task for inline data

Conditions for fast path:
- Single part object
- Inline data available
- Size <= 128KB
- Not encrypted/compressed/remote
- No range request

Expected impact: 2-3x improvement for small file GET latency.

Closes rustfs/backlog#767

Co-Authored-By: heihutu <heihutu@gmail.com>
 2026-06-28 07:48:17 +08:00
houseme 46e28cfac6 refactor(get): SF01 - use moka instead of dashmap for bucket cache 
Replace OnceLock + RwLock + HashMap with moka::sync::Cache for bucket
validation cache. moka provides built-in TTL support and is already
available in the workspace.

Changes:
- Add moka dependency to rustfs crate
- Replace manual TTL management with moka's time_to_live
- Simplify cache operations

Closes rustfs/backlog#766

Co-Authored-By: heihutu <heihutu@gmail.com>
 2026-06-28 07:40:38 +08:00
houseme 0bf32ab11a feat(get): SF07 - conditional metrics recording 
Gate hot path metrics behind get_stage_metrics_enabled() to reduce
overhead when metrics are not needed.

Changes:
- Conditional record_zero_copy_read
- Conditional manager.record_disk_operation
- Conditional manager.record_access
- Conditional manager.record_transfer

Expected impact: 20-50us reduction per GET request.

Closes rustfs/backlog#772

Co-Authored-By: heihutu <heihutu@gmail.com>
 2026-06-28 07:31:01 +08:00
houseme ce43a2189d feat(get): SF06 - conditional lifecycle check 
Only call resolve_put_object_expiration when the object has an
x-amz-expiration metadata marker. This avoids unnecessary lifecycle
configuration reads on every GET request.

Expected impact: 50-100us reduction per GET request.

Closes rustfs/backlog#771

Co-Authored-By: heihutu <heihutu@gmail.com>
 2026-06-28 07:29:40 +08:00
houseme c2acb9aeb6 feat(get): SF04 - remove unnecessary tokio::spawn in metadata fanout 
Replace tokio::spawn with direct async future in read_all_fileinfo_full_wait.
join_all already provides concurrency, so tokio::spawn adds unnecessary
task creation and scheduling overhead.

Changes:
- Remove tokio::spawn from metadata fanout futures
- Update result handling for direct future results

Expected impact: 16-32us reduction per GET request.

Closes rustfs/backlog#769

Co-Authored-By: heihutu <heihutu@gmail.com>
 2026-06-28 07:28:37 +08:00
houseme 295178df80 feat(get): SF03 - metadata cache TTL increase 
Increase metadata cache TTL from 250ms to 2s and capacity from 1024
to 4096 entries.

Changes:
- GET_OBJECT_METADATA_CACHE_TTL: 250ms -> 2s
- GET_OBJECT_METADATA_CACHE_MAX_ENTRIES: 1024 -> 4096

All mutation paths already call invalidate_get_object_metadata_cache,
so the longer TTL is safe.

Expected impact: 10-50x improvement for hot objects.

Closes rustfs/backlog#768

Co-Authored-By: heihutu <heihutu@gmail.com>
 2026-06-28 07:25:15 +08:00
houseme 6183775412 feat(get): SF01 - bucket validation cache 
Add 5s TTL cache for bucket validation to avoid repeated stat_volume()
calls on every GET request.

Changes:
- Add BUCKET_VALIDATED_CACHE (OnceLock + RwLock + HashMap)
- Add invalidate_bucket_validation_cache() for cache invalidation
- Add invalidate_all_bucket_validation_cache() for bulk invalidation
- Update get_validated_store() to use cache
- Add cache invalidation in execute_delete_bucket()

Expected impact: 3-5x improvement for small file GET latency.

Closes rustfs/backlog#766

Co-Authored-By: heihutu <heihutu@gmail.com>
 2026-06-28 07:23:36 +08:00
houseme 27468ebfa9 feat(get): consolidate GET performance optimization (#3972) 
* feat(get): consolidate GET performance optimization

Consolidated implementation of all GET performance optimizations into
a single, well-organized commit replacing the previous patch-on-patch
approach.

## Changes

### Configuration (set_disk/mod.rs)
- Consolidated all GET optimization flags into a single organized section
- Enabled by default: codec streaming, metadata early-stop, page cache reclaim
- Added codec streaming multipart flag (default: disabled)
- Added version-aware early-stop flag (default: disabled)
- Added adaptive duplex buffer sizing based on object size
- All flags use OnceLock caching with rollout percentage support

### Metadata Early-Stop (set_disk/read.rs)
- Delete marker early-stop when quorum agrees
- Version-aware early-stop for versioned GET requests
- MetadataQuorumAccumulator enhanced with:
  - delete_marker_votes tracking
  - requested_version_id and matching_version_votes tracking
  - version_early_stop_decision() method
- 6 new tests for version early-stop scenarios

### Codec Streaming (erasure/coding/decode_reader.rs)
- DualInFlight (2-stripe lookahead) enabled by default

### Decode Pipeline (erasure/coding/decode.rs)
- Stripe prefetch count configuration
- Bitrot-decode overlap configuration

### Disk Layer (disk/local.rs)
- O_DIRECT read configuration constants (preparation)

### Metrics (io-metrics/lib.rs)
- BytesPool acquisition/return metrics
- Metadata phase duration with early-stop label
- Total duration with reader_path label

### Diagnostics (diagnostics/)
- Early-stop reason constants
- Pool tier/outcome label constants

### Observability (.docker/observability/)
- 3 Grafana dashboards for GET optimization monitoring
- Prometheus alert rules (6 alerts: 3 critical, 3 warning)
- Updated README.md and README_ZH.md with usage docs

### Config (config/src/constants/runtime.rs)
- Page cache reclaim read enabled by default

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag |
| RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % |
| RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming |
| RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag |
| RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % |
| RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop |
| RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim |
| RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) |
| RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch |
| RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap |
| RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes |

## Rollback

All optimizations can be disabled via environment variables:
RUSTFS_GET_CODEC_STREAMING_ENABLE=false
RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false
RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(get): add stress test scripts for GET optimization validation

- quick-validate-get-optimization.sh: Quick 5-minute validation
- stress-test-get-optimization.sh: Full 30+ minute stress test
- README-stress-test.md: Usage documentation

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): align file cache reclaim defaults

* chore(deps): update redis and erasure codec

* test(ecstore): align decode fill policy default

* test(ecstore): align metadata early-stop default

* fix(ecstore): keep metadata early stop opt-in

---------

Co-authored-by: heihutu <heihutu@gmail.com>
 2026-06-28 07:14:07 +08:00
Zhengchao An 512418cda9 fix(ecstore): replace unbounded metadata cache with moka (#743) (#3970) 
fix(ecstore): replace unbounded metadata cache with moka

Replace the manual Arc<RwLock<HashMap>> metadata cache with
moka::future::Cache, which provides:
- Built-in LRU eviction when max_capacity is reached
- Automatic TTL expiry via time_to_live (250ms)
- Lock-free concurrent reads
- Non-blocking invalidation

Fixes the memory leak risk from unbounded HashMap and the
all-or-nothing eviction logic that cleared all entries at once.

Closes #743

Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-28 03:01:32 +08:00
houseme 175566f037 docs(get): record default switch readiness (#3967) 
* docs(get): record default switch readiness

* chore(docs): keep pr35 evaluation note local

* refactor(bench): trim pr35 readiness python
 2026-06-28 01:12:44 +08:00
Zhengchao An 7ae9697d42 docs: close final architecture audit gaps (#3966) 2026-06-28 00:07:42 +08:00
Zhengchao An 79234c030d docs: close architecture migration validation (#3965) 2026-06-27 23:42:51 +08:00
Zhengchao An 82bbef0b60 test: cover runtime and repair preservation (#3964) 2026-06-27 23:34:59 +08:00
houseme bf03ff2869 feat(get): add limited opt-in rollout gates (#3963) 
* feat(bench): harden cooled get ab harness

* feat(get): add limited opt-in rollout gates

* fix(get): tighten rollout gate fallbacks
 2026-06-27 23:13:02 +08:00
Zhengchao An 66ad138505 docs: close phase 7 global split plan (#3962) 2026-06-27 22:25:16 +08:00
houseme 47d05a1d6a feat(bench): harden cooled get ab harness (#3960) 2026-06-27 22:23:09 +08:00
Zhengchao An c7dc3d7974 refactor: move runtime fallback defaults to owners (#3961) 2026-06-27 22:21:09 +08:00
houseme 20f56af09c feat(get): tune output response handoff (#3956) 2026-06-27 21:46:38 +08:00
Zhengchao An 68d5d1d41d refactor: isolate specialized runtime fallbacks (#3958) 2026-06-27 21:35:47 +08:00
Zhengchao An 672f6e9ea9 refactor: remove scalar runtime fallbacks (#3957) 2026-06-27 20:45:13 +08:00
Zhengchao An 243a20b14e refactor: remove service resolver fallbacks (#3955) 2026-06-27 19:35:43 +08:00
Zhengchao An 4732f4ee2d refactor: expose runtime fallback boundaries (#3954) 2026-06-27 18:57:51 +08:00
Zhengchao An cdebdeca61 refactor: clean app context fallback signatures (#3953) 2026-06-27 18:30:06 +08:00
Zhengchao An d66391636b refactor: remove optional runtime handle fallbacks (#3952) 2026-06-27 18:07:09 +08:00
Zhengchao An 430aa8e8cb refactor: remove notification config fallbacks (#3951) 2026-06-27 17:43:05 +08:00
Zhengchao An ab9e98a9ca refactor: align core runtime facade helpers (#3948) 2026-06-27 17:14:25 +08:00
houseme de86025f2c feat(get): overlap read verify decode (#3945) 2026-06-27 15:14:44 +08:00
Zhengchao An dff0e467f4 refactor: align app runtime facade helpers (#3947) 2026-06-27 15:14:29 +08:00
Zhengchao An 7979ecb545 refactor: align storage runtime facade helpers (#3946) 2026-06-27 14:55:37 +08:00
Zhengchao An 261e16a446 refactor: route admin runtime sources through current helpers (#3944) 2026-06-27 14:19:18 +08:00
Zhengchao An cc41a50efe refactor: route admin runtime consumers through app context (#3943) 2026-06-27 13:38:21 +08:00
houseme 58b76a3d45 feat(get): add codec engine ab matrix (#3940) 
* upgrade version

* feat(get): add codec engine ab matrix

* chore(get): drop unrelated dependency drift

* upgrade version

* upgrade version

* fix cargo deny
 2026-06-27 13:27:16 +08:00
Zhengchao An 996e58fd9a refactor: route admin runtime config through app context (#3942) 2026-06-27 13:10:00 +08:00
Zhengchao An 1b3dea012e refactor: route ecstore runtime globals through facade (#3941) 2026-06-27 12:27:03 +08:00
Zhengchao An d3f1ff36af docs: record global state cleanup plan (#3939) 2026-06-27 12:04:59 +08:00
Zhengchao An fe9c0daf0f docs: add architecture readiness matrix (#3938) 2026-06-27 11:23:07 +08:00
Zhengchao An e1a4b9e0b6 refactor: batch cluster lock and health readiness (#3936) 2026-06-27 10:51:06 +08:00
GatewayJ 675597ec16 fix(ecstore): handle stalled recovery reads and listings (#3790) 
* fix(ecstore): handle stalled recovery reads and listings

* fix(rio): start HTTP stall timeout on read

* fix(ecstore): handle stalled reads and partial lists

* fix(ecstore): retire stalled shards and list errors

* fix(ecstore): preserve list merge lookahead entries

* fix(ecstore): bound zero-copy shard reads

* fix(ecstore): hedge stalled shard reads

* fix(ecstore): retire abandoned shard reads

* fix(ecstore): include part identity in metadata quorum

* fix(ecstore): validate heal shard sources

* fix(ecstore): verify reconstructed read shards

* chore(ecstore): log slow object read stages

* fix(heal): throttle auto heal during recovery

* fix(scanner): yield to foreground reads

* fix(scanner): track streaming object reads

* fix(ecstore): avoid false read heal fanout

* fix(ecstore): verify codec streaming reconstruction sources

* fix(ecstore): preserve quorum progress on slow shards

* fix(storage): restore read timeout facade

* fix(ecstore): retain fallback readers after quorum

* chore: allow decode helper argument lists

---------

Co-authored-by: overtrue <anzhengchao@gmail.com>
 2026-06-27 10:21:09 +08:00
Zhengchao An 3fb4dcd52e refactor: route cluster control plane readiness (#3935) 2026-06-27 09:47:30 +08:00
Zhengchao An 0a5b1b1b3a refactor: consolidate ecstore owner module layout (#3934) 
* refactor: shrink ecstore root owner facades

* refactor: remove ecstore core store root shims

* refactor: move ecstore erasure owner modules

* refactor: remove ecstore root rpc facade

* refactor: move ecstore services domain modules
 2026-06-27 09:03:20 +08:00