gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-28 17:18:58 +00:00
Code Issues Packages Projects Releases Wiki Activity
3,797 Commits 31 Branches 117 Tags
cd165ab1817b4c8d468904c4ef8673e56867eb4e
Commit Graph

591 Commits

Author SHA1 Message Date
Zhengchao An 710ae74cde perf: add S3 operations benchmark framework (#738) (#4005) 2026-06-28 18:02:41 +08:00
Zhengchao An ee82d6c026 test: add insta snapshot test for storage error display format (#740) (#4001) 
test: add insta snapshot test for storage error display format

Add snapshot test to detect unexpected changes in StorageError
display format. This catches output format regressions that
traditional assert tests might miss.

Refs #740
 2026-06-28 16:10:07 +08:00
houseme 27468ebfa9 feat(get): consolidate GET performance optimization (#3972) 
* feat(get): consolidate GET performance optimization

Consolidated implementation of all GET performance optimizations into
a single, well-organized commit replacing the previous patch-on-patch
approach.

## Changes

### Configuration (set_disk/mod.rs)
- Consolidated all GET optimization flags into a single organized section
- Enabled by default: codec streaming, metadata early-stop, page cache reclaim
- Added codec streaming multipart flag (default: disabled)
- Added version-aware early-stop flag (default: disabled)
- Added adaptive duplex buffer sizing based on object size
- All flags use OnceLock caching with rollout percentage support

### Metadata Early-Stop (set_disk/read.rs)
- Delete marker early-stop when quorum agrees
- Version-aware early-stop for versioned GET requests
- MetadataQuorumAccumulator enhanced with:
  - delete_marker_votes tracking
  - requested_version_id and matching_version_votes tracking
  - version_early_stop_decision() method
- 6 new tests for version early-stop scenarios

### Codec Streaming (erasure/coding/decode_reader.rs)
- DualInFlight (2-stripe lookahead) enabled by default

### Decode Pipeline (erasure/coding/decode.rs)
- Stripe prefetch count configuration
- Bitrot-decode overlap configuration

### Disk Layer (disk/local.rs)
- O_DIRECT read configuration constants (preparation)

### Metrics (io-metrics/lib.rs)
- BytesPool acquisition/return metrics
- Metadata phase duration with early-stop label
- Total duration with reader_path label

### Diagnostics (diagnostics/)
- Early-stop reason constants
- Pool tier/outcome label constants

### Observability (.docker/observability/)
- 3 Grafana dashboards for GET optimization monitoring
- Prometheus alert rules (6 alerts: 3 critical, 3 warning)
- Updated README.md and README_ZH.md with usage docs

### Config (config/src/constants/runtime.rs)
- Page cache reclaim read enabled by default

## Environment Variables

| Variable | Default | Description |
|----------|---------|-------------|
| RUSTFS_GET_CODEC_STREAMING_ENABLE | true | Codec streaming base flag |
| RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT | 100 | Codec streaming rollout % |
| RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE | false | Multipart codec streaming |
| RUSTFS_GET_METADATA_EARLY_STOP_ENABLE | true | Early-stop base flag |
| RUSTFS_GET_METADATA_EARLY_STOP_ROLLOUT_PCT | 100 | Early-stop rollout % |
| RUSTFS_GET_METADATA_VERSION_EARLY_STOP_ENABLE | false | Version-aware early-stop |
| RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE | true | Page cache reclaim |
| RUSTFS_OBJECT_DIRECT_IO_READ_ENABLE | false | O_DIRECT (preparation) |
| RUSTFS_GET_DECODE_STRIPE_PREFETCH_COUNT | 1 | Stripe prefetch |
| RUSTFS_GET_BITROT_DECODE_OVERLAP_ENABLE | false | Bitrot-decode overlap |
| RUSTFS_GET_CODEC_STREAMING_MAX_INFLIGHT | 2 | DualInFlight stripes |

## Rollback

All optimizations can be disabled via environment variables:
RUSTFS_GET_CODEC_STREAMING_ENABLE=false
RUSTFS_GET_METADATA_EARLY_STOP_ENABLE=false
RUSTFS_OBJECT_FILE_CACHE_RECLAIM_READ_ENABLE=false

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(get): add stress test scripts for GET optimization validation

- quick-validate-get-optimization.sh: Quick 5-minute validation
- stress-test-get-optimization.sh: Full 30+ minute stress test
- README-stress-test.md: Usage documentation

Co-Authored-By: heihutu <heihutu@gmail.com>

* test(ecstore): align file cache reclaim defaults

* chore(deps): update redis and erasure codec

* test(ecstore): align decode fill policy default

* test(ecstore): align metadata early-stop default

* fix(ecstore): keep metadata early stop opt-in

---------

Co-authored-by: heihutu <heihutu@gmail.com>
 2026-06-28 07:14:07 +08:00
Zhengchao An 512418cda9 fix(ecstore): replace unbounded metadata cache with moka (#743) (#3970) 
fix(ecstore): replace unbounded metadata cache with moka

Replace the manual Arc<RwLock<HashMap>> metadata cache with
moka::future::Cache, which provides:
- Built-in LRU eviction when max_capacity is reached
- Automatic TTL expiry via time_to_live (250ms)
- Lock-free concurrent reads
- Non-blocking invalidation

Fixes the memory leak risk from unbounded HashMap and the
all-or-nothing eviction logic that cleared all entries at once.

Closes #743

Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-28 03:01:32 +08:00
houseme 58b76a3d45 feat(get): add codec engine ab matrix (#3940) 
* upgrade version

* feat(get): add codec engine ab matrix

* chore(get): drop unrelated dependency drift

* upgrade version

* upgrade version

* fix cargo deny
 2026-06-27 13:27:16 +08:00
Zhengchao An 840d21d201 fix(deps): remove vulnerable thrift dependency (#3926) 2026-06-27 00:16:51 +08:00
cxymds 30957e2b51 fix(ecstore): throttle data movement under read pressure (#3906) 2026-06-26 17:48:09 +08:00
cxymds 90638bfc19 fix(heal): add backpressure to repair admission (#3900) 2026-06-26 15:19:37 +08:00
houseme f7a9a7142b chore: update deps and e2e cache action (#3885) 
* chore(deps): update workspace crate versions

* ci: update e2e s3tests cache action
 2026-06-26 04:19:28 +08:00
houseme 623fc801f1 feat(replication): support custom TLS for bucket targets (#3825) 
* feat(replication): support insecure https bucket targets

* feat(replication): support custom ca bucket targets

* test(replication): satisfy e2e clippy for TLS helpers

* fix(replication): avoid native root panic for custom trust stores

* test(replication): decouple private IP target test from TLS roots

* test(replication): use target TLS client in private IP unit test
 2026-06-24 20:23:20 +08:00
houseme 0a00d8d500 perf(server): lighten internode data-plane stack (#3735) 
* refactor(server): split internode dispatch scaffold

* test(server): cover internode dispatch prefix split

* refactor(server): name internode stack boundaries

* perf(server): skip internode request logging layer

* perf(server): skip internode trace layer

* perf(server): use lite internode request context

* feat(metrics): track internode rpc duration

* feat(ecstore): add put object stage summary logs

* test(metrics): update internode descriptor expectations

* fix(server): tighten internode path matching

* fix(pr): address review follow-up comments

* style(ecstore): simplify commit tail duration field

* refactor(ecstore): group put stage summary fields

* refactor(ecstore): inline put stage summary log

* fix(s3): return storage class for object attributes

* merge: sync latest main and resolve object attributes conflict

* fmt

* fix(server): remove duplicate rpc imports

* build(deps): bump memmap2 for RUSTSEC-2026-0186

* fix(s3select): align object_store with datafusion

* chore(deps): prune workspace dependencies

* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization

  Separate fuzz harness compilation from execution to eliminate redundant
  builds across targets. Introduce matrix-based parallel execution for
  PR smoke and nightly fuzz jobs.

  Changes:

  - Split CI workflow into `fuzz-build` (compile once) and matrix run jobs
    (`pr-fuzz-smoke`, `nightly-fuzz-corpus`) that execute targets in parallel
  - Add `BUILD_ONLY` mode to run_ci_targets.sh / run_nightly_targets.sh
  - Add run_single_target.sh for matrix jobs (no build phase)
  - Optimize `local_metadata` fuzz target: reduce prefix iterations from
    8-10 (4 functions each) to 5 critical prefixes (parser-only), cutting
    per-iteration cost by ~3-5x
  - Move archive path validation (`validate_extract_relative_path`,
    `normalize_extract_entry_key`) from `rustfs` to `rustfs-utils::path`,
    eliminating `rustfs` binary crate dependency from fuzz harness
  - Remove `rustfs` from fuzz/Cargo.toml (drops significant transitive deps)
  - Add unit tests for archive path validation in rustfs-utils
  - Update fuzz/README.md with new workflow and script documentation

  Expected CI improvement: PR smoke wall-clock from ~120min (frequent
  timeout) to ~40min; nightly from ~180min to ~60min.

* refactor(fuzz): consolidate scripts and fix prefix test alignment

Replace three duplicated shell scripts (run_ci_targets.sh,
run_nightly_targets.sh, run_single_target.sh) with a single
parameterized run.sh that supports BUILD_ONLY, SKIP_BUILD, and
MAX_TOTAL_TIME environment variables.

Fix local_metadata fuzz target prefix testing: replace always-true
'len > 0' guard with lengths aligned to xl.meta binary layout
(4/5/8/12 bytes for magic+version+header fields). Remove redundant
empty-slice test.

Hoist RUSTFLAGS to workflow top-level env to eliminate per-job
duplication. Update README with unified script documentation.

Net: -118 lines, zero functionality loss.

* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization

Restructure fuzz CI workflow to eliminate redundant compilation and
run targets in parallel via matrix strategy.

Workflow changes:
- Split into fuzz-build (compile once) and matrix run jobs
- PR smoke: 3 targets parallel, 60s each, timeout 30min (was 120min)
- Nightly: 3 targets parallel, 300s each, timeout 60min (was 180min)
- Pass compiled harness via actions/artifact between jobs
- Hoist RUSTFLAGS to workflow top-level env

Script consolidation:
- Replace 3 duplicated scripts with single parameterized run.sh
- Supports BUILD_ONLY, SKIP_BUILD, MAX_TOTAL_TIME env vars

Target optimizations:
- Remove rustfs binary crate from fuzz dependencies (was pulling
  979 transitive deps); move archive path validation to rustfs-utils
- Optimize local_metadata: reduce prefix iterations from 8-10x4
  calls to 5 prefixes with parser-only (no decompress), aligned
  with xl.meta binary layout (4/5/8/12 bytes)
- Add unit tests for archive path validation in rustfs-utils
- Update fuzz/README.md with unified script documentation

Expected: PR smoke wall-clock from ~120min (frequent timeout)
to ~40min; nightly from ~180min to ~60min.

* fix(rpc): resolve internode metrics via app context

* fmt

* ci: speed up fuzz smoke artifact restore

---------

Signed-off-by: houseme <housemecn@gmail.com>
 2026-06-23 21:36:39 +08:00
houseme 46775ae019 feat: harden runtime capability snapshots (#3784) 
* feat(admin): expose runtime capability snapshots

* feat(runtime): refine workload admission snapshots

* test(ci): align architecture migration checks

* build(deps): bump memmap2 for RUSTSEC-2026-0186

* build(deps): refresh cargo deny lockfile
 2026-06-23 18:10:49 +08:00
houseme 2f85ef4455 fix(security): extend outbound egress guard (#3744) 2026-06-22 19:20:31 +08:00
dependabot[bot] 5a601869f6 chore(deps): bump sysinfo from 0.39.3 to 0.39.4 in the dependencies group (#3712) 2026-06-22 09:53:57 +08:00
houseme 9e6d0c7292 chore(deps): refresh workflow actions and crate pins (#3696) 2026-06-21 20:29:17 +08:00
安正超 08b8f58e64 refactor: prune notify ecstore object alias (#3620) 2026-06-19 20:11:56 +08:00
houseme 8cf3c0bfbd fix(ecstore): support MinIO DARE fixture compatibility (#3590) 2026-06-19 10:13:39 +08:00
houseme 53ff19dd2d fix(kms): harden local master key storage (#3589) 2026-06-19 08:50:35 +08:00
安正超 b1c6578df1 refactor: narrow test harness compatibility surfaces (#3592) 2026-06-19 07:10:52 +08:00
安正超 c28fee0013 refactor: continue storage api contract cleanup (#3580) 
* refactor: move delete object contracts to storage api

* refactor: narrow store api compatibility exports

* refactor: route table catalog test through storage compat
 2026-06-18 22:42:02 +08:00
houseme 8cf11af649 fix(security): add shared outbound egress guard (#3567) 
* fix(security): block unsafe outbound webhook targets

* chore: keep issue-3557 plan local only
 2026-06-18 15:30:09 +08:00
安正超 54bbd05b25 refactor: move object list helper contracts (#3546) 2026-06-18 06:09:33 +08:00
houseme 87a3d107d6 test(property): add path validator invariants (#3541) 
test(property): add path validator invariants for #3525
 2026-06-18 02:14:24 +08:00
houseme bc769948ab test(property): add erasure recoverability invariants (#3540) 2026-06-18 01:34:07 +08:00
安正超 53337a6f71 refactor: move HTTP range helper contracts (#3533) 2026-06-18 00:47:23 +08:00
houseme cc84d9065e test(property): add filemeta invariants (#3536) 
* test(property): add filemeta invariants for #3523

* docs: relocate issue-3518 property testing plan

* docs: keep issue-3518 property plan local only
 2026-06-18 00:36:55 +08:00
houseme 841f86fe90 test(property): expand parser invariants (#3534) 
* test(property): harden parser invariants for #3522

* test(property): satisfy clippy for #3522
 2026-06-17 23:52:30 +08:00
安正超 919deeb816 refactor: move object option helper contracts (#3521) 2026-06-17 22:56:10 +08:00
Henry Guo 65c724c786 feat(table-catalog): add manifest reachability cleanup (#3484) 
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
 2026-06-15 22:29:23 +08:00
houseme adec195486 chore(scanner): enable pulsar telemetry feature (#3481) 2026-06-15 18:43:14 +08:00
dependabot[bot] fbf1f9e5b8 build(deps): bump brotli from 8.0.3 to 8.0.4 in the dependencies group (#3458) 
* build(deps): bump brotli from 8.0.3 to 8.0.4 in the dependencies group

Bumps the dependencies group with 1 update: [brotli](https://github.com/dropbox/rust-brotli).


Updates `brotli` from 8.0.3 to 8.0.4
- [Release notes](https://github.com/dropbox/rust-brotli/releases)
- [Commits](https://github.com/dropbox/rust-brotli/compare/8.0.3...8.0.4)

---
updated-dependencies:
- dependency-name: brotli
  dependency-version: 8.0.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* build(deps): bump s3s from cf4c3346 to c59b62f7ba in the dependencies group

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-15 12:06:01 +08:00
houseme 036741cb1c refactor(request-id): align contracts and lock field names (#3454) 
* refactor(request-id): align header log and trace contracts

* test(contract): lock external request-id field names

* chore(deps): drop unused rustfs-ecstore links

Co-Authored-By: heihutu <heihutu@gmail.com>
 2026-06-15 01:59:11 +08:00
cxymds 046d5386ba feat: stream object zip downloads (#3380) 
* feat: stream object zip downloads

* fix: stream zip downloads page by page

* fix: prepare zip downloads before streaming

---------

Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-14 14:26:44 +08:00
houseme e8012bd1ba refactor(logging): normalize admin telemetry and error messages (#3430) 2026-06-14 13:27:10 +08:00
houseme 0d68851f7a refactor(logging): standardize rustfs runtime events (#3396) 
* fix(rpc): adjust log levels and reduce noise in storage RPC layer

Issue #682: pool rebalance normal flow logs were using warn! instead
of info!. Additionally, several high-frequency RPC entry points (ping,
write_stream, read_at, walk_dir) were logging at info! level, causing
unnecessary log noise in production.

Changes:
- load_rebalance_meta: warn! → info! for normal flow (3 occurrences)
- ping body decode: info! → debug! (fires on every health check)
- write_stream/read_at/walk_dir entry: info! → debug!
- metrics.rs: lowercase error messages, add structured error field
- http_service.rs: add structured error field to walk_dir failure
- Add tracing::instrument fields for start_rebalance context

* refactor(logging): standardize rustfs runtime events

* build(deps): bump workspace dependency versions

* build(deps): pin time to 0.3.47

* build(deps): update postgres client versions
 2026-06-13 08:39:47 +08:00
安正超 434b71e569 feat(extension): add admin views (#3386) 2026-06-12 15:54:42 +08:00
安正超 1e7c376586 feat(targets): add extension schema adapter (#3385) 2026-06-12 15:27:55 +08:00
安正超 7d6d56a547 feat(extension): add schema contracts (#3384) 2026-06-12 15:07:17 +08:00
安正超 7146c893cb refactor(storage-api): remove legacy bucket DTO re-export (#3364) 2026-06-11 21:24:56 +08:00
安正超 2205991180 refactor(config): extract server config model (#3351) 2026-06-11 16:11:17 +08:00
houseme 7d38b0cf90 refactor(obs): unify local and otlp log output (#3352) 
wip(obs): start lg-002 output unification
 2026-06-11 16:10:54 +08:00
houseme 8d337c4e87 feat(obs): add logging redaction guard rails (#3349) 
* feat(obs): add logging redaction guard rails

* chore(docs): keep logging plans out of lg-001 pr

* test(obs): guard against unmasked access key logs

* test(obs): enforce masked access key log patterns
 2026-06-11 15:24:32 +08:00
houseme ee96e194a3 build(deps): bump s3s for header parsing fix (#3344) 2026-06-11 04:28:36 +00:00
安正超 87968275a7 refactor(storage): route maintenance inventory reads (#3337) 2026-06-11 10:32:48 +08:00
安正超 94c53af264 refactor(storage): use admin API for observability reads (#3335) 2026-06-11 08:13:15 +08:00
安正超 8a90bda16a refactor(admin): use storage admin drive counts (#3332) 2026-06-11 06:36:18 +08:00
安正超 d9ddd1bedc feat(storage-api): add disk inventory contract (#3330) 2026-06-10 16:33:31 +00:00
安正超 bb5d9565a6 feat(storage-api): add bucket DTO contract (#3314) 
* feat(storage-api): add bucket DTO contract

* ci(build): increase workflow timeout to 90 minutes

---------

Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-10 07:16:14 +00:00
houseme b9c924a6ed build(ci): update macOS runner and bump regex/uuid (#3318) 
* ci(build): use macos-26-intel for x86_64 job

* build(deps): bump regex and uuid

* ci(build): disable cross for macos x86_64 job

---------

Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
 2026-06-10 04:42:10 +00:00
houseme 68400933b5 chore(release): prepare 1.0.0-beta.8 (#3317) 
* chore(release): prepare 1.0.0-beta.8

* chore(release): align release assets for 1.0.0-beta.8
 2026-06-10 04:34:39 +00:00