gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-07-28 00:58:59 +00:00
Code Issues Packages Projects Releases Wiki Activity
4,822 Commits 12 Branches 117 Tags
342f9f94bc013b93b127704f797fe42e42498c10
Commit Graph

18 Commits

Author SHA1 Message Date
GatewayJ 0364523dad fix(s3select): enforce query and resource limits (#5028) 
* fix(s3select): enforce query and resource limits

* fix(s3select): close query resource limit gaps

* fix(s3select): preserve timeout and stream invariants

* fix(s3select): enforce staged query limits

* fix(s3select): preserve policy error compatibility

* fix(s3select): bound error source traversal
 2026-07-25 18:44:53 +08:00
GatewayJ 48b2f3d6e3 fix(s3select): preserve CSV input as strings (#5030) 
* fix(s3select): preserve CSV input as strings

* fix(s3select): address CSV schema review findings
 2026-07-20 21:02:54 +08:00
Zhengchao An 765e719aaf fix(s3select): configure session partitions (#4511) 2026-07-09 01:07:01 +08:00
Zhengchao An 31c6859965 chore: converge stale TODOs and apply safe fills (backlog#646) (#4322) 
Second TODO-convergence round over the current tree (backlog#646). All
line numbers in the old inventory had gone stale after the set_disk /
diagnostics / cluster refactors, so this re-scans and reduces the marker
count from 144 to 99.

STALE removals (comment describes already-implemented behavior, or dead
commented-out blocks) across ecstore (set_disk ops/core, store,
cluster/rpc, bucket/metadata_sys, services), iam, filemeta, s3select and
rustfs auth/object_usecase. No behavior change.

Safe fills, each verified:
- filemeta: replication_info_equals now also compares
  replication_state_internal (function currently has no callers; adds a
  regression test).
- bitrot: drop the confirmed-unused `_want` parameter from bitrot_verify
  and the now-unused `sum` on LocalDisk::bitrot_verify, removing a
  Bytes::copy_from_slice allocation. Streaming verify uses the file's
  embedded per-shard hash, never the passed sum.
- signer: rename v4_ignored_headers -> V4_IGNORED_HEADERS and drop the
  non_upper_case_globals allow.
- admin/heal: test_decode was #[ignore]d and used serde_urlencoded on a
  JSON body (would panic); rewire to serde_json::from_slice to match the
  production decode path, add assertions, un-ignore.

Verified: cargo fmt; cargo check on touched crates; tests pass
(filemeta, signer, bitrot, heal::test_decode); arch guardrail scripts
pass.
 2026-07-06 22:43:32 +08:00
houseme 0a00d8d500 perf(server): lighten internode data-plane stack (#3735) 
* refactor(server): split internode dispatch scaffold

* test(server): cover internode dispatch prefix split

* refactor(server): name internode stack boundaries

* perf(server): skip internode request logging layer

* perf(server): skip internode trace layer

* perf(server): use lite internode request context

* feat(metrics): track internode rpc duration

* feat(ecstore): add put object stage summary logs

* test(metrics): update internode descriptor expectations

* fix(server): tighten internode path matching

* fix(pr): address review follow-up comments

* style(ecstore): simplify commit tail duration field

* refactor(ecstore): group put stage summary fields

* refactor(ecstore): inline put stage summary log

* fix(s3): return storage class for object attributes

* merge: sync latest main and resolve object attributes conflict

* fmt

* fix(server): remove duplicate rpc imports

* build(deps): bump memmap2 for RUSTSEC-2026-0186

* fix(s3select): align object_store with datafusion

* chore(deps): prune workspace dependencies

* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization

  Separate fuzz harness compilation from execution to eliminate redundant
  builds across targets. Introduce matrix-based parallel execution for
  PR smoke and nightly fuzz jobs.

  Changes:

  - Split CI workflow into `fuzz-build` (compile once) and matrix run jobs
    (`pr-fuzz-smoke`, `nightly-fuzz-corpus`) that execute targets in parallel
  - Add `BUILD_ONLY` mode to run_ci_targets.sh / run_nightly_targets.sh
  - Add run_single_target.sh for matrix jobs (no build phase)
  - Optimize `local_metadata` fuzz target: reduce prefix iterations from
    8-10 (4 functions each) to 5 critical prefixes (parser-only), cutting
    per-iteration cost by ~3-5x
  - Move archive path validation (`validate_extract_relative_path`,
    `normalize_extract_entry_key`) from `rustfs` to `rustfs-utils::path`,
    eliminating `rustfs` binary crate dependency from fuzz harness
  - Remove `rustfs` from fuzz/Cargo.toml (drops significant transitive deps)
  - Add unit tests for archive path validation in rustfs-utils
  - Update fuzz/README.md with new workflow and script documentation

  Expected CI improvement: PR smoke wall-clock from ~120min (frequent
  timeout) to ~40min; nightly from ~180min to ~60min.

* refactor(fuzz): consolidate scripts and fix prefix test alignment

Replace three duplicated shell scripts (run_ci_targets.sh,
run_nightly_targets.sh, run_single_target.sh) with a single
parameterized run.sh that supports BUILD_ONLY, SKIP_BUILD, and
MAX_TOTAL_TIME environment variables.

Fix local_metadata fuzz target prefix testing: replace always-true
'len > 0' guard with lengths aligned to xl.meta binary layout
(4/5/8/12 bytes for magic+version+header fields). Remove redundant
empty-slice test.

Hoist RUSTFLAGS to workflow top-level env to eliminate per-job
duplication. Update README with unified script documentation.

Net: -118 lines, zero functionality loss.

* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization

Restructure fuzz CI workflow to eliminate redundant compilation and
run targets in parallel via matrix strategy.

Workflow changes:
- Split into fuzz-build (compile once) and matrix run jobs
- PR smoke: 3 targets parallel, 60s each, timeout 30min (was 120min)
- Nightly: 3 targets parallel, 300s each, timeout 60min (was 180min)
- Pass compiled harness via actions/artifact between jobs
- Hoist RUSTFLAGS to workflow top-level env

Script consolidation:
- Replace 3 duplicated scripts with single parameterized run.sh
- Supports BUILD_ONLY, SKIP_BUILD, MAX_TOTAL_TIME env vars

Target optimizations:
- Remove rustfs binary crate from fuzz dependencies (was pulling
  979 transitive deps); move archive path validation to rustfs-utils
- Optimize local_metadata: reduce prefix iterations from 8-10x4
  calls to 5 prefixes with parser-only (no decompress), aligned
  with xl.meta binary layout (4/5/8/12 bytes)
- Add unit tests for archive path validation in rustfs-utils
- Update fuzz/README.md with unified script documentation

Expected: PR smoke wall-clock from ~120min (frequent timeout)
to ~40min; nightly from ~180min to ~60min.

* fix(rpc): resolve internode metrics via app context

* fmt

* ci: speed up fuzz smoke artifact restore

---------

Signed-off-by: houseme <housemecn@gmail.com>
 2026-06-23 21:36:39 +08:00
GatewayJ 785d53fce8 fix(s3select): validate scan_range protocol and parquet overlap (#3176) 
* fix(s3select): enforce scan range protocol and parquet overlap

* fix(s3select): select row groups by start offset

---------

Co-authored-by: houseme <housemecn@gmail.com>
 2026-06-03 15:57:17 +00:00
GatewayJ b0646be756 feat(s3select): improve SelectObjectContent streaming (#3072) 
* feat(s3select): improve SelectObjectContent streaming

* fix(s3select): reject empty select expressions

* fix(s3select): address streaming review feedback

---------

Co-authored-by: loverustfs <hello@rustfs.com>
 2026-05-24 14:23:47 +00:00
houseme 59f41eb86a feat(obs): improve metrics coverage and dashboard performance (#2682) 2026-04-25 18:51:29 +00:00
houseme ec8f059506 feat: add timing and metrics for query execution phases (#2419) 
Signed-off-by: Shreyan Jain <shreyan11@duck.com>
Co-authored-by: Shreyan Jain <shreyan11@duck.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com>
 2026-04-07 20:32:32 +08:00
houseme af46a61fde build(deps): bump the dependencies group with 6 updates (#2303) 
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
 2026-03-27 12:47:48 +08:00
0xdx2 17b3054a77 feat(s3select): add JSON handling and flattening for EcObjectStore (#1930) 
Signed-off-by: 0xdx2 <xuedamon2@gmail.com>
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2026-02-24 18:05:34 +08:00
houseme 82a6e78845 Inject GlobalReadiness into HTTP server pipeline and gate traffic until FullReady (#1255) 2025-12-25 00:19:03 +08:00
houseme 7e75c9b1f5 remove unlinked file (#1258) 2025-12-24 23:37:43 +08:00
yihong fe4fabb195 fix: other two memory leak in the code base (#1160) 
Signed-off-by: yihong0618 <zouzou0208@gmail.com>
Co-authored-by: houseme <housemecn@gmail.com>
 2025-12-16 11:45:45 +08:00
houseme 18cd9a8b46 build(deps): bump the dependencies group with 5 updates (#896) 2025-11-20 13:04:24 +08:00
houseme 0e680eae31 fix typos and bump the dependencies group with 9 updates (#614) 
* fix typos

* build(deps): bump the dependencies group with 9 updates (#613)

Bumps the dependencies group with 9 updates:

| Package | From | To |
| --- | --- | --- |
| [axum](https://github.com/tokio-rs/axum) | `0.8.4` | `0.8.6` |
| [axum-extra](https://github.com/tokio-rs/axum) | `0.10.1` | `0.10.3` |
| [regex](https://github.com/rust-lang/regex) | `1.11.2` | `1.11.3` |
| [serde](https://github.com/serde-rs/serde) | `1.0.226` | `1.0.228` |
| [shadow-rs](https://github.com/baoyachi/shadow-rs) | `1.3.0` | `1.4.0` |
| [sysinfo](https://github.com/GuillaumeGomez/sysinfo) | `0.37.0` | `0.37.1` |
| [thiserror](https://github.com/dtolnay/thiserror) | `2.0.16` | `2.0.17` |
| [tokio-rustls](https://github.com/rustls/tokio-rustls) | `0.26.3` | `0.26.4` |
| [zeroize](https://github.com/RustCrypto/utils) | `1.8.1` | `1.8.2` |


Updates `axum` from 0.8.4 to 0.8.6
- [Release notes](https://github.com/tokio-rs/axum/releases)
- [Changelog](https://github.com/tokio-rs/axum/blob/main/CHANGELOG.md)
- [Commits](https://github.com/tokio-rs/axum/compare/axum-v0.8.4...axum-v0.8.6)

Updates `axum-extra` from 0.10.1 to 0.10.3
- [Release notes](https://github.com/tokio-rs/axum/releases)
- [Changelog](https://github.com/tokio-rs/axum/blob/main/CHANGELOG.md)
- [Commits](https://github.com/tokio-rs/axum/compare/axum-extra-v0.10.1...axum-extra-v0.10.3)

Updates `regex` from 1.11.2 to 1.11.3
- [Release notes](https://github.com/rust-lang/regex/releases)
- [Changelog](https://github.com/rust-lang/regex/blob/master/CHANGELOG.md)
- [Commits](https://github.com/rust-lang/regex/compare/1.11.2...1.11.3)

Updates `serde` from 1.0.226 to 1.0.228
- [Release notes](https://github.com/serde-rs/serde/releases)
- [Commits](https://github.com/serde-rs/serde/compare/v1.0.226...v1.0.228)

Updates `shadow-rs` from 1.3.0 to 1.4.0
- [Release notes](https://github.com/baoyachi/shadow-rs/releases)
- [Commits](https://github.com/baoyachi/shadow-rs/compare/1.3.0...v1.4.0)

Updates `sysinfo` from 0.37.0 to 0.37.1
- [Changelog](https://github.com/GuillaumeGomez/sysinfo/blob/master/CHANGELOG.md)
- [Commits](https://github.com/GuillaumeGomez/sysinfo/compare/v0.37.0...v0.37.1)

Updates `thiserror` from 2.0.16 to 2.0.17
- [Release notes](https://github.com/dtolnay/thiserror/releases)
- [Commits](https://github.com/dtolnay/thiserror/compare/2.0.16...2.0.17)

Updates `tokio-rustls` from 0.26.3 to 0.26.4
- [Release notes](https://github.com/rustls/tokio-rustls/releases)
- [Commits](https://github.com/rustls/tokio-rustls/compare/v/0.26.3...v/0.26.4)

Updates `zeroize` from 1.8.1 to 1.8.2
- [Commits](https://github.com/RustCrypto/utils/compare/zeroize-v1.8.1...zeroize-v1.8.2)

---
updated-dependencies:
- dependency-name: axum
  dependency-version: 0.8.6
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: axum-extra
  dependency-version: 0.10.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: regex
  dependency-version: 1.11.3
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: serde
  dependency-version: 1.0.228
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: shadow-rs
  dependency-version: 1.4.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
- dependency-name: sysinfo
  dependency-version: 0.37.1
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: thiserror
  dependency-version: 2.0.17
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: tokio-rustls
  dependency-version: 0.26.4
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
- dependency-name: zeroize
  dependency-version: 1.8.2
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
 2025-10-02 23:29:18 +08:00
安正超 d552210b59 feat: translate chinese to english (#402) 
* Checkpoint before follow-up message

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

* Translate project documentation and comments from Chinese to English

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

* Fix typo: "unparseable" to "unparsable" in version test comment

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

* Refactor compression test code with minor syntax improvements

Co-authored-by: anzhengchao <anzhengchao@gmail.com>

---------

Co-authored-by: Cursor Agent <cursoragent@cursor.com>
 2025-08-14 00:19:01 +08:00
houseme 5826396cd0 refactor: Restructure project layout and clean up dependencies (#30) 
This commit introduces a significant reorganization of the project structure to improve maintainability and clarity.

Key changes include:
- Adjusted the directory layout for a more logical module organization.
- Removed unused crate dependencies, reducing the overall project size and potentially speeding up build times.
- Updated import paths and configuration files to reflect the structural changes.
 2025-07-02 19:33:12 +08:00