0364523dad
fix(s3select): enforce query and resource limits ( #5028 )
...
* fix(s3select): enforce query and resource limits
* fix(s3select): close query resource limit gaps
* fix(s3select): preserve timeout and stream invariants
* fix(s3select): enforce staged query limits
* fix(s3select): preserve policy error compatibility
* fix(s3select): bound error source traversal
2026-07-25 18:44:53 +08:00
48b2f3d6e3
fix(s3select): preserve CSV input as strings ( #5030 )
...
* fix(s3select): preserve CSV input as strings
* fix(s3select): address CSV schema review findings
2026-07-20 21:02:54 +08:00
8ebedddfa1
fix(s3select): reject truncated object streams ( #5027 )
...
* fix(s3select): reject truncated object streams
* fix(s3select): validate raw stream before conversion
2026-07-19 15:34:20 +00:00
f3a7a4b0da
chore(deps): localize workspace dependency features ( #4888 )
...
Move workspace-level dependency feature lists into the member crates that consume each dependency while keeping required default-features flags at the workspace root.
Also refresh starshard to 2.2.2 via cargo update and cargo upgrade --exclude ratelimit.
Co-authored-by: heihutu <heihutu@gmail.com >
2026-07-16 03:55:27 +00:00
081c10f073
chore(deps): trim s3select datafusion features ( #4869 )
2026-07-15 12:17:26 +00:00
765e719aaf
fix(s3select): configure session partitions ( #4511 )
2026-07-09 01:07:01 +08:00
31c6859965
chore: converge stale TODOs and apply safe fills (backlog#646) ( #4322 )
...
Second TODO-convergence round over the current tree (backlog#646). All
line numbers in the old inventory had gone stale after the set_disk /
diagnostics / cluster refactors, so this re-scans and reduces the marker
count from 144 to 99.
STALE removals (comment describes already-implemented behavior, or dead
commented-out blocks) across ecstore (set_disk ops/core, store,
cluster/rpc, bucket/metadata_sys, services), iam, filemeta, s3select and
rustfs auth/object_usecase. No behavior change.
Safe fills, each verified:
- filemeta: replication_info_equals now also compares
replication_state_internal (function currently has no callers; adds a
regression test).
- bitrot: drop the confirmed-unused `_want` parameter from bitrot_verify
and the now-unused `sum` on LocalDisk::bitrot_verify, removing a
Bytes::copy_from_slice allocation. Streaming verify uses the file's
embedded per-shard hash, never the passed sum.
- signer: rename v4_ignored_headers -> V4_IGNORED_HEADERS and drop the
non_upper_case_globals allow.
- admin/heal: test_decode was #[ignore]d and used serde_urlencoded on a
JSON body (would panic); rewire to serde_json::from_slice to match the
production decode path, add assertions, un-ignore.
Verified: cargo fmt; cargo check on touched crates; tests pass
(filemeta, signer, bitrot, heal::test_decode); arch guardrail scripts
pass.
2026-07-06 22:43:32 +08:00
6dabbaab4d
refactor(deps): replace snafu and heal anyhow ( #4266 )
2026-07-05 00:31:37 +08:00
1b3dea012e
refactor: route ecstore runtime globals through facade ( #3941 )
2026-06-27 12:27:03 +08:00
72ae43cb90
refactor: segment external storage contract imports ( #3908 )
2026-06-26 16:44:59 +08:00
1f7e159388
refactor: segment external storage api boundaries ( #3903 )
2026-06-26 15:10:49 +08:00
c9614eb7cb
refactor: route ecstore storage api boundaries ( #3892 )
2026-06-26 09:35:18 +08:00
e37e367390
refactor: route remaining external storage boundaries ( #3889 )
2026-06-26 06:22:51 +08:00
0a00d8d500
perf(server): lighten internode data-plane stack ( #3735 )
...
* refactor(server): split internode dispatch scaffold
* test(server): cover internode dispatch prefix split
* refactor(server): name internode stack boundaries
* perf(server): skip internode request logging layer
* perf(server): skip internode trace layer
* perf(server): use lite internode request context
* feat(metrics): track internode rpc duration
* feat(ecstore): add put object stage summary logs
* test(metrics): update internode descriptor expectations
* fix(server): tighten internode path matching
* fix(pr): address review follow-up comments
* style(ecstore): simplify commit tail duration field
* refactor(ecstore): group put stage summary fields
* refactor(ecstore): inline put stage summary log
* fix(s3): return storage class for object attributes
* merge: sync latest main and resolve object attributes conflict
* fmt
* fix(server): remove duplicate rpc imports
* build(deps): bump memmap2 for RUSTSEC-2026-0186
* fix(s3select): align object_store with datafusion
* chore(deps): prune workspace dependencies
* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization
Separate fuzz harness compilation from execution to eliminate redundant
builds across targets. Introduce matrix-based parallel execution for
PR smoke and nightly fuzz jobs.
Changes:
- Split CI workflow into `fuzz-build` (compile once) and matrix run jobs
(`pr-fuzz-smoke`, `nightly-fuzz-corpus`) that execute targets in parallel
- Add `BUILD_ONLY` mode to run_ci_targets.sh / run_nightly_targets.sh
- Add run_single_target.sh for matrix jobs (no build phase)
- Optimize `local_metadata` fuzz target: reduce prefix iterations from
8-10 (4 functions each) to 5 critical prefixes (parser-only), cutting
per-iteration cost by ~3-5x
- Move archive path validation (`validate_extract_relative_path`,
`normalize_extract_entry_key`) from `rustfs` to `rustfs-utils::path`,
eliminating `rustfs` binary crate dependency from fuzz harness
- Remove `rustfs` from fuzz/Cargo.toml (drops significant transitive deps)
- Add unit tests for archive path validation in rustfs-utils
- Update fuzz/README.md with new workflow and script documentation
Expected CI improvement: PR smoke wall-clock from ~120min (frequent
timeout) to ~40min; nightly from ~180min to ~60min.
* refactor(fuzz): consolidate scripts and fix prefix test alignment
Replace three duplicated shell scripts (run_ci_targets.sh,
run_nightly_targets.sh, run_single_target.sh) with a single
parameterized run.sh that supports BUILD_ONLY, SKIP_BUILD, and
MAX_TOTAL_TIME environment variables.
Fix local_metadata fuzz target prefix testing: replace always-true
'len > 0' guard with lengths aligned to xl.meta binary layout
(4/5/8/12 bytes for magic+version+header fields). Remove redundant
empty-slice test.
Hoist RUSTFLAGS to workflow top-level env to eliminate per-job
duplication. Update README with unified script documentation.
Net: -118 lines, zero functionality loss.
* perf(fuzz): optimize CI runtime with build/run split and matrix parallelization
Restructure fuzz CI workflow to eliminate redundant compilation and
run targets in parallel via matrix strategy.
Workflow changes:
- Split into fuzz-build (compile once) and matrix run jobs
- PR smoke: 3 targets parallel, 60s each, timeout 30min (was 120min)
- Nightly: 3 targets parallel, 300s each, timeout 60min (was 180min)
- Pass compiled harness via actions/artifact between jobs
- Hoist RUSTFLAGS to workflow top-level env
Script consolidation:
- Replace 3 duplicated scripts with single parameterized run.sh
- Supports BUILD_ONLY, SKIP_BUILD, MAX_TOTAL_TIME env vars
Target optimizations:
- Remove rustfs binary crate from fuzz dependencies (was pulling
979 transitive deps); move archive path validation to rustfs-utils
- Optimize local_metadata: reduce prefix iterations from 8-10x4
calls to 5 prefixes with parser-only (no decompress), aligned
with xl.meta binary layout (4/5/8/12 bytes)
- Add unit tests for archive path validation in rustfs-utils
- Update fuzz/README.md with unified script documentation
Expected: PR smoke wall-clock from ~120min (frequent timeout)
to ~40min; nightly from ~180min to ~60min.
* fix(rpc): resolve internode metrics via app context
* fmt
* ci: speed up fuzz smoke artifact restore
---------
Signed-off-by: houseme <housemecn@gmail.com >
2026-06-23 21:36:39 +08:00
198fd4f150
refactor(runtime): route RustFS runtime consumers through storage owner ( #3756 )
2026-06-23 05:17:56 +08:00
db7ff8f513
refactor: expose external ECStore owner symbols ( #3748 )
2026-06-22 21:42:54 +08:00
418b5d04f9
refactor: route external ECStore imports through owners ( #3747 )
2026-06-22 20:57:10 +08:00
88a87b3e8f
refactor: centralize external ECStore facade aliases ( #3746 )
2026-06-22 20:37:13 +08:00
539c68778d
refactor: remove standalone compat bridge modules ( #3740 )
2026-06-22 17:35:58 +08:00
cca9e83a8b
refactor: use relative standalone compat consumers ( #3734 )
2026-06-22 15:07:58 +08:00
de158743c3
refactor: split compat facade imports ( #3716 )
2026-06-22 09:53:22 +08:00
d39815470e
refactor: prune consumer storage compat paths ( #3704 )
2026-06-22 02:31:30 +08:00
0985225448
refactor: consolidate ecstore public facade ( #3678 )
...
* refactor: expand ecstore compatibility facade
* test: enforce ecstore api facade imports
* refactor: hide legacy ecstore layout modules
* refactor: hide ecstore facade root modules
2026-06-21 09:09:11 +08:00
9b3fda9e30
refactor: route ecstore public surfaces through api ( #3673 )
2026-06-21 05:23:23 +08:00
afe46c01e7
refactor: prune consumer ecstore object aliases ( #3622 )
2026-06-19 21:33:00 +08:00
54ffbedbc8
refactor: remove ecstore operation compat facades ( #3608 )
2026-06-19 17:01:12 +08:00
a860f2b40c
refactor: narrow storage compatibility surfaces ( #3585 )
...
* refactor: narrow storage compatibility surfaces
* refactor: narrow observability compatibility surfaces
2026-06-19 00:56:14 +08:00
205f964dc5
refactor: tighten storage compat store_api aliases ( #3582 )
...
* refactor: collapse storage compat store_api modules
* chore: guard storage compat store_api aliases
2026-06-18 22:50:02 +08:00
c28fee0013
refactor: continue storage api contract cleanup ( #3580 )
...
* refactor: move delete object contracts to storage api
* refactor: narrow store api compatibility exports
* refactor: route table catalog test through storage compat
2026-06-18 22:42:02 +08:00
7b0cb9e725
refactor: prune storage compatibility re-export allowances ( #3579 )
...
* refactor: prune storage compatibility re-export allowances
* fix: reject whitespace-padded dot path segments
2026-06-18 21:14:24 +08:00
08371a6e09
refactor: narrow storage compatibility exports ( #3576 )
2026-06-18 18:31:14 +08:00
c098184c49
refactor: clean runtime and test storage boundaries ( #3573 )
...
* refactor: clean runtime observability select boundaries
* refactor: clean test harness fuzz storage boundaries
2026-06-18 18:09:25 +08:00
0e7e2660bb
refactor: clean remaining storage DTO imports ( #3568 )
2026-06-18 14:44:46 +08:00
acdf439371
refactor: clean external DTO consumers ( #3566 )
...
* refactor: clean external DTO consumers
* fix: handle empty erasure shard recovery
2026-06-18 14:22:44 +08:00
e5cad7ed20
refactor: move object operation contracts ( #3559 )
2026-06-18 09:48:13 +08:00
53337a6f71
refactor: move HTTP range helper contracts ( #3533 )
2026-06-18 00:47:23 +08:00
84f7027e90
refactor: add object store resolver for standalone crates ( #3437 )
2026-06-14 18:02:39 +08:00
f7724d223b
feat(rio): rio_v2 is compatible with minio for storing data. ( #3115 )
...
* Set up a compatibility layer for replacing old Rio components with new ones.
* fix(rio). compress range
* feat(rio). Add the experimental feature rio_v2 to support minio data at the binary level.
* feat(rio_v2): add sse-c test
* test compression component
* simple fix
* fix minlz encode
* fix metadata
* fix kms key cache error
* Update launch.json
* ci: set nix crate download user agent
* fix: gate obs pyroscope backend
* ignore minio test
* fix encrypt check
* fix
* fix
* fix
* Update object_usecase.rs
* Update ci.yml
* fix
* ci add rio-v2 test
* fix
* ci fix
* fix
* Reconstructed into a more reasonable compatibility mode
* fix
* fix
---------
Signed-off-by: houseme <housemecn@gmail.com >
Signed-off-by: 唐小鸭 <tangtang1251@qq.com >
Co-authored-by: houseme <housemecn@gmail.com >
Co-authored-by: cxymds <Cxymds@qq.com >
Co-authored-by: 安正超 <anzhengchao@gmail.com >
2026-06-08 11:59:14 +00:00
785d53fce8
fix(s3select): validate scan_range protocol and parquet overlap ( #3176 )
...
* fix(s3select): enforce scan range protocol and parquet overlap
* fix(s3select): select row groups by start offset
---------
Co-authored-by: houseme <housemecn@gmail.com >
2026-06-03 15:57:17 +00:00
b0646be756
feat(s3select): improve SelectObjectContent streaming ( #3072 )
...
* feat(s3select): improve SelectObjectContent streaming
* fix(s3select): reject empty select expressions
* fix(s3select): address streaming review feedback
---------
Co-authored-by: loverustfs <hello@rustfs.com >
2026-05-24 14:23:47 +00:00
59f41eb86a
feat(obs): improve metrics coverage and dashboard performance ( #2682 )
2026-04-25 18:51:29 +00:00
ec8f059506
feat: add timing and metrics for query execution phases ( #2419 )
...
Signed-off-by: Shreyan Jain <shreyan11@duck.com >
Co-authored-by: Shreyan Jain <shreyan11@duck.com >
Co-authored-by: heihutu <heihutu@gmail.com >
Co-authored-by: loverustfs <hello@rustfs.com >
Co-authored-by: cxymds <Cxymds@qq.com >
Co-authored-by: 安正超 <anzhengchao@gmail.com >
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com >
Co-authored-by: houseme <4829346+houseme@users.noreply.github.com >
2026-04-07 20:32:32 +08:00
af46a61fde
build(deps): bump the dependencies group with 6 updates ( #2303 )
...
Signed-off-by: houseme <housemecn@gmail.com >
Co-authored-by: heihutu <heihutu@gmail.com >
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
2026-03-27 12:47:48 +08:00
3c28f0a0ba
feat(metrics): migrate system monitoring from rustfs-obs to rustfs-metrics ( #2242 )
...
Co-authored-by: houseme <housemecn@gmail.com >
2026-03-20 18:52:33 +08:00
f4b523c236
build(deps): bump the dependencies group with 7 updates ( #2069 )
2026-03-04 00:42:03 +08:00
af6c32efac
refactor: improve code quality with safer error handling, trait decomposition, and dead code cleanup ( #1997 )
2026-02-28 01:19:47 +08:00
9d2b8822cf
refactor: stabilize heal format recovery integration tests ( #1984 )
...
Co-authored-by: houseme <housemecn@gmail.com >
2026-02-27 15:26:19 +08:00
17b3054a77
feat(s3select): add JSON handling and flattening for EcObjectStore ( #1930 )
...
Signed-off-by: 0xdx2 <xuedamon2@gmail.com >
Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com >
Co-authored-by: houseme <housemecn@gmail.com >
2026-02-24 18:05:34 +08:00
82a6e78845
Inject GlobalReadiness into HTTP server pipeline and gate traffic until FullReady ( #1255 )
2025-12-25 00:19:03 +08:00
7e75c9b1f5
remove unlinked file ( #1258 )
2025-12-24 23:37:43 +08:00
GatewayJ