Commit Graph

89 Commits

Author SHA1 Message Date
Zhengchao An c41062f278 fix(protocols): constant-time FormPost signature check and guard DLO/SLO range underflow (#4519) 2026-07-09 01:07:36 +08:00
Zhengchao An 87357a0fdd fix(swift): escape static-website listing to prevent stored/reflected XSS (#4515) 2026-07-09 01:07:19 +08:00
Henry Guo 506cd156bb fix(scanner): scope long walk timeouts (#4376)
* fix(scanner): scope long walk timeouts

* fix(scanner): bound IAM config walks

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
2026-07-08 17:06:51 +08:00
Zhengchao An b06f3df6b6 fix: repair main build (swift clippy assert + ecstore multipart test ctx) (#4441)
* fix(swift): replace assert!(false) with panic! in expiration_worker test

clippy::assertions_on_constants fails the swift clippy CI job under -D warnings.

* fix(ecstore): add missing ctx field to multipart lock test store

The multipart list-parts lock test (#4437) constructs ECStore without the
ctx field added by the Phase 5 InstanceContext work (backlog#939). Both
landed on main independently, leaving a semantic conflict that breaks the
ecstore lib-test build (E0063). Adopt the process bootstrap context,
matching the existing bootstrap_ctx() test in store/mod.rs.
2026-07-08 08:01:12 +00:00
Zhengchao An 7cd7c84e71 feat(swift): track object expiration (#4409)
fix(swift): wire object expiration tracking
2026-07-08 14:30:46 +08:00
Zhengchao An 3c3113619e fix(protocols): use constant-time secret comparison in FTPS and WebDAV auth (#4403)
The FTPS and WebDAV authentication handlers compared the client-supplied secret
key against the stored secret with `String::eq`, which short-circuits on the
first differing byte. A network attacker who knows (or enumerates) a valid
access key can recover the secret key byte-by-byte via response-timing analysis;
neither path is rate limited.

Switch both to a constant-time comparison using `subtle::ConstantTimeEq`, the
same primitive the SFTP handler and `rustfs/src/auth.rs::constant_time_eq`
already use. `subtle` is added to the `ftps` and `webdav` feature dependency
sets (it was previously gated on `sftp` only).

Addresses GHSA-3p3x-734c-h5vx.
2026-07-08 09:31:51 +08:00
Zhengchao An 07324e268b fix(protocols): allow clippy::type_complexity in expiration worker tests (#4395)
fix(protocols): allow clippy type_complexity in test mock struct

The MockExpirationObjectBackend test struct uses a nested generic type
that triggers clippy::type_complexity. Add #[allow(clippy::type_complexity)]
since this is test-only code where the type is inherent to the mock design.

Co-authored-by: houseme <housemecn@gmail.com>
2026-07-08 08:01:13 +08:00
Zhengchao An f021e4f321 fix(protocols): simplify swift expiration test types (#4385) 2026-07-08 05:15:30 +08:00
houseme 86940a9452 fix(protocols): resolve clippy type_complexity in swift expiration worker tests (#4393)
fix(protocols): factor swift expiration mock result type into alias

The swift feature clippy matrix on main fails with clippy::type_complexity on the MockExpirationObjectBackend test struct introduced with the expiration worker tests, blocking CI for every open PR. Introduce a MetadataResult type alias in the test module; no behavior change.

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-08 03:29:46 +08:00
Zhengchao An 62a31e4ec4 fix(storage): address pending metadata and health gaps (#4380) 2026-07-08 00:15:07 +08:00
houseme 2247823200 fix(runtime): remove tokio io-uring feature and add regression guard (#4364)
Drop the [target.'cfg(target_os = "linux")'.dependencies] tokio
"io-uring" feature from 6 crates and the rustfs binary. Because
.cargo/config.toml enables --cfg tokio_unstable globally, this feature
switched every Linux build's file I/O onto tokio's io_uring runtime
backend. Restricted Linux environments (Docker default seccomp, gVisor,
proot, old kernels) reject io_uring_setup with EACCES/ENOSYS, which
tokio surfaced as PermissionDenied and RustFS reported as
DiskAccessDenied at startup.

Add scripts/check_no_tokio_io_uring.sh so the feature cannot silently
return: it fails on any tokio dependency line enabling "io-uring", while
still allowing a future application-level io-uring crate dependency.
Wire it into make pre-commit/pre-pr/dev-check and CI.

Tracking: rustfs/backlog#890 (parent rustfs/backlog#897)

Co-authored-by: heihutu <heihutu@gmail.com>
2026-07-07 17:47:59 +08:00
Zhengchao An eb02486574 chore(swift): remove placeholder SSE module (#4330)
security(swift): remove placeholder SSE module (backlog#646)

The Swift `encryption` module was a non-functional stub: `encrypt_data`
returned the plaintext unchanged while labeling it AES-256-GCM in the
object metadata, and `generate_iv` derived the IV from a timestamp
rather than a CSPRNG. It had no production caller (only a `pub mod`
declaration and one integration test), so wiring it in as-is would have
silently shipped plaintext advertised as ciphertext.

We are not supporting Swift server-side encryption for now, so remove
the module outright rather than keep a dangerous stub around:
- delete crates/protocols/src/swift/encryption.rs
- drop `pub mod encryption;` from swift/mod.rs
- remove the encryption case (and unused import) from the swift
  integration test

The module reached main dubiously: it was introduced together with the
whole Swift API in commit 86e93624 ("fix(heal): canonicalize scanner
object-dir repairs (#3864)"), a 1665-file squash whose PR description
only covered the heal change and never mentioned Swift or SSE.

Verified: cargo fmt; cargo test -p rustfs-protocols --features swift
--test swift_simple_integration (10 passed); arch guardrail scripts pass.
2026-07-07 04:29:24 +08:00
houseme 9b69c6d14c fix(s3): preserve metadata listing extensions (#4261)
* chore(deps): update s3s to 0.14.1

* fix(s3): preserve metadata listing extensions

* fix(swift): make version names monotonic

* fix(s3): preserve v1 list pagination markers
2026-07-05 05:03:21 +08:00
houseme 25d80d7c60 feat(storage): harden internode data-path controls (#4224)
* fix(rio): propagate http writer shutdown errors

* fix(ecstore): unify remote lock rpc deadlines

* fix(storage): reject corrupt read multiple payloads

* feat(rio): add internode http tuning profiles

* feat(metrics): add internode baseline signals

* feat(ecstore): observe shard locality topology

* feat(ecstore): gate shard locality scheduling

* feat(ecstore): gate batch read version rpc

* feat(ecstore): observe batch processor adaptation

* feat(ecstore): gate batch processor observation

* docs: add get benchmark regression analysis

* docs: add issue 797 execution plan status

* fix(ecstore): require explicit batch rpc support

* fix(ecstore): honor documented batch read gate

* fix(ecstore): keep batch read gate stable per call

* chore: update workspace dependencies

* feat(ecstore): log batch read gate decisions

* feat(ecstore): count batch read gate decisions

* test(issue-797): add local internode A/B runner

* test(rio): fix tuning profile spelling fixture

* fix(protocols): adapt sftp channel open callbacks

* fix(metrics): wrap batch processor observation args

* chore(docs): keep issue notes local only

* fix(storage): address internode review feedback

* fix(storage): address internode data-path review findings

- Run the BatchReadVersion auto-mode unary fallback outside the batch
  RPC deadline so each read_version keeps its own per-op timeout and
  health accounting instead of racing the whole batch against one
  drive timeout.
- Cap adaptive batch-processor concurrency growth at a hard multiple
  of the configured baseline so sustained fast batches cannot ratchet
  past the configured limit.
- Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE,
  and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading
  the environment on hot paths.
- Skip shard read-cost collection in observe mode when stage metrics
  are disabled, and cache the local endpoint host list instead of
  rebuilding it on every read.
- Allow --warp-extra-args values starting with -- and drop the unused
  warp_hosts_csv helper in the issue-797 A/B runner.

* fix(storage): address internode data-path review findings

- Run the BatchReadVersion auto-mode unary fallback outside the batch
  RPC deadline so each read_version keeps its own per-op timeout and
  health accounting instead of racing the whole batch against one
  drive timeout.
- Cap adaptive batch-processor concurrency growth at a hard multiple
  of the configured baseline so sustained fast batches cannot ratchet
  past the configured limit.
- Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE,
  and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading
  the environment on hot paths.
- Skip shard read-cost collection in observe mode when stage metrics
  are disabled, and cache the local endpoint host list instead of
  rebuilding it on every read.
- Allow --warp-extra-args values starting with -- and drop the unused
  warp_hosts_csv helper in the issue-797 A/B runner.

Co-Authored-By: heihutu<heihutu@gmail.com>

* fix(storage): align buffer clamp test with media cap

* fix(ecstore): release optimized read locks before streaming

---------

Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
2026-07-03 17:08:15 +08:00
Zhengchao An 710ae74cde perf: add S3 operations benchmark framework (#738) (#4005) 2026-06-28 18:02:41 +08:00
Zhengchao An 05d201679c fix: replace unwrap() with expect() in more files (#729 batch 13) (#3993) 2026-06-28 11:45:13 +08:00
Zhengchao An f0ab812213 fix: replace unwrap() with expect() in remaining files (#729 batch 12) (#3992) 2026-06-28 11:45:03 +08:00
Zhengchao An 1b3dea012e refactor: route ecstore runtime globals through facade (#3941) 2026-06-27 12:27:03 +08:00
Zhengchao An 72ae43cb90 refactor: segment external storage contract imports (#3908) 2026-06-26 16:44:59 +08:00
Zhengchao An 1f7e159388 refactor: segment external storage api boundaries (#3903) 2026-06-26 15:10:49 +08:00
Zhengchao An c9614eb7cb refactor: route ecstore storage api boundaries (#3892) 2026-06-26 09:35:18 +08:00
Zhengchao An e37e367390 refactor: route remaining external storage boundaries (#3889) 2026-06-26 06:22:51 +08:00
Zhengchao An 70a2441407 refactor: route notify dispatch through app context (#3789)
* refactor: route notify dispatch through app context

* refactor: route admin IAM globals through app context (#3791)

* refactor: centralize IAM root credential access (#3792)
2026-06-23 20:05:28 +08:00
Zhengchao An 198fd4f150 refactor(runtime): route RustFS runtime consumers through storage owner (#3756) 2026-06-23 05:17:56 +08:00
Zhengchao An db7ff8f513 refactor: expose external ECStore owner symbols (#3748) 2026-06-22 21:42:54 +08:00
Zhengchao An 88a87b3e8f refactor: centralize external ECStore facade aliases (#3746) 2026-06-22 20:37:13 +08:00
Zhengchao An 539c68778d refactor: remove standalone compat bridge modules (#3740) 2026-06-22 17:35:58 +08:00
安正超 de158743c3 refactor: split compat facade imports (#3716) 2026-06-22 09:53:22 +08:00
安正超 d39815470e refactor: prune consumer storage compat paths (#3704) 2026-06-22 02:31:30 +08:00
安正超 0985225448 refactor: consolidate ecstore public facade (#3678)
* refactor: expand ecstore compatibility facade

* test: enforce ecstore api facade imports

* refactor: hide legacy ecstore layout modules

* refactor: hide ecstore facade root modules
2026-06-21 09:09:11 +08:00
安正超 9b3fda9e30 refactor: route ecstore public surfaces through api (#3673) 2026-06-21 05:23:23 +08:00
安正超 afe46c01e7 refactor: prune consumer ecstore object aliases (#3622) 2026-06-19 21:33:00 +08:00
安正超 54ffbedbc8 refactor: remove ecstore operation compat facades (#3608) 2026-06-19 17:01:12 +08:00
安正超 b14e49e84e refactor: narrow IAM and Swift compatibility surfaces (#3587)
* refactor: narrow IAM and Swift compatibility surfaces

* refactor: narrow heal and scanner compatibility surfaces (#3588)

* refactor: narrow RustFS runtime compatibility surfaces (#3591)
2026-06-19 03:06:57 +08:00
安正超 205f964dc5 refactor: tighten storage compat store_api aliases (#3582)
* refactor: collapse storage compat store_api modules

* chore: guard storage compat store_api aliases
2026-06-18 22:50:02 +08:00
安正超 c28fee0013 refactor: continue storage api contract cleanup (#3580)
* refactor: move delete object contracts to storage api

* refactor: narrow store api compatibility exports

* refactor: route table catalog test through storage compat
2026-06-18 22:42:02 +08:00
安正超 7b0cb9e725 refactor: prune storage compatibility re-export allowances (#3579)
* refactor: prune storage compatibility re-export allowances

* fix: reject whitespace-padded dot path segments
2026-06-18 21:14:24 +08:00
安正超 f3fcdd4ba2 refactor: narrow remaining storage compatibility exports (#3578) 2026-06-18 19:18:32 +08:00
安正超 c76e4dc6bc refactor: clean Swift ECStore boundaries (#3570) 2026-06-18 15:18:55 +08:00
安正超 0e7e2660bb refactor: clean remaining storage DTO imports (#3568) 2026-06-18 14:44:46 +08:00
安正超 acdf439371 refactor: clean external DTO consumers (#3566)
* refactor: clean external DTO consumers

* fix: handle empty erasure shard recovery
2026-06-18 14:22:44 +08:00
安正超 e5cad7ed20 refactor: move object operation contracts (#3559) 2026-06-18 09:48:13 +08:00
安正超 3fb4cb3d65 refactor: move list operations contract (#3550) 2026-06-18 08:48:18 +08:00
安正超 53337a6f71 refactor: move HTTP range helper contracts (#3533) 2026-06-18 00:47:23 +08:00
houseme 841f86fe90 test(property): expand parser invariants (#3534)
* test(property): harden parser invariants for #3522

* test(property): satisfy clippy for #3522
2026-06-17 23:52:30 +08:00
安正超 ed55857bca refactor: move bucket operations contract (#3507) 2026-06-17 09:10:38 +08:00
houseme efaf07d323 feat: preserve request ids across async recovery logs (#3451)
* feat(obs): promote request ids in structured logs

* refactor(tracing): propagate spans into request tasks

* test(ecstore): baseline recovery monitor log chains

* fix(replication): reduce startup resync log noise

* chore(docs): stop tracking local recovery baseline

* chore(obs): polish request id logging cleanup
2026-06-14 23:16:04 +08:00
安正超 84f7027e90 refactor: add object store resolver for standalone crates (#3437) 2026-06-14 18:02:39 +08:00
houseme e8012bd1ba refactor(logging): normalize admin telemetry and error messages (#3430) 2026-06-14 13:27:10 +08:00
houseme efa89a98ed refactor(logging): standardize protocol and observability events (#3419)
* refactor(logging): standardize object capacity events

* refactor(logging): standardize protocol server events

* refactor(logging): standardize swift protocol events

* refactor(logging): standardize observability events

* refactor(logging): move masking helper and extend guardrails
2026-06-14 07:14:45 +08:00