* feat(cache): add object data cache engine
* feat(cache): wire app-layer object cache flow
* refactor(cache): streamline app-layer cache flow
* refactor(cache): tighten cache flow internals
* refactor: address final clippy cleanup
* chore(deps): update quick-xml to 0.41.0
* feat(cache): wire object data cache env config
* fix(cache): gate materialize fill by cache plan
* chore(cache): add object data cache benchmark gate
* fix(cache): guard object cache fill size mismatches
* refactor(cache): streamline object cache body planning
* fix(cache): align object cache rollout config
* test(cache): cover buffered object cache benchmark
* test(cache): isolate object cache benchmark metrics
* test(cache): mark materialize rollout experimental
* test(cache): tighten object cache benchmark gate
* fix(cache): address review findings for object data cache
- singleflight: clean up leader entry on cancellation (Drop impl) so a dropped GET future can no longer wedge all subsequent fills for the same key; switch the fill map to a std Mutex and add a regression test
- adapter: honor RUSTFS_OBJECT_DATA_CACHE_ENABLE=true by defaulting to hit_only when no explicit mode is set (explicit mode still wins)
- planner: treat nil version UUIDs as "no value" per repo convention so unversioned objects key under the canonical "null" instead of fragmenting the key space
- multipart: invalidate the object cache on the quota-exceeded rollback delete after complete-multipart, closing a stale-cache window
- layering: move the disabled-cache fallback into app::context and drop the new infra->app layer-dependency baseline entry
* fix(cache): close invalidation races and drop full-cache scan on writes
- index: make identity-index insert/remove/prune atomic via starshard compute_if_present/compute_if_absent so concurrent fills can no longer drop each other's keys (lost keys made entries unreachable to invalidation until TTL); add a concurrency regression test
- fill: register the key in the identity index before the entry becomes visible in the cache and re-check the index afterwards, undoing the fill when an invalidation raced in between (new skipped_invalidation_race fill result)
- invalidate: with the index now authoritative, remove the full-cache iter() fallback that made every PUT/DELETE of a never-cached object O(total cache entries) (two scans per PUT, 2N per batch delete)
- materialize-fill: fail the GET instead of falling back to the partially consumed stream after a mid-read error (the fallback would send a body missing its prefix under a full-length Content-Length), and log the same size-mismatch warning as the sibling buffering paths
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(storage): fix media-dependent buffer clamp expectation
test_concurrency_manager_multi_factor_strategy_buffer_clamp asserted media_cap.min(MI_B), but the implementation's final safety clamp is [32KiB, media_cap.max(MI_B)] — deliberately so a media cap above 1MiB (NVMe's 2MiB default) stays effective. The test only passed on machines detected as SSD/Unknown (cap == 1MiB) and failed on NVMe-backed CI runners with 2MiB != 1MiB. Assert the media cap itself, which is what the strategy actually guarantees on every environment.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(storage): format buffer clamp assertion
* chore(logging): update tier guardrail path
---------
Signed-off-by: houseme <housemecn@gmail.com>
Co-authored-by: cxymds <cxymds@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(rio): propagate http writer shutdown errors
* fix(ecstore): unify remote lock rpc deadlines
* fix(storage): reject corrupt read multiple payloads
* feat(rio): add internode http tuning profiles
* feat(metrics): add internode baseline signals
* feat(ecstore): observe shard locality topology
* feat(ecstore): gate shard locality scheduling
* feat(ecstore): gate batch read version rpc
* feat(ecstore): observe batch processor adaptation
* feat(ecstore): gate batch processor observation
* docs: add get benchmark regression analysis
* docs: add issue 797 execution plan status
* fix(ecstore): require explicit batch rpc support
* fix(ecstore): honor documented batch read gate
* fix(ecstore): keep batch read gate stable per call
* chore: update workspace dependencies
* feat(ecstore): log batch read gate decisions
* feat(ecstore): count batch read gate decisions
* test(issue-797): add local internode A/B runner
* test(rio): fix tuning profile spelling fixture
* fix(protocols): adapt sftp channel open callbacks
* fix(metrics): wrap batch processor observation args
* chore(docs): keep issue notes local only
* fix(storage): address internode review feedback
* fix(storage): address internode data-path review findings
- Run the BatchReadVersion auto-mode unary fallback outside the batch
RPC deadline so each read_version keeps its own per-op timeout and
health accounting instead of racing the whole batch against one
drive timeout.
- Cap adaptive batch-processor concurrency growth at a hard multiple
of the configured baseline so sustained fast batches cannot ratchet
past the configured limit.
- Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE,
and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading
the environment on hot paths.
- Skip shard read-cost collection in observe mode when stage metrics
are disabled, and cache the local endpoint host list instead of
rebuilding it on every read.
- Allow --warp-extra-args values starting with -- and drop the unused
warp_hosts_csv helper in the issue-797 A/B runner.
* fix(storage): address internode data-path review findings
- Run the BatchReadVersion auto-mode unary fallback outside the batch
RPC deadline so each read_version keeps its own per-op timeout and
health accounting instead of racing the whole batch against one
drive timeout.
- Cap adaptive batch-processor concurrency growth at a hard multiple
of the configured baseline so sustained fast batches cannot ratchet
past the configured limit.
- Parse RUSTFS_INTERNODE_HTTP_* tuning, RUSTFS_BATCH_PROCESSOR_ADAPTIVE,
and RUSTFS_METADATA_BATCH_READ once per process instead of re-reading
the environment on hot paths.
- Skip shard read-cost collection in observe mode when stage metrics
are disabled, and cache the local endpoint host list instead of
rebuilding it on every read.
- Allow --warp-extra-args values starting with -- and drop the unused
warp_hosts_csv helper in the issue-797 A/B runner.
Co-Authored-By: heihutu<heihutu@gmail.com>
* fix(storage): align buffer clamp test with media cap
* fix(ecstore): release optimized read locks before streaming
---------
Co-authored-by: Zhengchao An <anzhengchao@gmail.com>
Speeds up the CI pipeline without changing what is tested:
- Drop the global CARGO_BUILD_JOBS=2 and per-job --jobs 2 flags, which
halved compiler parallelism on the 4-core runners.
- Stop embedding hashFiles(Cargo.lock) in rust-cache shared keys. The
action already fingerprints lockfiles internally; putting the hash in
the key prefix meant any PR that touched Cargo.lock started from a
completely cold cache instead of reusing unchanged dependencies.
- Give the e2e-tests job the full setup action with dependency caching.
Its migration-proof step compiles the e2e_test crate (which pulls in
most of the workspace) and previously did so cold on every run with
no cache, on a 2-core runner.
- Set profile.dev debug = "line-tables-only": keeps usable backtraces
while cutting compile/link time, target-dir size (faster cache
save/restore), and debug-binary artifact upload/download.
- Split fmt + repo-script checks into a compile-free quick-checks job
on ubuntu-latest so contributors get feedback in ~1 minute instead
of after the full test run.
- Collapse the five per-filter migration-proof cargo test reruns into
one filtered nextest invocation; the same tests already run in the
full nextest pass.
- Remove the touch rustfs/build.rs + forced rebuild from the debug
binary jobs (version stamping is irrelevant for e2e binaries) and
the unreachable Windows cross-compile branch in build.yml.
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>
* docs: document crypto RC version dependencies
Add comment explaining why aes-gcm and chacha20poly1305 use RC
versions and the migration path when stable versions are released.
Refs #732
* fix: remove all backlog links from code and comments
test: add insta snapshot test for storage error display format
Add snapshot test to detect unexpected changes in StorageError
display format. This catches output format regressions that
traditional assert tests might miss.
Refs #740
* feat(replication): support insecure https bucket targets
* feat(replication): support custom ca bucket targets
* test(replication): satisfy e2e clippy for TLS helpers
* fix(replication): avoid native root panic for custom trust stores
* test(replication): decouple private IP target test from TLS roots
* test(replication): use target TLS client in private IP unit test
* build(deps): bump brotli from 8.0.3 to 8.0.4 in the dependencies group
Bumps the dependencies group with 1 update: [brotli](https://github.com/dropbox/rust-brotli).
Updates `brotli` from 8.0.3 to 8.0.4
- [Release notes](https://github.com/dropbox/rust-brotli/releases)
- [Commits](https://github.com/dropbox/rust-brotli/compare/8.0.3...8.0.4)
---
updated-dependencies:
- dependency-name: brotli
dependency-version: 8.0.4
dependency-type: direct:production
update-type: version-update:semver-patch
dependency-group: dependencies
...
Signed-off-by: dependabot[bot] <support@github.com>
* build(deps): bump s3s from cf4c3346 to c59b62f7ba in the dependencies group
---------
Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
* chore(deps): bump workspace dependencies
- bump direct dependency versions in Cargo.toml\n- refresh Cargo.lock to resolve updated crates
* fix(deps): avoid pyroscope and pprof linker collision
- downgrade pyroscope from 2.0.6 to 2.0.5\n- keep a single pprof implementation path via pprof-pyroscope-fork\n- fix duplicate perf_signal_handler during test linking
* fix(deploy): simplify systemd service startup
* docs(deploy): add nixos systemd service example
* docs(deploy): add nixos service guide
* chore(deps): update pulsar and pyroscope
* docs(deploy): refine nixos service guidance
* fix(deploy): use explicit rustfs server entrypoint
* chore(deps): refresh workspace deps and linux fs_type gating
- refresh workspace dependency pins and lockfile updates
- remove now-unused crate dependency entries in multiple Cargo.toml files
- enable profiling export defaults in config and scripts/run.sh
- gate os::fs_type module/function/tests to Linux to avoid non-Linux dead_code warnings
* fix(utils): simplify fs_type linux gating
- keep fs_type module-level linux cfg in os::mod
- remove redundant linux cfg on get_fs_type and test module
* chore(deps): bump s3s git revision
- update workspace s3s dependency to rev 507e1312b211c3ddc214b03875d6fabd15d22ed5
- refresh Cargo.lock source entry for s3s
* chore(dev): allow mysql_async git source and env overrides
- allow mysql_async git source in deny.toml allow-git list
- make scripts/run.sh core env vars overrideable via existing shell env
* fix(utils): import get_fs_type in fs_type tests
- add explicit super::get_fs_type import in fs_type test module
- fix Linux E0425 unresolved function errors in unit tests
* chore(dev): tune run script observability defaults
- make profiling export env overrideable in scripts/run.sh
- set RUSTFS_OBS_SAMPLE_RATIO default from 2.0 to 1.0
- update allow-git review window comments in deny.toml
* test(obs): stabilize profiling env alias tests
houseme