gsadmin/rustfs
1
0
Fork 0
mirror of https://github.com/rustfs/rustfs.git synced 2026-08-28 07:57:01 +00:00
Code Issues Packages Projects Releases Wiki Activity
5,905 Commits 40 Branches 127 Tags
f51b06f0ae99ed5b9e3d6cabad87df221533a260
Commit Graph

5905 Commits

This Branch
This Branch
All Branches
Author SHA1 Message Date
Zhengchao An d63ca1f5f5 test(e2e): require exact retention errors (#6535) 2026-08-25 04:31:29 +08:00
houseme 4d43f1ea8a perf: optimize cgroup resource detection with single System instance (#6550) 
* perf: optimize cgroup resource detection with single System instance

Consolidate two sysinfo::System instantiations into one for CPU and
memory detection. Pre-compute the metrics basis string ("cgroup"/"host")
in ContainerResources to avoid per-snapshot String allocations in the
memory observability hot path.

Co-Authored-By: heihutu <heihutu@gmail.com>

* style: apply cargo fmt formatting

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
 2026-08-24 20:07:02 +00:00
Zhengchao An 5c6e1abe7e feat(connect): persist sanitized inventory snapshot (#6537) 
* feat(connect): persist sanitized inventory snapshot

* fix(connect): harden inventory persistence boundary

* fix(connect): harden inventory path anchors

* fix(connect): fail closed outside Linux

* fix(connect): gate inventory persistence to Linux

* fix(connect): keep runtime failure codes stable

* fix(connect): satisfy cross-platform lint

* fix(connect): preserve inventory persistence invariants

* fix(connect): preserve newer local inventory

* fix(connect): retain legacy inventory capture age

* chore(connect): document unsafe boundaries

* test(connect): secure inventory state fixtures

* fix(connect): preserve heartbeat runtime status
 2026-08-25 03:23:02 +08:00
houseme 3b0a28dd9b fix(memory): cgroup-aware resource detection for container environments (#6536) 
* fix(iam): raise recursion limit for migration test

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(memory): cgroup-aware resource detection for container environments

Issue #5803 reported memory RSS regression since beta.9:
- RSS memory steps ~+300 MiB on tiny S3 bursts and never returns
- Daily OOMKills in 1 GiB containers
- Root cause: RustFS uses host memory/CPU instead of container cgroup limits

Changes:
- Add cgroup_resources.rs: cgroup v1/v2 CPU and memory detection
- Add container_config.rs: container configuration with env overrides
- Fix memory_observability.rs: use effective memory (cgroup-aware)
- Fix server/runtime.rs: use cgroup-aware CPU detection for Tokio
- Cap max_blocking_threads to 256 for small containers (<=4 cores)
- Add new metrics: rustfs_memory_effective_total_bytes, rustfs_cgroup_*
- Add startup logging of detected container resources

New environment variables:
- RUSTFS_DISABLE_CGROUP_DETECTION: disable cgroup detection
- RUSTFS_OVERRIDE_CPU_CORES: override detected CPU cores
- RUSTFS_OVERRIDE_MEMORY_BYTES: override detected memory limit

Fixes: rustfs/rustfs#5803
Tracking: rustfs/backlog#2012

Co-Authored-By: heihutu <heihutu@gmail.com>

* style: apply cargo fmt formatting

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix: cross-platform compatibility for cgroup detection

- Move CHANGES_SUMMARY.md and FINAL_SUMMARY.md to docs/operations/
- Add platform-specific cgroup detection (Linux only)
- Non-Linux platforms (macOS, Windows) fall back to host values
- Add platform-specific tests for cgroup detection
- Remove unused imports for non-Linux builds

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix: clippy warnings for cgroup_resources

- Remove unused import super::CgroupResources
- Use derive(Default) instead of manual impl
- Remove redundant trim() before split_whitespace()
- Fix absurd_extreme_comparisons (quota <= 0 for u64)
- Use div_ceil() instead of manual implementation

Co-Authored-By: heihutu <heihutu@gmail.com>

* refactor: consolidate cgroup detection into single module

- Merge cgroup_resources.rs and container_config.rs into unified module
- Remove duplicate test file cgroup_resources_test.rs
- Remove redundant CHANGES_SUMMARY.md and FINAL_SUMMARY.md
- Simplify memory_observability.rs to use unified API
- Simplify server/runtime.rs to use unified API
- All cgroup detection logic now in single source of truth
- Environment variable overrides integrated into main module
- Clippy and fmt clean

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Co-authored-by: heihutu <heihutu@gmail.com>
 2026-08-25 00:45:30 +08:00
Zhengchao An 7be0d56be8 fix(storage): stabilize nextest regressions (#6543) 2026-08-24 23:34:18 +08:00
Zhengchao An dcdaa37b84 fix(e2e): box delete object errors (#6534) 2026-08-24 22:24:37 +08:00
Zhengchao An ea07c781c4 test(ilm): isolate suspended restore stack (#6533) 2026-08-24 22:08:55 +08:00
Zhengchao An 7a7871ca67 test(e2e): require exact object lock errors (#6532) 2026-08-24 22:08:07 +08:00
Henry Guo c80d970d58 fix(table-catalog): classify storage quorum as unavailable (#6531) 
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
 2026-08-24 21:52:53 +08:00
Zhengchao An 1f40c3ecd0 test(e2e): require remaining 404 absence oracles (#6530) 2026-08-24 21:25:27 +08:00
Zhengchao An 102fb767ce test(ci): stabilize Connect and health fixtures (#6529) 2026-08-24 21:10:33 +08:00
Zhengchao An afb2e4f728 fix(multipart): enforce complete part number limit (#6528) 2026-08-24 21:03:13 +08:00
唐小鸭 3d75e7b51f fix(ecstore): heap-allocate durable ILM receipt futures (#6527) 
PR #6369 awaits record_durable_ilm_decommission_progress/terminal inline
from save/delete_transition_transaction_record. Their state machines are
large and sit on the already-deep transition worker poll chain
(worker -> transition -> transaction record -> delete_config -> full
store delete fanout), which overflowed the default 2 MiB tokio worker
stack in debug builds: app::lifecycle_transition_api_test::
compensation_driven_complete_multipart_upload_still_transitions died
with SIGABRT in under a second (first-bad commit via git bisect
1.0.0-rc.3..1ec1a8d90: 34bbc1adb, #6369).

41546dee5 already unblocked the test by moving it onto a dedicated
32 MiB thread; this change removes the underlying stack growth so every
caller of the transaction-record helpers keeps its previous headroom.
With it, the test also passes on a plain 2 MiB tokio worker.
 2026-08-24 20:38:46 +08:00
Zhengchao An e4dfc6f45b fix(quota): release reconciled delete holds (#6526) 2026-08-24 20:34:42 +08:00
Zhengchao An c1c6a1e23f fix(storage): stabilize main regressions (#6525) 2026-08-24 20:26:03 +08:00
houseme a8be4f2695 fix(iam): raise recursion limit for migration test (#6524) 
Co-authored-by: heihutu <heihutu@gmail.com>
 2026-08-24 20:23:21 +08:00
Zhengchao An a1ebe9a3b3 test(ci): stabilize main fixture paths (#6523) 2026-08-24 19:34:02 +08:00
Zhengchao An e2193cc42c fix(ecstore): enforce monotonic transition cursors (#6522) 2026-08-24 19:33:58 +08:00
Zhengchao An 6f14a79089 fix(quota): reconcile matching scanner usage (#6521) 2026-08-24 19:33:53 +08:00
houseme de9e8faa27 fix(health): reflect node readiness in /health response body (#6520) 
The /health endpoint (liveness) was returning a hardcoded `ready: true`
in its response body regardless of actual node readiness state. This
caused a semantic contradiction with /health/ready (readiness), which
correctly reported readiness based on storage, IAM, lock quorum, and
peer health.

This led to confusing behavior in Kubernetes deployments where:
- /health returned 200 with `ready: true` (liveness)
- /health/ready returned 503 (readiness)
- Pods remained Running but were removed from Service endpoints

Changes:
- readiness_source_for_probe(Liveness) now returns Node readiness source
- health_check_state() for Liveness reflects actual readiness in body
  while keeping HTTP 200 status (process is alive)
- build_health_response_parts() for Liveness now includes dependency
  details and degradedReasons when readiness report is available

This ensures the `ready` field in /health body is truthful while
maintaining backward compatibility for liveness probe behavior.

Refs: rustfs/backlog#2011

Co-authored-by: heihutu <heihutu@gmail.com>
 2026-08-24 18:51:59 +08:00
houseme 2251f22c1a fix(test-utils): raise recursion limit for lib tests (#6518) 
Co-authored-by: heihutu <heihutu@gmail.com>
 2026-08-24 18:41:49 +08:00
Zhengchao An 70a6a9e8dc fix(ci): repair main test regressions (#6519) 2026-08-24 18:41:26 +08:00
Zhengchao An fe453b7f5b test(iam): create legacy migration bucket through store (#6517) 2026-08-24 18:33:08 +08:00
Zhengchao An ade7e320da test(scanner): align pristine startup fixtures (#6516) 2026-08-24 18:29:47 +08:00
Zhengchao An f6d69ce643 test(connect): use protected home for bootstrap fixtures (#6515) 2026-08-24 18:21:53 +08:00
houseme 762431c0c7 fix(tests): raise recursion limit for rustfs e2e crates (#6513) 
Co-authored-by: heihutu <heihutu@gmail.com>
 2026-08-24 18:12:28 +08:00
RustFS 1fb5d5a19d docs: remove ROSS Index badge from README (#6514) 
Drop the expired Q4 2025 Runa Capital badge from the English and Chinese project READMEs.

Co-authored-by: Cursor <cursoragent@cursor.com>
 2026-08-24 18:04:46 +08:00
Zhengchao An 8bd6d8c4db fix(build): restore mimalloc workspace dependency 2026-08-24 17:26:54 +08:00
Zhengchao An 8f2b91f79b fix(ci): refresh e2e full selection 2026-08-24 17:25:45 +08:00
cxymds c2b2b4ffd4 fix(ci): repair post-merge test gates 2026-08-24 17:01:29 +08:00
Zhengchao An 4ceed58be4 docs(operations): document rebalance impact assessment 2026-08-24 17:00:19 +08:00
Zhengchao An 507faf3a6a fix(ci): restore post-merge test gates 2026-08-24 16:42:03 +08:00
Henry Guo 1607e9a376 fix(table-catalog): return 503 when commit authority is unavailable 2026-08-24 16:12:57 +08:00
Henry Guo f06b004f2d fix(scanner): clarify follower status 2026-08-24 16:12:29 +08:00
cxymds 41546dee5d test(ilm): isolate multipart compensation stack 2026-08-24 16:11:29 +08:00
Zhengchao An 7d3f5545e7 fix(ci): repair post-merge build gates 2026-08-24 16:10:24 +08:00
Zhengchao An d293ed71e5 test(e2e): require authorization denial codes (#6497) 2026-08-24 14:35:28 +08:00
houseme 114bb4acec perf(ecstore): add bucket existence cache and allocator feature flags (#6496) 
## Bucket existence cache
- Add BucketExistenceCache in crates/ecstore/src/disk/fs.rs
- Cache bucket directory existence checks with 60s TTL
- Replace access() calls with cached_access() in local.rs
- Add invalidate_bucket_cache() for cache invalidation on create/delete
- Reduces statx syscalls by 89% (from 10,716/s to 1,186/s)

## Allocator feature flags
- Add mimalloc and jemalloc features to rustfs/Cargo.toml
- Default: system allocator (Rust built-in)
- --features mimalloc: mimalloc allocator
- --features jemalloc: jemalloc allocator
- Allows A/B testing different allocators

## Performance impact
- 1KiB PUT: 861 obj/s (unchanged, futex is main bottleneck)
- statx reduction: 89% (from 10,716/s to 1,186/s)
- Main bottleneck remains mimalloc internal synchronization

Ref: rustfs/backlog#2005
Ref: microsoft/mimalloc#1372

Co-authored-by: hector <hetor@rustfs.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: overtrue <anzhengchao@gmail.com>
 2026-08-24 14:35:04 +08:00
Zhengchao An 29272480bd test(e2e): refresh Linux full-suite selection (#6495) 2026-08-24 14:32:19 +08:00
Zhengchao An e136e95a20 test(e2e): fail closed on missing socket oracle (#6493) 2026-08-24 14:32:11 +08:00
Zhengchao An f4ce1a8b3a test(e2e): fail closed on compression disk probes (#6492) 2026-08-24 14:32:02 +08:00
Zhengchao An 9681f19bec test(ci): require dependency-aware readiness (#6491) 
* test(e2e): fail closed on runner readiness

* test(ci): require dependency-aware readiness
 2026-08-24 14:31:38 +08:00
Zhengchao An 20a9c12f86 test(e2e): fail closed on runner readiness (#6490) 2026-08-24 14:31:21 +08:00
Zhengchao An 86e969f63c fix(ecstore): complete rename tails after quorum ack (#6489) 2026-08-24 14:30:50 +08:00
Zhengchao An 51272d34dd fix(app): restore stable Rust 1.98 builds (#6487) 
* chore(scanner): narrow s3s DTO references

* fix(app): scope usage overlay import to tests

* fix(app): bound object-lock lookup future

* fix(log-analyzer): follow decommission migration logs
 2026-08-24 14:30:25 +08:00
Zhengchao An cd1363d519 fix(scanner): restore s3s footprint baseline (#6486) 
chore(scanner): narrow s3s DTO references
 2026-08-24 14:30:16 +08:00
Zhengchao An 5142775387 test(e2e): require 404 absence oracles (#6485) 2026-08-24 14:30:05 +08:00
dependabot[bot] 2ed08c8bad chore(deps): bump p256 from 0.13.2 to 0.14.0 in the dependencies group (#6481) 
* chore(deps): bump p256 from 0.13.2 to 0.14.0 in the dependencies group

Bumps the dependencies group with 1 update: [p256](https://github.com/RustCrypto/elliptic-curves).


Updates `p256` from 0.13.2 to 0.14.0
- [Commits](https://github.com/RustCrypto/elliptic-curves/compare/p256/v0.13.2...p256/v0.14.0)

---
updated-dependencies:
- dependency-name: p256
  dependency-version: 0.14.0
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* update crate version and remove rustfs-mimalloc-sys crate

* fix(connect): adapt p256 signing APIs

Use the p256 0.14 Generate trait for device key generation and update low-S normalization calls for ecdsa 0.17.

Remove an unused object usecase import so warning-deny builds stay clean.

Co-Authored-By: heihutu <heihutu@gmail.com>

* fix(connect): update p256 canonical signature checks

Co-Authored-By: heihutu <heihutu@gmail.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
 2026-08-24 14:29:56 +08:00
Henry Guo 60a0b1d6e7 fix(ecstore): avoid nested prefix listing amplification (#6473) 
* fix(ecstore): avoid nested prefix listing probes

* fix(app): scope usage overlay import to tests

---------

Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
 2026-08-24 14:29:35 +08:00
Zhengchao An 170a4c7640 fix(scanner): bootstrap pristine usage baseline (#6471) 
* fix(ecstore): fence pool metadata replica updates

* fix(ecstore): block decommission on unsafe pool metadata

* fix(ecstore): block writes after pool metadata save errors

* fix(ecstore): latch pool metadata writes before await

* fix(scanner): bootstrap pristine usage baseline
 2026-08-24 14:29:09 +08:00