* fix(utils): tolerate bavail greater than bfree on Linux
Treat f_bavail > f_bfree as a compatibility edge case instead of fatal corruption during startup disk info probing.
- keep corruption checks for other invalid counter relationships
- log a warning and clamp reserved blocks to 0 when bavail exceeds bfree
- add linux unit coverage for the compatibility and overflow paths
Refs #3025
* fix(utils): cap bavail to bfree for Linux statfs edge
* fix(utils): rate-limit Linux statfs compatibility warning
* test(utils): cover Linux statfs capacity math
Expand Linux statfs capacity math tests around normal, equal, zero, and fully-free block counter combinations.
Refs #3025
* fix(utils): avoid statfs warning guard reallocations
* test(utils): align statfs reserved block expectations
* fmt
* fix(utils): avoid warning guard path allocation
Check the warn-once set before allocating a PathBuf so repeated bavail/bfree compatibility warnings stay low-overhead.
Refs #3025
* chore(deps): bump workspace dependencies
- bump direct dependency versions in Cargo.toml\n- refresh Cargo.lock to resolve updated crates
* fix(deps): avoid pyroscope and pprof linker collision
- downgrade pyroscope from 2.0.6 to 2.0.5\n- keep a single pprof implementation path via pprof-pyroscope-fork\n- fix duplicate perf_signal_handler during test linking
* fix(ecstore): send valid ping body in remote locker
Build ping requests with a flatbuffer payload so health checks remain compatible with the ping response parser after restart.
* fix(bench): use multi-host warp target during failover
Normalize comma-separated warp host lists in run_object_batch_bench and let four-node failover bench pass BENCH_WARP_HOSTS so rolling restart does not pin load to a single restarting node.
* feat(health): add compat health probes with busy/KMS checks
- Add /health/live liveness probe endpoint
- Add busy protection (429) for readiness probes, gated by RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE
- Add KMS readiness check for /health/ready, gated by RUSTFS_HEALTH_COMPAT_KMS_READY_CHECK_ENABLE
- Add lock quorum status caching with TTL to reduce RPC pressure
- Consolidate health response building into build_health_response_parts
- Register /health/live in console router and readiness gate
- Remove MinIO references from newly added health code
* fix(health): decouple kms readiness from lock quorum
* fix(site-replication): refresh peer TLS client cache
- rebuild site-replication peer reqwest client when outbound TLS generation changes\n- keep cached client/failed state per generation for low-overhead requests\n- accept --tls-path as alias of tls inspect --path for operator compatibility\n- add targeted parser/cache tests\n\nRefs #2723
* docs(issue-2723): add private-ca cert rotation retest
- add executable retest checklist for private CA site-replication flows\n- cover tls inspect compatibility (--path and --tls-path)\n- include post-rotation and post-restart pass criteria\n\nRefs #2723
* test(site-replication): isolate global TLS cache test
- serialize generation-sensitive peer TLS cache test\n- snapshot and restore global outbound TLS generation and static cache\n- avoid order-dependent interactions with parallel test execution\n\nRefs #2723
* chore: remove docs file from tracking, keep locally
The docs directory is already in .gitignore but this file was
previously committed. Remove it from git index to prevent docs
content from being included in the PR.
Co-Authored-By: heihutu <heihutu@gmail.com>
* test(site-replication): cover ready client cache hit path
Add a focused unit test for unchanged-generation Ready cache entries in site_replication_peer_client_cache_hit to validate steady-state client reuse behavior highlighted in PR #3109 review feedback.
---------
Co-authored-by: loverustfs <hello@rustfs.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Treat empty ping bodies as liveness probes, add a startup cleanup barrier for early walk_dir calls, and delay immediate background cleanup/capacity timers to reduce transient restart-time VolumeNotFound noise.
Also downgrade expected missing-path producer results during startup from generic errors to warnings while preserving existing storage semantics.
* fix(ecstore): harden issue3031 multipart validation path
- clear stale multipart part destinations before rename fan-out
- add repeated part overwrite regression coverage
- reduce remote disk startup false-fault escalation to suspect-first
- refine remote locker diagnostics and lower scanner leader-lock log noise
- add a dedicated 4-node issue3031 docker validation script
* refactor(admin): inline console version json macro
- drop the unused serde_json::json import in admin console
- call serde_json::json! inline in version_handler
- keep the console version response behavior unchanged
* fix(remote-disk): recover suspect health on probe success
- record probe success during remote disk health checks so suspect drives recover
- use async_with_vars for the remote disk health probe test
- make the missing-listener test assert the state transition more robustly
* fix(lock): isolate retry attempt lock ids
Generate a fresh lock id for each distributed lock retry attempt so late pending cleanup from an earlier attempt cannot release a lock acquired by a later retry.
Also add a regression test covering late-success cleanup versus retried acquisition.
* style(lock): normalize retry race regression test
Normalize the distributed lock retry race regression test formatting without changing behavior.
* test(lock): remove quorum-order assumption
Make the retry race regression test assert the safety property without assuming which delayed client enters the retry quorum first.
* fix(readiness): gate on lock quorum health
Include distributed lock quorum in runtime readiness decisions and expose the signal in health responses.
Map namespace lock quorum failures to ServiceUnavailable instead of InternalError so clients can safely retry while keeping quorum enforcement unchanged.
* fix(ecstore): restore namespace lock error decoding
Add the missing from_u32 arm for NamespaceLockQuorumUnavailable and cover the numeric roundtrip in error tests.
* fix(deploy): simplify systemd service startup
* docs(deploy): add nixos systemd service example
* docs(deploy): add nixos service guide
* chore(deps): update pulsar and pyroscope
* docs(deploy): refine nixos service guidance
* fix(deploy): use explicit rustfs server entrypoint
The service template only handled ClusterIP and NodePort via if/else-if
branches. When service.type=LoadBalancer was set, no branch matched and
the type field was omitted from the rendered Service, causing Kubernetes
to silently default to ClusterIP.
Add the missing LoadBalancer branch with support for:
- loadBalancerIP: request a specific IP (e.g. Cilium LB-IPAM, MetalLB)
- loadBalancerClass: select a specific LB implementation
- loadBalancerSourceRanges: restrict client source IPs
This is particularly useful for bare-metal / on-prem clusters using
software load balancers (Cilium LB-IPAM, MetalLB, kube-vip) where
a stable routable IP is preferable to an Ingress for S3 clients.
Co-authored-by: Claude Sonnet 4.6 <noreply@anthropic.com>
Co-authored-by: houseme <housemecn@gmail.com>
Co-authored-by: 安正超 <anzhengchao@gmail.com>
Co-authored-by: cxymds <Cxymds@qq.com>
Co-authored-by: majinghe <42570491+majinghe@users.noreply.github.com>
* fix(heal): restore single disk data during deep heal
* fix(heal): clarify erasure heal step logging
* fix(heal): avoid duplicate deep object heals
* fix(ecstore): preserve volume-not-found walk errors
---------
Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com>
Co-authored-by: houseme <housemecn@gmail.com>
Object repair rebuilt missing data shards but left missing parity shards unrecreated. A parity-only repair could therefore complete without restoring the shard contents.
Add a repair-specific reconstruction path that regenerates parity after data shards are available, while keeping normal read decoding data-only. Also shut down repair writers after all blocks are written.
Constraint: Preserve read-path decode behavior and limit parity regeneration to repair.
Confidence: high
Scope-risk: moderate
Directive: Do not route read decoding through parity regeneration without measuring the cost.
Tested: cargo test -p rustfs-ecstore decode_data_and_parity -- --nocapture
Tested: cargo test -p rustfs-ecstore heal_reconstructs_missing_parity_shard -- --nocapture
Tested: cargo test -p rustfs-ecstore erasure_coding -- --nocapture
Tested: cargo test -p rustfs heal_object_marks_missing_shard_disk_dirty_for_capacity_manager -- --nocapture
Tested: cargo test -p rustfs-heal -- --nocapture
Tested: cargo clippy -p rustfs-ecstore --all-targets -- -D warnings
Tested: cargo fmt --all --check
Tested: make pre-commit
Not-tested: Live multi-node disk replacement outside local test harness
* Preserve multipart replication recovery state
Multipart uploads previously only scheduled replication after completion, leaving no persisted pending state for scanner recovery if the initial async work was lost. Persist the same pending replication metadata during multipart initialization and let completion evaluate the object metadata that was actually stored.
The scanner heal path also treated ordinary pending objects as delete-replication candidates. Restrict that path to delete markers and version purge state so pending objects remain eligible for object replication heal.
Constraint: Bucket replication recovery depends on persisted object metadata after the async queue is unavailable.
Rejected: Rely only on immediate completion-time scheduling | it cannot recover after process restart or worker loss.
Confidence: high
Scope-risk: moderate
Directive: Keep multipart upload initialization aligned with single PUT replication metadata semantics.
Tested: cargo fmt --all --check
Tested: cargo clippy --workspace --all-features --all-targets -- -D warnings
Tested: LC_ALL=en_US.UTF-8 LANG=en_US.UTF-8 make pre-commit
Tested: Runtime replication outage check confirmed multipart xl.meta stores PENDING status and timestamp.
* fix(replication): preserve version purge scanner state
Role-derived replication configs need target-scoped status strings before scanner heal can build per-target purge status. The duplicated replication-status assignment left version purge status unset, so scanner recovery could lose the target-level purge state.
Constraint: Scanner heal derives per-target purge decisions from version_purge_status_internal.
Rejected: Leave the duplicate as a harmless cleanup | it changes recovery behavior for role-only configs with version purge state.
Confidence: high
Scope-risk: narrow
Directive: Keep role-derived replication and version purge internal status mapping symmetric.
Tested: cargo fmt --all --check
Tested: cargo test -p rustfs-ecstore heal -- --nocapture
---------
Co-authored-by: wly <wlywly0735@126.com>
Co-authored-by: houseme <housemecn@gmail.com>
* perf(ecstore): use direct std writes for local disk
* fix(ecstore): avoid blocking disk writes in local disk path
* fix(ecstore): use tokio async write in local disk path
* chore(ecstore): remove unused AsyncWriteExt import
* fix(ecstore): preserve write semantics and avoid bytes copy
* fix(ecstore): optimize write_all_internal buffer paths
* fix(ecstore): sync write path in local disk
* fix(ecstore): align sync flag docs and avoid ref copy
* chore(rust): format local write path
---------
Co-authored-by: houseme <housemecn@gmail.com>
wood