The /health endpoint (liveness) was returning a hardcoded `ready: true`
in its response body regardless of actual node readiness state. This
caused a semantic contradiction with /health/ready (readiness), which
correctly reported readiness based on storage, IAM, lock quorum, and
peer health.
This led to confusing behavior in Kubernetes deployments where:
- /health returned 200 with `ready: true` (liveness)
- /health/ready returned 503 (readiness)
- Pods remained Running but were removed from Service endpoints
Changes:
- readiness_source_for_probe(Liveness) now returns Node readiness source
- health_check_state() for Liveness reflects actual readiness in body
while keeping HTTP 200 status (process is alive)
- build_health_response_parts() for Liveness now includes dependency
details and degradedReasons when readiness report is available
This ensures the `ready` field in /health body is truthful while
maintaining backward compatibility for liveness probe behavior.
Refs: rustfs/backlog#2011
Co-authored-by: heihutu <heihutu@gmail.com>
* fix(ecstore): fence pool metadata replica updates
* fix(ecstore): block decommission on unsafe pool metadata
* fix(ecstore): block writes after pool metadata save errors
* fix(ecstore): latch pool metadata writes before await
* fix(scanner): bootstrap pristine usage baseline
* fix(replication): tolerate orphaned resync intents at startup
Since #5215 (1.0.0-beta.12) startup reconciles every pending/started
resync intent in resync.bin against the bucket's configured targets and
aborts the whole server when an intent has no matching target ARN. A
resync whose remote target was later removed leaves exactly such an
orphan on disk, so every later start fails with "accepted replication
resync target ... is not configured" regardless of the binary version.
Skip orphaned intents with a warning instead of failing startup; the
resync routine already settles them to ResyncFailed. Cancel the intent
when its remote target is removed so the orphan is not created again.
Fixes#4784
* fix(replication): cancel removed-target resync under the admission lock
Canceling through this node's cached whole-bucket status map could
persist a map that predates another node's admission, erasing that
node's durable restart intent. Reload resync.bin under the bucket
admission lock, publish the fresh map, and only then mark the removed
target's intent canceled. Two-node regression covers the clobber.
* fix(replication): persist resync status via ETag CAS merge
mark_status, the periodic saver, admission, and removed-target
cancellation all persisted their node's cached whole-bucket map, so any
one node's stale cache could resurrect states another node had already
finalized (a canceled intent flipping back to Pending, an admission
vanishing). All resync.bin writers now go through update_resync_status_cas:
load the freshest document with its ETag, apply a per-target mutation
with staleness and canceled-is-terminal guards re-checked against the
persisted entry, and save conditionally, retrying on concurrent writes.
The periodic saver merges per target, letting terminal states and newer
admissions recorded elsewhere win. Cache convergence stays per-target so
locally running resyncs keep their authoritative progress counters.
Regressions: stale_peer_status_write_cannot_resurrect_canceled_intent
(node B's pre-cancel cache marking its own run Started must not revive
node A's canceled intent) plus unit coverage for the periodic-save merge.
* test(ecstore): rename resync test helper off the guarded contract name
fn resync_target is on the architecture guard's reserved list for
crates/replication operation contracts; the merge-test helper now reads
resync_target_state.
* fix(replication): serialize resync status updates
---------
Co-authored-by: overtrue <anzhengchao@gmail.com>
* fix(ecstore): fence pool metadata replica updates
* fix(ecstore): block decommission on unsafe pool metadata
* fix(ecstore): block writes after pool metadata save errors
* fix(ecstore): latch pool metadata writes before await
* test(e2e): activate data usage regressions
* test(e2e): bind data usage selection to Linux listing
* test(e2e): refresh data usage Darwin selection
---------
Co-authored-by: houseme <housemecn@gmail.com>
* fix(ecstore): persist unresolved decommission entries
* fix(ecstore): type decommission completion result
* fix(ecstore): allow intentional decommission listing signatures under strict clippy
The sftp/swift feature-matrix clippy gates run with -D warnings and
flag the unresolved-entry resolver (large Err payload by design, 8
context parameters) and the decommission listing driver (9 args).
Document why and align with the existing decommission_entry precedent.
fix(rustfs): drop sftp-dead scanner capability import and unwired heal_bucket trait method
The sftp feature matrix compiles fewer call sites: the plain
sign_ns_scanner_capability re-export had no remaining user, and
StoragePeerS3ClientExt::heal_bucket was superseded by
heal_bucket_with_fence when #6416 wired the fenced path.
Zhengchao An