* test(e2e): use the resumed decommission completion budget
* test(heal): exercise checkpoints written by rc.5
* test(heal): route upgrade fixtures through the storage facade
* fix(deps): update ctutils to address constant-time advisory
* test(lifecycle): gate recovery I/O while advancing test time
* fix(decommission): count settled entries once and back off repeated pauses
Follow-up to rustfs/rustfs#8824 for rustfs/backlog#2873 defect F.
- Count an entry's versions when the entry settles (copied and cleaned up,
or retained) instead of as each version copies. A partially moved entry
that is deferred, paused and resumed, or interrupted, no longer counts its
already-moved versions again when the resumed pass finds them on the
target.
- Hold the watchdog relaunch of a decommission that pauses again on the
same cause without progress, doubling from 30 s up to 15 min, and warn with
the cause. A first pause still relaunches on the next pass; the pause never
becomes a terminal failure.
- Log each drive's error when a delete version loses write quorum, so the
cause of a sub-quorum delete-marker copy is recorded instead of only its
rollback.
- Tests: pause/resume counting, a target whose metadata is below read
quorum, the copy-quorum pause cause, the relaunch backoff, and a rollback
test that asserts the restore attempt instead of a tracing log.
* fix(decommission): measure pause progress by cleaned entries
Review follow-up on the pause/resume counting and relaunch backoff.
- A pause now carries this generation's progress counters into the
pool metadata it persists, like a progress checkpoint. Before, everything
counted since the last checkpoint was dropped at the pause.
- A retained entry (one whose source stays) is relisted by every resumed
pass. A per-instance ledger counts its versions once per decommission
generation, so retained versions no longer grow per round in one process.
- The relaunch backoff measures progress by entries moved off the pool and
completed buckets, which a relisted retained entry cannot advance. Before,
the recounted retained entry looked like progress and reset the backoff.
- Tests: exact totals across a pause and a resume from persisted metadata;
a target kept below quorum defers the entry with nothing counted; a
retained entry plus a repeated cause escalates the backoff over three
pauses, with the watchdog path relaunching a first pause and holding a
repeat.
* fix(ecstore): bound retained decommission accounting
* feat(scanner): configure startup recovery backoff cap
* test(scanner): compare recovery retry count without narrowing
* docs(scanner): document recovery backoff cap and expose it in Helm
List scanner.recovery_backoff_cap in the runtime controls table, drop the
fixed ten-minute wording from the follower recovery section, and map
config.rustfs.scanner.recovery_backoff_cap_secs to
RUSTFS_SCANNER_RECOVERY_BACKOFF_CAP_SECS in the chart.
Run two cargo update and cargo upgrade rounds from origin/main.
Upgrade the hotpath crate family to 0.29.0 and refresh compatible
AWS, Redis, profiling, crypto, UUID, and certificate dependencies.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(quota): retire degraded floors from durable usage reports
* docs(quota): clarify retained usage cache refresh behavior
* fix(admin): report unknown bucket usage as null in GetBucketQuota
GetBucketQuota serialized missing usage as size 0, which contradicted the
D3 rule that unknown usage is never reported as zero and, after retiring
degraded floors, made a bucket near its limit look empty. Make size
optional and stop collapsing a missing or unreadable usage snapshot to 0.
* fix(scanner): recover dual corruption with retained epoch fencing
* fix(scanner): preserve recovery markers and legacy quota floors
* test(scanner): expect retained usage recovery marker state
* docs(scanner): register dual-corruption recovery evidence objects
Add the .bloomcycle.bin.dual-recovery/ evidence family to the persisted
objects table and describe its owner, write-once lifecycle and recovery
rule in Deletion And Recovery Rules.
* fix(s3): expose typed causes for unavailable responses
Keep the existing S3 retry contract while distinguishing namespace lock
timeouts, other lock refusals, and storage write quorum failures. Reuse one
bounded typed-source walk and document multipart session attribution.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(s3): preserve XML type on attributed errors
Keep application/xml when the pinned serializer replaces generated headers
with the custom cause map. Assert the media type on direct and wrapped lock
timeouts and namespace/write-quorum error responses.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): unify multipart generation fence lifetimes
Own registry registration before asynchronous incarnation validation so
cancellation cannot preserve a memo beyond real lifecycle-lock coverage.
Reuse successful validation for overlapping PUT and multipart operations,
and retain the full owner through detached part commits and orphan cleanup.
Keep cold validation concurrent and preserve authoritative snapshot,
borrowed-fence, and disabled-lock checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): remove redundant pending closure
Pass std::future::pending directly to the incarnation validator in the
cancellation regression test so strict Clippy accepts the test target.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(decommission): keep partially moved versions readable and defer lost-quorum copies
A version copy whose target write kept losing drive quorum was counted as a
permanent per-object failure, and a partially moved entry made every copied
version unreadable: the retained source lists all versions while the target
lists only the copied ones, and the cross-pool identity check treated that
version-list difference as a same-time conflict (SlowDownRead).
- Compare the version-list fields (num_versions, successor_mod_time,
is_latest) only for exact identities; data-movement equivalent copies of a
version stay readable while the entry is partially moved.
- Defer a delete marker, object or tiered version copy that exhausts its
in-place retries on a drive quorum error to the end-of-round replay instead
of counting it failed; a persistent loss pauses the decommission, which the
local decommission watchdog resumes.
- Skip the backup rename when rolling back a delete transaction whose journal
recorded no original metadata, so a marker created on a missing key no
longer logs a NotFound that reads as the cause of the failed write.
Refs rustfs/backlog#2873
* fix(ecstore): close copy quorum and tiered read gaps
* test(ecstore): synchronize capacity fixtures by object
* fix(s3): report payload SHA256 mismatches accurately
Distinguish signed payload SHA256 failures from MD5 and additional
checksum failures at the shared API error boundary. Preserve the
payload error through archive extraction and use typed s3s errors.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* refactor(s3): classify body failures in one bounded pass
Share typed body-error classification across direct and storage I/O.
Retain marker precedence while replacing repeated source-chain scans,
and map storage-wrapped short HashReader bodies to IncompleteBody.
Cover nested priorities, real short reads, and one traversal budget.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(s3): route upload stream errors through the app facade
Keep the real stream-error fixtures behind the existing test-only S3
boundary without increasing the direct s3s import footprint.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(s3): preserve typed body failures across archive reads
Route TAR entry failures through the shared archive error adapter and
preserve every classified body error. Keep generic archive parsing as
InvalidArgument without formatting discarded transport diagnostics.
Use the existing timeout marker for PUT and Snowball body guards. Add
regressions for real stream verification after sniffing and timeout
responses, and tighten the reduced S3 macro footprint baseline.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
An edit that omits the name, as mc admin replicate update --endpoint
does, was normalized with a name inferred from the new endpoint host and
overwrote the stored peer, renaming the site on every peer map while its
own state name stayed unchanged.
update_peer now keeps the stored name for a known deployment ID when the
edit carries none, covering the public edit, the endpoint-refresh commit
and internal peer deliveries. The public handler also fills the stored
name before planning so the fan-out and refresh journal carry it to
receivers running older versions. Add and join of unknown peers still
infer the name from the endpoint.
A set-remote-target update with the creds group on a bucket target that
belongs to a site-replication peer skipped the connection change, logged a
warning, and still returned 200. The Console's bucket replication form sends
exactly that request when an endpoint changes, so operators saw success while
the stored endpoint, credentials, and TLS settings stayed unchanged.
Return 400 InvalidRequest instead, naming the target and the evidence that
makes it peer-owned, and pointing at a site-replication edit. The ownership
probes are unchanged; the decision is extracted so it can be unit tested.
Document how to change a site's address: the local endpoint is recorded once
at setup and never re-derived, edits must reach every other peer at its stored
endpoint, so sites move one at a time (or old addresses are restored
temporarily), followed by a resync. Commands use rc because mc's resync does
not work against RustFS and its update renames the site.
GET /v3/info and /v3/storageinfo have wrapped the madmin InfoMessage and
StorageInfo under an `info` key since the cluster snapshot view (#3803).
madmin decodes these bodies directly, so `mc admin info` printed empty
values and `mc admin replicate resync start` rejected every alias because
the local deploymentID read as empty.
Requests on the MinIO-compatible /minio/admin prefix now receive the bare
madmin body, following the existing is_compat_admin_request convention.
The /rustfs/admin prefix keeps the envelope that rc (which requires the
`info` key), the console, e2e tests, and auto-testing scripts read.
* perf(s3): borrow options during delete preflight
Pass the existing request options by reference to the metadata preflight instead of cloning them solely for that borrow. Keep the getter's mutable snapshot copy and the later owned-delete copy.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): fence backlog writes against stale pool membership
Reject native scanner publications unless their captured writable pool cohort
matches the fenced durable topology. A smaller startup view must not replace
an expanded membership commit with its older stable rollback.
Keep single-replica test injection fenced by the complete current cohort and
cover rejected startup claims, unchanged CAS revisions, and recovery of the
acknowledged expanded commit.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): use current cohort for healthy scanner CAS writes
Keep the writer's cached topology stale while explicitly capturing the
surviving membership for healthy-target publication and CAS checks. Also
assert that the old cohort rejects healthy-target writes without publishing.
Preserve retiring-source rejection, payload and ETag checks, and canceled
source membership repair without changing production admission.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Record a bounded failure boundary for each non-successful logical remote
walk while preserving the existing outcome, error source and retry behavior.
Distinguish stream open/read, downstream write/flush, wrapper admission and
deadline failures, and unfinished operations dropped before completion.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Include single-pool, status, and expansion fixtures in the existing
native scanner reservation without changing budgets or assertions.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Fence native backlog writes against their captured pool membership and retain
queued replicas as read-only proof members during retirement. Stabilize the
topology test workload and bound pre-ACK recovery separately.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* chore(deps): upgrade workspace parking_lot to 0.13
Move all six direct workspace consumers to parking_lot 0.13 while
retaining the 0.12 dependency line required by upstream crates.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(connect): stabilize client failure classification
Use an ephemeral loopback listener that explicitly rejects proxy CONNECT
and direct TLS connection establishment instead of assuming fixed ports
will promptly refuse connections. Give classification its own bounded
transport and measurement budget while retaining the timeout tests.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Share the existing canonical identity resolver with a non-authoritative source eligibility query. Group borrowed intents and create owned anchors only for retired generations, preserving cache and responsibility contracts and moving proof keys.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): restore marker write quorum on decommission resume
Verify equivalent target markers at write quorum and republish partial
copies before completion. Preserve exact marker creation semantics and
legacy birth identities while retaining ordinary delete and purge guards.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(odm): run delete marker regression on a bounded stack
Construct and run the deep real-storage future on the existing large-stack
test worker so the default libtest stack does not overflow. Preserve the
registered test name, serial guard, and single-part/multipart assertions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): use marker majority for resumed copies
Confirm existing and republished delete markers at the configured-set
majority used by their delete transaction instead of the EC data quorum.
For a twelve-drive EC4 set this accepts seven equivalent markers while
still rejecting six, including when the online inventory is shortened.
Cover named/null markers on a real rebalance retry with retained source
history. The regression fails with InsufficientReadQuorum before the fix.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): keep catch-up fixture timestamps consistent
Admit the seeded catch-up using its observation timestamp so a durable
writer claim crossing a wall-clock second cannot spuriously rate-limit it.
Model an older claim timestamp to exercise that boundary deterministically.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Use a conservative work hint for verified and typed legacy events, retaining the original capacity-maintenance pass and metrics publication. Preserve durable identities, journal retention, retries, and shutdown behavior, with isolated real-runtime and queue lifecycle regressions.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore,filemeta): batch local walk_dir metadata reads and buffer metacache streams
* fix(ecstore): stream completed listing metadata reads
Publish each result before starting the next blocking read, preserving
per-object progress and cancellation without a timed chunk queue.
Keep dispatch batching, byte limits, and namespace/error stops intact.
Cover multiple batches, page boundaries, and oversized-envelope resumption.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore,filemeta): verify production listing streams
Replace the simulated directory-walk benchmark with the public local disk
path. Validate fixture output before timing and retain metacache-only
stream benchmarks. Pin fragmented reads, read coalescing, and canonical
UUID namespace probing against the reference behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): route listing benchmark through storage API
Centralize local listing benchmark imports in the shared benchmark
storage API boundary required by the architecture migration rules.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): keep catch-up fixture timestamps consistent
Admit the seeded catch-up using its observation timestamp so a durable
writer claim crossing a wall-clock second cannot spuriously rate-limit it.
Model an older claim timestamp to exercise that boundary deterministically.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: Hauser <housemecn@gmail.com>
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* chore(deps): refresh GCS, profiling and allocator dependencies
Update published GCS, Pyroscope, mimalloc, and PPMd crate versions.
Align the pinned erasure-fixture test recursion limit with ECStore
to compile GCS-enabled test targets.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* chore(deps): update AWS SDKs, Tokio and memmap2
Refresh compatible AWS service and Smithy dependencies, Tokio timers,
and memory mapping support with the regenerated lockfile.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): identify scanner retirement planner failure phases
Bind the existing native planner error context to each retirement phase while preserving typed retry classification and all storage fences. Exercise the actual wrapper in an isolated process across unavailable and typed failures.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): type the unavailable retirement planner error
Replace the new formatted other error with a private typed error so the error-construction ratchet remains intact. Preserve all phase messages and the existing native retry classification.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): cover unavailable planner error identity
Check the private unavailable cause for every phase through the existing IO boundary. Retain native handoff classification and verify same-phase quorum bucketing after fallible disk narrowing.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(obs): cover contended identity under netlink denial
Check that the production identity entry stays nonblocking while a name
writer is held and real netlink discovery is denied, then observes the
published name after release. Release the writer before joining so a
blocking regression reports a bounded failure.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(obs): keep identity contention behind common helpers
Move the held-writer fixture behind a test-only common helper and enable
it through the Linux x86_64 obs dev-dependency. Preserve the timeout,
writer release before reader join, and identity restoration ordering.
Exercise callback completion and unwind cleanup in the common regression.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Record the first dispatch or undispatched termination at the existing
external request boundary, with entry-gated timing and owned task scope.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(heal): reuse the verified repair proof queue
Retain proof events in place under the existing registry lock and keep
the pre-consumption capacity target and exact identity matching.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(common): move the final proof mismatch fixture
Avoid a redundant clone in the last owned mismatch fixture.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Chris