* fix(rio): ZigZag-encode signed delta varints and widen scratch buffer in compress_index
Index::append_to passes signed delta residuals (c_off and u_off, which are negative whenever a compressed block is smaller than est_block_uncomp / 2) to write_varint(buf, v: i64), which checks while v >= 0x80 without ZigZag encoding and truncates every negative delta to a single byte, while tmp = [0u8; 8] panics out-of-bounds on 9-10 byte varints.
Root Cause:
In crates/rio/src/compress_index.rs, Index::append_to delta-encodes u_off and c_off (c_off -= prev.compressed_offset + c_predict, where c_predict starts at est_block_uncomp / 2) as signed i64 residuals and passes them to write_varint(buf, v: i64). However, write_varint checks while v >= 0x80 on the signed i64 without ZigZag encoding (unlike Go binary.PutVarint / binary.Varint in the S2 index format): for any negative v, v >= 0x80 is false, so write_varint writes a single truncated byte v as u8 without continuation bits, either silently corrupting compressed_offset on Index::load (when (v as u8) < 0x80) or desynchronizing the varint stream (when (v as u8) >= 0x80). Additionally, tmp = [0u8; 8] at line 223 panics with an out-of-bounds index when encoding 9-10 byte varints, and read_varint lacks an overflow check on 10-byte inputs.
Fix:
Use ZigZag encoding and decoding in write_varint and read_varint, reject 64-bit varint overflows in read_varint, and widen the scratch buffer tmp in Index::append_to from [0u8; 8] to [0u8; 10].
Signed-off-by: czcbangkai <czcbangkai@google.com>
* test(rio): use est_block_uncomp field and compare IndexInfo fields in compress_index test
---------
Signed-off-by: czcbangkai <czcbangkai@google.com>
`LocalDisk::walk_dir` stored its own `limit_reached` into the
`producer_limit_reached` flag that every producer of a page shares, so a
walk reaching EOF after a sibling stopped at its per-disk limit cleared the
signal and the collector reported a complete page. Use `fetch_or` so the
flag answers "did any bounded walk stop early", whatever the finish order.
`MetaCacheEntry::is_in_dir` stripped the directory prefix repeatedly, so
`a/a/b` counted as a direct child of `a/`. `MetaCacheEntriesSorted::
forward_past` kept every entry when none sorted after the marker and split
on an index computed over the flattened entries. Both are masked today by
the collector's pre-filtering; fix them and pin the contracts with tests.
* fix(multipart): allow abort after partial commit
Read staging metadata at read quorum during Abort while retaining the
delete write quorum and generation checks. Bound post-fault lock recovery,
isolate quorum rows from peer breaker history, and cover the metadata and
deletion boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(multipart): release topology guards before metadata I/O
Read multipart metadata from owned disk snapshots so slow I/O does not
block registry renewal. Cover read and write lookups with a deterministic
paused-I/O regression.
Reuse one cluster per quorum matrix after observing fresh responses from
every peer. Preserve fault-hit and quorum assertions without repeatedly
starting clusters or disabling peer request breakers.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): avoid head-of-line blocking in delete recovery
Recover independent keys through a bounded rolling window while preserving per-key hint generations, admission budgets, cancellation retry, and the existing transaction decision engine.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): bound durable delete preparation and bootstrap recovery
Prepare independent Linux delete transactions with bounded concurrency while retaining per-key result order and serial fallback for overlapping, rollback, and legacy requests. Preserve startup discovery until bucket metadata is initialized, and add release performance and process-crash acceptance runners.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): verify clean exits in recovery acceptance
Reject forced or OOM exits when measuring clean startup and verify original ETags across mixed-layout crash recovery and a second restart.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): distinguish prepared and decided batch crashes
Observe real preparation and receipt states separately, retain non-vacuous acknowledged-delete sentinels, and require read-only physical convergence before S3 reads can trigger cleanup.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): move final delete recovery options
Move the options value into the batch-ordering regression's final case
instead of cloning it. Keep the earlier clone needed for reuse.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): replay delete receipts after interrupted proof GC
Retire only the exact empty transaction leaf before journal activation. Preserve occupied directories and prefix children, and cover partial cross-disk proof GC without a fresh delete. Tighten crash evidence to distinguish directory-only remnants from retained content.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bound implicit startup delay and publish config early
Limit warm-cache startup to the smaller of the cycle interval and one
minute before jitter, while preserving explicit startup settings.
Publish configuration gauges when applying runtime settings and retain
timeout diagnostics until an actual new cycle begins.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): unify startup waits and skip unchanged configuration
Bound read-only startup inspection and make startup and pre-leadership
retries respond to cancellation and configuration changes. Preserve one
startup deadline and discover replication behind lifecycle-only buckets.
Avoid irrelevant bucket listings and repeated per-disk reconfiguration.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): retain effective startup policy and fallback metrics
Keep the sampled startup deadline when configuration changes leave its
normalized policy unchanged. Initialize retained runtime metrics after
an invalid first refresh without suppressing the validation error or
replacing a newer accepted configuration.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
get_online_offline_disks_stats() counts and demotes all root_disk entries regardless of drive state, underflowing online_disks[ep] (0usize - 1) and double-counting offline_disks[ep] when a root_disk drive is offline or unknown alongside healthy non-root drives.
Root Cause:
In crates/ecstore/src/diagnostics/admin_server_info.rs: (1) root_disk_count counts every drive with root_disk = true even if the drive was already classified into offline_disks or unknown_disks; (2) the subsequent demotion loop unconditionally decrements online_disks[ep] and increments offline_disks[ep] for every disk.root_disk without checking whether disk.state was classified as online (ok or unformatted). When an endpoint has an offline or unknown root_disk drive and zero online drives, *online_disks.get_mut(ep) -= 1 underflows 0usize (panicking in debug/checked builds or wrapping to usize::MAX) and inflates offline_disks.
Fix:
Introduce is_online_drive_state() and only count and demote root_disk drives whose state was classified as online (ok or unformatted), preserving the offline and unknown bucket tallies and keeping online + offline + unknown equal to disks_info.len().
Signed-off-by: czcbangkai <czcbangkai@google.com>
* perf(rpc): remove redundant boxing from read streams
Keep the capped read stream concrete until StreamingBlob owns its type erasure. Add verified four-node TLS coverage for full, unaligned range and concurrent reads, with bounded client memory and explicit integrity checks.
Document protocol comparison commands and the old-node workaround during rolling upgrades.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): preserve TLS trust for alternate credentials
Derive credential-specific clients from the cluster configuration so they retain the generated CA. Exercise successful trusted reads and reject an invalid replacement identity instead of inheriting root credentials.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): collapse acceptance report path guard
Combine the optional report path and existing-file checks into a let chain
without changing the report or PID sidecar collision behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
HttpCompressionConfig::extract_filename_from_content_disposition() calls header_value.to_lowercase() and slices &header_value[start..] using the byte index from lower.find("filename="), panicking on multi-byte Unicode characters whose UTF-8 byte length changes under case mapping and matching unanchored *filename= parameter suffixes.
Root Cause:
In rustfs/src/server/compress.rs, extract_filename_from_content_disposition() computes lower = header_value.to_lowercase() and uses the byte offset idx from lower.find("filename=") to slice &header_value[start..]. Because Unicode case mapping can expand or shrink UTF-8 byte lengths (e.g., U+0130 "İ" is 2 bytes in UTF-8 but lowercases to 3 bytes; U+212A "K" is 3 bytes in UTF-8 but lowercases to 1 byte), start can land inside a multi-byte UTF-8 codepoint or past the end of header_value, causing a runtime panic. In addition, lower.find("filename=") matches any parameter ending in filename= (such as x-filename=).
Fix:
Use header_value.to_ascii_lowercase() so byte offsets remain 1:1 with header_value, and require filename= to appear either at the start of the header or after a semicolon parameter delimiter.
Signed-off-by: czcbangkai <czcbangkai@google.com>
StdBackend::pread_bytes caches FdCacheEntry { file, len } and skips fstat on cache hits, passing cached len directly to unsafe { mmap_options.map(file) }. However, StdBackend::open_write / UringBackend::open_write (create_file, append_file), LocalDisk::write_all_private (write_all), LocalDisk::rename_part, and directory LocalDisk::rename_file fail to invalidate FdCache, causing mmap past EOF (SIGBUS), false DiskError::FileCorrupt on grown files, or stale reads after renames.
Root Cause:
In crates/ecstore/src/disk/local.rs, StdBackend::pread_bytes caches open File handles and their initial fstat byte length in FdCache and uses the cached len directly for bounds checks and unsafe { mmap_options.map(file) }. While LocalDisk::rename_file (non-directory) and LocalDisk::delete invalidate FdCache, five mutation paths omit invalidation: (1) StdBackend::open_write (called by create_file with O_TRUNC and append_file), (2) LocalDisk::write_all_private (called by write_all with O_TRUNC), (3) LocalDisk::rename_part, (4) LocalDisk::rename_file when src_is_dir is true, and (5) UringBackend::open_write / unlink / rename which fail to forward invalidations to self.inner. Truncating a cached file across a page boundary causes mmap to map beyond EOF and trigger SIGBUS; growing a cached file causes false DiskError::FileCorrupt.
Fix:
Invalidate FdCache in StdBackend::open_write, UringBackend::open_write/unlink/rename, LocalDisk::write_all_private, LocalDisk::rename_part (both src and dst), and directory LocalDisk::rename_file (via invalidate_prefix on src and dst).
Signed-off-by: czcbangkai <czcbangkai@google.com>
* test(e2e): add MPU RPC quorum and recovery acceptance
Exercise separate storage and lock RPC delays across multipart phases,
including quorum boundaries, cancellation, bucket recreation, and bounded
eight-node load. Honor TMPDIR for isolated cluster drive directories.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(lock): probe healthy voters behind silent peers
Spread additional probes across the existing acquisition attempt budget
when the initial quorum-plus-spare wave stalls. Preserve quorum thresholds,
fast-path fan-out, and existing ownership and late-response cleanup.
Cover every four-node read-voter ordering, a stalled eight-node write
quorum, and normal fan-out. Extend MPU acceptance to EC12+4 at twelve and
eleven metadata acknowledgements.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Recover independent keys through a bounded rolling window while preserving per-key hint generations, admission budgets, cancellation retry, and the existing transaction decision engine.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(s3): preserve aws-chunked error responses
Map named aws-chunked failures through the shared bounded body classifier,
keeping typed source precedence and the dependency's S3 response contract.
Cover exact responses, upload publication, and valid signed/trailer limits.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(s3): classify upload stream errors once per cause
Reuse the typed UploadStreamError result for digest and incomplete-body
classification while preserving the shared classifier's marker priorities.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): refresh aws-chunked profile selections
Record the two new aws-chunked regression cases in smoke and full profiles.
Verify exact additions against the CI Linux inventory and native Darwin
listings, recovering each previous digest when those two cases are removed.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(rio): isolate internode streams with pooled HTTP/1.1
Default the internode HTTP data plane to reusable HTTP/1.1 connections so paused shard bodies cannot consume shared HTTP/2 receive credit. Retain explicit auto negotiation and the existing TLS, pooling, and recovery paths.
Refs rustfs/rustfs#8843 and rustfs/backlog#2934.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(rpc): frame bounded shard reads for connection reuse
Preserve the bounded read cap and stream-to-EOF sentinel while making connection completion independent of a chunked terminator. Cover exact reads without EOF polling, pooled TLS reuse, and partial-body truncation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(deps): migrate signer and OIDC to patched crypto
Pin the requested signer fix and upstream OIDC/OAuth2 crypto migrations to immutable commits. Preserve registry SDK type identity, remove the old bigint and elliptic-curve chains, and verify independent login state, nonce, and PKCE values.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
ConditionalCorsLayer took the bucket from the first path segment only.
With RUSTFS_SERVER_DOMAINS set, a virtual-hosted-style request such as
OPTIONS http://mybucket.localhost:9000/x.jpg was evaluated against a
bucket named "x.jpg", so a preflight never matched the bucket's CORS
configuration and returned 200 without Access-Control-* headers, while
the same request in path style succeeded.
Resolve the bucket from the host prefix first, reusing the virtual-host
matching that the API rate limiter already uses, and fall back to the
first path segment for path-style requests. A bucket-level request on a
virtual host ("/") is now treated as a bucket request instead of the
server root.
Co-authored-by: cozima <8464261+cozima@users.noreply.github.com>
Co-authored-by: Claude Opus 5.5 <noreply@anthropic.com>
* test(scanner): isolate async checkpoint log capture
Scope captured tracing subscribers to each future poll and cover the
competing-dispatcher failure without weakening checkpoint assertions.
Verify exact bounded erasure-probe accounting around the 64-entry limit.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): classify erasure candidates in one budgeted walk
Continue directory classification in cooperative 64-entry batches until
part evidence or EOF, retaining normal checkpoint and namespace work.
Defer metadata accounting and namespace reuse until classification ends.
Distinguish vanished opens from incomplete iteration and preserve quota,
sampling, failed-object and non-destructive heal behavior.
Pin test callsite interest alongside poll-scoped capture to prevent an
unsubscribed sibling from caching the checkpoint callsite as disabled.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): purge every explicit delete-marker version
Keep explicit marker removals on the canonical version-delete path so
same-key batches are not truncated and pending replication does not retain
source markers. Preserve marker identity and timestamps in the response.
Add real four-disk regressions for mixed version cleanup, duplicate marker
IDs, pending replication, and null/UUID marker removal.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(filemeta): index explicit delete-marker batches
Reuse the batch index and final compaction for ordinary marker removals.
Keep small marker-only histories on the scalar path and preserve fallback
for free versions and complex mutations.
Add scalar-equivalence, boundary, corruption and deterministic scan-count
coverage, a real-disk mixed-history regression, and reproducible marker
benchmark cases. Preserve existing benchmark defaults while honoring CLI
sampling overrides.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(filemeta): remove redundant marker fixture clone
Move the original metadata into its final test fixture after the last
borrow, avoiding the redundant clone reported by strict Clippy checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(storage): validate explicit marker deletion and recovery
Exercise UUID/null and duplicate explicit marker removals over real HTTP
with retries disabled, including immediate four-node visibility and exact
retained payloads. Cover all-drive PREPARED/APPLIED/DECISION process
termination, rollback/commit recovery, and old-ID retries across a second
restart without deleting a fresh marker.
Use flat sampling and fixed per-case fixture batches in the metadata
benchmark. Update profile selection digests for the added regressions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): synchronize crash fixture startup and flush readiness
Give disk setup and the durable flush boundary separate readiness gates,
keeping each phase bounded to ten seconds. Preserve the real SIGKILL and
restart assertions while reporting early child exits and retaining output.
Reap fixture children on assertion failure and cover early-exit diagnostics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): roll a wedged pause backlog forward to its proven ledger
For rustfs/backlog#2873 defect E.
Two interleaved writers could leave a replica state that no claim could
leave. Writer B read writer A's partial commit and selected the stable
rollback point. A's commit then completed and A verified it. B's rollback
stabilization overwrote every member it had read after the commit, and A's
stabilization reached only the remaining member. No commit survived and the
stable fields differed, so every claim and every decommission retirement
failed with "neither a surviving membership commit nor a stable rollback
point".
- When no commit survives and stable consensus fails, select the unique
newest stable ledger by (writer epoch, generation). A stable field is
written only after its ledger was selected as authority, so that ledger is
the last proven one. The next claim stabilizes it and advances normally.
- Keep failing closed for an invalid or unreadable replica, for two
different stable ledgers at the newest key, and for a missing replica that
a stored commit names as a member.
- A rollback proof written without an authoritative commit names only
members that already store a record, so a failed first write to a new
member cannot turn it into a named member that reads as lost.
- Retirement refuses a rolled-forward ledger until the writer has
stabilized it, as a retryable handoff conflict rather than a terminal
authority conflict.
- Tests replay the interleaving on real storage: it converges and retires,
it converges after another expansion with a failed first write to a new
member, and a same-key fork or a lost named member still refuses.
* fix(scanner): let retirement stabilize a rolled-forward pause backlog
Follow-up to the defect E roll-forward for rustfs/backlog#2873.
- Retirement no longer waits for a scanner writer. When no commit
survives and the native selector over every read member (survivors,
retiring source and queued pools) rolls forward, the planner commits and
stabilizes that ledger on the survivors under the writers' object fence.
The previous retryable handoff could exhaust its ~13.5 s budget long
before a writer re-claimed, and failed the decommission terminally.
- A writer's roll-forward refuses when a stored commit names a member that
is missing or outside its read, such as a retiring source excluded from
the writable set. Retirement, which reads that member, decides instead.
- A writer stabilizing a rolled-forward ledger writes missing members
first and stops at the first failure. A member that only read as missing
rejects the create, so no surviving proof can omit its newer record.
- Docs describe the actual roll-forward preconditions and the retirement
exit.
- Tests: retirement converges the wedge with no writer; a writer refuses to
roll forward past an unread source that holds the newest ledger and
retirement keeps it; a spurious missing read changes no record and the
next claim converges on the newest ledger.
* fix(decommission): create missing pause backlog survivors first
Follow-up to the defect E retirement roll-forward for rustfs/backlog#2873.
- Retirement creates survivors that read as missing one at a time, before
any stored survivor, and stops at the first failed create. A transient
create failure on a new pool's set no longer leaves every other survivor
holding a commit that names a still-missing member, which no later
selection could leave.
- A rolled-forward retirement first seeds a rollback proof that names only
the survivors that store a record; the commit still names every survivor.
- Retirement rolls forward only when the survivors cannot select a ledger
on their own or select the same one. Survivors that agree on a different
ledger keep the source-authority check.
- The scanner writer applies the same missing-first, stop-at-first-failure
order to every stabilization with a missing member, including a seed from
an old commit selected as the rollback proof.
- Tests: a failed new-pool create during retirement retries to convergence;
a spurious missing read cannot seed an old commit over a newer member;
the rolled-forward precedence over source stable records is pinned.
* docs(decommission): state the all-missing survivor limit of backlog roll-forward
* test(e2e): use the resumed decommission completion budget
* test(heal): exercise checkpoints written by rc.5
* test(heal): route upgrade fixtures through the storage facade
* fix(deps): update ctutils to address constant-time advisory
* test(lifecycle): gate recovery I/O while advancing test time
* fix(decommission): count settled entries once and back off repeated pauses
Follow-up to rustfs/rustfs#8824 for rustfs/backlog#2873 defect F.
- Count an entry's versions when the entry settles (copied and cleaned up,
or retained) instead of as each version copies. A partially moved entry
that is deferred, paused and resumed, or interrupted, no longer counts its
already-moved versions again when the resumed pass finds them on the
target.
- Hold the watchdog relaunch of a decommission that pauses again on the
same cause without progress, doubling from 30 s up to 15 min, and warn with
the cause. A first pause still relaunches on the next pass; the pause never
becomes a terminal failure.
- Log each drive's error when a delete version loses write quorum, so the
cause of a sub-quorum delete-marker copy is recorded instead of only its
rollback.
- Tests: pause/resume counting, a target whose metadata is below read
quorum, the copy-quorum pause cause, the relaunch backoff, and a rollback
test that asserts the restore attempt instead of a tracing log.
* fix(decommission): measure pause progress by cleaned entries
Review follow-up on the pause/resume counting and relaunch backoff.
- A pause now carries this generation's progress counters into the
pool metadata it persists, like a progress checkpoint. Before, everything
counted since the last checkpoint was dropped at the pause.
- A retained entry (one whose source stays) is relisted by every resumed
pass. A per-instance ledger counts its versions once per decommission
generation, so retained versions no longer grow per round in one process.
- The relaunch backoff measures progress by entries moved off the pool and
completed buckets, which a relisted retained entry cannot advance. Before,
the recounted retained entry looked like progress and reset the backoff.
- Tests: exact totals across a pause and a resume from persisted metadata;
a target kept below quorum defers the entry with nothing counted; a
retained entry plus a repeated cause escalates the backoff over three
pauses, with the watchdog path relaunching a first pause and holding a
repeat.
* fix(ecstore): bound retained decommission accounting
* feat(scanner): configure startup recovery backoff cap
* test(scanner): compare recovery retry count without narrowing
* docs(scanner): document recovery backoff cap and expose it in Helm
List scanner.recovery_backoff_cap in the runtime controls table, drop the
fixed ten-minute wording from the follower recovery section, and map
config.rustfs.scanner.recovery_backoff_cap_secs to
RUSTFS_SCANNER_RECOVERY_BACKOFF_CAP_SECS in the chart.
Run two cargo update and cargo upgrade rounds from origin/main.
Upgrade the hotpath crate family to 0.29.0 and refresh compatible
AWS, Redis, profiling, crypto, UUID, and certificate dependencies.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(quota): retire degraded floors from durable usage reports
* docs(quota): clarify retained usage cache refresh behavior
* fix(admin): report unknown bucket usage as null in GetBucketQuota
GetBucketQuota serialized missing usage as size 0, which contradicted the
D3 rule that unknown usage is never reported as zero and, after retiring
degraded floors, made a bucket near its limit look empty. Make size
optional and stop collapsing a missing or unreadable usage snapshot to 0.
* fix(scanner): recover dual corruption with retained epoch fencing
* fix(scanner): preserve recovery markers and legacy quota floors
* test(scanner): expect retained usage recovery marker state
* docs(scanner): register dual-corruption recovery evidence objects
Add the .bloomcycle.bin.dual-recovery/ evidence family to the persisted
objects table and describe its owner, write-once lifecycle and recovery
rule in Deletion And Recovery Rules.
Chris An
