* refactor(admin): complete the owned admin request boundary
Every type implementing the owned admin `Operation` now serves the legacy
router through one blanket impl that converts the s3s request and response
at that single edge, so handlers register with `AdminOperation(&Handler {})`
and need no adapter statics; `LegacyAdminAdapter` is removed.
`AdminRequest` gains the accessors handlers read today (`remote_addr`,
`client_info`, `request_context`, `req_info`, `req_info_mut`,
`install_req_info`, `envelope_mut`, `take_body`), the request helpers gain
envelope forms (`authenticate_envelope_credentials`,
`object_store_from_envelope`, the table-catalog `*_from_envelope` helpers,
`KmsAdminAudit::from_envelope`, `AccountAuditContext::from_envelope`,
`CallerIdentity::resolve_envelope`), `read_compatible_admin_body` accepts
either body type, and the owned `Body` gains `store_all_limited` and
`Default`. `admin::operation::test_support::TestRequest` builds owned
requests for handler tests. Unused accessors carry a per-method dead-code
expectation that each migrating handler batch removes.
No handler behavior changes; the remaining migrations follow in batches.
* chore(admin): add the T1.12 admin handler codemod
The script rewrites an admin handler file from the s3s request and response
types to the owned boundary (imports, signatures, request field and helper
calls, body access, response construction, and test request literals) and
leaves a TODO comment at every site it cannot rewrite mechanically. It is
idempotent and is removed once T1.12 completes.
* refactor(admin): migrate replication and KMS handlers to owned requests (#8951)
Move the T1.12 batch 2 admin handlers (site_replication, kms_keys, event,
replication, diagnostics, on_demand_migration, audit, account, kms_rekey,
kms_key_lifecycle, object_data_cache, tls_debug, integrity) from the s3s
request/response types to the owned AdminRequest/AdminResponse boundary.
Handler logic, authorization order, error codes and response bytes are
unchanged; the request/response plumbing now goes through the envelope
accessors and helpers introduced by the owned boundary.
The shared response helpers (utils::json_response, utils::empty_response,
handlers::admin_json_response, target_descriptor::build_json_response,
profile::authorize_profile_request) and the site replication repair
executor still answer with the s3s types, so the migrated files carry
owned-boundary forms of those helpers and convert the repair response at
its call edge. The streaming console-log body and the two devnull drains
move from the s3s byte stream to the owned Body stream/frame API.
Drop the dead_code expectations for the owned accessors and envelope
helpers this batch starts using, and lower the s3s footprint baseline to
the measured 99 files.
Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).
* refactor(admin): add owned forms of the shared admin response helpers
Handler batches were copying `json_response`, `empty_response`,
`admin_json_response`, and `build_json_response` into each migrated file
because the shared helpers still return the s3s response. Provide owned
forms next to each legacy helper (`owned_json_response`,
`owned_empty_response`, `owned_admin_json_response`,
`owned_build_json_response`, `authorize_profile_envelope`) with parity
tests against the legacy shape, derive `Debug` for `AdminResponse` so
tests can use `expect_err`, and fix three codemod rules: strip owned
string conversions in test credentials, annotate non-literal URI parses,
and leave a rebound request parameter immutable.
* refactor(admin): use the shared owned response helpers in batch 2 (#8954)
Replace the per-file owned-boundary copies of json_response,
empty_response, admin_json_response, build_json_response and
authorize_profile_request in the batch 2 admin handlers with the shared
owned_* helpers and authorize_profile_envelope that the boundary branch
now provides, and restore expect_err in the handler tests now that
AdminResponse implements Debug; the `.err().expect(..)` stand-ins were
failing the base branch's clippy step (clippy::err_expect under
-D warnings). The site replication repair call edge
keeps its conversion because the repair executor still answers with the
s3s response type.
Drop the dead_code expectations on the shared owned helpers this batch
now uses in non-test code.
Part of rustfs/backlog#2788 (rustfs/backlog#2734 T1.12).
* fix(admin): preserve legacy body limit error responses
* refactor(admin): move policy, ILM and batch 3 handlers to owned requests (#8956)
Batch 3 of T1.12 (rustfs/backlog#2788, parent rustfs/backlog#2734) moves
14 admin handler files to AdminRequest/AdminResponse and the owned
Operation trait: ilm_transition, policies, idp_compat, config_admin,
plugins_instances, profile_admin, group, heal, object_zip_download,
kms_key_metadata, durability, inspect_archive, sts and
gateway_key_inventory. Handler logic, authorization order, error codes
and response bytes are unchanged; test request literals use
test_support::TestRequest.
The handlers use the shared owned helpers (owned_json_response,
owned_admin_json_response, owned_build_json_response,
authorize_profile_envelope). The dead_code expectations on the
AdminRequest accessors and RequestEnvelope::uri_mut they now exercise
are removed. The legacy target_descriptor::build_json_response keeps
only its equivalence test caller and carries the matching expectation
until it retires.
object_zip_download builds the s3s request the S3 gate still reads from
the owned request facts; heal authenticates through
authenticate_envelope_credentials; the trace, inspect-archive and zip
bodies use Body::from_stream. extract_plugin_instance_filters takes the
URI so extensions.rs keeps calling it, and the two idp_compat
delegations stay on the legacy trait until their delegates migrate.
The s3s footprint baseline drops from 99 to 90 files.
* test(admin): anchor the replication diff body contract on the call
The source-contract test looked for `let body = read_compatible_admin_body(`
on one line; rustfmt wrapped that binding after the handler moved to the
owned request boundary, so the marker was no longer found. Anchor on the
call itself so formatting cannot change what the contract checks.
* refactor(admin): migrate table catalog and batch 1 to owned requests (#8955)
* fix(admin): preserve catalog body limit response bytes
Source: main 0f24511535
Target: gateway/integration 8cf6187ff4
Resolve the e2e-full selection digest conflict by recomputing both
platform digests over the merged nextest listing (685 darwin / 693
linux): main adds two rpc_fault_proxy unit tests, integration keeps its
dual-stack differential and anonymous-access additions. Raise the rustfs
test-count baseline row to the merged count (5932).
* test(e2e): tighten scoped multipart fault acceptance
Target Abort root deletion instead of child transaction cleanup, require
actual metadata peer hits, and keep faulted APIs single-attempt. Validate
bounded ordinary recovery and safe quorum refusal for partial mutations,
with the recovery enhancement tracked in backlog#2940.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(e2e): drive concurrent TLS reads independently
Give each coordinator its own supervised task and cooperative polling
budget while retaining the existing workload, deadline and byte/hash
assertions. Preserve node logs after fixture teardown and report the
operation, response phase and received bytes when a transfer times out.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(listing): honor persisted erasure quorum
Separate metadata admission and primary-reader sizing from each version's
persisted erasure requirements. Propagate agreed-entry uncertainty and
report LIST read-quorum failures as storage unavailability.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(listing): borrow metadata during quorum resolution
Keep validated metadata inspection and agreed-version selection in
filemeta. Resolve independent reader votes through borrowed slots and
retain owned buffers until a filtered result is required.
Preserve fixed-baseline legacy comparisons, cached latest fast paths,
quorum policy, and disk fan-out. Add ownership regressions and paired
release benchmarks for cold and cached version histories.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): remove redundant fallback clone
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scripts): collect scanner host telemetry on macOS
Use platform-supported iostat arguments and expose optional collector
failures without interrupting scanner-status collection.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): reuse validated metadata read path buffers
Share the checked bucket/object tuple across metadata read consumers and borrow clean normalization inputs without skipping filesystem validation. Keep Windows reconstruction and the second bucket check intact.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scripts): finalize scanner telemetry for failed runs
Initialize host telemetry artifacts before external captures so reused output directories cannot retain prior successes. Share normal and error-path finalization, reap collectors on request failure, and preserve the primary exit status even if status recording fails.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): distinguish unreadable recovery inventories
Treat only the exact system-prefix read-quorum result as incomplete during bounded polling. Preserve strict final reads and the shutdown/drain assertions, record failed observations, and cover interrupted empty streaks and unrelated errors.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(rio): ZigZag-encode signed delta varints and widen scratch buffer in compress_index
Index::append_to passes signed delta residuals (c_off and u_off, which are negative whenever a compressed block is smaller than est_block_uncomp / 2) to write_varint(buf, v: i64), which checks while v >= 0x80 without ZigZag encoding and truncates every negative delta to a single byte, while tmp = [0u8; 8] panics out-of-bounds on 9-10 byte varints.
Root Cause:
In crates/rio/src/compress_index.rs, Index::append_to delta-encodes u_off and c_off (c_off -= prev.compressed_offset + c_predict, where c_predict starts at est_block_uncomp / 2) as signed i64 residuals and passes them to write_varint(buf, v: i64). However, write_varint checks while v >= 0x80 on the signed i64 without ZigZag encoding (unlike Go binary.PutVarint / binary.Varint in the S2 index format): for any negative v, v >= 0x80 is false, so write_varint writes a single truncated byte v as u8 without continuation bits, either silently corrupting compressed_offset on Index::load (when (v as u8) < 0x80) or desynchronizing the varint stream (when (v as u8) >= 0x80). Additionally, tmp = [0u8; 8] at line 223 panics with an out-of-bounds index when encoding 9-10 byte varints, and read_varint lacks an overflow check on 10-byte inputs.
Fix:
Use ZigZag encoding and decoding in write_varint and read_varint, reject 64-bit varint overflows in read_varint, and widen the scratch buffer tmp in Index::append_to from [0u8; 8] to [0u8; 10].
Signed-off-by: czcbangkai <czcbangkai@google.com>
* test(rio): use est_block_uncomp field and compare IndexInfo fields in compress_index test
---------
Signed-off-by: czcbangkai <czcbangkai@google.com>
`LocalDisk::walk_dir` stored its own `limit_reached` into the
`producer_limit_reached` flag that every producer of a page shares, so a
walk reaching EOF after a sibling stopped at its per-disk limit cleared the
signal and the collector reported a complete page. Use `fetch_or` so the
flag answers "did any bounded walk stop early", whatever the finish order.
`MetaCacheEntry::is_in_dir` stripped the directory prefix repeatedly, so
`a/a/b` counted as a direct child of `a/`. `MetaCacheEntriesSorted::
forward_past` kept every entry when none sorted after the marker and split
on an index computed over the flattened entries. Both are masked today by
the collector's pre-filtering; fix them and pin the contracts with tests.
* fix(multipart): allow abort after partial commit
Read staging metadata at read quorum during Abort while retaining the
delete write quorum and generation checks. Bound post-fault lock recovery,
isolate quorum rows from peer breaker history, and cover the metadata and
deletion boundaries.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(multipart): release topology guards before metadata I/O
Read multipart metadata from owned disk snapshots so slow I/O does not
block registry renewal. Cover read and write lookups with a deterministic
paused-I/O regression.
Reuse one cluster per quorum matrix after observing fresh responses from
every peer. Preserve fault-hit and quorum assertions without repeatedly
starting clusters or disabling peer request breakers.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): avoid head-of-line blocking in delete recovery
Recover independent keys through a bounded rolling window while preserving per-key hint generations, admission budgets, cancellation retry, and the existing transaction decision engine.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(ecstore): bound durable delete preparation and bootstrap recovery
Prepare independent Linux delete transactions with bounded concurrency while retaining per-key result order and serial fallback for overlapping, rollback, and legacy requests. Preserve startup discovery until bucket metadata is initialized, and add release performance and process-crash acceptance runners.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): verify clean exits in recovery acceptance
Reject forced or OOM exits when measuring clean startup and verify original ETags across mixed-layout crash recovery and a second restart.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): distinguish prepared and decided batch crashes
Observe real preparation and receipt states separately, retain non-vacuous acknowledged-delete sentinels, and require read-only physical convergence before S3 reads can trigger cleanup.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(ecstore): move final delete recovery options
Move the options value into the batch-ordering regression's final case
instead of cloning it. Keep the earlier clone needed for reuse.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): replay delete receipts after interrupted proof GC
Retire only the exact empty transaction leaf before journal activation. Preserve occupied directories and prefix children, and cover partial cross-disk proof GC without a fresh delete. Tighten crash evidence to distinguish directory-only remnants from retained content.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): bound implicit startup delay and publish config early
Limit warm-cache startup to the smaller of the cycle interval and one
minute before jitter, while preserving explicit startup settings.
Publish configuration gauges when applying runtime settings and retain
timeout diagnostics until an actual new cycle begins.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): unify startup waits and skip unchanged configuration
Bound read-only startup inspection and make startup and pre-leadership
retries respond to cancellation and configuration changes. Preserve one
startup deadline and discover replication behind lifecycle-only buckets.
Avoid irrelevant bucket listings and repeated per-disk reconfiguration.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): retain effective startup policy and fallback metrics
Keep the sampled startup deadline when configuration changes leave its
normalized policy unchanged. Initialize retained runtime metrics after
an invalid first refresh without suppressing the validation error or
replacing a newer accepted configuration.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
get_online_offline_disks_stats() counts and demotes all root_disk entries regardless of drive state, underflowing online_disks[ep] (0usize - 1) and double-counting offline_disks[ep] when a root_disk drive is offline or unknown alongside healthy non-root drives.
Root Cause:
In crates/ecstore/src/diagnostics/admin_server_info.rs: (1) root_disk_count counts every drive with root_disk = true even if the drive was already classified into offline_disks or unknown_disks; (2) the subsequent demotion loop unconditionally decrements online_disks[ep] and increments offline_disks[ep] for every disk.root_disk without checking whether disk.state was classified as online (ok or unformatted). When an endpoint has an offline or unknown root_disk drive and zero online drives, *online_disks.get_mut(ep) -= 1 underflows 0usize (panicking in debug/checked builds or wrapping to usize::MAX) and inflates offline_disks.
Fix:
Introduce is_online_drive_state() and only count and demote root_disk drives whose state was classified as online (ok or unformatted), preserving the offline and unknown bucket tallies and keeping online + offline + unknown equal to disks_info.len().
Signed-off-by: czcbangkai <czcbangkai@google.com>
* perf(rpc): remove redundant boxing from read streams
Keep the capped read stream concrete until StreamingBlob owns its type erasure. Add verified four-node TLS coverage for full, unaligned range and concurrent reads, with bounded client memory and explicit integrity checks.
Document protocol comparison commands and the old-node workaround during rolling upgrades.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): preserve TLS trust for alternate credentials
Derive credential-specific clients from the cluster configuration so they retain the generated CA. Exercise successful trusted reads and reject an invalid replacement identity instead of inheriting root credentials.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): collapse acceptance report path guard
Combine the optional report path and existing-file checks into a let chain
without changing the report or PID sidecar collision behavior.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
StdBackend::pread_bytes caches FdCacheEntry { file, len } and skips fstat on cache hits, passing cached len directly to unsafe { mmap_options.map(file) }. However, StdBackend::open_write / UringBackend::open_write (create_file, append_file), LocalDisk::write_all_private (write_all), LocalDisk::rename_part, and directory LocalDisk::rename_file fail to invalidate FdCache, causing mmap past EOF (SIGBUS), false DiskError::FileCorrupt on grown files, or stale reads after renames.
Root Cause:
In crates/ecstore/src/disk/local.rs, StdBackend::pread_bytes caches open File handles and their initial fstat byte length in FdCache and uses the cached len directly for bounds checks and unsafe { mmap_options.map(file) }. While LocalDisk::rename_file (non-directory) and LocalDisk::delete invalidate FdCache, five mutation paths omit invalidation: (1) StdBackend::open_write (called by create_file with O_TRUNC and append_file), (2) LocalDisk::write_all_private (called by write_all with O_TRUNC), (3) LocalDisk::rename_part, (4) LocalDisk::rename_file when src_is_dir is true, and (5) UringBackend::open_write / unlink / rename which fail to forward invalidations to self.inner. Truncating a cached file across a page boundary causes mmap to map beyond EOF and trigger SIGBUS; growing a cached file causes false DiskError::FileCorrupt.
Fix:
Invalidate FdCache in StdBackend::open_write, UringBackend::open_write/unlink/rename, LocalDisk::write_all_private, LocalDisk::rename_part (both src and dst), and directory LocalDisk::rename_file (via invalidate_prefix on src and dst).
Signed-off-by: czcbangkai <czcbangkai@google.com>
* test(e2e): add MPU RPC quorum and recovery acceptance
Exercise separate storage and lock RPC delays across multipart phases,
including quorum boundaries, cancellation, bucket recreation, and bounded
eight-node load. Honor TMPDIR for isolated cluster drive directories.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(lock): probe healthy voters behind silent peers
Spread additional probes across the existing acquisition attempt budget
when the initial quorum-plus-spare wave stalls. Preserve quorum thresholds,
fast-path fan-out, and existing ownership and late-response cleanup.
Cover every four-node read-voter ordering, a stalled eight-node write
quorum, and normal fan-out. Extend MPU acceptance to EC12+4 at twelve and
eleven metadata acknowledgements.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
Recover independent keys through a bounded rolling window while preserving per-key hint generations, admission budgets, cancellation retry, and the existing transaction decision engine.
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(s3): preserve aws-chunked error responses
Map named aws-chunked failures through the shared bounded body classifier,
keeping typed source precedence and the dependency's S3 response contract.
Cover exact responses, upload publication, and valid signed/trailer limits.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(s3): classify upload stream errors once per cause
Reuse the typed UploadStreamError result for digest and incomplete-body
classification while preserving the shared classifier's marker priorities.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(e2e): refresh aws-chunked profile selections
Record the two new aws-chunked regression cases in smoke and full profiles.
Verify exact additions against the CI Linux inventory and native Darwin
listings, recovering each previous digest when those two cases are removed.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(rio): isolate internode streams with pooled HTTP/1.1
Default the internode HTTP data plane to reusable HTTP/1.1 connections so paused shard bodies cannot consume shared HTTP/2 receive credit. Retain explicit auto negotiation and the existing TLS, pooling, and recovery paths.
Refs rustfs/rustfs#8843 and rustfs/backlog#2934.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(rpc): frame bounded shard reads for connection reuse
Preserve the bounded read cap and stream-to-EOF sentinel while making connection completion independent of a chunked terminator. Cover exact reads without EOF polling, pooled TLS reuse, and partial-body truncation.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(deps): migrate signer and OIDC to patched crypto
Pin the requested signer fix and upstream OIDC/OAuth2 crypto migrations to immutable commits. Preserve registry SDK type identity, remove the old bigint and elliptic-curve chains, and verify independent login state, nonce, and PKCE values.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* test(scanner): isolate async checkpoint log capture
Scope captured tracing subscribers to each future poll and cover the
competing-dispatcher failure without weakening checkpoint assertions.
Verify exact bounded erasure-probe accounting around the 64-entry limit.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): classify erasure candidates in one budgeted walk
Continue directory classification in cooperative 64-entry batches until
part evidence or EOF, retaining normal checkpoint and namespace work.
Defer metadata accounting and namespace reuse until classification ends.
Distinguish vanished opens from incomplete iteration and preserve quota,
sampling, failed-object and non-destructive heal behavior.
Pin test callsite interest alongside poll-scoped capture to prevent an
unsubscribed sibling from caching the checkpoint callsite as disabled.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(ecstore): purge every explicit delete-marker version
Keep explicit marker removals on the canonical version-delete path so
same-key batches are not truncated and pending replication does not retain
source markers. Preserve marker identity and timestamps in the response.
Add real four-disk regressions for mixed version cleanup, duplicate marker
IDs, pending replication, and null/UUID marker removal.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* perf(filemeta): index explicit delete-marker batches
Reuse the batch index and final compaction for ordinary marker removals.
Keep small marker-only histories on the scalar path and preserve fallback
for free versions and complex mutations.
Add scalar-equivalence, boundary, corruption and deterministic scan-count
coverage, a real-disk mixed-history regression, and reproducible marker
benchmark cases. Preserve existing benchmark defaults while honoring CLI
sampling overrides.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(filemeta): remove redundant marker fixture clone
Move the original metadata into its final test fixture after the last
borrow, avoiding the redundant clone reported by strict Clippy checks.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(storage): validate explicit marker deletion and recovery
Exercise UUID/null and duplicate explicit marker removals over real HTTP
with retries disabled, including immediate four-node visibility and exact
retained payloads. Cover all-drive PREPARED/APPLIED/DECISION process
termination, rollback/commit recovery, and old-ID retries across a second
restart without deleting a fresh marker.
Use flat sampling and fixed per-case fixture batches in the metadata
benchmark. Update profile selection digests for the added regressions.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(heal): synchronize crash fixture startup and flush readiness
Give disk setup and the durable flush boundary separate readiness gates,
keeping each phase bounded to ten seconds. Preserve the real SIGKILL and
restart assertions while reporting early child exits and retaining output.
Reap fixture children on assertion failure and cover early-exit diagnostics.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
---------
Co-authored-by: heihutu <heihutu@gmail.com>
Co-authored-by: zhi22915 <qiuzgang@gmail.com>
* fix(scanner): roll a wedged pause backlog forward to its proven ledger
For rustfs/backlog#2873 defect E.
Two interleaved writers could leave a replica state that no claim could
leave. Writer B read writer A's partial commit and selected the stable
rollback point. A's commit then completed and A verified it. B's rollback
stabilization overwrote every member it had read after the commit, and A's
stabilization reached only the remaining member. No commit survived and the
stable fields differed, so every claim and every decommission retirement
failed with "neither a surviving membership commit nor a stable rollback
point".
- When no commit survives and stable consensus fails, select the unique
newest stable ledger by (writer epoch, generation). A stable field is
written only after its ledger was selected as authority, so that ledger is
the last proven one. The next claim stabilizes it and advances normally.
- Keep failing closed for an invalid or unreadable replica, for two
different stable ledgers at the newest key, and for a missing replica that
a stored commit names as a member.
- A rollback proof written without an authoritative commit names only
members that already store a record, so a failed first write to a new
member cannot turn it into a named member that reads as lost.
- Retirement refuses a rolled-forward ledger until the writer has
stabilized it, as a retryable handoff conflict rather than a terminal
authority conflict.
- Tests replay the interleaving on real storage: it converges and retires,
it converges after another expansion with a failed first write to a new
member, and a same-key fork or a lost named member still refuses.
* fix(scanner): let retirement stabilize a rolled-forward pause backlog
Follow-up to the defect E roll-forward for rustfs/backlog#2873.
- Retirement no longer waits for a scanner writer. When no commit
survives and the native selector over every read member (survivors,
retiring source and queued pools) rolls forward, the planner commits and
stabilizes that ledger on the survivors under the writers' object fence.
The previous retryable handoff could exhaust its ~13.5 s budget long
before a writer re-claimed, and failed the decommission terminally.
- A writer's roll-forward refuses when a stored commit names a member that
is missing or outside its read, such as a retiring source excluded from
the writable set. Retirement, which reads that member, decides instead.
- A writer stabilizing a rolled-forward ledger writes missing members
first and stops at the first failure. A member that only read as missing
rejects the create, so no surviving proof can omit its newer record.
- Docs describe the actual roll-forward preconditions and the retirement
exit.
- Tests: retirement converges the wedge with no writer; a writer refuses to
roll forward past an unread source that holds the newest ledger and
retirement keeps it; a spurious missing read changes no record and the
next claim converges on the newest ledger.
* fix(decommission): create missing pause backlog survivors first
Follow-up to the defect E retirement roll-forward for rustfs/backlog#2873.
- Retirement creates survivors that read as missing one at a time, before
any stored survivor, and stops at the first failed create. A transient
create failure on a new pool's set no longer leaves every other survivor
holding a commit that names a still-missing member, which no later
selection could leave.
- A rolled-forward retirement first seeds a rollback proof that names only
the survivors that store a record; the commit still names every survivor.
- Retirement rolls forward only when the survivors cannot select a ledger
on their own or select the same one. Survivors that agree on a different
ledger keep the source-authority check.
- The scanner writer applies the same missing-first, stop-at-first-failure
order to every stabilization with a missing member, including a seed from
an old commit selected as the rollback proof.
- Tests: a failed new-pool create during retirement retries to convergence;
a spurious missing read cannot seed an old commit over a newer member;
the rolled-forward precedence over source stable records is pinned.
* docs(decommission): state the all-missing survivor limit of backlog roll-forward
* test(e2e): use the resumed decommission completion budget
* test(heal): exercise checkpoints written by rc.5
* test(heal): route upgrade fixtures through the storage facade
* fix(deps): update ctutils to address constant-time advisory
* test(lifecycle): gate recovery I/O while advancing test time
* fix(decommission): count settled entries once and back off repeated pauses
Follow-up to rustfs/rustfs#8824 for rustfs/backlog#2873 defect F.
- Count an entry's versions when the entry settles (copied and cleaned up,
or retained) instead of as each version copies. A partially moved entry
that is deferred, paused and resumed, or interrupted, no longer counts its
already-moved versions again when the resumed pass finds them on the
target.
- Hold the watchdog relaunch of a decommission that pauses again on the
same cause without progress, doubling from 30 s up to 15 min, and warn with
the cause. A first pause still relaunches on the next pass; the pause never
becomes a terminal failure.
- Log each drive's error when a delete version loses write quorum, so the
cause of a sub-quorum delete-marker copy is recorded instead of only its
rollback.
- Tests: pause/resume counting, a target whose metadata is below read
quorum, the copy-quorum pause cause, the relaunch backoff, and a rollback
test that asserts the restore attempt instead of a tracing log.
* fix(decommission): measure pause progress by cleaned entries
Review follow-up on the pause/resume counting and relaunch backoff.
- A pause now carries this generation's progress counters into the
pool metadata it persists, like a progress checkpoint. Before, everything
counted since the last checkpoint was dropped at the pause.
- A retained entry (one whose source stays) is relisted by every resumed
pass. A per-instance ledger counts its versions once per decommission
generation, so retained versions no longer grow per round in one process.
- The relaunch backoff measures progress by entries moved off the pool and
completed buckets, which a relisted retained entry cannot advance. Before,
the recounted retained entry looked like progress and reset the backoff.
- Tests: exact totals across a pause and a resume from persisted metadata;
a target kept below quorum defers the entry with nothing counted; a
retained entry plus a repeated cause escalates the backoff over three
pauses, with the watchdog path relaunching a first pause and holding a
repeat.
* fix(ecstore): bound retained decommission accounting
Brings in #8825, #8831, #8836 and #8839. Conflicts were import lists and test blocks (both sides kept), plus the new restore_copyback request flag next to main's data-movement field. Main replaced completed_multipart_object_part with prepare_complete_multipart_parts, so the stale import is dropped.
Main's xl.meta fast-path scanner (inspect_v2_object_slice) predates #8836 and classified a transitioned version with a pending v1 restore copyback as having no physical data dir, which made find_unshared_data_dir_for_version drop the copyback data dir. The scanner now reads the copyback marker with get_consistent_bytes semantics (both aliases, ASCII case-insensitive, empty or conflicting values count as absent), and uses_data_dir keeps main's allocation-free transition check plus #8836's copyback exclusion. find_unshared_data_dir_for_version_keeps_pending_copyback_data_dir failed on the plain merge and passes now.
* feat(scanner): configure startup recovery backoff cap
* test(scanner): compare recovery retry count without narrowing
* docs(scanner): document recovery backoff cap and expose it in Helm
List scanner.recovery_backoff_cap in the runtime controls table, drop the
fixed ten-minute wording from the follower recovery section, and map
config.rustfs.scanner.recovery_backoff_cap_secs to
RUSTFS_SCANNER_RECOVERY_BACKOFF_CAP_SECS in the chart.
* fix(quota): retire degraded floors from durable usage reports
* docs(quota): clarify retained usage cache refresh behavior
* fix(admin): report unknown bucket usage as null in GetBucketQuota
GetBucketQuota serialized missing usage as size 0, which contradicted the
D3 rule that unknown usage is never reported as zero and, after retiring
degraded floors, made a bucket near its limit look empty. Make size
optional and stop collapsing a missing or unreadable usage snapshot to 0.
* fix(scanner): recover dual corruption with retained epoch fencing
* fix(scanner): preserve recovery markers and legacy quota floors
* test(scanner): expect retained usage recovery marker state
* docs(scanner): register dual-corruption recovery evidence objects
Add the .bloomcycle.bin.dual-recovery/ evidence family to the persisted
objects table and describe its owner, write-once lifecycle and recovery
rule in Deletion And Recovery Rules.
Chris An