mirror of
https://github.com/rustfs/rustfs.git
synced 2026-11-02 09:04:14 +00:00
e1986de4071fbef06fa7c96e67011663cd134676
613 Commits
| Author | SHA1 | Message | Date | |
|---|---|---|---|---|
 Hauser
|
e1986de407 |
chore(ecstore): integrate batch diagnostics for request stage work
Retain the reviewed diagnostic dependency while starting request-stage work from current main. This integration is not a performance qualification. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
7a3bf8cdbd |
fix(ecstore): resolve dangling latest versions in metadata updates (#9009)
* fix(ecstore): select the latest version at read quorum like MinIO GET and HEAD escalated latest-version selection to write quorum when enough metadata answered and skipped a newer version below write quorum on degraded reads. LIST and heal already pick the newer side of a split write at read quorum, so such keys listed but answered 503 until heal. Select with MinIO's getObjectFileInfo rule: the most common mod time that reaches the object's data blocks wins, ties go to the newer time, and one content identity at that time must reach the same quorum. Two identities tied at the selected time fail instead of resolving by hash order, and the selected identity's version-stack shape follows disk order. * fix(ecstore): resolve selection ties by disk order and guard writes Review of the read-quorum selection found three gaps: - Rejecting a same-time half split made explicit-version reads of a partially applied tagging update fail and blocked the retry that repairs it. Resolve ties to the identity on the lowest-index disk, for reads and heal alike, instead of rejecting or using map order. - Delete markers and empty versions can lower the common read quorum below the selected version's data blocks, so HEAD could report a version GET cannot decode. Require that many copies, as MinIO's getLatestFileInfo does. - A selection can now return exactly read-quorum disks, and partial metadata writes are not rolled back. Refuse an update before writing when its target disks cannot reach write quorum. * fix(ecstore): count identity copies in the data-blocks guard The first-candidate shortcut counts byte-identical copies, so copies that differ only in identity-neutral fields such as replication status were missing from the data-blocks check. When the shortcut's count falls short, count every copy of the selected content identity before refusing the version. * fix(ecstore): keep commit-proof selection for system-bucket reads Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an ordinary read of pool.bin as proof that the record committed. Under the read-quorum rule, a COMMIT that reached only half of a 2+2 set was served as committed, and the pool_meta_native_physical_* integration tests failed. Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's read-quorum rule, while is_meta_bucketname reads use the previous algorithm unchanged as select_committed_fileinfo, so a split stays unreadable until native recovery resolves it. * fix(ecstore): serve the latest readable version past dangling versions A failed versioned PUT whose rollback also failed can leave the new version on top of fewer drives than its data blocks. Reading only each drive's top version then left the committed version below it short of read quorum, so GET and HEAD of the latest version answered 503 although that version is intact in every stack. MinIO resolves the stacks in pickLatestQuorumFilesInfo. When every drive answered and an S3 latest read still cannot select a version, walk the newest version headers of each stack through exact-version reads: serve the readable version with the newest selected modification time, skip versions whose exact read proves them dangling (the NoSuchVersion an explicit GET reports) and queue heal for them, and keep the read-quorum error on any other outcome. Candidates are ordered by each version's newest modification time across drives, and at most eight exact reads are spent. Test PUTs now wait for every drive so tests that edit one drive's metadata no longer race the PUT's background tail writes. * fix(ecstore): resolve dangling latest versions in metadata updates A failed versioned PUT whose rollback also failed can leave the new version on fewer drives than its data blocks. GET and HEAD of the latest version already look past such versions, but read-before-write callers without a version id still answered 503: object tagging read each drive's top version with the quorum-error read, and put_object_metadata and the CopyObject source read selected from read_all_xl, whose merged latest version is missing on the other drives, so object_quorum_from_meta failed before selection. Object tagging now reads through a variant that runs the same walk past dangling versions as a latest GET while keeping the quorum error for a dangling key. put_object_metadata and the CopyObject source try the walk when their quorum or selection step fails with read quorum, before the existing dangling cleanup could report a key with a committed version absent. A version-only copy takes the served version's own copies and write quorum. The walk still applies only to S3 buckets, latest reads, and sets where every drive answered, and the update still targets only the served version's disks, so a version below write quorum is refused before writing. * fix(ecstore): refuse version-only copies below write quorum up front Review of the dangling-latest metadata updates found three gaps: - A version-only CopyObject wrote the new version and reverted on a write-quorum miss; refuse it before writing when the served version's disks cannot reach its write quorum, like update_object_meta_with_opts. - Replication status write-back, including the zero-write ValidateOnly mode, now keeps its read-quorum error and never walks past dangling versions, since the walk queues heal for the versions it skips. - The erasure-coding doc overstated that these callers never report a dangling key as absent; put_object_metadata and CopyObject keep their existing dangling cleanup when the walk finds no readable version. |
||
|
Hauser
|
32be316535 |
chore(ecstore): merge main into batch diagnostics
Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
f879946ead | chore: remove redundant review skills (#9022) | ||
|
Chris An
|
d88f524d9f |
fix(ecstore): serve the latest readable version past dangling versions (#9004)
* fix(ecstore): select the latest version at read quorum like MinIO GET and HEAD escalated latest-version selection to write quorum when enough metadata answered and skipped a newer version below write quorum on degraded reads. LIST and heal already pick the newer side of a split write at read quorum, so such keys listed but answered 503 until heal. Select with MinIO's getObjectFileInfo rule: the most common mod time that reaches the object's data blocks wins, ties go to the newer time, and one content identity at that time must reach the same quorum. Two identities tied at the selected time fail instead of resolving by hash order, and the selected identity's version-stack shape follows disk order. * fix(ecstore): resolve selection ties by disk order and guard writes Review of the read-quorum selection found three gaps: - Rejecting a same-time half split made explicit-version reads of a partially applied tagging update fail and blocked the retry that repairs it. Resolve ties to the identity on the lowest-index disk, for reads and heal alike, instead of rejecting or using map order. - Delete markers and empty versions can lower the common read quorum below the selected version's data blocks, so HEAD could report a version GET cannot decode. Require that many copies, as MinIO's getLatestFileInfo does. - A selection can now return exactly read-quorum disks, and partial metadata writes are not rolled back. Refuse an update before writing when its target disks cannot reach write quorum. * fix(ecstore): count identity copies in the data-blocks guard The first-candidate shortcut counts byte-identical copies, so copies that differ only in identity-neutral fields such as replication status were missing from the data-blocks check. When the shortcut's count falls short, count every copy of the selected content identity before refusing the version. * fix(ecstore): keep commit-proof selection for system-bucket reads Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an ordinary read of pool.bin as proof that the record committed. Under the read-quorum rule, a COMMIT that reached only half of a 2+2 set was served as committed, and the pool_meta_native_physical_* integration tests failed. Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's read-quorum rule, while is_meta_bucketname reads use the previous algorithm unchanged as select_committed_fileinfo, so a split stays unreadable until native recovery resolves it. * fix(ecstore): serve the latest readable version past dangling versions A failed versioned PUT whose rollback also failed can leave the new version on top of fewer drives than its data blocks. Reading only each drive's top version then left the committed version below it short of read quorum, so GET and HEAD of the latest version answered 503 although that version is intact in every stack. MinIO resolves the stacks in pickLatestQuorumFilesInfo. When every drive answered and an S3 latest read still cannot select a version, walk the newest version headers of each stack through exact-version reads: serve the readable version with the newest selected modification time, skip versions whose exact read proves them dangling (the NoSuchVersion an explicit GET reports) and queue heal for them, and keep the read-quorum error on any other outcome. Candidates are ordered by each version's newest modification time across drives, and at most eight exact reads are spent. Test PUTs now wait for every drive so tests that edit one drive's metadata no longer race the PUT's background tail writes. * chore(ci): shrink the error(format!) ratchet baseline for rebalance The rebalance control and meta modules now have fewer Error::other(format!) call sites than the baseline records, which fails the shrink-only ratchet in Quick Checks. Record the smaller counts. * fix(ecstore): preserve canonical order in latest-version fallback |
||
|
Hauser
|
f4daab72b0 |
fix(ecstore): route batch diagnostics through storage facade
Keep the diagnostic implementation private to its owner and expose explicit feature-gated helpers through the existing storage boundary. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
f96c94f950 |
fix(listing): treat key markers as literal keys (#8978)
* fix(listing): treat version markers as literal keys * fix(listing): keep start-after and V1 markers literal * docs(listing): state cursor compatibility removal condition * fix(listing): resume protected table pages from raw keys * test(listing): keep protected pagination coverage in app layer |
||
|
Chris An
|
acebee4ea3 |
fix(ecstore): select the latest version at read quorum like MinIO (#8982)
* fix(ecstore): select the latest version at read quorum like MinIO GET and HEAD escalated latest-version selection to write quorum when enough metadata answered and skipped a newer version below write quorum on degraded reads. LIST and heal already pick the newer side of a split write at read quorum, so such keys listed but answered 503 until heal. Select with MinIO's getObjectFileInfo rule: the most common mod time that reaches the object's data blocks wins, ties go to the newer time, and one content identity at that time must reach the same quorum. Two identities tied at the selected time fail instead of resolving by hash order, and the selected identity's version-stack shape follows disk order. * fix(ecstore): resolve selection ties by disk order and guard writes Review of the read-quorum selection found three gaps: - Rejecting a same-time half split made explicit-version reads of a partially applied tagging update fail and blocked the retry that repairs it. Resolve ties to the identity on the lowest-index disk, for reads and heal alike, instead of rejecting or using map order. - Delete markers and empty versions can lower the common read quorum below the selected version's data blocks, so HEAD could report a version GET cannot decode. Require that many copies, as MinIO's getLatestFileInfo does. - A selection can now return exactly read-quorum disks, and partial metadata writes are not rolled back. Refuse an update before writing when its target disks cannot reach write quorum. * fix(ecstore): count identity copies in the data-blocks guard The first-candidate shortcut counts byte-identical copies, so copies that differ only in identity-neutral fields such as replication status were missing from the data-blocks check. When the shortcut's count falls short, count every copy of the selected content identity before refusing the version. * fix(ecstore): keep commit-proof selection for system-bucket reads Pool metadata's PREPARE/COMMIT recovery and identity epochs treat an ordinary read of pool.bin as proof that the record committed. Under the read-quorum rule, a COMMIT that reached only half of a 2+2 set was served as committed, and the pool_meta_native_physical_* integration tests failed. Route reads through select_fileinfo_for_read: S3 buckets keep MinIO's read-quorum rule, while is_meta_bucketname reads use the previous algorithm unchanged as select_committed_fileinfo, so a split stays unreadable until native recovery resolves it. |
||
|
Chris An
|
e452d837dd | docs(scanner): clarify incomplete census diagnostics (#9002) | ||
|
Hauser
|
cf5a9e0ba3 |
docs: define the dirty-journal metadata source contract (#8984)
* docs: define the dirty-journal metadata source contract Specify the single node-owned journal scope: verified bootstrap, physical fencing and tail completion, reader-first replay, exact new responsibility settlement and retained legacy unknowns. Explicitly distinguish the proposed contract from the existing asynchronous hint implementation. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * docs: require receiving peers to revalidate source claims Keep user metadata, decoded records and authenticated transport distinct from storage-verified dirty-journal authority. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
d4d416ed47 |
fix(deps): upgrade russh and replace vulnerable zstd codecs (#8969)
* chore(deps): upgrade russh to 0.65.0 and refresh crates Run two cargo update and cargo upgrade rounds from origin/main. Adapt SFTP public-key callbacks to PublicKeyOffer and VerifiedPublicKey, preserving explicit rejection and password-only authentication. Cover the rejection policy with a pinned-host-key SSH handshake test. Refresh Arrow, Parquet, Brotli, ZIP, and compatible dependencies. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(deps): route legacy zstd consumers to fixed codecs Replace the old registry implementation with a private version-constraint adapter backed only by zstd 0.14.3 and zstd-safe 8.2.0. Preserve legacy features and Avro, Arrow IPC, Parquet, and Kafka formats. Cover abandoned contexts, legacy decode errors, seekable bounds, stored frames, consumer interoperability, and resolved implementation versions. Record the adapter removal condition in the compatibility register. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
fca714efe8 |
fix(listing): omit sub-quorum versions from ListObjectVersions (#8964)
Version listings bypassed listing quorum: SetDisks::list_path returned early into list_versions_authoritatively, whose resolver seeds unread slots as DiskNotFound. With any one drive of the set offline, every version held by fewer than ceil(n/2) drives (for example a deleted version kept by drives that were offline during the delete) failed the whole page with ErasureReadQuorum, surfaced as S3 503. Route merged version pages through the listing-quorum branch of SetDisks::list_path, as MinIO's listPath does with requestedVersions = 0, so such versions are omitted instead of failing the page. Keep the set-level floor: fewer than ceil(n/2) online drives still fails the page. Set walks (tier reference proofs, tier-delete journals, pool maintenance) and the recursive prefix-delete scan act on absence, so they keep the authoritative per-version scan and fail closed. |
||
|
Chris An
|
65bd2d575b |
refactor(listing): make cursors plain keys and opaque V2 tokens (#8950)
* refactor(listing): make cursors plain keys and opaque V2 tokens RustFS inherited MinIO's metacache cursor tag and appended `[rustfs_cache:v2,id:<uuid>,src:walker,gen:live]` to every internal continuation marker, although nothing ever consumed the id: four parsers had drifted apart, V1 NextMarker had to be stripped on the way out, and ListObjectVersions leaked the tag onto the wire as NextKeyMarker. ecstore stops assigning `list_id` in `ECStore::list_path` and `SetDisks::list_path_result`, drops `append_list_cache_id_to_marker` and the `cache_id` parameters of the pagination helpers, and removes the unused `ListPathOptions::encode_marker`; internal markers are the plain last key, last common prefix or last raw scanned key. The API layer builds the V2 NextContinuationToken as base64 of the server-side cursor in one place and decodes it in one place; V1 NextMarker is the decoded cursor and the `public_list_marker` tag stripper is gone; NextKeyMarker is the plain key. Base64 of the plain key is what nodes from earlier releases already read, so a rolling upgrade needs no switch. Compatibility, registered as `listing-cursor-legacy-tag`: `parse_marker` and `list_marker_key` still strip a trailing legacy tag from incoming markers so a client that paginates across the upgrade keeps its position. * fix(listing): preserve literal cache-tag cursor boundaries |
||
|
Hauser
|
fea5a584c3 |
fix(storage): fence multipart and metadata recovery (#8919)
* perf(ilm): back off idle tier-delete journal recovery Use bounded idle backoff only after complete empty journal and dispatch scans, while preserving immediate notifications and the retry cadence for observed work. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reconcile interrupted pool metadata publication Bind pool metadata CAS and Undo to exact native predecessors. Recover an interrupted physical publication only from an ordinary committed authority and complete verified identity, current and backup evidence, then require normal V3 CAS and quorum readback. Preserve known-conflict gates, startup and heal lock ordering, targeted admission and the original kill-recovery deadline. Wake proof-ready recovery immediately and bound parallel capacity sampling without removing mutation fences. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(ecstore): reuse proven pool metadata durability Publish native rollback backups directly in their owned UUID directories. Reuse a one-shot data-durability receipt only for an exact hardlink to the pinned, fully synchronized head and root. Preserve new-head and directory synchronization, native predecessor CAS, conditional Undo and uncertainty gates. Keep unknown, copied, raw-parent, cancelled and failed operations cold. Add bounded diagnostics, sync-call regressions and path-isolated test hooks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(multipart): exercise selective RPC delay and peer loss Add authenticated method-entry barriers for metadata and lifecycle locks under e2e-test-hooks. Verify real dispatch, write quorum, prior part payloads, lock drain and incarnation isolation without changing production budgets. Record cancellation separately from normal exits so late filesystem completion cannot overwrite Drop evidence. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * docs: define multipart delay gates and receipt invariants Document RPC entry evidence, peer-loss and cancellation oracles, remaining incident attribution limits, and immutable head inodes required for durability receipts. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(readiness): expose missing runtime drive slots Report configured slots without a runtime handle separately from unavailable inventoried drives. Preserve quorum and probe status semantics. Wait for every coordinator to regain its complete live inventory before multipart RPC fault windows and single-attempt recovery oracles. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): retry committed promotion convergence safely Require peer reload convergence on every worker attempt, including unchanged retries. Recheck durable identity, replicas, namespace fence, generation and cancellation before allowing any data movement. Keep known committed promotions active on a verified propagation pause. Release precise owners, retain queued successors and reuse the existing watchdog. Preserve uncertainty gates and terminal operation behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): guard native metadata mutation aliases Reject protected RenamePart source and destination paths before I/O. Classify the protected namespace through the native path resolver, and require canonical conditional RenameData and Undo keys so aliases cannot create separate lease domains. Preserve ordinary multipart and canonical CAS behavior, warm durability receipts and zero-allocation canonical fast paths. Cover real alias, byte identity, receipt and normal mutation behavior. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): register Darwin multipart RPC delay gates Update distributed membership from the actual Darwin listing. Preserve the full-profile digest and leave Linux unchanged until its own listing is produced, as required by the contributor guide. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(multipart): fence native publication and recovery Publish multipart data through private hardlinks and atomically consume the prepared source head. Bind exact compensation capabilities and native recovery to the physical cohort, source generation and durable publication state. Require fresh terminal confirmation for native replay and preserve indeterminate data when no safe recovery witness exists. Release logical guards only after validated irreversible quorum success while retaining physical mutation leases, accounting and resource ownership through actual callbacks. Preserve local instance proof authority and historical Copy metadata without granting healed donors fresh publication authority. Keep general movement identity strict while preserving directory capacity compatibility in a separate checked comparison. Cover native fault and cancellation boundaries, zero-byte objects, real Copy, heal and migration, exact physical readback and distributed RPC delay. Retain all quorum and durability requirements and document rollout limitations. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): fix ratchet scan and Windows fault hooks Move native pool metadata inventory tests after production items so the error-format ratchet sees the existing call site. Preserve the baseline instead of accepting a false count decrease. Make the directory-barrier failure hook available on every test platform and exercise the parent-barrier failure/retry cases in Windows CI. Keep production synchronization behavior and all existing assertions intact. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(protos): remove redundant recovery request clone Move the baseline into its final struct update while retaining the earlier clone needed to construct the first request variant. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): align RPC delay gates with native abort * fix(storage): validate native source recovery and CI gates Preserve wire layouts while bounding native response sizes and fix the last-use recovery request clone. Avoid destination replay reads only for fully observed initial native sources; keep upgraded and recovered sources conservative. Cover real legacy publication after a cold restart. Align upgrade acceptance with the native mixed-version mutation pause and use exact upload-scoped test completion evidence before advancing time. Keep all original quorum, ownership, byte and deadline assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): verify native abort under partial admission Delay native cleanup at the exact upload root after all-participant closure. Verify quorum-only admissions retain every source byte on refused Abort and complete the same upload without creating a missing original participant. Keep sub-quorum orphans and their discoverable identities isolated while a fresh all-participant upload proves service recovery. Route shared constants through the existing E2E storage facade. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): avoid formatting a fixed invalid upload path Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): align native MPU fault recovery assertions Verify real source participants, preserved orphan bytes and same-upload completion after quorum-only admission. Require full physical inventories so an expected read-quorum error cannot hide unrelated leakage. Preserve ordinary quorum and recovery budgets, and document listing availability through the affected set. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): keep MPU tail deadlines on the wall clock Global Tokio time advancement can expire drive-health probes while real filesystem callbacks are pending. Use a short real drive deadline with the existing physical-entry and cleanup barriers, retaining the byte, lock, resource-permit and final-shard assertions. Fixes #8937 Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(storage): align native fixtures with instance ownership Open independent disks for the simulated capacity-loss instance, scope four coordinator journal disks without rebinding source authority, and probe configured namespace keys while retaining physical I/O hooks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): isolate native multipart future construction Construct optional native publication futures behind synchronous heap boundaries so ordinary rename callers do not carry their state machines. Preserve native guards and ownership without adding allocations to ordinary PUT, copy or heal branches. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(heal): assert coordinator journal scope after rejoin Check the exact original four-member journal map after every rejoin and manager stop while preserving all sixteen native data slots. Reconnect expansion fails the test rather than being filtered away. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): bound object repair future construction Construct the normal and FileCorrupt deep-retry state machines behind a synchronous heap boundary. Preserve guards and cancellation while bounding the shared rename and deletion call stacks without increasing test stacks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): qualify fixture disk identity checks Call the same DiskAPI method explicitly so the independently reopened disk identity assertion compiles under every target and feature combination. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): isolate native inspection from mutation state Select a dedicated read-only inspection future behind the existing single Box allocation. Preserve validation, source ownership and the canonical inspection body while keeping mutation state out of inline read callers. Reject misrouted operations with typed errors and retain source evidence. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): pause abort time after native cleanup Wait for durable source deletion on the wall clock before advancing the capacity lease clock at the post-delete barrier. Paused time can otherwise expire the barrier while native filesystem cleanup is still pending. Retain lease-loss injection, durable inflight-byte accounting and restart reconciliation assertions. Document the test-util feature required to select the native decommission fixtures in package-scoped regressions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): reclaim data from owned multipart undo Retain exact runtime-only Undo ownership through deferred snapshot cleanup. Revalidate every target metadata version and the retirement anchor, keep the delete claim through the physical callback, and remove the anchor after the payload barrier. Preserve generic cleanup refusal when metadata is missing. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): preserve unformatted bootstrap routes Keep explicitly unformatted local paths reachable by an in-flight elected formatter while quarantining their service slots and disk-ID membership. Own bounded legacy-read handles so the actual asynchronous creator future remains Send. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(s3): bound CopyObject tracing poll frames Construct the business future behind an outlined single-Box boundary and lazily instrument its handle. Preserve Send, Unpin, span timing and cancellation without embedding concrete business-state construction in the tracing poll. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): observe recovery after direct legacy bootstrap Seed all sixteen real previous-release sources before attaching the same-port method observers. Require exact per-peer disk-route Unimplemented responses with zero publication calls, retain byte and cleanup assertions, and remove superseded raw transport. Register the two new regressions from actual Linux and macOS listings. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): prepare the homogeneous previous-release fleet Reuse the existing release baseline before creating the genuine legacy multipart sources. Restart old processes with their normal budgets and require every drive through the reduced-redundancy probes before installing the mixed-version observers and current fault budgets. Keep standard EC:4, single-attempt calls, all mixed-version byte and reclamation assertions, and their original deadlines unchanged. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): isolate release compatibility from fault profiles Keep normal component configuration for the observed mixed-release rollout, which injects no RPC delay or loss. Retain the short component profile in actual method-fault scenarios and preserve the single-attempt 45-second probe and every typed error, routing, source byte and reclamation assertion. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(mpu): use normal shutdown for planned release replacements Use the existing bounded SIGTERM path during homogeneous release preparation and the observed mixed-version rollout. Preserve every multipart inventory, routing, quorum and byte assertion, and keep actual crash recovery on its original hard-kill path. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * revert: retain the verified release compatibility fixture Restore the published fixture after the local configuration and lifecycle experiments failed to resolve the macOS startup failures. Keep their diagnostic records separate from the delivered implementation. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(copy): bound inherent handler polling stack Build the inherent CopyObject future through a synchronous noinline factory at the existing trait await. This keeps its large state out of the trait wrapper's active polling frame during historical self-copy. Preserve cancellation, tracing and post-write notifications, with one additional allocation restricted to COPY. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): validate authoritative quota and native pool heals Require actual authoritative quota usage in the existing readiness loop instead of treating HTTP success or unknown usage as a numeric baseline. Keep the original quota assertions and convergence budgets. Allow the owning pool metadata CAS to republish into a fresh valid data directory while retaining exact version, layout, completeness and shard fingerprints. Require every node's real metadata/read/write readiness before accepting recovery. Keep ordinary object manifests and the original 180-second recovery deadline unchanged, with negative oracle regressions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): distinguish certified native Abort recovery Require native participants with a persisted full closure certificate to retire their heads and reclaim every surviving source on retry. Keep uncertified legacy authority below identity quorum conservative, while retaining actual partial-delete and committed-object byte checks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Hauser
|
c610d11e8f |
fix(storage): guard MPU recovery and startup RPC admission (#8934)
* fix(ecstore): preserve uncertainty after partial multipart abort Keep recovery callers from reporting NoSuchUpload while minority primary metadata remains. Retain all quorum and deletion checks. Exercise actual partial Abort and explicitly retained late RPC packets after caller retirement, staging cleanup and bucket recreation. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(ecstore): prevent retired multipart staging resurrection Require the upload root to exist before preparing recovery publication. Preserve rooted directory walks for nested bases on non-Linux Unix and cover missing-source native work, configured aliases and pinned mixed-version coordinator recovery. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): use IPv4 peer aliases for pinned mixed recovery Keep direct IPv6 S3 listeners behind IPv4 RPC proxies for the pinned previous release, whose peer health resolver rejects bracketed IPv6 hosts. Preserve signed authorities, ports and strict recovery assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): separate pinned startup from MPU fault budgets Use the old fleet defaults during bootstrap, then apply the original short fault budgets to the current replacement. Keep direct S3 URLs aligned with IPv6 listeners and preserve all quorum and physical cleanup assertions. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): seed legacy MPU through transparent transport Retire raw proxy connections before enabling method faults so legacy streaming setup stays byte-transparent and every later scoped mutation must traverse the fault barrier. Verify raw and adapted protocol authority, bodies and trailers. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(e2e): retain direct listeners for legacy MPU peers Advertise same-port localhost peers for the pinned resolver while retaining direct IPv4 S3 listeners. Exact scoped RPC hit counts still reject bypassed injection and all physical authority assertions remain unchanged. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(server): keep startup gRPC on the authenticated router Use the hybrid protocol predicate for startup RPC admission so early control responses retain production authentication and owner validation. Keep REST FullReady gating and pinned replay capability rejection intact. Add deterministic regressions, scoped diagnostics and bounded legacy fixture resources. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ci): verify MPU recovery across upgrade source versions Run the exact sixteen-drive mixed MPU case for every checksum-verified upgrade source, independently of generic compatibility failures. Require one listed test and retain per-version logs while preserving the historical distributed source pin and every recovery assertion. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ci): trigger upgrade MPU checks for fixture changes Include distributed fixtures and the RPC fault proxy in the upgrade workflow paths so test-only changes schedule the strict recovery gate. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(storage): verify resend owners across MPU finalization Hold real part-transaction RPCs while another coordinator completes or aborts the upload, with both live and cancelled client response waiters. Require full object bytes, terminal upload state, and all staging roots to remain consistent after the part owner drains. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
299609bb83 |
fix(quota): keep reads available without snapshot-wide copies (#8928)
* fix(quota): keep quota reads available during scans Use optional scanner usage for read-only quota statistics across all quota configurations. Preserve strict legacy write admission and durable reservation enforcement while keeping unknown usage distinct from zero. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(usage): avoid snapshot copies for single-bucket reads Project owned snapshots, scalar bucket sizes and refresh-only results through the same TTL-bounded cache loader. Move loaded snapshots into the cache without changing refresh serialization or generation fencing. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Hauser
|
017b85a5bf |
fix(ecstore): preserve inactive multipart abort authority (#8920)
Retain per-disk recovery metadata through partial publication and require write-quorum proof for completion replay. Keep completed uploads terminal and strengthen RPC recovery acceptance. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris An
|
b23723a51f |
feat(heal): add live node admission feedback (#8897)
* feat(heal): add live node admission feedback * test(heal): use storage facade in admission regression * fix(heal): preserve retryable node admission deferrals |
||
|
Chris
|
da7dc74147 |
fix(quota): retire degraded floors from durable usage reports (#8832)
* fix(quota): retire degraded floors from durable usage reports * docs(quota): clarify retained usage cache refresh behavior * fix(admin): report unknown bucket usage as null in GetBucketQuota GetBucketQuota serialized missing usage as size 0, which contradicted the D3 rule that unknown usage is never reported as zero and, after retiring degraded floors, made a bucket near its limit look empty. Make size optional and stop collapsing a missing or unreadable usage snapshot to 0. |
||
|
Chris
|
db55098aa6 |
fix(scanner): recover dual corruption with retained epoch fencing (#8833)
* fix(scanner): recover dual corruption with retained epoch fencing * fix(scanner): preserve recovery markers and legacy quota floors * test(scanner): expect retained usage recovery marker state * docs(scanner): register dual-corruption recovery evidence objects Add the .bloomcycle.bin.dual-recovery/ evidence family to the persisted objects table and describe its owner, write-once lifecycle and recovery rule in Deletion And Recovery Rules. |
||
|
Chris
|
47e1850d7e |
fix(scanner): preserve remote checkpoint coverage identity (#8827)
* fix(scanner): preserve remote checkpoint coverage identity * test(scanner): verify remote plan identity propagation * test(rpc): disambiguate scanner capability body collection |
||
|
Chris
|
4e88ebcf30 |
fix(heal): preserve typed replacement readiness deferrals (#8823)
* fix(heal): preserve typed replacement readiness deferrals * docs(heal): align retry compatibility markers with guards |
||
|
Chris
|
8af4070d19 |
fix(decommission): move the free version of an expired tier version (#8777)
* fix(decommission): move the free version of an expired tier version Decommission only queued the lifecycle expiry of a transitioned version, so a source cleanup that ran first deleted the version itself and appended its tier free version after the entry's last pass. The post-check then failed the decommission until an operator re-ran it. Apply the expiry of a transitioned version synchronously, as rebalance does. Its free version is on the source before the cleanup preflight, which reports the entry changed, and the retried entry moves the record to a target in the same pass. An expiry that cannot be applied leaves the version live, so it moves like any other version. Refs rustfs/backlog#2873 * fix(decommission): move a delete-all expired tier version live Applying a delete-all expiry inline made its prefix delete wait for the bucket lifecycle write lock behind the entry's own read lock until the lock timeout, blocking every write in the bucket meanwhile. Move such a transitioned version live and leave the delete-all to lifecycle on the target. Cover a genuinely unapplied expiry and a free-version worker that consumes the source record before the entry moves it. Refs rustfs/backlog#2873 |
||
|
Hauser
|
c1de42af2c |
fix(heal): type legacy MRF decisions and defer diagnostics (#8778)
* fix(heal): explain unverified legacy heal outcomes Preserve storage diagnostics in canonical unknown outcomes and explain legacy payload identity limits through the shared local/peer status path. Borrow expected receipt identities while preserving all proof checks. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): keep diagnostics from pausing merged MRF repairs Require the actual MRF executor before publishing legacy pause events. An ordinary queued owner can retain a merged durable MRF notice without having performed the specialized deep check. Preserve its diagnostic without granting it authority to park the repair responsibility. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): type MRF outcomes and defer status diagnostics Carry storage-owned legacy observations in process and return a typed terminal result only after the qualified MRF Deep check. Publish pause events from that result and exact identities instead of source or text. Keep object names out of retry classification and compute only the status diagnostic selected by the canonical execution state. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris
|
d2eda195e6 |
fix(decommission): defer source cleanup quorum failures (#8760)
* fix(decommission): defer source cleanup quorum failures A quorum failure while deleting the source of an already copied entry was fatal and marked the whole pool failed, although the copied source is still in place and an uncommitted delete transaction is settled by delete recovery. Carry the cleanup error typed and resolve a quorum failure as Deferred: the entry is replayed at the end of the round, and a deferral that persists takes the existing capacity-blocked pause path instead of a terminal failure. Other cleanup errors and quorum errors from other stages stay terminal. Add set-level regressions for parent and child keys (`a/b`, `a/b/c`): parent cleanup leaves every child file untouched, and parent cleanup reaches quorum while the child is cleaned up or rewritten concurrently. * test(ecstore): keep source cleanup errors typed in prefix cleanup tests Return SourceCleanupError from the test helper instead of formatting it into Error::other, and shrink the error(format!) ratchet baseline for core/pools.rs, which lost one call site in the cleanup failure change. * fix(decommission): retry cleanup quorum failures and count replays once Retry a source cleanup quorum failure in place, up to three attempts with the existing source-cleanup backoff, before deferring the entry. Carry the versions an entry already counted on its deferral, so the end-of-round replay does not add a fully counted entry to items_decommissioned and bytes_done again. A deferral that persists pauses the pool through the existing deferred drain path. When source cleanup caused it, persist a reason led by "source cleanup blocked" without the carrier error's free-drive wording, and report waitingReason "source_cleanup" instead of "capacity". Add a source_cleanup test fault stage and a routine test that drives a cleanup quorum fault through deferral and replay to completion. * fix(decommission): wait seconds between cleanup quorum retries Give the in-place source cleanup quorum retry a fixed floor of one and then two seconds, with the existing jitter on top, so it can ride out a drive reconnect or health-check blip instead of retrying within a few hundred milliseconds. The wait stays bounded and cancel-safe. Find a source cleanup pause reason through stage wrappers, as the capacity-blocked check does, and add a routine test where cleanup keeps losing quorum: the decommission pauses with a source cleanup reason, the bucket stays queued and the source stays in place. Document the additive waitingReason value source_cleanup and its compatibility with older nodes reading the same pool.bin. * fix(decommission): never retry a lost source cleanup fence The cleanup retry classifier used is_quorum_error(), which also matches NamespaceLockQuorumUnavailable. The delete commit fence reports a lost namespace lock or mutation fence with that variant, so a lost fence was retried in place and only stopped later at the pre-cleanup fence check. A lost fence means this worker no longer owns the key; it must stop without retrying. Retry and defer only drive quorum failures: ErasureReadQuorum, ErasureWriteQuorum, InsufficientReadQuorum and InsufficientWriteQuorum. --------- Co-authored-by: Hauser <housemecn@gmail.com> |
||
|
Chris
|
77a36d92a3 |
fix(decommission): let a failed start roll back its own activation (#8752)
* fix(decommission): let a failed start roll back its own activation A fresh decommission start whose peer reload failed could not roll back: the pool metadata stale-write guard from #6604 refuses any save that clears a persisted active or queued decommission, so the pool stayed active with no worker until the watchdog or a leader restart picked it up. The rollback save now carries the activation that start committed. The guard exempts a pool only while its persisted record still equals that activation field for field, so a worker checkpoint or a later start keeps the rejection. A canceled record is not an active decommission and is cleared by the rollback as well. Every other save path passes no activation and is guarded as before. A peer may already have reloaded the activation and started a worker before the rollback. Saves are now fenced by decommission generation: a write carrying a start_time is rejected when the persisted record belongs to a later start, or has no decommission state and was updated after that start_time, so such a worker cannot make the pool active again. Its queued promotion and progress checkpoint also adopt the persisted record and stop the worker. Refs rustfs/backlog#2873 * fix(decommission): fence terminal writes of a withdrawn decommission Terminal records carry no start_time, so the generation fence did not cover them: after a start rollback, a peer worker of the rolled back generation that failed (for example on a bucket completion) could still persist a failed record over the restored pool. The merge now rejects a failed, canceled or complete record over a persisted record without decommission state; every attempt persists decommission state first, so no legitimate path writes a terminal record there. Peers also drop a withdrawn decommission sooner. A pool metadata reload replaces a running decommission with a newer record that withdrew it, even under an active worker; bucket completion adopts such a record like promotion and progress checkpoints do; and a withdrawn progress checkpoint stops the worker instead of logging a failed save and moving more entries. Tests cover the peer failure mark, a rollback retry that sees a worker checkpoint land between attempts, and the reload withdrawal. A heal fixture that wrote complete records over a fresh pool.bin now persists the active attempt first. Refs rustfs/backlog#2873 * fix(decommission): fence peer writes after a retry start rollback A retry start replaces a failed or canceled record that keeps its unresolved ledger, so its rollback restores that terminal record rather than clearing the pool. The withdrawal checks only recognized a record without decommission state, and the generation fence ignored a persisted record without a start_time. A peer worker of the rolled back retry could therefore checkpoint the pool back to active. The generation fence now compares the persisted last_update whenever the persisted record has no start_time of its own (cleared, terminal or queued). The progress checkpoint also checks the generation against the record read under the pool metadata lock, as bucket completion already does, and the save-path withdrawal check treats a terminal record newer than the running view as a withdrawal, so the peer adopts it and stops. A reload still never lets a newer terminal record replace an active worker's view, so after a retry's rollback a peer worker stops at its next bucket completion or progress checkpoint rather than on reload. Tests cover the retry rollback against a peer checkpoint and direct write, and a peer bucket completion after a fresh start's rollback. Refs rustfs/backlog#2873 * fix(decommission): keep a retry rollback's restored record intact After a retry start rolls back, the restored record is failed, so a peer worker's own failure mark passed every merge guard and replaced it. A failure mark now saves through save_current_pool_meta_checked, which runs the withdrawal check against the record read under the pool metadata lock and adopts that record instead of writing. The withdrawal check compared the restored record with the worker's in-memory last_update, which a rejected save had already raised, so the peer never adopted the rollback and the watchdog kept resuming it. It now compares against the running generation (start_time), and a rejected unresolved-entry save undoes its in-memory change. save_current_pool_meta_for_decommission_start is also compiled for the test-util feature again; a test-util helper still calls it, which broke the workspace clippy and Connect builds. Refs rustfs/backlog#2873 * fix(decommission): keep an unresolved entry when its own save fails The rollback of a rejected unresolved-entry save also dropped the entry on transient failures such as a quorum or lock-fence error, so the following failure mark persisted a failed record without it and an operator could clear a decommission that skipped an unreadable object. Keep the entry and undo only the last_update bump, which is all a withdrawn worker needs to still see the withdrawal. Refs rustfs/backlog#2873 |
||
|
唐小鸭
|
444eb5ddba |
fix(scanner): admit dirty scope across the bucket-listing window and pace full-walk fallbacks (#8690)
Under sustained writes nearly every cycle fell back to a full walk: scope admission required the global dirty generation to stay unchanged across the bucket listing, and after a Complete cycle the dirty-usage wake ran the next full walk immediately, back to back. - Track dirty-map removals with a process-local removal generation and admit a scoped walk when no entry was removed or replayed during the listing (`covers_listed_pending`). Writes to listed buckets in the window join the start snapshot; unlisted buckets dirtied after the cutoff stay pending and are never cleared by compare-and-remove certification. The strict `covers_all_pending` still gates segment-reuse evidence. - Classify scope fallbacks (removal during listing, generation exhausted, dirty-bucket overflow, no listed dirty bucket, baseline unavailable, peer scope unavailable), log `scoped_scan_fallback`, and report them in usage_freshness as additive serde(default) fields. - Pace the cycle after a Complete fallback while dirty buckets remain on the capped retry schedule clamped to the cycle interval, with the dirty wake disabled; the paced timer wake still requests a scope. - Update the per-bucket convergence design doc. No persisted format or internode protocol change. Co-authored-by: Chris <anzhengchao@gmail.com> |
||
|
GatewayJ
|
4dba635d48 | fix(table-catalog): authorize warehouse location claims (#8736) | ||
|
Chris
|
7cc4a9dbd0 |
fix(ecstore): leave no source free version for a moved tier version (#8732)
* fix(ecstore): leave no source free version for a moved tier version Data-movement source cleanup deletes each source version separately and never set skip_tier_free_version, so every transitioned version that decommission or rebalance had already copied to a target pool left a new tier free version on the source. Decommission then failed its post-check for any pool with tiered objects; rebalance left the record behind. The record claims the remote object the target copy still reads and stays blocked behind that live reference. Decommission and rebalance now pass the identities of transitioned versions whose copy committed on a target, and the cleanup skips the free version for exactly those. A transitioned version that did not move, such as one lifecycle expired during the move, still gets its free version because that record is its only remote-delete owner. Refs rustfs/backlog#2873, rustfs/backlog#2872 * fix(ecstore): keep the free version of a superseded moved source Data movement treats a source null version as complete when a newer unversioned object already sits on the target, without copying it. That outcome returned the same Ok as a committed copy, so a restored transitioned null version superseded this way was recorded as transferred and lost its source free version, leaving its remote object with no owner. The copy paths now return DataMovementCopy::Committed or ::Superseded, and decommission and rebalance record only committed copies of transitioned versions. A superseded source keeps its free version. Tests cover the superseded restored null version for rebalance and decommission, a restored and a null transitioned version moving by rebalance, an entry with one lifecycle-expired and one moved version, the per-version skip with two versions of one key, and a directory object. The tiered stale-target routine test from #8724 now expects the decommission to complete. The ILM ownership contract and the decommission notes describe the moved, superseded, expired and legacy cases. Refs rustfs/backlog#2873, rustfs/backlog#2872 * test(ecstore): build the decommission ILM fixtures without s3s The superseded and expiry-mix decommission tests named s3s request and lifecycle types directly, which added pools_test.rs to the ecstore files that reference s3s and failed the footprint ratchet. Set the restore request through ObjectOptions, and store the lifecycle rule as bucket metadata XML and load it with get_expiry_configs, as the decommission routine does. * test(ecstore): pin restored decommission copies and multipart supersession Decommission recorded a committed data copy of a restored transitioned version without any test failing if the record were dropped; add a routine-level decommission of a restored version that checks the post- check passes, the source keeps nothing, and the target keeps the full remote tuple. Add the multipart variant of the superseded restored null version (rebalance), and a unit test that separates committed, superseded and mismatched targets in the overwrite resume adjudication. |
||
|
Chris
|
5313b72baa |
fix(decommission): run the leader-side start to completion (#8723)
* fix(decommission): run the leader-side start to completion A forwarded decommission start awaits `ECStore::decommission` inline in the RPC handler. When the forwarding node's RPC times out, the stream is reset and the handler future is dropped wherever it is. Dropped after `pool.bin` records the pool as active but before the peers reload and the worker is spawned, the start left the pool active with no worker and peers on the old metadata, and a retry was rejected as already active. An admin client that disconnects can do the same. Run the start in its own task and let the caller await only its result. The start then always reaches its end: it fails before persisting anything, or persists the pool, has the peers reload and spawns the local workers, rolling back if the reload fails. A retry still cannot start twice: it waits on the start gate held by the running start and either reports the busy gate or is rejected because the pool is active. `decommission` now takes `self: &Arc<Self>` and spawns workers on that store instead of looking up the global object store, which is the same store in production and makes the path testable; the unused `require_decommission_store` helper is removed. Refs rustfs/backlog#2873 * fix(decommission): detach the leader-side start from its caller |
||
|
overtrue
|
a55f2f671b |
fix(decommission): keep status reads and starts off the movement gate
The data-movement operation gate is a write-preferring RwLock. A scanner publication lease stores a read guard on the leasing node, and the coordinator's lease-scoped rename, delete and validation on that node queued a second read. A movement writer queued in between (a decommission start, a status refresh, a peer reload) then waited for the lease while the coordinator's second read waited for the writer, so both stalled until the 60 s lease TTL. Admin status reads also took the movement writer, so every status call waited for the slowest in-flight copy or cleanup and stalled new movement reads while it queued. - Lease-scoped target admission shares the read admission the lease already holds instead of queueing a new read. - Pool and rebalance status reads publish under the gates only when they are free; otherwise they merge the persisted metadata into a private copy and publish nothing. - A decommission start bounds its start, activation and movement gate waits with one 30 s budget and fails with a retryable lock timeout before persisting anything; the admin handler maps it to 503 with Retry-After. Refs rustfs/backlog#2873 |
||
|
Chris
|
6f582b0548 |
fix(heal): decide object absence from storage evidence, not source (#8697)
* test(heal): add failing regressions for a source-independent absent verdict Add the regression tests for backlog#2900 that fail on the current code, where the same object-level not-found is treated as deleted on the plain heal path but as recreate_failed on the recreate path, and where the erasure-set healer, the task predicates and the error helpers disagree on wrapped not-found and transient inputs. Also add the per-object disposition cache module with its unit tests, the manager admission tests for it, the compat register entry for the display boundary, and the admission-invariant notes. The module and the manager tests are not declared yet, so nothing is wired into the build. * fix(heal): decide object absence from storage evidence, not request source Replace the five heal error classifiers (erasure-set healer outcome mapping, task-level not-found and lock predicates, error.rs recovery helpers and the storage existence-check matcher) with one typed classify_heal_error. An object-level not-found from the locked storage read is Absent for every source and every recreate_missing value; the recreate path, the usage-observation and scanner object-dir whitelists are gone, and display-text matching survives only in the marked classify_error_display boundary. Add a bounded, expiring per-object disposition memory consulted before queue admission so a best-effort producer cannot requeue an absent or terminally failed object on every tick; durable MRF, admin, forced, high-priority and dry-run requests bypass it. The read-repair producer no longer forces recreate_missing. * fix(heal): keep rendered grace windows out of task retries A grace window rendered into a task message belongs to an attempt that already finished; its owner schedules the next one. Only typed grace errors replay the task, as before the classifier merge. Update the ecstore read-repair black-box test: the producer no longer forces recreate_missing. * fix(heal): classify ambiguous legacy delete residue as permanent explicitly * fix(heal): scope disposition memory and bound absence parsing |
||
|
Chris
|
f063a8e862 |
fix(readiness): hold startup readiness until peers re-admit the node (#8704)
A restarted node published FullReady from its own view while survivors still counted its drives unavailable (stale DNS for the old pod IP, then the Offline->Returning->Online hysteresis). A rolling update could then stop the next pod and cost a set its read quorum (#8669). Add a read-only PeerDriveAdmission internode RPC that reports how a peer sees another host's drives, and gate the startup FullReady transition until a strict majority of each erasure set's peers count this node's drives online. The gate shares the startup budget, releases after 15s of unreachable peers, skips immediately when older peers cannot answer, and can be disabled with RUSTFS_STARTUP_PEER_READMISSION_ENABLE=false. |
||
|
Chris
|
9a1f3d2c4c |
fix(quota): keep ledger-admitted writes independent of scanner usage (#8699)
Hard quota admission is owned by the durable reservation ledger (decision D3, rustfs/backlog#2872), but the request-path quota check still read scanner usage on every write and mapped `UsageUnavailable` to a retryable 503 even for buckets the ledger admits. For a durable quota the checker now treats scanner usage as reporting input: an unavailable lookup yields `current_usage = None` and `allowed = true`, counted and logged at debug level, and admin quota stats report the usage as null instead of zero. Buckets without a reservation protocol keep the fail-closed snapshot path. Quotas saved before the reservation protocols are re-saved under protocol v2 with the same limit, type and created_at. The write goes through a new guarded save that keeps the admin save's lifecycle/transaction guards, incarnation check and fleet capability proof and additionally refuses when the quota revision moved since the cached read, so a concurrent quota edit wins. It is triggered lazily from the quota read path (background, one attempt per bucket with a retry interval, never blocking the read) and in bulk by `POST /rustfs/admin/v3/quota-migration`, which reports every bucket. The ADR's Consequences section inventories the remaining scanner-usage "authority" consumers with their meaning after D3, retirement condition and compatibility window; the operations guide documents the migration and the report-only usage semantics. Refs rustfs/backlog#2904, rustfs/backlog#2872, rustfs/rustfs#8678, rustfs/rustfs#5716, rustfs/rustfs#7350 Co-authored-by: Hauser <housemecn@gmail.com> |
||
|
Chris
|
7c75f319bc |
fix(scanner): stop treating an ordinary PUT as data movement in cycle termination (#8696)
* fix(scanner): stop treating an ordinary PUT as data movement in cycle termination A continuously written deployment never reached a `Complete` scanner cycle (#8494, #8305, #7108). Three places in the termination path vetoed the cycle for an ordinary PUT; each now lets the write shade its bucket, the path per-bucket certification (#8542) built, and keeps the data-movement fences. 1. Usage publication is fenced by data movement only. `scanner_data_usage_publication_blocked` no longer includes `namespace_commits_pending()`, so the post-walk probe, the local defer decision, the post-lease recheck and the usage-store route probes stop superseding or deferring a cycle for a pending rename fanout. The scanner publication leases (acquire, validate, target guard) drop the pending-commit rejection and the namespace-generation staleness check and keep the movement generation, the operation gate, generation exhaustion and the purpose/target restriction. A test-util probe lets fixtures wait for tails directly. 2. The plan identity is split into a coverage identity and a reuse identity. `scanner_activity_coverage_digest` (peer set, protocol, topology, movement, and a remote peer's generations) is what a bucket checkpoint is started and resumed under, through `ScannerBucketScanPlan::checkpoint_coverage_digest`, `DataUsageCacheReuseOptions::checkpoint_plan_digest` and `prepare_bucket_checkpoint_with_coverage`. The reuse identity (`scanner_activity_structural_digest`) keeps `instance_id` and `maintenance_generation`, so a restart or a maintenance change still invalidates the baseline and the set caches; a completed sweep is stored under it. `publication_blocked` leaves every digest and stays a guardrail veto. A remote worker still resumes under the single digest on the wire. 3. A bucket walk that stops early no longer makes the cycle `Incomplete` when the bucket has a certified value. The set scan stands in with the bucket's flattened value from its previous complete set cache (`carry_partial_bucket_values`), the cycle reports it pending (`ScannerBucketScanStatus::PartialCarried`), keeps it in the dirty map, and the next scoped cycle walks it because the baseline's pending list joins the dirty buckets. A bucket with no certified value keeps the cycle `Incomplete`; failed buckets are unchanged. No persisted file, generation, receipt or environment variable is added. Refs #8494, #8305, #7108, rustfs/backlog#2903. * chore(ecstore): gate the namespace commit probe to test builds The scanner publication barrier no longer consults InstanceContext::namespace_commits_pending, so it has no production caller left; keep it for tests and the test-util facade only. * test(rustfs): observe namespace commit ownership directly in node_service tests The scanner usage publication barrier is movement-only now, so it no longer reports a pending rename. The node_service tests that assert a cancelled handler keeps its physical owner, or that wait for rename tails to drain, read the namespace commit state through a test-only ECStore probe instead. |
||
|
Hauser
|
05b483b31a |
fix(storage): preserve scanner checkpoints and bound MRF journal I/O (#8692)
* fix(scanner): preserve checkpoints on identity read failures Distinguish confirmed missing incarnation sidecars from lookup errors before cache preparation. Preserve durable checkpoint progress when identity reads are unavailable and report post-scan uncertainty separately from replacement. Reuse the transaction-fenced reader for optional and required lookups while retaining existing missing, malformed, system-bucket and lock-loss behavior. Verify preservation and recovery through a real coordinator with the identity sidecar write-locked, including unchanged cache revisions and dirty work. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * refactor(ecstore): share required incarnation conversion Keep the missing-incarnation error contract in one conversion used by both required lookup entry points. This removes a duplicate formatted error site and satisfies the existing error-format ratchet without changing its baseline. Pin the full missing-identity error text in the real SetDisks regression. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(heal): bound cross-disk MRF checkpoint I/O Run independent journal disks with bounded concurrency while retaining each replica's payload, companion and manifest ordering and the mirror phases. Share encoded bodies, deduplicate local roots and collect every result without changing receipt deadlines or persistence failure semantics. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Chris
|
f3cef98afc |
fix(scanner): anchor the bitrot cadence and budget per-disk I/O (#8701)
* test(scanner): pin Normal mode when no bitrot start is recorded Regression for rustfs/backlog#2905: three consecutive cycles that find no recorded bitrot start currently run Deep through the selection window, and a Normal cycle never records the cadence start. * fix(scanner): anchor the bitrot cadence and budget per-disk I/O Decide the scan mode from a recorded Deep window start only. A cycle that finds no recorded start runs Normal and anchors the bitrot cadence in process memory instead of opening the selection window; the Deep cycle that follows the elapsed cadence records the window start exactly as before. A missing record used to be read as a window that started at cycle 0, which ran Deep for deep_window_cycles cycles whenever the start could not be read back (rustfs/rustfs#8015). The recorded start is also remembered across a skipped or failed save, and /v3/scanner/status reports the cadence state under a new bitrot object. Add a per-local-disk background I/O budget (operations and bytes per second, defaults by detected media, auto|off|<rate>) that DynamicSleeper charges before directory listings and object metadata reads as a hard ceiling layered on the foreground-priority disk admission gate, with runtime config keys, per-disk status, cycle metrics, pacing pressure, usage_freshness, Prometheus series, Helm values, admin config help, and operator docs. Refs rustfs/backlog#2905, rustfs/rustfs#8015, rustfs/rustfs#8377. * test(scanner): expect Normal retries on erasure sets without a bitrot start The pending-PUT retry fixture asserted that a 4-drive set revisits every bucket because the cycle ran Deep. Without a recorded Deep window the cycle now runs Normal on every layout and reuses the cold inventory. |
||
|
Chris
|
2534b6175b |
fix(scanner): startup gates back off, rebuild and run degraded (#8695)
* fix(scanner): give startup gates automatic exits instead of a pause Every fail-closed scanner gate now has a bounded automatic exit without deleting quarantine evidence or relaxing the leader, CAS, movement or usage-floor fences (rustfs/backlog#2901). - Recovery retries back off instead of pausing: after the short budget the observation reads `backoff`, stays retryable, names the next attempt and is capped at ten minutes; one success resets the count and a 24 h window restarts the budget (rustfs/rustfs#8297, #8670). The usage-floor failure no longer carries the retry count across attempts. - A corrupt `.bloomcycle.bin` is rebuilt on the iteration after its quarantine marker is durable: the fenced leadership claim CAS-replaces the quarantined revision under a higher epoch and the marker stays as evidence. `future_schema` and `cleanup-pending` remain blocked. - An unclaimable pause ledger runs the leader degraded: cycles run untracked behind the data-movement fence, `persistence_state` reports `degraded`, and the claim is retried at most every five minutes (rustfs/rustfs#8251). - The usage-state full-rebuild reset starts from the empty baseline when every usage slot is corrupt and no cycle state exists (rustfs/rustfs#8670, #6951). - A cycle-state save reconciles non-conflict failures against durable state and retries under the same epoch; an exhausted save reloads durable state in place. Only a newer owner or an unreadable primary ends the iteration, so peers see a changed fence only when ownership changed (rustfs/rustfs#8141). * test(scanner): align usage-state reset tests with the empty-baseline rebuild The full-rebuild reset no longer refuses corrupt or identity-less usage slots when no cycle state exists (rustfs/rustfs#8670). The two tests that encoded that refusal now check the new contract: the reset starts from the empty baseline with a bootstrap-pending marker, never an authoritative zero floor, and a valid backup still supplies the recovery floor. --------- Co-authored-by: cxymds <cxymds@gmail.com> |
||
|
Chris
|
90749fbb06 | ci: run feature lanes nightly and warm caches on a schedule (#8683) | ||
|
Chris
|
22740876b3 |
docs(architecture): record scanner usage authority and PUT-tail decisions (#8678)
Supersede the 2026-09-03 decision: hard quota admission is owned by the durable reservation ledger, and scanner data usage is best-effort reporting input. Record that a pending ordinary PUT commit is a write-barrier event rather than a usage-publication veto, and split the scanner plan identity into a coverage identity and a reuse identity. Refs rustfs/backlog#2872, rustfs/backlog#2903, rustfs/backlog#2904. |
||
|
Hauser
|
6681dd12e3 |
fix(scanner): expose mixed sweep checkpoint transitions (#8657)
* fix(scanner): expose mixed sweep checkpoint transitions Correlate checkpoint admission, incomplete sweeps, and successful terminal Partial persistence by bucket/source and cycle/leader epoch. Share the traversal-position predicate and local/remote save diagnostic without changing certification, acknowledgement, or persistence contracts. Extend real-disk cache regressions across cycles and leader changes and verify that failed terminal saves retain an older periodic frontier until a successful terminal save permits fresh certification. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * fix(scanner): report the aggregate incomplete sweep gate Expose the existing unresolved_objects decision in the incomplete-sweep event. Nested cache-entry failures can keep a sweep partial while both the root counter and the capped diagnostic failure map are empty. Extend the real-disk regression with diagnostics disabled and distinguish this case from mixed-only coverage without adding another cache walk. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
Henry Guo
|
ee534fa19b |
test(table-catalog): validate current backing contracts (#8559)
* test(table-catalog): qualify current backing contracts * fix(table-catalog): verify smoke cleanup after object rename * fix(ci): include the baseline writer test in full E2E selection * fix(ci): reserve test capacity for disk-backed lock probes --------- Co-authored-by: Henry Guo <marshawcoco@users.noreply.github.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Hauser
|
453be3dd84 |
fix(scanner): attribute incomplete census and lease waits (#8643)
* fix(scanner): attribute incomplete census and lease waits Report bucket sweep blockers and correlate partial remote work and rejected roots with their set and cycle. Keep distinct accepted bucket names in the result collector so completion counts and missing-bucket attribution share one source of truth, including during cancellation drain. Distinguish worker completion from coverage, retain peer/purpose context in lease failures without changing retry suffixes, and avoid claiming that an unverified leader fence changed or a waiting scan was complete. Document the census stages and preserve publication, dirty acknowledgement, remote failure classification and checkpoint safety rules. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(scanner): move set caches through the result collector Give the sole result collector exclusive ownership of the aggregate cache and return it together with the accepted bucket names after channel drain. Move that cache into complete persistence or incomplete updates instead of sharing a mutex and cloning the terminal cache. Return each accepted result's owned name from cache application so coverage tracking also avoids an extra name allocation. Preserve worker joins, cancellation drain, publication metadata and all certification fences. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> --------- Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
唐小鸭
|
222beaba00 |
fix(scanner): certify dirty usage per bucket and complete cycles with pending buckets (#8542)
* fix(scanner): treat replication status writebacks as status-class hits
A replication status writeback (persist_replication_state_if_current in
Update mode, including the heals the scanner queues during its own walk)
marked the bucket dirty on the namespace path: it advanced the dirty-usage
generation, woke the scanner, reached the durable dirty journal and
entered the activity snapshot. Any replicated bucket therefore made every
scanner cycle Superseded, the authoritative usage baseline was never
certified and usage freshness stayed stale for as long as replication ran
(#8375).
The writeback is now status class: record_status_class_mutation records
only the process-local producer coverage bit and nothing else. There is
no rescan queue for it, because a deployment with an active replication
rule already walks every bucket on every cycle
(ScannerMaintenanceFeatures::needs_regular_cycle) and that walk refreshes
replication_stats. The routing lives in a named, unit-tested function in
rustfs/src/storage/storage_api.rs; tier-expiration cleanup removes data
and stays on the namespace path.
No persisted format, protocol, admin API or configuration changes.
This is phase 1 of docs/architecture/scanner-per-bucket-convergence.md,
which this change adds together with its index entry.
Refs #8375, #8494, rustfs/backlog#2816.
* fix(scanner): refresh replication statistics during maintenance
Refresh compacted prefixes and completed bucket caches during replication maintenance. Bind set execution reuse to the observed replication statistics so a same-cycle retry cannot restore older counters.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* test(scanner): avoid expanding the legacy S3 DTO surface
Decode the existing replication configuration type from a JSON fixture instead of adding direct S3 DTO references to a new file. Preserve the compacted-prefix and same-cycle assertions and the footprint ratchet.
Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>
* feat(scanner): attribute activity guardrail vetoes per field
The post-walk cluster activity proof compared the whole activity snapshot,
so any namespace or dirty-usage generation movement vetoed publication.
Veto only on the fields that change which heap the walk described
(peer set, instance id, protocol version, topology, maintenance
generation, data movement, movement generation, publication blocked) and
name the field and host in the log and in a new usage-freshness counter.
Generation movement is a shade signal consumed per bucket. A remote peer
whose dirty-usage state moved keeps the whole-cycle outcome until scoped
acknowledgements can name its buckets.
* feat(data-usage): carry pending-bucket certification in DataUsageInfo
Add the additive fields a per-bucket certified scanner snapshot needs:
the sorted list and count of buckets whose dirty-usage generation moved
during the walk, and an explicit observational marker for snapshots whose
authoritative publication was vetoed. All three default when absent and
are skipped when empty, so older readers see the existing snapshot shape.
* feat(scanner): certify dirty usage per bucket and complete cycles with pending buckets
A structurally complete walk that the guardrail did not veto is now
`Complete` even when the dirty-usage generation moved during the walk.
Certification keeps the compare-and-remove semantics of
clear_dirty_usage_buckets: buckets whose generation still equals the start
snapshot are cleared, buckets written during the walk stay in the dirty map
and are published as the pending set, so usage_snapshot_converged becomes
`pending == 0`. The authoritative object is written for such cycles; the
observed route is selected by the explicit observational mark that vetoed
cycles carry instead of by the convergence flag, and the root publication
proof is read for every authoritative save. The post-lease activity recheck
compares the guardrail fields, with the peer dirty-usage fallback, instead
of the full snapshot digest.
* feat(scanner): accept unconverged complete publications as the cache baseline
complete_scanner_cache_baseline_plan_digest required the authoritative
snapshot to be converged, so a site whose cycles always left a bucket
pending could never scope a rescan and every cycle was a full walk. The
authoritative object is now a structural baseline whenever it carries a
convergence marker and no observational mark; bucket-cache reuse is then
governed by certification, because pending buckets stay in the dirty map
and are selected for the rescan while certified buckets reuse their set
cache. Observed candidates keep requiring the unconverged marker, and
segment reuse evidence keeps requiring a converged baseline.
Add the single-disk cycle test: a bucket written during the walk is the
only pending bucket, the others are certified, the publication is
authoritative and unconverged, and the next cycle is scoped to that
bucket; a held PUT tail still supersedes onto the observed route.
* feat(admin): expose scanner guardrail vetoes in usage freshness
Carry the attributed guardrail veto counter, timestamp and <field>@<host>
reason from the scanner metrics report into the madmin scanner status so
an operator can tell a rebalance or a PUT tail from a broken probe. The
fields are additive with serde defaults and merge like the deferral
fields.
* docs(scanner): record per-bucket certification in the design and changelog
Amend the convergence design where the implementation deviates: pending
buckets carry their walked value, maintenance_generation stays a veto,
guardrail localisation and the remaining phase-2 items ship separately,
last_cycle_result keeps its success label, and the admin surface gains the
guardrail veto fields. Update the publication contract for the post-lease
guardrail recheck and the explicit observational routing mark.
* refactor(scanner): drop the unreachable failed-bucket input from the pending set
A failed bucket never reaches a Complete cycle, so the pending set only
needs the start snapshot. Also remove a redundant clone in the data-usage
round-trip test.
* test(scanner): name the activity field mutation table type
* test(scanner): fill the post-walk gate in merged replication fixtures
The replication maintenance fixture merged from main builds a
ScannerCycleRequest literal and predates the test-only post_walk_gate
field; give it the default so the test target compiles on the rebased
branch.
* ci(nextest): run the admin target-repair fixtures on the ecstore base stack
repair_transaction_rejects_a_bucket_recreated_after_target_validation
overflowed the 2 MiB libtest default twice in a row on Linux CI for this
branch. Measured by sweeping RUST_MIN_STACK with the ci profile on macOS
arm64: main (
|
||
|
Hauser
|
f64cb816db |
fix(scanner): preserve checkpoint progress across object failures (#8642)
Separate completed traversal from persisted object health debt so leader handoffs reuse healthy subtrees and retry failures without losing progress. Fence old readers with a new receipt digest domain and preserve exact accounting under diagnostic eviction, compaction and punctuation ordering. Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
628663f468 |
fix(heal): recover replacements after pool removal (#8600)
* fix(heal): recover replacements after pool removal * test(e2e): gate upgrade probes on writable baseline nodes * test(e2e): register baseline readiness regression in smoke digest |
||
|
唐小鸭
|
d457378c02 |
feat(scanner): decouple bucket walkers from disk count (#8539)
* fix(scanner): treat replication status writebacks as status-class hits A replication status writeback (persist_replication_state_if_current in Update mode, including the heals the scanner queues during its own walk) marked the bucket dirty on the namespace path: it advanced the dirty-usage generation, woke the scanner, reached the durable dirty journal and entered the activity snapshot. Any replicated bucket therefore made every scanner cycle Superseded, the authoritative usage baseline was never certified and usage freshness stayed stale for as long as replication ran (#8375). The writeback is now status class: record_status_class_mutation records only the process-local producer coverage bit and nothing else. There is no rescan queue for it, because a deployment with an active replication rule already walks every bucket on every cycle (ScannerMaintenanceFeatures::needs_regular_cycle) and that walk refreshes replication_stats. The routing lives in a named, unit-tested function in rustfs/src/storage/storage_api.rs; tier-expiration cleanup removes data and stays on the namespace path. No persisted format, protocol, admin API or configuration changes. This is phase 1 of docs/architecture/scanner-per-bucket-convergence.md, which this change adds together with its index entry. Refs #8375, #8494, rustfs/backlog#2816. * fix(scanner): refresh replication statistics during maintenance Refresh compacted prefixes and completed bucket caches during replication maintenance. Bind set execution reuse to the observed replication statistics so a same-cycle retry cannot restore older counters. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(scanner): avoid expanding the legacy S3 DTO surface Decode the existing replication configuration type from a JSON fixture instead of adding direct S3 DTO references to a new file. Preserve the compacted-prefix and same-cycle assertions and the footprint ratchet. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * perf(scanner): keep raw enumeration bookkeeping linear per entry The page writer inserted into its building page with a push-and-sort on every entry and committed each page through a clone. Insert in sorted position and move the committed page into the index instead. Add a regression test that records 20,000 entries in one prefix and pins the cost to one hash per entry and well under a second, so the quadratic re-normalization removed by #8114 (rustfs#8113) cannot return once pacing builds on this walker. * feat(scanner): decouple bucket walkers from disk count Add the scanner key walkers_per_disk (RUSTFS_SCANNER_WALKERS_PER_DISK, 1 through 16, default 1). Each local coordinator disk of a set spawns that many bucket walkers; remote disks keep one. The walker count is what max_concurrent_disk_scans caps and the disk count stays the floor. The walkers of one disk share its scanner admission through LocalDiskAdmissionShare: the first walker to start a bucket takes the per-disk permit and the last to finish releases it, so a peer's remote request still sees one scan session per physical disk. Without the share the second walker was rejected as busy and retired. The disk_concurrency_budget log line reports walkers_per_disk, walkers and the reason for the effective limit. The default stays 1 because no local disk reports rotational media today; the design keeps a higher flash default as the intent. * docs(scanner): document walkers_per_disk in the Helm chart and configuration manuals RUSTFS_SCANNER_WALKERS_PER_DISK was only described in the scanner runtime controls table. Add it to the Helm chart (values, configmap rendering, README table, template test), the configuration crate's environment reference with operator guidance, and a short section in the operations manual on when to raise it and how to confirm it took effect. --------- Co-authored-by: Hauser <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> |
||
|
cxymds
|
ce00d8f7e5 |
feat(s3): support conditional bulk deletes (#8535)
* feat(s3): support conditional bulk deletes * fix(s3): report null marker identity in bulk delete responses * fix(s3): isolate conditional bulk delete actions and tier guards * fix(tier): preserve backend identity validation diagnostics --------- Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
唐小鸭
|
c9f88cff66 |
fix(scanner): treat replication status writebacks as status-class hints (#8509)
* fix(scanner): treat replication status writebacks as status-class hits A replication status writeback (persist_replication_state_if_current in Update mode, including the heals the scanner queues during its own walk) marked the bucket dirty on the namespace path: it advanced the dirty-usage generation, woke the scanner, reached the durable dirty journal and entered the activity snapshot. Any replicated bucket therefore made every scanner cycle Superseded, the authoritative usage baseline was never certified and usage freshness stayed stale for as long as replication ran (#8375). The writeback is now status class: record_status_class_mutation records only the process-local producer coverage bit and nothing else. There is no rescan queue for it, because a deployment with an active replication rule already walks every bucket on every cycle (ScannerMaintenanceFeatures::needs_regular_cycle) and that walk refreshes replication_stats. The routing lives in a named, unit-tested function in rustfs/src/storage/storage_api.rs; tier-expiration cleanup removes data and stays on the namespace path. No persisted format, protocol, admin API or configuration changes. This is phase 1 of docs/architecture/scanner-per-bucket-convergence.md, which this change adds together with its index entry. Refs #8375, #8494, rustfs/backlog#2816. * fix(scanner): refresh replication statistics during maintenance Refresh compacted prefixes and completed bucket caches during replication maintenance. Bind set execution reuse to the observed replication statistics so a same-cycle retry cannot restore older counters. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(scanner): avoid expanding the legacy S3 DTO surface Decode the existing replication configuration type from a JSON fixture instead of adding direct S3 DTO references to a new file. Preserve the compacted-prefix and same-cycle assertions and the footprint ratchet. Co-Authored-By: heihutu <heihutu@gmail.com> Co-Authored-By: zhi22915 <qiuzgang@gmail.com> * test(ecstore): seed the ordinary-timeout peer fixture under the default drive deadline The one-second ENV_DRIVE_MAX_TIMEOUT_DURATION in target_ordinary_timeout_keeps_its_physical_namespace_owner wrapped the whole test, so the fixture's write_metadata and the final read_version were bounded by it too. On a loaded CI runner the seed took longer than a second and the test failed with "seed real metadata: Timeout" before the paused rename under test ran. Only the rename now runs under the one-second deadline; the fixture and the tail read use the default. --------- Co-authored-by: Hauser <housemecn@gmail.com> Co-authored-by: heihutu <heihutu@gmail.com> Co-authored-by: zhi22915 <qiuzgang@gmail.com> Co-authored-by: overtrue <anzhengchao@gmail.com> |
||
|
Henry Guo
|
22c093e66b | feat(table-catalog): add opt-in periodic maintenance queueing (#8537) |