fix(scanner): isolate corrupt cycle state

.agents/skills/adversarial-validation/SKILL.md

@@ -1,45 +1,277 @@
 ---
 name: adversarial-validation
-description: Review a final RustFS diff adversarially when the user requests adversarial review, the root AGENTS.md classifies the change as high risk, or a substantial PR is being reviewed. Do not use for ordinary questions, diagnosis, planning, status, documentation-only work, or routine low-risk implementation.
+description: Execute the Adversarial Validation policy from the root AGENTS.md — run the applicable reviewer roles with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, design proposal, or agent-instruction change that alters execution before declaring it done.
 ---
 
-# RustFS Adversarial Validation
+# Adversarial Validation Playbooks
 
-Use the risk tier and review shape defined in the root `AGENTS.md`. This skill
-routes a review to RustFS-specific probes without loading unrelated domains.
+The policy — risk tiers, role list, protocol, exit criteria — lives in the
+root `AGENTS.md` under "Adversarial Validation (Default On)". Read it first;
+this skill does not restate it. This file adds the RustFS-specific probe
+playbook for each role: concrete attacks, where they apply, and the real
+shipped bug or rule that earns each probe its place.
 
-## Select Lenses
+## How to run a role
 
-Read only the references required by the diff:
+1. Pick the tier and the applicable roles per the root `AGENTS.md`.
+2. Run each role as an independent pass over the final diff — a parallel
+   reviewer agent where the tooling supports it, otherwise a fresh
+   sequential pass that starts from the diff and the nearest scoped
+   `AGENTS.md`, discarding the writing session's assumptions.
+3. Within a role, execute the probes whose domain the diff touches, plus any
+   attack the diff obviously invites that no probe lists — the playbook is a
+   floor, not a ceiling.
+4. Report findings (concrete failure scenario or named missing test, with
+   file:line) or the role's null report: "attacked X, Y, Z — no break
+   found". A bare pass is not a result.
 
-| Lens | When to read |
-|---|---|
-| [Correctness](references/correctness.md) | Every non-exempt adversarial review |
-| [Simplicity](references/simplicity.md) | Mechanical/standard changes and production growth |
-| [Test coverage](references/test-coverage.md) | Behavior or test changes |
-| [Security](references/security.md) | Authn/authz, IAM, RPC trust, paths, secrets, parsing, browser, encryption |
-| [Concurrency/durability](references/concurrency-durability.md) | Async shared state, locks, storage commit, cancellation, persisted queues |
-| [Compatibility](references/compatibility.md) | S3 surface, MinIO interop, metadata, wire/disk formats, mixed versions |
-| [Performance](references/performance.md) | Request/object hot paths, allocation, blocking work, fsync, fan-out |
+## Role playbooks
 
-Do not read all references as a precaution. A path name alone is insufficient;
-the changed behavior must touch the lens's domain.
+### Correctness adversary
 
-For a dedicated security audit or advisory analysis, use
-`security-advisory-lessons` instead of loading it automatically during every
-adversarial review.
+- For any change touching error aggregation or quorum decisions, build the exact disk-error slice at the quorum boundary: N disks where successes == quorum, then flip one success to an error (quorum-1) and separately inject None/nil placeholder entries into the slice. Trace whether reduce_errs (or the new equivalent) picks the placeholder as the dominant error or lets quorum-1 pass as success. Also check heal/write paths: does a per-target failure at quorum-1 return an explicit error, or silently degrade to success?
+  - Where: crates/ecstore/src/disk/error_reduce.rs; crates/ecstore/src/set_disk/{core,ops}; crates/heal
+  - Evidence: Commit 20d61c73b 'stop reduce_errs leaking nil placeholder as dominant error' (#4551) and 47c1e730c 'make erasure heal write quorum best-effort per target' (#4545); crates/ecstore/AGENTS.md: 'Do not weaken quorum checks... Prefer explicit failure over silent data corruption or implicit success.'
+- For any change in EC read/reconstruct/streaming code, trace the mid-stream error path: the first K shards read fine, then a shard turns out bitrot-corrupt or inconsistent after N bytes have already been sent to the client. Verify the error propagates as a stream error (client sees failure), not a clean end-of-body — a silently truncated GET body is data corruption. Also re-check byte accounting: sum of per-part bytes vs object size, and partNumber-to-offset routing for the first and last part.
+  - Where: crates/ecstore/src/set_disk/read.rs (reconstruct-read, inconsistent_source_indexes handling ~line 3827); codec streaming paths in crates/ecstore/src/erasure_coding
+  - Evidence: Known live bug: EC reconstruct-read failing on inconsistent shards mid-stream silently truncates GET body → client 'unexpected EOF'; commit 15808254d 'correct codec-streaming byte accounting and partNumber routing' (#4535).
+- For any listing/pagination change, construct the exact-boundary inputs: (a) exactly max_keys/max_uploads matching entries — assert the response contains max and is_truncated=false, then max+1 entries — assert exactly max returned with is_truncated=true and a correct continuation marker; (b) a delimiter listing where folding into CommonPrefixes re-fills a full page; (c) an object 'a' coexisting with prefix dir 'a/'. Off-by-one and dropped-truncation bugs live exactly at these boundaries.
+  - Where: crates/ecstore listing/merge paths (metacache, list_objects, list_multipart_uploads); rustfs/src/storage
+  - Evidence: Three recent real bugs: fefa70b31 'stop ListMultipartUploads from returning one upload past max-uploads' (#4447), d91f4d455 'report truncation when delimiter list re-folds a full page' (#4538), 7e1f7f242 'preserve CommonPrefixes when an object and same-named prefix dir coexist' (#4563).
+- Run the diff's logic with a directory object key (trailing slash, e.g. 'pre/dir/') as input. Check which layer encodes/decodes __XLDIR__ — set_disk never sees the trailing slash, so any trailing-slash branch added below the store layer is dead code and a wrong-layer bug. For delete paths, check whether options force a nil versionId onto directory keys: the resulting miss surfaces as version-not-found, not object-not-found, so callers matching only ObjectNotFound leak ghost directory entries.
+  - Where: crates/ecstore/src/store*.rs (store layer) vs crates/ecstore/src/set_disk/*; delete option construction (del_opts) and its callers
+  - Evidence: trailing-slash branches must live at store layer; del_opts injects nil version for dir keys, PR#4220 ghost-directory cleanup never fired on the real path (rustfs#4307, backlog#798 still OPEN).
+- Feed every new binary-UUID metadata read the three degenerate values: key absent, zero-length bytes, and 16 zero bytes (nil UUID). All three must mean 'no value' — any path that produces Uuid::nil() and then acts on it (e.g. sends it as a versionId) is a finding. For tier code specifically: with remote-tier version None or "", assert the tier GET/DELETE request carries no versionId parameter at all — sending versionId="" or nil yields NoSuchVersion against unversioned tier buckets.
+  - Where: crates/filemeta/src/filemeta/version.rs; crates/ecstore/src/bucket/lifecycle/; crates/ecstore/src/services/tier/; any new consumer of crates/utils/src/http/metadata_compat.rs
+  - Evidence: AGENTS.md Cross-Cutting Domain Invariants (defensive Uuid read pattern + unversioned-tier rule); docs/operations/tier-ilm-debugging.md ('nil transition_ver_id = corrupt legacy write-back, readers must filter'); commit 726f3dc18 'accept empty remote version_id in tier recovery paths' (#4552).
+- For each match/if-let on an error or algorithm enum touched by the diff, enumerate what the wildcard/else arm swallows. Inject the variants the author didn't think of — DiskNotFound during listing, an unsupported checksum algorithm, an Err from a cleanup rename/delete — and trace whether they degrade into 'not found', a wrong-but-plausible value, or silent success. Any error path that converges with the success path without logging and propagating is a finding.
+  - Where: crates/ecstore (listing, delete/rename cleanup); crates/checksums; error-mapping layers in rustfs/src/storage
+  - Evidence: Three recent real bugs of this exact shape: e0619e355 'stop treating DiskNotFound as object not-found in listing' (#4536), afaf8c681 'reject md5 instead of silently returning crc32' (#4513), f7d2b2563 'propagate disk delete/rename failures instead of swallowing them' (#4546).
+- For any change to version ordering, index lookup, or shard/part indexing: (a) call the accessor with index == len() and len()-1 — a get_idx-style bound must reject, not panic or wrap; (b) construct two versions with identical mod-times and check the sort tie-break is total and deterministic (equal keys must not compare as both before each other); (c) for EC shard math, compute shard size for object sizes 0, 1, blockSize-1, blockSize, blockSize+1 and cross-check total reconstructed length against the object size.
+  - Where: crates/filemeta/src/filemeta/*.rs (version sort, get_idx); crates/ecstore/src/erasure_coding shard-size math
+  - Evidence: Commit 8bfb00bc0 'guard get_idx bound and fix sorts_before tie-break' (#4509) — both bug classes shipped before; ecstore AGENTS.md high-risk designation for read/write/repair correctness.
+- Exercise the zero/empty end of every new size or count parameter: zero-length object PUT then GET (body must be empty, not error), part count 0, empty Vec of disks/entries into aggregation functions, and env/config values of 0 (must clamp or reject, never divide-by-zero or 'scan nothing and report zero usage'). Anywhere the diff computes a ratio, capacity, or progress percentage, plug in 0 and the max value.
+  - Where: crates/ecstore aggregation and scanner paths; crates/object-capacity; config/env parsing in touched crates
+  - Evidence: Commits 787cc77a7 'clamp zero capacity env values to safe defaults' (#4559) and 32b1094ec 'resolve a symlinked scan root instead of silently counting zero' (#4564) — zero-as-silent-wrong-answer is a recurring repo bug class.
+- For any diff touching multipart or object commit paths, order the operations on paper and attack the failure point between them: kill the process (or return Err) after the commit rename but before cleanup, and after cleanup but before commit. Verify the earlier-failure case leaves the object readable and the later-failure case leaves no half-visible object; part meta files must never be deleted before the commit is durable.
+  - Where: crates/ecstore multipart commit/cleanup (set_disk/ops); rustfs/src/storage multipart handlers
+  - Evidence: Commit c77c5f047 'defer multipart part.N.meta cleanup until after commit' (#4548) — cleanup-before-commit ordering already caused a real data-loss window; the #4221 durability work shows fsync/ordering bugs are endemic here.
 
-## Review Protocol
+Null report example: "Attacked quorum-1 error reduction, exact max-keys listing boundary, trailing-slash dir keys, nil-UUID tier versionId, and mid-stream reconstruct error propagation — no break found."
 
-1. Freeze the exact final diff/head and list the selected lenses.
-2. Run the review shape required by root `AGENTS.md`.
-3. For each selected lens, either report a concrete finding or a null verdict
-   naming the attacks performed.
-4. A finding needs `file:line`, a triggering input/state/interleaving, the wrong
-   outcome, and a focused fix or missing regression check.
-5. Fix or rebut every finding with code-path, test, or invariant evidence.
-6. After a non-trivial edit, rerun only lenses affected by that edit against the
-   new exact diff.
+### Simplicity adversary
 
-Do not turn a null verdict into a long checklist. Record concise evidence that
-the relevant failure classes were attacked.
+- Smaller-diff attack: inspect production growth separately from tests, fixtures, generated code, and documentation; test additions have no growth budget. Rewrite the production diff mentally (or in scratch) as the minimal equivalent edit. Report a finding only with a concrete smaller design that preserves correctness, compatibility, readability, and real boundaries; fewer lines alone are not evidence.
+  - Where: Any diff; extra scrutiny for crates/ecstore, crates/lock, rustfs/src/storage where 'preserve the existing control-flow shape' is an explicit rule
+  - Evidence: AGENTS.md 'Change Style for Existing Logic' (conditional extraction rule, preserve sensitive control flow, canonical modules) and 'Reuse Before You Write'; the Adversarial Validation roles list charters this attack.
+- Reuse-and-necessity attack: for each new helper, search `crates/utils`, `crates/common`, the touched crate, the likely domain owner, and relevant direct dependencies. A reimplementation is a finding, but forced reuse with mismatched normalization, error, backoff, or durability semantics is also a finding. Demand a nameable trigger for new defensive branches. Tests remain subject to validity and near-duplicate coverage review, never a size limit.
+  - Where: Any diff adding helpers, branches on decoded/peer data, or tests
+  - Evidence: AGENTS.md 'Reuse Before You Write' and 'Necessary Code Only'; GHSA-f4vq-9ffr-m8m3 (normalization-asymmetry traversal — why forced reuse of normalizing helpers on raw keys is itself an attack); docs/operations/tier-ilm-debugging.md nil-versionId incident (why boundary re-checks are load-bearing).
+- Replacement-and-comment attack: when the diff introduces a replacement path or representation, trace all callers and flag a superseded in-scope path left behind without a compatibility requirement. Keep one canonical core behind compatibility adapters. Comments must state non-obvious invariants completely without narration or change history. Never demand unrelated deletion or trade away correctness, compatibility, or readability to reduce the diff.
+
+Null report example: "Separated production growth from tests/docs, tested a smaller equivalent, checked helper reuse and superseded paths, and found no break."
+
+### Security reviewer
+
+- For every admin handler in the diff, grep the exact AdminAction constant it passes to validate_admin_request and confirm it names the operation the handler actually performs. Construct the escalation: a low-privileged user whose policy grants the wrong-but-adjacent action (e.g. Export while the handler Imports, or Update while it Lists) — if the mismatched constant lets them through, that is the bug. Also confirm read-only endpoints (metrics, list, server-info, diagnostics) still call an operation-specific admin authz path and not a mere 'credentials exist' check.
+  - Where: rustfs/src/admin/handlers/**, rustfs/src/admin/router registration; check_permissions / validate_admin_request / AdminAction::* call sites
+  - Evidence: GHSA-vcwh-pff9-64cc (ImportIam checked ExportIAMAction), GHSA-mm2q-qcmx-gw4w (ListServiceAccount used UpdateServiceAccountAdminAction), GHSA-f5cv-v44x-2xgf (/admin/v3/metrics accepted any authenticated IAM user). advisory-patterns.md 'Admin authorization and route exposure'; rustfs/src/admin/AGENTS.md 'route registration, whitelist, handler authz must agree'.
+- If the diff touches service-account or IAM import/update, treat parent, claims, accessKey, secretKey, status, policy names, and groups as attacker-controlled. Construct an ImportIam/create payload where parent points at root (or another user) and prove the code writes credentials without proving caller ownership or root authority. Separately, set deny_only=true (or 'no explicit deny') on a restricted account and check it does not skip the required allow check, letting it mint an unrestricted child.
+  - Where: crates/iam/, rustfs/src/admin/handlers (service account / import IAM), rustfs/src/auth.rs
+  - Evidence: GHSA-566f-q62r-wcr8 (attacker-controlled parent/claims/accessKey/secretKey → persistent backdoor under root), GHSA-xgr5-qc6w-vcg9 (deny_only=true skipped allow checks → privilege creation). crates/iam/AGENTS.md security boundaries; advisory-patterns.md 'IAM import, service accounts'.
+- For a changed protocol-frontend handler (FTP/FTPS/SFTP/WebDAV/gateway), enumerate ALL sibling command handlers in the same driver — not just the changed one. For each, confirm it calls the per-operation IAM authorize hook mapped to the correct S3 action (RETR→GetObject, SIZE/MDTM→HeadObject, MKD→CreateBucket, bucket probe→ListBucket/HeadBucket) BEFORE touching storage. Construct a denied-authz case and prove the storage backend is never reached.
+  - Where: crates/protocols/ (FtpsDriver, SftpDriver, WebDAV), authorize_operation call sites
+  - Evidence: GHSA-3g29-xff2-92vp (FTP RETR/SIZE/MDTM authenticated but skipped IAM), GHSA-g3vq-vv42-f647 (FTPS MKD called create_bucket without s3:CreateBucket). advisory-patterns.md: 'RustFS advisories show mixed guarded and unguarded siblings in the same driver.'
+- For any secret/token/signature/password comparison in the diff, check it uses a constant-time compare (e.g. subtle/constant_time_eq), not == or early-return byte loops. Then check the failure-response paths: construct an invalid-user request and an invalid-secret request and confirm they are indistinguishable (same error, no early length short-circuit) so an attacker cannot enumerate valid users or time-side-channel the secret.
+  - Where: crates/protocols/ (FTPS/WebDAV/FormPost auth), crates/credentials/, rustfs/src/auth.rs, RPC signature verification
+  - Evidence: GHSA-3p3x-734c-h5vx (FTPS/WebDAV early-return string equality + distinguishable invalid-user vs invalid-password). Fix commits 3c3113619 (constant-time FTPS/WebDAV) and c41062f27 (constant-time FormPost signature). 3p3x was fixed by PR #4403.
+- If the diff parses or transports secret-bearing config (env vars, key files, connection strings), grep every error-construction and format site on that value's path (`format!` feeding `Error::other`/`configuration_error`/`panic!`/`expect`) for interpolation of the raw value or of variables named like secret material. Construct the likeliest misconfiguration: the operator supplies the bare secret without the expected `<name>:` prefix (or with a stray newline) — if the parse-failure hint echoes the input, the secret lands in startup logs. Error strings are log content; the hint may name the env var and expected format, never the value. If the diff re-implements an existing parse helper, diff the two error paths — the duplicate is where the leak hides.
+  - Where: rustfs/src/init.rs (env plumbing), crates/kms/src/config.rs, crates/credentials/, any from_env/parse on secret values; mechanical backstop in scripts/check_logging_guardrails.sh (secret-interpolation check)
+  - Evidence: PR #5222 introduced `got: {secret_str}` in build_static_kms_config's format-hint error — a bare base64 key (the secret itself) would have been echoed into startup logs; fixed by PR #5243. The parallel parse in KmsConfig::from_env already omitted the value: the leak lived only in the duplicated copy (AGENTS.md 'Reuse Before You Write').
+- If the diff touches internode/RPC auth secret handling, trace whether the RPC HMAC secret can fall back to a public default (e.g. 'rustfsadmin', 'rustfs rpc') or be derived deterministically from the S3 root credentials. Construct the case where RUSTFS_RPC_SECRET is unset and confirm the code fails closed rather than silently using a default or a root-derived key. Verify RPC signing keys are independent random secrets, not reused across S3-root/RPC-HMAC/STS-JWT roles.
+  - Where: crates/credentials/, crates/ecstore/src/rpc/, internode auth setup
+  - Evidence: GHSA-r5qv-rc46-hv8q (fell back to 'rustfsadmin'), GHSA-75fx/68cw (RPC secret derivable from root creds → forgeable signatures), GHSA-h956 (hard-coded 'rustfs rpc'), GHSA-m77q (STS JWT reused root secret). Fix commit 7b2055405 (fail closed when deriving RPC secret from default credentials, PR#4402).
+- If the diff touches RPC/NodeService authentication, verify the HMAC payload binds the EXACT concrete gRPC method path (not a service prefix), the HTTP method surrogate, and a fresh timestamp. Construct captured valid metadata for method A and replay it to method B within the timestamp window — if it authorizes, the signature is under-bound. Also test stale timestamp, wrong path, wrong method, wrong secret.
+  - Where: crates/ecstore/src/rpc/, verify_rpc_signature / NodeServiceServer, x-rustfs-signature handling
+  - Evidence: GHSA-c667-rgrv-99vj (signed service prefix instead of concrete method path → cross-method replay in timestamp window). advisory-patterns.md 'RPC input validation and panic safety'; Minimum Regression Test Expectations lists replay across two methods.
+- For any RPC/gRPC handler or deserialization touched, feed empty bytes, truncated MessagePack/protobuf, invalid enum discriminants, and stale timestamps. Grep the deserialization path for unwrap()/expect()/panic-prone decode and prove malformed attacker payloads return a typed error, not a panic (remote DoS). Weak internode auth makes reachability worse, so combine with the RPC-secret probe.
+  - Where: crates/ecstore/src/rpc/, any #[derive(Deserialize)] decoded from wire bytes, RPC handler bodies
+  - Evidence: GHSA-gw2x-q739-qhcr (malformed GetMetrics reached unwrap() → remote DoS). advisory-patterns.md 'Treat all RPC payload bytes as attacker-controlled.' rust-code-quality skill: unwrap abuse.
+- Take any object key, RPC disk path, or archive/tar/zip entry name introduced or handled in the diff and construct traversal payloads: '../', URL-encoded '%2e%2e%2f', absolute paths, platform separators, empty components. Trace the value through parse → authz check → final storage path and prove (a) authz and storage normalize the SAME way, and (b) the canonicalized path cannot escape the bucket/prefix root. Attack the case where authz sees the raw attacker bucket but storage cleaning crosses into a victim bucket.
+  - Where: crates/ecstore (path join/canonicalize), rustfs/src/storage/, Snowball auto-extract / normalize_extract_entry_key, rpc read_file_stream
+  - Evidence: GHSA-pq29-69jg-9mxc (read_file_stream joined untrusted paths, no canonical boundary check), GHSA-8r6f-hmq2-28rg (traversal object keys bypassed authz), GHSA-f4vq-9ffr-m8m3 (Snowball '../victim-bucket/object' authorized raw path then storage crossed boundary). Note __XLDIR__ trailing-slash encoding is store-layer only.
+- If the diff touches multipart/copy or presigned POST, verify UploadPartCopy enforces source GetObject AND destination PutObject semantics equivalent to CopyObject, including copy-source policy conditions (not just independent source-read + dest-write). Construct a cross-bucket UploadPartCopy from a bucket the caller cannot read. For presigned POST, submit an upload that violates content-length-range, key prefix, or exact content-type and prove the server rejects it.
+  - Where: rustfs/src/storage / S3 API handlers: upload_part_copy, CompleteMultipartUpload, PostObject policy enforcement
+  - Evidence: GHSA-mx42-j6wv-px98 (UploadPartCopy missed source authz → cross-bucket exfil), GHSA-wfxj-ph3v-7mjf (missed destination copy-source policy constraint), GHSA-w5fh-f8xh-5x3p (presigned POST didn't enforce signed policy conditions). advisory-patterns.md 'S3 copy, multipart, and upload policy validation'.
+- Grep the diff for debug!/trace!/info!/error! and any ?value / {:?} on structs or response bodies that can carry secret_key, session_token, JWT claims, HMAC secrets, expected signatures, access keys beyond safe identifiers, or raw credential-bearing responses. Check custom Debug impls and merged-config dumps too. Construct the error path (invalid signature, failed auth) and confirm it does not log the secret or the derived authenticator. Verify audit/notify entries redact credential request headers.
+  - Where: rustfs/src/**, crates/iam/, crates/audit/, crates/notify/, crates/targets/, RPC signature error paths
+  - Evidence: GHSA-r54g (STS creds logged at info), GHSA-8cm2 (debug logs leaked tokens/secrets/JWT claims/raw STS bodies), GHSA-333v (invalid RPC signature log included HMAC secret + expected signature). Fix commit ee6f79110 (redact credential request headers from audit/notify, backlog#963). rustfs-logging-governance skill.
+- For any struct in the diff deserialized from untrusted input (S3 XML/JSON, lifecycle rules, bucket policy, replication config, RPC payload), check for #[serde(deny_unknown_fields)]. Construct a payload with a typo'd field (e.g. 'NoncurentDays') or an extra field and prove it is rejected, not silently ignored. Flag #[serde(default)] on security-critical fields (retention days, limits, permissions) lacking explicit post-deserialize validation, and any user-controlled integer cast with `as` (i32 as u32) — feed a negative value and check it doesn't wrap to a huge positive.
+  - Where: crates/policy/ (bucket policy), crates/ecstore lifecycle/ILM config, replication config structs, crates/protocols XML parsing
+  - Evidence: AGENTS.md 'Serde Safety'; advisory-patterns.md 'Serde deserialization' (no deny_unknown_fields found repo-wide; 'NoncurentDays' typo silently accepted; i32 as u32 wrap). Fix commit 1acd47f15 (SSE crash-loop DoS + credential reserved-char bypass, backlog#806).
+- If the diff touches SSE / encryption reader-writer composition, do not trust API metadata claiming encryption. Trace the reader wrapper order (HashReader / EncryptReader / compression / warp) and confirm EncryptReader is actually in the chain that writes to disk — construct the case where a helper unwraps a nested reader and bypasses encryption, storing plaintext. Require a regression test that inspects the ACTUAL stored bytes on disk, not just read-back. Also check encrypted-object checksums are not exposed.
+  - Where: rustfs/src/storage/ecfs.rs, crates/rio/ (reader wrappers), crates/kms/, SSE-C replication
+  - Evidence: GHSA-xrrf-67jm-3c2r (SSE metadata reported encryption while composition bypassed EncryptReader, stored plaintext). Fix commits a7b9659e7 (hide encrypted object checksums, #4529), 80cc3b1fc (preserve SSE-C checksum state, #4410). advisory-patterns.md 'SSE and on-disk storage invariants'.
+- If the diff touches CORS or the console/browser/object-preview surface: confirm default CORS does not reflect an arbitrary Origin while also sending Access-Control-Allow-Credentials: true — construct a request with a spoofed Origin and check the response. For preview, confirm attacker-controlled object content is origin-isolated (not rendered in a same-origin iframe with console creds), served with nosniff/CSP, and that preview trust derives from validated content-type + sandboxing, NOT from object name/extension (.pdf, .html). Separately, if aws:SourceIp is evaluated, spoof X-Forwarded-For / X-Real-IP as a direct (non-trusted-proxy) client and confirm the socket peer IP is used instead.
+  - Where: rustfs/src/server/layer.rs (CORS), console preview/auth code, aws:SourceIp / policy condition evaluation, X-Forwarded-For handling
+  - Evidence: GHSA-x5xv-223c-8vm7 (default CORS reflected arbitrary origins with credentials), GHSA-v9fg-3cr2-277j (preview rendered attacker HTML same-origin, exposed localStorage creds), GHSA-7gcx-wg4x-q9x6 (extension-based PDF detection bypassed sandbox), GHSA-fc6g-2gcp-2qrq (aws:SourceIp trusted client XFF). advisory-patterns.md 'Browser, CORS' + 'Trusted proxy'.
+
+Null report example: "Attacked admin action-constant matching in the two changed handlers (both call validate_admin_request with the exact AdminAction), the FTPS RETR/SIZE authz parity, and the new lifecycle struct's serde surface (has deny_unknown_fields) — no break found."
+
+### Concurrency/durability reviewer
+
+- For every new or moved lock acquisition, enumerate all other code paths that take any overlapping subset of those locks and construct the concrete ABBA interleaving (thread 1 holds A wants B, thread 2 holds B wants A). If the diff acquires 2+ locks without a comment documenting acquisition order, that alone is a finding.
+  - Where: crates/ecstore/** (namespace locks, set_disk, disk registry), crates/audit/**, crates/lock/**, any Mutex/RwLock pair in a diff
+  - Evidence: crates/ecstore/AGENTS.md 'Lock Ordering' (document order; same set in different orders = deadlock); real ABBA deadlock fixed in c0d5f938f (#4421, audit registry vs stream_cancellers)
+- If the diff touches the object write/commit path or a lock guard's lifetime, construct the timeline where the distributed lock is lost (heartbeat refresh fails / expiry) after shard writes but before the xl.meta rename commit — verify the commit is fenced on guard.is_lock_lost() (set_disk/ops/object.rs:874-880) and the diff does not move the commit outside the fenced region or drop the guard early.
+  - Where: crates/ecstore/src/set_disk/ops/object.rs, ops/multipart.rs, crates/lock/**
+  - Evidence: 1e6207c08 (#4406) fence write commit on lock loss; ddf197ba5 (#4388) heartbeat lock refresh; backlog#899 fencing comment in object.rs
+- For any change to file creation or write-then-rename: write out the exact syscall order (write tmp -> fdatasync tmp -> rename -> fsync parent dir -> fsync ancestor dirs on first object under a prefix) and simulate a power cut after each step. Flag any dropped/reordered sync, and check the change honors the durability gate (RUSTFS_DRIVE_SYNC_ENABLE, strict/relaxed/none modes, per-bucket overrides) instead of hardcoding one mode. The 'skip tmp parent fsync' optimization is only sound when the file is renamed out of tmp — verify that precondition still holds.
+  - Where: crates/ecstore/src/disk/local.rs, disk/os.rs, disk/fs.rs, crates/ecstore/src/bucket/durability.rs, set_disk/core/io_primitives.rs
+  - Evidence: PR #4221 (the repo previously had no fsync anywhere); 2df315baf/c081586e7 (#4493) fsync ancestor dirs; 062a68d15 (#4387) tmp-parent-fsync skip is rename-conditional; eaff17cad (#4397) durability modes; 54872d52d (#4478) rename_data crash harness exists — extend it for the diff
+- If the diff touches quorum counting or per-disk error aggregation, construct adversarial error vectors for reduce_errs: nil/placeholder entries, DiskNotFound mixed with FileNotFound, exactly quorum-1 agreeing errors — and show which dominant error wins. Specifically attack the case where offline-disk errors get counted as 'object does not exist', flipping a read/heal decision into data loss. Also check quorum monotonicity: a retry or heal pass must never conclude with a LOWER quorum than the original write.
+  - Where: crates/ecstore/src/disk/error_reduce.rs, crates/ecstore/src/api/mod.rs, set_disk read/heal paths
+  - Evidence: 20d61c73b (#4551) reduce_errs leaked nil placeholder as dominant error; e0619e355 (#4536) DiskNotFound treated as object-not-found in listing; quorum monotonicity flagged as open follow-up to #4221 (#4221 follow-up list); crates/ecstore/AGENTS.md 'Do not weaken quorum checks'
+- For any multi-disk fan-out (delete, rename, heal write, cleanup), trace each per-disk Result: find any `let _ =`, `.ok()`, or best-effort collapse that keeps a failed disk out of the quorum math. Construct the run where exactly write_quorum-1 disks succeed and prove the op still returns success — that is the bug. Conversely, for heal writes, check one bad target cannot fail the whole heal (best-effort per target).
+  - Where: crates/ecstore/src/set_disk/ops/*.rs, disk/disk_store.rs, crates/heal/**
+  - Evidence: f7d2b2563 (#4546) disk delete/rename failures were swallowed; 47c1e730c (#4545) heal write quorum made best-effort per target; 2b063b0c4 (#4400) disk-replacement heal missed versions
+- For every new .await placed between a state mutation and its cleanup/commit (or inside select!/timeout/spawned task that can be aborted), construct the cancellation point: client disconnects and the future is dropped exactly there. Enumerate what is left behind — tmp files, incremented counters never decremented, half-written xl.meta, a held permit/waiter — and verify cleanup runs in Drop or the state is re-entrant. Background loops the diff adds must have a hard outer timeout so a wedged awaitee cannot pin them forever.
+  - Where: crates/ecstore/** write paths, crates/object-capacity/** scanners, crates/audit/**, anything using tokio::select! or spawn+abort
+  - Evidence: d608e320f io_uring cancel-safety spike (cancellation known-hard here); 5a372557e (#4533) wedged scans needed hard outer timeout; 7b87d4d13 (#4520) waiter-count leak; e44bece00 (#4497) audit start race + paused drops
+- If the diff touches metacache/list producers or cursor resume, construct the interleaving where the producer task completes (or errors) while a reader with a saved cursor comes back for the next page — verify a completed producer is tolerated (no error, no hang) and that a re-folded/full page reports truncation instead of silently ending the listing early. Also feed a corrupt/oversized length prefix into any metacache decode the diff touches.
+  - Where: crates/ecstore/src/cache_value/metacache_set.rs, crates/ecstore/src/store/list_objects.rs, crates/filemeta/**
+  - Evidence: 91a23361e (#4531) tolerate completed metacache producers; d91f4d455 (#4538) delimiter re-fold dropped truncation flag; d2c100fd3 (#4226) corrupt length-prefix guard
+- For multipart changes, construct concurrent operations on the SAME uploadId: put_object_part racing put_object_part (same part number), abort racing complete between the parts listing and the commit rename, and list-parts racing cleanup. Verify every metadata read/list/abort holds the per-uploadId lock, and that part.N.meta cleanup is deferred until AFTER the commit rename — cleanup before commit loses parts on a crash between the two.
+  - Where: crates/ecstore/src/set_disk/ops/multipart.rs, rustfs/src/storage/
+  - Evidence: 3bc8d79fe (#4329) serialize put_object_part per uploadId; 7fb95d4fc (#4428) unlocked upload metadata reads/aborts; 93ffbdb9b (#4437) unlocked part listings; c77c5f047 (#4548) part.N.meta cleanup moved after commit
+- Any cleanup/rollback logic added near a commit: verify it runs strictly AFTER the commit is durable, is best-effort (its failure must not fail an already-committed write), and is safe under retry — i.e., re-running it after a partial first attempt must never delete the newly-committed data dir or the last surviving copy.
+  - Where: crates/ecstore/src/set_disk/ops/object.rs (rename_data tail), ops/multipart.rs, disk cleanup helpers
+  - Evidence: afc7f1d6f/d908243e6 (#4386, backlog#898) post-commit old-data-dir cleanup had to be made best-effort; e7cc719c1 (#4389) speculative tmp cleanup moved off hot path
+- If the diff does read-modify-write on any persisted shared state (bucket metadata, notify/target config, queue store), construct two concurrent writers: show whether the second write silently discards the first (lost update) — RMW must be serialized or CAS-guarded. For persisted queues/replay, construct crash-mid-replay and prove entries are neither lost nor delivered twice without an idempotency key.
+  - Where: crates/notify/**, crates/targets/** (queue store, SQL backends), crates/ecstore/src/bucket/metadata_sys.rs
+  - Evidence: 2490d4ee2 (#4425) persisted config RMW lost updates; 08e44b95f (#4505) queue store crash-safety + replay lifecycle; e008cc5da (#4500) SQL backend idempotency
+- If the diff touches erasure decode/reconstruct or streaming GET, construct the failure mid-stream: shards become inconsistent (or a disk read fails) after N bytes of the body have already been sent — verify the stream surfaces an error to the client instead of ending cleanly at a truncated length. Silent truncation on a 200 response is the known failure mode.
+  - Where: crates/ecstore/src/set_disk/read.rs (reconstruct-read validation, historically ~line 3117), crates/ecstore/src/erasure/coding/decode.rs
+  - Evidence: Known open bug: EC reconstruct-read 'inconsistent shards' mid-GET silently truncates body -> client unexpected EOF; crates/ecstore/AGENTS.md 'explicit failure over silent corruption'
+
+Null report example: "Attacked lock ordering on the new disk-registry mutex pair, lock-loss fencing across the moved commit, power-cut points around the added rename, and cancellation at the two new awaits — no break found; quorum math and metacache paths untouched by this diff."
+
+### Compatibility reviewer
+
+- Grep the diff for raw 'x-rustfs-internal-' or 'x-minio-internal-' string literals used with map.insert/remove/get instead of the metadata_compat helpers. If found, construct the MinIO-written object case: a metadata map containing ONLY 'X-Minio-Internal-<suffix>' (mixed case, no RustFS key) and trace the diff's read path — does it miss the value? Then construct the removal case: does remove leave the twin key behind so a stale MinIO-key value resurrects on next read? Also check the value-type trap: get_bytes has NO case-insensitive fallback (unlike get_str), so a diff that moves a suffix from FileInfo.metadata (String) to meta_sys (Vec<u8>) silently loses mixed-case MinIO keys.
+  - Where: Any code touching FileInfo.metadata / user_defined / meta_sys: crates/ecstore/, crates/filemeta/, rustfs/src/storage/, crates/utils/src/http/metadata_compat.rs
+  - Evidence: Repo-wide invariant in AGENTS.md 'Cross-Cutting Domain Invariants' and CLAUDE.md; helpers and the asymmetry are pinned by tests test_str_lookup_accepts_minio_metadata_case and test_get_bytes_no_case_insensitive_fallback in crates/utils/src/http/metadata_compat.rs
+- For any diff reading a binary UUID from internal metadata (transitioned-versionID, tier-free-versionID, data_dir), trace the three degenerate inputs — key absent, value empty (b""), value nil UUID — through to the outgoing tier/S3 request. The bug shape to hunt: unwrap_or_default() or Uuid::from_slice(..).unwrap_or(Uuid::nil()) turning 'no value' into Uuid::nil(), which then gets serialized as ?versionId=00000000-... and the remote tier returns NoSuchVersion. The required pattern is .and_then(|v| Uuid::from_slice(&v).ok()).filter(|u| !u.is_nil()).
+  - Where: crates/ecstore/src/bucket/lifecycle/ (bucket_lifecycle_ops.rs, tier_sweeper.rs), crates/ecstore/src/services/tier/warm_backend_*.rs, crates/filemeta/src/filemeta/version.rs
+  - Evidence: Historical production bug documented in docs/operations/tier-ilm-debugging.md ('Nil-UUID versionId sent to tier'); regression tests live in crates/filemeta/src/filemeta/version.rs; follow-up fix 726f3dc18 'accept empty remote version_id in tier recovery paths' (#4552)
+- For any diff touching tier or replication GET/DELETE against a remote S3 target, enumerate BOTH directions of the versionId contract and trace each: (a) remote version None/"" means the tier bucket is unversioned — the request must carry NO versionId parameter at all (not an empty one, not nil); (b) remote version Some(v) on a versioned target — the versionId MUST be sent, especially on version-purge deletes, or the delete lands on the wrong version / creates a delete marker instead of purging. Check whether the diff collapses these cases through a single Option/String conversion that loses the distinction.
+  - Where: crates/ecstore/src/services/tier/warm_backend*.rs, crates/ecstore/src/bucket/lifecycle/tier_sweeper.rs, replication code under crates/ecstore/src/bucket/
+  - Evidence: Invariant in AGENTS.md and docs/operations/tier-ilm-debugging.md; real bug fixed by 0fad35645 'send versionId on version-purge deletes to generic S3 targets' (#4401) — the versioned direction, and #4552 — the unversioned direction
+- If the diff changes xl.meta encoding (adds/reorders msgpack header fields, touches FileMeta::marshal_msg or codec.rs encode paths), attack downgrade and cross-vendor parse: encode an object with the new code and decode it with (a) the meta_ver<=3 read path and (b) the real-MinIO fixture tests from #4377. Then check the header signature: is it recomputed over the new bytes, or copied/hardcoded? MinIO validates it; a stale or zero signature makes MinIO reject the file. Finally verify XL_META_VERSION was not silently bumped — old RustFS/MinIO nodes reject meta_ver > 3 during a rolling upgrade.
+  - Where: crates/filemeta/src/filemeta.rs (XL_HEADER_VERSION/XL_META_VERSION, lines ~46-54), crates/filemeta/src/filemeta/codec.rs (check_xl2_v1, decode_xl_headers)
+  - Evidence: Real bug 073bc9675 'compute header signature instead of hardcoding zero' (#4343); format contract pinned in docs/architecture/minio-file-format-compat.md (write meta_ver 3, read <=3, XL2 magic); parity fixtures from a91d9cefc (#4377)
+- If the diff touches xl.meta / FileInfo decode (into_fileinfo, version parsing, part arrays), construct hostile foreign input: a MinIO- or corruption-shaped msgpack with a parts-count that disagrees with the etags/sizes array lengths, missing optional fields, and a meta_ver 2 object with legacy checksum. Trace whether the new code indexes past an array, panics, or fabricates default values instead of returning a decode error. Run the pinned legacy fixtures (test_issue_2265_legacy_meta_v2_object_compatibility, test_issue_2288) plus the #4377 real-MinIO xl.meta parse tests against the diff.
+  - Where: crates/filemeta/src/ (fileinfo.rs, filemeta.rs, filemeta/codec.rs, filemeta/version.rs)
+  - Evidence: Real bug 7efacbdf9 'validate part array lengths in into_fileinfo' (#4382); legacy meta_ver 2 regression fixtures at crates/filemeta/src/filemeta.rs (~:1130-:1174) cited by docs/architecture/minio-file-format-compat.md
+- If the diff 'corrects' a formula, constant, or layout that is a byte-for-byte MinIO port (shard-size math, bitrot hash interleaving, erasure distribution, inline-data prefix), treat the correction itself as the bug: verify against legacy on-disk data before accepting. Concretely: run crates/ecstore/tests/legacy_bitrot_read_test.rs and the ECA-18 pinning tests; check whether existing objects written by old RustFS or MinIO still verify byte-for-byte. Known trap examples: bitrot_shard_file_size's bare return for non-streaming algorithms is CORRECT MinIO whole-file behavior, and the 32-byte prefix on inline data is the HighwayHash256 bitrot hash, not corruption.
+  - Where: crates/ecstore/src/erasure/coding/bitrot.rs, crates/ecstore/src/io_support/bitrot.rs, crates/filemeta/src/fileinfo.rs
+  - Evidence: f96314a1d 'pin streaming-only bitrot layout invariant (ECA-18)' (#4553) — audit explicitly decided NOT to change the formula because it breaks legacy interop; #4377 proved the inline-data prefix is the bitrot hash
+- For any diff that copies object metadata into an S3-client-visible surface (GET/HEAD response headers, notification event userMetadata, ListObjects/replication payloads, copy-object metadata directives), construct an object carrying internal keys under BOTH prefixes and in non-canonical casing ('X-Minio-Internal-Compression') and confirm every one is stripped via is_internal_key (which is case-insensitive) — not by an exact-match filter on one prefix. Leaked internal keys are an API-semantics break and an information leak.
+  - Where: rustfs/src/storage/, crates/notify/, replication and copy_object paths in crates/ecstore/
+  - Evidence: Real bug cf8929189 'strip rustfs/minio internal metadata from event userMetadata' (#4419); is_internal_key contract in crates/utils/src/http/metadata_compat.rs
+- If the diff touches crates/protos (node.proto, models.fbs) or internode RPC request/response structs, attack the rolling-upgrade interleaving: an old node sends a message without the new field to a new node, and a new node sends the extended message to an old node. Verify proto field numbers are only appended (never reused/renumbered), FlatBuffers tables are only extended at the end, and that an absent new field decodes to a safe default on the receiving side — 'safe' meaning it must not be interpreted as success/authorization (RPC errors fail closed) and must not flip a quorum decision.
+  - Where: crates/protos/ (node.proto, models.fbs, generated/), gRPC transport and dispatch in the internode layer
+  - Evidence: 6f613317f 'optimize gRPC transport' (#4337) shows the wire layer churns; security advisory 68cw fixed by PR#4402 established RPC fail-closed as a repo rule (see .agents/skills/security-advisory-lessons)
+- For S3 handler diffs, replay the request shapes real clients actually send, not just the canonical one: mc and aws-sdk differ on path normalization (root '//' ListBuckets), virtual-host vs path style, and header casing. Then attack every pagination boundary the diff touches: request exactly max-keys/max-uploads/max-parts items and verify the response returns exactly N (not N+1), sets IsTruncated correctly, and yields a NextMarker/KeyMarker that resumes without skipping or duplicating — construct the N+1st-item case explicitly.
+  - Where: rustfs/src/storage/ S3 handlers, listing paths in crates/ecstore/src/store/ and set_disk/
+  - Evidence: Real bugs 511ad31ba 'normalize root double-slash ListBuckets requests' (#4336) and fefa70b31 'stop ListMultipartUploads from returning one upload past max-uploads' (#4447); docs/architecture/s3-compatibility-matrix.md is the compat source of truth
+- If the diff changes bucket-metadata (.metadata.bin) or IAM/config parsing structs, run it against the real MinIO RELEASE.2025-07-23 fixtures: the msgpack blob uses PascalCase field names, so any serde rename, field-type change, or derive tweak silently drops MinIO-written fields instead of erroring. Verify parse_all_configs still loads all ten config types from the fixture without loss, and that drop-in migration still decrypts MinIO-encrypted IAM/server config rather than treating ciphertext as corrupt.
+  - Where: crates/ecstore/src/bucket/metadata*, crates/ecstore/src/bucket/migration.rs, IAM/config load paths in crates/iam/ and crates/config/
+  - Evidence: Fixture parity test parses_real_minio_bucket_metadata_blob_without_loss from a91d9cefc (#4377); real bug 717cdd2ab 'decrypt MinIO IAM & server config on drop-in migration' (#4358); format matrix in docs/architecture/minio-file-format-compat.md
+- If the diff adds a compatibility shim, legacy fallback, wrapper, or old-endpoint alias (grep the diff for 'legacy', 'fallback', 'compat', 'deprecated'), verify two things: (1) it carries a RUSTFS_COMPAT_TODO(<task-id>) marker with an exact removal condition and a matching entry in the register — an unmarked shim becomes permanent dead weight; (2) the fallback's default direction is safe for old data: e.g. a new decode path must fall back to the legacy decode for old objects by default, not gate legacy reads behind an opt-in flag that makes existing data unreadable after upgrade.
+  - Where: Anywhere in the diff; register at docs/architecture/compat-cleanup-register.md; recent example: allow_inplace_legacy_fallback flag in the ecstore erasure codec streaming path
+  - Evidence: docs/architecture/compat-cleanup-register.md review checklist; d232a46b4 wired legacy decode prefetch behind a default-OFF gate while keeping legacy reads working (#4542), with arity fallout fixed in 05890d6e2 (#4573)
+
+Null report example: "Attacked dual-key metadata writes/removals against MinIO-only-key objects, nil/empty transitioned-versionID paths to the tier, xl.meta encode against meta_ver<=3 decoders and the #4377 real-MinIO fixtures, and proto field-number evolution for old-node/new-node RPC — no compatibility break found."
+
+### Performance reviewer
+
+- For each `.clone()` or allocation added to a per-request/per-object path, identify the copied data and execution frequency. Report a finding only for a concrete repeated cost or benchmark regression. Recommend borrowing, moving, `Bytes`/`Arc`, `Cow`, or capacity reservation only when it reduces that cost without obscuring ownership or APIs.
+  - Where: crates/ecstore/src/set_disk/**, crates/ecstore/src/store*.rs, rustfs/src/storage/, crates/filemeta/, request handlers in rustfs/src/
+  - Evidence: crates/ecstore/AGENTS.md 'Allocation Discipline in Hot Paths'; .agents/skills/rust-code-quality/SKILL.md requires a concrete hot-path cost rather than a proxy metric
+- For every new sync_all/sync_data/fdatasync/flush/File::sync call in the diff, trace the call chain to DurabilityMode / RUSTFS_DRIVE_SYNC_ENABLE resolution (crates/ecstore/src/disk/local.rs:291 DurabilityMode, :347 resolve_durability_mode) and to per-bucket durability overrides. Construct the run where the operator sets mode=none (or legacy RUSTFS_DRIVE_SYNC_ENABLE=false) and the new fsync still fires — that is an ungated durability cost and a regression on 4KiB writes.
+  - Where: crates/ecstore/src/disk/local.rs, crates/ecstore/src/bucket/durability.rs, crates/ecstore/src/set_disk/** (rename_data/commit paths), any crate doing tokio::fs or std::fs writes
+  - Evidence: #4221 fsync work caused a measured -10% 4KiB write regression (#814 investigation), later gated; durability modes added in eaff17cad (#4397), per-bucket tier overrides in 13e48d93a (#4407); 2df315baf (#4493) shows even ancestor-dir fsyncs are routed through the gate
+- Attack blocking-work placement from both directions: (a) find new synchronous fs calls, hashing, or EC encode/decode executed directly on an async runtime thread without spawn_blocking/block_in_place — construct the stall (a slow disk blocks a worker thread and every task queued on it); (b) find new code that splits one logical disk operation into multiple spawn_blocking hops per object — each hop is a threadpool round-trip, so K hops x N objects multiplies latency. Demand the author justify the placement with the size of the work, not habit.
+  - Where: crates/ecstore/src/disk/local.rs, crates/ecstore/src/erasure_coding/, crates/ecstore/src/bitrot/, crates/rio/
+  - Evidence: 608ab14d7 (#4554, HP-12) folded metadata open+fstat+read into a single spawn_blocking because per-op hops were measurably slow; 8fc637fb1 (#4484) moved the short EC encode inline because block_in_place cost exceeded the work — direction depends on measured work size
+- For each lock acquisition the diff adds or relocates, mark the guard's live range and list every .await and disk/RPC call inside it. Construct the contention interleaving: N concurrent requests serialize on the guard while the holder waits on IO; for namespace/multipart commit locks, compute worst-case hold time (fsync + rename per disk) against the lock's timeout. Also diff the acquisition order against other paths taking the same locks (ABBA).
+  - Where: crates/ecstore/src/set_disk/** (commit/rename paths), crates/lock/, crates/audit/ registry, any RwLock/Mutex in per-request paths
+  - Evidence: crates/ecstore/AGENTS.md 'Lock Ordering'; c0d5f938f (#4421) fixed a real ABBA deadlock between registry and stream_cancellers; #4370 history: fsync-heavy serial cross-disk commits held a lock long enough to blow test timeouts (#4370)
+- Trace exactly what executes inside the PUT commit critical section (under the object write lock, between tmp write and rename_data completion) before vs after the diff. Any newly added work there — cleanup, extra stat, additional rename, O_DIRECT write, logging — is an attack target: construct the per-PUT latency delta and demand it be moved off the critical section or parallelized across disks.
+  - Where: crates/ecstore/src/set_disk/ops/*, crates/ecstore/src/disk/local.rs rename_data path
+  - Evidence: Three real optimizations removed exactly this class of regression: e7cc719c1 (#4389) moved speculative PUT-tail tmp cleanup off the hot path, 92c8c6db7 (#4411) moved O_DIRECT shard-writes off the commit critical section, 651ccac13 (#4487) parallelized tmp xl.meta write and shard fdatasync on commit
+- Find any new loop in a batch API that performs a per-item stat/read/RPC sequentially. Construct the concrete blowup: a 1000-key DeleteObjects or a full listing page -> 1000 serial round-trips added by the diff. Demand either a gate (skip when not needed) or bounded parallelism; for startup/load paths, check for accidental O(n^2) (re-scanning the full list per item).
+  - Where: crates/ecstore/src/store_delete*.rs / batch object APIs, listing/metacache paths, crates/iam/ store loading, crates/heal/
+  - Evidence: a413729b1 (#4398) had to gate and parallelize the DeleteObjects per-object stat fanout after it shipped serial; 16a91c35e (#4537) fixed O(n^2) IAM startup load by chunking — both were diff-introduced fanouts of this exact shape
+- For every buffer the diff allocates on the encode/decode/shard path, check: is it sized with with_capacity to the EC-expanded block (not the logical size, not default-grown)? Does it copy into a fresh Vec where Bytes::slice/clone (refcount) or the io-core buffer pool would avoid the copy? Does the diff read hash and data in separate passes where one pass suffices? Construct the per-block byte-copy count before vs after. If the diff touches the io-core pool, verify gauge accounting still balances.
+  - Where: crates/ecstore/src/erasure_coding/, crates/ecstore/src/bitrot/, crates/io-core/src/pool.rs, crates/rio/
+  - Evidence: 92bf55ce6 (#4396) fixed a real regression by right-sizing BytesMut encode ingest capacity to the EC-expanded block; 47bee8b31 (#4475) merged bitrot hash+data into one read pass; 7fa3d0d4b (#4534) shows pool gauge accounting is easy to drift when touching buffer reuse
+- For every logging or instrumentation statement the diff adds, classify the call site frequency: per-request, per-object, per-shard, or per-block. Anything info!/warn!/error! at per-object frequency or higher is a finding — construct the flood (one listing under client cancellation, one 10k-object heal) and count emitted lines. New metrics/timers on the data path must be feature-gated, not always-on. Run scripts/check_logging_guardrails.sh on the diff.
+  - Where: any per-request/per-object code, especially crates/ecstore listing and heal loops, rustfs/src/storage/ handlers; scripts/check_logging_guardrails.sh
+  - Evidence: .agents/skills/rustfs-logging-governance/SKILL.md (trace level for hot-path/repetitive success events); d25ddb0e1 (#4372) fixed real listing-cancellation error-log noise; hotpath instrumentation is deliberately feature-gated (3f13d098b #4394, f262fcfce #4541 HP-14)
+- Count how many times the diff's request path parses or fetches the same metadata: xl.meta/FileMeta decoded more than once per object, bucket metadata (metadata_sys) re-fetched inside a per-object loop, or the dual x-rustfs-internal/x-minio-internal key lookup re-run repeatedly on the same map. Construct the per-request parse count before vs after; a second full FileMeta decode per GET is a finding.
+  - Where: crates/filemeta/, crates/ecstore/src/set_disk/** read paths, crates/ecstore/src/bucket/metadata_sys.rs, crates/utils/src/http/metadata_compat.rs
+  - Evidence: 608ab14d7 (#4554) exists because redundant metadata-read syscall sequences per object were measurable; CLAUDE.md dual-key metadata convention makes repeated get_bytes lookups an easy hidden double-parse
+- If the diff touches PUT/GET/commit/erasure paths and claims 'no perf impact', demand numbers, not assertion: run the criterion benches (cargo bench -p ecstore — comparison_benchmark, erasure_benchmark, rename_data_meta_benchmark, single_block_non_inline_benchmark per crates/ecstore/benches/) against origin/main, and for end-to-end paths the warp A/B relative-budget gate (scripts/run_hotpath_warp_ab.sh --baseline-ref origin/main, as .github/workflows/performance-ab.yml runs it). Probe specifically at 4KiB object size — that is where the last real regression hid.
+  - Where: crates/ecstore/benches/, .github/workflows/performance-ab.yml, scripts/run_hotpath_warp_ab.sh
+  - Evidence: crates/ecstore/AGENTS.md: 'Benchmark-sensitive changes should include measurable rationale'; performance-ab.yml (215747022 #4480) is the repo's own relative-budget gate; the #4221 regression was only visible at 4KiB writes (#814 bisect)
+
+Null report example: "Attacked the new rename_data commit-section work, durability-gate routing of the added fdatasync, guard live-range across the shard-write awaits, and per-object clone count in the PUT path; ran comparison_benchmark + rename_data_meta_benchmark vs origin/main (4KiB delta within noise) — no break found."
+
+### Test-coverage skeptic
+
+- For every testable behavior claim in the PR description, revert that hunk and name the focused test or executable check that detects the revert. If no reasonable check exists, require the reason and residual risk from the validation floor. Especially verify the check exercises the real production path, not a lookalike helper.
+  - Where: All crates; highest value in crates/ecstore, rustfs/src/storage, crates/heal
+  - Evidence: AGENTS.md testable-behavior exit criterion. Real bug: PR #4220 (ghost-directory cleanup) merged with green tests but its fix never executed on the real delete path — required follow-up rustfs#4307, backlog#798 stayed OPEN. The tests exercised a path the production flow never took.
+- Read each added/modified test and confirm it asserts the real outcome (returned value, stored bytes, error variant), not merely 'call succeeded' or 'no panic'. Flag any test whose only observable is that the function returned, and any `assert!(result.is_err())` that never checks WHICH error. Then check: does the test prove the exploit/failure form is denied, or only that the intended form still works?
+  - Where: crates/e2e_test (security_boundary_test.rs pattern), and every #[cfg(test)] module in the diff
+  - Evidence: Commit dee8e4e63 (#4466) had to rewrite 277 lines of crates/e2e_test/src/security_boundary_test.rs because 'security boundary tests' passed without asserting real outcomes. .agents/skills/rust-code-quality/SKILL.md requires an observable failure criterion; .agents/skills/security-advisory-lessons/SKILL.md asks whether the exploit form is denied.
+- When the diff adds a boolean/mode parameter or config flag, find the test that fails if the flag's effect is INVERTED inside the changed function. Tests that were mechanically updated to pass `false`/default at every call site assert nothing about the new behavior. Execute the check: flip the flag's branch in the source and confirm at least one test goes red for each branch.
+  - Where: crates/ecstore/src/set_disk/ (e.g. build_codec_streaming_part_reader), any function gaining a parameter
+  - Evidence: Commit 05890d6e2 (#4573): PR #4560 added a 15th param allow_inplace_legacy_fallback; the arity tests were fixed by passing `false` everywhere — they assert Err outcomes independent of the flag, so the fallback behavior itself has no revert-detecting test at those sites.
+- Mutation spot-check on error propagation: for each newly added `?`, `return Err`, or error-mapping line, mentally replace it with `Ok(default)`/ignore and ask which test fails. The swallowed-error bug class recurs in this repo and always ships with green tests — a fix that propagates errors needs a test that injects the failure (faulty disk, failed rename, dispatch error) and asserts the caller sees Err.
+  - Where: crates/ecstore (disk delete/rename, reduce_errs), crates/audit, crates/notify, crates/targets
+  - Evidence: Three recent fixes for the same class: f7d2b2563 (#4546, disk delete/rename failures swallowed), dbc628f16 (#4424, audit dispatch failures swallowed), 20d61c73b (#4551, reduce_errs leaking nil placeholder as dominant error). All existed while tests were green.
+- Any test touching GET/read/reconstruct/stream paths must assert the FULL body content and exact length against a known value, not status-ok or first-bytes. Construct the degraded-read case (missing/inconsistent shards forcing EC reconstruction) and assert byte-for-byte equality; a mid-stream failure that truncates the body passes every test that only checks headers or the first chunk.
+  - Where: crates/ecstore/src/set_disk/read.rs and ops/, crates/rio, crates/e2e_test GET scenarios
+  - Evidence: Known live bug: EC reconstruct-read verification failure mid-GET at set_disk read path silently truncates the body → client 'unexpected EOF'; version-independent, undetected by existing suites because none assert full-body integrity under shard inconsistency.
+- For on-disk / on-wire format changes (xl.meta, .metadata.bin, bitrot framing), reject round-trip-only tests: a struct serialized and deserialized by the same code under test cannot catch format drift. Demand the test parse a REAL captured fixture from crates/filemeta/tests/fixtures, crates/ecstore/tests/fixtures, or crates/rio-v2/tests/minio_fixture_lab — or capture a new one from a single-disk MinIO instance (RELEASE.2025-07-23 procedure from #4377).
+  - Where: crates/filemeta, crates/ecstore (headers, msgpack bucket metadata), crates/rio-v2, migration code
+  - Evidence: Commit 073bc9675 (#4343): filemeta header signature was hardcoded to zero — round-trip tests passed for months. Commit a91d9cefc (#4377) established the real-MinIO fixture convention (inline/versioned/multipart xl.meta, HighwayHash256-prefixed inline bodies) precisely because synthetic fixtures proved nothing about interop.
+- Attack new concurrency tests for flakiness-by-construction: grep the added tests for `sleep(`, fixed timeouts under ~30s on lock acquisition, and use of shared global state (disk registry, lock client, GLOBAL_*). Serialized cross-disk commits exceed small lock timeouts under full-suite CI disk load. If the test shares global state or saturates IO, it must join the `ecstore-serial-flaky` nextest test-group in .config/nextest.toml (note: serial_test's #[serial] does NOT work — nextest runs each test in its own process). Require readiness polling, never fixed sleeps.
+  - Where: crates/ecstore tests, crates/e2e_test, .config/nextest.toml
+  - Evidence: Commit 2dfa3d3c3 (#4370): concurrent_resend test flaked with Lock(Timeout 5s) on CI — six legitimate serialized cross-disk commits under IO pressure needed 30s. Commit 7c701d9f2 (#4558) created the nextest test-group after bucket_delete_* raced make_bucket into InsufficientWriteQuorum. 65849740f (#4213) deflaked global-state contamination. crates/e2e_test/AGENTS.md: 'readiness checks and explicit polling over fixed sleep-based timing'.
+- If the diff writes internal object metadata, run the dual-key mutation: delete the `x-minio-internal-<suffix>` write (keeping only `x-rustfs-internal-`) and check whether any test fails. Because `get_bytes` prefers the RustFS key, every read-back test stays green while MinIO interop is silently broken — coverage must include an assertion that BOTH keys are present in the stored metadata map.
+  - Where: crates/utils/src/http/metadata_compat.rs and all its callers in crates/ecstore and rustfs/src/storage
+  - Evidence: CLAUDE.md domain convention: metadata must be written under both x-rustfs-internal- and x-minio-internal- keys for MinIO interop; get_bytes prefers the RustFS key, making the MinIO-key half of the invariant invisible to read-back tests.
+- For changed quorum/version/UUID logic, name the tests covering the specific poison values: quorum−1 disks, nil UUID, absent vs empty vs nil-serialized UUID bytes, remote-tier version_id of None/"" (unversioned tier bucket → no versionId sent), and the same metadata read on both MetaObject and MetaDeleteMarker version types. Mutation check: remove a `.filter(|u| !u.is_nil())` guard from the diff and confirm a test fails; if none does, the nil-UUID class is uncovered.
+  - Where: crates/ecstore (tier recovery, heal, quorum paths), crates/filemeta, code reading UUIDs from xl.meta metadata
+  - Evidence: Commit 726f3dc18 (#4552) fixed rejection of empty remote version_id in tier recovery. CLAUDE.md invariant: absent/empty/nil UUID all mean 'no value', not Uuid::nil(). docs/operations/tier-ilm-debugging.md: None/"" tier version means unversioned bucket. df9cbc4ed (#4427): unvalidated distribution values caused shuffle index panic — edge values reached production untested.
+- For any pagination/limit/truncation change, construct the exact-boundary test: result count == max (page exactly full), max+1, and a delimiter re-fold that lands precisely on the page boundary — assert both the item count AND the is_truncated/continuation marker. Off-by-one at the page boundary is a recurring shipped bug here.
+  - Where: crates/ecstore listing paths (list_objects, ListMultipartUploads, metacache), S3 handlers in rustfs/src/storage
+  - Evidence: Two shipped boundary bugs: fefa70b31 (#4447) ListMultipartUploads returned one upload past max-uploads; d91f4d455 (#4538) delimiter re-fold of a full page lost the truncation flag. Both survived existing tests because no test pinned n == max exactly.
+- A green focused test is evidence only for the targets it builds. Follow the `AGENTS.md` validation tier: add package-scoped Clippy or broader test-target compilation only when changed targets, features, or dependents remain uncovered; do not require a workspace-wide build by default.
+  - Where: All crates; especially concurrent-branch merges into crates/ecstore
+  - Evidence: #4322 broke main because only cargo test ran (field_reassign_with_default is clippy-only). b06f3df6b (#4441) and 05890d6e2 (#4573): test code broke the workspace test build (E0061) on main after textually-clean merges, failing CI for every open PR.
+
+Null report example: "Attacked revert-detection for all 3 claimed behaviors (each has a named test that fails on revert), flag-inversion on the new fallback parameter (both branches covered in codec_streaming tests), full-body assertions on the changed GET path, and n==max pagination boundary — no coverage gap found."
+
+## Sources and maintenance
+
+Probes are distilled from shipped bugs in git history (commit/PR references
+above), GitHub security advisories (see the security-advisory-lessons
+skill), scoped `AGENTS.md` rules, and invariants under `docs/architecture/`
+and `docs/operations/`. Line numbers drift; re-locate the invariant. Merge
+new incidents into an existing probe when they share a failure class; add a
+new probe only for a distinct attack, rather than growing the root policy.

.agents/skills/adversarial-validation/references/compatibility.md

@@ -1,24 +0,0 @@
-# Compatibility Lens
-
-- Internal metadata uses `metadata_compat` helpers for dual RustFS/MinIO keys,
-  including mixed casing and removal of both twins.
-- Binary UUID metadata treats absent, empty, and nil as no value. Unversioned
-  remote tiers receive no `versionId`; versioned purge requests retain the real
-  version ID.
-- `xl.meta` changes preserve supported header/meta versions, recompute
-  signatures, decode legacy fixtures, and remain readable by old RustFS/MinIO.
-- Foreign/corrupt metadata validates parallel array lengths and missing fields;
-  it returns a decode error rather than indexing, panicking, or fabricating data.
-- Do not “correct” byte-for-byte MinIO ports without legacy fixture evidence.
-  Bitrot framing, shard math, distribution, and inline prefixes are contracts.
-- Client-visible metadata/events strip both internal prefixes
-  case-insensitively.
-- Proto fields are appended, never reused/renumbered; FlatBuffers tables extend
-  compatibly and absent new fields fail closed where authorization/quorum is
-  involved.
-- Replay real client request shapes and exact pagination boundaries for S3
-  handler changes.
-- Bucket metadata/IAM/config parsing remains compatible with pinned real MinIO
-  fixtures and encrypted migration data.
-- Compatibility shims use `RUSTFS_COMPAT_TODO(<task-id>)`, have a removal
-  condition, and default toward reading old data safely.

.agents/skills/adversarial-validation/references/concurrency-durability.md

@@ -1,23 +0,0 @@
-# Concurrency and Durability Lens
-
-- For every changed lock, enumerate overlapping lock sets and construct the
-  ABBA interleaving. Multiple-lock order must be documented and consistent.
-- Mark guard lifetimes and every `.await`, disk, and RPC call inside them.
-  Estimate contention and timeout behavior under concurrent requests.
-- Object commits remain fenced if the distributed lock is lost after shard
-  writes and before metadata rename.
-- For write/rename changes, trace `write tmp -> sync tmp -> rename -> sync parent
-  -> sync required ancestors`; simulate a crash after each step and honor the
-  configured durability gate.
-- Multi-disk fan-out counts every result. Quorum-minus-one cannot become success;
-  heal remains best-effort per target where that is the established contract.
-- At every new cancellable await between mutation and cleanup/commit, drop the
-  future and inspect leftover files, counters, permits, and replay state.
-- Multipart operations on the same upload ID are serialized where required;
-  abort/complete/list races cannot delete parts before durable commit.
-- Post-commit cleanup is best-effort, retry-safe, and cannot fail an already
-  committed write or delete the last surviving copy.
-- Persisted read-modify-write uses serialization/CAS. Queue replay is crash-safe
-  and duplicate delivery has an idempotency contract.
-- Streaming reconstruction failures after partial output surface as errors, not
-  successful EOF.

.agents/skills/adversarial-validation/references/correctness.md

@@ -1,29 +0,0 @@
-# Correctness Lens
-
-Attack the changed behavior, not every subsystem in the repository.
-
-- Trace new error paths to the caller. Inject the ignored/wildcard variants and
-  verify they cannot become success, not-found, or a plausible default.
-- Exercise zero/empty/missing, maximum, and exact-boundary inputs for every
-  changed count, size, index, page limit, or optional value.
-- For aggregation/quorum changes, test exactly quorum and quorum-minus-one with
-  mixed disk errors and nil/placeholder entries.
-- For listing/pagination, test `n == max`, `n == max + 1`, delimiter folding,
-  continuation markers, and object/prefix name collisions.
-- For EC/read/streaming changes, inject failure after partial output and verify
-  the client receives an error rather than a clean truncated body. Assert exact
-  bytes and length.
-- For multipart/object commits, fail before/after rename and cleanup; committed
-  data must remain readable and pre-commit cleanup must not destroy parts.
-- For version/index ordering, test `len - 1`, `len`, equal timestamps, missing
-  versions, and deterministic tie-breaking.
-- For directory-object behavior, trace `__XLDIR__` at the store layer; branches
-  below the layer that sees trailing slashes are dead.
-- For binary UUID metadata, absent, empty, and nil all mean no value. Never send
-  nil/empty `versionId` to an unversioned tier.
-- For agent rules/skill routers, test a trigger matrix covering ordinary
-  inquiry, low-risk implementation, explicit review, high-risk code, PR
-  creation, release, and post-PR monitoring. Each case must select only the
-  intended workflow and retain required safety/authorization boundaries.
-
-Null verdicts name only the probes relevant to the diff.

.agents/skills/adversarial-validation/references/performance.md

@@ -1,20 +0,0 @@
-# Performance Lens
-
-- For added clones/allocations on request/object/block paths, quantify copied
-  data and frequency. Recommend borrowing, move, `Bytes`/`Arc`, `Cow`, or
-  capacity reservation only for a concrete repeated cost.
-- Route every new sync/flush through the durability-mode and bucket override
-  gates; mode `none` must not pay the new fsync.
-- Keep blocking filesystem/CPU work off async runtime threads, but do not split
-  one small operation into many `spawn_blocking` round trips.
-- Measure lock hold time across I/O and compare acquisition order for ABBA.
-- Keep cleanup, extra stat/rename, and diagnostics out of the PUT commit critical
-  section when they need not be there.
-- Detect per-item serial I/O/RPC in batch APIs and accidental quadratic scans;
-  use a gate or bounded concurrency when the concrete fan-out warrants it.
-- Count buffer growth and byte copies in EC/bitrot paths; preserve pool gauge
-  balance and avoid repeated metadata decode/fetch per object.
-- Repetitive success logs stay at `trace`; metrics/instrumentation on hot paths
-  require an existing gate.
-- Claims of no impact on PUT/GET/commit/erasure paths need relevant benchmark or
-  A/B evidence, especially for 4 KiB objects.

.agents/skills/adversarial-validation/references/security.md

@@ -1,31 +0,0 @@
-# Security Lens
-
-Use `security-advisory-lessons` only for a dedicated advisory/security audit.
-For an ordinary matched diff, attack these boundaries:
-
-- Admin routes: route registration, whitelist, handler authn, and the exact
-  `AdminAction` must agree. Read-only diagnostics still require admin authz.
-- IAM/service accounts: treat parent, claims, keys, groups, status, and policy
-  names as attacker-controlled; prove ownership/root authority before writes.
-- Protocol frontends: every changed/sibling command authorizes the matching S3
-  action before reaching storage.
-- Secrets/signatures: use constant-time comparison, normalize public failures,
-  keep RPC/root/STS keys independent, and fail closed when secrets are absent.
-- RPC: bind signatures to the exact method/path and timestamp; reject replay,
-  stale, malformed, truncated, and invalid-enum payloads without panic.
-- Paths/object/archive entries: reject traversal, absolute/platform escapes,
-  and normalization differences between authz and storage.
-- Copy/multipart/presigned POST: enforce source, destination, version-aware
-  actions, copy-source conditions, and every signed policy condition.
-- Logging/errors: never expose credentials, tokens, expected signatures, raw
-  secret-bearing input, or merged configs—including via `Debug` and parse errors.
-- Untrusted serde: reject unknown fields where compatible and validate
-  security-critical defaults/ranges before numeric conversion.
-- SSE/browser/CORS/trusted proxy: inspect stored ciphertext and wrapper order;
-  isolate user content; never reflect credentialed arbitrary origins or trust
-  forwarded identity from direct clients.
-- Object Lock: unreadable/fabricated/unparsable metadata fails closed across
-  foreground, lifecycle, scanner, and force-delete paths.
-
-Security findings distinguish unauthenticated compromise from a
-low-privileged authenticated bypass.

.agents/skills/adversarial-validation/references/simplicity.md

@@ -1,22 +0,0 @@
-# Simplicity Lens
-
-- Compare the production diff with the smallest equivalent local edit. Fewer
-  lines alone are not evidence; the replacement must preserve correctness,
-  compatibility, readability, and real boundaries.
-- Search the touched crate, domain owner, `crates/utils`, `crates/common`, and
-  relevant dependencies for each new helper, constant, wrapper, or fixture.
-- Reject forced reuse when normalization, error, deadline, or durability
-  semantics differ.
-- Require a concrete trigger for every new defensive branch. Keep boundary
-  checks for disk/RPC/version data and checks immediately before destructive
-  actions.
-- Flag one-caller helpers only when they merely forward or split a short linear
-  flow without adding domain naming, invariant isolation, or useful context.
-- Ensure a replacement removes the superseded in-scope path or keeps one
-  canonical core behind a documented compatibility adapter.
-- Remove narration/change-history comments; preserve concise safety, lock,
-  durability, and compatibility invariants.
-- Treat tests, fixtures, generated code, and documentation separately from
-  production growth. Do not optimize away meaningful regression coverage.
-
-A finding must include a concrete smaller design, not a style preference.

.agents/skills/adversarial-validation/references/test-coverage.md

@@ -1,24 +0,0 @@
-# Test-Coverage Lens
-
-- For every behavior claim, name the focused test/check that fails if the
-  changed hunk is reverted. If none is practical, require the reason and
-  residual risk.
-- Confirm tests exercise the real production path and assert returned values,
-  exact bytes, stored state, or the specific error variant—not only success,
-  `is_err()`, or no panic.
-- For new flags/modes, verify each branch and ask which test fails if the branch
-  is inverted.
-- For new error propagation, inject the failure and assert the caller observes
-  it; mentally replacing `?`/`return Err` with success must break a test.
-- Streaming GET tests assert the complete body and length under degraded reads.
-- Disk/wire-format tests use pinned foreign/legacy fixtures; same-code
-  round-trips are insufficient for compatibility.
-- Concurrency tests use readiness polling, isolate global state, and avoid fixed
-  sleeps or unrealistically short timeouts. Use nextest groups when process-level
-  serialization is required.
-- Internal metadata tests assert both RustFS and MinIO keys, not only read-back
-  through a helper that prefers one key.
-- Boundary companions are distinct coverage: `n == max` vs `max + 1`, and
-  absent vs empty vs nil UUID.
-- A focused test proves only the targets/features it builds. Add compilation or
-  Clippy only for uncovered changed targets.

.agents/skills/code-change-verification/SKILL.md

@@ -1,12 +1,11 @@
 ---
 name: code-change-verification
-description: Review a commit, PR, or merged patch when the user requests ordinary code-change verification. Do not combine with adversarial-validation; use that skill instead for explicitly adversarial, substantial, or high-risk RustFS reviews.
+description: Verify code changes by identifying correctness, regression, security, and performance risks from diffs or patches, then produce prioritized findings with file/line evidence and concrete fixes. Use when reviewing commits, PRs, and merged patches before/after release.
 ---
 
 # Code Change Verification
 
-Use this skill for an ordinary requested review. If the root policy or user calls
-for adversarial validation, use `adversarial-validation` instead of running both.
+Use this skill to review code changes consistently before merge, before release, and during incident follow-up.
 
 ## Quick Start
 
@@ -80,3 +79,4 @@ Run the full checklist in [rust-code-quality](../rust-code-quality/SKILL.md) —
   - Impact: ...
   - Fix suggestion: ...
   - Validation: ...
+

.agents/skills/code-change-verification/agents/openai.yaml

@@ -1,4 +1,4 @@
 interface:
   display_name: "Code Change Verification"
   short_description: "Prioritize risks and verify code changes before merge."
-  default_prompt: "Use $code-change-verification for an ordinary requested diff review with prioritized findings."
+  default_prompt: "Inspect a patch or diff, identify correctness/security/regression risks, and return prioritized findings with file/line evidence and fixes."

.agents/skills/pr-creation-checker/SKILL.md

@@ -1,46 +1,97 @@
 ---
 name: pr-creation-checker
-description: Perform the final RustFS PR preflight and draft compliant English title/body metadata immediately before creating or updating a PR. Do not use during implementation or as a second general code review.
+description: Prepare PR-ready diffs by validating scope, checking required verification steps, drafting a compliant English PR title/body, and surfacing blockers before opening or updating a pull request in RustFS.
 ---
 
 # PR Creation Checker
 
-Use this skill only at the PR boundary. Reuse completed diff review and
-verification evidence; do not reread the repository or rerun equivalent checks.
+Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whether a branch is ready for PR.
 
-## Preflight
+## Read sources of truth first
 
-1. Confirm the branch is based on current `origin/main` and contains only the
-   intended task diff.
-2. Inspect `git diff --stat`, `git diff --check`, and changed file names for
-   secrets, logs, generated artifacts, or unrelated edits.
-3. Confirm the checks selected by root `AGENTS.md` passed on the final diff.
-   Do not replace focused behavioral tests with a generic gate or rerun checks
-   already covered by an unchanged umbrella run.
-4. Read `.github/pull_request_template.md`. Consult `Makefile`, `.config/make/`,
-   or CI only when the required command/current gate is uncertain.
-5. Return `BLOCKED` for an unclean scope, missing required evidence, failed
-   required checks, or non-compliant metadata.
+- Read `AGENTS.md`.
+- Read `.github/pull_request_template.md`.
+- Use `Makefile` and `.config/make/` for local quality commands.
+- Use `.github/workflows/ci.yml` for CI expectations.
+- Do not restate long command matrices or template sections from memory when the files exist.
 
-## Metadata
+## Workflow
 
-- Title: English Conventional Commit, at most 72 characters, with no tool
-  prefix.
-- Body: English, exact template headings, `N/A` where needed, concise rationale,
-  actual verification commands, and material risks/rollback notes.
-- Use repository-relative paths; never include local absolute paths.
-- Keep prose paragraphs on one logical line and never include the literal
-  sequence `\n`.
-- Use a temporary body file with `gh pr create --body-file` or
-  `gh pr edit --body-file`; never pass multiline Markdown inline.
+1. Collect PR context
+- Confirm base branch, current branch, change goal, and scope.
+- Confirm whether the task is: draft a new PR, update an existing PR, or preflight-check readiness.
+- Confirm whether the branch includes only intended changes.
 
-## Output
+2. Inspect change scope
+- Review the diff and summarize what changed.
+- Inspect `git diff --stat` and `git diff --numstat`; assess production-code growth separately. Tests, fixtures, generated code, and documentation have no growth budget. Treat line counts as signals, not quotas.
+- Call out unrelated edits, generated artifacts, logs, or secrets as blockers.
+- Mark risky areas explicitly: auth, storage, config, network, migrations, breaking changes.
+- Use the simplicity-adversary verdict instead of producing a per-symbol inventory. Block growth only when the review identifies duplication or gives a concrete smaller design that preserves correctness, compatibility, readability, and real boundaries.
+- Confirm replacement implementations remove the superseded in-scope path or adapt compatibility at the boundary to one canonical core.
+- Scan the diff for newly added string literals and confirm whether they duplicate values already defined as constants/enums/typed wrappers in the same module or shared modules.
+- Treat introducing a new hardcoded literal where a project constant already exists as a likely regression risk; require either a refactor to reuse the constant or an explicit exception explanation in the PR body.
 
-- Status: `READY` or `BLOCKED`.
-- Title.
-- Complete PR body.
-- Verification commands and results.
-- Risks or `N/A`.
+3. Verify readiness requirements
+- Select checks from `AGENTS.md` "Verification Before PR" based on the final diff's risk tier. Do not replace a focused behavioral test with `make pre-commit`, or a required high-risk `make pre-pr` with a narrower gate.
+- For focused verification, state why the selected tier is sufficient and list the scope-specific commands in the PR body.
+- If `make` is unavailable, use the equivalent commands from `.config/make/`.
+- Add scope-specific verification commands when the changed area needs more than the baseline.
+- If required checks fail, stop and return `BLOCKED`.
 
-Immediately before the GitHub write, repeat only the five preflight checks above
-against the final head.
+4. Draft PR metadata
+- Write the PR title in English using Conventional Commits and keep it within 72 characters.
+- If a generic PR workflow suggests a different title format, ignore it and follow the repository rule instead.
+- In RustFS, do not use tool-specific prefixes such as `[codex]` when the repository requires Conventional Commits.
+- Keep the PR body in English.
+- Use the exact section headings from `.github/pull_request_template.md`.
+- Fill non-applicable sections with `N/A`.
+- Include verification commands in the PR description.
+- Do not include local filesystem paths in the PR body unless the user explicitly asks for them.
+- Prefer repo-relative paths, command names, and concise summaries over machine-specific paths such as `/Users/...`.
+
+5. Prepare reviewer context
+- Summarize why the change exists.
+- Summarize what was verified.
+- Call out risks, rollout notes, config impact, and rollback notes when applicable.
+- Mention assumptions or missing context instead of guessing.
+
+6. Prepare CLI-safe output
+- When proposing `gh pr create` or `gh pr edit`, use `--body-file`, never inline `--body` for multiline markdown.
+- Return a ready-to-save PR body plus a short title.
+- If not ready, return blockers first and list the minimum steps needed to unblock.
+
+## Output format
+
+### Status
+- `READY` or `BLOCKED`
+
+### Title
+- `<type>(<scope>): <summary>`
+
+### PR Body
+- Reproduce the repository template headings exactly.
+- Fill every section.
+- Omit local absolute paths unless explicitly required.
+
+### Verification
+- List each command run.
+- State pass/fail.
+
+### Risks
+- List breaking changes, config changes, migration impact, or `N/A`.
+
+## Blocker rules
+
+- Return `BLOCKED` if the checks required by the `AGENTS.md` validation tier have not passed.
+- Return `BLOCKED` if a documentation-only, agent-instruction-only, or local developer-tooling-only change lacks focused verification for the changed surface.
+- Return `BLOCKED` if the diff contains unrelated changes that are not acknowledged.
+- Return `BLOCKED` if required template sections are missing.
+- Return `BLOCKED` if the title/body is not in English.
+- Return `BLOCKED` if the title does not follow the repository's Conventional Commit rule.
+- Return `BLOCKED` if the diff introduces string literals that should use existing constants but did not.
+- Return `BLOCKED` for production-code growth only when the review identifies a duplicated or superseded implementation, or supplies a concrete smaller design with equivalent semantics. Fewer lines alone are not evidence.
+
+## Reference
+
+- Use [pr-readiness-checklist.md](references/pr-readiness-checklist.md) for a short final pass before opening or editing the PR.

.agents/skills/pr-creation-checker/agents/openai.yaml

@@ -1,4 +1,4 @@
 interface:
   display_name: "PR Creation Checker"
   short_description: "Draft RustFS-ready PRs with checks, template, and blockers."
-  default_prompt: "Use $pr-creation-checker for final PR preflight and compliant English title/body metadata."
+  default_prompt: "Inspect a branch or diff, verify required PR checks, and produce a compliant English PR title/body plus blockers or readiness status."

.agents/skills/pr-creation-checker/references/pr-readiness-checklist.md

@@ -0,0 +1,16 @@
+# PR Readiness Checklist
+
+- Confirm the branch is based on current `main`.
+- Confirm the diff matches the stated scope.
+- Confirm no secrets, logs, temp files, or unrelated refactors are included.
+- Confirm the checks required by the `AGENTS.md` validation tier passed.
+- For focused verification, confirm it covered the changed surface and the PR body explains why the selected tier is sufficient.
+- Confirm extra verification commands are listed for risky changes.
+- Confirm the PR title uses Conventional Commits and stays within 72 characters.
+- Confirm the PR title does not use tool-specific prefixes such as `[codex]`.
+- Confirm the PR body is in English.
+- Confirm the PR body keeps the exact headings from `.github/pull_request_template.md`.
+- Confirm non-applicable sections are filled with `N/A`.
+- Confirm the PR body does not include local absolute paths unless explicitly required.
+- Confirm multiline GitHub CLI commands use `--body-file`.
+- Confirm new hardcoded string literals were not introduced for values already represented by existing constants/enums (including protocol labels, error identifiers, headers, and metric names), or record a justified exception.

.agents/skills/rust-code-quality/SKILL.md

@@ -1,12 +1,11 @@
 ---
 name: rust-code-quality
-description: Run a focused Rust quality review when the user requests one, when reviewing a Rust PR/commit, or when another selected review workflow delegates Rust-specific checks. Do not auto-load for every implementation edit.
+description: Enforce Rust-specific code quality rules on every Rust change. Use before merge to catch unwrap abuse, silent truncation, unnecessary cloning, lock ordering violations, recursion risks, and error type anti-patterns.
 ---
 
 # Rust Code Quality Gate
 
-Use this skill for a dedicated Rust review to cover rules that `cargo clippy`
-does not catch.
+Use this skill on every Rust code change to enforce quality rules that `cargo clippy` does not catch.
 
 ## Quick Start
 
@@ -46,7 +45,7 @@ rg -n 'unwrap_or_default\(\)|unwrap_or\(' <changed-files>
 
 ## Manual Review Checklist
 
-For the Rust diff under review, verify:
+For every Rust code change, verify:
 
 ### Error Handling
 - [ ] Every production `unwrap()` or `expect()` is infallible by type or a checked invariant; explain only non-obvious invariants, using an existing type, a useful `expect` message, or a concise comment

.agents/skills/rustfs-logging-governance/SKILL.md

@@ -1,34 +1,107 @@
 ---
 name: rustfs-logging-governance
-description: Add or review RustFS `tracing` events with the repository field shape, level policy, privacy boundaries, and guardrails. Use when a change adds or edits a tracing macro/instrumentation site or the logging guardrail script.
+description: Standardize and review RustFS logging with structured `tracing` events, lower noise on hot paths, preserve security-sensitive diagnostics, and extend guardrails to prevent legacy logging patterns from returning. Use whenever a change adds or edits any `tracing` macro call (`error!`/`warn!`/`info!`/`debug!`/`trace!`/`#[instrument]`) — including a single log line added in passing while fixing unrelated logic, which is how most new log sites enter the repo — and when reviewing RustFS logs, startup/config diagnostics, cloud metadata logs, request validation logs, or `scripts/check_logging_guardrails.sh`.
 ---
 
 # RustFS Logging Governance
 
-Apply this skill only to changed logging sites; do not turn a local log edit into
-a broad logging cleanup.
+Use this skill when RustFS logging needs to be added, cleaned up, reviewed, or protected against regressions.
 
-## Workflow
+## Quick Start
 
-1. Read the changed function/module context and classify the site as lifecycle,
-   request/hot path, fallback, external fetch, or summary.
-2. Match neighboring structured events and reuse existing `EVENT_*`,
-   `LOG_COMPONENT_*`, and `LOG_SUBSYSTEM_*` constants.
-3. Put stable fields first (`event`, `component`, `subsystem`, `state`/`result`,
-   then context) and a short label last.
-4. Select the level by operational meaning:
-   - `error`: behavior/security-affecting failure;
-   - `warn`: degraded/fallback/operator-actionable state;
-   - `info`: low-frequency lifecycle/mode change;
-   - `debug`: targeted diagnostics;
-   - `trace`: repetitive request/object/shard success paths.
-5. Never log secrets, tokens, auth headers, credential payloads, raw
-   attacker-controlled bodies, or merged config dumps. Error strings and
-   `Debug` output are log surfaces too.
-6. Prefer one aggregate summary over inventories or startup banners.
-7. Run `./scripts/check_logging_guardrails.sh` and the checks selected by root
-   `AGENTS.md`.
+1. Identify the files whose logs are changing.
+2. Scan current `tracing` or `log` macros before editing.
+3. Convert sentence-style logs to short event-style logs.
+4. Demote hot-path success logs unless operators truly need them at `info`.
+5. Preserve failure, fallback, and security-relevant diagnostics.
+6. Update `scripts/check_logging_guardrails.sh` when a broad cleanup removes a legacy pattern class.
+7. Validate with formatting, targeted checks/tests, and the logging guardrail script.
 
-Read [logging-governance.md](references/logging-governance.md) only for a broad
-logging audit, event-model migration, or guardrail expansion. Ordinary single-
-site edits do not require the full workspace scope map.
+## Core Workflow
+
+### 1. Scope the logging surface
+
+- Read the changed module in full before touching log lines.
+- Classify the log site:
+  - lifecycle/startup
+  - request or validation path
+  - background loop or hot path
+  - fallback/degraded behavior
+  - cloud metadata or external fetch path
+  - metrics/config summary
+- Do not rewrite business logic to make logging easier.
+
+### 2. Use the RustFS event shape
+
+- Prefer fields first, message second.
+- Use short labels, not prose paragraphs.
+- Default field shape:
+  - `event`
+  - `component`
+  - `subsystem`
+  - `state` or `result`
+  - key context fields
+- Reuse stable field names and avoid inventing near-duplicates.
+
+See `references/logging-governance.md` for the event model, level policy, and anti-pattern list.
+
+### 3. Choose the right level
+
+- `error`: operation failure that affects behavior or security guarantees.
+- `warn`: degraded path, fallback, suspicious input, or operator-actionable misconfiguration.
+- `info`: low-frequency lifecycle or mode selection.
+- `debug`: targeted diagnostics and low-volume detail.
+- `trace`: hot-path and repetitive success-path events.
+
+When in doubt, lower the verbosity of normal success paths and keep structured detail in fields.
+
+### 4. Preserve security and privacy boundaries
+
+- Do not log secrets, tokens, auth headers, raw credential payloads, or merged config dumps.
+- Avoid logging raw forwarded headers or full trusted network inventories above `debug`.
+- Keep warning/error logs useful without echoing attacker-controlled payloads unnecessarily.
+
+### 5. Keep summaries aggregated
+
+- Replace multi-line startup banners or checklist logs with one structured event.
+- If metrics already express a concept, avoid duplicating it with many `info!` lines.
+- Prefer counts, modes, and sources over inventories unless debug detail is truly needed.
+
+### 6. Update guardrails when needed
+
+- Broad logging cleanup should usually extend `scripts/check_logging_guardrails.sh`.
+- Add forbidden patterns only for styles the repo has intentionally retired:
+  - sentence-style lifecycle logs
+  - noisy hot-path `info!`
+  - checklist-style summary logs
+  - legacy fallback wording that has been replaced by structured fields
+- Keep guardrails concrete and grep-friendly.
+
+### 7. Validate manually
+
+Use the smallest relevant set:
+
+```bash
+cargo fmt --all --check
+./scripts/check_logging_guardrails.sh
+cargo check -p <affected-crate>
+cargo test -p <affected-crate>
+```
+
+For broader Rust changes, add:
+
+```bash
+./scripts/check_unsafe_code_allowances.sh
+./scripts/check_architecture_migration_rules.sh
+cargo clippy -p <affected-crates> --all-targets -- -D warnings
+```
+
+## RustFS-Specific Notes
+
+- The durable RustFS logging direction is `event + component + subsystem + state/result + key context fields`.
+- `crates/concurrency` and `crates/trusted-proxies` are examples of this style for lifecycle, fallback, and cloud metadata logs.
+- `scripts/check_logging_guardrails.sh` is the enforcement point for preventing removed log styles from returning.
+
+## References
+
+- Read `references/logging-governance.md` when you need the detailed field set, anti-pattern examples, or guardrail update checklist.

.agents/skills/rustfs-logging-governance/references/logging-governance.md

@@ -1,62 +1,285 @@
-# Logging Audit and Migration Reference
+# RustFS Logging Governance Reference
 
-Read this reference only for a broad logging audit, an event-model migration,
-or a change to `scripts/check_logging_guardrails.sh`. Use `Cargo.toml` for the
-current workspace/crate list instead of maintaining one here.
+## Workspace Scope Map
 
-## Audit by Operational Role
+Use `Cargo.toml` `[workspace].members` as the source of truth for crate membership. When doing a broad logging sweep, classify crates by operational role so logs stay consistent within each role.
 
-- Server/protocol/admin: lifecycle, authorization failures, request boundaries,
-  and degraded subsystems; avoid normal request success at `info`.
-- Storage/heal/scanner/capacity: integrity failures and aggregate lifecycle;
-  avoid per-object, per-shard, and folder iteration noise.
-- IAM/policy/credentials/KMS/crypto: safe identifiers and enforcement results;
-  never emit secrets, claims, payloads, or expected authenticators.
-- Notify/audit/targets: target lifecycle and batch/backpressure summaries; avoid
-  per-event success logs.
-- Locking/concurrency/I/O foundations: contention anomalies and state changes;
-  prefer metrics for high-frequency worker/permit signals.
-- Shared type/schema crates: log at the operational caller boundary unless the
-  crate itself owns the failure context.
+### Core Server And Request Handling
 
-## Event Shape
+- `rustfs`
+  - Role: top-level server, startup, auth, admin wiring, S3 request handling.
+  - Logging focus: startup lifecycle, config summaries, authn/authz failures, protocol entrypoints, degraded subsystems.
+- `crates/protocols`
+  - Role: protocol integrations such as FTP, SFTP, WebDAV, and related server-side protocol layers.
+  - Logging focus: listener lifecycle, per-protocol enablement/disablement, request bridge failures.
+- `crates/madmin`
+  - Role: admin API contracts and management interfaces.
+  - Logging focus: admin action boundaries, validation failures, compatibility warnings.
+- `crates/trusted-proxies`
+  - Role: forwarded IP trust, proxy chain validation, cloud metadata sources.
+  - Logging focus: direct/trusted/fallback decisions, degraded metadata fetches, aggregated config summaries.
+- `crates/keystone`
+  - Role: Keystone auth integration.
+  - Logging focus: integration enablement, upstream auth failures, config safety without credential leakage.
 
-Prefer stable fields in this order when available:
+### Storage, Healing, And Data Plane
 
-1. `event`
-2. `component`
-3. `subsystem`
-4. `state` or `result`
-5. stable context such as mode, duration, reason, counts, safe identifiers, or
-   capacity/permit values
-6. short message label
+- `crates/ecstore`
+  - Role: erasure-coded storage implementation and peer/store initialization.
+  - Logging focus: disk/peer lifecycle, storage fallback, object I/O failures, avoid per-object noise.
+- `crates/heal`
+  - Role: healing orchestration and repair workflows.
+  - Logging focus: scheduler lifecycle, repair decisions, backlog or skipped work summaries, avoid repetitive task spam at `info`.
+- `crates/scanner`
+  - Role: data integrity scanning and health monitoring.
+  - Logging focus: scan lifecycle, compaction/deep-heal transitions, lag/backlog, noisy folder iteration should stay at `debug/trace`.
+- `crates/object-capacity`
+  - Role: capacity scan and refresh core.
+  - Logging focus: refresh lifecycle, degraded capacity sources, aggregate stats rather than per-object chatter.
+- `crates/filemeta`
+  - Role: file metadata parsing and helpers.
+  - Logging focus: parse failures, schema/format mismatch, avoid dumping raw metadata payloads.
+- `crates/storage-api`
+  - Role: storage contracts and shared data plane interfaces.
+  - Logging focus: contract mismatch and boundary diagnostics, usually low-volume.
+- `crates/checksums`
+  - Role: checksum helpers and validation.
+  - Logging focus: integrity failures and compatibility mismatches, not per-chunk success logs.
+- `crates/zip`
+  - Role: ZIP handling and compression helpers.
+  - Logging focus: parse/extract failures, archive path safety issues, avoid verbose file-by-file success logs.
 
-Reuse the module's constants and neighboring field names. Do not create aliases
-for the same concept.
+### Security, Identity, And Policy
 
-## Patterns to Retire
+- `crates/iam`
+  - Role: identity and access management.
+  - Logging focus: authz decision boundaries, imported payload safety, do not leak principals, secrets, or claims.
+- `crates/policy`
+  - Role: policy modeling and evaluation.
+  - Logging focus: deny/allow decision context, parser/validation failures, no raw secret-bearing request dumps.
+- `crates/credentials`
+  - Role: credential handling.
+  - Logging focus: never log secrets or tokens; only safe identifiers and redacted states.
+- `crates/kms`
+  - Role: key management service integration.
+  - Logging focus: init/health/fallback, key-source availability, never log key material.
+- `crates/crypto`
+  - Role: cryptographic helpers and security primitives.
+  - Logging focus: only algorithm or mode state, not plaintext, ciphertext, or secret-derived material.
+- `crates/security-governance`
+  - Role: security governance contracts.
+  - Logging focus: policy/state transitions and enforcement diagnostics.
+- `crates/signer`
+  - Role: request signing helpers.
+  - Logging focus: signature validation failures without expected-signature leakage.
 
-- sentence-style lifecycle announcements;
-- startup banners and checklist lines;
-- repetitive success logs at `info`/`debug`;
-- raw inventories when an aggregate count is sufficient;
-- fallback prose with values embedded in the message;
-- `?value`/`Debug` output for credential-bearing or attacker-controlled data;
-- logging a parse input when the malformed input may itself be a secret.
+### Notifications, Audit, And Targets
 
-## Guardrail Changes
+- `crates/notify`
+  - Role: notification dispatch, runtime facade, notifier implementations.
+  - Logging focus: target lifecycle, dispatch summaries, stream lag/backpressure, avoid per-event success spam.
+- `crates/audit`
+  - Role: audit target fan-out and audit pipeline management.
+  - Logging focus: pipeline lifecycle, target availability, batch dispatch summaries, avoid noisy "started successfully" prose.
+- `crates/targets`
+  - Role: target-specific configuration and utilities used by fan-out style systems.
+  - Logging focus: target selection, config validation, per-target degraded state.
+- `crates/s3-types`
+  - Role: S3 event and type definitions.
+  - Logging focus: usually minimal; keep logging at integration boundaries rather than low-level type crates.
+- `crates/s3-ops`
+  - Role: S3 operation definitions and mapping.
+  - Logging focus: mapping/contract failures, unsupported combinations, not normal-path request spam.
 
-When expanding `scripts/check_logging_guardrails.sh`:
+### Concurrency, Locking, And Runtime Foundations
 
-1. Add only files/patterns intentionally migrated in the same change.
-2. Keep patterns concrete and grep-friendly.
-3. Do not encode a style that remains valid elsewhere as a global ban.
-4. Run the guardrail script and the root validation tier.
-5. Treat the script as a floor; manually verify level, field shape, and privacy.
+- `crates/concurrency`
+  - Role: timeout, locking, backpressure, and I/O scheduling facade.
+  - Logging focus: lifecycle transitions and degraded states, not high-frequency worker/permit churn at `info`.
+- `crates/lock`
+  - Role: distributed locking implementation.
+  - Logging focus: lock lifecycle, contention anomalies, lock ordering or timeout diagnostics.
+- `crates/tls-runtime`
+  - Role: shared TLS runtime foundation.
+  - Logging focus: certificate lifecycle, reload/fallback, validation failures without sensitive dumps.
+- `crates/obs`
+  - Role: observability helpers.
+  - Logging focus: this crate shapes other crates' telemetry conventions; avoid recursive or redundant summaries.
+- `crates/io-core`
+  - Role: zero-copy I/O core primitives.
+  - Logging focus: keep very sparse; prefer metrics unless failures are actionable.
+- `crates/io-metrics`
+  - Role: I/O metrics collection.
+  - Logging focus: typically minimal; metrics should carry the hot-path signal.
+- `crates/rio`
+  - Role: Rust I/O utility layer.
+  - Logging focus: compatibility or runtime boundary failures, not fast-path internals.
+- `crates/rio-v2`
+  - Role: next-generation I/O compatibility layer.
+  - Logging focus: migration/feature-mode differences and degraded fallback between I/O paths.
+- `crates/utils`
+  - Role: shared helpers.
+  - Logging focus: usually avoid direct logging in generic helpers unless the helper is itself an operational boundary.
+- `crates/common`
+  - Role: shared data structures and helpers.
+  - Logging focus: same principle as `utils`; prefer callers to log context-rich events.
+- `crates/config`
+  - Role: configuration management.
+  - Logging focus: config source, fallback, validation, and summary aggregation; avoid dumping merged configs.
+- `crates/data-usage`
+  - Role: shared data usage models and algorithms.
+  - Logging focus: refresh lifecycle, summary stats, and degraded reads.
 
-Useful search seeds for the changed surface:
+### Schema, Contracts, And API Support
+
+- `crates/protos`
+  - Role: protobuf definitions.
+  - Logging focus: usually none inside the crate; emit logs at decode/use boundaries.
+- `crates/extension-schema`
+  - Role: extension schema contracts.
+  - Logging focus: schema validation and compatibility mismatches.
+- `crates/s3select-api`
+  - Role: S3 Select API interfaces.
+  - Logging focus: request validation and unsupported feature boundaries.
+- `crates/s3select-query`
+  - Role: S3 Select query engine.
+  - Logging focus: query parse/planning/execution failures, avoid row-level spam.
+- `crates/protocols`
+  - Role: non-S3 protocol support.
+  - Logging focus: see core server section; keep per-request verbosity below `info`.
+
+### Testing And Non-Production Crates
+
+- `crates/e2e_test`
+  - Role: end-to-end tests.
+  - Logging focus: test clarity matters more than production governance, but avoid copying test-only logging style into production crates.
+
+## Current Guardrail Coverage Map
+
+`scripts/check_logging_guardrails.sh` currently enforces retired patterns in these high-signal areas:
+
+- `rustfs/src/main.rs`
+- `rustfs/src/startup_iam.rs`
+- `rustfs/src/auth.rs`
+- `rustfs/src/protocols/client.rs`
+- `crates/audit/src/pipeline.rs`
+- `crates/audit/src/system.rs`
+- `crates/audit/src/global.rs`
+- `crates/notify/src/config_manager.rs`
+- `crates/notify/src/runtime_facade.rs`
+- `crates/notify/src/notifier.rs`
+- `crates/ecstore/src/store/peer.rs`
+- `crates/ecstore/src/store/init.rs`
+- `crates/ecstore/src/tier/tier.rs`
+- `crates/concurrency/src/workers.rs`
+- `crates/concurrency/src/manager.rs`
+- `crates/concurrency/src/lock.rs`
+- `crates/concurrency/src/deadlock.rs`
+- `crates/trusted-proxies/src/global.rs`
+- `crates/trusted-proxies/src/config/loader.rs`
+- `crates/trusted-proxies/src/proxy/metrics.rs`
+- `crates/trusted-proxies/src/proxy/validator.rs`
+- `crates/trusted-proxies/src/proxy/chain.rs`
+- `crates/trusted-proxies/src/middleware/service.rs`
+- `crates/trusted-proxies/src/cloud/detector.rs`
+- `crates/trusted-proxies/src/cloud/ranges.rs`
+- `crates/trusted-proxies/src/cloud/metadata/aws.rs`
+- `crates/trusted-proxies/src/cloud/metadata/azure.rs`
+- `crates/trusted-proxies/src/cloud/metadata/gcp.rs`
+
+When expanding coverage, prefer crates with:
+
+- repeated sentence-style lifecycle logs
+- high-frequency success-path `info!`
+- startup/config checklist banners
+- security-sensitive fallback wording
+- external fetch/retry/fallback flows
+
+That typically means the next broad candidates are `rustfs`, `crates/notify`, `crates/audit`, `crates/targets`, `crates/heal`, and `crates/scanner`.
+
+## Event Model
+
+Prefer this structure when the fields are available:
+
+- `event`
+- `component`
+- `subsystem`
+- `state` or `result`
+- stable context fields such as:
+  - `enabled`
+  - `implementation`
+  - `validation_mode`
+  - `peer_ip`
+  - `client_ip`
+  - `proxy_hops`
+  - `duration_ms`
+  - `fallback`
+  - `reason`
+  - `range_count`
+  - `hold_time_ms`
+  - `available_slots`
+  - `total_slots`
+  - `permits_in_use`
+
+## Level Policy
+
+- `error`: the operation fails and callers or security guarantees are affected.
+- `warn`: a degraded path, fallback, suspicious request, or operator-actionable config issue occurs.
+- `info`: a low-frequency lifecycle or mode transition occurs.
+- `debug`: useful diagnostics exist but normal operators do not need them all the time.
+- `trace`: hot-path and repetitive success-path details occur.
+
+## Preferred Patterns
+
+- Use a short message label:
+  - `"trusted proxy validation failed"`
+  - `"concurrency manager state changed"`
+  - `"trusted proxy cloud metadata loaded"`
+- Put key meaning into fields, not only the message text.
+- Aggregate config or metrics summaries into one log event.
+
+## Retired Patterns
+
+These should usually be removed or replaced:
+
+- Sentence-style lifecycle logs:
+  - `info!("Concurrency manager stopped")`
+  - `info!("Trusted Proxies module initialized")`
+- Checklist or banner logs:
+  - `info!("=== Application Configuration ===")`
+  - `info!("Available metrics:")`
+- Hot-path noise:
+  - `info!("worker take, {}", *available)`
+  - `debug!("Proxy validation successful in {:?}", duration)`
+- Legacy fallback prose:
+  - `"Request from private network but not trusted: ..."`
+  - `"Cloud metadata fetching is disabled"`
+
+## Guardrail Update Checklist
+
+When extending `scripts/check_logging_guardrails.sh`:
+
+1. Add the touched files to `checked_files`.
+2. Add only legacy patterns that have been intentionally retired.
+3. Keep patterns literal and grep-friendly.
+4. Run the guardrail script after changes.
+5. Avoid adding patterns for logs that are still valid elsewhere in the repo.
+
+## Validation Checklist
+
+For logging-only changes:
 
 ```bash
-rg -n 'error!|warn!|info!|debug!|trace!|#\[instrument' <changed-paths>
-rg -n '\?[^,)]|secret|token|credential|authorization|merged_config' <changed-paths>
+cargo fmt --all --check
+./scripts/check_logging_guardrails.sh
+cargo check -p <affected-crate>
+cargo test -p <affected-crate>
+```
+
+For broader Rust changes:
+
+```bash
+./scripts/check_unsafe_code_allowances.sh
+./scripts/check_architecture_migration_rules.sh
+cargo clippy -p <affected-crates> --all-targets -- -D warnings
 ```

.agents/skills/rustfs-release-publish/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: rustfs-release-publish
-description: "Run the end-to-end RustFS console gate, version bump, preview validation, and final-tag publication pipeline. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)."
+description: "End-to-end RustFS release pipeline: first publish any merged-but-unreleased rustfs/console changes and wait for its latest Release asset, then bump RustFS version files on main directly to the final target, publish a visible GitHub prerelease from a preview tag without updating latest channels, validate it, and publish the final tag on the SAME commit. Use whenever the user wants to release/publish a RustFS version (发版/发布)."
 ---
 # RustFS Release Publish (preview-validated pipeline)
 

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: rustfs-release-version-bump
-description: "Prepare the version-file and release-asset bump for an exact RustFS alpha/beta/stable target, with verification and optional commit/push/PR delivery. Use for an explicit version bump or when invoked by the release-publish workflow."
+description: "Publish a RustFS alpha/beta/stable release with an auditable flow: confirm target version and scope, update workspace and release assets (including strict rustfs.spec changelog identity/date/version format), run required verification, and finish with commit, push, and GitHub PR creation."
 ---
 # RustFS Release Version Bump
 
@@ -81,7 +81,10 @@ Only drop a file when the current repository release process clearly no longer r
 
 4. Verify before shipping
 - Run:
+- `cargo fmt --all`
+- `cargo fmt --all --check`
 - `make pre-commit`
+- If verification passes, run `cargo clean`.
 - If `make pre-commit` fails, return `BLOCKED` with root cause and do not silently widen scope to fix unrelated issues unless user asks.
 
 5. Commit strategy
@@ -106,7 +109,10 @@ Only drop a file when the current repository release process clearly no longer r
 - `git diff --name-only origin/main...HEAD`
 - `git diff --stat origin/main...HEAD`
 - `rg -n "<old_version>|<new_version>" Cargo.toml Cargo.lock README.md README_ZH.md flake.nix helm/rustfs/Chart.yaml rustfs.spec`
+- `cargo fmt --all`
+- `cargo fmt --all --check`
 - `make pre-commit`
+- `cargo clean`
 
 ## Output contract
 

.agents/skills/rustfs-release-version-bump/agents/openai.yaml

@@ -1,4 +1,4 @@
 interface:
   display_name: "RustFS Release Bump"
   short_description: "Prepare RustFS release branches like PR #2957."
-  default_prompt: "Use $rustfs-release-version-bump to prepare and verify an exact RustFS release-version bump."
+  default_prompt: "Use $rustfs-release-version-bump to prepare a RustFS release version, ask about any unclear version policy, and finish the commit/push/PR flow."

.agents/skills/security-advisory-lessons/SKILL.md

@@ -1,40 +1,170 @@
 ---
 name: security-advisory-lessons
-description: Perform a dedicated RustFS security/advisory review for authn/authz, IAM, RPC trust, paths, secrets, browser isolation, encryption, Object Lock, or other security boundaries. Use only when the user requests a security/advisory review or an adversarial review explicitly escalates to the full advisory map; do not auto-load solely because code touches a sensitive path.
+description: Apply RustFS security lessons distilled from repository GitHub Security Advisories. Use when making or reviewing RustFS code changes, doing security checks, handling PR review for auth/authz, IAM, storage, RPC, logging, CORS, console/browser, encryption, policy, or endpoint changes, and when deciding which security regression tests are required.
 ---
 
 # RustFS Security Advisory Lessons
 
-Use this skill as the deep security lens. For a normal adversarial review with a
-matched security surface, the concise security reference under
-`adversarial-validation` is sufficient.
+Use this skill as a RustFS-specific security lens before changing or approving code. For the distilled advisory lessons and review patterns, read [advisory-patterns.md](references/advisory-patterns.md).
 
-## Workflow
-
-1. Freeze the exact diff/head and identify the changed trust boundaries.
-2. Read [advisory-patterns.md](references/advisory-patterns.md), then apply only
-   the matching sections. Useful headings are
-   auth/admin, IAM/STS/OIDC, policy/plugins, S3/copy/multipart, protocols, paths,
-   secrets/logging/RPC, browser/CORS/proxy, SSE, Object Lock, and serde.
-3. Trace unauthenticated, low-privilege, wrong-action/owner/bucket, malformed,
-   and default-config cases. Security decisions must fail closed.
-4. Require a focused negative regression test for the bypass/exploit form, not
-   only the intended success path. State residual risk when a test is impractical.
-5. Report proven vulnerabilities separately from defense-in-depth hardening.
-
-When advisory currentness matters, fetch the live inventory instead of treating
-the reference as a status mirror:
+When currentness matters, fetch the live advisory inventory instead of relying on this skill as a status mirror:
 
 ```bash
 gh api repos/rustfs/rustfs/security-advisories --paginate \
   --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
 ```
 
-Fetch an individual advisory only when the live summary indicates a new or
-changed lesson.
+Fetch full advisory details only when the live summary suggests a new or changed lesson:
 
-## Finding Standard
+```bash
+gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
+```
 
-Each finding includes severity, `file:line`, attacker prerequisites, concrete
-input/path, impact, smallest safe fix, and a regression check. Do not exaggerate
-unauthenticated impact when the actual issue requires authenticated low privilege.
+For the full pattern map, read [advisory-patterns.md](references/advisory-patterns.md).
+
+## Workflow
+
+### 1. Scope the change
+- Identify touched routes, protocol frontends, handlers, storage paths, credentials, logs, browser surfaces, CI/release code, and policy checks.
+- Treat these paths as security-sensitive by default: `rustfs/src/admin/`, `rustfs/src/storage/`, `rustfs/src/auth.rs`, `rustfs/src/server/layer.rs`, `crates/iam/`, `crates/policy/`, `crates/credentials/`, `crates/ecstore/src/rpc/`, `crates/protocols/`, `crates/rio/`, OIDC/STS federation code, and console preview/auth code.
+
+### 2. Map to advisory classes
+- Read [advisory-patterns.md](references/advisory-patterns.md) for matching GHSA lessons.
+- Do not rely on advisory titles alone. Confirm whether the issue is authentication, authorization, input validation, storage invariant, browser isolation, logging, or operational hardening.
+
+### 3. Verify fail-closed behavior
+- Check that unauthenticated, wrong-permission, cross-user, cross-bucket, malformed-input, and default-config cases fail explicitly.
+- Prefer exact action/permission checks over broad helper calls or inferred ownership.
+- Confirm lower storage/RPC layers do not bypass checks done in upper layers.
+
+### 4. Require regression evidence
+- For behavior changes, add focused negative tests that reproduce the advisory class.
+- For sensitive fixes, include tests for the bypass form, not only the happy path.
+- If a test is impractical, explain the residual risk and provide a manual verification command.
+
+### 5. Report clearly
+- Lead with concrete findings and file/line evidence.
+- Separate proven vulnerabilities from hardening risks.
+- Avoid exaggerating unauthenticated impact when the code actually rejects unauthenticated requests but allows a low-privileged authenticated bypass.
+
+## Advisory-Derived Guardrails
+
+### Auth and admin authorization
+- Every admin or diagnostic route needs an explicit authn and authz story. Route registration, router whitelist, and handler-level authorization must agree.
+- Match the admin action to the operation exactly. Copy-paste action constants are a known RustFS vulnerability class.
+- Avoid authentication-only helpers for state-changing admin APIs; use `validate_admin_request` or the established equivalent with the right `AdminAction`.
+- Read-only admin APIs such as metrics, server info, and diagnostics still require admin authorization; checking only that credentials exist is not enough.
+- Replication admin reads can expose remote target credentials; list/get target endpoints require replication/admin authorization and must not return secrets to low-privilege callers.
+- Do not assume admin-action `Resource` scoping constrains blast radius unless the policy engine actually enforces resources for that action.
+
+### IAM and service accounts
+- Treat imported IAM payload fields as attacker-controlled: `parent`, `claims`, `accessKey`, `secretKey`, status, policy names, and groups.
+- For service account create/update/import, prove parent ownership or root/admin authority before writing credentials or claims; an action permission alone must not allow choosing root or another user as `target_user`.
+- Treat IAM export packages as credential disclosure surfaces; never include plaintext user or service-account secret keys unless the caller is allowed to recover those secrets and the export format is intentionally sealed.
+- Do not let `deny_only` or "no explicit deny" become an allow decision that skips required allow checks.
+- Test cross-user list/update/import flows with wrong, correct, self, parent, and root identities.
+
+### STS, OIDC, and federation flows
+- Every STS endpoint must have an explicit authentication story: SigV4 where required, OIDC token verification for web identity, and role/session policy validation before issuing credentials.
+- For web identity, the JWT is the credential; exemption from SigV4 is not itself an authentication bypass. Treat pre-verification claims only as untrusted routing hints, bound token size, normalize public failures, rate-limit discovery, and issue credentials only after signature, issuer, audience, and expiration checks.
+- JWT session tokens must be signed and verified by a trusted issuer/key path, not by service-account-controlled material or a reused root secret.
+- JWT verification must enforce required claims and expiration for every bearer token path; "allow missing exp" is never acceptable for user-presented credentials.
+- Public OIDC bootstrap and callback routes must treat `Host`, `X-Forwarded-Proto`, redirect targets, `state`, and callback parameters as untrusted; credential-bearing redirects require a configured, allowlisted origin.
+- OIDC discovery and validation URLs are SSRF sinks. Resolve and classify hostnames at connection time, reject rebinding to loopback/private/link-local ranges, and do not rely on literal string checks.
+
+### IAM policy conditions and plugins
+- Treat request headers as attacker-controlled even after SigV4; callers sign their own spoofed headers. Do not merge them into server-derived condition keys such as identity, groups, version ID, signature version, JWT, or LDAP claims.
+- Keep the condition-key namespace explicit. Reserved server-derived keys must reject or ignore colliding headers, while intentional request-header keys such as `s3:x-amz-*` remain available.
+- Quantified IAM condition tests need partially overlapping multi-value sets. Fully contained and fully disjoint sets cannot distinguish `ForAllValues` from `ForAnyValue` bugs.
+- External policy plugins must receive the same security context as built-in policy evaluation. If OPA or another plugin depends on existing object tags, load and pass `ExistingObjectTag/*` before the plugin decision.
+
+### S3 object actions, copy, multipart, and presigned POST
+- Version-aware object requests need version-aware actions. Explicit `versionId` reads and copy sources must authorize `s3:GetObjectVersion`, not only `s3:GetObject`.
+- Multipart copy must enforce source `GetObject` and destination `PutObject` semantics equivalent to `CopyObject`, including copy-source and policy conditions.
+- Do not let `CreateMultipartUpload`, `UploadPartCopy`, `CompleteMultipartUpload`, or `AbortMultipartUpload` return success without authorization.
+- Fallbacks from version actions to non-version actions must still pass the same public-access-block, anonymous-deny, and post-authorization gates as a direct allow.
+- Presigned POST policies are server-side contracts. Enforce `content-length-range`, key prefix, exact metadata/content-type, and all signed policy conditions.
+
+### Protocol frontends and IAM parity
+- FTP/FTPS, SFTP, gateway, and other protocol drivers must enforce IAM per operation before calling storage backends; authentication to a protocol listener is not authorization.
+- Match protocol commands to the same S3 actions as HTTP, such as `RETR` to `GetObject`, `SIZE`/`MDTM` to `HeadObject`, `MKD` to `CreateBucket`, and bucket probes to `ListBucket` or `HeadBucket`.
+- Review every handler in a protocol driver, not only the changed handler, because RustFS advisories show mixed guarded and unguarded siblings in the same driver.
+- Regression tests for protocol frontends should deny the shared authorization hook and prove the backend is not reached for the denied command.
+- Compare protocol secrets in constant time, normalize invalid-user and invalid-secret failures where practical, and add rate limiting before exposing password-style protocol endpoints.
+
+### Paths, object keys, and filesystem access
+- Never join untrusted bucket/object/RPC path strings onto filesystem roots without normalization and boundary checks.
+- Reject or safely handle `..`, absolute paths, URL-encoded traversal, platform separators, empty components, and paths that canonicalize outside the intended root.
+- Validate both S3 object-key paths and internode/RPC disk paths; storage helpers can bypass S3 authorization if they trust already-parsed paths.
+- Archive auto-extract paths are object keys too. Validate tar/zip entry names before IAM checks and before storage writes, and prove cleaned paths cannot cross bucket or prefix boundaries.
+
+### Secrets, default credentials, and crypto
+- Do not ship hard-coded shared tokens, HMAC secrets, private keys, or production test keys.
+- Defaults for root credentials and internode/RPC auth must fail closed for network-reachable deployments or generate per-install random secrets; warnings alone are not a security boundary.
+- Keep cryptographic roles separated: root S3 credentials, RPC HMAC keys, and STS/JWT signing keys must not be reused or deterministically derived from each other.
+- License or token validation must use signatures with embedded public/verifying keys only; do not use private-key decryption as authenticity.
+- Plan key rotation and key IDs when removing exposed keys.
+
+### Logging and debug output
+- Logs must never include access keys beyond safe identifiers, secret keys, session tokens, JWT claims, HMAC secrets, expected signatures, license secrets, or raw response bodies containing credentials.
+- Treat `Debug` implementations, `?value` tracing, merged config dumps, and dependency-level HTTP body logging as leak surfaces.
+- Error and panic messages are log content: they propagate through `?` and get printed by `error!`/startup logging far from where they were constructed. Never interpolate a raw config or credential value into an error string.
+- A value that fails secret-format parsing is usually the secret itself (e.g. a bare base64 key missing its `<name>:` prefix), so a parse-failure hint must name the env var or file and the expected format, never echo the input. Redacting `Debug` impls does not cover this channel.
+- Add log-capture tests or targeted unit tests for redaction wrappers when changing credential structs or response bodies.
+
+### RPC, parsing, and panic safety
+- Treat all RPC payload bytes as attacker-controlled. Replace `unwrap`, `expect`, and panic-prone deserialization with typed errors.
+- Malformed request tests should cover empty bytes, truncated MessagePack/protobuf, invalid enum values, stale timestamps, and invalid signatures.
+- RPC authentication must be independently strong; do not depend on S3 admin credentials unless the fallback is explicit and safe.
+- RPC signatures must bind the exact generated gRPC method path, timestamp, and request method. Service-prefix signatures must not authorize a different concrete NodeService call.
+
+### Browser, CORS, and console surfaces
+- Do not reflect arbitrary `Origin` while also allowing credentials. Default CORS should be no CORS unless explicitly configured.
+- Do not render user-controlled object content in a same-origin iframe with console credentials available to JavaScript.
+- Prefer origin separation for object preview/download, `nosniff`, CSP, strict content-type handling, and avoiding durable credentials in `localStorage`.
+- Preview safety must be based on trusted content type and sandboxing, not object names or extensions such as `.pdf`.
+- Console license/version-like metadata endpoints should expose only coarse public data unless authenticated, especially subject names and expiration timestamps.
+
+### Profiling, debug, and health endpoints
+- Profiling and debug endpoints are not health checks. They require admin auth, opt-in enablement, rate limiting, and safe responses.
+- Do not return absolute filesystem paths or other deployment layout in unauthenticated or low-privilege responses.
+- Ensure health endpoint allowlists cannot accidentally include expensive diagnostics.
+
+### Trusted proxy and network identity
+- Only honor `X-Forwarded-For` or `X-Real-IP` when the request came from a configured trusted proxy.
+- Apply the same trusted-proxy rule to scheme and host derivation; direct clients must not control security-sensitive redirects through `Host`, `X-Forwarded-Host`, or `X-Forwarded-Proto`.
+- Direct clients must use the socket peer address for `aws:SourceIp` and policy condition evaluation.
+- Add tests for direct spoofed headers and trusted-proxy headers.
+
+### SSE and storage invariants
+- Encryption metadata is not proof that bytes were encrypted on disk.
+- When touching reader/writer wrappers such as hashing, encryption, compression, or warp readers, verify wrapper order and inspect stored bytes in regression tests.
+- Avoid helper shortcuts that unwrap nested readers and accidentally bypass encryption or integrity layers.
+
+### Object Lock and retention invariants
+- Object Lock state must fail closed when bucket metadata is unreadable, fabricated, or unparsable. Only a confirmed absence of Object Lock configuration may permit unprotected deletes or writes.
+- Do not collapse metadata read faults, missing persisted metadata, parse failures, and genuinely absent Object Lock config into one "not configured" result.
+- Retention enforcement must cover foreground deletes, batch deletes, force-delete helpers, default-retention materialization on PUT, lifecycle expiry, scanner sweeps, and all-versions expiry.
+
+## Review Prompts
+
+Use these prompts while reviewing a diff:
+
+- Could a low-privileged authenticated user reach this path with the wrong action, parent, bucket, or source object?
+- Does a non-HTTP protocol path call the same authorization boundary as the S3 API before touching storage?
+- Does a public/default/empty config change security behavior from fail-closed to fail-open?
+- Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
+- Does this response contain stored replication, remote target, or service credentials that need redaction or stricter authorization?
+- Does any error constructor or `format!` interpolate a variable that can hold secret material, including a config parse error that echoes the raw input?
+- Does an IAM export/import path expose or trust plaintext credential secrets beyond the caller's intended authority?
+- Can this STS/OIDC path issue credentials without SigV4, trusted issuer validation, allowlisted redirects, or trusted-proxy host/scheme handling?
+- Can a service-account or STS token omit `exp`, forge `sessionPolicy`, or use a principal-controlled key as signing authority?
+- Does this outbound validation path resolve attacker-supplied hostnames and reject private, loopback, link-local, and rebound addresses at the actual connection boundary?
+- Is an archive entry, object key, or policy resource normalized differently between authorization and storage?
+- Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
+- Does an explicit object version, fallback action, or plugin authorization path pass through the same action and post-authorization gates as the direct S3 path?
+- Can a caller-controlled header populate a condition key that should be derived only by the server?
+- Do condition tests include partially overlapping multi-value inputs for quantified operators?
+- Does unreadable bucket metadata make Object Lock or retention enforcement fail closed rather than disappear?
+- Does a preview or browser-surface fix preserve the original security invariant when adding alternate viewers or file-type detection?
+- Does the test prove the exploit form is denied, or only that the intended form still works?

.agents/skills/security-advisory-lessons/agents/openai.yaml

@@ -1,4 +1,4 @@
 interface:
   display_name: "Security Advisory Lessons"
   short_description: "Apply advisory lessons in reviews."
-  default_prompt: "Use $security-advisory-lessons for a dedicated RustFS security review grounded in past advisories."
+  default_prompt: "Review code changes against past RustFS security advisory lessons and report concrete risks, missing tests, and recommended fixes."

AGENTS.md

@@ -1,255 +1,402 @@
-# RustFS Agent Instructions
+# RustFS Agent Instructions (Global)
 
-This file contains repository-wide rules. Use the nearest subdirectory
-`AGENTS.md` for path-specific invariants.
+This root file keeps repository-wide rules only.
+Use the nearest subdirectory `AGENTS.md` for path-specific guidance.
 
-## Precedence
+## Rule Precedence
 
 1. System/developer instructions.
-2. The current user request.
-3. The nearest `AGENTS.md`.
-4. This file.
+2. Current user/task instructions.
+3. The nearest `AGENTS.md` in the current path.
+4. This file (global defaults).
 
-## Operating Model
+If repo-level instructions conflict, follow the nearest file and keep behavior aligned with CI.
 
-- Inquiry, diagnosis, review, and planning tasks are read-only unless the user
-  explicitly requests changes.
-- For implementation, read the relevant code, tests, and local guidance, then
-  make the smallest change that satisfies the request.
-- State assumptions only when they affect behavior or verification. Ask only
-  when a wrong assumption would materially change the result.
-- Do not load every skill or inspect unrelated modules preemptively. Select a
-  skill only when its description directly matches the request or changed
-  surface.
-- Avoid repeated reads and equivalent verification commands once enough
-  evidence exists.
+## Execution Discipline
+
+- Read the relevant existing code, tests, and local guidance before changing behavior. For new helpers or test setup, that read includes `crates/utils`, `crates/common`, and the touched crate's own `test_util`/fixtures (see Reuse Before You Write).
+- State assumptions when they affect the implementation or verification path.
+- If a task has multiple plausible interpretations, list the options briefly and choose the narrowest reasonable path; ask when the ambiguity would make the change risky.
+- For multi-step work, keep the plan minimal and tied to verifiable outcomes.
+- Avoid redundant file reads, repeated commands, and unnecessary exploratory work once enough context is available.
+- A good result is a minimal diff with clear assumptions, no over-engineering, and independent verification that survives Adversarial Validation (below).
 
 ## Worktree and Disk Hygiene
 
-- Start implementation from the latest `origin/main` and confirm the requested
-  change is not already present.
-- An existing clean, isolated task worktree is sufficient. Create another
-  worktree only when the current checkout is shared, dirty with unrelated work,
-  or belongs to another task.
-- Never commit from a shared checkout. Use an `overtrue/` feature branch unless
-  the user requests another name.
-- Check free space before artifact-heavy builds, tests, coverage, or downloads.
-  Re-check before a broad gate when space is tight.
-- Remove only task-owned temporary/build artifacts. Never delete another task's
-  worktree or uncommitted data.
-- At handoff, mention disk or cleanup details only when they affected execution
-  or artifacts/worktrees remain intentionally.
+- Unless the requester explicitly says otherwise, treat every new implementation task as isolated work: fetch the latest `origin/main`, confirm the requested change is not already present there, and create a dedicated feature branch and worktree from that exact upstream commit before editing. Do not implement new work directly in the primary checkout or reuse a worktree from another task.
+- Check available disk space before creating the worktree or starting dependency downloads, builds, tests, coverage, or other artifact-heavy commands. For long-running or artifact-heavy work, re-check disk usage at natural phase boundaries and before broad validation; if remaining space may not safely accommodate the next command, stop and reclaim task-owned artifacts before continuing.
+- Keep cleanup scoped and safe: remove generated build/test/coverage artifacts and temporary files created by the task when they are no longer needed, and never delete another task's worktree or uncommitted files. Prefer shared dependency caches where supported instead of duplicating large artifacts across worktrees.
+- At handoff, report the disk-space checks, cleanup performed, and any retained worktree or artifacts with the reason they are still needed.
 
-## Change Style
+## PR Lifecycle Monitoring
 
-- Preserve existing control flow unless changing it is required for correctness.
-- Prefer a direct local edit over new files, wrappers, managers, or speculative
-  abstractions.
-- Add a helper only when it removes current duplication, names a real domain
-  boundary, or isolates a non-trivial invariant.
-- Remove an in-scope path superseded by the change. If compatibility requires it,
-  adapt at the boundary to one canonical core and use the repository's
-  `RUSTFS_COMPAT_TODO` policy.
-- Comments explain non-obvious invariants or reasons. Do not narrate code or
-  record change history.
-- Mention unrelated problems when useful; do not fix them in a narrow task.
+- Creating or updating a PR is not the terminal state. Unless the requester explicitly limits the task to PR creation, monitor the PR through its terminal state: merged, closed, or explicitly handed off because progress requires user or maintainer action.
+- While the task is active, monitor CI/check runs, review decisions and unresolved threads, mergeability and conflicts, and unexpected head/base changes. Prefer event-driven or bounded waits provided by the current environment over frequent polling; report only state changes, actionable failures, or meaningful prolonged delays.
+- Investigate every failing check and review comment before changing code. Fix failures attributable to the task, run the verification required for the new diff, push the update, respond to or resolve the corresponding review threads, and resume monitoring. Do not weaken checks, dismiss valid feedback, or retry flaky failures merely to obtain a green result.
+- Treat opening, green CI, approval, and mergeability as intermediate states. Never merge without the required reviewer approval or explicit authority. If progress depends on credentials, infrastructure, a maintainer decision, or another external action, report the exact blocker and the evidence already collected.
+- If the current execution environment cannot remain active until the next PR event, use a supported automation, monitor, or thread wakeup when available and within scope. Otherwise leave an explicit handoff containing the PR, current state, next event to observe, and pending cleanup; do not imply that background monitoring exists when none is scheduled.
+- After observing a merge, verify the commits are preserved on the upstream base, ensure the worktree is clean, remove the dedicated worktree, prune stale worktree metadata, and delete the local task branch when it is no longer in use. For a closed or abandoned PR, preserve any unmerged work unless deletion was explicitly authorized. Do not delete remote branches unless explicitly requested or repository automation owns that cleanup.
 
-## Reuse and Boundary Rules
+## Autonomy and Approval Boundaries
 
-- Before adding helpers, constants, fixtures, or wrappers, search the touched
-  crate, the domain-owning crate, `crates/utils`, `crates/common`, and relevant
-  direct dependencies.
-- Reuse requires matching semantics: normalization, error types, deadlines,
-  durability, and compatibility must fit the call site. A narrowly named local
-  helper is better than forced reuse with different semantics.
-- Validate untrusted input at its trust boundary, then trust the validated type.
-  Values crossing disk, RPC, persistence, or version boundaries remain
-  untrusted at every consumer.
-- Re-check boundary values immediately before destructive actions such as
-  delete, overwrite, or quorum decisions.
-- Every new branch needs a concrete triggering input/state. For decoded or peer
-  data, corruption and mixed-version input are valid triggers.
-- Required values must return a typed error when absent or corrupt; do not use a
-  default that converts corruption into a plausible result.
-- Attach error context once where it is actionable. Do not erase typed errors
-  below aggregation or quorum layers.
+- Inquiry tasks (answer, explain, review, diagnose, plan): report findings; do not change files unless a fix is explicitly requested.
+- Action tasks (change, build, fix): make in-scope local changes without asking for approval.
+- Ask for confirmation before destructive or hard-to-reverse operations (force-pushes, history rewrites, deleting data or branches), merging a PR (reviewer approval required), or any material expansion of the requested scope.
+
+## Communication and Language
+
+- Respond in the same language used by the requester.
+- Keep source code, comments, commit messages, and PR title/body in English.
+- Be concise. Avoid sycophantic openers, closing fluff, and verbose status reporting.
+
+## Change Style for Existing Logic
+
+- Start with the smallest direct, local edit. Add production files, types, traits, helpers, wrappers, or abstraction layers only when current behavior requires them. Extraction must remove present duplication, enforce a real boundary, or materially clarify a non-trivial flow; anticipated reuse is not enough.
+- Use Rust's default module file layout (`mod foo;` with `foo.rs` or `foo/mod.rs`/`foo/*.rs`).
+  Avoid `#[path = "..."]` for module inclusion; move files into the canonical module tree instead.
+  If an unavoidable generated-code, FFI, or test-fixture exception remains, keep it local and document why the canonical layout cannot work.
+- Solve only the requested problem; do not add speculative features, configurability, or adjacent improvements.
+- Prefer editing existing code over rewriting files or reshaping unrelated logic.
+- Modify only what is required. Remove any in-scope path or representation superseded by the change. If compatibility or rollback requires retention, adapt at the boundary to one canonical core and follow the repository's `RUSTFS_COMPAT_TODO` removal policy; never delete unrelated code merely to improve addition/deletion statistics.
+- Preserve the existing control-flow and logic shape when fixing bugs or addressing review comments, especially in init, distributed coordination, locking, metadata, and concurrency paths.
+- Do not refactor existing code only to make it easier to unit test.
+- Keep fixes narrowly aligned with the requested behavior; avoid semantic-adjacent rewrites while touching sensitive paths.
+- Keep code elegant, concise, and direct. Prefer the smallest readable design and existing abstractions over parallel managers, factories, adapters, or wrappers added only to make the design look extensible.
+- Comments state non-obvious reasons, assumptions, and invariants in the shortest complete form. Their length follows the invariant's complexity: `SAFETY`, lock ordering, durability, and compatibility contracts may need a short list of conditions. Never narrate the next line, restate a signature, or record change history; move durable design rationale to architecture or operations documentation.
+- Mention unrelated issues when useful, but do not fix them as part of a narrow task.
+
+## Reuse Before You Write
+
+Search for an existing implementation before writing a new one; extend what exists instead of duplicating it:
+
+- **Helpers and utilities** (path/string handling, hashing, retry, env parsing, IO wrappers): check `crates/utils`, `crates/common`, the touched crate, the likely domain-owning crate, and relevant direct workspace dependencies from `Cargo.toml`. Search snake_case signatures with a focused term. Reimplementing an existing workspace helper — or hand-rolling what `std`, `tokio`, or an existing dependency already provides — is a review finding, not a style preference.
+- **Reuse requires matching semantics, not a matching name**: before adopting a helper, check its normalization (`clean` resolves `.`/`..` — never apply it to raw S3 object keys), error type, backoff/deadline behavior, and durability gating against the call site. When semantics differ, a new narrowly-named helper with a comment naming the rejected lookalike is the correct outcome. The inverse also holds: workspace wrappers exist because raw `std`/`tokio` semantics were insufficient (durability gates, retries) — prefer the wrapper over the raw call.
+- **Constants and fixed tokens** (protocol labels, error identifiers, header keys, event names, metric names, command tags): search for existing constants/enums that already represent the same semantic value and reuse them. If a value is truly new, define one local constant near related logic; never scatter the literal across sites. When changing existing behavior, align naming and format with the established constants.
+- **Test scaffolding**: reuse existing test utilities and fixtures (the touched crate's own `test_util` module and `tests/fixtures`, or `crates/test-utils`) instead of writing new setup code — run `rg -l '<fn-under-test>' <crate>/src <crate>/tests` before writing a test. A new test must pin a failure mode no existing test covers. Near-duplicate means same code path AND same poison-value class: this repo's boundary companions (n==max vs max+1, absent vs empty vs nil UUID bytes, MetaObject vs MetaDeleteMarker) are distinct by definition and must all be written.
+
+## Necessary Code Only
+
+Net-new code — files, types, branches, comments — is cost to justify, not progress:
+
+- Inspect production-code additions separately. Tests, fixtures, generated code, and documentation do not count as production-code growth. Line counts are signals, not quotas: new production structures must map to a current requirement, and a blocker requires a concrete smaller design that preserves correctness, compatibility, readability, and real boundaries.
+- Validate at the trust boundary — untrusted client input, bytes read from disk, RPC payloads, config (see Serde Safety and Cross-Cutting Domain Invariants) — then trust the type: do not re-check what the type system or a validated upstream layer already guarantees, and cite the establishing check (`file:line`) when the guarantee is not obvious.
+- The exception is load-bearing: a value that crossed a persistence, RPC, or version boundary is never guaranteed by the code on the other side — a peer may be older or buggy, disk bytes may be corrupt — so the Cross-Cutting Domain Invariant patterns apply at every consumer, and re-checks immediately before a destructive action (delete, overwrite, quorum decision) stay. Deleting an existing guard is a behavior change requiring adversarial review, not cleanup.
+- Every new branch needs a nameable trigger: a concrete input, state, or failure that reaches it — for boundary-crossing values, corrupt or stale persisted/peer data is always nameable. If you cannot name one, do not write the branch. If the case is truly unreachable, encode the invariant in the type; where that is impossible, return a typed internal error (fail closed). `debug_assert!` is acceptable only for pure internal arithmetic on values that never crossed a disk/RPC/config boundary — never as the sole guard on decoded or peer-supplied data.
+- Never substitute a default where the value is required (e.g. `unwrap_or_default()` on metadata that must exist) — that converts corruption into a wrong answer. Return the typed error instead: explicit failure over implicit success.
+- Attach error context once, at the layer where it is actionable: re-wrapping equivalent context at every hop is noise, and expanding a fallible chain into nested `match` blocks where `?` or a combinator suffices is a finding. Never add context by converting a typed error into a generic variant below an error-aggregation or quorum layer (`reduce_errs` classifies by variant equality) — context there belongs in a `tracing` event, not the error value.
 
 ## Sources of Truth
 
-- Workspace membership: `Cargo.toml`.
-- Local gates: `Makefile` and `.config/make/`.
-- CI gates: `.github/workflows/ci.yml`.
-- PR format: `.github/pull_request_template.md`.
-- Architecture routing: `ARCHITECTURE.md` and `docs/architecture/README.md`.
-- Agent skills: `.agents/skills/*/SKILL.md`.
+- Workspace layout and crate membership: `Cargo.toml` (`[workspace].members`)
+- Local quality commands: `Makefile` and `.config/make/`
+- CI quality gates: `.github/workflows/ci.yml`
+- PR template: `.github/pull_request_template.md`
+- High-level architecture and crate map: `ARCHITECTURE.md`
+- Migration guardrails, readiness contracts, support matrices:
+  `docs/architecture/README.md` (routes by audience)
+- Shared agent skills (all tools): `.agents/skills/` — each `SKILL.md` carries
+  a frontmatter `description` stating when it applies. Scan the descriptions
+  before starting a task and follow any skill that matches, even if your tool
+  does not auto-load skills:
+  `grep -m1 '^description:' .agents/skills/*/SKILL.md`
+  Claude Code reads them through the `.claude/skills` symlink; add new skills
+  to `.agents/skills/` only, never as separate copies per tool
 
-Do not commit one-shot plans, trackers, migration ledgers, benchmark snapshots,
-or agent scratch notes. Durable architecture belongs under `docs/architecture/`,
-operations under `docs/operations/`, and testing references under
-`docs/testing/`. `scripts/check_no_planning_docs.sh` enforces this boundary.
+Avoid duplicating long crate lists or command matrices in instruction files.
+Reference the source files above instead.
 
-## Verification
+Do not commit planning-type documents — one-shot implementation/optimization
+plans, task trackers, migration-progress ledgers, phase/PR templates,
+issue-scoped benchmark-result snapshots or optimization conclusions, or
+agent-generated working notes (e.g. anything a `superpowers`/scratch workflow
+produces). Keep that work in the issue tracker or your local worktree, not in
+the repository. Only durable reference — the architecture set under
+`docs/architecture/`, repeatable operational runbooks under `docs/operations/`,
+and the test-suite references under `docs/testing/` — belongs in version
+control; `.gitignore` ignores everything else under `docs/` by default, so a new
+plan file will not be tracked unless someone force-adds it — don't.
+`scripts/check_no_planning_docs.sh` (wired into `make pre-commit`/`pre-pr` and
+CI) fails the build if anything is committed under `docs/superpowers/`, even via
+`git add -f`.
 
-Select checks from the final task-owned diff. Scoped `AGENTS.md` files may add a
-concrete path-specific check, but must not replace this tiering with a generic
-full-workspace gate.
+## Verification Before PR
 
-### Documentation and Instructions
+Convert changes into independently verifiable outcomes. This section controls
+agent-run local validation; preparing a commit or PR does not by itself require
+the broadest gate. Inspect only the final task-owned diff, classify it by
+behavioral impact rather than line count or path alone, and run the smallest
+set of checks that provides meaningful coverage. Do not let unrelated
+worktree changes or a generic contributor checklist expand the scope.
+For non-exempt changes, complete the applicable multi-role adversarial review
+before running `make pre-pr` (or an equivalent full gate). Resolve or rebut
+every finding first, then run the gate against the reviewed final diff.
 
-For prose, comments, agent instructions, and skill metadata that cannot affect
-runtime/build output:
+### Validation floor
 
-- Run `git diff --check`.
-- Run the relevant documentation guard or skill validator when applicable.
-- Skip Cargo formatting, compilation, Clippy, tests, `make pre-commit`, and
-  `make pre-pr`.
+- Every change that is not documentation-only must finish with
+  `cargo fmt --all --check` passing. An umbrella gate that runs this exact
+  check satisfies the requirement; do not run it twice. Use `cargo fmt --all`
+  only when formatting needs to be fixed. Run the configured formatter or
+  validator for other changed languages when one exists.
+- Documentation-only or instruction-only means all task-owned changes are
+  prose or documentation assets and cannot affect runtime, builds, CI,
+  dependencies, generated code, or tests. Run `git diff --check` and any
+  relevant documentation guard, but skip Cargo formatting, compilation,
+  Clippy, tests, `make pre-commit`, and `make pre-pr`.
+- Behavior changes require relevant existing or new tests. Prefer the most
+  focused test or affected package. A passing targeted test can also provide
+  sufficient compilation coverage when it builds every changed target and
+  feature involved; do not add a redundant `cargo check` in that case.
+- `cargo check` supplements compilation coverage; it never substitutes for a
+  behavioral test. If a relevant test cannot reasonably be added or run, use
+  the narrowest compilation check and report the reason and remaining risk.
 
-### Non-Behavioral Source Changes
+### Validation tiers
 
-- Run the formatter/validator for the changed language.
-- Add compilation or doctests only when syntax or executable examples changed.
+1. **Documentation/instruction-only:** Apply the exemption above. Run a guard
+   such as `make doc-paths-check` only when it is relevant to the edited text.
+2. **Non-behavioral source change:** For comments, formatting, or another
+   demonstrably non-executable change, run the formatting floor. Compilation,
+   Clippy, and tests may be skipped only when the edit cannot affect
+   compilation or runtime behavior; run targeted doctests if executable
+   documentation examples changed.
+3. **Localized or bounded behavior change:** Run the formatting floor and the
+   narrowest relevant tests. Add package-scoped `cargo check` or Clippy only
+   for changed targets, features, APIs, error handling, async behavior, or
+   control flow not already covered. When several crates are affected but the
+   dependency set is identifiable, validate those packages and known
+   dependents instead of the whole workspace. Use `make pre-commit` only when
+   a repository-wide fast gate adds useful confidence beyond those checks.
+4. **Broad or high-risk change:** After the applicable adversarial review has
+   completed, run `make pre-pr` only when targeted coverage cannot bound the
+   impact, including:
+   - dependency, feature, build-script, procedural-macro, code-generation,
+     toolchain, or CI changes that alter compilation or the test matrix;
+   - cross-crate public APIs, shared foundational code, or broad refactors with
+     an unbounded dependent set;
+   - locking, storage durability or formats, erasure coding, replication,
+     RPC/protocol compatibility, IAM/KMS/auth, cryptography, or other
+     security-sensitive behavior;
+   - a targeted check that reveals wider impact, an explicit user request, or
+     a release policy that requires the full gate.
 
-### Localized Behavior Changes
+Documentation-only and non-behavioral classifications take precedence over
+path-based triggers. A small diff can still be high-risk, while a CI comment,
+manifest comment, or release-note edit does not require full validation.
 
-- Run `cargo fmt --all --check` for Rust changes.
-- Run the narrowest test that exercises the changed behavior.
-- Add package-scoped `cargo check` or Clippy only for targets, features, public
-  APIs, error handling, or control flow not compiled by the focused test.
-- Use `make pre-commit` only when its repository-wide fast checks add confidence
-  beyond the focused checks.
+`make pre-pr` includes `make pre-commit` coverage. Never run both for the same
+unchanged diff, and do not repeat equivalent checks during PR preparation or
+because a local hook already ran them. Rerun only checks whose scope is affected
+by later edits. Full workspace checks do not replace a relevant integration or
+E2E test for changed behavior; run that focused test when required and
+available, or report why it was not run and the remaining risk.
 
-### Broad or High-Risk Changes
+If `make` is unavailable, run the equivalent checks defined under
+`.config/make/`. At handoff, list the checks actually run, checks intentionally
+skipped, and the reason for the selected tier.
 
-After the required adversarial review, run `make pre-pr` when targeted coverage
-cannot bound the impact, including dependency/toolchain/build-matrix changes,
-unbounded cross-crate APIs, or locking, durability, erasure coding, replication,
-RPC, IAM/KMS/auth, cryptography, on-disk/on-wire, and S3-visible behavior.
+After build-based verification completes, clean generated build artifacts before wrapping up to avoid unnecessary disk usage.
+Do not open a PR with code changes when the required checks fail.
+Make a failing check pass by fixing the cause, never by weakening the gate:
+do not loosen or skip a guard script, add entries to a baseline or allowance
+list, suppress a lint with `#[allow]`, mark a failing test `#[ignore]`, or
+delete or relax a failing assertion to get green. If a check itself is wrong,
+change it deliberately and state the rationale in the PR.
 
-`make pre-pr` includes `make pre-commit`; never run both for the same unchanged
-diff. Do not repeat a check already covered by a successful umbrella gate.
-Rerun only checks affected by later edits.
+For flaky tests, do not paper over them with retries. Follow the flake policy
+in [docs/testing/README.md](docs/testing/README.md) (open an issue within 24h,
+quarantine with an issue link, fix or delete within 30 days); the local
+`default` nextest profile never retries.
 
-Never weaken a gate to get green: do not add baselines/allowances, suppress
-lints, ignore tests, or relax assertions unless changing that policy is itself
-the reviewed task. Follow `docs/testing/README.md` for flaky tests.
+## Adversarial Validation (Default On)
 
-## Adversarial Validation
+Every non-exempt output (see Risk tiers) — code change, bug fix, or
+design/solution proposal — passes multi-role adversarial review before it
+counts as done.
+Author confidence is not evidence: each role's job is to refute the change,
+not to bless it.
 
-Adversarial validation applies to final implementation diffs, explicitly
-requested adversarial/design reviews, and agent-instruction changes that alter
-execution. Ordinary questions, diagnoses, status reports, non-adversarial code
-reviews, and low-risk planning do not trigger it.
+### Risk tiers
 
-Risk and review shape:
+Pick the tier from the riskiest file touched; when in doubt, pick the higher.
 
-- **Exempt:** documentation, comments, formatting, or typos with no runtime,
-  build, test, or agent-execution effect.
-- **Mechanical:** renames, moves, test/tooling-only changes, and agent-rule
-  changes. Run correctness and simplicity lenses.
-- **Standard:** localized behavior changes. Run one integrated final-diff pass
-  covering correctness, simplicity, and test coverage; add only domain lenses
-  matched by the diff.
-- **High risk / substantial PR review:** high risk includes locking,
-  erasure/quorum/heal, replication, multipart, RPC, lifecycle/tiering,
-  persistence/fsync, IAM/KMS/auth, cryptography, on-disk/on-wire formats, and
-  S3-visible semantics. Cover all applicable lenses using exactly two
-  independent reviewers when delegation is explicitly authorized. Split the
-  lenses between them. Otherwise perform two fresh sequential passes.
+- **Exempt:** docs/comments, formatting, and typos that cannot affect runtime,
+  builds, tests, or agent execution. Skip this section.
+- **Mechanical:** pure renames, file moves, test-only or tooling changes, and
+  agent-instruction changes that alter execution —
+  correctness and simplicity adversaries only.
+- **Standard (the default):** any change that affects behavior.
+- **High risk:** touches locking, erasure coding, quorum/heal, replication,
+  multipart, RPC, lifecycle/tiering, metadata formats (`xl.meta`),
+  persistence/fsync, IAM/KMS/auth, on-disk or on-wire formats, or
+  S3 API-visible behavior.
 
-Available domain lenses are security, concurrency/durability, compatibility,
-and performance. Select `.agents/skills/adversarial-validation/SKILL.md` for an
-explicit adversarial request, a high-risk change, or a substantial PR review;
-then read only its matching role references. A routine standard pass does not
-load the playbook unless the reviewer needs a RustFS-specific probe.
+### Roles
 
-A finding must name a concrete input/state/interleaving and wrong outcome, or a
-specific missing regression check, with `file:line`. Resolve it by fixing the
-diff or rebutting it with code-path/test/invariant evidence. After a non-trivial
-fix, rerun only affected lenses.
+Run each applicable role as an independent pass over the final diff (or
+proposal text) — parallel reviewer agents where the tooling supports them,
+otherwise sequential passes that each start fresh from the diff and the
+nearest scoped `AGENTS.md`, discarding the writing session's assumptions.
+Each role either produces findings or reports "attacked X, Y, Z — no break
+found"; a bare pass is not a result. Repo-specific attack probes for every
+role live in `.agents/skills/adversarial-validation/` — run them, they
+encode this repo's shipped bugs.
 
-For high-risk PRs, record one concise verdict per covered lens in the PR body.
+- **Correctness adversary** — construct a concrete input/state/interleaving
+  that yields wrong output, data loss, or a crash. Probe error paths and edge
+  values (empty, nil UUID, zero-length, quorum−1, missing version).
+- **Simplicity adversary** — same behavior, less code. Hunt reimplemented helpers, rewrites where an in-place edit suffices, speculative abstractions, defensive branches with no nameable trigger, redundant error wrapping, near-duplicate tests, and narration comments. A one-caller helper is a finding only when it merely forwards or splits a short linear flow without adding domain naming, boundary isolation, an invariant, or useful error context. Report a concrete smaller replacement; fewer lines alone are not evidence.
+- **Security reviewer** — authn/authz bypass, injection, secret leakage,
+  untrusted deserialization (see Serde Safety), path traversal, timing leaks.
+- **Concurrency/durability reviewer** — lock ordering, races, cancellation,
+  partial failure, retry/idempotency, crash and power-loss ordering.
+- **Compatibility reviewer** — S3 API surface, MinIO interop, on-disk and
+  on-wire formats, mixed-version upgrade/downgrade paths.
+- **Performance reviewer** — allocation and cloning on hot paths, lock hold
+  time across IO, sync or CPU-heavy work on async runtime threads, added
+  fsync/flush outside the durability gate, hot-path logging noise. A
+  measurable regression on a per-request or per-object path is a finding.
+- **Test-coverage skeptic** — for each testable behavior claim, name the test
+  or executable check that detects a revert; then name a changed line that
+  could be wrong while all checks stay green. If a focused check is not
+  reasonable, require the reason and residual risk from the validation floor.
+  Test additions have no line-count or growth budget.
 
-## Pull Request Lifecycle
+Standard tier: correctness adversary + simplicity adversary + test-coverage
+skeptic, plus every role whose domain the diff touches (async or
+shared-state code → concurrency; parsing of untrusted input → security;
+public crate API shape → compatibility; per-request or per-object hot paths
+→ performance).
+High risk: all seven roles.
 
-- Creating or updating a PR includes one immediate snapshot of checks,
-  mergeability, reviews, and unresolved threads.
-- Unless the user explicitly requests monitoring, a release workflow requires
-  it, or an automation already owns it, hand off after the PR is open with the
-  current state and next event to watch. Do not delay ordinary handoff with
-  fixed quiet-period sleeps.
-- For requested monitoring, use event-driven or bounded waits. Report only state
-  changes, actionable failures, or a meaningful prolonged delay.
-- Investigate failures/comments before changing code. Fix task-attributable
-  issues, rerun affected verification, push, reply or resolve the thread, then
-  resume the requested monitor.
-- Never merge without required reviewer approval or explicit authority.
-- After an observed merge, verify the commit reached the base, then clean the
-  task worktree/branch when safe. Preserve unmerged work for closed PRs unless
-  deletion was explicitly authorized.
+### Protocol
+
+1. A finding states a concrete failure scenario (input/state → wrong
+   outcome) or names a missing test, with severity and file:line. "Looks
+   risky" is not a finding.
+2. Resolve every finding: fix it, or rebut it with evidence — a test, a
+   traced code path, or a cited invariant. Restated intent and "unlikely"
+   are not rebuttals.
+3. After non-trivial fixes, re-run the roles whose domain the fix touched.
+4. For proposals with no diff, roles attack assumptions, failure modes,
+   migration/rollback, and testability instead — including the simplest
+   rejected alternative and the blast radius when the design fails.
+
+### Exit criteria
+
+- Every applicable role has run; every finding is fixed or rebutted with
+  evidence.
+- Every testable behavior change has a focused regression check. Exceptions
+  follow the validation floor and state why a check is impractical and what
+  risk remains.
+- After the applicable adversarial review has completed, the Verification
+  Before PR gates pass; adversarial review supplements those gates, never
+  replaces them.
+- High risk only: record a one-line verdict per role in the PR description.
 
 ## Git and PR Baseline
 
-- Follow Conventional Commits; keep the subject at most 72 characters.
-- Source comments, commits, PR titles, and PR bodies are in English.
-- Keep every heading from `.github/pull_request_template.md`; use `N/A` where
-  needed and include commands actually run.
-- Use `--body-file` for multiline `gh pr create`/`gh pr edit` content.
-- PR/issue/discussion content must not contain the literal sequence `\n` or
-  hard-wrapped prose paragraphs.
-- Do not include local absolute paths or tool-specific labels/prefixes in GitHub
-  content.
-- Resolve review threads after the underlying issue is fixed. If declining a
-  suggestion, reply with a short evidence-based reason.
+- Use feature branches based on the latest `main`.
+- Assume other agent sessions work this repository concurrently. Never commit
+  in a shared checkout; do all work on a dedicated feature branch, preferably
+  in a dedicated worktree.
+- Immediately before branching, fetch `origin/main` and branch from it;
+  confirm the target issue is not already fixed there before writing code.
+- Follow Conventional Commits, with subject length <= 72 characters.
+- Keep PR title and description in English.
+- Use `.github/pull_request_template.md` and keep all section headings.
+- Use `N/A` for non-applicable template sections.
+- Include verification commands in the PR description.
+- When using `gh pr create`/`gh pr edit`, write the markdown body to a file
+  and pass `--body-file`; multiline inline `--body` is unsafe — backticks and
+  shell expansion can corrupt content or trigger unintended commands.
+  Pattern: `cat > /tmp/pr_body.md <<'EOF' ... EOF`, then
+  `--body-file /tmp/pr_body.md` (keep the file outside the checkout).
+- Do not include the literal sequence `\n` in any GitHub issue, pull request, or discussion comment.
+- Do not hard-wrap prose in PR/issue/discussion bodies; write each paragraph as a
+  single line and let it reflow. GitHub renders single newlines inside a paragraph
+  as line breaks, so mid-sentence wrapping shows up as ugly breaks. Only break lines
+  for list items, code blocks, and deliberate separators.
+- After fixing code review comments or CI findings, always mark corresponding review
+  comments/threads as resolved before returning to the user.
+- In handling review comments, confirm the underlying issue before changing code.
+  If a suggested change is not appropriate for behavior or risk, reply with a
+  concise rationale instead of blindly applying it.
 
 ## Security Baseline
 
 - Never commit secrets, credentials, or key material.
 - Use environment variables or vault tooling for sensitive configuration.
-- For localhost-sensitive tests, bypass proxies explicitly.
-- Untrusted S3 XML/JSON, lifecycle, policy, replication, and RPC structures use
-  strict deserialization where compatibility permits. Security-critical
-  defaults require explicit validation.
+- For localhost-sensitive tests, verify proxy settings to avoid traffic leakage.
 
 ## Logging
 
-For every added or edited `tracing` call:
+Applies to **every** `tracing` macro you add or edit, including a single line
+added in passing while fixing something else — not only to log-focused changes.
 
-- Reuse the module's `EVENT_*`, `LOG_COMPONENT_*`, and `LOG_SUBSYSTEM_*`
-  constants and field shape.
-- Put fields first and a short label last.
-- Use `error` for behavior/security failure, `warn` for degradation/fallback,
-  `info` for low-frequency lifecycle, `debug` for diagnostics, and `trace` for
-  repetitive request/object success paths.
-- Never log secrets, credential payloads, or merged configs.
+- Fields first, message second: `event`, `component`, `subsystem`,
+  `result`/`state`, then key context. The message is a short label, not a
+  sentence with values interpolated into it.
+- Reuse the existing `EVENT_*` / `LOG_COMPONENT_*` / `LOG_SUBSYSTEM_*`
+  constants of the module you are editing; match the shape of the log sites
+  already in that file rather than introducing a second style next to them.
+- Level policy: `error` for behavior/security-affecting failures, `warn` for
+  degraded or fallback paths, `info` for low-frequency lifecycle, `debug` for
+  targeted diagnostics, `trace` for hot paths. Per-object and per-request
+  success paths are `trace`.
+- Never log secrets, tokens, credential payloads, or merged config dumps.
+- `scripts/check_logging_guardrails.sh` enforces a subset of this on the files
+  it lists; passing it is a floor, not evidence the log matches the house style.
 
-Use `.agents/skills/rustfs-logging-governance/SKILL.md` for logging changes.
+See `.agents/skills/rustfs-logging-governance/SKILL.md` for the full event
+model, level policy, and guardrail-update checklist.
 
-## Cross-Cutting Storage Invariants
+## Tools
 
-- Write internal object metadata under both `x-rustfs-internal-<suffix>` and
-  `x-minio-internal-<suffix>` using
-  `crates/utils/src/http/metadata_compat.rs` helpers.
-- Read binary UUID metadata with
-  `.and_then(|v| Uuid::from_slice(&v).ok()).filter(|u| !u.is_nil())`; absent,
-  empty, and nil all mean no value.
-- Remote-tier version `None` or `""` means an unversioned bucket; send no
-  `versionId` on tier GET/DELETE.
-- `DataUsageCacheInfo` and `DataUsageEntry` keep their hand-written map
-  serialization and new fields remain `#[serde(default)]` for older readers.
+### xl.meta decode tool Quick Use
 
-## Naming
+```
+cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/file/xl.meta"
+```
 
-Use Rust API naming: `SCREAMING_SNAKE_CASE` constants/statics, `snake_case`
-functions/variables, and `PascalCase` types. Do not rename unrelated existing
-violations.
+## Serde Safety
 
-## Scoped Guidance
+- Add `#[serde(deny_unknown_fields)]` to structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs).
+- When `deny_unknown_fields` is impractical (backward compatibility), at minimum log unknown fields at `warn` level.
+- Never use `#[serde(default)]` on security-critical fields without explicit validation of the resulting value.
 
-Before editing, locate the nearest instructions with:
+## Cross-Cutting Domain Invariants
+
+- Write internal object metadata under **both** `x-rustfs-internal-<suffix>`
+  and `x-minio-internal-<suffix>` keys (MinIO interop). Use the helpers in
+  `crates/utils/src/http/metadata_compat.rs` (`get_bytes` prefers the RustFS
+  key); never write only one of the two.
+- Read binary UUID metadata defensively:
+  `.and_then(|v| Uuid::from_slice(&v).ok()).filter(|u| !u.is_nil())` —
+  absent, empty, and nil all mean "no value", never `Uuid::nil()`.
+- A remote-tier version of `None`/`""` means the tier bucket is unversioned:
+  send **no** `versionId` on tier GET/DELETE.
+- Structs persisted in the scanner data-usage cache (`DataUsageCacheInfo`,
+  `DataUsageEntry`) carry a hand-written map-encoded `Serialize`. MessagePack
+  encodes derived structs as arrays, where an appended field makes the whole
+  cache a decode error for older readers — keep new fields `#[serde(default)]`
+  and keep the map encoding rather than reverting to `derive(Serialize)`.
+
+## Naming Conventions
+
+- Follow Rust API Guidelines for naming: `SCREAMING_SNAKE_CASE` for statics and constants, `snake_case` for functions and variables, `PascalCase` for types.
+- Do not use camelCase or Hungarian notation (e.g., `globalDeploymentIDPtr` → `GLOBAL_DEPLOYMENT_ID`).
+- If existing code violates naming conventions, do not widen the violation in new code. Do not rename existing symbols as part of an unrelated task; mention the violation instead (see Change Style for Existing Logic).
+
+## Scoped Guidance in This Repository
+
+Many crates and modules carry their own `AGENTS.md` with path-specific rules
+(security boundaries, lock ordering, domain invariants). Before editing a
+path, check for the nearest one:
 
 ```bash
 git ls-files '*AGENTS.md'
 ```
 
-The nearest file wins for domain invariants. Keep generic workflow and
-validation policy in this root file.
+The nearest file wins. Do not maintain a hand-written index of these files
+here — it goes stale.

Cargo.lock

@@ -3843,6 +3843,7 @@ dependencies = [
  "s3s",
  "serde",
  "serde_json",
+ "serial_test",
  "sha2 0.11.0",
  "suppaftp",
  "time",
@@ -9256,7 +9257,6 @@ dependencies = [
  "url",
  "urlencoding",
  "uuid",
- "x509-parser",
  "zeroize",
  "zip",
  "zstd",
@@ -9920,6 +9920,7 @@ dependencies = [
  "rustfs-config",
  "rustfs-io-metrics",
  "rustfs-utils",
+ "serial_test",
  "temp-env",
  "tempfile",
  "tokio",
@@ -10292,6 +10293,7 @@ dependencies = [
  "s3s",
  "serde",
  "serde_json",
+ "serial_test",
  "sha2 0.11.0",
  "temp-env",
  "tempfile",

Cargo.toml

@@ -204,7 +204,6 @@ rsa = { version = "=0.10.0-rc.18" }
 rustls = { default-features = false, version = "0.23.43" }
 rustls-native-certs = "0.8"
 rustls-pki-types = "1.15.1"
-x509-parser = "0.18.1"
 sha1 = "0.11.0"
 sha2 = "0.11.0"
 subtle = "2.6"

crates/AGENTS.md

@@ -19,9 +19,7 @@ Applies to all paths under `crates/`.
 
 - Document lock acquisition order when a module uses multiple locks. Never acquire the same set of locks in different orders across code paths.
 - Never hold a `tokio::sync::RwLock`/`Mutex` write guard across `.await` points unless the critical section is unavoidably async and the hold time is bounded.
-- Prefer direct atomic `fetch_*` operations for unconditional updates and
-  `compare_exchange` loops only for conditional updates such as peaks or
-  adaptive state.
+- Prefer `compare_exchange` loops over load-then-store for concurrent counters (peak values, adaptive heuristics).
 - When resetting multi-field atomic statistics, use a version/sequence counter or accept that concurrent readers may see partial snapshots; document the tradeoff.
 - `std::sync::Mutex` is acceptable in async context only when held for a brief, non-`await`-containing critical section. If in doubt, use `tokio::sync::Mutex`.
 
@@ -42,9 +40,7 @@ Applies to all paths under `crates/`.
 - Keep unit tests close to the module they test.
 - Keep integration tests under each crate's `tests/` directory.
 - Add regression tests for bug fixes and behavior changes.
-- Every test needs an observable failure criterion. Direct assertions,
-  delegated assertions, snapshots/properties, `#[should_panic]`, and meaningful
-  `Result` failures are all valid; a call that can silently succeed is not.
+- Every test function must contain at least one `assert!`/`assert_eq!`/`assert_matches!`. A test that only calls code without asserting is not a test.
 - In tests, prefer `.expect("context: what was being tested")` over bare `.unwrap()`. A test failure should tell you which operation failed and with what input.
 
 ## Async and Performance

crates/audit/AGENTS.md

@@ -50,3 +50,4 @@ crate.
   - `cargo test -p rustfs-audit`
   - Focused: `cargo test -p rustfs-audit --test pipeline_layer_test`
   - Focused: `cargo test -p rustfs-audit pipeline`
+- Full gate before commit: `make pre-commit`

crates/config/src/constants/runtime.rs

@@ -57,13 +57,6 @@ pub const DEFAULT_MAX_IO_EVENTS_PER_TICK: usize = 1024;
 pub const DEFAULT_EVENT_INTERVAL: u32 = 61;
 pub const DEFAULT_RNG_SEED: Option<u64> = None; // None means random
 
-/// Dedicated blocking thread pool for fsync/fdatasync operations.
-/// When > 1, fsync operations are isolated from the main blocking pool to
-/// prevent device-bound fsync from starving read operations (pread/stat/open).
-/// Default 0 means auto (no isolation, use main runtime).
-pub const ENV_FSYNC_BLOCKING_THREADS: &str = "RUSTFS_RUNTIME_FSYNC_BLOCKING_THREADS";
-pub const DEFAULT_FSYNC_BLOCKING_THREADS: usize = 0;
-
 // Dial9 Tokio Telemetry Default values
 pub const DEFAULT_RUNTIME_DIAL9_ENABLED: bool = false; // Disabled by default
 pub const DEFAULT_RUNTIME_DIAL9_OUTPUT_DIR: &str = "/var/log/rustfs/telemetry";

crates/e2e_test/AGENTS.md

@@ -28,3 +28,4 @@ follow.
 ## Suggested Validation
 
 - `cargo test --package e2e_test`
+- Full gate before commit: `make pre-commit`

crates/e2e_test/Cargo.toml

@@ -96,6 +96,7 @@ tokio-stream = { workspace = true }
 rustfs-madmin.workspace = true
 rustfs-filemeta.workspace = true
 bytes = { workspace = true, features = ["serde"] }
+serial_test = { workspace = true }
 aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
 aws-sdk-sts = { workspace = true, default-features = false, features = ["default-https-client", "rt-tokio"] }
 aws-config = { workspace = true }

crates/e2e_test/src/common.rs

@@ -30,7 +30,6 @@ use reqwest::StatusCode;
 use rustfs_signer::constants::UNSIGNED_PAYLOAD;
 use rustfs_signer::sign_v4;
 use s3s::Body;
-use serde_json;
 use std::ffi::OsStr;
 use std::fs as stdfs;
 use std::io::ErrorKind;
@@ -1584,156 +1583,6 @@ impl Drop for RustFSTestClusterEnvironment {
     }
 }
 
-/// Send a SigV4-signed HTTP request and return the raw `reqwest::Response`.
-///
-/// Unlike [`signed_s3_request`], this variant accepts `body: Option<Vec<u8>>`
-/// (binary-safe) and reorders parameters so that `access_key`/`secret_key`
-/// appear before the body — matching the convention used by the replication
-/// extension and object-lambda e2e suites.
-pub(crate) async fn signed_request(
-    method: http::Method,
-    url: &str,
-    access_key: &str,
-    secret_key: &str,
-    body: Option<Vec<u8>>,
-    content_type: Option<&str>,
-) -> Result<reqwest::Response, Box<dyn std::error::Error + Send + Sync>> {
-    let uri = url.parse::<http::Uri>()?;
-    let authority = uri.authority().ok_or("request URL missing authority")?.to_string();
-    let mut request = http::Request::builder().method(method.clone()).uri(uri);
-    request = request.header(HOST, authority);
-    request = request.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
-    if let Some(content_type) = content_type {
-        request = request.header(CONTENT_TYPE, content_type);
-    }
-
-    let content_len = body.as_ref().map(|body| body.len() as i64).unwrap_or_default();
-    let signed = sign_v4(request.body(Body::empty())?, content_len, access_key, secret_key, "", "us-east-1");
-
-    let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
-    let client = local_http_client();
-    let mut request_builder = client.request(reqwest_method, url);
-    for (name, value) in signed.headers() {
-        request_builder = request_builder.header(name, value);
-    }
-    if let Some(body) = body {
-        request_builder = request_builder.body(body);
-    }
-
-    Ok(request_builder.send().await?)
-}
-
-/// Like [`signed_request`], but uses a caller-supplied `reqwest::Client`
-/// instead of the shared [`local_http_client`].
-pub(crate) async fn signed_request_with_client(
-    client: &reqwest::Client,
-    method: http::Method,
-    url: &str,
-    access_key: &str,
-    secret_key: &str,
-    body: Option<Vec<u8>>,
-    content_type: Option<&str>,
-) -> Result<reqwest::Response, Box<dyn std::error::Error + Send + Sync>> {
-    let uri = url.parse::<http::Uri>()?;
-    let authority = uri.authority().ok_or("request URL missing authority")?.to_string();
-    let mut request = http::Request::builder().method(method.clone()).uri(uri);
-    request = request.header(HOST, authority);
-    request = request.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
-    if let Some(content_type) = content_type {
-        request = request.header(CONTENT_TYPE, content_type);
-    }
-
-    let content_len = body.as_ref().map(|body| body.len() as i64).unwrap_or_default();
-    let signed = sign_v4(request.body(Body::empty())?, content_len, access_key, secret_key, "", "us-east-1");
-
-    let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
-    let mut request_builder = client.request(reqwest_method, url);
-    for (name, value) in signed.headers() {
-        request_builder = request_builder.header(name, value);
-    }
-    if let Some(body) = body {
-        request_builder = request_builder.body(body);
-    }
-
-    Ok(request_builder.send().await?)
-}
-
-/// Like [`signed_request`], but includes a `session_token` in the
-/// `x-amz-security-token` header and passes it to the SigV4 signer.
-pub(crate) async fn signed_request_with_session_token(
-    method: http::Method,
-    url: &str,
-    access_key: &str,
-    secret_key: &str,
-    session_token: &str,
-    body: Option<Vec<u8>>,
-    content_type: Option<&str>,
-) -> Result<reqwest::Response, Box<dyn std::error::Error + Send + Sync>> {
-    let uri = url.parse::<http::Uri>()?;
-    let authority = uri.authority().ok_or("request URL missing authority")?.to_string();
-    let mut request = http::Request::builder().method(method.clone()).uri(uri);
-    request = request.header(HOST, authority);
-    request = request.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
-    if !session_token.is_empty() {
-        request = request.header("x-amz-security-token", session_token);
-    }
-    if let Some(content_type) = content_type {
-        request = request.header(CONTENT_TYPE, content_type);
-    }
-
-    let content_len = body.as_ref().map(|body| body.len() as i64).unwrap_or_default();
-    let signed = sign_v4(
-        request.body(Body::empty())?,
-        content_len,
-        access_key,
-        secret_key,
-        session_token,
-        "us-east-1",
-    );
-
-    let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
-    let client = local_http_client();
-    let mut request_builder = client.request(reqwest_method, url);
-    for (name, value) in signed.headers() {
-        request_builder = request_builder.header(name, value);
-    }
-    if let Some(body) = body {
-        request_builder = request_builder.body(body);
-    }
-
-    Ok(request_builder.send().await?)
-}
-
-/// Create a new user via the admin API.
-pub(crate) async fn admin_create_user(
-    env: &RustFSTestEnvironment,
-    username: &str,
-    secret_key: &str,
-) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    let url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", env.url, username);
-    let body = serde_json::json!({
-        "secretKey": secret_key,
-        "status": "enabled"
-    });
-    let response = signed_request(
-        http::Method::PUT,
-        &url,
-        &env.access_key,
-        &env.secret_key,
-        Some(body.to_string().into_bytes()),
-        Some("application/json"),
-    )
-    .await?;
-
-    if response.status() != reqwest::StatusCode::OK {
-        let status = response.status();
-        let body = response.text().await.unwrap_or_default();
-        return Err(format!("create user failed: {status} {body}").into());
-    }
-
-    Ok(())
-}
-
 #[cfg(test)]
 mod tests {
     use super::*;

crates/e2e_test/src/degraded_read_eof_regression_test.rs

@@ -55,6 +55,7 @@ mod tests {
     use aws_sdk_s3::Client;
     use aws_sdk_s3::primitives::ByteStream;
     use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
+    use serial_test::serial;
     use sha2::{Digest, Sha256};
     use std::error::Error;
     use tokio::time::{Duration, timeout};
@@ -268,6 +269,7 @@ mod tests {
     /// stripes) and a multipart object (3 parts × 5 MiB) must GET back as a
     /// full, byte-identical body with the correct Content-Length. No early EOF.
     #[tokio::test]
+    #[serial]
     async fn degraded_read_large_objects_with_one_disk_offline_return_full_body() -> TestResult {
         init_logging();
         info!("dist-13 (a): large-object degraded read with one of four disks offline");
@@ -333,6 +335,7 @@ mod tests {
     /// mid-stream — the exact window the fixes had to reconstruct through rather
     /// than truncate.
     #[tokio::test]
+    #[serial]
     async fn degraded_read_reconstructs_through_midstream_bitrot_within_quorum() -> TestResult {
         init_logging();
         info!("dist-13 (b): mid-stream bitrot within quorum must reconstruct a full body");
@@ -390,6 +393,7 @@ mod tests {
     /// Content-Length. `get_checked` panics on that forbidden outcome, so this
     /// test fails loudly if the truncation bug ever returns.
     #[tokio::test]
+    #[serial]
     async fn beyond_quorum_degraded_read_never_silently_truncates() -> TestResult {
         init_logging();
         info!("dist-13 (c): beyond-quorum degraded read must fail, never 200+truncated");

crates/e2e_test/src/get_stream_failure_observability_test.rs

@@ -51,6 +51,7 @@ mod tests {
     use aws_sdk_s3::Client;
     use aws_sdk_s3::primitives::ByteStream;
     use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
+    use serial_test::serial;
     use std::error::Error;
     use tokio::time::{Duration, timeout};
     use tracing::info;
@@ -128,6 +129,7 @@ mod tests {
     /// the body — and assert the server log names the object, at the log level a
     /// default deployment actually runs with.
     #[tokio::test]
+    #[serial]
     async fn midstream_get_failure_is_logged_with_the_object_at_default_log_level() -> TestResult {
         init_logging();
         info!("rustfs#4784: a mid-stream GET failure must name its object in the source log");

crates/e2e_test/src/inline_fast_path_cluster_test.rs

@@ -46,6 +46,7 @@ use prost::Message;
 use rustfs_signer::constants::UNSIGNED_PAYLOAD;
 use rustfs_signer::sign_v4;
 use s3s::Body;
+use serial_test::serial;
 use std::collections::BTreeMap;
 use std::convert::Infallible;
 use std::error::Error;
@@ -1694,6 +1695,7 @@ fn assert_storage_layout(
 }
 
 #[tokio::test]
+#[serial]
 async fn four_node_inline_storage_and_get_boundaries() -> TestResult {
     init_logging();
 
@@ -1765,6 +1767,7 @@ async fn four_node_inline_storage_and_get_boundaries() -> TestResult {
 }
 
 #[tokio::test]
+#[serial]
 async fn four_node_empty_legacy_volumes_start_as_fresh() -> TestResult {
     init_logging();
 
@@ -1802,6 +1805,7 @@ async fn four_node_empty_legacy_volumes_start_as_fresh() -> TestResult {
 }
 
 #[tokio::test]
+#[serial]
 async fn four_node_inline_fallback_controls() -> TestResult {
     init_logging();
 
@@ -1866,6 +1870,7 @@ async fn four_node_inline_fallback_controls() -> TestResult {
 }
 
 #[tokio::test]
+#[serial]
 async fn four_node_compressed_inline_fallback() -> TestResult {
     init_logging();
 
@@ -1900,6 +1905,7 @@ async fn four_node_compressed_inline_fallback() -> TestResult {
 /// Multipart disk compression is live again, so a compression-enabled cluster classifies multipart objects as compressed and the roundtrip (full GET plus partNumber GET) must still return the original bytes.
 /// Reverting the multipart compression fix must fail this test.
 #[tokio::test]
+#[serial]
 async fn four_node_multipart_disk_compression_roundtrip() -> TestResult {
     init_logging();
 
@@ -1946,6 +1952,7 @@ async fn four_node_multipart_disk_compression_roundtrip() -> TestResult {
 /// read costs on the order of the covering part's block size against a ~5 MiB
 /// object.
 #[tokio::test]
+#[serial]
 async fn four_node_compressed_multipart_tail_range_reads_are_bounded() -> TestResult {
     init_logging();
 
@@ -2012,6 +2019,7 @@ async fn four_node_compressed_multipart_tail_range_reads_are_bounded() -> TestRe
 }
 
 #[tokio::test]
+#[serial]
 async fn four_node_mixed_msgpack_compat_mode_preserves_fallback_controls() -> TestResult {
     init_logging();
 
@@ -2115,6 +2123,7 @@ async fn four_node_mixed_msgpack_compat_mode_preserves_fallback_controls() -> Te
 }
 
 #[tokio::test]
+#[serial]
 async fn four_node_add_tier_converges() -> TestResult {
     init_logging();
 
@@ -2133,6 +2142,7 @@ async fn four_node_add_tier_converges() -> TestResult {
 }
 
 #[tokio::test]
+#[serial]
 async fn four_node_add_tier_converges_after_offline_node_restart_without_second_mutation() -> TestResult {
     init_logging();
 
@@ -2154,6 +2164,7 @@ async fn four_node_add_tier_converges_after_offline_node_restart_without_second_
 }
 
 #[tokio::test]
+#[serial]
 async fn four_node_manual_transition_job_status_survives_node_restart() -> TestResult {
     init_logging();
 
@@ -2228,6 +2239,7 @@ async fn four_node_manual_transition_job_status_survives_node_restart() -> TestR
 }
 
 #[tokio::test]
+#[serial]
 async fn four_node_manual_transition_distributed_admission_conflict_reports_status_and_backpressure() -> TestResult {
     init_logging();
 
@@ -2369,6 +2381,7 @@ async fn four_node_manual_transition_distributed_admission_conflict_reports_stat
 }
 
 #[tokio::test]
+#[serial]
 #[ignore = "manual #1508 evidence harness: starts a 4-node cluster, a remote tier, and an in-flight transition job"]
 async fn four_node_manual_transition_rollout_non_empty_restart_readback() -> TestResult {
     init_logging();
@@ -2473,6 +2486,7 @@ async fn four_node_manual_transition_rollout_non_empty_restart_readback() -> Tes
 }
 
 #[tokio::test]
+#[serial]
 async fn four_node_mixed_msgpack_compat_mode_preserves_fallback_controls_during_transition() -> TestResult {
     init_logging();
 
@@ -2584,6 +2598,7 @@ async fn four_node_mixed_msgpack_compat_mode_preserves_fallback_controls_during_
 }
 
 #[tokio::test]
+#[serial]
 async fn four_node_transitioned_inline_fallback() -> TestResult {
     init_logging();
 

crates/e2e_test/src/kms/common.rs

@@ -40,7 +40,7 @@ use std::time::Duration;
 use tokio::fs;
 use tokio::net::TcpStream;
 use tokio::time::sleep;
-use tracing::{debug, error, info, warn};
+use tracing::{debug, error, info};
 
 // KMS-specific constants
 pub const TEST_BUCKET: &str = "kms-test-bucket";
@@ -177,49 +177,6 @@ pub async fn get_kms_status(
     Ok(status)
 }
 
-/// Poll the KMS status endpoint until the backend reports ready or the timeout
-/// expires.  Replaces hard-coded `sleep(Duration::from_secs(3))` startup waits
-/// with an active readiness probe so tests start as soon as KMS is usable
-/// (typically < 1 s) instead of always waiting the full 3 s.
-///
-/// Uses exponential back-off starting at 200 ms (doubling each attempt, capped
-/// at 1 s) up to a total wall-clock budget of 5 s.
-pub async fn wait_for_kms_ready(
-    base_url: &str,
-    access_key: &str,
-    secret_key: &str,
-) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    let total_deadline = Duration::from_secs(5);
-    let start = tokio::time::Instant::now();
-    let mut backoff = Duration::from_millis(200);
-    let max_backoff = Duration::from_secs(1);
-    let mut first_attempt = true;
-
-    loop {
-        if !first_attempt {
-            if start.elapsed() >= total_deadline {
-                return Err("KMS failed to become ready within 5 seconds".into());
-            }
-            sleep(backoff).await;
-            backoff = (backoff * 2).min(max_backoff);
-        }
-        first_attempt = false;
-
-        match get_kms_status(base_url, access_key, secret_key).await {
-            Ok(status) => {
-                info!("KMS is ready (status: {})", status);
-                return Ok(());
-            }
-            Err(e) => {
-                if start.elapsed() >= total_deadline {
-                    return Err(format!("KMS did not become ready within 5 s: last error: {e}").into());
-                }
-                warn!(error = %e, elapsed_ms = start.elapsed().as_millis() as u64, "KMS not ready yet, retrying…");
-            }
-        }
-    }
-}
-
 /// Create a default KMS key for testing and return the created key ID
 pub async fn create_default_key(
     base_url: &str,
@@ -904,13 +861,6 @@ impl LocalKMSTestEnvironment {
         Ok(default_key_id.to_string())
     }
 
-    /// Poll the KMS status endpoint until the backend reports ready.
-    ///
-    /// Prefer this over a fixed `sleep` after calling `start_rustfs_for_local_kms`.
-    pub async fn wait_for_kms_ready(&self) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-        wait_for_kms_ready(&self.base_env.url, &self.base_env.access_key, &self.base_env.secret_key).await
-    }
-
     /// Configure Local KMS backend with a predefined default key
     pub async fn configure_local_kms(&self) -> Result<String, Box<dyn std::error::Error + Send + Sync>> {
         // Use a fixed, predictable default key ID

crates/e2e_test/src/kms/kms_vault_test.rs

@@ -19,6 +19,7 @@
 //! multipart upload behaviour.
 
 use crate::common::{TEST_BUCKET, init_logging};
+use serial_test::serial;
 use tokio::time::{Duration, sleep};
 use tracing::{error, info};
 
@@ -61,6 +62,7 @@ impl VaultKmsTestContext {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_vault_kms_end_to_end() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
     init_logging();
     if skip_if_kms_admin_tool_unavailable("test_vault_kms_end_to_end") {
@@ -116,6 +118,7 @@ async fn test_vault_kms_end_to_end() -> Result<(), Box<dyn std::error::Error + S
 }
 
 #[tokio::test]
+#[serial]
 async fn test_vault_kms_key_isolation() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
     init_logging();
     if skip_if_kms_admin_tool_unavailable("test_vault_kms_key_isolation") {
@@ -202,6 +205,7 @@ async fn test_vault_kms_key_isolation() -> Result<(), Box<dyn std::error::Error
 }
 
 #[tokio::test]
+#[serial]
 async fn test_vault_kms_large_file() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
     init_logging();
     if skip_if_kms_admin_tool_unavailable("test_vault_kms_large_file") {
@@ -266,6 +270,7 @@ async fn test_vault_kms_large_file() -> Result<(), Box<dyn std::error::Error + S
 }
 
 #[tokio::test]
+#[serial]
 async fn test_vault_kms_multipart_upload() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
     init_logging();
     if skip_if_kms_admin_tool_unavailable("test_vault_kms_multipart_upload") {
@@ -296,6 +301,7 @@ async fn test_vault_kms_multipart_upload() -> Result<(), Box<dyn std::error::Err
 }
 
 #[tokio::test]
+#[serial]
 async fn test_vault_kms_key_operations() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
     init_logging();
     if skip_if_kms_admin_tool_unavailable("test_vault_kms_key_operations") {

crates/e2e_test/src/object_lambda_test.rs

@@ -12,11 +12,12 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use crate::common::{RustFSTestClusterEnvironment, RustFSTestEnvironment, init_logging, local_http_client, signed_request};
+use crate::common::{RustFSTestClusterEnvironment, RustFSTestEnvironment, init_logging, local_http_client};
 use aws_sdk_s3::primitives::ByteStream;
 use http::header::{CONTENT_TYPE, HOST};
 use reqwest::StatusCode;
-use rustfs_signer::pre_sign_v4;
+use rustfs_signer::constants::UNSIGNED_PAYLOAD;
+use rustfs_signer::{pre_sign_v4, sign_v4};
 use rustfs_utils::egress::ENV_OUTBOUND_ALLOW_ORIGINS;
 use s3s::Body;
 use std::collections::HashMap;
@@ -226,6 +227,39 @@ async fn presigned_get_request(
     Ok(local_http_client().get(signed.uri().to_string()).send().await?)
 }
 
+async fn signed_request(
+    method: http::Method,
+    url: &str,
+    access_key: &str,
+    secret_key: &str,
+    body: Option<Vec<u8>>,
+    content_type: Option<&str>,
+) -> Result<reqwest::Response, Box<dyn Error + Send + Sync>> {
+    let uri = url.parse::<http::Uri>()?;
+    let authority = uri.authority().ok_or("request URL missing authority")?.to_string();
+    let mut request = http::Request::builder().method(method.clone()).uri(uri);
+    request = request.header(HOST, authority);
+    request = request.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
+    if let Some(content_type) = content_type {
+        request = request.header(CONTENT_TYPE, content_type);
+    }
+
+    let content_len = body.as_ref().map(|body| body.len() as i64).unwrap_or_default();
+    let signed = sign_v4(request.body(Body::empty())?, content_len, access_key, secret_key, "", "us-east-1");
+
+    let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
+    let client = local_http_client();
+    let mut request_builder = client.request(reqwest_method, url);
+    for (name, value) in signed.headers() {
+        request_builder = request_builder.header(name, value);
+    }
+    if let Some(body) = body {
+        request_builder = request_builder.body(body);
+    }
+
+    Ok(request_builder.send().await?)
+}
+
 async fn configure_webhook_target(
     env: &RustFSTestEnvironment,
     target_name: &str,

crates/e2e_test/src/policy/policy_variables_test.rs

@@ -17,6 +17,7 @@
 use crate::common::{awscurl_delete, awscurl_put, init_logging};
 use crate::policy::test_env::PolicyTestEnvironment;
 use aws_sdk_s3::primitives::ByteStream;
+use serial_test::serial;
 use tracing::info;
 
 /// Helper function to create a regular user with given credentials
@@ -121,6 +122,7 @@ async fn cleanup_user_and_policy(env: &PolicyTestEnvironment, username: &str, po
 
 /// Test AWS policy variables with single-value scenarios
 #[tokio::test(flavor = "multi_thread")]
+#[serial]
 #[ignore = "Starts a rustfs server; enable when running full E2E"]
 pub async fn test_aws_policy_variables_single_value() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
     test_aws_policy_variables_single_value_impl().await
@@ -273,6 +275,7 @@ pub async fn test_aws_policy_variables_single_value_impl_with_env(
 
 /// Test AWS policy variables with multi-value scenarios
 #[tokio::test(flavor = "multi_thread")]
+#[serial]
 #[ignore = "Starts a rustfs server; enable when running full E2E"]
 pub async fn test_aws_policy_variables_multi_value() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
     test_aws_policy_variables_multi_value_impl().await
@@ -398,6 +401,7 @@ pub async fn test_aws_policy_variables_multi_value_impl_with_env(
 
 /// Test AWS policy variables with variable concatenation
 #[tokio::test(flavor = "multi_thread")]
+#[serial]
 #[ignore = "Starts a rustfs server; enable when running full E2E"]
 pub async fn test_aws_policy_variables_concatenation() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
     test_aws_policy_variables_concatenation_impl().await
@@ -487,6 +491,7 @@ pub async fn test_aws_policy_variables_concatenation_impl_with_env(
 
 /// Test AWS policy variables with nested scenarios
 #[tokio::test(flavor = "multi_thread")]
+#[serial]
 #[ignore = "Starts a rustfs server; enable when running full E2E"]
 pub async fn test_aws_policy_variables_nested() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
     test_aws_policy_variables_nested_impl().await
@@ -504,6 +509,7 @@ pub async fn test_aws_policy_variables_nested_impl() -> Result<(), Box<dyn std::
 
 /// Test AWS policy variables with STS temporary credentials
 #[tokio::test(flavor = "multi_thread")]
+#[serial]
 #[ignore = "Starts a rustfs server; enable when running full E2E"]
 pub async fn test_aws_policy_variables_sts() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
     test_aws_policy_variables_sts_impl().await
@@ -699,6 +705,7 @@ pub async fn test_aws_policy_variables_sts_impl_with_env(
 
 /// Test AWS policy variables with deny scenarios
 #[tokio::test(flavor = "multi_thread")]
+#[serial]
 #[ignore = "Starts a rustfs server; enable when running full E2E"]
 pub async fn test_aws_policy_variables_deny() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
     test_aws_policy_variables_deny_impl().await

crates/e2e_test/src/policy/test_runner.rs

@@ -14,6 +14,7 @@
 
 use crate::common::init_logging;
 use crate::policy::test_env::PolicyTestEnvironment;
+use serial_test::serial;
 use std::time::Instant;
 use tokio::time::{Duration, sleep};
 use tracing::{error, info};
@@ -212,6 +213,7 @@ impl PolicyTestSuite {
 
 /// Test suite
 #[tokio::test]
+#[serial]
 #[ignore = "Connects to existing rustfs server"]
 async fn test_policy_critical_suite() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
     let config = TestSuiteConfig {

crates/e2e_test/src/protocols/webdav_core.rs

@@ -41,6 +41,7 @@ use reqwest::Client;
 use rustfs_signer::constants::UNSIGNED_PAYLOAD;
 use rustfs_signer::sign_v4;
 use s3s::Body;
+use serial_test::serial;
 use tokio::process::Command;
 use tracing::info;
 
@@ -820,6 +821,7 @@ pub async fn test_webdav_core_operations() -> Result<()> {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_webdav_core_operations_direct() -> Result<()> {
     test_webdav_core_operations().await
 }

crates/e2e_test/src/reliability_disk_fault_test.rs

@@ -27,6 +27,7 @@ mod tests {
     use aws_sdk_s3::Client;
     use aws_sdk_s3::primitives::ByteStream;
     use aws_sdk_s3::types::{BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, VersioningConfiguration};
+    use serial_test::serial;
     use sha2::{Digest, Sha256};
     use std::collections::HashSet;
     use std::error::Error;
@@ -156,6 +157,7 @@ mod tests {
     /// content, degraded writes must succeed, and everything must still
     /// verify after the disk returns.
     #[tokio::test]
+    #[serial]
     async fn test_degraded_read_write_with_one_disk_offline() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         init_logging();
         info!("Reliability: degraded read/write with one of four disks offline");
@@ -208,6 +210,7 @@ mod tests {
     /// bytes to a reader: per-shard bitrot checksums reject the bad shard and
     /// the object is reconstructed from the remaining shards.
     #[tokio::test]
+    #[serial]
     async fn test_bitrot_corrupted_shard_read_returns_correct_data() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         init_logging();
         info!("Reliability: GET must read through a bitrot-corrupted shard");
@@ -250,6 +253,7 @@ mod tests {
     /// heal, and require the replaced disk to be rebuilt and all content to
     /// verify against the sha256 manifest.
     #[tokio::test]
+    #[serial]
     async fn test_fresh_disk_replacement_heals_after_sigkill_restart() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         init_logging();
         info!("Reliability: fresh-disk replacement heals after SIGKILL restart");
@@ -323,6 +327,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_versioned_shard_census_selects_each_version_data_dir() -> Result<(), Box<dyn Error + Send + Sync>> {
         init_logging();
         info!("Reliability: physical shard census selects the requested object version");

crates/e2e_test/src/replacement_privileged_e2e_test.rs

@@ -29,6 +29,7 @@ mod tests {
     use aws_sdk_s3::primitives::ByteStream;
     use aws_sdk_s3::types::{BucketVersioningStatus, CompletedMultipartUpload, CompletedPart, VersioningConfiguration};
     use http::Method;
+    use serial_test::serial;
     use sha2::{Digest, Sha256};
     use std::collections::BTreeSet;
     use std::error::Error;
@@ -1060,6 +1061,7 @@ mod tests {
     /// Linux mount namespaces are per-thread; keep mount setup and process
     /// spawning on one OS thread so child RustFS nodes inherit the test mounts.
     #[tokio::test(flavor = "current_thread")]
+    #[serial]
     #[ignore = "requires Linux root/CAP_SYS_ADMIN and RUSTFS_PRIVILEGED_REPLACEMENT_E2E=1"]
     async fn test_privileged_3x4_auto_replacement_rebuilds_ec8_plus_4_without_admin_heal()
     -> Result<(), Box<dyn Error + Send + Sync>> {
@@ -1073,6 +1075,7 @@ mod tests {
     /// Linux mount namespaces are per-thread; keep mount setup and process
     /// spawning on one OS thread so child RustFS nodes inherit the test mounts.
     #[tokio::test(flavor = "current_thread")]
+    #[serial]
     #[ignore = "requires Linux root/CAP_SYS_ADMIN and RUSTFS_PRIVILEGED_REPLACEMENT_E2E=1"]
     async fn test_privileged_3x4_auto_replacement_rebuilds_ec6_plus_6_without_admin_heal()
     -> Result<(), Box<dyn Error + Send + Sync>> {

crates/e2e_test/src/replication_extension_test.rs

@@ -13,9 +13,8 @@
 // limitations under the License.
 
 use crate::common::{
-    RustFSTestEnvironment, admin_create_user, awscurl_available, awscurl_post_sts_form_urlencoded, init_logging,
-    local_http_client, replication_fast_env, rustfs_binary_path, signed_request, signed_request_with_client,
-    signed_request_with_session_token,
+    RustFSTestEnvironment, awscurl_available, awscurl_post_sts_form_urlencoded, init_logging, local_http_client,
+    replication_fast_env, rustfs_binary_path,
 };
 use crate::fake_s3_target::{
     FAKE_ACCESS_KEY, FAKE_SECRET_KEY, FakeS3Target, FaultAction as FakeTargetFault, Operation as FakeTargetOperation,
@@ -36,7 +35,7 @@ use base64::{Engine, engine::general_purpose::STANDARD as BASE64_STANDARD};
 use bytes::Bytes;
 use flate2::read::GzDecoder;
 use futures::{Stream, StreamExt};
-use http::header::CONTENT_ENCODING;
+use http::header::{CONTENT_ENCODING, CONTENT_TYPE, HOST};
 use http_body_util::{BodyExt, Full};
 use hyper::body::Incoming;
 use hyper::server::conn::http1;
@@ -57,6 +56,9 @@ use rustfs_madmin::{
     AddServiceAccountReq, ListServiceAccountsResp, PeerInfo, PeerSite, ReplicateAddStatus, ReplicateEditStatus,
     ReplicateRemoveStatus, SRRemoveReq, SRResyncOpStatus, SRStatusInfo, SiteReplicationInfo, SyncStatus,
 };
+use rustfs_signer::constants::UNSIGNED_PAYLOAD;
+use rustfs_signer::sign_v4;
+use s3s::Body;
 use s3s::header::X_AMZ_REPLICATION_STATUS;
 use sha2::{Digest, Sha256};
 use std::collections::BTreeMap;
@@ -385,6 +387,116 @@ struct ReplicationResetStatusTarget {
     object: String,
 }
 
+async fn signed_request(
+    method: http::Method,
+    url: &str,
+    access_key: &str,
+    secret_key: &str,
+    body: Option<Vec<u8>>,
+    content_type: Option<&str>,
+) -> Result<reqwest::Response, Box<dyn Error + Send + Sync>> {
+    let uri = url.parse::<http::Uri>()?;
+    let authority = uri.authority().ok_or("request URL missing authority")?.to_string();
+    let mut request = http::Request::builder().method(method.clone()).uri(uri);
+    request = request.header(HOST, authority);
+    request = request.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
+    if let Some(content_type) = content_type {
+        request = request.header(CONTENT_TYPE, content_type);
+    }
+
+    let content_len = body.as_ref().map(|body| body.len() as i64).unwrap_or_default();
+    let signed = sign_v4(request.body(Body::empty())?, content_len, access_key, secret_key, "", "us-east-1");
+
+    let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
+    let client = local_http_client();
+    let mut request_builder = client.request(reqwest_method, url);
+    for (name, value) in signed.headers() {
+        request_builder = request_builder.header(name, value);
+    }
+    if let Some(body) = body {
+        request_builder = request_builder.body(body);
+    }
+
+    Ok(request_builder.send().await?)
+}
+
+async fn signed_request_with_client(
+    client: &reqwest::Client,
+    method: http::Method,
+    url: &str,
+    access_key: &str,
+    secret_key: &str,
+    body: Option<Vec<u8>>,
+    content_type: Option<&str>,
+) -> Result<reqwest::Response, Box<dyn Error + Send + Sync>> {
+    let uri = url.parse::<http::Uri>()?;
+    let authority = uri.authority().ok_or("request URL missing authority")?.to_string();
+    let mut request = http::Request::builder().method(method.clone()).uri(uri);
+    request = request.header(HOST, authority);
+    request = request.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
+    if let Some(content_type) = content_type {
+        request = request.header(CONTENT_TYPE, content_type);
+    }
+
+    let content_len = body.as_ref().map(|body| body.len() as i64).unwrap_or_default();
+    let signed = sign_v4(request.body(Body::empty())?, content_len, access_key, secret_key, "", "us-east-1");
+
+    let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
+    let mut request_builder = client.request(reqwest_method, url);
+    for (name, value) in signed.headers() {
+        request_builder = request_builder.header(name, value);
+    }
+    if let Some(body) = body {
+        request_builder = request_builder.body(body);
+    }
+
+    Ok(request_builder.send().await?)
+}
+
+async fn signed_request_with_session_token(
+    method: http::Method,
+    url: &str,
+    access_key: &str,
+    secret_key: &str,
+    session_token: &str,
+    body: Option<Vec<u8>>,
+    content_type: Option<&str>,
+) -> Result<reqwest::Response, Box<dyn Error + Send + Sync>> {
+    let uri = url.parse::<http::Uri>()?;
+    let authority = uri.authority().ok_or("request URL missing authority")?.to_string();
+    let mut request = http::Request::builder().method(method.clone()).uri(uri);
+    request = request.header(HOST, authority);
+    request = request.header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
+    if !session_token.is_empty() {
+        request = request.header("x-amz-security-token", session_token);
+    }
+    if let Some(content_type) = content_type {
+        request = request.header(CONTENT_TYPE, content_type);
+    }
+
+    let content_len = body.as_ref().map(|body| body.len() as i64).unwrap_or_default();
+    let signed = sign_v4(
+        request.body(Body::empty())?,
+        content_len,
+        access_key,
+        secret_key,
+        session_token,
+        "us-east-1",
+    );
+
+    let reqwest_method = reqwest::Method::from_bytes(method.as_str().as_bytes())?;
+    let client = local_http_client();
+    let mut request_builder = client.request(reqwest_method, url);
+    for (name, value) in signed.headers() {
+        request_builder = request_builder.header(name, value);
+    }
+    if let Some(body) = body {
+        request_builder = request_builder.body(body);
+    }
+
+    Ok(request_builder.send().await?)
+}
+
 fn extract_xml_tag(xml: &str, tag: &str) -> Option<String> {
     let open = format!("<{tag}>");
     let close = format!("</{tag}>");
@@ -904,6 +1016,35 @@ fn create_user_s3_client(env: &RustFSTestEnvironment, access_key: &str, secret_k
     Client::from_conf(config)
 }
 
+async fn admin_create_user(
+    env: &RustFSTestEnvironment,
+    username: &str,
+    secret_key: &str,
+) -> Result<(), Box<dyn Error + Send + Sync>> {
+    let url = format!("{}/rustfs/admin/v3/add-user?accessKey={}", env.url, username);
+    let body = serde_json::json!({
+        "secretKey": secret_key,
+        "status": "enabled"
+    });
+    let response = signed_request(
+        http::Method::PUT,
+        &url,
+        &env.access_key,
+        &env.secret_key,
+        Some(body.to_string().into_bytes()),
+        Some("application/json"),
+    )
+    .await?;
+
+    if response.status() != StatusCode::OK {
+        let status = response.status();
+        let body = response.text().await.unwrap_or_default();
+        return Err(format!("create user failed: {status} {body}").into());
+    }
+
+    Ok(())
+}
+
 async fn admin_add_canned_policy(
     env: &RustFSTestEnvironment,
     policy_name: &str,

crates/ecstore/AGENTS.md

@@ -49,3 +49,4 @@ Applies to `crates/ecstore/`.
 ## Suggested Validation
 
 - `cargo test -p rustfs-ecstore`
+- Full gate before commit: `make pre-commit`

crates/ecstore/src/bucket/bucket_target_sys.rs

@@ -866,7 +866,7 @@ impl BucketTargetSys {
             return Some(cli);
         }
 
-        // TODO(backlog): spawn an async task to proactively reload the replication target
+        // TODO: spawn a task to reload the target
         if self.is_reloading_target(bucket, arn).await {
             return None;
         }

crates/ecstore/src/cluster/rpc/peer_s3_client.rs

@@ -454,7 +454,7 @@ impl S3PeerSys {
                 }
             }
             topology_complete &= bucket_map.values().all(|count| *count >= quorum);
-            // TODO(backlog): integrate MRF backlog stats into scanner bucket listing
+            // TODO: MRF
         }
 
         let mut buckets: Vec<BucketInfo> = result_map.into_values().collect();

crates/ecstore/src/cluster/rpc/remote_disk.rs

@@ -2406,7 +2406,7 @@ impl DiskAPI for RemoteDisk {
             return errors;
         }
 
-        // TODO(backlog): replace string errors with typed `StorageError` variants
+        // TODO: use Error not string
 
         let result = self
             .execute_with_timeout(

crates/ecstore/src/core/sets.rs

@@ -249,7 +249,7 @@ impl Sets {
 
         self.connect_disks().await;
 
-        // TODO(backlog): make monitor_and_connect interval configurable instead of hardcoded 15s
+        // TODO: config interval
         let mut interval = tokio::time::interval(Duration::from_secs(15));
         loop {
             tokio::select! {

crates/ecstore/src/disk/local.rs

@@ -5215,8 +5215,8 @@ impl LocalDisk {
 
         let cache = Cache::new(update_fn, Duration::from_secs(1), Opts::default());
 
-        // TODO(backlog): add O_DIRECT I/O support for performance-critical paths
-        // TODO(backlog): populate DiskInfo in constructor
+        // TODO: DIRECT support
+        // TODD: DiskInfo
         let mut disk = Self {
             root: root.clone(),
             publication_root,
@@ -5751,7 +5751,7 @@ impl LocalDisk {
 
         // return Ok(());
 
-        // TODO(backlog): make disk space checks and trash cleanup event-driven instead of poll-based
+        // TODO: async notifications for disk space checks and trash cleanup
 
         let trash_path = self.io_get_object_path(RUSTFS_META_TMP_DELETED_BUCKET, Uuid::new_v4().to_string().as_str())?;
         // if let Some(parent) = trash_path.parent() {
@@ -5997,7 +5997,7 @@ impl LocalDisk {
 
     #[hotpath::measure(impl_type = "LocalDisk")]
     async fn read_all_data(&self, volume: &str, volume_dir: impl AsRef<Path>, file_path: impl AsRef<Path>) -> Result<Vec<u8>> {
-        // TODO(backlog): add configurable timeout for read_all_data operations
+        // TODO: timeout support
         let (data, _) = self.read_all_data_with_dmtime(volume, volume_dir, file_path).await?;
         Ok(data)
     }
@@ -6674,7 +6674,7 @@ impl LocalDisk {
             return Ok(());
         }
 
-        // TODO(backlog): add directory listing lock to prevent concurrent enumeration
+        // TODO: add lock
 
         let stall = opts.stall_timeout_duration();
 
@@ -8796,7 +8796,7 @@ impl DiskAPI for LocalDisk {
         Ok(entries)
     }
 
-    // TODO(backlog): support io.writer cancellation and early termination in walk_dir
+    // FIXME: TODO: io.writer TODO cancel
     #[tracing::instrument(level = "trace", skip_all)]
     async fn walk_dir<W: AsyncWrite + Unpin + Send>(&self, opts: WalkDirOptions, wr: &mut W) -> Result<()> {
         self.wait_for_startup_cleanup().await;
@@ -9880,7 +9880,7 @@ impl DiskAPI for LocalDisk {
                 );
                 return Err(e);
             }
-            // TODO(backlog): add post-setup disk health verification
+            // TODO: health check
         }
         Ok(())
     }

crates/ecstore/src/disk/os.rs

@@ -315,7 +315,7 @@ pub async fn fsync_dir(dir: impl AsRef<Path>) -> io::Result<()> {
     #[cfg(unix)]
     {
         let dir = dir.as_ref().to_path_buf();
-        fsync_spawn_blocking(move || fsync_dir_std(dir)).await?
+        tokio::task::spawn_blocking(move || fsync_dir_std(dir)).await?
     }
 
     #[cfg(not(unix))]
@@ -683,7 +683,7 @@ async fn fsync_open_dst_dir_group(group: &DstDirFsyncGroup) -> io::Result<()> {
     #[cfg(test)]
     let dir = group.dir.clone();
     let dir_file = group.dir_file.clone();
-    fsync_spawn_blocking(move || {
+    tokio::task::spawn_blocking(move || {
         #[cfg(test)]
         {
             if let Some(kind) = fsync_dir_recorder::take_grouped_failure(&dir) {
@@ -1080,44 +1080,6 @@ const TEST_GLOBAL_FILE_SYNCS: usize = 64;
 
 static FILE_SYNC_PERMITS: LazyLock<Semaphore> = LazyLock::new(|| Semaphore::new(global_file_sync_limit()));
 static DISK_FILE_SYNC_LIMITERS: LazyLock<Mutex<HashMap<PathBuf, Weak<Semaphore>>>> = LazyLock::new(|| Mutex::new(HashMap::new()));
-
-/// Dedicated tokio runtime for fsync/fdatasync blocking operations. When
-/// configured with >1 threads, isolates device-bound fsync from the main
-/// blocking pool so reads (pread/stat/open) are not starved. `None` means
-/// fall back to the main runtime (zero behavior change).
-static FSYNC_RUNTIME: LazyLock<Option<tokio::runtime::Runtime>> = LazyLock::new(|| {
-    let threads =
-        rustfs_utils::get_env_usize(rustfs_config::ENV_FSYNC_BLOCKING_THREADS, rustfs_config::DEFAULT_FSYNC_BLOCKING_THREADS);
-    if threads <= 1 {
-        return None;
-    }
-    let mut builder = tokio::runtime::Builder::new_multi_thread();
-    builder
-        .worker_threads(num_cpus::get().min(8))
-        .max_blocking_threads(threads)
-        .thread_name("rustfs-fsync")
-        .thread_stack_size(512 * 1024)
-        .enable_all();
-    match builder.build() {
-        Ok(rt) => {
-            tracing::info!(threads, "fsync dedicated blocking pool enabled");
-            Some(rt)
-        }
-        Err(err) => {
-            tracing::warn!(%err, "failed to build fsync runtime, falling back to main pool");
-            None
-        }
-    }
-});
-
-/// Spawn a blocking task on the fsync-dedicated runtime if configured,
-/// otherwise fall back to the main tokio blocking pool.
-fn fsync_spawn_blocking<T: Send + 'static>(f: impl FnOnce() -> T + Send + 'static) -> tokio::task::JoinHandle<T> {
-    match FSYNC_RUNTIME.as_ref() {
-        Some(rt) => rt.spawn_blocking(f),
-        None => tokio::task::spawn_blocking(f),
-    }
-}
 static DISK_VOLUME_MUTATION_LOCKS: LazyLock<Mutex<HashMap<PathBuf, Weak<RwLock<()>>>>> =
     LazyLock::new(|| Mutex::new(HashMap::new()));
 type NamespaceMutationLock = AsyncMutex<()>;
@@ -1255,7 +1217,7 @@ where
     F: FnOnce() -> io::Result<T> + Send + 'static,
 {
     let (disk_permit, global_permit) = acquire_file_sync_permits(disk_permits).await?;
-    let result = fsync_spawn_blocking(move || {
+    let result = tokio::task::spawn_blocking(move || {
         let _disk_permit = disk_permit;
         work()
     })
@@ -2184,7 +2146,7 @@ async fn run_blocking_namespace_file_sync_operation_with_global<T: Send + 'stati
         wait_started,
     );
     let disk_permit = admission.disk_permit.clone();
-    let result = fsync_spawn_blocking(move || {
+    let result = tokio::task::spawn_blocking(move || {
         let _lease = lease;
         let _disk_permit = disk_permit;
         operation()

crates/ecstore/src/layout/endpoints.rs

@@ -249,7 +249,7 @@ impl PoolEndpointList {
             endpoint.set_set_index(0);
             endpoint.set_disk_index(0);
 
-            // TODO(backlog): check for cross-device mounts in single-drive setup
+            // TODO Check for cross device mounts if any.
 
             return Ok(Self {
                 inner: vec![Endpoints::from(vec![endpoint])],
@@ -264,7 +264,7 @@ impl PoolEndpointList {
                 // Convert args to endpoints
                 let mut eps = Endpoints::try_from(set_layout.as_slice())?;
 
-                // TODO(backlog): check for cross-device mounts in multi-pool setup
+                // TODO Check for cross device mounts if any.
 
                 for (disk_idx, ep) in eps.as_mut().iter_mut().enumerate() {
                     ep.set_pool_index(pool_idx);

crates/ecstore/src/object_api/types.rs

@@ -1091,7 +1091,7 @@ impl ObjectInfo {
                     }
                 };
 
-                // TODO(backlog): handle VersionPurgeStatus in object listing
+                // TODO:VersionPurgeStatus
                 let versioned = vcfg.clone().map(|v| v.0.versioned(&entry.name)).unwrap_or_default();
                 objects.push(ObjectInfo::from_file_info(&fi, bucket, &entry.name, versioned));
 

crates/ecstore/src/set_disk/ops/multipart.rs

@@ -1575,7 +1575,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
         let parts_metadata = vec![fi.clone(); disks.len()];
 
         if !user_defined.contains_key("content-type") {
-            // TODO(backlog): detect content-type from part data when header is missing
+            // TODO: get content-type
         }
 
         if let Some(sc) = user_defined.get(AMZ_STORAGE_CLASS)
@@ -1971,7 +1971,7 @@ impl crate::storage_api_contracts::multipart::MultipartOperations for SetDisks {
                 return Err(Error::InvalidPart(p.part_num, ext_part.etag.clone(), p.etag.clone().unwrap_or_default()));
             }
 
-            // TODO(backlog): integrate encryption verification during complete multipart
+            // TODO: crypto
 
             if (i < uploaded_parts.len() - 1)
                 && !(opts.data_movement && ext_part.actual_size < 0)

crates/ecstore/src/set_disk/ops/object.rs

@@ -6161,7 +6161,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
 
         join_all(rollback_futures).await;
 
-        // TODO(backlog): support partial object deletion for multi-part objects
+        // TODO: add_partial
 
         if let Some(api) = opts.tier_delete_journal_api.as_ref() {
             for (idx, je) in persisted_journal_entries {
@@ -6371,7 +6371,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
             }
         }
 
-        // TODO(backlog): integrate lifecycle evaluation before object deletion
+        // TODO: Lifecycle
 
         let mut version_found = true;
         // delete_object_version below derives its own majority quorum from the
@@ -6465,7 +6465,7 @@ impl crate::storage_api_contracts::object::ObjectOperations for SetDisks {
                 mark_deleted: mark_delete,
                 mod_time: Some(mod_time),
                 replication_state_internal: opts.delete_replication.as_ref().map(replication_state_to_filemeta),
-                ..Default::default() // TODO(backlog): populate transition state on delete markers
+                ..Default::default() // TODO: Transition
             };
 
             fi.set_tier_free_version_id(&find_vid.to_string());

crates/ecstore/src/store/bucket.rs

@@ -601,7 +601,7 @@ impl ECStore {
 
     #[instrument(skip(self))]
     pub(super) async fn handle_list_bucket(&self, opts: &BucketOptions) -> Result<Vec<BucketInfo>> {
-        // TODO(backlog): support cached bucket listing via opts.cached
+        // TODO: opts.cached
 
         let mut buckets = self.peer_sys.list_bucket(opts).await?;
 

crates/ecstore/src/store/list_objects.rs

@@ -4673,7 +4673,7 @@ async fn gather_results(
             entry.name = entry.name.replace("\\", "/");
         }
 
-        // TODO(backlog): integrate rx.recv() for incremental listing results
+        // TODO: rx.recv()
 
         if let Some(marker) = &opts.marker
             && ((!opts.include_marker && &entry.name <= marker) || (opts.include_marker && &entry.name < marker))
@@ -4703,7 +4703,7 @@ async fn gather_results(
             continue;
         }
 
-        // TODO(backlog): integrate lifecycle evaluation during object listing
+        // TODO: Lifecycle
 
         entries.push(Some(entry));
         candidate_entries += 1;

crates/ecstore/src/store/multipart.rs

@@ -332,7 +332,7 @@ impl ECStore {
         let expected_incarnation_id = opts.expected_bucket_incarnation_id;
 
         if request.prefix.is_empty() {
-            // TODO(backlog): return cached multipart listing when prefix is empty
+            // TODO: return from cache
         }
 
         if self.single_pool() {
@@ -610,7 +610,7 @@ impl ECStore {
         let (opts, _bucket_lifecycle_guard) = self.guard_multipart_bucket_incarnation(bucket, opts).await?;
         let opts = &opts;
 
-        // TODO(backlog): defer DeleteUploadID to background for faster abort response
+        // TODO: defer DeleteUploadID
 
         if self.single_pool() {
             return self.pools[0].abort_multipart_upload(bucket, object, upload_id, opts).await;

crates/ecstore/src/store/rebalance.rs

@@ -385,7 +385,7 @@ impl ECStore {
     }
 
     pub(super) async fn is_suspended(&self, idx: usize) -> bool {
-        // TODO(backlog): acquire pool metadata lock for consistent suspension check
+        // TODO: LOCK
 
         let pool_meta = self.pool_meta.read().await;
 

crates/heal/src/heal/erasure_healer.rs

@@ -373,11 +373,6 @@ impl ErasureSetHealer {
         set_disk_id: &str,
         buckets: &[String],
     ) -> Result<(ResumeManager, CheckpointManager)> {
-        if self.replacement_task_id.is_none() && CheckpointManager::is_blocked(&self.disk, task_id).await {
-            return Err(Error::TaskExecutionFailed {
-                message: format!("Resume task {task_id} has a blocked checkpoint"),
-            });
-        }
         // check if resume state exists
         let has_resume_state = if self.replacement_task_id.is_some() {
             ResumeManager::has_replacement_intent(&self.disk, task_id).await

crates/heal/src/heal/resume.rs

@@ -51,7 +51,6 @@ const RESUME_STATE_FILE: &str = "ahm_resume_state.json";
 const REPLACEMENT_INTENT_FILE: &str = "ahm_replacement_intent.json";
 const RESUME_PROGRESS_FILE: &str = "ahm_progress.json";
 pub(super) const RESUME_CHECKPOINT_FILE: &str = "ahm_checkpoint.json";
-pub(super) const RESUME_CHECKPOINT_BLOCKED_FILE: &str = "ahm_checkpoint.blocked";
 const REPLACEMENT_COMPLETION_PROOF_FILE: &str = "ahm_replacement_completion_proof.json";
 const REPLACEMENT_RECOVERY_DIR: &str = "ahm-replacement";
 const REPLACEMENT_INTENT_SEAL_FILE: &str = "ahm_replacement_intent_seal";

crates/heal/src/heal/resume/checkpoint.rs

@@ -18,14 +18,13 @@ use std::collections::HashSet;
 use std::path::Path;
 use std::sync::{Arc, Mutex};
 use std::time::{SystemTime, UNIX_EPOCH};
-use tokio::sync::{Mutex as AsyncMutex, RwLock};
+use tokio::sync::RwLock;
 use tracing::{debug, warn};
 
-use super::super::storage_api::owner::{EcstoreConditionalFileUpdate, EcstoreDiskAPI, EcstoreDiskBytes};
-use super::super::{BUCKET_META_PREFIX, DiskStore, HealDiskExt, RUSTFS_META_BUCKET};
+use super::super::{BUCKET_META_PREFIX, DiskStore, HealDiskExt as _, RUSTFS_META_BUCKET};
 use super::{
-    LOG_COMPONENT_HEAL, LOG_SUBSYSTEM_RESUME, PersistThrottle, RESUME_CHECKPOINT_BLOCKED_FILE, RESUME_CHECKPOINT_FILE,
-    delete_resume_file, path_to_str, validate_resume_task_id,
+    LOG_COMPONENT_HEAL, LOG_SUBSYSTEM_RESUME, PersistThrottle, RESUME_CHECKPOINT_FILE, delete_resume_file, path_to_str,
+    validate_resume_task_id,
 };
 
 const EVENT_HEAL_CHECKPOINT_STATE: &str = "heal_checkpoint_state";
@@ -117,108 +116,17 @@ pub struct CheckpointManager {
     disk: DiskStore,
     checkpoint: Arc<RwLock<ResumeCheckpoint>>,
     throttle: Mutex<PersistThrottle>,
-    save_lock: AsyncMutex<()>,
-    last_saved: Mutex<Option<EcstoreDiskBytes>>,
 }
 
 impl CheckpointManager {
-    fn blocked_path(task_id: &str) -> std::path::PathBuf {
-        Path::new(BUCKET_META_PREFIX).join(format!("{task_id}_{RESUME_CHECKPOINT_BLOCKED_FILE}"))
-    }
-
-    /// Return whether a checkpoint was permanently isolated after a malformed
-    /// or unsupported snapshot was observed.
-    pub(crate) async fn is_blocked(disk: &DiskStore, task_id: &str) -> bool {
-        if validate_resume_task_id(task_id).is_err() {
-            return false;
-        }
-        let blocked_path = Self::blocked_path(task_id);
-        let Ok(path) = path_to_str(&blocked_path) else {
-            return false;
-        };
-        match HealDiskExt::read_all(disk.as_ref(), RUSTFS_META_BUCKET, path).await {
-            Ok(_) => true,
-            Err(crate::heal::DiskError::FileNotFound) => false,
-            Err(_) => true,
-        }
-    }
-
-    /// Validate the checkpoint while enumerating resumable state. This reads
-    /// the checkpoint once and also isolates malformed or unsupported data.
-    pub(crate) async fn is_resumable(disk: &DiskStore, task_id: &str) -> bool {
-        if validate_resume_task_id(task_id).is_err() || Self::is_blocked(disk, task_id).await {
-            return false;
-        }
-        let file_path = Path::new(BUCKET_META_PREFIX).join(format!("{task_id}_{RESUME_CHECKPOINT_FILE}"));
-        let Ok(path) = path_to_str(&file_path) else {
-            return false;
-        };
-        match HealDiskExt::read_all(disk.as_ref(), RUSTFS_META_BUCKET, path).await {
-            Ok(bytes) if bytes.is_empty() => true,
-            Ok(bytes) => Self::load_from_data(disk.clone(), task_id, bytes.to_vec()).await.is_ok(),
-            Err(crate::heal::DiskError::FileNotFound) => true,
-            Err(_) => false,
-        }
-    }
-
-    async fn block_invalid_snapshot(disk: &DiskStore, task_id: &str) {
-        // This marker is intentionally version-agnostic: an unsupported reader
-        // must stop selector retries until an operator cleans up the snapshot.
-        let blocked_path = Self::blocked_path(task_id);
-        let Ok(path) = path_to_str(&blocked_path) else {
-            return;
-        };
-        let result = EcstoreDiskAPI::compare_and_update_file(
-            disk.as_ref(),
-            RUSTFS_META_BUCKET,
-            path,
-            None,
-            Some(EcstoreDiskBytes::from_static(b"blocked")),
-        )
-        .await;
-        match result {
-            Ok(EcstoreConditionalFileUpdate::Updated | EcstoreConditionalFileUpdate::Mismatch) => {}
-            Ok(EcstoreConditionalFileUpdate::Missing) => warn!(
-                target: "rustfs::heal::resume",
-                event = EVENT_HEAL_CHECKPOINT_STATE,
-                component = LOG_COMPONENT_HEAL,
-                subsystem = LOG_SUBSYSTEM_RESUME,
-                task_id,
-                state = "blocked_marker_write_failed",
-                error = "marker target disappeared",
-                "Heal checkpoint could not persist its blocked marker"
-            ),
-            Err(error) => warn!(
-                target: "rustfs::heal::resume",
-                event = EVENT_HEAL_CHECKPOINT_STATE,
-                component = LOG_COMPONENT_HEAL,
-                subsystem = LOG_SUBSYSTEM_RESUME,
-                task_id,
-                state = "blocked_marker_write_failed",
-                error = %error,
-                "Heal checkpoint could not persist its blocked marker"
-            ),
-        }
-    }
-
     /// create new checkpoint manager
     pub async fn new(disk: DiskStore, task_id: String) -> Result<Self> {
         validate_resume_task_id(&task_id)?;
-        let checkpoint_volume = format!("{RUSTFS_META_BUCKET}/{BUCKET_META_PREFIX}");
-        if let Err(error) = EcstoreDiskAPI::make_volume(disk.as_ref(), &checkpoint_volume).await
-            && error != crate::heal::DiskError::VolumeExists
-        {
-            return Err(Error::TaskExecutionFailed {
-                message: format!("Failed to create checkpoint volume: {error}"),
-            });
-        }
         let checkpoint = ResumeCheckpoint::new(task_id);
         let manager = Self {
             disk,
             checkpoint: Arc::new(RwLock::new(checkpoint)),
             throttle: Mutex::new(PersistThrottle::new()),
-            save_lock: AsyncMutex::new(()),
-            last_saved: Mutex::new(None),
         };
 
         // save initial checkpoint
@@ -232,7 +140,6 @@ impl CheckpointManager {
                 error = %e,
                 "Heal checkpoint persistence failed"
             );
-            return Err(e);
         }
         Ok(manager)
     }
@@ -241,22 +148,11 @@ impl CheckpointManager {
     pub async fn load_from_disk(disk: DiskStore, task_id: &str) -> Result<Self> {
         validate_resume_task_id(task_id)?;
         let checkpoint_data = Self::read_checkpoint_file(&disk, task_id).await?;
-        Self::load_from_data(disk, task_id, checkpoint_data).await
-    }
-
-    async fn load_from_data(disk: DiskStore, task_id: &str, checkpoint_data: Vec<u8>) -> Result<Self> {
-        validate_resume_task_id(task_id)?;
-        let mut checkpoint: ResumeCheckpoint = match serde_json::from_slice(&checkpoint_data) {
-            Ok(checkpoint) => checkpoint,
-            Err(error) => {
-                Self::block_invalid_snapshot(&disk, task_id).await;
-                return Err(Error::TaskExecutionFailed {
-                    message: format!("Failed to deserialize checkpoint: {error}"),
-                });
-            }
-        };
+        let mut checkpoint: ResumeCheckpoint =
+            serde_json::from_slice(&checkpoint_data).map_err(|e| Error::TaskExecutionFailed {
+                message: format!("Failed to deserialize checkpoint: {e}"),
+            })?;
         if checkpoint.task_id != task_id {
-            Self::block_invalid_snapshot(&disk, task_id).await;
             return Err(Error::TaskExecutionFailed {
                 message: "Resume checkpoint task id does not match filename".to_string(),
             });
@@ -267,7 +163,6 @@ impl CheckpointManager {
         // identities. Discard the stale sets and position, then stamp the
         // current schema so the scan restarts cleanly.
         if checkpoint.schema_version > CURRENT_CHECKPOINT_SCHEMA {
-            Self::block_invalid_snapshot(&disk, task_id).await;
             return Err(Error::TaskExecutionFailed {
                 message: format!(
                     "Checkpoint schema {} is newer than supported schema {CURRENT_CHECKPOINT_SCHEMA}",
@@ -299,8 +194,6 @@ impl CheckpointManager {
             disk,
             checkpoint: Arc::new(RwLock::new(checkpoint)),
             throttle: Mutex::new(PersistThrottle::new()),
-            save_lock: AsyncMutex::new(()),
-            last_saved: Mutex::new(Some(EcstoreDiskBytes::from(checkpoint_data))),
         })
     }
 
@@ -311,7 +204,7 @@ impl CheckpointManager {
         }
         let file_path = Path::new(BUCKET_META_PREFIX).join(format!("{task_id}_{RESUME_CHECKPOINT_FILE}"));
         match path_to_str(&file_path) {
-            Ok(path_str) => match HealDiskExt::read_all(disk.as_ref(), RUSTFS_META_BUCKET, path_str).await {
+            Ok(path_str) => match disk.read_all(RUSTFS_META_BUCKET, path_str).await {
                 Ok(data) => !data.is_empty(),
                 Err(_) => false,
             },
@@ -399,7 +292,6 @@ impl CheckpointManager {
 
         let checkpoint_file = Path::new(BUCKET_META_PREFIX).join(format!("{task_id}_{RESUME_CHECKPOINT_FILE}"));
         delete_resume_file(&self.disk, &checkpoint_file).await?;
-        delete_resume_file(&self.disk, &Self::blocked_path(&task_id)).await?;
 
         debug!(
             target: "rustfs::heal::resume",
@@ -415,126 +307,21 @@ impl CheckpointManager {
 
     /// save checkpoint to disk
     async fn save_checkpoint(&self) -> Result<()> {
-        // Serialize saves and take the snapshot only after acquiring the lock:
-        // a slower writer must not publish a snapshot taken before a newer one.
-        let _save_guard = self.save_lock.lock().await;
-        let checkpoint = self.checkpoint.read().await.clone();
+        let checkpoint = self.checkpoint.read().await;
         validate_resume_task_id(&checkpoint.task_id)?;
-        let checkpoint_data =
-            EcstoreDiskBytes::from(serde_json::to_vec(&checkpoint).map_err(|e| Error::TaskExecutionFailed {
-                message: format!("Failed to serialize checkpoint: {e}"),
-            })?);
+        let checkpoint_data = serde_json::to_vec(&*checkpoint).map_err(|e| Error::TaskExecutionFailed {
+            message: format!("Failed to serialize checkpoint: {e}"),
+        })?;
 
         let file_path = Path::new(BUCKET_META_PREFIX).join(format!("{}_{}", checkpoint.task_id, RESUME_CHECKPOINT_FILE));
 
         let path_str = path_to_str(&file_path)?;
-        let last_saved = self
-            .last_saved
-            .lock()
-            .map_err(|_| Error::TaskExecutionFailed {
-                message: "Checkpoint save state lock is poisoned; refusing to save".to_string(),
-            })?
-            .clone();
-        let update = EcstoreDiskAPI::compare_and_update_file(
-            self.disk.as_ref(),
-            RUSTFS_META_BUCKET,
-            path_str,
-            last_saved.clone(),
-            Some(checkpoint_data.clone()),
-        )
-        .await
-        .map_err(|e| Error::TaskExecutionFailed {
-            message: format!("Failed to save checkpoint: {e}"),
-        })?;
-
-        let expected = match update {
-            EcstoreConditionalFileUpdate::Updated => None,
-            EcstoreConditionalFileUpdate::Missing => {
-                return Err(Error::TaskExecutionFailed {
-                    message: "Checkpoint was removed after this manager saved it; refusing to recreate it".to_string(),
-                });
-            }
-            EcstoreConditionalFileUpdate::Mismatch => {
-                // A healthy manager normally completes the CAS above without
-                // another read or JSON parse. Inspect only after a mismatch so
-                // corruption and future schemas cannot be overwritten blindly.
-                let existing = match HealDiskExt::read_all(self.disk.as_ref(), RUSTFS_META_BUCKET, path_str).await {
-                    Ok(existing) => existing,
-                    Err(crate::heal::DiskError::FileNotFound) => {
-                        return Err(Error::TaskExecutionFailed {
-                            message: "Checkpoint was removed after this manager saved it; refusing to recreate it".to_string(),
-                        });
-                    }
-                    Err(error) => {
-                        return Err(Error::TaskExecutionFailed {
-                            message: format!("Failed to inspect checkpoint after CAS mismatch: {error}"),
-                        });
-                    }
-                };
-
-                if existing.is_empty() && last_saved.is_none() {
-                    Some(existing)
-                } else {
-                    let current: ResumeCheckpoint = match serde_json::from_slice(&existing) {
-                        Ok(current) => current,
-                        Err(error) => {
-                            Self::block_invalid_snapshot(&self.disk, &checkpoint.task_id).await;
-                            return Err(Error::TaskExecutionFailed {
-                                message: format!("Existing checkpoint is corrupt: {error}"),
-                            });
-                        }
-                    };
-                    if current.task_id != checkpoint.task_id {
-                        Self::block_invalid_snapshot(&self.disk, &checkpoint.task_id).await;
-                        return Err(Error::TaskExecutionFailed {
-                            message: "Existing checkpoint task id does not match filename".to_string(),
-                        });
-                    }
-                    if current.schema_version > CURRENT_CHECKPOINT_SCHEMA {
-                        Self::block_invalid_snapshot(&self.disk, &checkpoint.task_id).await;
-                        return Err(Error::TaskExecutionFailed {
-                            message: format!(
-                                "Existing checkpoint schema {} is newer than supported schema {CURRENT_CHECKPOINT_SCHEMA}",
-                                current.schema_version
-                            ),
-                        });
-                    }
-                    if last_saved.as_ref().is_none_or(|saved| saved.as_ref() != existing.as_ref()) {
-                        return Err(Error::TaskExecutionFailed {
-                            message: "Checkpoint changed since this manager loaded it; refusing to overwrite newer progress"
-                                .to_string(),
-                        });
-                    }
-                    Some(existing)
-                }
-            }
-        };
-
-        if let Some(expected) = expected {
-            match EcstoreDiskAPI::compare_and_update_file(
-                self.disk.as_ref(),
-                RUSTFS_META_BUCKET,
-                path_str,
-                Some(expected),
-                Some(checkpoint_data.clone()),
-            )
+        self.disk
+            .write_all(RUSTFS_META_BUCKET, path_str, checkpoint_data.into())
             .await
             .map_err(|e| Error::TaskExecutionFailed {
-                message: format!("Failed to save checkpoint after CAS mismatch: {e}"),
-            })? {
-                EcstoreConditionalFileUpdate::Updated => {}
-                EcstoreConditionalFileUpdate::Missing | EcstoreConditionalFileUpdate::Mismatch => {
-                    return Err(Error::TaskExecutionFailed {
-                        message: "Checkpoint changed while saving; refusing to overwrite newer progress".to_string(),
-                    });
-                }
-            }
-        }
-
-        let mut last_saved = self.last_saved.lock().map_err(|_| Error::TaskExecutionFailed {
-            message: "Checkpoint save state lock is poisoned after save".to_string(),
-        })?;
-        *last_saved = Some(checkpoint_data);
+                message: format!("Failed to save checkpoint: {e}"),
+            })?;
 
         debug!(
             target: "rustfs::heal::resume",
@@ -554,7 +341,7 @@ impl CheckpointManager {
         let file_path = Path::new(BUCKET_META_PREFIX).join(format!("{task_id}_{RESUME_CHECKPOINT_FILE}"));
 
         let path_str = path_to_str(&file_path)?;
-        HealDiskExt::read_all(disk.as_ref(), RUSTFS_META_BUCKET, path_str)
+        disk.read_all(RUSTFS_META_BUCKET, path_str)
             .await
             .map(|bytes| bytes.to_vec())
             .map_err(|e| Error::TaskExecutionFailed {

crates/heal/src/heal/resume/tests.rs

@@ -1675,264 +1675,6 @@ async fn future_resume_and_checkpoint_schemas_are_rejected() {
     temp_dir.close().expect("remove schema test directory");
 }
 
-#[tokio::test]
-async fn checkpoint_save_does_not_replace_a_non_empty_truncated_snapshot() {
-    let (temp_dir, disk) = schema_test_disk().await;
-    let task_id = ResumeUtils::generate_task_id();
-    let manager = CheckpointManager::new(disk.clone(), task_id.clone())
-        .await
-        .expect("create checkpoint manager");
-    let checkpoint_path = format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_FILE}");
-    let truncated = b"{\"schema_version\":5,\"task_id\":";
-    disk.write_all(RUSTFS_META_BUCKET, &checkpoint_path, truncated.as_slice().into())
-        .await
-        .expect("write truncated checkpoint fixture");
-
-    let error = manager
-        .update_position(2, 7)
-        .await
-        .expect_err("a truncated checkpoint must fail closed during save");
-    assert!(error.to_string().contains("Existing checkpoint is corrupt"));
-    assert_eq!(
-        disk.read_all(RUSTFS_META_BUCKET, &checkpoint_path)
-            .await
-            .expect("read truncated checkpoint fixture"),
-        truncated.as_slice()
-    );
-    assert!(CheckpointManager::is_blocked(&disk, &task_id).await);
-    temp_dir.close().expect("remove checkpoint save test directory");
-}
-
-#[tokio::test]
-async fn checkpoint_save_does_not_replace_a_future_schema_snapshot() {
-    let (temp_dir, disk) = schema_test_disk().await;
-    let task_id = ResumeUtils::generate_task_id();
-    let manager = CheckpointManager::new(disk.clone(), task_id.clone())
-        .await
-        .expect("create checkpoint manager");
-    let checkpoint_path = format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_FILE}");
-    let mut future = ResumeCheckpoint::new(task_id.clone());
-    future.schema_version = CURRENT_CHECKPOINT_SCHEMA + 1;
-    let future_bytes = serde_json::to_vec(&future).expect("serialize future checkpoint fixture");
-    disk.write_all(RUSTFS_META_BUCKET, &checkpoint_path, future_bytes.clone().into())
-        .await
-        .expect("write future checkpoint fixture");
-
-    let error = manager
-        .update_position(2, 7)
-        .await
-        .expect_err("a future schema must fail closed during save");
-    assert!(error.to_string().contains("Existing checkpoint schema"));
-    assert_eq!(
-        disk.read_all(RUSTFS_META_BUCKET, &checkpoint_path)
-            .await
-            .expect("read future checkpoint fixture"),
-        future_bytes
-    );
-    assert!(CheckpointManager::is_blocked(&disk, &task_id).await);
-    temp_dir.close().expect("remove future schema test directory");
-}
-
-#[tokio::test]
-async fn new_checkpoint_manager_rebuilds_an_empty_snapshot() {
-    let (temp_dir, disk) = schema_test_disk().await;
-    let task_id = ResumeUtils::generate_task_id();
-    let checkpoint_path = format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_FILE}");
-    disk.write_all(RUSTFS_META_BUCKET, &checkpoint_path, EcstoreDiskBytes::new())
-        .await
-        .expect("write empty checkpoint fixture");
-
-    let manager = CheckpointManager::new(disk.clone(), task_id.clone())
-        .await
-        .expect("a new manager must rebuild an empty checkpoint");
-    manager
-        .update_position(3, 11)
-        .await
-        .expect("rebuilt checkpoint must remain writable");
-    assert!(CheckpointManager::has_checkpoint(&disk, &task_id).await);
-    temp_dir.close().expect("remove empty checkpoint test directory");
-}
-
-#[tokio::test]
-async fn deleted_checkpoint_is_not_recreated_by_an_old_manager() {
-    let (temp_dir, disk) = schema_test_disk().await;
-    let task_id = ResumeUtils::generate_task_id();
-    let manager = CheckpointManager::new(disk.clone(), task_id.clone())
-        .await
-        .expect("create checkpoint manager");
-    manager.cleanup().await.expect("delete checkpoint fixture");
-
-    let error = manager
-        .update_position(1, 2)
-        .await
-        .expect_err("an old manager must not resurrect a deleted checkpoint");
-    assert!(error.to_string().contains("removed after this manager saved it"));
-    assert!(!CheckpointManager::has_checkpoint(&disk, &task_id).await);
-    temp_dir.close().expect("remove deleted checkpoint test directory");
-}
-
-#[tokio::test]
-async fn an_empty_blocked_marker_still_blocks_resume_selection() {
-    let (temp_dir, disk) = schema_test_disk().await;
-    let task_id = ResumeUtils::generate_task_id();
-    let manager = CheckpointManager::new(disk.clone(), task_id.clone())
-        .await
-        .expect("create checkpoint manager");
-    let blocked_path = format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_BLOCKED_FILE}");
-    disk.write_all(RUSTFS_META_BUCKET, &blocked_path, EcstoreDiskBytes::new())
-        .await
-        .expect("write empty blocked marker fixture");
-
-    assert!(CheckpointManager::is_blocked(&disk, &task_id).await);
-    assert!(!CheckpointManager::is_resumable(&disk, &task_id).await);
-    // Recovery requires replacing/cleaning the snapshot, then removing the
-    // marker; ordinary selector retries are intentionally not an unlock path.
-    manager.cleanup().await.expect("clean blocked checkpoint");
-    assert!(!CheckpointManager::is_blocked(&disk, &task_id).await);
-    temp_dir.close().expect("remove empty blocked marker test directory");
-}
-
-#[tokio::test]
-async fn resumable_selector_skips_healthy_tasks_with_blocked_markers() {
-    let (temp_dir, disk) = schema_test_disk().await;
-    let tasks = [
-        (ResumeUtils::generate_task_id(), EcstoreDiskBytes::new()),
-        (ResumeUtils::generate_task_id(), EcstoreDiskBytes::from_static(b"blocked")),
-    ];
-    for (task_id, marker) in &tasks {
-        ResumeManager::new(
-            disk.clone(),
-            task_id.clone(),
-            "erasure_set".to_string(),
-            "pool_0_set_0".to_string(),
-            vec!["bucket".to_string()],
-        )
-        .await
-        .expect("create healthy resume state");
-        CheckpointManager::new(disk.clone(), task_id.clone())
-            .await
-            .expect("create healthy checkpoint");
-        let checkpoint_path = format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_FILE}");
-        let checkpoint_bytes = disk
-            .read_all(RUSTFS_META_BUCKET, &checkpoint_path)
-            .await
-            .expect("read healthy checkpoint before blocking");
-        let marker_path = format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_BLOCKED_FILE}");
-        disk.write_all(RUSTFS_META_BUCKET, &marker_path, marker.clone())
-            .await
-            .expect("write blocked marker");
-
-        assert!(
-            ResumeUtils::get_resumable_tasks(&disk)
-                .await
-                .expect("filter blocked healthy task")
-                .is_empty()
-        );
-        assert_eq!(
-            disk.read_all(RUSTFS_META_BUCKET, &checkpoint_path)
-                .await
-                .expect("read healthy checkpoint after blocking"),
-            checkpoint_bytes
-        );
-    }
-    temp_dir.close().expect("remove blocked selector test directory");
-}
-
-#[tokio::test]
-async fn stale_checkpoint_manager_cannot_overwrite_newer_progress() {
-    let (temp_dir, disk) = schema_test_disk().await;
-    let task_id = ResumeUtils::generate_task_id();
-    let first = CheckpointManager::new(disk.clone(), task_id.clone())
-        .await
-        .expect("create first checkpoint manager");
-    let second = CheckpointManager::load_from_disk(disk.clone(), &task_id)
-        .await
-        .expect("load second checkpoint manager");
-
-    second
-        .update_position(4, 20)
-        .await
-        .expect("persist newer checkpoint progress");
-    let error = first
-        .update_position(1, 3)
-        .await
-        .expect_err("stale checkpoint manager must not overwrite newer progress");
-    assert!(error.to_string().contains("newer progress"));
-
-    let persisted = CheckpointManager::load_from_disk(disk.clone(), &task_id)
-        .await
-        .expect("load newer checkpoint progress")
-        .get_checkpoint()
-        .await;
-    assert_eq!(persisted.current_bucket_index, 4);
-    assert_eq!(persisted.current_object_index, 20);
-    temp_dir.close().expect("remove stale manager test directory");
-}
-
-#[tokio::test]
-async fn resumable_selector_isolates_future_and_corrupt_checkpoints() {
-    let (temp_dir, disk) = schema_test_disk().await;
-    let future_task = ResumeUtils::generate_task_id();
-    let corrupt_task = ResumeUtils::generate_task_id();
-    for task_id in [&future_task, &corrupt_task] {
-        ResumeManager::new(
-            disk.clone(),
-            task_id.to_string(),
-            "erasure_set".to_string(),
-            "pool_0_set_0".to_string(),
-            vec!["bucket".to_string()],
-        )
-        .await
-        .expect("create resumable state fixture");
-    }
-
-    let future_path = format!("{BUCKET_META_PREFIX}/{future_task}_{RESUME_CHECKPOINT_FILE}");
-    let mut future = ResumeCheckpoint::new(future_task.clone());
-    future.schema_version = CURRENT_CHECKPOINT_SCHEMA + 1;
-    let future_bytes = serde_json::to_vec(&future).expect("serialize future checkpoint fixture");
-    disk.write_all(RUSTFS_META_BUCKET, &future_path, future_bytes.clone().into())
-        .await
-        .expect("write future checkpoint fixture");
-    let corrupt_path = format!("{BUCKET_META_PREFIX}/{corrupt_task}_{RESUME_CHECKPOINT_FILE}");
-    let corrupt_bytes = b"{truncated";
-    disk.write_all(RUSTFS_META_BUCKET, &corrupt_path, corrupt_bytes.as_slice().into())
-        .await
-        .expect("write corrupt checkpoint fixture");
-
-    assert!(
-        ResumeUtils::get_resumable_tasks(&disk)
-            .await
-            .expect("filter malformed resumable tasks")
-            .is_empty()
-    );
-    assert!(
-        ResumeUtils::get_resumable_tasks(&disk)
-            .await
-            .expect("filter blocked resumable tasks")
-            .is_empty()
-    );
-    for (task_id, path, bytes) in [
-        (&future_task, future_path, future_bytes),
-        (&corrupt_task, corrupt_path, corrupt_bytes.to_vec()),
-    ] {
-        assert_eq!(
-            disk.read_all(RUSTFS_META_BUCKET, &path)
-                .await
-                .expect("read isolated checkpoint bytes"),
-            bytes
-        );
-        let blocked_path = format!("{BUCKET_META_PREFIX}/{task_id}_{RESUME_CHECKPOINT_BLOCKED_FILE}");
-        assert!(
-            !disk
-                .read_all(RUSTFS_META_BUCKET, &blocked_path)
-                .await
-                .expect("read checkpoint blocked marker")
-                .is_empty()
-        );
-    }
-    temp_dir.close().expect("remove selector isolation test directory");
-}
-
 #[test]
 fn test_persist_throttle_batches_until_threshold() {
     let mut throttle = PersistThrottle::new();

crates/heal/src/heal/resume/utils.rs

@@ -21,7 +21,7 @@ use uuid::Uuid;
 use super::super::{BUCKET_META_PREFIX, DiskError, DiskStore, HealDiskExt as _, RUSTFS_META_BUCKET};
 use super::replacement::{ReplacementPhase, ReplacementRecoveryRecord};
 use super::{
-    CheckpointManager, EVENT_HEAL_RESUME_STATE, LOG_COMPONENT_HEAL, LOG_SUBSYSTEM_RESUME, REPLACEMENT_COMPLETION_PROOF_FILE,
+    EVENT_HEAL_RESUME_STATE, LOG_COMPONENT_HEAL, LOG_SUBSYSTEM_RESUME, REPLACEMENT_COMPLETION_PROOF_FILE,
     REPLACEMENT_INTENT_FILE, RESUME_STATE_FILE, ResumeManager, ResumeStateFile, is_replacement_intent, path_to_str,
     replacement_recovery_corruption_for_state_load, replacement_recovery_dir, validate_resume_task_id,
 };
@@ -67,7 +67,6 @@ impl ResumeUtils {
                 // Extract task ID from filename: {task_id}_ahm_resume_state.json
                 if let Some(task_id) = entry.strip_suffix(&format!("_{RESUME_STATE_FILE}"))
                     && validate_resume_task_id(task_id).is_ok()
-                    && CheckpointManager::is_resumable(disk, task_id).await
                 {
                     task_ids.push(task_id.to_string());
                 }

crates/iam/AGENTS.md

@@ -24,3 +24,4 @@ Applies to `crates/iam/`.
 ## Suggested Validation
 
 - `cargo test -p rustfs-iam`
+- Full gate before commit: `make pre-commit`

crates/lifecycle/Cargo.toml

@@ -69,7 +69,7 @@ uuid = { workspace = true, features = ["v4", "serde", "fast-rng", "macro-diagnos
 [dev-dependencies]
 metrics-util = { workspace = true, features = ["debugging"] }
 proptest = "1"
-serial_test = { workspace = true }
+serial_test.workspace = true
 temp-env.workspace = true
 tokio = { workspace = true, features = ["macros", "fs", "rt-multi-thread"] }
 

crates/lifecycle/src/core.rs

@@ -1564,6 +1564,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn abort_incomplete_multipart_upload_due_accepts_zero_days() {
         let initiated = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
         let lc = BucketLifecycleConfiguration {
@@ -1624,6 +1625,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn predict_expiration_selects_closest_expiry_for_put_object() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
         let lc = BucketLifecycleConfiguration {
@@ -1870,6 +1872,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn empty_transition_vectors_are_not_active_or_due() {
         let lc = BucketLifecycleConfiguration {
             expiry_updated_at: None,
@@ -1935,6 +1938,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn eval_inner_keeps_latest_object_before_days_due() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
         let lc = BucketLifecycleConfiguration {
@@ -1968,6 +1972,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn eval_inner_transitions_latest_object_after_days_due() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
         let lc = BucketLifecycleConfiguration {
@@ -2005,6 +2010,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn eval_inner_transitions_latest_object_after_date_due() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
         let transition_date = base_time - Duration::days(1);
@@ -2044,6 +2050,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn eval_inner_selects_earliest_due_among_multiple_past_due_events() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
         // Two enabled rules both yield a past-due DeleteAction and a third yields a
@@ -2157,6 +2164,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn eval_inner_expires_noncurrent_version_after_due() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
         let lc = BucketLifecycleConfiguration {
@@ -2194,6 +2202,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn eval_inner_skips_noncurrent_expiration_without_successor() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).expect("valid fixed test timestamp");
         let lc = BucketLifecycleConfiguration {
@@ -2229,6 +2238,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn eval_inner_missing_successor_does_not_skip_noncurrent_transition() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).expect("valid fixed test timestamp");
         let lc = BucketLifecycleConfiguration {
@@ -2271,6 +2281,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn eval_inner_noncurrent_expiration_one_day_respects_due_boundary() {
         let successor_time = datetime!(2025-06-15 12:00:00 UTC);
         let due = expected_expiry_time(successor_time, 1);
@@ -2312,6 +2323,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn eval_inner_expires_noncurrent_version_immediately_when_zero_days() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
         let lc = BucketLifecycleConfiguration {
@@ -2349,6 +2361,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn eval_inner_transitions_noncurrent_version_after_due() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
         let lc = BucketLifecycleConfiguration {
@@ -2424,6 +2437,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn evaluator_honors_newer_noncurrent_versions_retention_count() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
         let lc = Arc::new(BucketLifecycleConfiguration {
@@ -2712,6 +2726,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn expired_object_delete_marker_ignores_marker_with_noncurrent_versions_present() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
         let lc = BucketLifecycleConfiguration {
@@ -2788,6 +2803,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn expired_object_delete_marker_deletes_only_delete_marker_immediately() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
         let lc = BucketLifecycleConfiguration {
@@ -2865,6 +2881,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn expiration_days_deletes_only_expired_delete_marker_when_due() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
         let lc = BucketLifecycleConfiguration {
@@ -2915,6 +2932,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn expiration_days_uses_earliest_due_rule_for_expired_delete_marker() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
         let make_rule = |id: &str, days| LifecycleRule {
@@ -3245,6 +3263,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn del_marker_expiration_deletes_marker_and_older_versions_when_due() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).expect("fixed timestamp should be valid");
         let lc = BucketLifecycleConfiguration {
@@ -3284,6 +3303,7 @@ mod tests {
     // --- TASK-003 tests: Round up to next UTC processing boundary ---
 
     #[test]
+    #[serial]
     fn expected_expiry_time_rounds_up_to_next_midnight_utc() {
         with_default_ilm_process_time(|| {
             // Object created at 2025-01-15T10:30:45Z, expire in 30 days
@@ -3299,6 +3319,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn expected_expiry_time_immediate_expiry_returns_epoch() {
         with_default_ilm_process_time(|| {
             let mod_time = datetime!(2025-06-01 12:00:00 UTC);
@@ -3308,6 +3329,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn expected_expiry_time_preserves_exact_midnight_boundary() {
         with_default_ilm_process_time(|| {
             let mod_time = datetime!(2025-03-01 00:00:00 UTC);
@@ -3317,6 +3339,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn expected_expiry_time_rounds_end_of_day_to_following_midnight() {
         with_default_ilm_process_time(|| {
             let mod_time = datetime!(2025-06-15 23:59:59 UTC);
@@ -3326,6 +3349,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn expected_expiry_time_uses_canonical_process_time_boundary() {
         let mod_time = datetime!(2025-01-15 10:30:45 UTC);
 
@@ -3338,6 +3362,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn expected_expiry_time_uses_deprecated_process_time_alias() {
         let mod_time = datetime!(2025-01-15 10:30:45 UTC);
 
@@ -3350,6 +3375,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn expected_expiry_time_uses_default_boundary_when_process_time_is_zero_or_invalid() {
         let mod_time = datetime!(2025-01-15 10:30:45 UTC);
 
@@ -3372,6 +3398,7 @@ mod tests {
 
     // (a) Default path (env unset) is byte-identical: one day == 86400s.
     #[test]
+    #[serial]
     fn ilm_day_secs_defaults_to_86400_when_unset() {
         temp_env::with_var_unset(ENV_ILM_DEBUG_DAY_SECS, || {
             assert_eq!(ilm_day_secs(), DEFAULT_ILM_DAY_SECS);
@@ -3400,6 +3427,7 @@ mod tests {
 
     // (b) End-to-end env read scales the day length.
     #[test]
+    #[serial]
     fn ilm_day_secs_scales_when_env_set() {
         temp_env::with_var(ENV_ILM_DEBUG_DAY_SECS, Some("2"), || {
             assert_eq!(ilm_day_secs(), 2);
@@ -3408,6 +3436,7 @@ mod tests {
 
     // (c) Invalid env value falls back to 86400.
     #[test]
+    #[serial]
     fn ilm_day_secs_falls_back_on_invalid_env() {
         temp_env::with_var(ENV_ILM_DEBUG_DAY_SECS, Some("bogus"), || {
             assert_eq!(ilm_day_secs(), DEFAULT_ILM_DAY_SECS);
@@ -3420,6 +3449,7 @@ mod tests {
     // Deadline math scales: with a 1s day and PROCESS_TIME unset, a Days=1 rule is
     // due 1s after mod_time (rounded up to the next 1s boundary => same instant).
     #[test]
+    #[serial]
     fn expected_expiry_time_scales_with_debug_day_secs() {
         let mod_time = datetime!(2025-01-15 10:30:45 UTC);
         temp_env::with_var(ENV_ILM_DEBUG_DAY_SECS, Some("1"), || {
@@ -3435,6 +3465,7 @@ mod tests {
 
     // days == 0 still yields the immediate-expiry sentinel regardless of the switch.
     #[test]
+    #[serial]
     fn expected_expiry_time_zero_days_ignores_debug_day_secs() {
         let mod_time = datetime!(2025-06-01 12:00:00 UTC);
         temp_env::with_var(ENV_ILM_DEBUG_DAY_SECS, Some("2"), || {
@@ -3445,6 +3476,7 @@ mod tests {
     // (③) Interaction with an explicit RUSTFS_ILM_PROCESS_TIME: the deadline offset
     // uses the accelerated day length, but the rounding boundary honors PROCESS_TIME.
     #[test]
+    #[serial]
     fn expected_expiry_time_debug_day_secs_respects_explicit_process_time() {
         let mod_time = datetime!(2025-01-15 10:30:00 UTC);
         // day == 10s, but round up to the next 60s (PROCESS_TIME) boundary.
@@ -3461,6 +3493,7 @@ mod tests {
 
     // (③) With the switch unset, an explicit PROCESS_TIME behaves exactly as before.
     #[test]
+    #[serial]
     fn expected_expiry_time_unset_debug_day_secs_matches_legacy_process_time() {
         let mod_time = datetime!(2025-01-15 10:30:45 UTC);
         temp_env::with_var_unset(ENV_ILM_DEBUG_DAY_SECS, || {
@@ -3488,6 +3521,7 @@ mod tests {
 
     // The abort-incomplete-multipart deadline path also scales through the switch.
     #[test]
+    #[serial]
     fn abort_incomplete_multipart_due_scales_with_debug_day_secs() {
         use s3s::dto::AbortIncompleteMultipartUpload;
         let initiated = datetime!(2025-01-15 10:30:45 UTC);
@@ -3532,6 +3566,7 @@ mod tests {
     // (⑤ evaluator seam) A Days=1 rule fires under RUSTFS_ILM_DEBUG_DAY_SECS=1 once
     // `now` advances a few seconds past a mod_time only ~seconds in the past.
     #[test]
+    #[serial]
     fn eval_inner_expires_days_one_rule_under_debug_day_secs() {
         let lc = BucketLifecycleConfiguration {
             expiry_updated_at: None,
@@ -3580,6 +3615,7 @@ mod tests {
 
     // Absolute Date-based rules must NOT scale with the switch (regression guard).
     #[test]
+    #[serial]
     fn eval_inner_date_rule_ignores_debug_day_secs() {
         let expiry_date = datetime!(2025-06-01 00:00:00 UTC);
         let lc = BucketLifecycleConfiguration {
@@ -3837,6 +3873,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn eval_inner_triggers_delete_all_versions_when_expired_object_all_versions_set() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
         let lc = BucketLifecycleConfiguration {
@@ -3875,6 +3912,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn expired_object_all_versions_does_not_apply_to_current_delete_marker() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).expect("fixed timestamp should be valid");
         let lc = BucketLifecycleConfiguration {
@@ -3904,6 +3942,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn eval_inner_uses_delete_action_when_all_versions_not_set() {
         let base_time = OffsetDateTime::from_unix_timestamp(1_000_000).unwrap();
         let lc = BucketLifecycleConfiguration {
@@ -4022,6 +4061,7 @@ mod tests {
         use super::*;
         use proptest::prelude::*;
         use s3s::dto::{NoncurrentVersionExpiration, Tag};
+        use serial_test::serial;
 
         const DAY_SECS: i64 = 86400;
 
@@ -4252,6 +4292,7 @@ mod tests {
             /// combination, and must be deterministic: the same input
             /// evaluated twice yields an identical event.
             #[test]
+            #[serial]
             fn eval_inner_never_panics_and_is_deterministic(
                 rules in prop::collection::vec(arb_rule(), 0..4),
                 obj in arb_object_opts(),
@@ -4391,6 +4432,7 @@ mod tests {
             /// candidate set — earliest due wins, ties prefer delete-class —
             /// and must be `NoneAction` exactly when that set is empty.
             #[test]
+            #[serial]
             fn eval_inner_winner_matches_selection_oracle(
                 rules in prop::collection::vec(arb_selection_rule(), 0..5),
                 mod_off in 0i64..(2 * DAY_SECS),
@@ -4444,6 +4486,7 @@ mod tests {
             /// non-decreasing in `days` (days == 0 maps to UNIX_EPOCH, below
             /// any post-1970 deadline).
             #[test]
+            #[serial]
             fn expected_expiry_time_is_monotonic_in_days(
                 mod_off in 0i64..(3650 * DAY_SECS),
                 d1 in 0i32..2000,
@@ -4465,6 +4508,7 @@ mod tests {
             /// to the next whole-day boundary: the result is day-aligned, not
             /// before `mod_time + days`, and less than one boundary beyond it.
             #[test]
+            #[serial]
             fn expected_expiry_time_lands_on_default_day_boundary(
                 mod_off in 0i64..(3650 * DAY_SECS),
                 days in 1i32..2000,
@@ -4482,6 +4526,7 @@ mod tests {
             /// to that boundary instead: aligned to it, never early, and less
             /// than one boundary late.
             #[test]
+            #[serial]
             fn expected_expiry_time_lands_on_explicit_process_boundary(
                 mod_off in 0i64..(365 * DAY_SECS),
                 days in 1i32..400,

crates/madmin/src/trace.rs

@@ -12,16 +12,18 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use std::{collections::HashMap, time::Duration};
+
+use jiff::Timestamp;
 use serde::{Deserialize, Serialize};
 
-/// Bitflag helper for service trace categories.
-///
-/// Each variant occupies a single bit so that a `TraceType` value can represent
-/// an arbitrary combination of categories via bitwise OR.
+use crate::heal_commands::HealResultItem;
+
 #[derive(Debug, Clone, Copy, Serialize, Deserialize, Default)]
 pub struct TraceType(u64);
 
 impl TraceType {
+    // Define some constants
     pub const OS: TraceType = TraceType(1 << 0);
     pub const STORAGE: TraceType = TraceType(1 << 1);
     pub const S3: TraceType = TraceType(1 << 2);
@@ -38,13 +40,15 @@ impl TraceType {
     pub const FTP: TraceType = TraceType(1 << 13);
     pub const ILM: TraceType = TraceType(1 << 14);
 
-    /// All trace categories combined. Must be updated when adding new variants.
+    // MetricsAll must be last.
     pub const ALL: TraceType = TraceType((1 << 15) - 1);
 
     pub fn new(t: u64) -> Self {
         Self(t)
     }
+}
 
+impl TraceType {
     pub fn contains(&self, x: &TraceType) -> bool {
         (self.0 & x.0) == x.0
     }
@@ -72,38 +76,140 @@ impl TraceType {
     }
 }
 
+#[derive(Debug, Default, Clone, Serialize, Deserialize)]
+pub struct TraceInfo {
+    #[serde(rename = "type")]
+    trace_type: u64,
+    #[serde(rename = "nodename")]
+    node_name: String,
+    #[serde(rename = "funcname")]
+    func_name: String,
+    #[serde(rename = "time")]
+    time: Timestamp,
+    #[serde(rename = "path")]
+    path: String,
+    #[serde(rename = "dur")]
+    duration: Duration,
+    #[serde(rename = "bytes", skip_serializing_if = "Option::is_none")]
+    bytes: Option<i64>,
+    #[serde(rename = "msg", skip_serializing_if = "Option::is_none")]
+    message: Option<String>,
+    #[serde(rename = "error", skip_serializing_if = "Option::is_none")]
+    error: Option<String>,
+    #[serde(rename = "custom", skip_serializing_if = "Option::is_none")]
+    custom: Option<HashMap<String, String>>,
+    #[serde(rename = "http", skip_serializing_if = "Option::is_none")]
+    http: Option<TraceHTTPStats>,
+    #[serde(rename = "healResult", skip_serializing_if = "Option::is_none")]
+    heal_result: Option<HealResultItem>,
+}
+
+impl TraceInfo {
+    pub fn mask(&self) -> u64 {
+        TraceType::new(self.trace_type).mask()
+    }
+}
+
+#[derive(Debug, Default, Clone, Serialize, Deserialize)]
+pub struct TraceInfoLegacy {
+    trace_info: TraceInfo,
+    #[serde(rename = "request")]
+    req_info: Option<TraceRequestInfo>,
+    #[serde(rename = "response")]
+    resp_info: Option<TraceResponseInfo>,
+    #[serde(rename = "stats")]
+    call_stats: Option<TraceCallStats>,
+    #[serde(rename = "storageStats")]
+    storage_stats: Option<StorageStats>,
+    #[serde(rename = "osStats")]
+    os_stats: Option<OSStats>,
+}
+
+#[derive(Debug, Default, Clone, Serialize, Deserialize)]
+pub struct StorageStats {
+    path: String,
+    duration: Duration,
+}
+
+#[derive(Debug, Default, Clone, Serialize, Deserialize)]
+pub struct OSStats {
+    path: String,
+    duration: Duration,
+}
+
+#[derive(Debug, Default, Clone, Serialize, Deserialize)]
+pub struct TraceHTTPStats {
+    req_info: TraceRequestInfo,
+    resp_info: TraceResponseInfo,
+    call_stats: TraceCallStats,
+}
+
+#[derive(Debug, Default, Clone, Serialize, Deserialize)]
+pub struct TraceCallStats {
+    input_bytes: i32,
+    output_bytes: i32,
+    latency: Duration,
+    time_to_first_byte: Duration,
+}
+
+#[derive(Debug, Default, Clone, Serialize, Deserialize)]
+pub struct TraceRequestInfo {
+    time: Timestamp,
+    proto: String,
+    method: String,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    path: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    raw_query: Option<String>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    headers: Option<HashMap<String, String>>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    body: Option<Vec<u8>>,
+    client: String,
+}
+
+#[derive(Debug, Default, Clone, Serialize, Deserialize)]
+pub struct TraceResponseInfo {
+    time: Timestamp,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    headers: Option<HashMap<String, String>>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    body: Option<Vec<u8>>,
+    #[serde(skip_serializing_if = "Option::is_none")]
+    status_code: Option<i32>,
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
 
     #[test]
-    fn trace_type_contains_and_overlaps() {
-        let mut combined = TraceType::default();
-        combined.merge(&TraceType::S3);
-        combined.merge(&TraceType::HEALING);
+    fn trace_timestamps_serialize_as_rfc3339_utc() {
+        let timestamp = Timestamp::constant(1_700_000_000, 123_456_000);
+        let trace = TraceInfo {
+            time: timestamp,
+            http: Some(TraceHTTPStats {
+                req_info: TraceRequestInfo {
+                    time: timestamp,
+                    ..Default::default()
+                },
+                resp_info: TraceResponseInfo {
+                    time: timestamp,
+                    ..Default::default()
+                },
+                ..Default::default()
+            }),
+            ..Default::default()
+        };
 
-        assert!(combined.contains(&TraceType::S3));
-        assert!(combined.contains(&TraceType::HEALING));
-        assert!(!combined.contains(&TraceType::SCANNER));
-        assert!(combined.overlaps(&TraceType::S3));
-        assert!(combined.overlaps(&TraceType::HEALING));
-        assert!(!combined.overlaps(&TraceType::SCANNER));
-    }
-
-    #[test]
-    fn trace_type_set_if() {
-        let mut tt = TraceType::default();
-        tt.set_if(true, &TraceType::OS);
-        tt.set_if(false, &TraceType::S3);
-        assert!(tt.contains(&TraceType::OS));
-        assert!(!tt.contains(&TraceType::S3));
-    }
-
-    #[test]
-    fn trace_type_single_type() {
-        assert!(TraceType::S3.single_type());
-        let mut combined = TraceType::S3;
-        combined.merge(&TraceType::HEALING);
-        assert!(!combined.single_type());
+        let value = serde_json::to_value(trace).expect("trace should serialize");
+        assert_eq!(value["time"], "2023-11-14T22:13:20.123456Z");
+        assert_eq!(value["http"]["req_info"]["time"], "2023-11-14T22:13:20.123456Z");
+        assert_eq!(value["http"]["resp_info"]["time"], "2023-11-14T22:13:20.123456Z");
+        let trace: TraceInfo = serde_json::from_value(value).expect("trace should deserialize");
+        assert_eq!(trace.time, timestamp);
+        let http = trace.http.expect("http trace should deserialize");
+        assert_eq!(http.req_info.time, timestamp);
+        assert_eq!(http.resp_info.time, timestamp);
     }
 }

crates/notify/AGENTS.md

@@ -55,3 +55,4 @@ shared plugin/runtime primitives from `rustfs-targets`.
   - Focused: `cargo test -p rustfs-notify runtime_facade`
   - Focused: `cargo test -p rustfs-notify runtime_view`
   - Focused: `cargo test -p rustfs-notify config_manager`
+- Full gate before commit: `make pre-commit`

crates/object-capacity/Cargo.toml

@@ -73,6 +73,7 @@ walkdir = { workspace = true }
 
 [dev-dependencies]
 criterion = { workspace = true, features = ["html_reports"] }
+serial_test = { workspace = true }
 temp-env = { workspace = true, features = ["async_closure"] }
 tempfile = { workspace = true }
 tokio = { workspace = true, features = ["test-util", "macros", "fs", "rt-multi-thread"] }

crates/object-capacity/src/capacity_manager.rs

@@ -1484,6 +1484,7 @@ mod tests {
         ENV_CAPACITY_SAMPLE_RATE, ENV_CAPACITY_STAT_TIMEOUT, ENV_CAPACITY_WRITE_FREQUENCY_THRESHOLD,
         ENV_CAPACITY_WRITE_TRIGGER_DELAY,
     };
+    use serial_test::serial;
     use std::sync::Arc;
     use std::sync::atomic::{AtomicUsize, Ordering};
 
@@ -1668,6 +1669,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn test_config_getter_defaults() {
         for (env_var, getter, default, _, _) in config_getter_cases() {
             temp_env::with_var(env_var, None::<&str>, || {
@@ -1677,6 +1679,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn test_config_getter_env_overrides() {
         for (env_var, getter, _, override_value, expected) in config_getter_cases() {
             temp_env::with_var(env_var, Some(override_value), || {
@@ -1686,6 +1689,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn test_zero_env_values_clamp_to_defaults() {
         // A zero threshold makes small disks report 0 bytes; a zero timeout
         // (with dynamic timeout off) makes every scan fail. Both must fall
@@ -1705,6 +1709,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_update_capacity_preserves_retrieval_metadata() {
         let manager = HybridCapacityManager::from_env();
 
@@ -1720,6 +1725,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_record_write_operation() {
         let manager = HybridCapacityManager::from_env();
 
@@ -1730,6 +1736,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_write_frequency_window() {
         let manager = HybridCapacityManager::from_env();
 
@@ -1817,6 +1824,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn test_recent_write_count_ignores_future_buckets() {
         let record = WriteRecord::new();
         record.write_buckets[0].store(120, 3);
@@ -1830,6 +1838,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_needs_fast_update() {
         let manager = HybridCapacityManager::from_env();
 
@@ -1846,6 +1855,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_cache_age_tracking() {
         let manager = HybridCapacityManager::from_env();
 
@@ -1865,6 +1875,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_data_source_tracking() {
         let manager = HybridCapacityManager::from_env();
 
@@ -1880,6 +1891,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_needs_fast_update_waits_for_write_trigger_delay() {
         let manager = create_isolated_manager(HybridStrategyConfig {
             scheduled_update_interval: Duration::from_secs(60),
@@ -1910,6 +1922,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_needs_fast_update_respects_enable_write_trigger() {
         let manager = create_isolated_manager(HybridStrategyConfig {
             scheduled_update_interval: Duration::from_secs(60),
@@ -1936,6 +1949,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_concurrent_access() {
         let manager = Arc::new(HybridCapacityManager::from_env());
         let mut handles = Vec::new();
@@ -1962,6 +1976,7 @@ mod tests {
     // exact under heavy same-second contention or the frequency window (and the
     // write-trigger decision) would undercount.
     #[tokio::test(flavor = "multi_thread", worker_threads = 8)]
+    #[serial]
     async fn test_record_write_operation_lock_free_is_exact_under_contention() {
         let manager = Arc::new(HybridCapacityManager::from_env());
         let mut handles = Vec::new();
@@ -1986,6 +2001,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_performance_overhead() {
         let manager = Arc::new(HybridCapacityManager::from_env());
         let start = Instant::now();
@@ -2002,6 +2018,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_refresh_or_join_singleflight() {
         let manager = Arc::new(HybridCapacityManager::from_env());
         let calls = Arc::new(AtomicUsize::new(0));
@@ -2041,6 +2058,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_refresh_or_join_recovers_after_leader_cancellation() {
         let manager = Arc::new(HybridCapacityManager::from_env());
 
@@ -2069,6 +2087,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_refresh_or_join_cancelled_leader_unblocks_joiner() {
         let manager = Arc::new(HybridCapacityManager::from_env());
 
@@ -2096,6 +2115,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_spawn_refresh_if_needed_deduplicates_background_refresh() {
         let manager = Arc::new(HybridCapacityManager::from_env());
         let calls = Arc::new(AtomicUsize::new(0));
@@ -2133,6 +2153,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_record_write_operation_with_scope_token_marks_dirty_disks() {
         let manager = create_isolated_manager(HybridStrategyConfig::default());
         let token = uuid::Uuid::new_v4();
@@ -2156,6 +2177,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_get_dirty_disks_drains_global_dirty_scope_registry() {
         let manager = create_isolated_manager(HybridStrategyConfig::default());
         record_global_dirty_scope(CapacityScope {
@@ -2175,6 +2197,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_update_capacity_recomputes_total_from_disk_cache_for_subset_refresh() {
         let manager = create_isolated_manager(HybridStrategyConfig::default());
 
@@ -2285,6 +2308,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_update_capacity_degraded_full_refresh_merges_cache_and_does_not_oscillate() {
         let manager = create_isolated_manager(HybridStrategyConfig::default());
 
@@ -2330,6 +2354,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_update_capacity_degraded_with_empty_per_disk_serves_merged_cache() {
         let manager = create_isolated_manager(HybridStrategyConfig::default());
         manager.update_capacity(full_two_disk_update(), DataSource::RealTime).await;
@@ -2359,6 +2384,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_update_capacity_degraded_without_complete_cache_keeps_partial_sum() {
         let manager = create_isolated_manager(HybridStrategyConfig::default());
 
@@ -2399,6 +2425,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_commit_keeps_dirty_marks_recorded_after_scan_start() {
         let manager = create_isolated_manager(HybridStrategyConfig::default());
         let disk = scope_disk("node-a", "/tmp/disk-a");
@@ -2429,6 +2456,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_commit_clears_dirty_marks_recorded_before_scan_start() {
         let manager = create_isolated_manager(HybridStrategyConfig::default());
         let disk = scope_disk("node-a", "/tmp/disk-a");
@@ -2449,6 +2477,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_retain_dirty_disks_within_drops_ghost_entries() {
         let manager = create_isolated_manager(HybridStrategyConfig::default());
         let local = scope_disk("node-a", "/tmp/disk-a");
@@ -2467,6 +2496,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_spawn_refresh_recovers_from_construction_panic() {
         let manager = create_isolated_manager(HybridStrategyConfig::default());
 
@@ -2533,6 +2563,7 @@ mod tests {
     }
 
     #[tokio::test(start_paused = true)]
+    #[serial]
     async fn test_refresh_or_join_joiner_times_out_when_leader_wedges() {
         let manager = create_isolated_manager(HybridStrategyConfig::default());
 
@@ -2560,6 +2591,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_refresh_or_join_returns_cluster_total_for_dirty_subset() {
         let manager = create_isolated_manager(HybridStrategyConfig::default());
 
@@ -2641,6 +2673,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_config_from_env() {
         let config = HybridStrategyConfig::from_env();
 
@@ -2654,6 +2687,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_config_from_env_with_override() {
         temp_env::with_var(ENV_CAPACITY_SCHEDULED_INTERVAL, Some("600"), || {
             let config = HybridStrategyConfig::from_env();

crates/object-capacity/src/scan.rs

@@ -1069,6 +1069,7 @@ mod tests {
     #[cfg(unix)]
     use rustfs_config::ENV_CAPACITY_FOLLOW_SYMLINKS;
     use rustfs_config::{ENV_CAPACITY_MAX_FILES_THRESHOLD, ENV_CAPACITY_SAMPLE_RATE};
+    use serial_test::serial;
 
     /// Reference implementation using unbounded `u128` arithmetic, clamped to
     /// `u64::MAX`, used as the source of truth for the sampling extrapolation.
@@ -1273,6 +1274,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_get_dir_size_async_nonexistent_directory() {
         let result = get_dir_size_async(Path::new("/nonexistent/path")).await;
         assert!(result.is_err());
@@ -1646,6 +1648,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_metadata_incomplete_aggregate_does_not_replace_disk_cache() {
         use std::fs::File;
         use std::io::Write;
@@ -1780,6 +1783,7 @@ mod tests {
 
     #[cfg(unix)]
     #[tokio::test]
+    #[serial]
     async fn test_get_dir_size_async_ignores_symlink_targets_when_follow_disabled() {
         use std::fs::File;
         use std::io::Write;
@@ -1805,6 +1809,7 @@ mod tests {
 
     #[cfg(unix)]
     #[tokio::test]
+    #[serial]
     async fn test_get_dir_size_async_counts_symlink_targets_when_follow_enabled() {
         use std::fs::File;
         use std::io::Write;

crates/obs/src/metrics/collectors/bucket_replication.rs

@@ -75,7 +75,7 @@ pub struct BucketReplicationBandwidthStats {
 }
 
 #[derive(Debug, Clone, Default)]
-pub struct BucketReplicationMetricsSnapshot {
+pub struct BucketReplicationStats {
     pub bucket: String,
     pub total_failed_bytes: u64,
     pub total_failed_count: u64,
@@ -107,7 +107,7 @@ pub struct BucketReplicationMetricsSnapshot {
 
 #[derive(Debug, Clone, Default)]
 pub(crate) struct BucketReplicationRuntimeStats {
-    pub(crate) stats: BucketReplicationMetricsSnapshot,
+    pub(crate) stats: BucketReplicationStats,
     pub(crate) target_flows: Vec<BucketReplicationTargetFlowStats>,
 }
 
@@ -182,7 +182,7 @@ fn push_proxy_request_result_metrics(
     }
 }
 
-pub fn collect_bucket_replication_metrics(stats: &[BucketReplicationMetricsSnapshot]) -> Vec<PrometheusMetric> {
+pub fn collect_bucket_replication_metrics(stats: &[BucketReplicationStats]) -> Vec<PrometheusMetric> {
     if stats.is_empty() {
         return Vec::new();
     }
@@ -572,7 +572,7 @@ mod tests {
     #[test]
     fn test_collect_bucket_replication_metrics() {
         let stats = vec![BucketReplicationRuntimeStats {
-            stats: BucketReplicationMetricsSnapshot {
+            stats: BucketReplicationStats {
                 bucket: "b1".to_string(),
                 total_failed_bytes: 64,
                 total_failed_count: 2,
@@ -876,7 +876,7 @@ mod tests {
 
     #[test]
     fn test_collect_bucket_replication_metrics_empty() {
-        let stats: Vec<BucketReplicationMetricsSnapshot> = Vec::new();
+        let stats: Vec<BucketReplicationStats> = Vec::new();
         let metrics = collect_bucket_replication_metrics(&stats);
         assert!(metrics.is_empty());
     }

crates/obs/src/metrics/collectors/mod.rs

@@ -48,7 +48,7 @@ pub(crate) use bucket_replication::{
     BucketReplicationTargetFlowStats, collect_bucket_replication_backlog_metrics, collect_bucket_replication_runtime_metrics,
 };
 pub use bucket_replication::{
-    BucketReplicationBandwidthStats, BucketReplicationMetricsSnapshot, BucketReplicationTargetStats,
+    BucketReplicationBandwidthStats, BucketReplicationStats, BucketReplicationTargetStats,
     collect_bucket_replication_bandwidth_metrics, collect_bucket_replication_metrics,
 };
 pub use cluster::{ClusterStats, collect_cluster_metrics};
@@ -68,8 +68,8 @@ pub(crate) use notification::collect_notification_runtime_metrics;
 pub use notification::{NotificationStats, collect_notification_metrics};
 pub(crate) use notification_target::{NotificationTargetRuntimeStats, collect_notification_target_runtime_metrics};
 pub use notification_target::{NotificationTargetStats, collect_notification_target_metrics};
-pub use replication::{ReplicationMetricsSnapshot, collect_replication_metrics};
 pub(crate) use replication::{ReplicationRuntimeStats, collect_replication_runtime_metrics};
+pub use replication::{ReplicationStats, collect_replication_metrics};
 pub(crate) use request::{ApiRequestMetricSupport, ApiRequestStats, collect_request_metrics};
 pub use resource::{ResourceStats, collect_resource_metrics};
 pub(crate) use scanner::{ScannerRuntimeStats, collect_scanner_runtime_metrics};

crates/obs/src/metrics/collectors/replication.rs

@@ -22,7 +22,7 @@ use crate::metrics::schema::replication::*;
 
 /// Replication statistics.
 #[derive(Debug, Clone, Default)]
-pub struct ReplicationMetricsSnapshot {
+pub struct ReplicationStats {
     /// Average number of active replication workers
     pub average_active_workers: f64,
     /// Average queued bytes since server start
@@ -54,13 +54,13 @@ pub struct ReplicationMetricsSnapshot {
 #[derive(Debug, Clone, Default)]
 pub(crate) struct ReplicationRuntimeStats {
     pub(crate) server: String,
-    pub(crate) stats: ReplicationMetricsSnapshot,
+    pub(crate) stats: ReplicationStats,
 }
 
 /// Collects replication metrics from the given stats.
 ///
 /// Returns a vector of Prometheus metrics for replication statistics.
-pub fn collect_replication_metrics(stats: &ReplicationMetricsSnapshot) -> Vec<PrometheusMetric> {
+pub fn collect_replication_metrics(stats: &ReplicationStats) -> Vec<PrometheusMetric> {
     vec![
         PrometheusMetric::from_descriptor(&REPLICATION_AVERAGE_ACTIVE_WORKERS_MD, stats.average_active_workers),
         PrometheusMetric::from_descriptor(&REPLICATION_AVERAGE_QUEUED_BYTES_MD, stats.average_queued_bytes as f64),
@@ -120,7 +120,7 @@ mod tests {
 
     #[test]
     fn test_collect_replication_metrics() {
-        let stats = ReplicationMetricsSnapshot {
+        let stats = ReplicationStats {
             average_active_workers: 8.5,
             average_queued_bytes: 1024 * 1024 * 40,
             average_queued_count: 240,
@@ -182,7 +182,7 @@ mod tests {
 
     #[test]
     fn test_collect_replication_metrics_default() {
-        let stats = ReplicationMetricsSnapshot::default();
+        let stats = ReplicationStats::default();
         let metrics = collect_replication_metrics(&stats);
 
         assert_eq!(metrics.len(), 13);
@@ -194,7 +194,7 @@ mod tests {
 
     #[test]
     fn replication_stats_struct_literal_keeps_legacy_fields() {
-        let stats = ReplicationMetricsSnapshot {
+        let stats = ReplicationStats {
             average_active_workers: 1.0,
             average_queued_bytes: 2,
             average_queued_count: 3,

crates/obs/src/metrics/scheduler.rs

@@ -2811,14 +2811,14 @@ mod tests {
     #[test]
     fn replication_proxy_bucket_keys_detect_removed_buckets() {
         let previous = repl_proxy_bucket_live_keys(&[BucketReplicationRuntimeStats {
-            stats: crate::metrics::BucketReplicationMetricsSnapshot {
+            stats: crate::metrics::BucketReplicationStats {
                 bucket: "photos".to_string(),
                 ..Default::default()
             },
             ..Default::default()
         }]);
         let current = repl_proxy_bucket_live_keys(&[BucketReplicationRuntimeStats {
-            stats: crate::metrics::BucketReplicationMetricsSnapshot {
+            stats: crate::metrics::BucketReplicationStats {
                 bucket: "logs".to_string(),
                 ..Default::default()
             },

crates/obs/src/metrics/stats_collector.rs

@@ -21,12 +21,12 @@
 use crate::metrics::collectors::scanner::{ScannerActiveBucketDriveStats, ScannerBucketDriveResultStats, ScannerSourceWorkStats};
 use crate::metrics::collectors::{
     ApiRequestMetricSupport, ApiRequestStats, BucketReplicationBacklogStats, BucketReplicationBandwidthStats,
-    BucketReplicationMetricsSnapshot, BucketReplicationRuntimeStats, BucketReplicationTargetBacklogStats,
-    BucketReplicationTargetFlowStats, BucketReplicationTargetStats, BucketStats, BucketUsageStats, ClusterConfigStats,
-    ClusterHealthStats, ClusterStats, ClusterUsageStats, CompressionClusterStats, CpuStats, DiskStats, DriveCountStats,
-    DriveDetailedStats, DriveRuntimeDetailedStats, ErasureSetStats, HostNetworkStats, IamStats, IlmActionTaskStats,
-    IlmBackpressureStats, IlmQueueTaskStats, IlmRuntimeStats, IlmStats, IlmTaskEventStats, MemoryStats, NetworkStats,
-    ProcessStats, ProcessStatusType, ReplicationMetricsSnapshot, ResourceStats, ScannerRuntimeStats, ScannerStats,
+    BucketReplicationRuntimeStats, BucketReplicationStats, BucketReplicationTargetBacklogStats, BucketReplicationTargetFlowStats,
+    BucketReplicationTargetStats, BucketStats, BucketUsageStats, ClusterConfigStats, ClusterHealthStats, ClusterStats,
+    ClusterUsageStats, CompressionClusterStats, CpuStats, DiskStats, DriveCountStats, DriveDetailedStats,
+    DriveRuntimeDetailedStats, ErasureSetStats, HostNetworkStats, IamStats, IlmActionTaskStats, IlmBackpressureStats,
+    IlmQueueTaskStats, IlmRuntimeStats, IlmStats, IlmTaskEventStats, MemoryStats, NetworkStats, ProcessStats, ProcessStatusType,
+    ReplicationStats, ResourceStats, ScannerRuntimeStats, ScannerStats,
 };
 use crate::metrics::runtime_sources::{ObsIlmRuntimeSnapshot, bucket_monitor_handle, iam_metrics_snapshot, ilm_runtime_snapshot};
 use crate::metrics::{
@@ -266,7 +266,7 @@ fn bucket_replication_detail_from_snapshot(stats: ObsBucketReplicationStatsSnaps
 
     BucketReplicationRuntimeStats {
         target_flows,
-        stats: BucketReplicationMetricsSnapshot {
+        stats: BucketReplicationStats {
             bucket,
             total_failed_bytes: stats.total_failed_bytes,
             total_failed_count: stats.total_failed_count,
@@ -298,7 +298,7 @@ fn bucket_replication_detail_from_snapshot(stats: ObsBucketReplicationStatsSnaps
     }
 }
 
-async fn obs_site_replication_stats() -> ReplicationMetricsSnapshot {
+async fn obs_site_replication_stats() -> ReplicationStats {
     let current_data_transfer_rate = obs_bucket_replication_bandwidth_stats()
         .into_iter()
         .flatten()
@@ -306,7 +306,7 @@ async fn obs_site_replication_stats() -> ReplicationMetricsSnapshot {
         .sum::<f64>();
     let stats = obs_replication_site_stats_snapshot(current_data_transfer_rate).await;
 
-    ReplicationMetricsSnapshot {
+    ReplicationStats {
         average_active_workers: stats.average_active_workers,
         average_queued_bytes: stats.average_queued_bytes,
         average_queued_count: stats.average_queued_count,
@@ -648,7 +648,7 @@ pub fn collect_bucket_replication_bandwidth_stats() -> Vec<BucketReplicationBand
 }
 
 /// Collect bucket and target level replication stats from the global replication runtime.
-pub async fn collect_bucket_replication_detail_stats() -> Vec<BucketReplicationMetricsSnapshot> {
+pub async fn collect_bucket_replication_detail_stats() -> Vec<BucketReplicationStats> {
     obs_bucket_replication_stats_snapshot()
         .await
         .into_iter()
@@ -662,7 +662,7 @@ pub(crate) async fn collect_bucket_replication_stats_bundle()
 }
 
 /// Collect site-level replication stats from the global replication runtime.
-pub async fn collect_replication_stats() -> ReplicationMetricsSnapshot {
+pub async fn collect_replication_stats() -> ReplicationStats {
     obs_site_replication_stats().await
 }
 

crates/policy/AGENTS.md

@@ -23,3 +23,4 @@ Applies to `crates/policy/`.
 ## Suggested Validation
 
 - `cargo test -p rustfs-policy`
+- Full gate before commit: `make pre-commit`

crates/scanner/Cargo.toml

@@ -103,7 +103,8 @@ hex-simd.workspace = true
 
 [dev-dependencies]
 tracing-subscriber = { workspace = true, features = ["json", "env-filter", "time"] }
-temp-env = { workspace = true, features = ["async_closure"] }
+serial_test = { workspace = true }
+temp-env = { workspace = true }
 tempfile = { workspace = true }
 uuid = { workspace = true, features = ["v4", "serde", "fast-rng", "macro-diagnostics"] }
 tokio = { workspace = true, features = ["test-util", "fs", "rt-multi-thread"] }

crates/scanner/src/data_usage_define.rs

@@ -146,6 +146,11 @@ pub static LEGACY_DATA_USAGE_OBJ_NAME_PATH: LazyLock<String> =
 pub static DATA_USAGE_BLOOM_NAME_PATH: LazyLock<String> =
     LazyLock::new(|| format!("{BUCKET_META_PREFIX}{SLASH_SEPARATOR}{DATA_USAGE_BLOOM_NAME}"));
 
+/// Durable companion object for a cycle-state object which cannot be decoded.
+/// The primary object is deliberately never replaced or deleted by recovery.
+pub static DATA_USAGE_BLOOM_RECOVERY_PATH: LazyLock<String> =
+    LazyLock::new(|| format!("{}.recovery-required.json", DATA_USAGE_BLOOM_NAME_PATH.as_str()));
+
 pub static BACKGROUND_HEAL_INFO_PATH: LazyLock<String> =
     LazyLock::new(|| format!("{BUCKET_META_PREFIX}{SLASH_SEPARATOR}.background-heal.json"));
 

crates/scanner/src/lib.rs

@@ -75,7 +75,10 @@ pub use remote_scanner::{
 };
 pub use runtime_config::{apply_scanner_runtime_config, scanner_runtime_config_status, validate_scanner_runtime_config};
 pub use rustfs_common::last_minute;
-pub use scanner::{ScannerCycleScheduleStatus, init_data_scanner, scanner_cycle_schedule_status, scanner_topology_digest};
+pub use scanner::{
+    ScannerCycleRecoveryMarker, ScannerCycleRecoveryStatus, ScannerCycleScheduleStatus, init_data_scanner,
+    reset_scanner_cycle_recovery, scanner_cycle_recovery_status, scanner_cycle_schedule_status, scanner_topology_digest,
+};
 pub use scanner_io::{
     ScannerDirtyUsageAckError, ScannerDirtyUsageState, acknowledge_dirty_usage_generation, clear_dirty_usage_bucket,
     record_dirty_usage_bucket, record_scanner_maintenance_change, scanner_activity_epoch, scanner_dirty_usage_state,
@@ -599,8 +602,10 @@ impl ScannerConfigObjectDelete for ECStore {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use serial_test::serial;
 
     #[tokio::test]
+    #[serial]
     async fn runtime_tier_names_serves_cached_arc_within_ttl() {
         reset_tier_name_cache_for_test();
         // The tier config manager is unconfigured in unit tests, so the
@@ -614,6 +619,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn foreground_read_guard_tracks_stream_lifetime() {
         reset_foreground_read_activity_for_test();
         assert_eq!(current_foreground_read_activity(), 0);
@@ -627,6 +633,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn foreground_read_activity_keeps_larger_signal() {
         reset_foreground_read_activity_for_test();
         let _guard = ForegroundReadGuard::new();
@@ -639,6 +646,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn scanner_runtime_guard_tracks_runtime_lifetime() {
         reset_scanner_runtime_instances_for_test();
         assert!(!scanner_runtime_initialized());

crates/scanner/src/runtime_config.rs

@@ -868,6 +868,7 @@ mod tests {
         SCANNER_CYCLE_MAX_DIRECTORIES, SCANNER_CYCLE_MAX_DURATION, SCANNER_CYCLE_MAX_OBJECTS, SCANNER_DELAY, SCANNER_IDLE_MODE,
         SCANNER_SPEED, SCANNER_SUB_SYS, ScannerSpeed,
     };
+    use serial_test::serial;
     use std::collections::HashMap;
     use std::time::Duration;
     use temp_env::{with_var, with_var_unset};
@@ -915,6 +916,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn scanner_runtime_config_uses_persisted_values_when_env_is_unset() {
         let config = server_config_with_scanner(&[
             (SCANNER_SPEED, "slow"),
@@ -942,6 +944,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn scanner_runtime_config_normalizes_persisted_default_speed() {
         let config = server_config_with_scanner(&[(SCANNER_SPEED, "default")]);
 
@@ -957,6 +960,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn scanner_runtime_config_prefers_env_over_persisted_config() {
         let config = server_config_with_scanner(&[(SCANNER_SPEED, "slowest"), (SCANNER_CYCLE, "600")]);
 
@@ -973,6 +977,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn scanner_runtime_config_prefers_heal_bitrot_cycle_over_scanner_compat_config() {
         let config = server_config_with_scanner_and_heal(&[(SCANNER_BITROT_CYCLE, "3600")], &[(HEAL_BITROT_CYCLE, "off")]);
 
@@ -985,6 +990,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn scanner_runtime_config_marks_scanner_bitrot_cycle_as_compat_source() {
         let config = server_config_with_scanner(&[(SCANNER_BITROT_CYCLE, "3600")]);
 
@@ -1001,6 +1007,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn scanner_runtime_config_normalizes_persisted_default_bitrot_cycles() {
         let default_cycle = DEFAULT_HEAL_BITROT_CYCLE_SECS.to_string();
         for config in [
@@ -1025,6 +1032,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn scanner_runtime_config_validation_rejects_invalid_persisted_speed_with_env_override() {
         let config = server_config_with_scanner(&[(SCANNER_SPEED, "warp")]);
 
@@ -1058,6 +1066,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn scanner_runtime_config_uses_derived_delay_for_excessive_env_override() {
         let config = server_config_with_scanner(&[(SCANNER_SPEED, "slow")]);
 
@@ -1078,6 +1087,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn scanner_runtime_config_status_reports_value_sources() {
         let config = server_config_with_scanner(&[(SCANNER_CYCLE_MAX_OBJECTS, "100"), (SCANNER_CACHE_SAVE_TIMEOUT, "5")]);
 
@@ -1098,6 +1108,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn applied_runtime_config_is_the_authoritative_scheduler_state() {
         let config = server_config_with_scanner(&[(SCANNER_CYCLE, "321")]);
 
@@ -1114,6 +1125,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn scanner_runtime_config_status_reports_persisted_pacing_overrides() {
         let config = server_config_with_scanner(&[("delay", "3.5"), ("max_wait", "7")]);
 
@@ -1135,6 +1147,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn scanner_runtime_config_status_prefers_env_pacing_overrides() {
         let config = server_config_with_scanner(&[("delay", "3.5"), ("max_wait", "7")]);
 
@@ -1156,6 +1169,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn scanner_runtime_config_status_preserves_subsecond_max_wait() {
         let config = server_config_with_scanner(&[(SCANNER_SPEED, "fast")]);
 

crates/scanner/src/scanner.rs

@@ -54,9 +54,7 @@ use rustfs_config::{ENV_SCANNER_CYCLE, ENV_SCANNER_SPEED, ENV_SCANNER_START_DELA
 use rustfs_data_usage::observed_data_usage_is_newer;
 use serde::{Deserialize, Serialize};
 use sha2::{Digest as _, Sha256};
-#[cfg(test)]
-use tokio::sync::Notify;
-use tokio::sync::mpsc;
+use tokio::sync::{Notify, mpsc};
 use tokio::time::{Duration, Instant};
 use tokio_util::sync::CancellationToken;
 use tokio_util::task::AbortOnDropHandle;
@@ -104,6 +102,13 @@ const CLEAN_IDLE_BACKOFF_FACTOR: u32 = 2;
 /// unavailable peer cannot drive a tight retry loop.
 const SCANNER_RETRY_BASE_INTERVAL: Duration = Duration::from_secs(5);
 const SCANNER_RETRY_MAX_INTERVAL: Duration = Duration::from_secs(30 * 60);
+/// A transient backend outage remains self-healing after the short retry
+/// budget is exhausted, but the probe is intentionally sparse until storage
+/// recovers or an operator reset wakes the scanner.
+const SCANNER_CYCLE_RECOVERY_PAUSED_INTERVAL: Duration = Duration::from_secs(5 * 60);
+/// Permanent recovery states still get a sparse status probe so a reset that
+/// races the wait registration cannot leave the scanner asleep forever.
+const SCANNER_CYCLE_RECOVERY_BLOCKED_PROBE_INTERVAL: Duration = Duration::from_secs(5 * 60);
 const SCANNER_LEADER_LOCK_POLL_INTERVAL: Duration = Duration::from_secs(1);
 #[cfg(not(test))]
 const SCANNER_LOCK_LOSS_SHUTDOWN_TIMEOUT: Duration = Duration::from_secs(30);
@@ -125,6 +130,12 @@ type ScannerCycleStatePersistTestHook = (u64, Arc<Notify>);
 static SCANNER_CYCLE_STATE_PERSIST_TEST_HOOK: LazyLock<StdMutex<Option<ScannerCycleStatePersistTestHook>>> =
     LazyLock::new(|| StdMutex::new(None));
 
+static SCANNER_CYCLE_RECOVERY_WAKE: LazyLock<Notify> = LazyLock::new(Notify::new);
+
+pub(super) fn notify_scanner_cycle_recovery_wake() {
+    SCANNER_CYCLE_RECOVERY_WAKE.notify_waiters();
+}
+
 #[cfg(test)]
 struct ScannerCycleStatePersistTestHookGuard;
 
@@ -576,19 +587,21 @@ pub async fn init_data_scanner(ctx: CancellationToken, storeapi: Arc<ECStore>) {
             tokio::time::sleep(sleep_time).await;
         }
 
+        let mut transient_backoff = ScannerRetryBackoff::default();
+        let mut recovery_retry_count = 0_u32;
         loop {
             if ctx_clone.is_cancelled() {
                 break;
             }
 
-            if let Err(e) = run_data_scanner_with_maintenance_state(
+            let run_result = run_data_scanner_with_maintenance_state(
                 ctx_clone.clone(),
                 storeapi_clone.clone(),
                 startup_features,
                 startup_maintenance_generation,
             )
-            .await
-            {
+            .await;
+            if let Err(e) = &run_result {
                 error!(
                     target: "rustfs::scanner",
                     event = EVENT_SCANNER_CYCLE_STATE,
@@ -599,11 +612,52 @@ pub async fn init_data_scanner(ctx: CancellationToken, storeapi: Arc<ECStore>) {
                     "Scanner runtime iteration failed"
                 );
             }
+            let recovery_status = scanner_cycle_recovery_status();
+            if recovery_status.retryable {
+                recovery_retry_count = recovery_retry_count.saturating_add(1);
+                let _ = record_scanner_cycle_recovery_retry(recovery_retry_count);
+            } else {
+                recovery_retry_count = 0;
+            }
+
+            let recovery_status = scanner_cycle_recovery_status();
+            if recovery_status.state == "paused" {
+                transient_backoff.record_retryable_cycle(false);
+                tokio::select! {
+                    _ = ctx_clone.cancelled() => break,
+                    _ = SCANNER_CYCLE_RECOVERY_WAKE.notified() => {},
+                    _ = tokio::time::sleep(SCANNER_CYCLE_RECOVERY_PAUSED_INTERVAL) => {},
+                }
+                recovery_retry_count = 0;
+                continue;
+            }
+            if !recovery_status.retryable
+                && matches!(recovery_status.state.as_str(), "blocked" | "recovery-required" | "cleanup-pending")
+            {
+                transient_backoff.record_retryable_cycle(false);
+                tokio::select! {
+                    _ = ctx_clone.cancelled() => break,
+                    _ = SCANNER_CYCLE_RECOVERY_WAKE.notified() => {},
+                    _ = tokio::time::sleep(SCANNER_CYCLE_RECOVERY_BLOCKED_PROBE_INTERVAL) => {},
+                }
+                continue;
+            }
+
+            let retry_delay = if recovery_status.retryable || run_result.is_err() {
+                transient_backoff.record_retryable_cycle(true);
+                transient_backoff
+                    .retry_interval(scanner_cycle_interval())
+                    .unwrap_or(SCANNER_RETRY_BASE_INTERVAL)
+            } else {
+                transient_backoff.record_retryable_cycle(false);
+                randomized_cycle_delay()
+            };
             // Backoff before retrying after lock contention or scanner-level failures.
             // Keep this cancellation-aware so shutdown is not delayed by backoff sleep.
             tokio::select! {
                 _ = ctx_clone.cancelled() => break,
-                _ = tokio::time::sleep(randomized_cycle_delay()) => {}
+                _ = SCANNER_CYCLE_RECOVERY_WAKE.notified() => {},
+                _ = tokio::time::sleep(retry_delay) => {}
             }
         }
     });
@@ -1606,40 +1660,22 @@ async fn run_data_scanner_with_maintenance_state(
         observe_scanner_activity(&storeapi, distributed, &mut scanner_activity_seen).await;
     }
 
-    let (buf, mut cycle_revision) = match read_config_with_revision(storeapi.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str()).await {
-        Ok((buf, revision)) => (buf.unwrap_or_default(), revision),
-        Err(err) => {
-            error!(
-                target: "rustfs::scanner",
-                event = EVENT_SCANNER_PERSIST_STATE,
-                component = LOG_COMPONENT_SCANNER,
-                subsystem = LOG_SUBSYSTEM_RUNTIME,
-                path = %&*DATA_USAGE_BLOOM_NAME_PATH,
-                state = "revision_load_failed",
-                error = %err,
-                "Scanner cycle state revision load failed"
-            );
-            global_metrics().set_cycle(None).await;
-            return Ok(());
-        }
-    };
-    let (mut cycle_info, mut leader_epoch) = match decode_scanner_cycle_state_for_startup(&buf) {
-        Ok(state) => state,
-        Err(err) => {
-            error!(
-                target: "rustfs::scanner",
-                event = EVENT_SCANNER_PERSIST_STATE,
-                component = LOG_COMPONENT_SCANNER,
-                subsystem = LOG_SUBSYSTEM_RUNTIME,
-                path = %&*DATA_USAGE_BLOOM_NAME_PATH,
-                state = "cycle_decode_failed",
-                error = %err,
-                "Scanner stopped because persisted cycle state is invalid"
-            );
-            global_metrics().set_cycle(None).await;
-            return Ok(());
-        }
-    };
+    let (mut cycle_info, mut leader_epoch, mut cycle_revision) =
+        match load_scanner_cycle_state_for_startup(storeapi.clone()).await {
+            ScannerCycleStateStartup::Ready {
+                cycle,
+                leader_epoch,
+                revision,
+            } => (cycle, leader_epoch, revision),
+            ScannerCycleStateStartup::Blocked => {
+                global_metrics().set_cycle(None).await;
+                return Ok(());
+            }
+            ScannerCycleStateStartup::Transient(err) => {
+                global_metrics().set_cycle(None).await;
+                return Err(err);
+            }
+        };
     let usage_floor = match persisted_usage_floor(storeapi.clone()).await {
         Ok(floor) => floor,
         Err(err) => {
@@ -2219,7 +2255,12 @@ pub(crate) use activity::{
 pub(crate) use activity::{ScannerCycleOutcome, scanner_cycle_outcome_with_pending_maintenance};
 #[cfg(test)]
 pub(crate) use cycle_state::encode_scanner_cycle_fence_for_test;
-pub(crate) use cycle_state::{current_scanner_leader_epoch, decode_persisted_scanner_cycle_fence};
+pub use cycle_state::{
+    ScannerCycleRecoveryMarker, ScannerCycleRecoveryStatus, reset_scanner_cycle_recovery, scanner_cycle_recovery_status,
+};
+pub(crate) use cycle_state::{
+    current_scanner_leader_epoch, decode_persisted_scanner_cycle_fence, load_scanner_cycle_state_for_startup,
+};
 pub use heal_info::{BackgroundHealInfo, read_background_heal_info, save_background_heal_info};
 pub use usage_store::store_data_usage_in_backend;
 

crates/scanner/src/scanner/cycle_state.rs

@@ -13,6 +13,952 @@
 // limitations under the License.
 /// Scanner cycle-state codec, persisted usage floors, and cycle-state persistence.
 use super::*;
+use crate::ScannerGetObjectReader;
+use crate::data_usage_define::DATA_USAGE_BLOOM_RECOVERY_PATH;
+use crate::storage_api::owner::ObjectIO as _;
+use tokio::io::AsyncReadExt as _;
+
+const SCANNER_CYCLE_RECOVERY_SCHEMA_VERSION: u16 = 1;
+const MAX_SCANNER_CYCLE_STATE_BYTES: u64 = 1024 * 1024;
+pub(super) const MAX_SCANNER_CYCLE_RECOVERY_RETRIES: u32 = 5;
+const METRIC_SCANNER_CYCLE_RECOVERY_REQUIRED: &str = "rustfs_scanner_cycle_recovery_required";
+const METRIC_SCANNER_CYCLE_RECOVERY_RETRY_COUNT: &str = "rustfs_scanner_cycle_recovery_retry_count";
+
+#[derive(Clone, Debug, Default, Serialize)]
+pub struct ScannerCycleRecoveryStatus {
+    /// The immutable primary object whose revision is being guarded.
+    pub path: String,
+    /// The companion marker/quarantine object containing the recovery evidence.
+    pub quarantine_path: Option<String>,
+    pub state: String,
+    pub classification: Option<String>,
+    pub primary_revision: Option<String>,
+    pub generation: Option<u64>,
+    pub leader_epoch: Option<u64>,
+    pub first_detected_at_unix_secs: Option<u64>,
+    pub last_attempt_at_unix_secs: Option<u64>,
+    pub retry_count: u64,
+    pub max_retries: u32,
+    /// Whether the scanner may retry this state automatically.
+    pub retryable: bool,
+    pub reason: Option<String>,
+}
+
+static SCANNER_CYCLE_RECOVERY_STATUS: LazyLock<RwLock<ScannerCycleRecoveryStatus>> = LazyLock::new(|| {
+    RwLock::new(ScannerCycleRecoveryStatus {
+        path: DATA_USAGE_BLOOM_NAME_PATH.clone(),
+        quarantine_path: Some(DATA_USAGE_BLOOM_RECOVERY_PATH.clone()),
+        state: "healthy".to_string(),
+        max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES,
+        ..Default::default()
+    })
+});
+
+pub fn scanner_cycle_recovery_status() -> ScannerCycleRecoveryStatus {
+    SCANNER_CYCLE_RECOVERY_STATUS
+        .read()
+        .unwrap_or_else(|poisoned| poisoned.into_inner())
+        .clone()
+}
+
+fn set_scanner_cycle_recovery_status(status: ScannerCycleRecoveryStatus) {
+    let recovery_required = if matches!(status.state.as_str(), "blocked" | "paused" | "recovery-required" | "cleanup-pending") {
+        1.0
+    } else {
+        0.0
+    };
+    metrics::gauge!(METRIC_SCANNER_CYCLE_RECOVERY_REQUIRED).set(recovery_required);
+    metrics::gauge!(METRIC_SCANNER_CYCLE_RECOVERY_RETRY_COUNT).set(status.retry_count as f64);
+    *SCANNER_CYCLE_RECOVERY_STATUS
+        .write()
+        .unwrap_or_else(|poisoned| poisoned.into_inner()) = status;
+}
+
+pub(super) fn record_scanner_cycle_recovery_retry(attempt: u32) -> bool {
+    let mut status = scanner_cycle_recovery_status();
+    status.retry_count = u64::from(attempt);
+    status.last_attempt_at_unix_secs = Some(unix_now_secs());
+    if attempt >= MAX_SCANNER_CYCLE_RECOVERY_RETRIES {
+        status.state = "paused".to_string();
+        status.retryable = false;
+        status.reason = Some("scanner cycle recovery retry budget reached; sparse backend probes continue".to_string());
+        set_scanner_cycle_recovery_status(status);
+        false
+    } else {
+        status.retryable = true;
+        set_scanner_cycle_recovery_status(status);
+        true
+    }
+}
+
+fn unix_now_secs() -> u64 {
+    u64::try_from(Utc::now().timestamp()).unwrap_or(0)
+}
+
+fn recovery_status(state: &str, reason: Option<&str>, retryable: bool) -> ScannerCycleRecoveryStatus {
+    ScannerCycleRecoveryStatus {
+        path: DATA_USAGE_BLOOM_NAME_PATH.clone(),
+        quarantine_path: Some(DATA_USAGE_BLOOM_RECOVERY_PATH.clone()),
+        state: state.to_string(),
+        max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES,
+        retryable,
+        last_attempt_at_unix_secs: Some(unix_now_secs()),
+        reason: reason.map(str::to_string),
+        ..Default::default()
+    }
+}
+
+#[derive(Clone, Debug, Serialize, Deserialize)]
+#[serde(deny_unknown_fields)]
+pub struct ScannerCycleRecoveryMarker {
+    pub schema_version: u16,
+    pub primary_revision: String,
+    pub generation: u64,
+    pub leader_epoch: u64,
+    pub classification: String,
+    pub first_detected_at_unix_secs: u64,
+    pub last_attempt_at_unix_secs: u64,
+    pub retry_count: u64,
+    pub reason: String,
+    pub path: String,
+    pub quarantine_path: String,
+    /// `blocked` means the marker guards the primary revision; `cleanup-pending`
+    /// means an operator reset is in progress and must remain fenced across a
+    /// restart, even if the primary object is subsequently rewritten.
+    #[serde(default = "default_recovery_marker_state")]
+    pub state: String,
+}
+
+fn default_recovery_marker_state() -> String {
+    "blocked".to_string()
+}
+
+#[derive(Debug, Deserialize)]
+struct ScannerCycleRecoveryMarkerCompat {
+    schema_version: Option<u16>,
+    primary_revision: Option<String>,
+    classification: Option<String>,
+    first_detected_at_unix_secs: Option<u64>,
+    last_attempt_at_unix_secs: Option<u64>,
+    retry_count: Option<u64>,
+    reason: Option<String>,
+    path: Option<String>,
+    quarantine_path: Option<String>,
+    state: Option<String>,
+}
+
+#[derive(Debug)]
+pub(crate) enum ScannerCycleStateStartup {
+    Ready {
+        cycle: CurrentCycle,
+        leader_epoch: u64,
+        revision: DataUsageCacheRevision,
+    },
+    Blocked,
+    Transient(ScannerError),
+}
+
+#[derive(Debug, thiserror::Error)]
+enum CycleRecoveryMarkerReadError {
+    #[error("cycle recovery marker backend read failed: {0}")]
+    Backend(#[source] EcstoreError),
+    #[error("invalid cycle recovery marker: {0}")]
+    Invalid(&'static str),
+    #[error("cycle recovery marker revision changed while publishing")]
+    Conflict,
+}
+
+#[derive(Debug, thiserror::Error)]
+enum CycleStateBodyReadError {
+    #[error("scanner cycle state exceeds the bounded object size")]
+    TooLarge,
+    #[error("scanner cycle state body read failed: {0}")]
+    Backend(#[source] EcstoreError),
+}
+
+fn recovery_status_from_marker(marker: &ScannerCycleRecoveryMarker, state: &str) -> ScannerCycleRecoveryStatus {
+    ScannerCycleRecoveryStatus {
+        path: marker.path.clone(),
+        quarantine_path: Some(marker.quarantine_path.clone()),
+        state: state.to_string(),
+        classification: Some(marker.classification.clone()),
+        primary_revision: Some(marker.primary_revision.clone()),
+        generation: Some(marker.generation),
+        leader_epoch: Some(marker.leader_epoch),
+        first_detected_at_unix_secs: Some(marker.first_detected_at_unix_secs),
+        last_attempt_at_unix_secs: Some(marker.last_attempt_at_unix_secs),
+        retry_count: marker.retry_count,
+        max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES,
+        retryable: false,
+        reason: Some(marker.reason.clone()),
+    }
+}
+
+fn marker_matches_revision(marker: &ScannerCycleRecoveryMarker, revision: &DataUsageCacheRevision) -> bool {
+    matches!(revision, DataUsageCacheRevision::Etag(etag) if marker.primary_revision == *etag)
+}
+
+fn validate_recovery_marker(marker: &ScannerCycleRecoveryMarker) -> Result<(), &'static str> {
+    if marker.schema_version != SCANNER_CYCLE_RECOVERY_SCHEMA_VERSION {
+        return Err("cycle recovery marker schema is unsupported");
+    }
+    if marker.primary_revision.is_empty() {
+        return Err("cycle recovery marker has no primary revision");
+    }
+    if marker.path != *DATA_USAGE_BLOOM_NAME_PATH {
+        return Err("cycle recovery marker path does not match the scanner scope");
+    }
+    if marker.quarantine_path != *DATA_USAGE_BLOOM_RECOVERY_PATH {
+        return Err("cycle recovery marker quarantine path does not match the scanner scope");
+    }
+    if !matches!(marker.classification.as_str(), "corrupt" | "future_schema") {
+        return Err("cycle recovery marker classification is invalid");
+    }
+    if !matches!(marker.state.as_str(), "blocked" | "cleanup-pending") {
+        return Err("cycle recovery marker state is invalid");
+    }
+    Ok(())
+}
+
+/// Decode only the stable scope and revision fields needed by an authenticated
+/// full-rescan reset. Startup keeps the strict decoder above so a newer marker
+/// cannot be interpreted as a trusted cursor; reset deliberately rebuilds from
+/// the persisted usage floor instead.
+pub(super) fn decode_recovery_marker_for_reset(
+    data: &[u8],
+    marker_revision: &DataUsageCacheRevision,
+) -> Result<ScannerCycleRecoveryMarker, ScannerError> {
+    if !matches!(marker_revision, DataUsageCacheRevision::Etag(_)) {
+        return Err(ScannerError::Other("cycle recovery marker has no object revision".to_string()));
+    }
+    let compat = serde_json::from_slice::<ScannerCycleRecoveryMarkerCompat>(data).ok();
+    let _schema_version = compat.as_ref().and_then(|marker| marker.schema_version);
+    let primary_revision = compat
+        .as_ref()
+        .and_then(|marker| marker.primary_revision.clone())
+        .filter(|revision| !revision.is_empty())
+        .unwrap_or_default();
+    let path = compat
+        .as_ref()
+        .and_then(|marker| marker.path.clone())
+        .unwrap_or_else(|| DATA_USAGE_BLOOM_NAME_PATH.clone());
+    let quarantine_path = compat
+        .as_ref()
+        .and_then(|marker| marker.quarantine_path.clone())
+        .unwrap_or_else(|| DATA_USAGE_BLOOM_RECOVERY_PATH.clone());
+    if path != *DATA_USAGE_BLOOM_NAME_PATH || quarantine_path != *DATA_USAGE_BLOOM_RECOVERY_PATH {
+        return Err(ScannerError::Other(
+            "cycle recovery marker path does not match the scanner scope".to_string(),
+        ));
+    }
+    let classification = match compat.as_ref().and_then(|marker| marker.classification.as_deref()) {
+        Some("corrupt") => "corrupt",
+        Some("future_schema") | None => "future_schema",
+        Some(_) => "future_schema",
+    };
+    let state = match compat.as_ref().and_then(|marker| marker.state.as_deref()) {
+        Some("cleanup-pending") => "cleanup-pending",
+        _ => "blocked",
+    };
+    let now = unix_now_secs();
+    Ok(ScannerCycleRecoveryMarker {
+        schema_version: SCANNER_CYCLE_RECOVERY_SCHEMA_VERSION,
+        primary_revision,
+        // Cursor and epoch values from an unknown marker are audit-only data;
+        // the reset path intentionally rebuilds both from the verified usage
+        // floor instead of carrying them across a version boundary.
+        generation: 0,
+        leader_epoch: 0,
+        classification: classification.to_string(),
+        first_detected_at_unix_secs: compat
+            .as_ref()
+            .and_then(|marker| marker.first_detected_at_unix_secs)
+            .unwrap_or(now),
+        last_attempt_at_unix_secs: compat
+            .as_ref()
+            .and_then(|marker| marker.last_attempt_at_unix_secs)
+            .unwrap_or(now),
+        retry_count: compat.as_ref().and_then(|marker| marker.retry_count).unwrap_or(0),
+        reason: compat
+            .as_ref()
+            .and_then(|marker| marker.reason.clone())
+            .unwrap_or_else(|| "operator requested full scanner rescan".to_string()),
+        path: DATA_USAGE_BLOOM_NAME_PATH.clone(),
+        quarantine_path: DATA_USAGE_BLOOM_RECOVERY_PATH.clone(),
+        state: state.to_string(),
+    })
+}
+
+async fn read_cycle_state_body(reader: &mut ScannerGetObjectReader) -> Result<Vec<u8>, CycleStateBodyReadError> {
+    let max_len = usize::try_from(MAX_SCANNER_CYCLE_STATE_BYTES).unwrap_or(usize::MAX);
+    let mut data = Vec::new();
+    reader
+        .take(MAX_SCANNER_CYCLE_STATE_BYTES.saturating_add(1))
+        .read_to_end(&mut data)
+        .await
+        .map_err(|err| CycleStateBodyReadError::Backend(EcstoreError::other(err)))?;
+    if data.len() > max_len {
+        return Err(CycleStateBodyReadError::TooLarge);
+    }
+    Ok(data)
+}
+
+fn cycle_state_classification(buf: &[u8]) -> (&'static str, &'static str) {
+    if buf.len() >= 16 && &buf[8..12] == b"RSCY" && &buf[8..16] != SCANNER_CYCLE_STATE_MAGIC {
+        ("future_schema", "scanner cycle state schema is newer than this reader")
+    } else {
+        ("corrupt", "scanner cycle state failed validation")
+    }
+}
+
+fn cycle_state_generation_and_epoch(buf: &[u8]) -> (u64, u64) {
+    let generation = buf
+        .get(..8)
+        .and_then(|bytes| bytes.try_into().ok())
+        .map(u64::from_le_bytes)
+        .unwrap_or(0);
+    let leader_epoch = if buf.len() >= SCANNER_CYCLE_STATE_HEADER_LEN && &buf[8..16] == SCANNER_CYCLE_STATE_MAGIC {
+        u64::from_le_bytes(buf[16..24].try_into().unwrap_or([0; 8]))
+    } else {
+        0
+    };
+    (generation, leader_epoch)
+}
+
+async fn persist_cycle_recovery_marker(
+    storeapi: Arc<impl ScannerObjectIO>,
+    primary_revision: &DataUsageCacheRevision,
+    generation: u64,
+    leader_epoch: u64,
+    classification: &'static str,
+    reason: &'static str,
+) -> Result<ScannerCycleRecoveryMarker, CycleRecoveryMarkerReadError> {
+    let now = unix_now_secs();
+    let (existing, existing_revision) = match read_cycle_recovery_marker_bytes(storeapi.clone()).await {
+        Ok(result) => result,
+        Err(err) => return Err(err),
+    };
+    let existing_marker = existing
+        .as_deref()
+        .and_then(|bytes| serde_json::from_slice::<ScannerCycleRecoveryMarker>(bytes).ok());
+    let primary_revision = match primary_revision {
+        DataUsageCacheRevision::Etag(etag) => etag.clone(),
+        DataUsageCacheRevision::Missing => {
+            return Err(CycleRecoveryMarkerReadError::Invalid("cycle state recovery requires a primary revision"));
+        }
+    };
+    let marker = ScannerCycleRecoveryMarker {
+        schema_version: SCANNER_CYCLE_RECOVERY_SCHEMA_VERSION,
+        primary_revision: primary_revision.clone(),
+        generation,
+        leader_epoch,
+        classification: classification.to_string(),
+        first_detected_at_unix_secs: existing_marker
+            .as_ref()
+            .filter(|marker| marker.primary_revision == primary_revision)
+            .map(|marker| marker.first_detected_at_unix_secs)
+            .unwrap_or(now),
+        last_attempt_at_unix_secs: now,
+        retry_count: existing_marker
+            .as_ref()
+            .filter(|marker| marker.primary_revision == primary_revision)
+            .map(|marker| marker.retry_count.saturating_add(1))
+            .unwrap_or(0),
+        reason: reason.to_string(),
+        path: DATA_USAGE_BLOOM_NAME_PATH.clone(),
+        quarantine_path: DATA_USAGE_BLOOM_RECOVERY_PATH.clone(),
+        state: "blocked".to_string(),
+    };
+    let bytes = serde_json::to_vec(&marker).map_err(|_| CycleRecoveryMarkerReadError::Invalid("marker serialization failed"))?;
+    let save_result = save_config_with_preconditions(
+        storeapi.clone(),
+        DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(),
+        bytes,
+        existing_revision.preconditions(),
+    )
+    .await;
+    match save_result {
+        Ok(_) => Ok(marker),
+        Err(EcstoreError::PreconditionFailed) => Err(CycleRecoveryMarkerReadError::Conflict),
+        Err(err) => Err(CycleRecoveryMarkerReadError::Backend(err)),
+    }
+}
+
+async fn read_cycle_recovery_marker_bytes(
+    storeapi: Arc<impl ScannerObjectIO>,
+) -> Result<(Option<Vec<u8>>, DataUsageCacheRevision), CycleRecoveryMarkerReadError> {
+    let mut reader = match storeapi
+        .get_object_reader(
+            RUSTFS_META_BUCKET,
+            DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(),
+            None,
+            http::HeaderMap::new(),
+            &ScannerObjectOptions {
+                no_lock: true,
+                ..Default::default()
+            },
+        )
+        .await
+    {
+        Ok(reader) => reader,
+        Err(
+            EcstoreError::FileNotFound
+            | EcstoreError::VolumeNotFound
+            | EcstoreError::ObjectNotFound(_, _)
+            | EcstoreError::BucketNotFound(_)
+            | EcstoreError::ConfigNotFound,
+        ) => {
+            return Ok((None, DataUsageCacheRevision::Missing));
+        }
+        Err(err) => return Err(CycleRecoveryMarkerReadError::Backend(err)),
+    };
+    let revision = reader
+        .object_info
+        .etag
+        .as_ref()
+        .filter(|etag| !etag.is_empty())
+        .cloned()
+        .map(DataUsageCacheRevision::Etag)
+        .ok_or(CycleRecoveryMarkerReadError::Invalid("marker has no revision"))?;
+    if reader.object_info.is_dir || reader.object_info.size < 0 || reader.object_info.size > 64 * 1024 {
+        return Err(CycleRecoveryMarkerReadError::Invalid("marker exceeds the bounded object size"));
+    }
+    let mut data = Vec::new();
+    (&mut reader)
+        .take(64 * 1024 + 1)
+        .read_to_end(&mut data)
+        .await
+        .map_err(|err| CycleRecoveryMarkerReadError::Backend(EcstoreError::other(err)))?;
+    if data.len() > 64 * 1024 {
+        return Err(CycleRecoveryMarkerReadError::Invalid("marker exceeds the bounded object size"));
+    }
+    if data.is_empty() {
+        return Err(CycleRecoveryMarkerReadError::Invalid("marker is empty"));
+    }
+    Ok((Some(data), revision))
+}
+
+async fn quarantine_invalid_cycle_state(
+    storeapi: Arc<impl ScannerObjectIO>,
+    revision: &DataUsageCacheRevision,
+    buf: &[u8],
+) -> ScannerCycleStateStartup {
+    let (classification, reason) = cycle_state_classification(buf);
+    let (generation, leader_epoch) = cycle_state_generation_and_epoch(buf);
+    quarantine_invalid_cycle_state_with_reason(storeapi, revision, generation, leader_epoch, classification, reason).await
+}
+
+async fn quarantine_invalid_cycle_state_with_reason(
+    storeapi: Arc<impl ScannerObjectIO>,
+    revision: &DataUsageCacheRevision,
+    generation: u64,
+    leader_epoch: u64,
+    classification: &'static str,
+    reason: &'static str,
+) -> ScannerCycleStateStartup {
+    let now = unix_now_secs();
+    let base_status = ScannerCycleRecoveryStatus {
+        path: DATA_USAGE_BLOOM_NAME_PATH.clone(),
+        quarantine_path: Some(DATA_USAGE_BLOOM_RECOVERY_PATH.clone()),
+        state: "recovery-required".to_string(),
+        classification: Some(classification.to_string()),
+        primary_revision: match revision {
+            DataUsageCacheRevision::Etag(etag) => Some(etag.clone()),
+            DataUsageCacheRevision::Missing => None,
+        },
+        generation: Some(generation),
+        leader_epoch: Some(leader_epoch),
+        first_detected_at_unix_secs: Some(now),
+        last_attempt_at_unix_secs: Some(now),
+        retry_count: 0,
+        max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES,
+        retryable: true,
+        reason: Some(reason.to_string()),
+    };
+    set_scanner_cycle_recovery_status(base_status);
+    match persist_cycle_recovery_marker(storeapi, revision, generation, leader_epoch, classification, reason).await {
+        Ok(marker) => set_scanner_cycle_recovery_status(recovery_status_from_marker(&marker, "blocked")),
+        Err(CycleRecoveryMarkerReadError::Backend(_)) => {
+            // Keep the poison object untouched and retry marker creation with the
+            // bounded startup backoff; recovery-required never becomes healthy.
+            return ScannerCycleStateStartup::Transient(ScannerError::Other(
+                "failed to persist scanner cycle recovery marker".to_string(),
+            ));
+        }
+        Err(CycleRecoveryMarkerReadError::Conflict) => {
+            set_scanner_cycle_recovery_status(recovery_status(
+                "transient",
+                Some("cycle recovery marker revision changed while publishing"),
+                true,
+            ));
+            return ScannerCycleStateStartup::Transient(ScannerError::Other(
+                "cycle recovery marker revision changed while publishing".to_string(),
+            ));
+        }
+        Err(CycleRecoveryMarkerReadError::Invalid(reason)) => {
+            set_scanner_cycle_recovery_status(recovery_status("recovery-required", Some(reason), false));
+            return ScannerCycleStateStartup::Blocked;
+        }
+    }
+    ScannerCycleStateStartup::Blocked
+}
+
+async fn mark_cycle_recovery_cleanup_pending(
+    storeapi: Arc<ECStore>,
+    mut marker: ScannerCycleRecoveryMarker,
+    marker_revision: &DataUsageCacheRevision,
+) -> Result<(ScannerCycleRecoveryMarker, DataUsageCacheRevision), ScannerError> {
+    marker.state = "cleanup-pending".to_string();
+    marker.last_attempt_at_unix_secs = unix_now_secs();
+    let bytes = serde_json::to_vec(&marker)
+        .map_err(|err| ScannerError::Other(format!("failed to encode cycle recovery marker: {err}")))?;
+    let info = save_config_with_preconditions(
+        storeapi.clone(),
+        DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(),
+        bytes,
+        marker_revision.preconditions(),
+    )
+    .await
+    .map_err(|err| ScannerError::Other(format!("failed to mark cycle recovery cleanup pending: {err}")))?;
+    let revision = info
+        .etag
+        .filter(|etag| !etag.is_empty())
+        .map(DataUsageCacheRevision::Etag)
+        .ok_or_else(|| ScannerError::Other("cycle recovery marker save returned no revision".to_string()))?;
+    Ok((marker, revision))
+}
+
+pub(crate) async fn load_scanner_cycle_state_for_startup(storeapi: Arc<impl ScannerObjectIO>) -> ScannerCycleStateStartup {
+    let marker = match read_cycle_recovery_marker_bytes(storeapi.clone()).await {
+        Ok((None, _)) => None,
+        Ok((Some(data), marker_revision)) => match serde_json::from_slice::<ScannerCycleRecoveryMarker>(&data) {
+            Ok(marker) => match validate_recovery_marker(&marker) {
+                Ok(()) => Some((marker, marker_revision)),
+                Err(reason) => {
+                    set_scanner_cycle_recovery_status(recovery_status("recovery-required", Some(reason), false));
+                    return ScannerCycleStateStartup::Blocked;
+                }
+            },
+            Err(_) => {
+                set_scanner_cycle_recovery_status(recovery_status(
+                    "recovery-required",
+                    Some("cycle recovery marker is invalid"),
+                    false,
+                ));
+                return ScannerCycleStateStartup::Blocked;
+            }
+        },
+        Err(CycleRecoveryMarkerReadError::Backend(err)) => {
+            let status = recovery_status("transient", Some("cycle recovery marker I/O is temporarily unavailable"), true);
+            set_scanner_cycle_recovery_status(status);
+            return ScannerCycleStateStartup::Transient(ScannerError::Other(format!(
+                "failed to read scanner cycle recovery marker: {err}"
+            )));
+        }
+        Err(CycleRecoveryMarkerReadError::Invalid(reason)) => {
+            set_scanner_cycle_recovery_status(recovery_status("recovery-required", Some(reason), false));
+            return ScannerCycleStateStartup::Blocked;
+        }
+        Err(CycleRecoveryMarkerReadError::Conflict) => {
+            set_scanner_cycle_recovery_status(recovery_status(
+                "transient",
+                Some("cycle recovery marker revision changed while being inspected"),
+                true,
+            ));
+            return ScannerCycleStateStartup::Transient(ScannerError::Other(
+                "cycle recovery marker revision changed while being inspected".to_string(),
+            ));
+        }
+    };
+
+    let mut reader = match storeapi
+        .get_object_reader(
+            RUSTFS_META_BUCKET,
+            DATA_USAGE_BLOOM_NAME_PATH.as_str(),
+            None,
+            http::HeaderMap::new(),
+            &ScannerObjectOptions {
+                no_lock: true,
+                ..Default::default()
+            },
+        )
+        .await
+    {
+        Ok(reader) => reader,
+        Err(
+            EcstoreError::FileNotFound
+            | EcstoreError::VolumeNotFound
+            | EcstoreError::ObjectNotFound(_, _)
+            | EcstoreError::BucketNotFound(_)
+            | EcstoreError::ConfigNotFound,
+        ) => {
+            if let Some((marker, _)) = marker {
+                let state = if marker.state == "cleanup-pending" {
+                    "cleanup-pending"
+                } else {
+                    "recovery-required"
+                };
+                set_scanner_cycle_recovery_status(recovery_status_from_marker(&marker, state));
+                return ScannerCycleStateStartup::Blocked;
+            }
+            set_scanner_cycle_recovery_status(recovery_status("healthy", None, false));
+            return ScannerCycleStateStartup::Ready {
+                cycle: CurrentCycle::default(),
+                leader_epoch: 0,
+                revision: DataUsageCacheRevision::Missing,
+            };
+        }
+        Err(err) => {
+            set_scanner_cycle_recovery_status(recovery_status("transient", Some("cycle state could not be inspected"), true));
+            return ScannerCycleStateStartup::Transient(ScannerError::Other(format!(
+                "failed to inspect scanner cycle state: {err}"
+            )));
+        }
+    };
+    let revision = reader
+        .object_info
+        .etag
+        .as_ref()
+        .filter(|etag| !etag.is_empty())
+        .cloned()
+        .map(DataUsageCacheRevision::Etag);
+    let Some(revision) = revision else {
+        set_scanner_cycle_recovery_status(recovery_status("recovery-required", Some("cycle state has no revision"), false));
+        return ScannerCycleStateStartup::Blocked;
+    };
+    let max_size = i64::try_from(MAX_SCANNER_CYCLE_STATE_BYTES).unwrap_or(i64::MAX);
+    if reader.object_info.is_dir || reader.object_info.size < 0 || reader.object_info.size > max_size {
+        return quarantine_invalid_cycle_state_with_reason(
+            storeapi,
+            &revision,
+            0,
+            0,
+            "corrupt",
+            "scanner cycle state object is oversized or not a regular object",
+        )
+        .await;
+    }
+    if let Some((marker, _)) = marker
+        .as_ref()
+        .filter(|(marker, _)| marker.state == "cleanup-pending" || marker_matches_revision(marker, &revision))
+    {
+        let state = if marker.state == "cleanup-pending" {
+            "cleanup-pending"
+        } else {
+            "blocked"
+        };
+        set_scanner_cycle_recovery_status(recovery_status_from_marker(marker, state));
+        return ScannerCycleStateStartup::Blocked;
+    }
+    let data = match read_cycle_state_body(&mut reader).await {
+        Ok(data) => data,
+        Err(CycleStateBodyReadError::TooLarge) => {
+            return quarantine_invalid_cycle_state_with_reason(
+                storeapi,
+                &revision,
+                0,
+                0,
+                "corrupt",
+                "scanner cycle state exceeds the bounded object size",
+            )
+            .await;
+        }
+        Err(CycleStateBodyReadError::Backend(err)) => {
+            set_scanner_cycle_recovery_status(recovery_status("transient", Some("cycle state read failed"), true));
+            return ScannerCycleStateStartup::Transient(ScannerError::Other(format!(
+                "failed to read scanner cycle state: {err}"
+            )));
+        }
+    };
+    if data.is_empty() {
+        return quarantine_invalid_cycle_state_with_reason(
+            storeapi,
+            &revision,
+            0,
+            0,
+            "corrupt",
+            "scanner cycle state object is empty",
+        )
+        .await;
+    }
+    match decode_scanner_cycle_state_for_startup(&data) {
+        Ok((cycle, leader_epoch)) => {
+            set_scanner_cycle_recovery_status(recovery_status("healthy", None, false));
+            ScannerCycleStateStartup::Ready {
+                cycle,
+                leader_epoch,
+                revision,
+            }
+        }
+        Err(_) => quarantine_invalid_cycle_state(storeapi, &revision, &data).await,
+    }
+}
+
+/// Reset a blocked cycle state after an operator has explicitly requested a full
+/// usage rebuild. The primary object is changed first with its observed ETag;
+/// the recovery marker is removed only when its own ETag still matches.
+pub async fn reset_scanner_cycle_recovery(ctx: CancellationToken, storeapi: Arc<ECStore>) -> Result<(), ScannerError> {
+    let lock = storeapi
+        .new_ns_lock(RUSTFS_META_BUCKET, "leader.lock")
+        .await
+        .map_err(|err| ScannerError::Other(format!("failed to acquire scanner leader lock: {err}")))?;
+    let guard = lock
+        .get_write_lock_quiet(Duration::from_secs(5))
+        .await
+        .map_err(|err| ScannerError::Other(format!("scanner leader lock is busy: {err}")))?;
+
+    if guard.is_lock_lost() {
+        return Err(ScannerError::Other("scanner leader lock was lost before recovery reset".to_string()));
+    }
+
+    let (marker_data, marker_revision) = read_cycle_recovery_marker_bytes(storeapi.clone())
+        .await
+        .map_err(|err| ScannerError::Other(format!("failed to read cycle recovery marker: {err}")))?;
+    let marker_data = marker_data.ok_or_else(|| ScannerError::Other("scanner cycle recovery marker is absent".to_string()))?;
+    let (marker, force_full_rescan) = match serde_json::from_slice::<ScannerCycleRecoveryMarker>(&marker_data) {
+        Ok(marker) if validate_recovery_marker(&marker).is_ok() => (marker, false),
+        _ => (decode_recovery_marker_for_reset(&marker_data, &marker_revision)?, true),
+    };
+
+    if guard.is_lock_lost() {
+        return Err(ScannerError::Other(
+            "scanner leader lock was lost while reading recovery state".to_string(),
+        ));
+    }
+
+    let (mut primary_reader, primary_revision) = match storeapi
+        .get_object_reader(
+            RUSTFS_META_BUCKET,
+            DATA_USAGE_BLOOM_NAME_PATH.as_str(),
+            None,
+            http::HeaderMap::new(),
+            &ScannerObjectOptions {
+                no_lock: true,
+                ..Default::default()
+            },
+        )
+        .await
+    {
+        Ok(reader) => {
+            let revision = reader
+                .object_info
+                .etag
+                .as_ref()
+                .filter(|etag| !etag.is_empty())
+                .cloned()
+                .ok_or_else(|| ScannerError::Other("scanner cycle state has no revision".to_string()))?;
+            (Some(reader), DataUsageCacheRevision::Etag(revision))
+        }
+        Err(
+            EcstoreError::FileNotFound
+            | EcstoreError::VolumeNotFound
+            | EcstoreError::ObjectNotFound(_, _)
+            | EcstoreError::BucketNotFound(_)
+            | EcstoreError::ConfigNotFound,
+        ) => (None, DataUsageCacheRevision::Missing),
+        Err(err) => return Err(ScannerError::Other(format!("failed to inspect scanner cycle state: {err}"))),
+    };
+    let marker_guards_primary =
+        force_full_rescan || marker.state == "cleanup-pending" || marker_matches_revision(&marker, &primary_revision);
+    if !marker_guards_primary && let Some(mut reader) = primary_reader.take() {
+        // A newer, independently fenced primary is authoritative. A
+        // full-rescan reset must not overwrite that progress; it only
+        // removes the stale recovery marker after validating the state.
+        let max_size = i64::try_from(MAX_SCANNER_CYCLE_STATE_BYTES).unwrap_or(i64::MAX);
+        if reader.object_info.is_dir || reader.object_info.size < 0 || reader.object_info.size > max_size {
+            return Err(ScannerError::Other("scanner cycle state changed since recovery was recorded".to_string()));
+        }
+        let data = read_cycle_state_body(&mut reader)
+            .await
+            .map_err(|err| ScannerError::Other(format!("scanner cycle state changed since recovery was recorded: {err}")))?;
+        if data.is_empty() || decode_scanner_cycle_state_for_startup(&data).is_err() {
+            return Err(ScannerError::Other("scanner cycle state changed since recovery was recorded".to_string()));
+        }
+        storeapi
+            .delete_config_object(
+                RUSTFS_META_BUCKET,
+                DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(),
+                ScannerObjectOptions {
+                    delete_prefix: true,
+                    delete_prefix_object: true,
+                    no_lock: true,
+                    http_preconditions: Some(marker_revision.preconditions()),
+                    ..Default::default()
+                },
+            )
+            .await
+            .map_err(|err| ScannerError::Other(format!("failed to clear stale cycle recovery marker: {err}")))?;
+        set_scanner_cycle_recovery_status(recovery_status("healthy", None, false));
+        super::notify_scanner_cycle_recovery_wake();
+        return Ok(());
+    }
+
+    if guard.is_lock_lost() {
+        return Err(ScannerError::Other(
+            "scanner leader lock was lost before rebuilding cycle state".to_string(),
+        ));
+    }
+
+    let floor = persisted_usage_floor(storeapi.clone()).await?;
+    // A full rescan must not trust a cursor recovered from a corrupt, future,
+    // or mixed-version marker. The durable usage floor is the only verified
+    // starting point; marker generation/epoch fields remain audit evidence.
+    let next = floor.next_cycle;
+    if next == u64::MAX {
+        return Err(ScannerError::Other("scanner cycle counter is exhausted".to_string()));
+    }
+    let leader_epoch = floor
+        .leader_epoch
+        .checked_add(1)
+        .ok_or_else(|| ScannerError::Other("scanner leader epoch is exhausted".to_string()))?;
+    let cycle = CurrentCycle {
+        next,
+        ..Default::default()
+    };
+    let data = encode_scanner_cycle_state(&cycle, leader_epoch)
+        .map_err(|err| ScannerError::Other(format!("failed to encode rebuilt scanner cycle state: {err}")))?;
+    // Persist the cleanup-pending phase before rewriting the primary. If the
+    // process dies after the rewrite, startup still sees a durable fence and
+    // cannot mistake the partially completed reset for a healthy state.
+    let (marker, marker_revision) = if marker.state == "cleanup-pending" {
+        (marker, marker_revision)
+    } else {
+        mark_cycle_recovery_cleanup_pending(storeapi.clone(), marker, &marker_revision).await?
+    };
+    let rebuilt_info = save_config_with_preconditions(
+        storeapi.clone(),
+        DATA_USAGE_BLOOM_NAME_PATH.as_str(),
+        data,
+        primary_revision.preconditions(),
+    )
+    .await
+    .map_err(|err| ScannerError::Other(format!("failed to persist rebuilt scanner cycle state: {err}")))?;
+    let rebuilt_revision = rebuilt_info
+        .etag
+        .filter(|etag| !etag.is_empty())
+        .ok_or_else(|| ScannerError::Other("rebuilt scanner cycle state has no revision".to_string()))?;
+    if guard.is_lock_lost() {
+        return Err(ScannerError::Other(
+            "scanner leader lock was lost after rebuilding cycle state".to_string(),
+        ));
+    }
+    if let Err(err) = fence_scanner_usage_epoch(&ctx, storeapi.clone(), leader_epoch).await {
+        set_scanner_cycle_recovery_status(ScannerCycleRecoveryStatus {
+            path: DATA_USAGE_BLOOM_NAME_PATH.clone(),
+            quarantine_path: Some(DATA_USAGE_BLOOM_RECOVERY_PATH.clone()),
+            state: "cleanup-pending".to_string(),
+            classification: Some(marker.classification.clone()),
+            primary_revision: Some(rebuilt_revision.clone()),
+            generation: Some(next),
+            leader_epoch: Some(leader_epoch),
+            first_detected_at_unix_secs: Some(marker.first_detected_at_unix_secs),
+            last_attempt_at_unix_secs: Some(unix_now_secs()),
+            retry_count: marker.retry_count,
+            max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES,
+            retryable: false,
+            reason: Some("cycle state rebuilt but usage epoch fencing failed".to_string()),
+        });
+        return Err(err);
+    }
+
+    let current_revision = storeapi
+        .get_object_reader(
+            RUSTFS_META_BUCKET,
+            DATA_USAGE_BLOOM_NAME_PATH.as_str(),
+            None,
+            http::HeaderMap::new(),
+            &ScannerObjectOptions {
+                no_lock: true,
+                ..Default::default()
+            },
+        )
+        .await
+        .map_err(|err| ScannerError::Other(format!("failed to verify rebuilt scanner cycle state: {err}")))?
+        .object_info
+        .etag
+        .filter(|etag| !etag.is_empty())
+        .ok_or_else(|| ScannerError::Other("rebuilt scanner cycle state lost its revision".to_string()))?;
+    if current_revision != rebuilt_revision {
+        set_scanner_cycle_recovery_status(ScannerCycleRecoveryStatus {
+            path: DATA_USAGE_BLOOM_NAME_PATH.clone(),
+            quarantine_path: Some(DATA_USAGE_BLOOM_RECOVERY_PATH.clone()),
+            state: "cleanup-pending".to_string(),
+            classification: Some(marker.classification.clone()),
+            primary_revision: Some(current_revision),
+            generation: Some(next),
+            leader_epoch: Some(leader_epoch),
+            first_detected_at_unix_secs: Some(marker.first_detected_at_unix_secs),
+            last_attempt_at_unix_secs: Some(unix_now_secs()),
+            retry_count: marker.retry_count,
+            max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES,
+            retryable: false,
+            reason: Some("rebuilt scanner cycle state changed before marker cleanup".to_string()),
+        });
+        return Err(ScannerError::Other(
+            "rebuilt scanner cycle state changed before recovery marker cleanup".to_string(),
+        ));
+    }
+
+    if guard.is_lock_lost() {
+        set_scanner_cycle_recovery_status(ScannerCycleRecoveryStatus {
+            path: DATA_USAGE_BLOOM_NAME_PATH.clone(),
+            quarantine_path: Some(DATA_USAGE_BLOOM_RECOVERY_PATH.clone()),
+            state: "cleanup-pending".to_string(),
+            classification: Some(marker.classification.clone()),
+            primary_revision: Some(rebuilt_revision.clone()),
+            generation: Some(next),
+            leader_epoch: Some(leader_epoch),
+            retry_count: marker.retry_count,
+            max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES,
+            retryable: false,
+            reason: Some("cycle state rebuilt but recovery marker was not cleared".to_string()),
+            ..Default::default()
+        });
+        return Err(ScannerError::Other(
+            "scanner leader lock was lost before clearing recovery marker".to_string(),
+        ));
+    }
+
+    if let Err(err) = storeapi
+        .delete_config_object(
+            RUSTFS_META_BUCKET,
+            DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(),
+            ScannerObjectOptions {
+                delete_prefix: true,
+                delete_prefix_object: true,
+                no_lock: true,
+                http_preconditions: Some(marker_revision.preconditions()),
+                ..Default::default()
+            },
+        )
+        .await
+    {
+        set_scanner_cycle_recovery_status(ScannerCycleRecoveryStatus {
+            path: DATA_USAGE_BLOOM_NAME_PATH.clone(),
+            quarantine_path: Some(DATA_USAGE_BLOOM_RECOVERY_PATH.clone()),
+            state: "cleanup-pending".to_string(),
+            classification: Some(marker.classification.clone()),
+            primary_revision: Some(rebuilt_revision.clone()),
+            generation: Some(next),
+            leader_epoch: Some(leader_epoch),
+            retry_count: marker.retry_count,
+            max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES,
+            retryable: false,
+            reason: Some("cycle state rebuilt but recovery marker cleanup failed".to_string()),
+            ..Default::default()
+        });
+        return Err(ScannerError::Other(format!("failed to clear cycle recovery marker: {err}")));
+    }
+    set_scanner_cycle_recovery_status(ScannerCycleRecoveryStatus {
+        path: DATA_USAGE_BLOOM_NAME_PATH.clone(),
+        quarantine_path: Some(DATA_USAGE_BLOOM_RECOVERY_PATH.clone()),
+        state: "healthy".to_string(),
+        max_retries: MAX_SCANNER_CYCLE_RECOVERY_RETRIES,
+        ..Default::default()
+    });
+    super::notify_scanner_cycle_recovery_wake();
+    Ok(())
+}
 
 #[derive(Debug, thiserror::Error)]
 pub(super) enum ScannerCycleStateError {

crates/scanner/src/scanner/tests.rs

@@ -15,11 +15,13 @@
 use super::*;
 use crate::EcstoreResult;
 use crate::{
-    Endpoint, EndpointServerPools, Endpoints, InstanceContext, PoolEndpoints, ScannerGetObjectReader as GetObjectReader,
-    ScannerObjectInfo as ObjectInfo, ScannerObjectOptions as ObjectOptions, ScannerPutObjReader as PutObjReader,
-    init_bucket_metadata_sys_for_scanner_tests, init_ecstore_config_for_scanner_tests, init_local_disks_with_instance_ctx,
+    DATA_USAGE_BLOOM_RECOVERY_PATH, Endpoint, EndpointServerPools, Endpoints, InstanceContext, PoolEndpoints,
+    ScannerGetObjectReader as GetObjectReader, ScannerObjectInfo as ObjectInfo, ScannerObjectOptions as ObjectOptions,
+    ScannerPutObjReader as PutObjReader, init_bucket_metadata_sys_for_scanner_tests, init_ecstore_config_for_scanner_tests,
+    init_local_disks_with_instance_ctx,
 };
-use std::collections::HashMap;
+use serial_test::serial;
+use std::collections::{HashMap, HashSet};
 use std::io::Cursor;
 use std::task::Poll;
 use temp_env::{with_var, with_var_unset};
@@ -151,6 +153,7 @@ impl Drop for ScannerDefaultCycleGuard {
 struct MemoryConfigStore {
     objects: Mutex<HashMap<String, Vec<u8>>>,
     revisions: Mutex<HashMap<String, u64>>,
+    non_regular_objects: Mutex<HashSet<String>>,
     fail_put_number: Mutex<HashMap<String, usize>>,
     object_not_found_put_number: Mutex<HashMap<String, usize>>,
     error_after_commit_put_number: Mutex<HashMap<String, usize>>,
@@ -191,12 +194,16 @@ impl crate::storage_api::scanner_io::ObjectIO for MemoryConfigStore {
             .get(&key)
             .cloned()
             .ok_or(EcstoreError::FileNotFound)?;
-        let revision = *self.revisions.lock().await.entry(key).or_insert(1);
+        let data_len = i64::try_from(data.len()).expect("memory test object length should fit in i64");
+        let revision = *self.revisions.lock().await.entry(key.clone()).or_insert(1);
+        let is_dir = self.non_regular_objects.lock().await.contains(&key);
 
         Ok(GetObjectReader {
             stream: Box::new(Cursor::new(data)),
             object_info: ObjectInfo {
                 etag: Some(format!("memory-{revision}")),
+                size: data_len,
+                is_dir,
                 ..Default::default()
             },
             buffered_body: None,
@@ -361,6 +368,7 @@ fn test_initial_scanner_delay_uses_configured_start_delay() {
 }
 
 #[test]
+#[serial]
 fn test_initial_scanner_delay_uses_cycle_without_explicit_start_delay() {
     with_var(ENV_SCANNER_CYCLE, Some("120"), || {
         crate::runtime_config::refresh_scanner_runtime_config_for_tests();
@@ -407,6 +415,7 @@ fn test_initial_scanner_delay_keeps_delay_for_replication_without_buckets() {
 }
 
 #[test]
+#[serial]
 fn test_scanner_cycle_max_duration_uses_env() {
     with_var(ENV_SCANNER_CYCLE_MAX_DURATION_SECS, Some("42"), || {
         assert_eq!(scanner_cycle_max_duration(), Some(Duration::from_secs(42)));
@@ -414,6 +423,7 @@ fn test_scanner_cycle_max_duration_uses_env() {
 }
 
 #[test]
+#[serial]
 fn test_scanner_cycle_max_duration_default_is_disabled() {
     with_var_unset(ENV_SCANNER_CYCLE_MAX_DURATION_SECS, || {
         assert_eq!(scanner_cycle_max_duration(), None);
@@ -457,6 +467,7 @@ async fn test_scanner_cycle_budget_drop_cancels_child_without_elapsed() {
 }
 
 #[test]
+#[serial]
 fn test_scanner_cycle_budget_config_uses_work_budget_env() {
     with_var(ENV_SCANNER_CYCLE_MAX_OBJECTS, Some("100"), || {
         with_var(ENV_SCANNER_CYCLE_MAX_DIRECTORIES, Some("25"), || {
@@ -468,6 +479,7 @@ fn test_scanner_cycle_budget_config_uses_work_budget_env() {
 }
 
 #[test]
+#[serial]
 fn test_scanner_cycle_budget_config_disables_zero_work_budgets() {
     with_var(ENV_SCANNER_CYCLE_MAX_OBJECTS, Some("0"), || {
         with_var(ENV_SCANNER_CYCLE_MAX_DIRECTORIES, Some("0"), || {
@@ -510,6 +522,7 @@ fn test_scan_cycle_partial_source_maps_budget_reason() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_mark_scan_cycle_idle_clears_published_cycle_state() {
     let mut cycle_info = CurrentCycle {
         current: 12,
@@ -538,6 +551,7 @@ async fn test_mark_scan_cycle_idle_clears_published_cycle_state() {
 }
 
 #[tokio::test]
+#[serial]
 async fn scanner_cycle_metrics_guard_covers_published_first_cycle_lifetime() {
     let cycle_started = Utc::now() - chrono::Duration::seconds(5);
     let mut cycle_info = CurrentCycle {
@@ -564,6 +578,7 @@ async fn scanner_cycle_metrics_guard_covers_published_first_cycle_lifetime() {
 }
 
 #[tokio::test]
+#[serial]
 async fn scanner_cycle_metrics_guard_keeps_active_cycle_published_during_finalization() {
     let mut cycle_info = CurrentCycle {
         current: 12,
@@ -588,6 +603,7 @@ async fn scanner_cycle_metrics_guard_keeps_active_cycle_published_during_finaliz
 }
 
 #[tokio::test]
+#[serial]
 async fn scanner_cycle_metrics_guard_drop_clears_activity() {
     let guard = ScannerCycleMetricsGuard::new(CurrentCycle {
         current: 12,
@@ -605,6 +621,7 @@ async fn scanner_cycle_metrics_guard_drop_clears_activity() {
 }
 
 #[tokio::test]
+#[serial]
 async fn run_data_scanner_cycle_publishes_activity_for_owner_lifetime() {
     let (_temp_dir, store) = setup_scanner_cycle_store().await;
     let ctx = CancellationToken::new();
@@ -655,6 +672,7 @@ async fn run_data_scanner_cycle_publishes_activity_for_owner_lifetime() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_finalize_partial_scan_cycle_advances_and_persists_counter() {
     let store = Arc::new(MemoryConfigStore::default());
     let ctx = CancellationToken::new();
@@ -690,6 +708,7 @@ async fn test_finalize_partial_scan_cycle_advances_and_persists_counter() {
 }
 
 #[tokio::test]
+#[serial]
 async fn scanner_cycle_recovers_to_newer_durable_cache_floor() {
     let store = Arc::new(MemoryConfigStore::default());
     let ctx = CancellationToken::new();
@@ -729,6 +748,7 @@ async fn scanner_cycle_recovers_to_newer_durable_cache_floor() {
 }
 
 #[tokio::test]
+#[serial]
 async fn scanner_cycle_rejects_invalid_cache_floor() {
     let store = Arc::new(MemoryConfigStore::default());
     let ctx = CancellationToken::new();
@@ -823,6 +843,327 @@ fn scanner_startup_fails_closed_on_nonempty_corrupt_cycle_state() {
     assert!(encode_scanner_cycle_state(&exhausted, 7).is_err());
 }
 
+#[tokio::test]
+#[serial]
+async fn corrupt_cycle_state_is_quarantined_once() {
+    let store = Arc::new(MemoryConfigStore::default());
+    let state_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str());
+    store.objects.lock().await.insert(state_key.clone(), vec![1]);
+    store.revisions.lock().await.insert(state_key.clone(), 7);
+
+    assert!(matches!(
+        load_scanner_cycle_state_for_startup(store.clone()).await,
+        ScannerCycleStateStartup::Blocked
+    ));
+    let marker_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str());
+    let marker_data = store
+        .objects
+        .lock()
+        .await
+        .get(&marker_key)
+        .cloned()
+        .expect("corrupt state must leave a durable recovery marker");
+    let marker: ScannerCycleRecoveryMarker = serde_json::from_slice(&marker_data).expect("marker should be valid JSON");
+    assert_eq!(marker.primary_revision, "memory-7");
+    assert_eq!(marker.path, DATA_USAGE_BLOOM_NAME_PATH.as_str());
+    assert_eq!(marker.quarantine_path, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str());
+    assert_eq!(marker.classification, "corrupt");
+
+    // A second startup sees the matching marker before consuming the poison body.
+    assert!(matches!(
+        load_scanner_cycle_state_for_startup(store.clone()).await,
+        ScannerCycleStateStartup::Blocked
+    ));
+
+    // Replacing the primary object advances its revision; the stale marker must
+    // not quarantine the newer, valid state.
+    let cycle = CurrentCycle {
+        next: 9,
+        ..Default::default()
+    };
+    let encoded = encode_scanner_cycle_state(&cycle, 3).expect("valid state should encode");
+    store.objects.lock().await.insert(state_key.clone(), encoded);
+    store.revisions.lock().await.insert(state_key, 8);
+    assert!(matches!(
+        load_scanner_cycle_state_for_startup(store).await,
+        ScannerCycleStateStartup::Ready {
+            cycle: CurrentCycle { next: 9, .. },
+            leader_epoch: 3,
+            ..
+        }
+    ));
+}
+
+#[tokio::test]
+#[serial]
+async fn empty_cycle_state_object_is_quarantined_as_corrupt() {
+    let store = Arc::new(MemoryConfigStore::default());
+    let state_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str());
+    store.objects.lock().await.insert(state_key.clone(), Vec::new());
+    store.revisions.lock().await.insert(state_key, 6);
+
+    assert!(matches!(
+        load_scanner_cycle_state_for_startup(store).await,
+        ScannerCycleStateStartup::Blocked
+    ));
+    assert_eq!(scanner_cycle_recovery_status().classification.as_deref(), Some("corrupt"));
+    assert!(
+        scanner_cycle_recovery_status()
+            .reason
+            .as_deref()
+            .is_some_and(|reason| reason.contains("empty"))
+    );
+}
+
+#[tokio::test]
+#[serial]
+async fn future_cycle_state_schema_is_recovery_required() {
+    let store = Arc::new(MemoryConfigStore::default());
+    let state_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str());
+    let mut future = 17_u64.to_le_bytes().to_vec();
+    future.extend_from_slice(b"RSCYC999");
+    future.extend_from_slice(&4_u64.to_le_bytes());
+    future.extend_from_slice(&[0x90]);
+    store.objects.lock().await.insert(state_key.clone(), future);
+    store.revisions.lock().await.insert(state_key, 13);
+
+    assert!(matches!(
+        load_scanner_cycle_state_for_startup(store).await,
+        ScannerCycleStateStartup::Blocked
+    ));
+    assert_eq!(scanner_cycle_recovery_status().classification.as_deref(), Some("future_schema"));
+}
+
+#[tokio::test]
+#[serial]
+async fn concurrent_leaders_cannot_quarantine_newer_cycle_state() {
+    let store = Arc::new(MemoryConfigStore::default());
+    let state_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str());
+    store.objects.lock().await.insert(state_key.clone(), vec![1]);
+    store.revisions.lock().await.insert(state_key, 4);
+
+    let (first, second) = tokio::join!(
+        load_scanner_cycle_state_for_startup(store.clone()),
+        load_scanner_cycle_state_for_startup(store.clone()),
+    );
+    assert!(matches!(first, ScannerCycleStateStartup::Blocked));
+    assert!(matches!(second, ScannerCycleStateStartup::Blocked));
+
+    let marker_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str());
+    let marker_data = store
+        .objects
+        .lock()
+        .await
+        .get(&marker_key)
+        .cloned()
+        .expect("one contender must publish the recovery marker");
+    let marker: ScannerCycleRecoveryMarker = serde_json::from_slice(&marker_data).expect("marker should decode");
+    assert_eq!(marker.primary_revision, "memory-4");
+}
+
+#[tokio::test]
+#[serial]
+async fn cleanup_pending_marker_blocks_a_rewritten_primary_after_restart() {
+    let store = Arc::new(MemoryConfigStore::default());
+    let state_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str());
+    let marker_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str());
+    let encoded = encode_scanner_cycle_state(
+        &CurrentCycle {
+            next: 12,
+            ..Default::default()
+        },
+        8,
+    )
+    .expect("valid state should encode");
+    store.objects.lock().await.insert(state_key.clone(), encoded);
+    store.revisions.lock().await.insert(state_key, 22);
+    let marker = ScannerCycleRecoveryMarker {
+        schema_version: 1,
+        primary_revision: "memory-21".to_string(),
+        generation: 11,
+        leader_epoch: 7,
+        classification: "corrupt".to_string(),
+        first_detected_at_unix_secs: 1,
+        last_attempt_at_unix_secs: 2,
+        retry_count: 1,
+        reason: "reset in progress".to_string(),
+        path: DATA_USAGE_BLOOM_NAME_PATH.clone(),
+        quarantine_path: DATA_USAGE_BLOOM_RECOVERY_PATH.clone(),
+        state: "cleanup-pending".to_string(),
+    };
+    store
+        .objects
+        .lock()
+        .await
+        .insert(marker_key.clone(), serde_json::to_vec(&marker).expect("marker should encode"));
+    store.revisions.lock().await.insert(marker_key, 3);
+
+    assert!(matches!(
+        load_scanner_cycle_state_for_startup(store).await,
+        ScannerCycleStateStartup::Blocked
+    ));
+    assert_eq!(scanner_cycle_recovery_status().state, "cleanup-pending");
+}
+
+#[test]
+fn full_rescan_reset_accepts_unknown_marker_fields_without_trusting_cursor() {
+    let marker = br#"{
+        "schema_version": 99,
+        "primary_revision": "memory-7",
+        "generation": 9000,
+        "leader_epoch": 9000,
+        "classification": "new-future-classification",
+        "first_detected_at_unix_secs": 1,
+        "last_attempt_at_unix_secs": 2,
+        "retry_count": 9,
+        "reason": "future marker",
+        "path": "buckets/.bloomcycle.bin",
+        "quarantine_path": "buckets/.bloomcycle.bin.recovery-required.json",
+        "future_field": {"cursor": "untrusted"}
+    }"#;
+    let decoded =
+        super::cycle_state::decode_recovery_marker_for_reset(marker, &DataUsageCacheRevision::Etag("memory-3".to_string()))
+            .expect("full-rescan compatibility decoder should accept additive fields");
+    assert_eq!(decoded.primary_revision, "memory-7");
+    assert_eq!(decoded.classification, "future_schema");
+    assert_eq!(decoded.generation, 0);
+    assert_eq!(decoded.leader_epoch, 0);
+    assert_eq!(decoded.state, "blocked");
+
+    let malformed =
+        super::cycle_state::decode_recovery_marker_for_reset(b"{not-json", &DataUsageCacheRevision::Etag("memory-4".to_string()))
+            .expect("a full-rescan reset must recover even when the marker is malformed");
+    assert!(malformed.primary_revision.is_empty());
+    assert_eq!(malformed.classification, "future_schema");
+}
+
+#[tokio::test]
+#[serial]
+async fn full_rescan_reset_rebuilds_after_malformed_marker_without_trusting_cursor() {
+    let (_temp_dir, store) = setup_scanner_cycle_store().await;
+    save_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str(), vec![0xff, 0x00, 0x01])
+        .await
+        .expect("corrupt cycle state should be persisted");
+    save_config(store.clone(), DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(), br#"{not-json"#.to_vec())
+        .await
+        .expect("malformed marker should be persisted");
+
+    reset_scanner_cycle_recovery(CancellationToken::new(), store.clone())
+        .await
+        .expect("full-rescan reset should recover malformed marker");
+
+    let state = read_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str())
+        .await
+        .expect("rebuilt cycle state should remain durable");
+    let (cycle, leader_epoch) = decode_scanner_cycle_state(&state).expect("rebuilt cycle state should decode");
+    assert_eq!(cycle.next, 0, "reset must use the verified usage floor, not marker cursor");
+    assert_eq!(leader_epoch, 1);
+    assert!(matches!(
+        read_config(store, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()).await,
+        Err(EcstoreError::ConfigNotFound)
+    ));
+}
+
+#[tokio::test]
+#[serial]
+async fn full_rescan_reset_rebuilds_when_primary_cycle_state_is_missing() {
+    let (_temp_dir, store) = setup_scanner_cycle_store().await;
+    let marker = ScannerCycleRecoveryMarker {
+        schema_version: 1,
+        primary_revision: "memory-missing".to_string(),
+        generation: u64::MAX,
+        leader_epoch: u64::MAX,
+        classification: "corrupt".to_string(),
+        first_detected_at_unix_secs: 1,
+        last_attempt_at_unix_secs: 2,
+        retry_count: 0,
+        reason: "missing primary".to_string(),
+        path: DATA_USAGE_BLOOM_NAME_PATH.clone(),
+        quarantine_path: DATA_USAGE_BLOOM_RECOVERY_PATH.clone(),
+        state: "blocked".to_string(),
+    };
+    save_config(
+        store.clone(),
+        DATA_USAGE_BLOOM_RECOVERY_PATH.as_str(),
+        serde_json::to_vec(&marker).expect("marker should encode"),
+    )
+    .await
+    .expect("marker should be persisted");
+
+    reset_scanner_cycle_recovery(CancellationToken::new(), store.clone())
+        .await
+        .expect("full-rescan reset should recreate missing primary");
+    let state = read_config(store.clone(), DATA_USAGE_BLOOM_NAME_PATH.as_str())
+        .await
+        .expect("missing primary should be rebuilt");
+    let (cycle, leader_epoch) = decode_scanner_cycle_state(&state).expect("rebuilt cycle state should decode");
+    assert_eq!(cycle.next, 0);
+    assert_eq!(leader_epoch, 1);
+    assert!(matches!(
+        read_config(store, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()).await,
+        Err(EcstoreError::ConfigNotFound)
+    ));
+}
+
+#[tokio::test]
+#[serial]
+async fn corrupt_cycle_state_rename_or_marker_failure_stays_recovery_required() {
+    let store = Arc::new(MemoryConfigStore::default());
+    let state_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str());
+    let marker_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str());
+    store.objects.lock().await.insert(state_key.clone(), vec![1]);
+    store.revisions.lock().await.insert(state_key, 9);
+    store.fail_put_number.lock().await.insert(marker_key, 1);
+
+    assert!(matches!(
+        load_scanner_cycle_state_for_startup(store.clone()).await,
+        ScannerCycleStateStartup::Transient(_)
+    ));
+    let status = scanner_cycle_recovery_status();
+    assert_eq!(status.state, "recovery-required");
+    assert!(status.retryable);
+    assert!(
+        store
+            .objects
+            .lock()
+            .await
+            .contains_key(&memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str()))
+    );
+}
+
+#[tokio::test]
+#[serial]
+async fn oversized_or_symlinked_cycle_state_is_rejected() {
+    let store = Arc::new(MemoryConfigStore::default());
+    let key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_NAME_PATH.as_str());
+    store.objects.lock().await.insert(key.clone(), vec![0; 1024 * 1024 + 1]);
+    store.revisions.lock().await.insert(key.clone(), 11);
+
+    assert!(matches!(
+        load_scanner_cycle_state_for_startup(store.clone()).await,
+        ScannerCycleStateStartup::Blocked
+    ));
+    assert_eq!(scanner_cycle_recovery_status().classification.as_deref(), Some("corrupt"));
+    assert!(
+        scanner_cycle_recovery_status()
+            .reason
+            .as_deref()
+            .is_some_and(|reason| reason.contains("oversized"))
+    );
+
+    let marker_key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_BLOOM_RECOVERY_PATH.as_str());
+    store.objects.lock().await.remove(&marker_key);
+    store.objects.lock().await.insert(key.clone(), vec![1]);
+    store.revisions.lock().await.insert(key.clone(), 12);
+    store.non_regular_objects.lock().await.insert(key);
+    // The object contract exposes a non-regular object as `is_dir`; local
+    // backends reject symlink/reparse entries before they become an object.
+    assert!(matches!(
+        load_scanner_cycle_state_for_startup(store).await,
+        ScannerCycleStateStartup::Blocked
+    ));
+}
+
 #[tokio::test]
 async fn scanner_startup_uses_primary_and_backup_usage_floor() {
     let store = Arc::new(MemoryConfigStore::default());
@@ -999,6 +1340,7 @@ async fn scanner_usage_floor_fails_closed_on_corrupt_or_exhausted_usage_state()
 }
 
 #[tokio::test]
+#[serial]
 async fn scanner_usage_backup_uses_durable_cycle_cadence_across_tasks() {
     let store = Arc::new(MemoryConfigStore::default());
     let ctx = CancellationToken::new();
@@ -1072,6 +1414,7 @@ fn scanner_cycle_advance_fails_before_reserved_exhausted_value() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_finalize_partial_scan_cycle_reports_persist_failure() {
     let store = Arc::new(MemoryConfigStore::default());
     let ctx = CancellationToken::new();
@@ -1095,6 +1438,7 @@ async fn test_finalize_partial_scan_cycle_reports_persist_failure() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_persist_scanner_cycle_state_reconciles_newer_winner() {
     let store = Arc::new(MemoryConfigStore::default());
     let ctx = CancellationToken::new();
@@ -1517,6 +1861,7 @@ async fn test_usage_save_route_barrier_prevents_missing_snapshot_creation() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_usage_route_barrier_precedes_durable_reconciliation() {
     let store = Arc::new(MemoryConfigStore::default());
     let key = memory_config_key(RUSTFS_META_BUCKET, DATA_USAGE_OBJ_NAME_PATH.as_str());
@@ -1544,6 +1889,7 @@ async fn test_usage_route_barrier_precedes_durable_reconciliation() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_deferred_usage_save_keeps_last_real_save_metric() {
     let metrics = global_metrics();
     metrics.record_scanner_usage_save_result(ScannerUsageSaveResult::Success);
@@ -2607,6 +2953,7 @@ fn scanner_cycle_cache_floor_stays_pending_during_deferred_usage_publication() {
 }
 
 #[test]
+#[serial]
 fn finalizing_a_saved_cycle_acknowledges_its_exact_dirty_snapshot() {
     crate::scanner_io::clear_dirty_usage_bucket("photos");
     crate::scanner_io::record_dirty_usage_bucket("photos");
@@ -2633,6 +2980,7 @@ fn finalizing_a_saved_cycle_acknowledges_its_exact_dirty_snapshot() {
 }
 
 #[test]
+#[serial]
 fn finalizing_a_deferred_usage_save_keeps_dirty_work_pending() {
     crate::scanner_io::clear_dirty_usage_bucket("photos");
     crate::scanner_io::record_dirty_usage_bucket("photos");
@@ -2675,6 +3023,7 @@ async fn scanner_cycle_keeps_remote_pending_acknowledgement() {
 }
 
 #[test]
+#[serial]
 fn finalizing_an_already_durable_cycle_acknowledges_its_exact_dirty_snapshot() {
     crate::scanner_io::clear_dirty_usage_bucket("photos");
     crate::scanner_io::record_dirty_usage_bucket("photos");
@@ -2689,6 +3038,7 @@ fn finalizing_an_already_durable_cycle_acknowledges_its_exact_dirty_snapshot() {
 }
 
 #[test]
+#[serial]
 fn finalizing_a_prior_same_cycle_snapshot_keeps_new_dirty_work_pending() {
     crate::scanner_io::clear_dirty_usage_bucket("photos");
     crate::scanner_io::record_dirty_usage_bucket("photos");
@@ -2704,6 +3054,7 @@ fn finalizing_a_prior_same_cycle_snapshot_keeps_new_dirty_work_pending() {
 }
 
 #[test]
+#[serial]
 fn finalizing_a_durable_superseded_snapshot_keeps_dirty_work_pending() {
     crate::scanner_io::clear_dirty_usage_bucket("photos");
     crate::scanner_io::record_dirty_usage_bucket("photos");
@@ -2719,6 +3070,7 @@ fn finalizing_a_durable_superseded_snapshot_keeps_dirty_work_pending() {
 }
 
 #[test]
+#[serial]
 fn data_usage_persist_wait_covers_cache_retries_and_backup() {
     with_var(rustfs_config::ENV_SCANNER_CACHE_SAVE_TIMEOUT_SECS, Some("7"), || {
         crate::runtime_config::refresh_scanner_runtime_config_for_tests();
@@ -2771,6 +3123,7 @@ async fn maintenance_feature_inspection_preserves_base_cycle_after_timeout() {
 }
 
 #[tokio::test(start_paused = true)]
+#[serial]
 async fn stable_maintenance_detection_preserves_base_cycle_after_timeout() {
     let ctx = CancellationToken::new();
 
@@ -2836,6 +3189,7 @@ async fn maintenance_feature_inspection_stops_on_cancellation() {
 }
 
 #[test]
+#[serial]
 fn test_cycle_interval_prefers_explicit_cycle_override() {
     with_var(ENV_SCANNER_SPEED, Some("slowest"), || {
         with_var(ENV_SCANNER_CYCLE, Some("42"), || {
@@ -2845,6 +3199,7 @@ fn test_cycle_interval_prefers_explicit_cycle_override() {
 }
 
 #[test]
+#[serial]
 fn test_cycle_interval_prefers_explicit_cycle_over_default_cycle() {
     let _guard = ScannerDefaultCycleGuard::set(TEST_DEFAULT_SCANNER_CYCLE_SECS);
 
@@ -2854,6 +3209,7 @@ fn test_cycle_interval_prefers_explicit_cycle_over_default_cycle() {
 }
 
 #[test]
+#[serial]
 fn test_cycle_interval_uses_scanner_default_speed_override_when_unconfigured() {
     let _guard = ScannerDefaultSpeedGuard::set(ScannerSpeed::Slowest);
 
@@ -2863,6 +3219,7 @@ fn test_cycle_interval_uses_scanner_default_speed_override_when_unconfigured() {
 }
 
 #[test]
+#[serial]
 fn test_cycle_interval_prefers_explicit_speed_over_default_speed_override() {
     let _guard = ScannerDefaultSpeedGuard::set(ScannerSpeed::Slowest);
 
@@ -2880,6 +3237,7 @@ fn test_cycle_interval_prefers_explicit_speed_over_default_speed_override() {
 }
 
 #[test]
+#[serial]
 fn test_cycle_interval_uses_default_cycle_override_when_unconfigured() {
     let _guard = ScannerDefaultCycleGuard::set(TEST_DEFAULT_SCANNER_CYCLE_SECS);
 
@@ -2975,6 +3333,24 @@ fn superseded_retry_backoff_grows_from_the_default_cycle() {
     }
 }
 
+#[tokio::test(start_paused = true)]
+async fn corrupt_cycle_state_backoff_uses_virtual_clock() {
+    let mut backoff = ScannerRetryBackoff::default();
+    backoff.record_retryable_cycle(true);
+    let first_delay = backoff
+        .retry_interval(Duration::from_secs(60))
+        .expect("the first recovery retry should be scheduled");
+    assert_eq!(first_delay, Duration::from_secs(5));
+
+    let deadline = Instant::now() + first_delay;
+    assert!(Instant::now() < deadline);
+    tokio::time::advance(first_delay).await;
+    assert!(Instant::now() >= deadline);
+
+    backoff.record_retryable_cycle(true);
+    assert_eq!(backoff.retry_interval(Duration::from_secs(60)), Some(Duration::from_secs(10)));
+}
+
 #[test]
 fn scanner_cycle_wait_plan_drives_growth_resets_and_bitrot_cap() {
     let runtime_config = ScannerRuntimeConfig {
@@ -3043,6 +3419,7 @@ fn scanner_cycle_wait_plan_drives_growth_resets_and_bitrot_cap() {
 }
 
 #[test]
+#[serial]
 fn scanner_cycle_schedule_status_reports_effective_backoff() {
     record_scanner_cycle_schedule(Duration::from_millis(86_400_001), true, 2_048, true, 7);
 
@@ -3322,6 +3699,7 @@ fn dirty_usage_wakes_are_disabled_for_explicit_cycle_policy() {
 }
 
 #[test]
+#[serial]
 fn clean_idle_cap_preserves_default_bitrot_coverage_window() {
     let config = ScannerRuntimeConfig {
         bitrot_cycle: Some(Duration::from_secs(30 * 24 * 60 * 60)),
@@ -3351,6 +3729,7 @@ fn clean_idle_cap_allows_policy_max_when_bitrot_is_disabled() {
 }
 
 #[test]
+#[serial]
 fn clean_idle_cap_never_shortens_the_base_cycle() {
     let config = ScannerRuntimeConfig {
         bitrot_cycle: Some(Duration::from_secs(60)),
@@ -3364,6 +3743,7 @@ fn clean_idle_cap_never_shortens_the_base_cycle() {
 }
 
 #[test]
+#[serial]
 fn test_cycle_interval_keeps_default_cycle_with_explicit_speed() {
     let _guard = ScannerDefaultCycleGuard::set(TEST_DEFAULT_SCANNER_CYCLE_SECS);
 
@@ -3381,6 +3761,7 @@ fn test_cycle_interval_keeps_default_cycle_with_explicit_speed() {
 }
 
 #[test]
+#[serial]
 fn test_cycle_interval_prefers_explicit_start_delay_over_default_cycle() {
     let _guard = ScannerDefaultCycleGuard::set(TEST_DEFAULT_SCANNER_CYCLE_SECS);
 
@@ -3394,6 +3775,7 @@ fn test_cycle_interval_prefers_explicit_start_delay_over_default_cycle() {
 }
 
 #[test]
+#[serial]
 fn test_cycle_interval_supports_minio_speed_alias() {
     with_var_unset(ENV_SCANNER_SPEED, || {
         with_var_unset(ENV_SCANNER_CYCLE, || {
@@ -3407,6 +3789,7 @@ fn test_cycle_interval_supports_minio_speed_alias() {
 }
 
 #[test]
+#[serial]
 fn test_cycle_interval_supports_minio_cycle_alias() {
     with_var_unset(ENV_SCANNER_CYCLE, || {
         with_var_unset(ENV_SCANNER_START_DELAY_SECS, || {
@@ -3426,6 +3809,7 @@ fn test_randomized_cycle_delay_handles_small_start_delay() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_wait_for_next_scanner_cycle_wakes_for_dirty_usage() {
     crate::scanner_io::clear_dirty_usage_buckets_for_tests();
 
@@ -3451,6 +3835,7 @@ async fn test_wait_for_next_scanner_cycle_wakes_for_dirty_usage() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_wait_for_next_scanner_cycle_sees_unattempted_dirty_usage() {
     crate::scanner_io::clear_dirty_usage_buckets_for_tests();
     let dirty_generation = crate::scanner_io::dirty_usage_generation();
@@ -3472,6 +3857,7 @@ async fn test_wait_for_next_scanner_cycle_sees_unattempted_dirty_usage() {
 }
 
 #[tokio::test(start_paused = true)]
+#[serial]
 async fn test_wait_for_next_scanner_cycle_retries_stable_dirty_usage_on_timer() {
     crate::scanner_io::clear_dirty_usage_buckets_for_tests();
     crate::scanner_io::record_dirty_usage_bucket("photos");
@@ -3493,6 +3879,7 @@ async fn test_wait_for_next_scanner_cycle_retries_stable_dirty_usage_on_timer()
 }
 
 #[tokio::test(start_paused = true)]
+#[serial]
 async fn test_wait_for_next_scanner_cycle_can_defer_dirty_wakes_until_timer() {
     crate::scanner_io::clear_dirty_usage_buckets_for_tests();
     let ctx = CancellationToken::new();
@@ -3511,6 +3898,7 @@ async fn test_wait_for_next_scanner_cycle_can_defer_dirty_wakes_until_timer() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_wait_for_next_scanner_cycle_wakes_for_repeated_dirty_bucket() {
     crate::scanner_io::clear_dirty_usage_buckets_for_tests();
     crate::scanner_io::record_dirty_usage_bucket("photos");
@@ -3536,6 +3924,7 @@ async fn test_wait_for_next_scanner_cycle_wakes_for_repeated_dirty_bucket() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_wait_for_next_scanner_cycle_reschedules_for_runtime_config() {
     crate::scanner_io::clear_dirty_usage_buckets_for_tests();
     let observed_generation = crate::runtime_config::scanner_runtime_config_generation();
@@ -3563,6 +3952,7 @@ async fn test_wait_for_next_scanner_cycle_reschedules_for_runtime_config() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_wait_for_next_scanner_cycle_reschedules_for_maintenance_change() {
     crate::scanner_io::clear_dirty_usage_buckets_for_tests();
     let observed_generation = crate::scanner_io::scanner_maintenance_generation();
@@ -3806,6 +4196,7 @@ fn scanner_activity_after_a_cycle_restores_the_base_interval() {
 }
 
 #[tokio::test(start_paused = true)]
+#[serial]
 async fn distributed_clean_idle_wait_wakes_at_base_interval_for_remote_activity() {
     crate::scanner_io::clear_dirty_usage_buckets_for_tests();
     let ctx = CancellationToken::new();
@@ -3833,6 +4224,7 @@ async fn distributed_clean_idle_wait_wakes_at_base_interval_for_remote_activity(
 }
 
 #[tokio::test(start_paused = true)]
+#[serial]
 async fn superseded_retry_wait_defers_dirty_cluster_activity_until_timer() {
     crate::scanner_io::clear_dirty_usage_buckets_for_tests();
     let ctx = CancellationToken::new();
@@ -3860,6 +4252,7 @@ async fn superseded_retry_wait_defers_dirty_cluster_activity_until_timer() {
 }
 
 #[tokio::test(start_paused = true)]
+#[serial]
 async fn distributed_clean_idle_wait_blocks_backoff_for_unpropagated_maintenance() {
     crate::scanner_io::clear_dirty_usage_buckets_for_tests();
     let ctx = CancellationToken::new();
@@ -3886,6 +4279,7 @@ async fn distributed_clean_idle_wait_blocks_backoff_for_unpropagated_maintenance
 }
 
 #[tokio::test(start_paused = true)]
+#[serial]
 async fn distributed_clean_idle_wait_fails_closed_when_a_peer_is_unverifiable() {
     crate::scanner_io::clear_dirty_usage_buckets_for_tests();
     let ctx = CancellationToken::new();
@@ -3912,6 +4306,7 @@ async fn distributed_clean_idle_wait_fails_closed_when_a_peer_is_unverifiable()
 }
 
 #[tokio::test(start_paused = true)]
+#[serial]
 async fn distributed_clean_idle_wait_keeps_the_extended_deadline_when_peers_are_clean() {
     crate::scanner_io::clear_dirty_usage_buckets_for_tests();
     let ctx = CancellationToken::new();
@@ -3939,6 +4334,7 @@ async fn distributed_clean_idle_wait_keeps_the_extended_deadline_when_peers_are_
 }
 
 #[tokio::test(start_paused = true)]
+#[serial]
 async fn scanner_activity_probe_wait_is_cancellation_aware() {
     crate::scanner_io::clear_dirty_usage_buckets_for_tests();
     let ctx = CancellationToken::new();
@@ -3969,6 +4365,7 @@ async fn scanner_activity_probe_wait_is_cancellation_aware() {
 }
 
 #[tokio::test(start_paused = true)]
+#[serial]
 async fn scanner_activity_probe_wait_stops_after_leader_lock_loss() {
     crate::scanner_io::clear_dirty_usage_buckets_for_tests();
     let ctx = CancellationToken::new();
@@ -4000,6 +4397,7 @@ async fn scanner_activity_probe_wait_stops_after_leader_lock_loss() {
 }
 
 #[test]
+#[serial]
 fn test_get_cycle_scan_mode_runs_deep_until_selection_window_completes() {
     with_var(ENV_SCANNER_BITROT_CYCLE_SECS, Some("3600"), || {
         let mode = get_cycle_scan_mode(10, 0, Some(Utc::now()), bitrot_scan_cycle());
@@ -4008,6 +4406,7 @@ fn test_get_cycle_scan_mode_runs_deep_until_selection_window_completes() {
 }
 
 #[test]
+#[serial]
 fn test_get_cycle_scan_mode_respects_elapsed_bitrot_cycle() {
     with_var(ENV_SCANNER_BITROT_CYCLE_SECS, Some("3600"), || {
         let recent = Utc::now() - chrono::Duration::minutes(30);
@@ -4019,6 +4418,7 @@ fn test_get_cycle_scan_mode_respects_elapsed_bitrot_cycle() {
 }
 
 #[test]
+#[serial]
 fn test_get_cycle_scan_mode_can_disable_periodic_deep_scan() {
     with_var(ENV_SCANNER_BITROT_CYCLE_SECS, Some("off"), || {
         assert_eq!(get_cycle_scan_mode(1, 0, None, bitrot_scan_cycle()), HealScanMode::Normal);
@@ -4026,6 +4426,7 @@ fn test_get_cycle_scan_mode_can_disable_periodic_deep_scan() {
 }
 
 #[test]
+#[serial]
 fn test_background_heal_info_for_scan_start_marks_deep_active() {
     let now = Utc::now();
     let info =
@@ -4038,6 +4439,7 @@ fn test_background_heal_info_for_scan_start_marks_deep_active() {
 }
 
 #[test]
+#[serial]
 fn test_background_heal_info_for_scan_start_keeps_deep_window_start() {
     with_var_unset(ENV_SCANNER_BITROT_CYCLE_SECS, || {
         let started_at = Utc::now();

crates/scanner/src/scanner_folder/tests.rs

@@ -18,6 +18,7 @@ use super::*;
 use crate::storage_api::VersionPurgeStatusType;
 use crate::{DiskOption, Endpoint, STORAGE_FORMAT_FILE, TierStats, new_disk, storageclass};
 use rustfs_filemeta::{FileInfo, FileMeta};
+use serial_test::serial;
 use std::io::Write;
 #[cfg(unix)]
 use std::os::unix::fs::{PermissionsExt, symlink};
@@ -355,6 +356,7 @@ impl Drop for TestGuard {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_should_skip_failed_respects_ttl() {
     let (mut scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir);
@@ -376,6 +378,7 @@ async fn test_should_skip_failed_respects_ttl() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_record_failed_ttl_zero_noop() {
     let (mut scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard::new(0, 100, &mut scanner, temp_dir);
@@ -464,6 +467,7 @@ fn test_should_account_replication_stats_only_for_live_object_versions() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_heal_replication_only_queues_pending_null_deletes() {
     async fn replication_skipped_count() -> u64 {
         global_metrics()
@@ -712,6 +716,7 @@ async fn test_scanner_heal_admission_accounting_maps_deep_scan_to_bitrot() {
 }
 
 #[test]
+#[serial]
 fn test_excessive_version_alert_thresholds_use_env() {
     with_var(rustfs_config::ENV_SCANNER_ALERT_EXCESS_VERSIONS, Some("3"), || {
         with_var(rustfs_config::ENV_SCANNER_ALERT_EXCESS_VERSION_SIZE, Some("100"), || {
@@ -726,6 +731,7 @@ fn test_excessive_version_alert_thresholds_use_env() {
 }
 
 #[test]
+#[serial]
 fn test_excessive_folders_threshold_uses_env() {
     with_var(rustfs_config::ENV_SCANNER_ALERT_EXCESS_FOLDERS, Some("3"), || {
         crate::runtime_config::refresh_scanner_runtime_config_for_tests();
@@ -735,6 +741,7 @@ fn test_excessive_folders_threshold_uses_env() {
 }
 
 #[test]
+#[serial]
 fn test_excessive_folders_threshold_default_supports_pbs_layout() {
     with_var_unset(rustfs_config::ENV_SCANNER_ALERT_EXCESS_FOLDERS, || {
         crate::runtime_config::refresh_scanner_runtime_config_for_tests();
@@ -744,6 +751,7 @@ fn test_excessive_folders_threshold_default_supports_pbs_layout() {
 }
 
 #[test]
+#[serial]
 fn test_scanner_yield_every_n_objects_uses_env() {
     with_var(rustfs_config::ENV_SCANNER_YIELD_EVERY_N_OBJECTS, Some("32"), || {
         crate::runtime_config::refresh_scanner_runtime_config_for_tests();
@@ -753,6 +761,7 @@ fn test_scanner_yield_every_n_objects_uses_env() {
 }
 
 #[test]
+#[serial]
 fn test_scanner_yield_every_n_objects_uses_default() {
     with_var_unset(rustfs_config::ENV_SCANNER_YIELD_EVERY_N_OBJECTS, || {
         crate::runtime_config::refresh_scanner_runtime_config_for_tests();
@@ -879,6 +888,7 @@ fn test_order_folders_for_resume_reports_stale_hint() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_record_failed_prunes_to_max_entries() {
     let (mut scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard::new(1000, 2, &mut scanner, temp_dir);
@@ -910,6 +920,7 @@ async fn test_record_failed_prunes_to_max_entries() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_prune_failed_objects_cache_drops_expired() {
     let (mut scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard::new(5, 10, &mut scanner, temp_dir);
@@ -933,6 +944,7 @@ async fn test_prune_failed_objects_cache_drops_expired() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_prune_failed_objects_max_zero_keeps_fresh() {
     let (mut scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard::new(60, 0, &mut scanner, temp_dir);
@@ -1689,6 +1701,7 @@ async fn test_heal_actions_returns_actual_size_without_inline_heal() {
 }
 
 #[tokio::test]
+#[serial]
 #[cfg(unix)]
 async fn test_scan_folder_skips_unreadable_child_directory() {
     let (mut scanner, temp_dir) = build_test_scanner().await;
@@ -1721,6 +1734,7 @@ async fn test_scan_folder_skips_unreadable_child_directory() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_scan_folder_exits_when_abandoned_child_listing_finishes() {
     let (mut scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
@@ -1799,6 +1813,7 @@ async fn test_scan_folder_exits_when_abandoned_child_listing_finishes() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_scan_folder_xl_meta_named_directory_uses_namespace_descent() {
     let (mut scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
@@ -1844,6 +1859,7 @@ async fn test_scan_folder_xl_meta_named_directory_uses_namespace_descent() {
 }
 
 #[tokio::test(flavor = "current_thread")]
+#[serial]
 async fn test_scan_folder_corrupt_xl_meta_stops_erasure_data_dir_descent() {
     let logs = CapturedLogs::default();
     let subscriber = tracing_subscriber::fmt()
@@ -2005,6 +2021,7 @@ async fn test_scan_folder_corrupt_xl_meta_stops_erasure_data_dir_descent() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_scan_folder_missing_xl_meta_stops_erasure_data_dir_descent() {
     let (mut scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
@@ -2082,6 +2099,7 @@ async fn test_scan_folder_missing_xl_meta_stops_erasure_data_dir_descent() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_scan_folder_uuid_namespace_part_name_directory_is_not_data_dir() {
     let (mut scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
@@ -2143,6 +2161,7 @@ async fn test_scan_folder_uuid_namespace_part_name_directory_is_not_data_dir() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_scan_folder_non_erasure_metadata_keeps_namespace_descent() {
     let (mut scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
@@ -2184,6 +2203,7 @@ async fn test_scan_folder_non_erasure_metadata_keeps_namespace_descent() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_scan_folder_compacted_parent_sends_partial_update() {
     let (mut scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
@@ -2225,6 +2245,7 @@ async fn test_scan_folder_compacted_parent_sends_partial_update() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_scan_data_folder_cancelled_before_scan_clears_current_path() {
     let (scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard {
@@ -2269,6 +2290,7 @@ async fn test_scan_data_folder_cancelled_before_scan_clears_current_path() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_scan_data_folder_returns_partial_cache_on_budget_cancel() {
     let (mut scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard::new(60, 100, &mut scanner, temp_dir.clone());
@@ -2324,6 +2346,7 @@ async fn test_scan_data_folder_returns_partial_cache_on_budget_cancel() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_scan_data_folder_reports_invalid_checkpoint_ignored_once() {
     let (scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard {
@@ -2368,6 +2391,7 @@ async fn test_scan_data_folder_reports_invalid_checkpoint_ignored_once() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_scan_data_folder_resume_hint_prioritizes_next_existing_folder() {
     let (scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard {
@@ -2441,6 +2465,7 @@ async fn test_scan_data_folder_resume_hint_prioritizes_next_existing_folder() {
 }
 
 #[tokio::test]
+#[serial]
 async fn scan_data_folder_missing_bucket_returns_partial() {
     let (scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard {
@@ -2492,6 +2517,7 @@ async fn scan_data_folder_missing_bucket_returns_partial() {
 }
 
 #[tokio::test]
+#[serial]
 async fn scan_data_folder_missing_scan_root_returns_partial() {
     let (scanner, temp_dir) = build_test_scanner().await;
     tokio::fs::remove_dir_all(&temp_dir)
@@ -2537,6 +2563,7 @@ async fn scan_data_folder_missing_scan_root_returns_partial() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_scan_data_folder_resume_hint_orders_across_new_and_existing_folders() {
     let (scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard {
@@ -2605,6 +2632,7 @@ async fn test_scan_data_folder_resume_hint_orders_across_new_and_existing_folder
 }
 
 #[tokio::test]
+#[serial]
 async fn test_scan_data_folder_partial_object_budget_accumulates_progress() {
     let (scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard {
@@ -2687,6 +2715,7 @@ async fn test_scan_data_folder_partial_object_budget_accumulates_progress() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_partial_compacted_entry_does_not_carry_children() {
     let (mut scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard {
@@ -2732,6 +2761,7 @@ async fn test_partial_compacted_entry_does_not_carry_children() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_partial_entry_does_not_carry_missing_old_child() {
     let (mut scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard {
@@ -2764,6 +2794,7 @@ async fn test_partial_entry_does_not_carry_missing_old_child() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_legacy_windows_cache_rebuilds_and_round_trips_portable_keys() {
     let (scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard {
@@ -2830,6 +2861,7 @@ async fn test_legacy_windows_cache_rebuilds_and_round_trips_portable_keys() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_scan_data_folder_success_clears_resume_hint() {
     let (scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard {
@@ -2872,6 +2904,7 @@ async fn test_scan_data_folder_success_clears_resume_hint() {
 }
 
 #[tokio::test]
+#[serial]
 async fn test_scan_data_folder_keeps_unresolved_objects_partial() {
     let (scanner, temp_dir) = build_test_scanner().await;
     let _guard = TestGuard {
@@ -2918,6 +2951,7 @@ async fn test_scan_data_folder_keeps_unresolved_objects_partial() {
 }
 
 #[tokio::test]
+#[serial]
 #[cfg(unix)]
 async fn test_scan_folder_ignores_symlinked_child_directory() {
     let (mut scanner, temp_dir) = build_test_scanner().await;

crates/scanner/src/scanner_io/tests.rs

@@ -27,6 +27,7 @@ use crate::{
     init_local_disks_with_instance_ctx, new_disk, path2_bucket_object_with_base_path,
 };
 use rustfs_filemeta::FileInfo;
+use serial_test::serial;
 use temp_env::with_var;
 use time::OffsetDateTime;
 use uuid::Uuid;
@@ -102,6 +103,7 @@ async fn setup_two_pool_scanner_store() -> (tempfile::TempDir, Arc<ECStore>) {
 }
 
 #[tokio::test]
+#[serial]
 async fn scanner_cache_locks_block_same_source_workers() {
     let (_temp_dir, store) = setup_two_pool_scanner_store().await;
     let set = &store.pools[0].disk_set[0];
@@ -128,6 +130,7 @@ async fn scanner_cache_locks_block_same_source_workers() {
 }
 
 #[tokio::test]
+#[serial]
 async fn scanner_cache_locks_allow_cross_source_workers() {
     let (_temp_dir, store) = setup_two_pool_scanner_store().await;
     let first_set = &store.pools[0].disk_set[0];
@@ -146,6 +149,7 @@ async fn scanner_cache_locks_allow_cross_source_workers() {
 }
 
 #[tokio::test]
+#[serial]
 async fn scanner_cycle_is_deferred_while_rebalance_is_active() {
     let (_temp_dir, store) = setup_two_pool_scanner_store().await;
     let mut pool_stats = vec![EcstoreRebalanceStats::default(); store.pools.len()];
@@ -181,6 +185,7 @@ async fn scanner_cycle_is_deferred_while_rebalance_is_active() {
 }
 
 #[tokio::test]
+#[serial]
 async fn scanner_cycle_is_deferred_while_terminal_decommission_is_blocked() {
     let (_temp_dir, store) = setup_two_pool_scanner_store().await;
     for decommission in [
@@ -225,6 +230,7 @@ async fn data_usage_publish_fails_when_receiver_is_closed() {
 }
 
 #[tokio::test]
+#[serial]
 async fn multi_pool_scanner_cycle_publishes_combined_usage() {
     let (_temp_dir, store) = setup_two_pool_scanner_store().await;
     let bucket = format!("scanner-union-{}", Uuid::new_v4().simple());
@@ -272,6 +278,7 @@ async fn multi_pool_scanner_cycle_publishes_combined_usage() {
 }
 
 #[tokio::test]
+#[serial]
 async fn multi_pool_scanner_cycle_zero_fills_bucket_absent_from_first_pool() {
     let (_temp_dir, store) = setup_two_pool_scanner_store().await;
     let bucket = format!("scanner-second-pool-{}", Uuid::new_v4().simple());
@@ -359,6 +366,7 @@ fn object_lock_config_enabled_accepts_enabled_only() {
 }
 
 #[test]
+#[serial]
 fn dirty_usage_snapshot_clear_preserves_newer_generation() {
     clear_dirty_usage_buckets_for_tests();
     record_dirty_usage_bucket("photos");
@@ -373,6 +381,7 @@ fn dirty_usage_snapshot_clear_preserves_newer_generation() {
 }
 
 #[test]
+#[serial]
 fn dirty_usage_generation_acknowledgement_preserves_newer_mutations() {
     clear_dirty_usage_buckets_for_tests();
     record_dirty_usage_bucket("photos");
@@ -398,6 +407,7 @@ fn dirty_usage_generation_acknowledgement_preserves_newer_mutations() {
 }
 
 #[test]
+#[serial]
 fn dirty_usage_generation_acknowledgement_rejects_stale_process_and_future_generation() {
     clear_dirty_usage_buckets_for_tests();
     record_dirty_usage_bucket("photos");
@@ -427,6 +437,7 @@ fn dirty_usage_generation_acknowledgement_rejects_stale_process_and_future_gener
 }
 
 #[test]
+#[serial]
 fn dirty_usage_snapshot_detects_uncovered_generation() {
     clear_dirty_usage_buckets_for_tests();
     record_dirty_usage_bucket("photos");
@@ -451,6 +462,7 @@ fn generation_saturates_instead_of_wrapping() {
 }
 
 #[test]
+#[serial]
 fn dirty_usage_snapshot_clears_a_stably_absent_bucket_after_durable_save() {
     clear_dirty_usage_buckets_for_tests();
     record_dirty_usage_bucket("photos");
@@ -472,6 +484,7 @@ fn dirty_usage_snapshot_clears_a_stably_absent_bucket_after_durable_save() {
 }
 
 #[test]
+#[serial]
 fn dirty_usage_snapshot_preserves_an_absent_bucket_recorded_after_listing_started() {
     clear_dirty_usage_buckets_for_tests();
     let generation_before_bucket_list = dirty_usage_generation();
@@ -486,6 +499,7 @@ fn dirty_usage_snapshot_preserves_an_absent_bucket_recorded_after_listing_starte
 }
 
 #[test]
+#[serial]
 fn deleting_a_clean_bucket_invalidates_an_inflight_usage_snapshot() {
     clear_dirty_usage_buckets_for_tests();
     let snapshot = snapshot_dirty_usage_buckets(&[bucket_info("photos")], dirty_usage_generation());
@@ -499,6 +513,7 @@ fn deleting_a_clean_bucket_invalidates_an_inflight_usage_snapshot() {
 }
 
 #[test]
+#[serial]
 fn deleting_a_bucket_during_listing_invalidates_the_resulting_usage_snapshot() {
     clear_dirty_usage_buckets_for_tests();
     let generation_before_bucket_list = dirty_usage_generation();
@@ -512,6 +527,7 @@ fn deleting_a_bucket_during_listing_invalidates_the_resulting_usage_snapshot() {
 }
 
 #[test]
+#[serial]
 fn scanner_maintenance_change_advances_generation_and_marks_usage_dirty() {
     clear_dirty_usage_buckets_for_tests();
     let generation = scanner_maintenance_generation();
@@ -524,6 +540,7 @@ fn scanner_maintenance_change_advances_generation_and_marks_usage_dirty() {
 }
 
 #[test]
+#[serial]
 fn dirty_usage_clear_excludes_failed_buckets() {
     clear_dirty_usage_buckets_for_tests();
     record_dirty_usage_bucket("photos");
@@ -555,6 +572,7 @@ fn dirty_usage_clear_plan_excludes_cache_save_failures() {
 }
 
 #[test]
+#[serial]
 fn dirty_usage_is_acknowledged_only_after_durable_usage_confirmation() {
     clear_dirty_usage_buckets_for_tests();
     record_dirty_usage_bucket("photos");
@@ -572,6 +590,7 @@ fn dirty_usage_is_acknowledged_only_after_durable_usage_confirmation() {
 }
 
 #[test]
+#[serial]
 fn clear_dirty_usage_bucket_removes_deleted_bucket_marker() {
     clear_dirty_usage_buckets_for_tests();
     record_dirty_usage_bucket("photos");
@@ -898,30 +917,35 @@ async fn bucket_cache_pending_heal_reaches_cycle_maintenance_state() {
 }
 
 #[test]
+#[serial]
 fn scanner_concurrency_limit_preserves_available_when_unconfigured() {
     crate::reset_foreground_read_activity_for_test();
     assert_eq!(scanner_concurrency_limit(0, 4), 4);
 }
 
 #[test]
+#[serial]
 fn scanner_concurrency_limit_caps_to_configured_value() {
     crate::reset_foreground_read_activity_for_test();
     assert_eq!(scanner_concurrency_limit(2, 4), 2);
 }
 
 #[test]
+#[serial]
 fn scanner_concurrency_limit_never_exceeds_available_work() {
     crate::reset_foreground_read_activity_for_test();
     assert_eq!(scanner_concurrency_limit(8, 4), 4);
 }
 
 #[test]
+#[serial]
 fn scanner_concurrency_limit_handles_no_available_work() {
     crate::reset_foreground_read_activity_for_test();
     assert_eq!(scanner_concurrency_limit(2, 0), 0);
 }
 
 #[test]
+#[serial]
 fn scanner_concurrency_limit_yields_to_foreground_reads() {
     crate::reset_foreground_read_activity_for_test();
     crate::set_foreground_read_activity(8);
@@ -931,6 +955,7 @@ fn scanner_concurrency_limit_yields_to_foreground_reads() {
 }
 
 #[test]
+#[serial]
 fn scanner_concurrency_limit_yields_to_streaming_reads() {
     crate::reset_foreground_read_activity_for_test();
     let _guard = crate::ForegroundReadGuard::new();
@@ -954,6 +979,7 @@ fn increment_atomic_usize_saturates_at_max() {
 }
 
 #[test]
+#[serial]
 fn scanner_max_concurrent_set_scans_uses_env_cap() {
     with_var(ENV_SCANNER_MAX_CONCURRENT_SET_SCANS, Some("2"), || {
         crate::runtime_config::refresh_scanner_runtime_config_for_tests();
@@ -963,6 +989,7 @@ fn scanner_max_concurrent_set_scans_uses_env_cap() {
 }
 
 #[test]
+#[serial]
 fn scanner_max_concurrent_disk_scans_uses_env_cap() {
     with_var(ENV_SCANNER_MAX_CONCURRENT_DISK_SCANS, Some("1"), || {
         crate::runtime_config::refresh_scanner_runtime_config_for_tests();

crates/scanner/src/sleeper.rs

@@ -258,6 +258,7 @@ impl SleepTimer {
 #[cfg(test)]
 mod tests {
     use super::*;
+    use serial_test::serial;
     use temp_env::{with_var, with_var_unset};
 
     struct ScannerDefaultSpeedGuard;
@@ -325,6 +326,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn test_refresh_from_env_applies_speed_and_idle_mode_for_next_cycle() {
         let prev_mode = SCANNER_IDLE_MODE.load(Ordering::Relaxed);
         SCANNER_IDLE_MODE.store(true, Ordering::Relaxed);
@@ -344,6 +346,7 @@ mod tests {
     }
 
     #[test]
+    #[serial]
     fn test_refresh_from_env_uses_default_speed_override_when_speed_unset() {
         let _guard = ScannerDefaultSpeedGuard::set(ScannerSpeed::Slowest);
         let s = DynamicSleeper::new(ScannerSpeed::Default);
@@ -359,6 +362,7 @@ mod tests {
     }
 
     #[tokio::test(start_paused = true)]
+    #[serial]
     async fn test_fastest_never_sleeps() {
         let prev_mode = SCANNER_IDLE_MODE.load(Ordering::Relaxed);
         SCANNER_IDLE_MODE.store(true, Ordering::Relaxed);
@@ -372,6 +376,7 @@ mod tests {
     }
 
     #[tokio::test(start_paused = true)]
+    #[serial]
     async fn test_idle_mode_off_skips_sleep() {
         let prev_mode = SCANNER_IDLE_MODE.load(Ordering::Relaxed);
         SCANNER_IDLE_MODE.store(false, Ordering::Relaxed);

crates/scanner/tests/lifecycle_integration_test.rs

@@ -14,6 +14,7 @@
 
 #![recursion_limit = "256"]
 
+use futures::FutureExt;
 use rustfs_config::ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT;
 use rustfs_scanner::scanner_folder::ScannerItem;
 use rustfs_scanner::scanner_io::ScannerIODisk;
@@ -22,8 +23,10 @@ use rustfs_scanner::{
     scanner::init_data_scanner,
 };
 use s3s::dto::RestoreRequest;
+use serial_test::serial;
 use std::{
     collections::HashMap,
+    env,
     path::{Path, PathBuf},
     sync::{Arc, Once, OnceLock},
     time::Duration,
@@ -532,15 +535,31 @@ async fn wait_for_transition(ecstore: &Arc<ECStore>, bucket: &str, object: &str,
     }
 }
 
-// Run `test_fn` with `ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT`
-// set to `"1"` for its duration. `temp_env` serializes environment mutations
-// globally, preventing data races when multiple tests run in parallel.
+// SAFETY: this helper is used only by `#[serial]` tests and runs under the single-threaded Tokio
+// runtime (`worker_threads = 1`), so no concurrent test can mutate process environment during the
+// `env::set_var` / `env::remove_var` window.
+#[allow(unsafe_code)]
 async fn with_forced_immediate_enqueue_timeout<F, Fut>(test_fn: F)
 where
     F: FnOnce() -> Fut,
     Fut: std::future::Future<Output = ()>,
 {
-    temp_env::async_with_vars([(ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT, Some("1"))], test_fn()).await;
+    let original = env::var_os(ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT);
+    unsafe {
+        env::set_var(ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT, "1");
+    }
+    let result = std::panic::AssertUnwindSafe(test_fn()).catch_unwind().await;
+    match original {
+        Some(value) => unsafe {
+            env::set_var(ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT, value);
+        },
+        None => unsafe {
+            env::remove_var(ENV_TEST_FORCE_IMMEDIATE_TRANSITION_ENQUEUE_TIMEOUT);
+        },
+    }
+    if let Err(err) = result {
+        std::panic::resume_unwind(err);
+    }
 }
 
 mod serial_tests {
@@ -573,6 +592,7 @@ mod serial_tests {
     ///     body (GET won) or a clean object/version-not-found (expiry won). A
     ///     tier-fetch failure -- the #3491 symptom -- is never tolerated.
     #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+    #[serial]
     #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-2)"]
     async fn test_expire_transitioned_object_never_races_concurrent_get() {
         let (_disk_paths, ecstore) = setup_isolated_test_env(false).await;
@@ -718,6 +738,7 @@ mod serial_tests {
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+    #[serial]
     #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial"]
     async fn rejected_transition_candidate_is_recovered_from_persisted_delete_journal() {
         let (_disk_paths, ecstore) = setup_isolated_test_env(false).await;
@@ -804,6 +825,7 @@ mod serial_tests {
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+    #[serial]
     #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial"]
     async fn cancelled_before_cleanup_store_resolution_persists_journal() {
         let (_disk_paths, ecstore) = setup_isolated_test_env(false).await;
@@ -897,6 +919,7 @@ mod serial_tests {
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+    #[serial]
     #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial"]
     async fn rejected_transition_cleanup_durability_matrix() {
         #[derive(Clone, Copy)]
@@ -1036,6 +1059,7 @@ mod serial_tests {
     }
 
     #[test]
+    #[serial]
     #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"]
     fn test_transition_and_restore_flows() {
         std::thread::Builder::new()
@@ -1361,6 +1385,7 @@ mod serial_tests {
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+    #[serial]
     #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"]
     async fn test_scanner_enqueues_free_version_cleanup_for_stale_transitioned_object() {
         let (disk_paths, ecstore) = setup_isolated_test_env(false).await;
@@ -1421,6 +1446,7 @@ mod serial_tests {
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+    #[serial]
     #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"]
     async fn test_scanner_cleanup_still_works_after_immediate_compensation_transition() {
         let (disk_paths, ecstore) = setup_isolated_test_env(false).await;
@@ -1478,6 +1504,7 @@ mod serial_tests {
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+    #[serial]
     #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"]
     async fn test_existing_object_backfill_is_idempotent_after_immediate_compensation_transition() {
         let (_disk_paths, ecstore) = setup_isolated_test_env(false).await;
@@ -1520,6 +1547,7 @@ mod serial_tests {
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+    #[serial]
     #[ignore = "FAILING on main: excluded from the serial ILM lane pending a fix, see rustfs/backlog#1148 (ilm-1 partial)"]
     async fn test_noncurrent_expiry_still_works_after_immediate_compensation_transition() {
         let (disk_paths, ecstore) = setup_isolated_test_env(true).await;
@@ -1603,6 +1631,7 @@ mod serial_tests {
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+    #[serial]
     #[ignore = "FAILING on main: excluded from the serial ILM lane pending a fix, see rustfs/backlog#1148 (ilm-1 partial)"]
     async fn test_noncurrent_transition_still_works_after_immediate_compensation_transition() {
         let (disk_paths, ecstore) = setup_isolated_test_env(true).await;
@@ -1685,6 +1714,7 @@ mod serial_tests {
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+    #[serial]
     #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"]
     async fn test_modeled_versioned_delete_creates_delete_marker_after_immediate_compensation_transition() {
         let (_disk_paths, ecstore) = setup_isolated_test_env(true).await;
@@ -1732,6 +1762,7 @@ mod serial_tests {
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+    #[serial]
     #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"]
     async fn test_modeled_delete_marker_cleanup_after_immediate_compensation_transition() {
         let (disk_paths, ecstore) = setup_isolated_test_env(true).await;
@@ -1808,6 +1839,7 @@ mod serial_tests {
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+    #[serial]
     #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"]
     async fn test_scanner_expires_zero_day_current_version() {
         let (disk_paths, ecstore) = setup_isolated_test_env(false).await;
@@ -1834,6 +1866,7 @@ mod serial_tests {
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+    #[serial]
     #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"]
     async fn test_put_object_immediately_enqueues_zero_day_current_expiry() {
         let (_disk_paths, ecstore) = setup_isolated_test_env(true).await;
@@ -1871,6 +1904,7 @@ mod serial_tests {
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+    #[serial]
     #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"]
     async fn test_scanner_expires_zero_day_noncurrent_version() {
         let (disk_paths, ecstore) = setup_isolated_test_env(false).await;
@@ -1937,6 +1971,7 @@ mod serial_tests {
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+    #[serial]
     #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"]
     async fn test_put_object_immediately_enqueues_zero_day_noncurrent_expiry() {
         let (_disk_paths, ecstore) = setup_isolated_test_env(true).await;
@@ -1997,6 +2032,7 @@ mod serial_tests {
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+    #[serial]
     async fn test_background_scanner_expires_zero_day_current_version() {
         let (_disk_paths, ecstore) = setup_isolated_test_env(true).await;
 
@@ -2020,6 +2056,7 @@ mod serial_tests {
     }
 
     #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+    #[serial]
     #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-1)"]
     async fn test_background_scanner_expires_zero_day_current_version_for_exact_key_prefix() {
         let (_disk_paths, ecstore) = setup_isolated_test_env(true).await;
@@ -2085,6 +2122,7 @@ mod serial_tests {
     /// tier object is untouched (zero `remove` calls) -> GET streams from the
     /// tier again -> a second restore succeeds.
     #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+    #[serial]
     #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-8)"]
     async fn test_restore_chain_local_read_expiry_keeps_remote_and_allows_re_restore() {
         let (_disk_paths, ecstore) = setup_test_env().await;
@@ -2216,6 +2254,7 @@ mod serial_tests {
     /// parts) must reassemble the exact part layout: part count and sizes,
     /// the multipart ETag, and byte-identical content across part boundaries.
     #[tokio::test(flavor = "multi_thread", worker_threads = 1)]
+    #[serial]
     #[ignore = "global-state ILM integration test: runs serialized in the CI ILM Integration (serial) lane, see ci.yml test-ilm-integration-serial and rustfs/backlog#1148 (ilm-8)"]
     async fn test_multipart_restore_preserves_parts_and_etag() {
         let (_disk_paths, ecstore) = setup_test_env().await;

crates/targets/AGENTS.md

@@ -75,3 +75,4 @@ run commands.
   - `cargo test -p rustfs-targets plugin`
   - `cargo test -p rustfs-targets runtime`
   - `cargo test -p rustfs-targets control_plane`
+- Full gate before commit: `make pre-commit`

protocol/agent/v1/fixtures/auth/MANIFEST.sha256

@@ -1,6 +1,5 @@
 3d602080f7ca4c32ba9e37ad1a32665c78560726b30aeee08fd9e95eb2f36194  accept-vectors.json
 d3c19946288717088145592e0e8d6f2fa684443ba2f73d4c7bc49c415d6dd051  certificate-profile.json
-299a2ae34a8ca74bcf31deeb53a08f9eff279efa09d3f5358a0cf10866fe1a5d  error-codes.json
+060485263c51003274c056a0e04bec1b7d76157cf599ba79eebe040bc7cee71b  error-codes.json
 43fe297ffb512b1b9f4af62f1832f3aa3905157893bfdc3dcc6d56f5a98aaef6  reject-vectors.json
-0cf26a7332fa6e3f57390e081f2cceead3236f6ca57f7038e3b55c2582af1733  rotation-proof.json
-7c07100460fa23fca482466df9ca22f91c7f26b36087a7207a001f7d987f527e  surface-separation.json
+b946175b094f4a8d75091b652fbe3d4327c9c795f28e02c96e1ab90a429e418d  surface-separation.json

protocol/agent/v1/fixtures/auth/error-codes.json

@@ -2,7 +2,7 @@
   "protocolVersion": "v1",
   "fixtureSet": "auth",
   "fixture": "error-codes",
-  "description": "Frozen ErrorInfo reasons for agent authentication, negotiation, and credential-rotation authorization. Clients branch on status and reason, never on message.",
+  "description": "Frozen ErrorInfo reasons for agent authentication and negotiation. Clients branch on status and reason, never on message.",
   "domain": "rustfs.connect",
   "detailType": "type.googleapis.com/google.rpc.ErrorInfo",
   "disclosureRules": [
@@ -81,24 +81,6 @@
       "httpStatus": 401,
       "status": "UNAUTHENTICATED",
       "meaning": "A browser session cookie was presented to an authenticated agent operation. The agent surface never accepts it."
-    },
-    {
-      "reason": "ROTATION_CREDENTIAL_NOT_CURRENT",
-      "httpStatus": 409,
-      "status": "ABORTED",
-      "meaning": "The authenticated certificate is valid for ordinary agent operations but is not the device's current ACTIVE credential and therefore cannot authorize another rotation."
-    },
-    {
-      "reason": "ROTATION_REQUEST_CONFLICT",
-      "httpStatus": 409,
-      "status": "ABORTED",
-      "meaning": "The requestId already belongs to a rotation with different transcript inputs and cannot be reused."
-    },
-    {
-      "reason": "ROTATION_PROOF_INVALID",
-      "httpStatus": 401,
-      "status": "UNAUTHENTICATED",
-      "meaning": "The credential proof does not verify over the canonical rotation transcript under the presented certificate key."
     }
   ]
 }

protocol/agent/v1/fixtures/auth/rotation-proof.json

@@ -1,306 +0,0 @@
-{
-  "protocolVersion": "v1",
-  "fixtureSet": "auth",
-  "fixture": "rotation-proof",
-  "description": "Frozen proof-of-possession transcript for rotating an online device credential. The current certificate key authorizes one new certificate request for one device and one idempotent request.",
-  "operation": {
-    "method": "POST",
-    "path": "/agent/clusterDevices/{device}:rotateCredential",
-    "operationId": "rotateClusterDeviceCredential",
-    "authenticatedSurface": "/agent/*",
-    "currentCredentialRequiredForNewRotation": true,
-    "outgoingOverlapCredentialMayAuthenticateOrdinaryOperations": true,
-    "outgoingOverlapCredentialMayRotate": false,
-    "outgoingOverlapCredentialMayReplayItsCompletedRotation": true
-  },
-  "replayPolicy": {
-    "recordState": "COMPLETED",
-    "bindingFields": [
-      "currentCertificateFingerprint",
-      "clusterDeviceName",
-      "requestId",
-      "certificateRequestSha256"
-    ],
-    "exactMatch": "returnStoredResult",
-    "mismatchReason": "ROTATION_REQUEST_CONFLICT",
-    "retentionLowerBound": "outgoingCredential.validUntil",
-    "sideEffects": "No issuer call, credential write, or overlap extension."
-  },
-  "completedReplayRecord": {
-    "state": "COMPLETED",
-    "currentCertificateFingerprint": "1bc816e2d285c52ee3a0aa06dc7aec17e788626043e795259a67f2436f970d09",
-    "clusterDeviceName": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/clusters/0198f4b0-2b00-7d20-9e31-3f4a5b6c7d81/clusterDevices/0198f4b0-3c00-7e30-8f41-4a5b6c7d8e92",
-    "requestId": "7c5d6e2a-8b91-4f03-a4d5-6e7f8091a2b3",
-    "certificateRequestSha256": "kVS-bXYxD6F22cZNy4Vnpgb5gFZbr4EGP5GvIz7uOaw",
-    "resultReference": "credential-rotation-result-01"
-  },
-  "transcript": {
-    "domain": "RUSTFS-CONNECT-CREDENTIAL-ROTATION-V1",
-    "domainTerminator": "0x0a",
-    "fieldSeparator": "0x3a",
-    "fieldTerminator": "0x0a",
-    "fieldCount": 4,
-    "encoding": "US-ASCII",
-    "normalisationPermitted": false,
-    "fieldOrder": [
-      "currentCertificateFingerprint",
-      "clusterDeviceName",
-      "requestId",
-      "certificateRequestSha256"
-    ],
-    "fields": [
-      {
-        "name": "currentCertificateFingerprint",
-        "position": 1,
-        "source": "the exact certificate accepted by trusted ingress and resolved by Connect",
-        "pattern": "^[0-9a-f]{64}$",
-        "binds": "the one currently active credential authorizing the rotation",
-        "absenceWouldAllow": "A proof captured from one current certificate to authorize another credential after the device rotated."
-      },
-      {
-        "name": "clusterDeviceName",
-        "position": 2,
-        "source": "the authenticated identity after it is matched to the target resource",
-        "pattern": "^organizations/[0-9a-f-]{36}/clusters/[0-9a-f-]{36}/clusterDevices/[0-9a-f-]{36}$",
-        "binds": "the organization, cluster, and device being rotated",
-        "absenceWouldAllow": "A proof produced by one device to be presented against another device resource."
-      },
-      {
-        "name": "requestId",
-        "position": 3,
-        "source": "the request body",
-        "pattern": "^[0-9a-f]{8}-[0-9a-f]{4}-4[0-9a-f]{3}-[89ab][0-9a-f]{3}-[0-9a-f]{12}$",
-        "binds": "the single idempotent rotation attempt",
-        "absenceWouldAllow": "A captured proof to be replayed as a fresh attempt instead of the same idempotent request."
-      },
-      {
-        "name": "certificateRequestSha256",
-        "position": 4,
-        "source": "recomputed over the exact PKCS#10 DER octets in certificateRequest",
-        "pattern": "^[A-Za-z0-9_-]{43}$",
-        "binds": "the next device public key and certificate request",
-        "absenceWouldAllow": "A valid proof to be combined with an attacker certificate request."
-      }
-    ]
-  },
-  "signature": {
-    "algorithmField": "proof.algorithm",
-    "algorithmEnumeration": ["ES256"],
-    "curve": "P-256",
-    "signatureEncoding": "fixed-width-r-s",
-    "signatureLengthBytes": 64,
-    "signatureTransferEncoding": "base64url-unpadded",
-    "signatureValuePattern": "^[A-Za-z0-9_-]{86}$",
-    "lowSRequired": true,
-    "verifyingKeySource": "the SubjectPublicKeyInfo of the exact presented certificate accepted by trusted ingress",
-    "newKeyPossession": "the PKCS#10 self-signature is verified separately under the public key inside certificateRequest",
-    "sharedEncodingContract": "protocol/agent/v1/registration-proof.md#the-signature"
-  },
-  "verificationOrder": [
-    {
-      "stage": "protocolVersion",
-      "rule": "protocolVersion must be the supported major version.",
-      "reason": "UNSUPPORTED_PROTOCOL"
-    },
-    {
-      "stage": "encoding",
-      "rule": "proof.algorithm and proof.value obey the shared ES256 fixed-width low-S contract.",
-      "reasons": ["UNSUPPORTED_ALGORITHM", "SIGNATURE_MALFORMED", "SIGNATURE_NOT_CANONICAL"]
-    },
-    {
-      "stage": "certificateRequest",
-      "rule": "certificateRequest is one self-signed PKCS#10 request whose public key is P-256.",
-      "reasons": ["CERTIFICATE_REQUEST_MALFORMED", "DEVICE_KEY_UNSUPPORTED"]
-    },
-    {
-      "stage": "proof",
-      "rule": "Connect rebuilds the transcript and verifies proof.value under the presented certificate public key.",
-      "reason": "ROTATION_PROOF_INVALID"
-    },
-    {
-      "stage": "replay",
-      "rule": "A completed record bound to the presented credential, device, requestId, and CSR digest returns its stored result without issuing again. A requestId bound to different transcript inputs is refused.",
-      "reason": "ROTATION_REQUEST_CONFLICT"
-    },
-    {
-      "stage": "credential",
-      "rule": "After replay lookup misses, the authenticated certificate must be the device current ACTIVE credential, not a credential in the outgoing overlap.",
-      "reason": "ROTATION_CREDENTIAL_NOT_CURRENT"
-    }
-  ],
-  "reasonSources": {
-    "auth": ["UNSUPPORTED_PROTOCOL", "ROTATION_CREDENTIAL_NOT_CURRENT", "ROTATION_REQUEST_CONFLICT", "ROTATION_PROOF_INVALID"],
-    "sharedFromRegistration": [
-      "UNSUPPORTED_ALGORITHM",
-      "SIGNATURE_MALFORMED",
-      "SIGNATURE_NOT_CANONICAL",
-      "CERTIFICATE_REQUEST_MALFORMED",
-      "DEVICE_KEY_UNSUPPORTED"
-    ]
-  },
-  "example": {
-    "inputs": {
-      "currentCertificateFingerprint": "1bc816e2d285c52ee3a0aa06dc7aec17e788626043e795259a67f2436f970d09",
-      "clusterDeviceName": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/clusters/0198f4b0-2b00-7d20-9e31-3f4a5b6c7d81/clusterDevices/0198f4b0-3c00-7e30-8f41-4a5b6c7d8e92",
-      "requestId": "7c5d6e2a-8b91-4f03-a4d5-6e7f8091a2b3",
-      "certificateRequestSha256": "kVS-bXYxD6F22cZNy4Vnpgb5gFZbr4EGP5GvIz7uOaw"
-    },
-    "artifacts": {
-      "currentPublicKeySpki": "MFkwEwYHKoZIzj0CAQYIKoZIzj0DAQcDQgAENEpjuFZTqp0Hxh/OWV3TGkluNjCo15dk+4CozuR6aT9Vaxhkb2M9nhaVGfk8+aSiSIiKFSCsYonKl8jh743Qow==",
-      "currentPublicKeyFingerprint": "28608e223c75ed89e72041f12afae4fc1cd3f1d3bcca38cbb62d1a41687610c2",
-      "certificateRequest": "MIIBHjCBxAIBADBiMRswGQYDVQQDDBJpZ25vcmVkLWJ5LWNvbm5lY3QxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAAQWPosvIKr5V3CpTUdLfMQxDx31B3SglLKyqRg/oH3J+PhUnqf7pDWW1sTkP2aRIDjAwRn0DpLrz405CcvGHvYEoAAwCgYIKoZIzj0EAwIDSQAwRgIhAMJzXo/CK4E9BfjOxP35he9LLlqENhK7HTzZQTuIgLX2AiEAwOZHibk5HEijTWcJ/UT117nssfJesWZVOWwz/KTIpi8=",
-      "otherCertificateRequest": "MIIBHTCBxAIBADBiMRswGQYDVQQDDBJpZ25vcmVkLWJ5LWNvbm5lY3QxCzAJBgNVBAYTAkFVMRMwEQYDVQQIDApTb21lLVN0YXRlMSEwHwYDVQQKDBhJbnRlcm5ldCBXaWRnaXRzIFB0eSBMdGQwWTATBgcqhkjOPQIBBggqhkjOPQMBBwNCAATi0KiZGcbPJixDiQN/aIVky7xqaYhjmwAn1aozrC2eanaDENYQ3D5zw5qoZNIp8YUk/CETZrba3C5KYR5ocdlLoAAwCgYIKoZIzj0EAwIDSAAwRQIhAL0ZtvWyTSu18RF5J4ZVIuOGjJpJwSdP+87CVxNKJJduAiAahIruc3FLtO3RI7B8Ome8IsVDUpSAQilThjdOeDMsYg=="
-    },
-    "canonicalTranscript": "RUSTFS-CONNECT-CREDENTIAL-ROTATION-V1\n64:1bc816e2d285c52ee3a0aa06dc7aec17e788626043e795259a67f2436f970d09\n148:organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/clusters/0198f4b0-2b00-7d20-9e31-3f4a5b6c7d81/clusterDevices/0198f4b0-3c00-7e30-8f41-4a5b6c7d8e92\n36:7c5d6e2a-8b91-4f03-a4d5-6e7f8091a2b3\n43:kVS-bXYxD6F22cZNy4Vnpgb5gFZbr4EGP5GvIz7uOaw\n",
-    "canonicalTranscriptLengthBytes": 346,
-    "canonicalTranscriptSha256": "e5f0e9cd0d5d420bc7e51217512de76e1cf4dceb1d67804b6518c4fb5d9fe434"
-  },
-  "acceptVectors": [
-    {
-      "name": "current credential authorizes one new certificate request",
-      "requestId": "7c5d6e2a-8b91-4f03-a4d5-6e7f8091a2b3",
-      "proof": "vgz7Hm-xZ5JQ7B3oqUEKXB4tQxEzWZ988iPxAklaIZIsd4nSTUv5YriqES5yKGHvKOgWPyWjvrgI0j0T-fYEsQ",
-      "expected": {"accepted": true, "reason": null, "stage": "credential", "replayed": false}
-    },
-    {
-      "name": "the same certificate request under a new requestId needs its own proof",
-      "requestId": "82e1f3a4-9b05-4c67-8d90-1e2f3a4b5c6d",
-      "proof": "s-rkF9w41sqgdQNkOlhjcLYlrNHN_HL5xh4C3k_1a9ZQu7Uq0LoET_x7i4Rw3xQ75ec6B6tfqZ3XsHTS0FfN7Q",
-      "expected": {"accepted": true, "reason": null, "stage": "credential", "replayed": false}
-    },
-    {
-      "name": "outgoing credential replays its completed byte-equivalent rotation",
-      "requestId": "7c5d6e2a-8b91-4f03-a4d5-6e7f8091a2b3",
-      "proof": "vgz7Hm-xZ5JQ7B3oqUEKXB4tQxEzWZ988iPxAklaIZIsd4nSTUv5YriqES5yKGHvKOgWPyWjvrgI0j0T-fYEsQ",
-      "replayRecord": "completedExample",
-      "mutation": {"currentCredential": false},
-      "expected": {"accepted": true, "reason": null, "stage": "replay", "replayedResultReference": "credential-rotation-result-01"}
-    }
-  ],
-  "rejectVectors": [
-    {
-      "name": "outgoing overlap credential attempts another rotation",
-      "stage": "credential",
-      "requestId": "82e1f3a4-9b05-4c67-8d90-1e2f3a4b5c6d",
-      "proof": "s-rkF9w41sqgdQNkOlhjcLYlrNHN_HL5xh4C3k_1a9ZQu7Uq0LoET_x7i4Rw3xQ75ec6B6tfqZ3XsHTS0FfN7Q",
-      "mutation": {"currentCredential": false},
-      "expected": {"accepted": false, "reason": "ROTATION_CREDENTIAL_NOT_CURRENT"}
-    },
-    {
-      "name": "rotation declares protocol v2",
-      "stage": "protocolVersion",
-      "mutation": {"protocolVersion": "v2"},
-      "expected": {"accepted": false, "reason": "UNSUPPORTED_PROTOCOL"}
-    },
-    {
-      "name": "proof declares ES384",
-      "stage": "encoding",
-      "mutation": {"proofAlgorithm": "ES384"},
-      "expected": {"accepted": false, "reason": "UNSUPPORTED_ALGORITHM"}
-    },
-    {
-      "name": "DER encoded proof",
-      "stage": "encoding",
-      "mutation": {"proofValue": "MEUCIQC-DPseb7FnklDsHeipQQpcHi1DETNZn3zyI_ECSVohkgIgLHeJ0k1L-WK4qhEucihh7yjoFj8lo764CNI9E_n2BLE"},
-      "expected": {"accepted": false, "reason": "SIGNATURE_MALFORMED"}
-    },
-    {
-      "name": "padded base64url proof",
-      "stage": "encoding",
-      "mutation": {"proofValue": "vgz7Hm-xZ5JQ7B3oqUEKXB4tQxEzWZ988iPxAklaIZIsd4nSTUv5YriqES5yKGHvKOgWPyWjvrgI0j0T-fYEsQ=="},
-      "expected": {"accepted": false, "reason": "SIGNATURE_MALFORMED"}
-    },
-    {
-      "name": "protocol negotiation precedes proof encoding and stale credential state",
-      "stage": "protocolVersion",
-      "mutation": {"currentCredential": false, "protocolVersion": "v2", "proofValue": "not-a-signature"},
-      "expected": {"accepted": false, "reason": "UNSUPPORTED_PROTOCOL"}
-    },
-    {
-      "name": "declared algorithm precedes signature bytes",
-      "stage": "encoding",
-      "mutation": {"proofAlgorithm": "ES384", "proofValue": "not-a-signature"},
-      "expected": {"accepted": false, "reason": "UNSUPPORTED_ALGORITHM"}
-    },
-    {
-      "name": "malleated high-S proof",
-      "stage": "encoding",
-      "mutation": {"proofValue": "vgz7Hm-xZ5JQ7B3oqUEKXB4tQxEzWZ988iPxAklaIZLTiHYssrQGnkdV7tGN154Qk_7kboFz38zq542vAm0goA"},
-      "expected": {"accepted": false, "reason": "SIGNATURE_NOT_CANONICAL"}
-    },
-    {
-      "name": "proof over the rotation fields uses the registration domain",
-      "stage": "proof",
-      "mutation": {"proofValue": "as27GKXEKXtym-BU8NUl0BYhJkEUooZWPadrxwOWa40pAaN6p4VxKCCFggZl4ZQsl5CUtaMoMaxM_HzT1c1qUA"},
-      "expected": {"accepted": false, "reason": "ROTATION_PROOF_INVALID"}
-    },
-    {
-      "name": "proof verification precedes stale credential state",
-      "stage": "proof",
-      "mutation": {"currentCredential": false, "proofValue": "as27GKXEKXtym-BU8NUl0BYhJkEUooZWPadrxwOWa40pAaN6p4VxKCCFggZl4ZQsl5CUtaMoMaxM_HzT1c1qUA"},
-      "expected": {"accepted": false, "reason": "ROTATION_PROOF_INVALID"}
-    },
-    {
-      "name": "new certificate request key signs instead of the current credential key",
-      "stage": "proof",
-      "mutation": {"proofValue": "QEN8wMQQk5HjriDw2aeAxIfsRvrWgIjGR3KlpK81Y34kaTAacO-6_bOMAjQ0hMMIk-YYN6MDCw0Via_Ri0X0BA"},
-      "expected": {"accepted": false, "reason": "ROTATION_PROOF_INVALID"}
-    },
-    {
-      "name": "proof is moved to another current certificate",
-      "stage": "proof",
-      "mutation": {"currentCertificateFingerprint": "2bc816e2d285c52ee3a0aa06dc7aec17e788626043e795259a67f2436f970d09"},
-      "expected": {"accepted": false, "reason": "ROTATION_PROOF_INVALID"}
-    },
-    {
-      "name": "proof is moved to another device",
-      "stage": "proof",
-      "mutation": {"clusterDeviceName": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/clusters/0198f4b0-2b00-7d20-9e31-3f4a5b6c7d81/clusterDevices/0198f4b0-3c00-7e30-8f41-4a5b6c7d8e93"},
-      "expected": {"accepted": false, "reason": "ROTATION_PROOF_INVALID"}
-    },
-    {
-      "name": "proof is replayed under another requestId",
-      "stage": "proof",
-      "mutation": {"requestId": "82e1f3a4-9b05-4c67-8d90-1e2f3a4b5c6d"},
-      "expected": {"accepted": false, "reason": "ROTATION_PROOF_INVALID"}
-    },
-    {
-      "name": "proof is combined with another certificate request",
-      "stage": "proof",
-      "mutation": {"certificateRequest": "otherCertificateRequest"},
-      "expected": {"accepted": false, "reason": "ROTATION_PROOF_INVALID"}
-    },
-    {
-      "name": "completed replay belongs to another certificate fingerprint",
-      "stage": "replay",
-      "replayRecord": "completedExample",
-      "replayRecordMutation": {"currentCertificateFingerprint": "2bc816e2d285c52ee3a0aa06dc7aec17e788626043e795259a67f2436f970d09"},
-      "mutation": {"currentCredential": false},
-      "expected": {"accepted": false, "reason": "ROTATION_REQUEST_CONFLICT"}
-    },
-    {
-      "name": "completed replay belongs to another device",
-      "stage": "replay",
-      "replayRecord": "completedExample",
-      "replayRecordMutation": {"clusterDeviceName": "organizations/0198f4b0-1a00-7c10-8d21-2e3f4a5b6c70/clusters/0198f4b0-2b00-7d20-9e31-3f4a5b6c7d81/clusterDevices/0198f4b0-3c00-7e30-8f41-4a5b6c7d8e93"},
-      "mutation": {"currentCredential": false},
-      "expected": {"accepted": false, "reason": "ROTATION_REQUEST_CONFLICT"}
-    },
-    {
-      "name": "completed replay belongs to another requestId",
-      "stage": "replay",
-      "replayRecord": "completedExample",
-      "replayRecordMutation": {"requestId": "82e1f3a4-9b05-4c67-8d90-1e2f3a4b5c6d"},
-      "mutation": {"currentCredential": false},
-      "expected": {"accepted": false, "reason": "ROTATION_REQUEST_CONFLICT"}
-    },
-    {
-      "name": "completed replay belongs to another CSR digest",
-      "stage": "replay",
-      "replayRecord": "completedExample",
-      "replayRecordMutation": {"certificateRequestSha256": "PXxfNGWtIbrqVocag6lOLDDR41AjpHXJ7UcAWkbTSAs"},
-      "mutation": {"currentCredential": false},
-      "expected": {"accepted": false, "reason": "ROTATION_REQUEST_CONFLICT"}
-    }
-  ]
-}

protocol/agent/v1/fixtures/auth/surface-separation.json

@@ -49,7 +49,7 @@
       "schemeType": "mutualTLS",
       "forbiddenSecuritySchemes": ["sessionCookie"],
       "defaultSecurity": ["agentMutualTls"],
-      "publicOperations": ["getProtocolStatus", "exchangeRegistrationToken"]
+      "publicOperations": ["getProtocolStatus"]
     },
     {
       "document": "openapi/control.json",

protocol/agent/v1/fixtures/fixture-sets.json

@@ -11,7 +11,7 @@
     {
       "name": "auth",
       "status": "populated",
-      "purpose": "Client certificate profile, RFC 9440 header profile, authentication and credential-rotation proof vectors, surface separation, and the frozen error reason registry."
+      "purpose": "Client certificate profile, RFC 9440 header profile, authentication accept and reject vectors, surface separation, and the frozen error reason registry."
     },
     {
       "name": "version",

rustfs/Cargo.toml

@@ -288,7 +288,6 @@ zstd.workspace = true
 # Cryptography and Security
 rustls = { workspace = true, default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
 rustls-pki-types = { workspace = true }
-x509-parser = { workspace = true }
 subtle = { workspace = true }
 jiff = { workspace = true, features = ["serde"] }
 time = { workspace = true, features = ["parsing", "formatting", "serde", "macros"] }

rustfs/src/admin/AGENTS.md

@@ -25,3 +25,4 @@ Applies to `rustfs/src/admin/`.
 ## Suggested Validation
 
 - Admin handler and routing tests under `rustfs/src/admin/`
+- Full gate before commit: `make pre-commit`

rustfs/src/admin/handlers/mod.rs

@@ -126,6 +126,7 @@ mod tests {
         let _list_remote_target_handler = replication::ListRemoteTargetHandler {};
         let _remove_remote_target_handler = replication::RemoveRemoteTargetHandler {};
         let _scanner_status_handler = scanner::ScannerStatusHandler {};
+        let _scanner_cycle_state_reset_handler = scanner::ScannerCycleStateResetHandler {};
         let _ilm_expiry_status_handler = scanner::IlmExpiryStatusHandler {};
         let _manual_transition_handler = ilm_transition::ManualTransitionRunHandler {};
         let _manual_transition_status_handler = ilm_transition::ManualTransitionJobStatusHandler {};

rustfs/src/admin/handlers/scanner.rs

@@ -13,8 +13,11 @@
 // limitations under the License.
 
 use crate::admin::auth::authorize_admin_request;
+use crate::admin::handlers::supervise_admin_mutation;
 use crate::admin::router::{AdminOperation, Operation, S3Router};
-use crate::admin::runtime_sources::current_scanner_metrics_report;
+use crate::admin::runtime_sources::{
+    app_context_from_req, current_object_store_handle_for_context, current_scanner_metrics_report,
+};
 use crate::module_switches::{ENV_SCANNER_ENABLED, scanner_enabled_from_env};
 use crate::server::ADMIN_PREFIX;
 use chrono::Utc;
@@ -22,11 +25,13 @@ use http::{HeaderMap, HeaderValue};
 use hyper::{Method, StatusCode};
 use matchit::Params;
 use rustfs_common::metrics::{ScannerLifecycleExpirySnapshot, ScannerMaintenanceControlSnapshot, ScannerMetricsReport};
+use rustfs_config::MAX_ADMIN_REQUEST_BODY_SIZE;
 use rustfs_credentials::Credentials;
 use rustfs_policy::policy::action::{Action, AdminAction};
 use s3s::header::CONTENT_TYPE;
 use s3s::{Body, S3Error, S3ErrorCode, S3Request, S3Response, S3Result, s3_error};
-use serde::Serialize;
+use serde::{Deserialize, Serialize};
+use tokio_util::sync::CancellationToken;
 
 const JSON_CONTENT_TYPE: &str = "application/json";
 
@@ -38,6 +43,13 @@ struct ScannerStatusResponse {
     metrics: ScannerMetricsReport,
     cycle_schedule: rustfs_scanner::ScannerCycleScheduleStatus,
     runtime_config: rustfs_scanner::runtime_config::ScannerRuntimeConfigStatus,
+    cycle_recovery: rustfs_scanner::ScannerCycleRecoveryStatus,
+}
+
+#[derive(Debug, Deserialize)]
+#[serde(deny_unknown_fields)]
+struct ScannerCycleResetRequest {
+    mode: String,
 }
 
 #[derive(Debug, Serialize)]
@@ -117,6 +129,7 @@ fn scanner_status_response(
         metrics,
         cycle_schedule,
         runtime_config,
+        cycle_recovery: rustfs_scanner::scanner::scanner_cycle_recovery_status(),
     }
 }
 
@@ -144,6 +157,11 @@ pub fn register_scanner_route(r: &mut S3Router<AdminOperation>) -> std::io::Resu
         format!("{ADMIN_PREFIX}/v3/scanner/status").as_str(),
         AdminOperation(&ScannerStatusHandler {}),
     )?;
+    r.insert(
+        Method::POST,
+        format!("{ADMIN_PREFIX}/v3/scanner/cycle-state/reset").as_str(),
+        AdminOperation(&ScannerCycleStateResetHandler {}),
+    )?;
     r.insert(
         Method::GET,
         format!("{ADMIN_PREFIX}/v3/ilm/expiry/status").as_str(),
@@ -163,6 +181,13 @@ async fn validate_scanner_status_request(req: &S3Request<Body>) -> S3Result<Cred
     authorize_admin_request(req, vec![Action::AdminAction(AdminAction::ServerInfoAdminAction)]).await
 }
 
+async fn validate_scanner_reset_request(req: &S3Request<Body>) -> S3Result<Credentials> {
+    if req.credentials.is_none() {
+        return Err(s3_error!(InvalidRequest, "missing credentials"));
+    }
+    authorize_admin_request(req, vec![Action::AdminAction(AdminAction::ConfigUpdateAdminAction)]).await
+}
+
 fn json_response(body: Vec<u8>) -> S3Result<S3Response<(StatusCode, Body)>> {
     let mut headers = HeaderMap::new();
     let content_type = HeaderValue::from_str(JSON_CONTENT_TYPE)
@@ -192,6 +217,37 @@ impl Operation for ScannerStatusHandler {
 
 pub struct IlmExpiryStatusHandler {}
 
+pub struct ScannerCycleStateResetHandler {}
+
+#[async_trait::async_trait]
+impl Operation for ScannerCycleStateResetHandler {
+    async fn call(&self, mut req: S3Request<Body>, _params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
+        let _cred = validate_scanner_reset_request(&req).await?;
+        let body = req
+            .input
+            .store_all_limited(MAX_ADMIN_REQUEST_BODY_SIZE)
+            .await
+            .map_err(|err| S3Error::with_message(S3ErrorCode::InvalidRequest, format!("invalid reset request body: {err}")))?;
+        let reset = serde_json::from_slice::<ScannerCycleResetRequest>(&body)
+            .map_err(|err| S3Error::with_message(S3ErrorCode::InvalidRequest, format!("invalid reset request body: {err}")))?;
+        if reset.mode != "full-rescan" {
+            return Err(S3Error::with_message(S3ErrorCode::InvalidRequest, "reset mode must be full-rescan"));
+        }
+        let context = app_context_from_req(&req)
+            .ok_or_else(|| S3Error::with_message(S3ErrorCode::InternalError, "storage layer not initialized"))?;
+        let store = current_object_store_handle_for_context(Some(context.as_ref()))
+            .ok_or_else(|| S3Error::with_message(S3ErrorCode::InternalError, "storage layer not initialized"))?;
+        supervise_admin_mutation("scanner cycle state reset", async move {
+            rustfs_scanner::scanner::reset_scanner_cycle_recovery(CancellationToken::new(), store)
+                .await
+                .map_err(|err| S3Error::with_message(S3ErrorCode::InternalError, err.to_string()))?;
+            Ok::<_, S3Error>(())
+        })
+        .await?;
+        json_response(br#"{"status":"reset","mode":"full-rescan"}"#.to_vec())
+    }
+}
+
 #[async_trait::async_trait]
 impl Operation for IlmExpiryStatusHandler {
     async fn call(&self, req: S3Request<Body>, _params: Params<'_, '_>) -> S3Result<S3Response<(StatusCode, Body)>> {
@@ -237,6 +293,38 @@ mod tests {
         assert_eq!(err.message(), Some("missing credentials"));
     }
 
+    #[tokio::test]
+    async fn scanner_reset_gate_rejects_missing_credentials() {
+        let req = S3Request {
+            input: Body::from(String::new()),
+            method: Method::POST,
+            uri: http::Uri::from_static("/rustfs/admin/v3/scanner/cycle-state/reset"),
+            headers: HeaderMap::new(),
+            extensions: http::Extensions::new(),
+            credentials: None,
+            region: None,
+            service: None,
+            trailing_headers: None,
+        };
+
+        let err = validate_scanner_reset_request(&req)
+            .await
+            .expect_err("a reset request without credentials must be rejected");
+        assert_eq!(err.code(), &S3ErrorCode::InvalidRequest);
+        assert_eq!(err.message(), Some("missing credentials"));
+    }
+
+    #[test]
+    fn admin_reset_requires_full_rescan_or_verified_cursor() {
+        let full_rescan: ScannerCycleResetRequest =
+            serde_json::from_str(r#"{"mode":"full-rescan"}"#).expect("full rescan must be accepted");
+        assert_eq!(full_rescan.mode, "full-rescan");
+        let cursor: ScannerCycleResetRequest =
+            serde_json::from_str(r#"{"mode":"cursor"}"#).expect("mode validation belongs to the handler");
+        assert_ne!(cursor.mode, "full-rescan");
+        assert!(serde_json::from_str::<ScannerCycleResetRequest>(r#"{"mode":"full-rescan","cursor":"untrusted"}"#).is_err());
+    }
+
     #[test]
     fn scanner_disabled_reason_reports_startup_env_key() {
         assert_eq!(scanner_disabled_reason(true), None);
@@ -304,6 +392,11 @@ mod tests {
         assert_eq!(encoded["cycle_schedule"]["effective_interval_seconds"], 0);
         assert_eq!(encoded["cycle_schedule"]["clean_idle_backoff_enabled"], false);
         assert_eq!(encoded["cycle_schedule"]["clean_idle_backoff_multiplier"], 1);
+        assert_eq!(encoded["cycle_recovery"]["state"], "healthy");
+        assert_eq!(
+            encoded["cycle_recovery"]["quarantine_path"],
+            rustfs_scanner::DATA_USAGE_BLOOM_RECOVERY_PATH.as_str()
+        );
     }
 
     #[test]

rustfs/src/admin/route_policy.rs

@@ -428,6 +428,12 @@ pub const ADMIN_ROUTE_POLICY_SPECS: &[AdminRouteSpec] = &[
     admin(HttpMethod::Get, "/rustfs/admin/v3/config", CONFIG_UPDATE, RouteRiskLevel::High),
     admin(HttpMethod::Put, "/rustfs/admin/v3/config", CONFIG_UPDATE, RouteRiskLevel::High),
     admin(HttpMethod::Get, "/rustfs/admin/v3/scanner/status", SERVER_INFO, RouteRiskLevel::Sensitive),
+    admin(
+        HttpMethod::Post,
+        "/rustfs/admin/v3/scanner/cycle-state/reset",
+        CONFIG_UPDATE,
+        RouteRiskLevel::High,
+    ),
     admin(
         HttpMethod::Get,
         "/rustfs/admin/v3/ilm/expiry/status",
@@ -2020,6 +2026,12 @@ mod tests {
         assert_not_action(HttpMethod::Get, "/rustfs/admin/v3/ilm/expiry/status", SET_TIER);
     }
 
+    #[test]
+    fn route_policy_requires_config_update_for_scanner_cycle_reset() {
+        assert_action(HttpMethod::Post, "/rustfs/admin/v3/scanner/cycle-state/reset", CONFIG_UPDATE);
+        assert_not_action(HttpMethod::Post, "/rustfs/admin/v3/scanner/cycle-state/reset", SERVER_INFO);
+    }
+
     #[test]
     fn route_policy_uses_tier_actions_for_transition_routes() {
         assert_action(HttpMethod::Post, "/rustfs/admin/v3/ilm/transition/run", SET_TIER);

rustfs/src/admin/route_registration_test.rs

@@ -243,6 +243,7 @@ fn expected_admin_route_matrix() -> Vec<RouteMatrixEntry> {
         admin_route(Method::GET, "/v3/config"),
         admin_route(Method::PUT, "/v3/config"),
         admin_route(Method::GET, "/v3/scanner/status"),
+        admin_route(Method::POST, "/v3/scanner/cycle-state/reset"),
         admin_route(Method::GET, "/v3/audit/target/list"),
         admin_route_sample(
             Method::PUT,
@@ -879,6 +880,7 @@ fn test_register_routes_cover_representative_admin_paths() {
     assert_route(&router, Method::GET, &admin_path("/v3/config"));
     assert_route(&router, Method::PUT, &admin_path("/v3/config"));
     assert_route(&router, Method::GET, &admin_path("/v3/scanner/status"));
+    assert_route(&router, Method::POST, &admin_path("/v3/scanner/cycle-state/reset"));
     assert_route(&router, Method::GET, &admin_path("/v3/ilm/expiry/status"));
     assert_route(&router, Method::POST, &admin_path("/v3/ilm/transition/run"));
     assert_route(
@@ -1367,6 +1369,7 @@ fn test_admin_alias_paths_match_existing_admin_routes() {
         (Method::GET, compat_admin_alias_path("/v3/config")),
         (Method::PUT, compat_admin_alias_path("/v3/config")),
         (Method::GET, compat_admin_alias_path("/v3/scanner/status")),
+        (Method::POST, compat_admin_alias_path("/v3/scanner/cycle-state/reset")),
         (Method::GET, compat_admin_alias_path("/v3/ilm/expiry/status")),
     ] {
         assert!(

rustfs/src/connect/client.rs

@@ -1,566 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-use std::time::Duration;
-
-use base64::Engine as _;
-use reqwest::{Client, StatusCode, Url};
-use rustls::RootCertStore;
-use rustls::pki_types::{CertificateDer, pem::PemObject as _};
-use serde::Deserialize;
-use uuid::Uuid;
-use zeroize::Zeroizing;
-
-use super::credential_store::{
-    CompletedRegistration, CredentialStore, CredentialStoreError, DeviceCredential, PendingRegistration, PendingRotation,
-};
-use super::identity::{IdentityError, RegistrationTranscript};
-use super::identity_store::{IdentityStore, StoreError};
-use super::registration::{
-    CredentialResponse, CredentialValidationError, ExpectedDevice, RegistrationRequest, RegistrationToken, RotationRequest,
-    certificate_fingerprint, certificate_request_matches, public_key_fingerprint, validate_credential,
-    validate_stored_credential,
-};
-
-const MAX_ATTEMPTS: usize = 3;
-const MAX_RESPONSE_BYTES: usize = 1024 * 1024;
-const ROTATION_THRESHOLD_SECONDS: i64 = 8 * 60 * 60;
-
-pub struct ConnectConfig<'a> {
-    pub endpoint: &'a str,
-    pub root_ca_pem: &'a [u8],
-    pub timeout: Duration,
-}
-
-pub struct ConnectClient {
-    endpoint: Url,
-    roots: RootCertStore,
-    root_certificates: Vec<CertificateDer<'static>>,
-    client: Client,
-    timeout: Duration,
-}
-
-impl ConnectClient {
-    pub fn from_optional_config(config: Option<ConnectConfig<'_>>) -> Result<Option<Self>, ClientError> {
-        config.map(Self::new).transpose()
-    }
-
-    pub fn new(config: ConnectConfig<'_>) -> Result<Self, ClientError> {
-        let mut endpoint = Url::parse(config.endpoint).map_err(|_| ClientError::Endpoint)?;
-        if endpoint.scheme() != "https"
-            || endpoint.cannot_be_a_base()
-            || !endpoint.username().is_empty()
-            || endpoint.password().is_some()
-            || endpoint.query().is_some()
-            || endpoint.fragment().is_some()
-        {
-            return Err(ClientError::Endpoint);
-        }
-        if !endpoint.path().ends_with('/') {
-            let path = format!("{}/", endpoint.path());
-            endpoint.set_path(&path);
-        }
-
-        let root_certificates = CertificateDer::pem_slice_iter(config.root_ca_pem)
-            .collect::<Result<Vec<_>, _>>()
-            .map_err(|_| ClientError::RootCertificate)?;
-        if root_certificates.is_empty() {
-            return Err(ClientError::RootCertificate);
-        }
-        let mut roots = RootCertStore::empty();
-        let (accepted, rejected) = roots.add_parsable_certificates(root_certificates.clone());
-        if accepted != root_certificates.len() || rejected != 0 {
-            return Err(ClientError::RootCertificate);
-        }
-
-        let client = build_client(&root_certificates, config.timeout, None)?;
-        Ok(Self {
-            endpoint,
-            roots,
-            root_certificates,
-            client,
-            timeout: config.timeout,
-        })
-    }
-
-    pub async fn register(
-        &self,
-        identity_store: &IdentityStore,
-        credential_store: &CredentialStore,
-        token: &RegistrationToken,
-    ) -> Result<DeviceCredential, ClientError> {
-        let _lock = credential_store.lock().await?;
-        if let Some((credential, _)) = self.load_valid_credential(identity_store, credential_store)? {
-            ensure_credential_time(&credential, unix_now())?;
-            return Ok(credential);
-        }
-
-        let identity = identity_store.load_or_create()?;
-        let candidate = PendingRegistration {
-            token_uid: token.registration_token_uid.clone(),
-            request_id: Uuid::new_v4().to_string(),
-            certificate_request: identity.certificate_request_base64()?,
-            previous_credential_fingerprint: None,
-            next_public_key_sha256: None,
-        };
-        let pending = credential_store.claim_pending_registration(&candidate)?;
-        if pending.token_uid != token.registration_token_uid
-            || pending.previous_credential_fingerprint.is_some()
-            || pending.next_public_key_sha256.is_some()
-            || !is_request_id(&pending.request_id)
-            || !certificate_request_matches(&pending.certificate_request, &identity)?
-        {
-            return Err(ClientError::PendingRegistration);
-        }
-
-        let credential = match self.exchange_registration(token, &pending, &identity).await {
-            Ok(credential) => credential,
-            Err(error @ (ClientError::AccessRevoked { .. } | ClientError::Rejected { .. })) => {
-                credential_store.clear_pending_registration()?;
-                return Err(error);
-            }
-            Err(error) => return Err(error),
-        };
-        credential_store.save(&credential)?;
-        credential_store.clear_pending_registration()?;
-        Ok(credential)
-    }
-
-    pub async fn reenroll(
-        &self,
-        identity_store: &IdentityStore,
-        credential_store: &CredentialStore,
-        token: &RegistrationToken,
-    ) -> Result<DeviceCredential, ClientError> {
-        let _lock = credential_store.lock().await?;
-        let (credential, _) = self
-            .load_valid_credential(identity_store, credential_store)?
-            .ok_or(ClientError::NotRegistered)?;
-        let fingerprint = certificate_fingerprint(&credential.certificate)?;
-        if credential_store.load_completed_registration()?.is_some_and(|completed| {
-            completed.token_uid == token.registration_token_uid && completed.credential_fingerprint == fingerprint
-        }) {
-            return Ok(credential);
-        }
-        credential_store.clear_pending_rotation()?;
-        let next = identity_store.load_or_create_next()?;
-        let next_fingerprint = public_key_fingerprint(&next);
-        let candidate = PendingRegistration {
-            token_uid: token.registration_token_uid.clone(),
-            request_id: Uuid::new_v4().to_string(),
-            certificate_request: next.certificate_request_base64()?,
-            previous_credential_fingerprint: Some(fingerprint.clone()),
-            next_public_key_sha256: Some(next_fingerprint.clone()),
-        };
-        let pending = credential_store.claim_pending_registration(&candidate)?;
-        if pending.token_uid != token.registration_token_uid
-            || pending.previous_credential_fingerprint.as_deref() != Some(&fingerprint)
-            || pending.next_public_key_sha256.as_deref() != Some(&next_fingerprint)
-            || !is_request_id(&pending.request_id)
-            || !certificate_request_matches(&pending.certificate_request, &next)?
-        {
-            return Err(ClientError::PendingRegistration);
-        }
-        let enrolled = match self.exchange_registration(token, &pending, &next).await {
-            Ok(credential) => credential,
-            Err(error @ (ClientError::AccessRevoked { .. } | ClientError::Rejected { .. })) => {
-                credential_store.clear_pending_registration()?;
-                identity_store.clear_next()?;
-                return Err(error);
-            }
-            Err(error) => return Err(error),
-        };
-        credential_store.save(&enrolled)?;
-        identity_store.commit_next(&next)?;
-        credential_store.save_completed_registration(&CompletedRegistration {
-            token_uid: token.registration_token_uid.clone(),
-            credential_fingerprint: certificate_fingerprint(&enrolled.certificate)?,
-        })?;
-        credential_store.clear_pending_registration()?;
-        Ok(enrolled)
-    }
-
-    async fn exchange_registration(
-        &self,
-        token: &RegistrationToken,
-        pending: &PendingRegistration,
-        identity: &super::identity::DeviceIdentity,
-    ) -> Result<DeviceCredential, ClientError> {
-        let csr_der = base64::engine::general_purpose::STANDARD
-            .decode(&pending.certificate_request)
-            .map_err(|_| ClientError::PendingRegistration)?;
-        let transcript = RegistrationTranscript::build(
-            &token.registration_token_uid,
-            &token.organization_uid,
-            &token.cluster_uid,
-            &pending.request_id,
-            &token.challenge_nonce,
-            token.expires_unix,
-            &csr_der,
-        )?;
-        let proof = identity.sign_registration(&transcript);
-        let body = RegistrationRequest::new(token, &pending.request_id, &pending.certificate_request, &proof);
-        let url = self.url("./registrationTokens:exchange")?;
-        let response = self
-            .send(StatusCode::CREATED, || self.client.post(url.clone()).json(&body))
-            .await?;
-        let cluster = format!("organizations/{}/clusters/{}", token.organization_uid, token.cluster_uid);
-        let credential = validate_credential(
-            response,
-            identity,
-            &self.roots,
-            &self.root_certificates,
-            ExpectedDevice::Registration { cluster: &cluster },
-        )?;
-        Ok(credential)
-    }
-
-    pub async fn rotate_if_due(
-        &self,
-        identity_store: &IdentityStore,
-        credential_store: &CredentialStore,
-        now_unix: i64,
-    ) -> Result<Option<DeviceCredential>, ClientError> {
-        let _lock = credential_store.lock().await?;
-        let (credential, identity) = self
-            .load_valid_credential(identity_store, credential_store)?
-            .ok_or(ClientError::NotRegistered)?;
-        if credential_store.load_pending_registration()?.is_some() {
-            return Err(ClientError::PendingRegistration);
-        }
-        ensure_credential_time(&credential, now_unix)?;
-        if credential.not_after_unix - now_unix > ROTATION_THRESHOLD_SECONDS {
-            return Ok(None);
-        }
-        let fingerprint = certificate_fingerprint(&credential.certificate)?;
-        let next = identity_store.load_or_create_next()?;
-        let candidate = PendingRotation {
-            credential_fingerprint: fingerprint.clone(),
-            device_name: credential.name.clone(),
-            request_id: Uuid::new_v4().to_string(),
-            certificate_request: next.certificate_request_base64()?,
-            next_public_key_sha256: public_key_fingerprint(&next),
-        };
-        let pending = credential_store.claim_pending_rotation(&candidate)?;
-        if pending.credential_fingerprint != fingerprint
-            || pending.device_name != credential.name
-            || !is_request_id(&pending.request_id)
-            || pending.next_public_key_sha256 != public_key_fingerprint(&next)
-            || !certificate_request_matches(&pending.certificate_request, &next)?
-        {
-            return Err(ClientError::PendingRotation);
-        }
-        let body = RotationRequest::new(
-            &identity,
-            &fingerprint,
-            &credential.name,
-            &pending.request_id,
-            &pending.certificate_request,
-        )?;
-        let private_key = identity.to_pkcs8_pem()?;
-        let mut identity_pem = Zeroizing::new(Vec::with_capacity(credential.certificate_chain.len() + private_key.len() + 1));
-        identity_pem.extend_from_slice(credential.certificate_chain.as_bytes());
-        identity_pem.push(b'\n');
-        identity_pem.extend_from_slice(private_key.as_bytes());
-        let tls_identity = reqwest::Identity::from_pem(&identity_pem).map_err(|_| ClientError::IdentityCertificate)?;
-        let client = build_client(&self.root_certificates, self.timeout, Some(tls_identity))?;
-        let path = format!("clusterDevices/{}:rotateCredential", credential.uid);
-        let url = self.url(&path)?;
-        let response = self.send(StatusCode::OK, || client.post(url.clone()).json(&body)).await?;
-        let rotated = validate_credential(
-            response,
-            &next,
-            &self.roots,
-            &self.root_certificates,
-            ExpectedDevice::Rotation { name: &credential.name },
-        )?;
-        if rotated.name != credential.name || rotated.uid != credential.uid {
-            return Err(ClientError::Credential(CredentialValidationError::Identity));
-        }
-        credential_store.save(&rotated)?;
-        identity_store.commit_next(&next)?;
-        credential_store.clear_pending_rotation()?;
-        Ok(Some(rotated))
-    }
-
-    fn load_valid_credential(
-        &self,
-        identity_store: &IdentityStore,
-        credential_store: &CredentialStore,
-    ) -> Result<Option<(DeviceCredential, super::identity::DeviceIdentity)>, ClientError> {
-        let Some(credential) = credential_store.load()? else {
-            return Ok(None);
-        };
-        let current = identity_store.load()?.ok_or(ClientError::IdentityMissing)?;
-        if let Some(pending) = credential_store.load_pending_registration()? {
-            let Some(previous) = pending.previous_credential_fingerprint.as_deref() else {
-                if pending.next_public_key_sha256.is_some()
-                    || !is_request_id(&pending.request_id)
-                    || !certificate_request_matches(&pending.certificate_request, &current)?
-                {
-                    return Err(ClientError::PendingRegistration);
-                }
-                validate_stored_credential(&credential, &current, &self.roots, &self.root_certificates)?;
-                credential_store.clear_pending_registration()?;
-                return Ok(Some((credential, current)));
-            };
-            let next_fingerprint = pending
-                .next_public_key_sha256
-                .as_deref()
-                .ok_or(ClientError::PendingRegistration)?;
-            if !is_request_id(&pending.request_id) {
-                return Err(ClientError::PendingRegistration);
-            }
-            let fingerprint = certificate_fingerprint(&credential.certificate)?;
-            if fingerprint == previous {
-                validate_stored_credential(&credential, &current, &self.roots, &self.root_certificates)?;
-                let next = identity_store.load_next()?.ok_or(ClientError::PendingRegistration)?;
-                if public_key_fingerprint(&next) != next_fingerprint
-                    || !certificate_request_matches(&pending.certificate_request, &next)?
-                {
-                    return Err(ClientError::PendingRegistration);
-                }
-                return Ok(Some((credential, current)));
-            }
-            if public_key_fingerprint(&current) == next_fingerprint {
-                if !certificate_request_matches(&pending.certificate_request, &current)? {
-                    return Err(ClientError::PendingRegistration);
-                }
-                validate_stored_credential(&credential, &current, &self.roots, &self.root_certificates)?;
-            } else {
-                let next = identity_store.load_next()?.ok_or(ClientError::PendingRegistration)?;
-                if public_key_fingerprint(&next) != next_fingerprint
-                    || !certificate_request_matches(&pending.certificate_request, &next)?
-                {
-                    return Err(ClientError::PendingRegistration);
-                }
-                validate_stored_credential(&credential, &next, &self.roots, &self.root_certificates)?;
-                identity_store.commit_next(&next)?;
-            }
-            credential_store.save_completed_registration(&CompletedRegistration {
-                token_uid: pending.token_uid,
-                credential_fingerprint: fingerprint,
-            })?;
-            credential_store.clear_pending_registration()?;
-            let current = identity_store.load()?.ok_or(ClientError::IdentityMissing)?;
-            return Ok(Some((credential, current)));
-        }
-        let Some(pending) = credential_store.load_pending_rotation()? else {
-            validate_stored_credential(&credential, &current, &self.roots, &self.root_certificates)?;
-            return Ok(Some((credential, current)));
-        };
-        let fingerprint = certificate_fingerprint(&credential.certificate)?;
-        if pending.device_name != credential.name || !is_request_id(&pending.request_id) {
-            return Err(ClientError::PendingRotation);
-        }
-        if fingerprint == pending.credential_fingerprint {
-            validate_stored_credential(&credential, &current, &self.roots, &self.root_certificates)?;
-            let next = identity_store.load_next()?.ok_or(ClientError::PendingRotation)?;
-            if pending.next_public_key_sha256 != public_key_fingerprint(&next)
-                || !certificate_request_matches(&pending.certificate_request, &next)?
-            {
-                return Err(ClientError::PendingRotation);
-            }
-            return Ok(Some((credential, current)));
-        }
-
-        if public_key_fingerprint(&current) == pending.next_public_key_sha256 {
-            if !certificate_request_matches(&pending.certificate_request, &current)? {
-                return Err(ClientError::PendingRotation);
-            }
-            validate_stored_credential(&credential, &current, &self.roots, &self.root_certificates)?;
-        } else {
-            let next = identity_store.load_next()?.ok_or(ClientError::PendingRotation)?;
-            if public_key_fingerprint(&next) != pending.next_public_key_sha256
-                || !certificate_request_matches(&pending.certificate_request, &next)?
-            {
-                return Err(ClientError::PendingRotation);
-            }
-            validate_stored_credential(&credential, &next, &self.roots, &self.root_certificates)?;
-            identity_store.commit_next(&next)?;
-        }
-        credential_store.clear_pending_rotation()?;
-        let current = identity_store.load()?.ok_or(ClientError::IdentityMissing)?;
-        Ok(Some((credential, current)))
-    }
-
-    async fn send<F>(&self, success: StatusCode, mut request: F) -> Result<CredentialResponse, ClientError>
-    where
-        F: FnMut() -> reqwest::RequestBuilder,
-    {
-        let mut last_status = None;
-        for attempt in 0..MAX_ATTEMPTS {
-            match request().send().await {
-                Ok(response) if response.status() == success => return decode_response(response).await,
-                Ok(response) if matches!(response.status(), StatusCode::UNAUTHORIZED | StatusCode::FORBIDDEN) => {
-                    let status = response.status();
-                    let reason = decode_reason(response).await;
-                    return Err(ClientError::AccessRevoked { status, reason });
-                }
-                Ok(response) if matches!(response.status(), StatusCode::REQUEST_TIMEOUT | StatusCode::TOO_MANY_REQUESTS) => {
-                    last_status = Some(response.status());
-                }
-                Ok(response) if response.status().is_client_error() => {
-                    let status = response.status();
-                    let reason = decode_reason(response).await;
-                    return Err(ClientError::Rejected { status, reason });
-                }
-                Ok(response) if response.status().is_server_error() => {
-                    last_status = Some(response.status());
-                }
-                Ok(response) => {
-                    let status = response.status();
-                    let reason = decode_reason(response).await;
-                    return Err(ClientError::Rejected { status, reason });
-                }
-                Err(error) if !error.is_timeout() && !error.is_connect() => return Err(ClientError::Transport(error)),
-                Err(_) => {}
-            }
-
-            if attempt + 1 < MAX_ATTEMPTS {
-                tokio::time::sleep(Duration::from_millis(50 * (attempt as u64 + 1))).await;
-            }
-        }
-        Err(ClientError::Unavailable { status: last_status })
-    }
-
-    fn url(&self, path: &str) -> Result<Url, ClientError> {
-        self.endpoint.join(path).map_err(|_| ClientError::Endpoint)
-    }
-}
-
-fn is_request_id(value: &str) -> bool {
-    Uuid::parse_str(value).is_ok_and(|uuid| uuid.get_version() == Some(uuid::Version::Random) && uuid.to_string() == value)
-}
-
-fn unix_now() -> i64 {
-    std::time::SystemTime::now()
-        .duration_since(std::time::UNIX_EPOCH)
-        .map_or(0, |duration| duration.as_secs() as i64)
-}
-
-fn ensure_credential_time(credential: &DeviceCredential, now_unix: i64) -> Result<(), ClientError> {
-    if credential.not_before_unix > now_unix {
-        return Err(ClientError::CredentialNotYetValid);
-    }
-    if credential.not_after_unix <= now_unix {
-        return Err(ClientError::CredentialExpired);
-    }
-    Ok(())
-}
-
-fn build_client(
-    roots: &[CertificateDer<'static>],
-    timeout: Duration,
-    identity: Option<reqwest::Identity>,
-) -> Result<Client, ClientError> {
-    let certificates = roots
-        .iter()
-        .map(|root| reqwest::Certificate::from_der(root.as_ref()))
-        .collect::<Result<Vec<_>, _>>()?;
-    let mut builder = Client::builder()
-        .https_only(true)
-        .redirect(reqwest::redirect::Policy::none())
-        .timeout(timeout)
-        .tls_certs_only(certificates);
-    if let Some(identity) = identity {
-        builder = builder.identity(identity);
-    }
-    builder.build().map_err(ClientError::Transport)
-}
-
-async fn decode_response(mut response: reqwest::Response) -> Result<CredentialResponse, ClientError> {
-    let body = read_body(&mut response).await?;
-    serde_json::from_slice(&body).map_err(|_| ClientError::Response)
-}
-
-async fn read_body(response: &mut reqwest::Response) -> Result<Vec<u8>, ClientError> {
-    let mut body = Vec::new();
-    while let Some(chunk) = response.chunk().await.map_err(ClientError::Transport)? {
-        if body.len() + chunk.len() > MAX_RESPONSE_BYTES {
-            return Err(ClientError::ResponseTooLarge);
-        }
-        body.extend_from_slice(&chunk);
-    }
-    Ok(body)
-}
-
-#[derive(Deserialize)]
-struct ErrorEnvelope {
-    #[serde(default)]
-    details: Vec<ErrorDetail>,
-}
-
-#[derive(Deserialize)]
-struct ErrorDetail {
-    #[serde(default)]
-    reason: String,
-}
-
-async fn decode_reason(mut response: reqwest::Response) -> Option<String> {
-    let body = read_body(&mut response).await.ok()?;
-    serde_json::from_slice::<ErrorEnvelope>(&body)
-        .ok()?
-        .details
-        .into_iter()
-        .find_map(|detail| (!detail.reason.is_empty()).then_some(detail.reason))
-}
-
-#[derive(Debug, thiserror::Error)]
-pub enum ClientError {
-    #[error("Connect endpoint must be an HTTPS base URL without credentials, query, or fragment")]
-    Endpoint,
-    #[error("Connect root CA configuration is invalid")]
-    RootCertificate,
-    #[error(
-        "Connect registration has a pending attempt for a different token; restore the original protected token configuration"
-    )]
-    PendingRegistration,
-    #[error(
-        "Connect credential rotation has an unfinished attempt for a different current certificate; inspect the local credential store"
-    )]
-    PendingRotation,
-    #[error("RustFS is not registered with Connect")]
-    NotRegistered,
-    #[error("the Connect device private key is missing; restore device.key before using the stored certificate")]
-    IdentityMissing,
-    #[error("the Connect device certificate has expired; call ConnectClient::reenroll with a fresh registration token")]
-    CredentialExpired,
-    #[error("the Connect device certificate is not yet valid; fix local clock skew or call ConnectClient::reenroll")]
-    CredentialNotYetValid,
-    #[error("the stored Connect certificate and device private key cannot form a TLS identity")]
-    IdentityCertificate,
-    #[error(
-        "Connect rejected the device credential with HTTP {status}; reason={reason:?}; call ConnectClient::reenroll with a fresh registration token if revoked"
-    )]
-    AccessRevoked { status: StatusCode, reason: Option<String> },
-    #[error("Connect rejected the request with HTTP {status}; reason={reason:?}")]
-    Rejected { status: StatusCode, reason: Option<String> },
-    #[error("Connect remained unavailable after bounded retries; last_status={status:?}")]
-    Unavailable { status: Option<StatusCode> },
-    #[error("Connect response exceeded the 1 MiB credential-response limit")]
-    ResponseTooLarge,
-    #[error("Connect returned an invalid credential response")]
-    Response,
-    #[error(transparent)]
-    Transport(#[from] reqwest::Error),
-    #[error(transparent)]
-    Identity(#[from] IdentityError),
-    #[error(transparent)]
-    IdentityStore(#[from] StoreError),
-    #[error(transparent)]
-    CredentialStore(#[from] CredentialStoreError),
-    #[error(transparent)]
-    Credential(#[from] CredentialValidationError),
-}