docs: add MinIO replication compatibility review and P1 remediation plan

.agents/skills/adversarial-validation/SKILL.md

@@ -1,45 +1,277 @@
 ---
 name: adversarial-validation
-description: Review a final RustFS diff adversarially when the user requests adversarial review, the root AGENTS.md classifies the change as high risk, or a substantial PR is being reviewed. Do not use for ordinary questions, diagnosis, planning, status, documentation-only work, or routine low-risk implementation.
+description: Execute the Adversarial Validation policy from the root AGENTS.md — run the seven reviewer roles (correctness, simplicity, security, concurrency/durability, compatibility, performance, test coverage) with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, or design proposal before declaring it done.
 ---
 
-# RustFS Adversarial Validation
+# Adversarial Validation Playbooks
 
-Use the risk tier and review shape defined in the root `AGENTS.md`. This skill
-routes a review to RustFS-specific probes without loading unrelated domains.
+The policy — risk tiers, role list, protocol, exit criteria — lives in the
+root `AGENTS.md` under "Adversarial Validation (Default On)". Read it first;
+this skill does not restate it. This file adds the RustFS-specific probe
+playbook for each role: concrete attacks, where they apply, and the real
+shipped bug or rule that earns each probe its place.
 
-## Select Lenses
+## How to run a role
 
-Read only the references required by the diff:
+1. Pick the tier and the applicable roles per the root `AGENTS.md`.
+2. Run each role as an independent pass over the final diff — a parallel
+   reviewer agent where the tooling supports it, otherwise a fresh
+   sequential pass that starts from the diff and the nearest scoped
+   `AGENTS.md`, discarding the writing session's assumptions.
+3. Within a role, execute the probes whose domain the diff touches, plus any
+   attack the diff obviously invites that no probe lists — the playbook is a
+   floor, not a ceiling.
+4. Report findings (concrete failure scenario or named missing test, with
+   file:line) or the role's null report: "attacked X, Y, Z — no break
+   found". A bare pass is not a result.
 
-| Lens | When to read |
-|---|---|
-| [Correctness](references/correctness.md) | Every non-exempt adversarial review |
-| [Simplicity](references/simplicity.md) | Mechanical/standard changes and production growth |
-| [Test coverage](references/test-coverage.md) | Behavior or test changes |
-| [Security](references/security.md) | Authn/authz, IAM, RPC trust, paths, secrets, parsing, browser, encryption |
-| [Concurrency/durability](references/concurrency-durability.md) | Async shared state, locks, storage commit, cancellation, persisted queues |
-| [Compatibility](references/compatibility.md) | S3 surface, MinIO interop, metadata, wire/disk formats, mixed versions |
-| [Performance](references/performance.md) | Request/object hot paths, allocation, blocking work, fsync, fan-out |
+## Role playbooks
 
-Do not read all references as a precaution. A path name alone is insufficient;
-the changed behavior must touch the lens's domain.
+### Correctness adversary
 
-For a dedicated security audit or advisory analysis, use
-`security-advisory-lessons` instead of loading it automatically during every
-adversarial review.
+- For any change touching error aggregation or quorum decisions, build the exact disk-error slice at the quorum boundary: N disks where successes == quorum, then flip one success to an error (quorum-1) and separately inject None/nil placeholder entries into the slice. Trace whether reduce_errs (or the new equivalent) picks the placeholder as the dominant error or lets quorum-1 pass as success. Also check heal/write paths: does a per-target failure at quorum-1 return an explicit error, or silently degrade to success?
+  - Where: crates/ecstore/src/disk/error_reduce.rs; crates/ecstore/src/set_disk/{core,ops}; crates/heal
+  - Evidence: Commit 20d61c73b 'stop reduce_errs leaking nil placeholder as dominant error' (#4551) and 47c1e730c 'make erasure heal write quorum best-effort per target' (#4545); crates/ecstore/AGENTS.md: 'Do not weaken quorum checks... Prefer explicit failure over silent data corruption or implicit success.'
+- For any change in EC read/reconstruct/streaming code, trace the mid-stream error path: the first K shards read fine, then a shard turns out bitrot-corrupt or inconsistent after N bytes have already been sent to the client. Verify the error propagates as a stream error (client sees failure), not a clean end-of-body — a silently truncated GET body is data corruption. Also re-check byte accounting: sum of per-part bytes vs object size, and partNumber-to-offset routing for the first and last part.
+  - Where: crates/ecstore/src/set_disk/read.rs (reconstruct-read, inconsistent_source_indexes handling ~line 3827); codec streaming paths in crates/ecstore/src/erasure_coding
+  - Evidence: Known live bug: EC reconstruct-read failing on inconsistent shards mid-stream silently truncates GET body → client 'unexpected EOF'; commit 15808254d 'correct codec-streaming byte accounting and partNumber routing' (#4535).
+- For any listing/pagination change, construct the exact-boundary inputs: (a) exactly max_keys/max_uploads matching entries — assert the response contains max and is_truncated=false, then max+1 entries — assert exactly max returned with is_truncated=true and a correct continuation marker; (b) a delimiter listing where folding into CommonPrefixes re-fills a full page; (c) an object 'a' coexisting with prefix dir 'a/'. Off-by-one and dropped-truncation bugs live exactly at these boundaries.
+  - Where: crates/ecstore listing/merge paths (metacache, list_objects, list_multipart_uploads); rustfs/src/storage
+  - Evidence: Three recent real bugs: fefa70b31 'stop ListMultipartUploads from returning one upload past max-uploads' (#4447), d91f4d455 'report truncation when delimiter list re-folds a full page' (#4538), 7e1f7f242 'preserve CommonPrefixes when an object and same-named prefix dir coexist' (#4563).
+- Run the diff's logic with a directory object key (trailing slash, e.g. 'pre/dir/') as input. Check which layer encodes/decodes __XLDIR__ — set_disk never sees the trailing slash, so any trailing-slash branch added below the store layer is dead code and a wrong-layer bug. For delete paths, check whether options force a nil versionId onto directory keys: the resulting miss surfaces as version-not-found, not object-not-found, so callers matching only ObjectNotFound leak ghost directory entries.
+  - Where: crates/ecstore/src/store*.rs (store layer) vs crates/ecstore/src/set_disk/*; delete option construction (del_opts) and its callers
+  - Evidence: trailing-slash branches must live at store layer; del_opts injects nil version for dir keys, PR#4220 ghost-directory cleanup never fired on the real path (rustfs#4307, backlog#798 still OPEN).
+- Feed every new binary-UUID metadata read the three degenerate values: key absent, zero-length bytes, and 16 zero bytes (nil UUID). All three must mean 'no value' — any path that produces Uuid::nil() and then acts on it (e.g. sends it as a versionId) is a finding. For tier code specifically: with remote-tier version None or "", assert the tier GET/DELETE request carries no versionId parameter at all — sending versionId="" or nil yields NoSuchVersion against unversioned tier buckets.
+  - Where: crates/filemeta/src/filemeta/version.rs; crates/ecstore/src/bucket/lifecycle/; crates/ecstore/src/services/tier/; any new consumer of crates/utils/src/http/metadata_compat.rs
+  - Evidence: AGENTS.md Cross-Cutting Domain Invariants (defensive Uuid read pattern + unversioned-tier rule); docs/operations/tier-ilm-debugging.md ('nil transition_ver_id = corrupt legacy write-back, readers must filter'); commit 726f3dc18 'accept empty remote version_id in tier recovery paths' (#4552).
+- For each match/if-let on an error or algorithm enum touched by the diff, enumerate what the wildcard/else arm swallows. Inject the variants the author didn't think of — DiskNotFound during listing, an unsupported checksum algorithm, an Err from a cleanup rename/delete — and trace whether they degrade into 'not found', a wrong-but-plausible value, or silent success. Any error path that converges with the success path without logging and propagating is a finding.
+  - Where: crates/ecstore (listing, delete/rename cleanup); crates/checksums; error-mapping layers in rustfs/src/storage
+  - Evidence: Three recent real bugs of this exact shape: e0619e355 'stop treating DiskNotFound as object not-found in listing' (#4536), afaf8c681 'reject md5 instead of silently returning crc32' (#4513), f7d2b2563 'propagate disk delete/rename failures instead of swallowing them' (#4546).
+- For any change to version ordering, index lookup, or shard/part indexing: (a) call the accessor with index == len() and len()-1 — a get_idx-style bound must reject, not panic or wrap; (b) construct two versions with identical mod-times and check the sort tie-break is total and deterministic (equal keys must not compare as both before each other); (c) for EC shard math, compute shard size for object sizes 0, 1, blockSize-1, blockSize, blockSize+1 and cross-check total reconstructed length against the object size.
+  - Where: crates/filemeta/src/filemeta/*.rs (version sort, get_idx); crates/ecstore/src/erasure_coding shard-size math
+  - Evidence: Commit 8bfb00bc0 'guard get_idx bound and fix sorts_before tie-break' (#4509) — both bug classes shipped before; ecstore AGENTS.md high-risk designation for read/write/repair correctness.
+- Exercise the zero/empty end of every new size or count parameter: zero-length object PUT then GET (body must be empty, not error), part count 0, empty Vec of disks/entries into aggregation functions, and env/config values of 0 (must clamp or reject, never divide-by-zero or 'scan nothing and report zero usage'). Anywhere the diff computes a ratio, capacity, or progress percentage, plug in 0 and the max value.
+  - Where: crates/ecstore aggregation and scanner paths; crates/object-capacity; config/env parsing in touched crates
+  - Evidence: Commits 787cc77a7 'clamp zero capacity env values to safe defaults' (#4559) and 32b1094ec 'resolve a symlinked scan root instead of silently counting zero' (#4564) — zero-as-silent-wrong-answer is a recurring repo bug class.
+- For any diff touching multipart or object commit paths, order the operations on paper and attack the failure point between them: kill the process (or return Err) after the commit rename but before cleanup, and after cleanup but before commit. Verify the earlier-failure case leaves the object readable and the later-failure case leaves no half-visible object; part meta files must never be deleted before the commit is durable.
+  - Where: crates/ecstore multipart commit/cleanup (set_disk/ops); rustfs/src/storage multipart handlers
+  - Evidence: Commit c77c5f047 'defer multipart part.N.meta cleanup until after commit' (#4548) — cleanup-before-commit ordering already caused a real data-loss window; the #4221 durability work shows fsync/ordering bugs are endemic here.
 
-## Review Protocol
+Null report example: "Attacked quorum-1 error reduction, exact max-keys listing boundary, trailing-slash dir keys, nil-UUID tier versionId, and mid-stream reconstruct error propagation — no break found."
 
-1. Freeze the exact final diff/head and list the selected lenses.
-2. Run the review shape required by root `AGENTS.md`.
-3. For each selected lens, either report a concrete finding or a null verdict
-   naming the attacks performed.
-4. A finding needs `file:line`, a triggering input/state/interleaving, the wrong
-   outcome, and a focused fix or missing regression check.
-5. Fix or rebut every finding with code-path, test, or invariant evidence.
-6. After a non-trivial edit, rerun only lenses affected by that edit against the
-   new exact diff.
+### Simplicity adversary
 
-Do not turn a null verdict into a long checklist. Record concise evidence that
-the relevant failure classes were attacked.
+- Smaller-diff attack: rewrite the diff's change mentally (or actually, in scratch) as the minimal in-place edit and compare. Flag as findings: a helper function with exactly one caller introduced by this diff; a file rewrite where a 3-line edit inside the existing control flow suffices; reshaped control flow in init/locking/metadata/quorum paths beyond what the fix requires; new string literals duplicating existing constants (grep the token first); #[path] module inclusion. If the smaller diff achieves identical behavior, report it with the concrete replacement.
+  - Where: Any diff; extra scrutiny for crates/ecstore, crates/lock, rustfs/src/storage where 'preserve the existing control-flow shape' is an explicit rule
+  - Evidence: AGENTS.md 'Change Style for Existing Logic' (one-off helper ban, preserve control-flow shape in distributed/locking/metadata paths, no #[path]) and 'Reuse Before You Write' (constants clause); the Adversarial Validation roles list charters the simplicity adversary with exactly this attack.
+- Reuse-and-necessity attack: for each new helper the diff introduces, run `ls crates/utils/src crates/common/src` and `rg -i 'fn \w*<term>'` over those dirs plus the touched crate (snake_case signatures — a full-text single-word grep drowns, a multi-word phrase returns nothing). A reimplementation of an existing workspace utility, or of plain std/tokio behavior no wrapper refines, is a finding — but so is forced reuse with mismatched semantics (normalization such as `clean` resolving `.`/`..` against raw S3 keys, error type, backoff, durability gating). For each new defensive branch, demand the nameable trigger and flag re-validation of what a validated upstream layer on the SAME path already guarantees — excluding the Cross-Cutting Domain Invariant patterns (nil/empty/absent UUID, dual metadata keys, unversioned-tier versionId) and re-checks before destructive actions, which are load-bearing even when redundant on the happy path. For each new test, flag near-duplicates pinning the same code path AND poison-value class as an existing test — boundary companions (n==max vs max+1, absent vs empty vs nil UUID, MetaObject vs MetaDeleteMarker) are never near-duplicates; the test-coverage skeptic playbook below mandates them.
+  - Where: Any diff adding helpers, branches on decoded/peer data, or tests; helper checks against crates/utils, crates/common, and the touched crate
+  - Evidence: AGENTS.md 'Reuse Before You Write' and 'Necessary Code Only'; GHSA-f4vq-9ffr-m8m3 (normalization-asymmetry traversal — why forced reuse of normalizing helpers on raw keys is itself an attack); docs/operations/tier-ilm-debugging.md nil-versionId incident (why boundary re-checks are load-bearing).
+
+Null report example: "Rewrote the diff as an in-place edit (no smaller equivalent exists), grepped both new helpers against crates/utils, crates/common, and the touched crate (no existing equivalent; call-site semantics checked), verified the two new defensive branches name concrete corrupt-input triggers, and checked the added tests against the existing suite (each pins a distinct poison-value class) — no break found."
+
+### Security reviewer
+
+- For every admin handler in the diff, grep the exact AdminAction constant it passes to validate_admin_request and confirm it names the operation the handler actually performs. Construct the escalation: a low-privileged user whose policy grants the wrong-but-adjacent action (e.g. Export while the handler Imports, or Update while it Lists) — if the mismatched constant lets them through, that is the bug. Also confirm read-only endpoints (metrics, list, server-info, diagnostics) still call an operation-specific admin authz path and not a mere 'credentials exist' check.
+  - Where: rustfs/src/admin/handlers/**, rustfs/src/admin/router registration; check_permissions / validate_admin_request / AdminAction::* call sites
+  - Evidence: GHSA-vcwh-pff9-64cc (ImportIam checked ExportIAMAction), GHSA-mm2q-qcmx-gw4w (ListServiceAccount used UpdateServiceAccountAdminAction), GHSA-f5cv-v44x-2xgf (/admin/v3/metrics accepted any authenticated IAM user). advisory-patterns.md 'Admin authorization and route exposure'; rustfs/src/admin/AGENTS.md 'route registration, whitelist, handler authz must agree'.
+- If the diff touches service-account or IAM import/update, treat parent, claims, accessKey, secretKey, status, policy names, and groups as attacker-controlled. Construct an ImportIam/create payload where parent points at root (or another user) and prove the code writes credentials without proving caller ownership or root authority. Separately, set deny_only=true (or 'no explicit deny') on a restricted account and check it does not skip the required allow check, letting it mint an unrestricted child.
+  - Where: crates/iam/, rustfs/src/admin/handlers (service account / import IAM), rustfs/src/auth.rs
+  - Evidence: GHSA-566f-q62r-wcr8 (attacker-controlled parent/claims/accessKey/secretKey → persistent backdoor under root), GHSA-xgr5-qc6w-vcg9 (deny_only=true skipped allow checks → privilege creation). crates/iam/AGENTS.md security boundaries; advisory-patterns.md 'IAM import, service accounts'.
+- For a changed protocol-frontend handler (FTP/FTPS/SFTP/WebDAV/gateway), enumerate ALL sibling command handlers in the same driver — not just the changed one. For each, confirm it calls the per-operation IAM authorize hook mapped to the correct S3 action (RETR→GetObject, SIZE/MDTM→HeadObject, MKD→CreateBucket, bucket probe→ListBucket/HeadBucket) BEFORE touching storage. Construct a denied-authz case and prove the storage backend is never reached.
+  - Where: crates/protocols/ (FtpsDriver, SftpDriver, WebDAV), authorize_operation call sites
+  - Evidence: GHSA-3g29-xff2-92vp (FTP RETR/SIZE/MDTM authenticated but skipped IAM), GHSA-g3vq-vv42-f647 (FTPS MKD called create_bucket without s3:CreateBucket). advisory-patterns.md: 'RustFS advisories show mixed guarded and unguarded siblings in the same driver.'
+- For any secret/token/signature/password comparison in the diff, check it uses a constant-time compare (e.g. subtle/constant_time_eq), not == or early-return byte loops. Then check the failure-response paths: construct an invalid-user request and an invalid-secret request and confirm they are indistinguishable (same error, no early length short-circuit) so an attacker cannot enumerate valid users or time-side-channel the secret.
+  - Where: crates/protocols/ (FTPS/WebDAV/FormPost auth), crates/credentials/, rustfs/src/auth.rs, RPC signature verification
+  - Evidence: GHSA-3p3x-734c-h5vx (FTPS/WebDAV early-return string equality + distinguishable invalid-user vs invalid-password). Fix commits 3c3113619 (constant-time FTPS/WebDAV) and c41062f27 (constant-time FormPost signature). 3p3x was fixed by PR #4403.
+- If the diff parses or transports secret-bearing config (env vars, key files, connection strings), grep every error-construction and format site on that value's path (`format!` feeding `Error::other`/`configuration_error`/`panic!`/`expect`) for interpolation of the raw value or of variables named like secret material. Construct the likeliest misconfiguration: the operator supplies the bare secret without the expected `<name>:` prefix (or with a stray newline) — if the parse-failure hint echoes the input, the secret lands in startup logs. Error strings are log content; the hint may name the env var and expected format, never the value. If the diff re-implements an existing parse helper, diff the two error paths — the duplicate is where the leak hides.
+  - Where: rustfs/src/init.rs (env plumbing), crates/kms/src/config.rs, crates/credentials/, any from_env/parse on secret values; mechanical backstop in scripts/check_logging_guardrails.sh (secret-interpolation check)
+  - Evidence: PR #5222 introduced `got: {secret_str}` in build_static_kms_config's format-hint error — a bare base64 key (the secret itself) would have been echoed into startup logs; fixed by PR #5243. The parallel parse in KmsConfig::from_env already omitted the value: the leak lived only in the duplicated copy (AGENTS.md 'Reuse Before You Write').
+- If the diff touches internode/RPC auth secret handling, trace whether the RPC HMAC secret can fall back to a public default (e.g. 'rustfsadmin', 'rustfs rpc') or be derived deterministically from the S3 root credentials. Construct the case where RUSTFS_RPC_SECRET is unset and confirm the code fails closed rather than silently using a default or a root-derived key. Verify RPC signing keys are independent random secrets, not reused across S3-root/RPC-HMAC/STS-JWT roles.
+  - Where: crates/credentials/, crates/ecstore/src/rpc/, internode auth setup
+  - Evidence: GHSA-r5qv-rc46-hv8q (fell back to 'rustfsadmin'), GHSA-75fx/68cw (RPC secret derivable from root creds → forgeable signatures), GHSA-h956 (hard-coded 'rustfs rpc'), GHSA-m77q (STS JWT reused root secret). Fix commit 7b2055405 (fail closed when deriving RPC secret from default credentials, PR#4402).
+- If the diff touches RPC/NodeService authentication, verify the HMAC payload binds the EXACT concrete gRPC method path (not a service prefix), the HTTP method surrogate, and a fresh timestamp. Construct captured valid metadata for method A and replay it to method B within the timestamp window — if it authorizes, the signature is under-bound. Also test stale timestamp, wrong path, wrong method, wrong secret.
+  - Where: crates/ecstore/src/rpc/, verify_rpc_signature / NodeServiceServer, x-rustfs-signature handling
+  - Evidence: GHSA-c667-rgrv-99vj (signed service prefix instead of concrete method path → cross-method replay in timestamp window). advisory-patterns.md 'RPC input validation and panic safety'; Minimum Regression Test Expectations lists replay across two methods.
+- For any RPC/gRPC handler or deserialization touched, feed empty bytes, truncated MessagePack/protobuf, invalid enum discriminants, and stale timestamps. Grep the deserialization path for unwrap()/expect()/panic-prone decode and prove malformed attacker payloads return a typed error, not a panic (remote DoS). Weak internode auth makes reachability worse, so combine with the RPC-secret probe.
+  - Where: crates/ecstore/src/rpc/, any #[derive(Deserialize)] decoded from wire bytes, RPC handler bodies
+  - Evidence: GHSA-gw2x-q739-qhcr (malformed GetMetrics reached unwrap() → remote DoS). advisory-patterns.md 'Treat all RPC payload bytes as attacker-controlled.' rust-code-quality skill: unwrap abuse.
+- Take any object key, RPC disk path, or archive/tar/zip entry name introduced or handled in the diff and construct traversal payloads: '../', URL-encoded '%2e%2e%2f', absolute paths, platform separators, empty components. Trace the value through parse → authz check → final storage path and prove (a) authz and storage normalize the SAME way, and (b) the canonicalized path cannot escape the bucket/prefix root. Attack the case where authz sees the raw attacker bucket but storage cleaning crosses into a victim bucket.
+  - Where: crates/ecstore (path join/canonicalize), rustfs/src/storage/, Snowball auto-extract / normalize_extract_entry_key, rpc read_file_stream
+  - Evidence: GHSA-pq29-69jg-9mxc (read_file_stream joined untrusted paths, no canonical boundary check), GHSA-8r6f-hmq2-28rg (traversal object keys bypassed authz), GHSA-f4vq-9ffr-m8m3 (Snowball '../victim-bucket/object' authorized raw path then storage crossed boundary). Note __XLDIR__ trailing-slash encoding is store-layer only.
+- If the diff touches multipart/copy or presigned POST, verify UploadPartCopy enforces source GetObject AND destination PutObject semantics equivalent to CopyObject, including copy-source policy conditions (not just independent source-read + dest-write). Construct a cross-bucket UploadPartCopy from a bucket the caller cannot read. For presigned POST, submit an upload that violates content-length-range, key prefix, or exact content-type and prove the server rejects it.
+  - Where: rustfs/src/storage / S3 API handlers: upload_part_copy, CompleteMultipartUpload, PostObject policy enforcement
+  - Evidence: GHSA-mx42-j6wv-px98 (UploadPartCopy missed source authz → cross-bucket exfil), GHSA-wfxj-ph3v-7mjf (missed destination copy-source policy constraint), GHSA-w5fh-f8xh-5x3p (presigned POST didn't enforce signed policy conditions). advisory-patterns.md 'S3 copy, multipart, and upload policy validation'.
+- Grep the diff for debug!/trace!/info!/error! and any ?value / {:?} on structs or response bodies that can carry secret_key, session_token, JWT claims, HMAC secrets, expected signatures, access keys beyond safe identifiers, or raw credential-bearing responses. Check custom Debug impls and merged-config dumps too. Construct the error path (invalid signature, failed auth) and confirm it does not log the secret or the derived authenticator. Verify audit/notify entries redact credential request headers.
+  - Where: rustfs/src/**, crates/iam/, crates/audit/, crates/notify/, crates/targets/, RPC signature error paths
+  - Evidence: GHSA-r54g (STS creds logged at info), GHSA-8cm2 (debug logs leaked tokens/secrets/JWT claims/raw STS bodies), GHSA-333v (invalid RPC signature log included HMAC secret + expected signature). Fix commit ee6f79110 (redact credential request headers from audit/notify, backlog#963). rustfs-logging-governance skill.
+- For any struct in the diff deserialized from untrusted input (S3 XML/JSON, lifecycle rules, bucket policy, replication config, RPC payload), check for #[serde(deny_unknown_fields)]. Construct a payload with a typo'd field (e.g. 'NoncurentDays') or an extra field and prove it is rejected, not silently ignored. Flag #[serde(default)] on security-critical fields (retention days, limits, permissions) lacking explicit post-deserialize validation, and any user-controlled integer cast with `as` (i32 as u32) — feed a negative value and check it doesn't wrap to a huge positive.
+  - Where: crates/policy/ (bucket policy), crates/ecstore lifecycle/ILM config, replication config structs, crates/protocols XML parsing
+  - Evidence: AGENTS.md 'Serde Safety'; advisory-patterns.md 'Serde deserialization' (no deny_unknown_fields found repo-wide; 'NoncurentDays' typo silently accepted; i32 as u32 wrap). Fix commit 1acd47f15 (SSE crash-loop DoS + credential reserved-char bypass, backlog#806).
+- If the diff touches SSE / encryption reader-writer composition, do not trust API metadata claiming encryption. Trace the reader wrapper order (HashReader / EncryptReader / compression / warp) and confirm EncryptReader is actually in the chain that writes to disk — construct the case where a helper unwraps a nested reader and bypasses encryption, storing plaintext. Require a regression test that inspects the ACTUAL stored bytes on disk, not just read-back. Also check encrypted-object checksums are not exposed.
+  - Where: rustfs/src/storage/ecfs.rs, crates/rio/ (reader wrappers), crates/kms/, SSE-C replication
+  - Evidence: GHSA-xrrf-67jm-3c2r (SSE metadata reported encryption while composition bypassed EncryptReader, stored plaintext). Fix commits a7b9659e7 (hide encrypted object checksums, #4529), 80cc3b1fc (preserve SSE-C checksum state, #4410). advisory-patterns.md 'SSE and on-disk storage invariants'.
+- If the diff touches CORS or the console/browser/object-preview surface: confirm default CORS does not reflect an arbitrary Origin while also sending Access-Control-Allow-Credentials: true — construct a request with a spoofed Origin and check the response. For preview, confirm attacker-controlled object content is origin-isolated (not rendered in a same-origin iframe with console creds), served with nosniff/CSP, and that preview trust derives from validated content-type + sandboxing, NOT from object name/extension (.pdf, .html). Separately, if aws:SourceIp is evaluated, spoof X-Forwarded-For / X-Real-IP as a direct (non-trusted-proxy) client and confirm the socket peer IP is used instead.
+  - Where: rustfs/src/server/layer.rs (CORS), console preview/auth code, aws:SourceIp / policy condition evaluation, X-Forwarded-For handling
+  - Evidence: GHSA-x5xv-223c-8vm7 (default CORS reflected arbitrary origins with credentials), GHSA-v9fg-3cr2-277j (preview rendered attacker HTML same-origin, exposed localStorage creds), GHSA-7gcx-wg4x-q9x6 (extension-based PDF detection bypassed sandbox), GHSA-fc6g-2gcp-2qrq (aws:SourceIp trusted client XFF). advisory-patterns.md 'Browser, CORS' + 'Trusted proxy'.
+
+Null report example: "Attacked admin action-constant matching in the two changed handlers (both call validate_admin_request with the exact AdminAction), the FTPS RETR/SIZE authz parity, and the new lifecycle struct's serde surface (has deny_unknown_fields) — no break found."
+
+### Concurrency/durability reviewer
+
+- For every new or moved lock acquisition, enumerate all other code paths that take any overlapping subset of those locks and construct the concrete ABBA interleaving (thread 1 holds A wants B, thread 2 holds B wants A). If the diff acquires 2+ locks without a comment documenting acquisition order, that alone is a finding.
+  - Where: crates/ecstore/** (namespace locks, set_disk, disk registry), crates/audit/**, crates/lock/**, any Mutex/RwLock pair in a diff
+  - Evidence: crates/ecstore/AGENTS.md 'Lock Ordering' (document order; same set in different orders = deadlock); real ABBA deadlock fixed in c0d5f938f (#4421, audit registry vs stream_cancellers)
+- If the diff touches the object write/commit path or a lock guard's lifetime, construct the timeline where the distributed lock is lost (heartbeat refresh fails / expiry) after shard writes but before the xl.meta rename commit — verify the commit is fenced on guard.is_lock_lost() (set_disk/ops/object.rs:874-880) and the diff does not move the commit outside the fenced region or drop the guard early.
+  - Where: crates/ecstore/src/set_disk/ops/object.rs, ops/multipart.rs, crates/lock/**
+  - Evidence: 1e6207c08 (#4406) fence write commit on lock loss; ddf197ba5 (#4388) heartbeat lock refresh; backlog#899 fencing comment in object.rs
+- For any change to file creation or write-then-rename: write out the exact syscall order (write tmp -> fdatasync tmp -> rename -> fsync parent dir -> fsync ancestor dirs on first object under a prefix) and simulate a power cut after each step. Flag any dropped/reordered sync, and check the change honors the durability gate (RUSTFS_DRIVE_SYNC_ENABLE, strict/relaxed/none modes, per-bucket overrides) instead of hardcoding one mode. The 'skip tmp parent fsync' optimization is only sound when the file is renamed out of tmp — verify that precondition still holds.
+  - Where: crates/ecstore/src/disk/local.rs, disk/os.rs, disk/fs.rs, crates/ecstore/src/bucket/durability.rs, set_disk/core/io_primitives.rs
+  - Evidence: PR #4221 (the repo previously had no fsync anywhere); 2df315baf/c081586e7 (#4493) fsync ancestor dirs; 062a68d15 (#4387) tmp-parent-fsync skip is rename-conditional; eaff17cad (#4397) durability modes; 54872d52d (#4478) rename_data crash harness exists — extend it for the diff
+- If the diff touches quorum counting or per-disk error aggregation, construct adversarial error vectors for reduce_errs: nil/placeholder entries, DiskNotFound mixed with FileNotFound, exactly quorum-1 agreeing errors — and show which dominant error wins. Specifically attack the case where offline-disk errors get counted as 'object does not exist', flipping a read/heal decision into data loss. Also check quorum monotonicity: a retry or heal pass must never conclude with a LOWER quorum than the original write.
+  - Where: crates/ecstore/src/disk/error_reduce.rs, crates/ecstore/src/api/mod.rs, set_disk read/heal paths
+  - Evidence: 20d61c73b (#4551) reduce_errs leaked nil placeholder as dominant error; e0619e355 (#4536) DiskNotFound treated as object-not-found in listing; quorum monotonicity flagged as open follow-up to #4221 (#4221 follow-up list); crates/ecstore/AGENTS.md 'Do not weaken quorum checks'
+- For any multi-disk fan-out (delete, rename, heal write, cleanup), trace each per-disk Result: find any `let _ =`, `.ok()`, or best-effort collapse that keeps a failed disk out of the quorum math. Construct the run where exactly write_quorum-1 disks succeed and prove the op still returns success — that is the bug. Conversely, for heal writes, check one bad target cannot fail the whole heal (best-effort per target).
+  - Where: crates/ecstore/src/set_disk/ops/*.rs, disk/disk_store.rs, crates/heal/**
+  - Evidence: f7d2b2563 (#4546) disk delete/rename failures were swallowed; 47c1e730c (#4545) heal write quorum made best-effort per target; 2b063b0c4 (#4400) disk-replacement heal missed versions
+- For every new .await placed between a state mutation and its cleanup/commit (or inside select!/timeout/spawned task that can be aborted), construct the cancellation point: client disconnects and the future is dropped exactly there. Enumerate what is left behind — tmp files, incremented counters never decremented, half-written xl.meta, a held permit/waiter — and verify cleanup runs in Drop or the state is re-entrant. Background loops the diff adds must have a hard outer timeout so a wedged awaitee cannot pin them forever.
+  - Where: crates/ecstore/** write paths, crates/object-capacity/** scanners, crates/audit/**, anything using tokio::select! or spawn+abort
+  - Evidence: d608e320f io_uring cancel-safety spike (cancellation known-hard here); 5a372557e (#4533) wedged scans needed hard outer timeout; 7b87d4d13 (#4520) waiter-count leak; e44bece00 (#4497) audit start race + paused drops
+- If the diff touches metacache/list producers or cursor resume, construct the interleaving where the producer task completes (or errors) while a reader with a saved cursor comes back for the next page — verify a completed producer is tolerated (no error, no hang) and that a re-folded/full page reports truncation instead of silently ending the listing early. Also feed a corrupt/oversized length prefix into any metacache decode the diff touches.
+  - Where: crates/ecstore/src/cache_value/metacache_set.rs, crates/ecstore/src/store/list_objects.rs, crates/filemeta/**
+  - Evidence: 91a23361e (#4531) tolerate completed metacache producers; d91f4d455 (#4538) delimiter re-fold dropped truncation flag; d2c100fd3 (#4226) corrupt length-prefix guard
+- For multipart changes, construct concurrent operations on the SAME uploadId: put_object_part racing put_object_part (same part number), abort racing complete between the parts listing and the commit rename, and list-parts racing cleanup. Verify every metadata read/list/abort holds the per-uploadId lock, and that part.N.meta cleanup is deferred until AFTER the commit rename — cleanup before commit loses parts on a crash between the two.
+  - Where: crates/ecstore/src/set_disk/ops/multipart.rs, rustfs/src/storage/
+  - Evidence: 3bc8d79fe (#4329) serialize put_object_part per uploadId; 7fb95d4fc (#4428) unlocked upload metadata reads/aborts; 93ffbdb9b (#4437) unlocked part listings; c77c5f047 (#4548) part.N.meta cleanup moved after commit
+- Any cleanup/rollback logic added near a commit: verify it runs strictly AFTER the commit is durable, is best-effort (its failure must not fail an already-committed write), and is safe under retry — i.e., re-running it after a partial first attempt must never delete the newly-committed data dir or the last surviving copy.
+  - Where: crates/ecstore/src/set_disk/ops/object.rs (rename_data tail), ops/multipart.rs, disk cleanup helpers
+  - Evidence: afc7f1d6f/d908243e6 (#4386, backlog#898) post-commit old-data-dir cleanup had to be made best-effort; e7cc719c1 (#4389) speculative tmp cleanup moved off hot path
+- If the diff does read-modify-write on any persisted shared state (bucket metadata, notify/target config, queue store), construct two concurrent writers: show whether the second write silently discards the first (lost update) — RMW must be serialized or CAS-guarded. For persisted queues/replay, construct crash-mid-replay and prove entries are neither lost nor delivered twice without an idempotency key.
+  - Where: crates/notify/**, crates/targets/** (queue store, SQL backends), crates/ecstore/src/bucket/metadata_sys.rs
+  - Evidence: 2490d4ee2 (#4425) persisted config RMW lost updates; 08e44b95f (#4505) queue store crash-safety + replay lifecycle; e008cc5da (#4500) SQL backend idempotency
+- If the diff touches erasure decode/reconstruct or streaming GET, construct the failure mid-stream: shards become inconsistent (or a disk read fails) after N bytes of the body have already been sent — verify the stream surfaces an error to the client instead of ending cleanly at a truncated length. Silent truncation on a 200 response is the known failure mode.
+  - Where: crates/ecstore/src/set_disk/read.rs (reconstruct-read validation, historically ~line 3117), crates/ecstore/src/erasure/coding/decode.rs
+  - Evidence: Known open bug: EC reconstruct-read 'inconsistent shards' mid-GET silently truncates body -> client unexpected EOF; crates/ecstore/AGENTS.md 'explicit failure over silent corruption'
+
+Null report example: "Attacked lock ordering on the new disk-registry mutex pair, lock-loss fencing across the moved commit, power-cut points around the added rename, and cancellation at the two new awaits — no break found; quorum math and metacache paths untouched by this diff."
+
+### Compatibility reviewer
+
+- Grep the diff for raw 'x-rustfs-internal-' or 'x-minio-internal-' string literals used with map.insert/remove/get instead of the metadata_compat helpers. If found, construct the MinIO-written object case: a metadata map containing ONLY 'X-Minio-Internal-<suffix>' (mixed case, no RustFS key) and trace the diff's read path — does it miss the value? Then construct the removal case: does remove leave the twin key behind so a stale MinIO-key value resurrects on next read? Also check the value-type trap: get_bytes has NO case-insensitive fallback (unlike get_str), so a diff that moves a suffix from FileInfo.metadata (String) to meta_sys (Vec<u8>) silently loses mixed-case MinIO keys.
+  - Where: Any code touching FileInfo.metadata / user_defined / meta_sys: crates/ecstore/, crates/filemeta/, rustfs/src/storage/, crates/utils/src/http/metadata_compat.rs
+  - Evidence: Repo-wide invariant in AGENTS.md 'Cross-Cutting Domain Invariants' and CLAUDE.md; helpers and the asymmetry are pinned by tests test_str_lookup_accepts_minio_metadata_case and test_get_bytes_no_case_insensitive_fallback in crates/utils/src/http/metadata_compat.rs
+- For any diff reading a binary UUID from internal metadata (transitioned-versionID, tier-free-versionID, data_dir), trace the three degenerate inputs — key absent, value empty (b""), value nil UUID — through to the outgoing tier/S3 request. The bug shape to hunt: unwrap_or_default() or Uuid::from_slice(..).unwrap_or(Uuid::nil()) turning 'no value' into Uuid::nil(), which then gets serialized as ?versionId=00000000-... and the remote tier returns NoSuchVersion. The required pattern is .and_then(|v| Uuid::from_slice(&v).ok()).filter(|u| !u.is_nil()).
+  - Where: crates/ecstore/src/bucket/lifecycle/ (bucket_lifecycle_ops.rs, tier_sweeper.rs), crates/ecstore/src/services/tier/warm_backend_*.rs, crates/filemeta/src/filemeta/version.rs
+  - Evidence: Historical production bug documented in docs/operations/tier-ilm-debugging.md ('Nil-UUID versionId sent to tier'); regression tests live in crates/filemeta/src/filemeta/version.rs; follow-up fix 726f3dc18 'accept empty remote version_id in tier recovery paths' (#4552)
+- For any diff touching tier or replication GET/DELETE against a remote S3 target, enumerate BOTH directions of the versionId contract and trace each: (a) remote version None/"" means the tier bucket is unversioned — the request must carry NO versionId parameter at all (not an empty one, not nil); (b) remote version Some(v) on a versioned target — the versionId MUST be sent, especially on version-purge deletes, or the delete lands on the wrong version / creates a delete marker instead of purging. Check whether the diff collapses these cases through a single Option/String conversion that loses the distinction.
+  - Where: crates/ecstore/src/services/tier/warm_backend*.rs, crates/ecstore/src/bucket/lifecycle/tier_sweeper.rs, replication code under crates/ecstore/src/bucket/
+  - Evidence: Invariant in AGENTS.md and docs/operations/tier-ilm-debugging.md; real bug fixed by 0fad35645 'send versionId on version-purge deletes to generic S3 targets' (#4401) — the versioned direction, and #4552 — the unversioned direction
+- If the diff changes xl.meta encoding (adds/reorders msgpack header fields, touches FileMeta::marshal_msg or codec.rs encode paths), attack downgrade and cross-vendor parse: encode an object with the new code and decode it with (a) the meta_ver<=3 read path and (b) the real-MinIO fixture tests from #4377. Then check the header signature: is it recomputed over the new bytes, or copied/hardcoded? MinIO validates it; a stale or zero signature makes MinIO reject the file. Finally verify XL_META_VERSION was not silently bumped — old RustFS/MinIO nodes reject meta_ver > 3 during a rolling upgrade.
+  - Where: crates/filemeta/src/filemeta.rs (XL_HEADER_VERSION/XL_META_VERSION, lines ~46-54), crates/filemeta/src/filemeta/codec.rs (check_xl2_v1, decode_xl_headers)
+  - Evidence: Real bug 073bc9675 'compute header signature instead of hardcoding zero' (#4343); format contract pinned in docs/architecture/minio-file-format-compat.md (write meta_ver 3, read <=3, XL2 magic); parity fixtures from a91d9cefc (#4377)
+- If the diff touches xl.meta / FileInfo decode (into_fileinfo, version parsing, part arrays), construct hostile foreign input: a MinIO- or corruption-shaped msgpack with a parts-count that disagrees with the etags/sizes array lengths, missing optional fields, and a meta_ver 2 object with legacy checksum. Trace whether the new code indexes past an array, panics, or fabricates default values instead of returning a decode error. Run the pinned legacy fixtures (test_issue_2265_legacy_meta_v2_object_compatibility, test_issue_2288) plus the #4377 real-MinIO xl.meta parse tests against the diff.
+  - Where: crates/filemeta/src/ (fileinfo.rs, filemeta.rs, filemeta/codec.rs, filemeta/version.rs)
+  - Evidence: Real bug 7efacbdf9 'validate part array lengths in into_fileinfo' (#4382); legacy meta_ver 2 regression fixtures at crates/filemeta/src/filemeta.rs (~:1130-:1174) cited by docs/architecture/minio-file-format-compat.md
+- If the diff 'corrects' a formula, constant, or layout that is a byte-for-byte MinIO port (shard-size math, bitrot hash interleaving, erasure distribution, inline-data prefix), treat the correction itself as the bug: verify against legacy on-disk data before accepting. Concretely: run crates/ecstore/tests/legacy_bitrot_read_test.rs and the ECA-18 pinning tests; check whether existing objects written by old RustFS or MinIO still verify byte-for-byte. Known trap examples: bitrot_shard_file_size's bare return for non-streaming algorithms is CORRECT MinIO whole-file behavior, and the 32-byte prefix on inline data is the HighwayHash256 bitrot hash, not corruption.
+  - Where: crates/ecstore/src/erasure/coding/bitrot.rs, crates/ecstore/src/io_support/bitrot.rs, crates/filemeta/src/fileinfo.rs
+  - Evidence: f96314a1d 'pin streaming-only bitrot layout invariant (ECA-18)' (#4553) — audit explicitly decided NOT to change the formula because it breaks legacy interop; #4377 proved the inline-data prefix is the bitrot hash
+- For any diff that copies object metadata into an S3-client-visible surface (GET/HEAD response headers, notification event userMetadata, ListObjects/replication payloads, copy-object metadata directives), construct an object carrying internal keys under BOTH prefixes and in non-canonical casing ('X-Minio-Internal-Compression') and confirm every one is stripped via is_internal_key (which is case-insensitive) — not by an exact-match filter on one prefix. Leaked internal keys are an API-semantics break and an information leak.
+  - Where: rustfs/src/storage/, crates/notify/, replication and copy_object paths in crates/ecstore/
+  - Evidence: Real bug cf8929189 'strip rustfs/minio internal metadata from event userMetadata' (#4419); is_internal_key contract in crates/utils/src/http/metadata_compat.rs
+- If the diff touches crates/protos (node.proto, models.fbs) or internode RPC request/response structs, attack the rolling-upgrade interleaving: an old node sends a message without the new field to a new node, and a new node sends the extended message to an old node. Verify proto field numbers are only appended (never reused/renumbered), FlatBuffers tables are only extended at the end, and that an absent new field decodes to a safe default on the receiving side — 'safe' meaning it must not be interpreted as success/authorization (RPC errors fail closed) and must not flip a quorum decision.
+  - Where: crates/protos/ (node.proto, models.fbs, generated/), gRPC transport and dispatch in the internode layer
+  - Evidence: 6f613317f 'optimize gRPC transport' (#4337) shows the wire layer churns; security advisory 68cw fixed by PR#4402 established RPC fail-closed as a repo rule (see .agents/skills/security-advisory-lessons)
+- For S3 handler diffs, replay the request shapes real clients actually send, not just the canonical one: mc and aws-sdk differ on path normalization (root '//' ListBuckets), virtual-host vs path style, and header casing. Then attack every pagination boundary the diff touches: request exactly max-keys/max-uploads/max-parts items and verify the response returns exactly N (not N+1), sets IsTruncated correctly, and yields a NextMarker/KeyMarker that resumes without skipping or duplicating — construct the N+1st-item case explicitly.
+  - Where: rustfs/src/storage/ S3 handlers, listing paths in crates/ecstore/src/store/ and set_disk/
+  - Evidence: Real bugs 511ad31ba 'normalize root double-slash ListBuckets requests' (#4336) and fefa70b31 'stop ListMultipartUploads from returning one upload past max-uploads' (#4447); docs/architecture/s3-compatibility-matrix.md is the compat source of truth
+- If the diff changes bucket-metadata (.metadata.bin) or IAM/config parsing structs, run it against the real MinIO RELEASE.2025-07-23 fixtures: the msgpack blob uses PascalCase field names, so any serde rename, field-type change, or derive tweak silently drops MinIO-written fields instead of erroring. Verify parse_all_configs still loads all ten config types from the fixture without loss, and that drop-in migration still decrypts MinIO-encrypted IAM/server config rather than treating ciphertext as corrupt.
+  - Where: crates/ecstore/src/bucket/metadata*, crates/ecstore/src/bucket/migration.rs, IAM/config load paths in crates/iam/ and crates/config/
+  - Evidence: Fixture parity test parses_real_minio_bucket_metadata_blob_without_loss from a91d9cefc (#4377); real bug 717cdd2ab 'decrypt MinIO IAM & server config on drop-in migration' (#4358); format matrix in docs/architecture/minio-file-format-compat.md
+- If the diff adds a compatibility shim, legacy fallback, wrapper, or old-endpoint alias (grep the diff for 'legacy', 'fallback', 'compat', 'deprecated'), verify two things: (1) it carries a RUSTFS_COMPAT_TODO(<task-id>) marker with an exact removal condition and a matching entry in the register — an unmarked shim becomes permanent dead weight; (2) the fallback's default direction is safe for old data: e.g. a new decode path must fall back to the legacy decode for old objects by default, not gate legacy reads behind an opt-in flag that makes existing data unreadable after upgrade.
+  - Where: Anywhere in the diff; register at docs/architecture/compat-cleanup-register.md; recent example: allow_inplace_legacy_fallback flag in the ecstore erasure codec streaming path
+  - Evidence: docs/architecture/compat-cleanup-register.md review checklist; d232a46b4 wired legacy decode prefetch behind a default-OFF gate while keeping legacy reads working (#4542), with arity fallout fixed in 05890d6e2 (#4573)
+
+Null report example: "Attacked dual-key metadata writes/removals against MinIO-only-key objects, nil/empty transitioned-versionID paths to the tier, xl.meta encode against meta_ver<=3 decoders and the #4377 real-MinIO fixtures, and proto field-number evolution for old-node/new-node RPC — no compatibility break found."
+
+### Performance reviewer
+
+- For every `.clone()` the diff adds or moves onto a per-request/per-object path, open the cloned type and count heap fields (String, Vec, HashMap, Bytes). If >5 heap fields or it contains an EC block buffer, construct the cost: N concurrent PUTs x M objects -> N*M deep copies per second. Demand Arc-wrapping of heavy fields or pass-by-reference; also flag new `String` allocations in header/path/signature parsing where `&str`/`Cow<str>` suffices.
+  - Where: crates/ecstore/src/set_disk/**, crates/ecstore/src/store*.rs, rustfs/src/storage/, crates/filemeta/, request handlers in rustfs/src/
+  - Evidence: crates/ecstore/AGENTS.md 'Allocation Discipline in Hot Paths' (no Clone on >5-heap-field structs, Arc for large buffers, &str/Cow for temporary computations); .agents/skills/rust-code-quality/SKILL.md ranks 'unnecessary clone in hot path' as P1 must-fix
+- For every new sync_all/sync_data/fdatasync/flush/File::sync call in the diff, trace the call chain to DurabilityMode / RUSTFS_DRIVE_SYNC_ENABLE resolution (crates/ecstore/src/disk/local.rs:291 DurabilityMode, :347 resolve_durability_mode) and to per-bucket durability overrides. Construct the run where the operator sets mode=none (or legacy RUSTFS_DRIVE_SYNC_ENABLE=false) and the new fsync still fires — that is an ungated durability cost and a regression on 4KiB writes.
+  - Where: crates/ecstore/src/disk/local.rs, crates/ecstore/src/bucket/durability.rs, crates/ecstore/src/set_disk/** (rename_data/commit paths), any crate doing tokio::fs or std::fs writes
+  - Evidence: #4221 fsync work caused a measured -10% 4KiB write regression (#814 investigation), later gated; durability modes added in eaff17cad (#4397), per-bucket tier overrides in 13e48d93a (#4407); 2df315baf (#4493) shows even ancestor-dir fsyncs are routed through the gate
+- Attack blocking-work placement from both directions: (a) find new synchronous fs calls, hashing, or EC encode/decode executed directly on an async runtime thread without spawn_blocking/block_in_place — construct the stall (a slow disk blocks a worker thread and every task queued on it); (b) find new code that splits one logical disk operation into multiple spawn_blocking hops per object — each hop is a threadpool round-trip, so K hops x N objects multiplies latency. Demand the author justify the placement with the size of the work, not habit.
+  - Where: crates/ecstore/src/disk/local.rs, crates/ecstore/src/erasure_coding/, crates/ecstore/src/bitrot/, crates/rio/
+  - Evidence: 608ab14d7 (#4554, HP-12) folded metadata open+fstat+read into a single spawn_blocking because per-op hops were measurably slow; 8fc637fb1 (#4484) moved the short EC encode inline because block_in_place cost exceeded the work — direction depends on measured work size
+- For each lock acquisition the diff adds or relocates, mark the guard's live range and list every .await and disk/RPC call inside it. Construct the contention interleaving: N concurrent requests serialize on the guard while the holder waits on IO; for namespace/multipart commit locks, compute worst-case hold time (fsync + rename per disk) against the lock's timeout. Also diff the acquisition order against other paths taking the same locks (ABBA).
+  - Where: crates/ecstore/src/set_disk/** (commit/rename paths), crates/lock/, crates/audit/ registry, any RwLock/Mutex in per-request paths
+  - Evidence: crates/ecstore/AGENTS.md 'Lock Ordering'; c0d5f938f (#4421) fixed a real ABBA deadlock between registry and stream_cancellers; #4370 history: fsync-heavy serial cross-disk commits held a lock long enough to blow test timeouts (#4370)
+- Trace exactly what executes inside the PUT commit critical section (under the object write lock, between tmp write and rename_data completion) before vs after the diff. Any newly added work there — cleanup, extra stat, additional rename, O_DIRECT write, logging — is an attack target: construct the per-PUT latency delta and demand it be moved off the critical section or parallelized across disks.
+  - Where: crates/ecstore/src/set_disk/ops/*, crates/ecstore/src/disk/local.rs rename_data path
+  - Evidence: Three real optimizations removed exactly this class of regression: e7cc719c1 (#4389) moved speculative PUT-tail tmp cleanup off the hot path, 92c8c6db7 (#4411) moved O_DIRECT shard-writes off the commit critical section, 651ccac13 (#4487) parallelized tmp xl.meta write and shard fdatasync on commit
+- Find any new loop in a batch API that performs a per-item stat/read/RPC sequentially. Construct the concrete blowup: a 1000-key DeleteObjects or a full listing page -> 1000 serial round-trips added by the diff. Demand either a gate (skip when not needed) or bounded parallelism; for startup/load paths, check for accidental O(n^2) (re-scanning the full list per item).
+  - Where: crates/ecstore/src/store_delete*.rs / batch object APIs, listing/metacache paths, crates/iam/ store loading, crates/heal/
+  - Evidence: a413729b1 (#4398) had to gate and parallelize the DeleteObjects per-object stat fanout after it shipped serial; 16a91c35e (#4537) fixed O(n^2) IAM startup load by chunking — both were diff-introduced fanouts of this exact shape
+- For every buffer the diff allocates on the encode/decode/shard path, check: is it sized with with_capacity to the EC-expanded block (not the logical size, not default-grown)? Does it copy into a fresh Vec where Bytes::slice/clone (refcount) or the io-core buffer pool would avoid the copy? Does the diff read hash and data in separate passes where one pass suffices? Construct the per-block byte-copy count before vs after. If the diff touches the io-core pool, verify gauge accounting still balances.
+  - Where: crates/ecstore/src/erasure_coding/, crates/ecstore/src/bitrot/, crates/io-core/src/pool.rs, crates/rio/
+  - Evidence: 92bf55ce6 (#4396) fixed a real regression by right-sizing BytesMut encode ingest capacity to the EC-expanded block; 47bee8b31 (#4475) merged bitrot hash+data into one read pass; 7fa3d0d4b (#4534) shows pool gauge accounting is easy to drift when touching buffer reuse
+- For every logging or instrumentation statement the diff adds, classify the call site frequency: per-request, per-object, per-shard, or per-block. Anything info!/warn!/error! at per-object frequency or higher is a finding — construct the flood (one listing under client cancellation, one 10k-object heal) and count emitted lines. New metrics/timers on the data path must be feature-gated, not always-on. Run scripts/check_logging_guardrails.sh on the diff.
+  - Where: any per-request/per-object code, especially crates/ecstore listing and heal loops, rustfs/src/storage/ handlers; scripts/check_logging_guardrails.sh
+  - Evidence: .agents/skills/rustfs-logging-governance/SKILL.md (trace level for hot-path/repetitive success events); d25ddb0e1 (#4372) fixed real listing-cancellation error-log noise; hotpath instrumentation is deliberately feature-gated (3f13d098b #4394, f262fcfce #4541 HP-14)
+- Count how many times the diff's request path parses or fetches the same metadata: xl.meta/FileMeta decoded more than once per object, bucket metadata (metadata_sys) re-fetched inside a per-object loop, or the dual x-rustfs-internal/x-minio-internal key lookup re-run repeatedly on the same map. Construct the per-request parse count before vs after; a second full FileMeta decode per GET is a finding.
+  - Where: crates/filemeta/, crates/ecstore/src/set_disk/** read paths, crates/ecstore/src/bucket/metadata_sys.rs, crates/utils/src/http/metadata_compat.rs
+  - Evidence: 608ab14d7 (#4554) exists because redundant metadata-read syscall sequences per object were measurable; CLAUDE.md dual-key metadata convention makes repeated get_bytes lookups an easy hidden double-parse
+- If the diff touches PUT/GET/commit/erasure paths and claims 'no perf impact', demand numbers, not assertion: run the criterion benches (cargo bench -p ecstore — comparison_benchmark, erasure_benchmark, rename_data_meta_benchmark, single_block_non_inline_benchmark per crates/ecstore/benches/) against origin/main, and for end-to-end paths the warp A/B relative-budget gate (scripts/run_hotpath_warp_ab.sh --baseline-ref origin/main, as .github/workflows/performance-ab.yml runs it). Probe specifically at 4KiB object size — that is where the last real regression hid.
+  - Where: crates/ecstore/benches/, .github/workflows/performance-ab.yml, scripts/run_hotpath_warp_ab.sh
+  - Evidence: crates/ecstore/AGENTS.md: 'Benchmark-sensitive changes should include measurable rationale'; performance-ab.yml (215747022 #4480) is the repo's own relative-budget gate; the #4221 regression was only visible at 4KiB writes (#814 bisect)
+
+Null report example: "Attacked the new rename_data commit-section work, durability-gate routing of the added fdatasync, guard live-range across the shard-write awaits, and per-object clone count in the PUT path; ran comparison_benchmark + rename_data_meta_benchmark vs origin/main (4KiB delta within noise) — no break found."
+
+### Test-coverage skeptic
+
+- For every behavior claim in the PR description, revert that hunk (git stash / manual undo of the changed lines) and name the exact test (`cargo test -p <crate> <test_name>`) that fails. If no test fails on revert, the behavior is untested — file a finding, not a note. Especially verify the test exercises the REAL production call path, not a lookalike helper.
+  - Where: All crates; highest value in crates/ecstore, rustfs/src/storage, crates/heal
+  - Evidence: AGENTS.md exit criterion 'Every behavior change has a test that fails without it'. Real bug: PR #4220 (ghost-directory cleanup) merged with green tests but its fix never executed on the real delete path — required follow-up rustfs#4307, backlog#798 stayed OPEN. The tests exercised a path the production flow never took.
+- Read each added/modified test and confirm it asserts the real outcome (returned value, stored bytes, error variant), not merely 'call succeeded' or 'no panic'. Flag any test whose only observable is that the function returned, and any `assert!(result.is_err())` that never checks WHICH error. Then check: does the test prove the exploit/failure form is denied, or only that the intended form still works?
+  - Where: crates/e2e_test (security_boundary_test.rs pattern), and every #[cfg(test)] module in the diff
+  - Evidence: Commit dee8e4e63 (#4466) had to rewrite 277 lines of crates/e2e_test/src/security_boundary_test.rs because 'security boundary tests' passed without asserting real outcomes. .agents/skills/rust-code-quality/SKILL.md checklist: 'Every test function has at least one assert!'; .agents/skills/security-advisory-lessons/SKILL.md: 'Does the test prove the exploit form is denied, or only that the intended form still works?'
+- When the diff adds a boolean/mode parameter or config flag, find the test that fails if the flag's effect is INVERTED inside the changed function. Tests that were mechanically updated to pass `false`/default at every call site assert nothing about the new behavior. Execute the check: flip the flag's branch in the source and confirm at least one test goes red for each branch.
+  - Where: crates/ecstore/src/set_disk/ (e.g. build_codec_streaming_part_reader), any function gaining a parameter
+  - Evidence: Commit 05890d6e2 (#4573): PR #4560 added a 15th param allow_inplace_legacy_fallback; the arity tests were fixed by passing `false` everywhere — they assert Err outcomes independent of the flag, so the fallback behavior itself has no revert-detecting test at those sites.
+- Mutation spot-check on error propagation: for each newly added `?`, `return Err`, or error-mapping line, mentally replace it with `Ok(default)`/ignore and ask which test fails. The swallowed-error bug class recurs in this repo and always ships with green tests — a fix that propagates errors needs a test that injects the failure (faulty disk, failed rename, dispatch error) and asserts the caller sees Err.
+  - Where: crates/ecstore (disk delete/rename, reduce_errs), crates/audit, crates/notify, crates/targets
+  - Evidence: Three recent fixes for the same class: f7d2b2563 (#4546, disk delete/rename failures swallowed), dbc628f16 (#4424, audit dispatch failures swallowed), 20d61c73b (#4551, reduce_errs leaking nil placeholder as dominant error). All existed while tests were green.
+- Any test touching GET/read/reconstruct/stream paths must assert the FULL body content and exact length against a known value, not status-ok or first-bytes. Construct the degraded-read case (missing/inconsistent shards forcing EC reconstruction) and assert byte-for-byte equality; a mid-stream failure that truncates the body passes every test that only checks headers or the first chunk.
+  - Where: crates/ecstore/src/set_disk/read.rs and ops/, crates/rio, crates/e2e_test GET scenarios
+  - Evidence: Known live bug: EC reconstruct-read verification failure mid-GET at set_disk read path silently truncates the body → client 'unexpected EOF'; version-independent, undetected by existing suites because none assert full-body integrity under shard inconsistency.
+- For on-disk / on-wire format changes (xl.meta, .metadata.bin, bitrot framing), reject round-trip-only tests: a struct serialized and deserialized by the same code under test cannot catch format drift. Demand the test parse a REAL captured fixture from crates/filemeta/tests/fixtures, crates/ecstore/tests/fixtures, or crates/rio-v2/tests/minio_fixture_lab — or capture a new one from a single-disk MinIO instance (RELEASE.2025-07-23 procedure from #4377).
+  - Where: crates/filemeta, crates/ecstore (headers, msgpack bucket metadata), crates/rio-v2, migration code
+  - Evidence: Commit 073bc9675 (#4343): filemeta header signature was hardcoded to zero — round-trip tests passed for months. Commit a91d9cefc (#4377) established the real-MinIO fixture convention (inline/versioned/multipart xl.meta, HighwayHash256-prefixed inline bodies) precisely because synthetic fixtures proved nothing about interop.
+- Attack new concurrency tests for flakiness-by-construction: grep the added tests for `sleep(`, fixed timeouts under ~30s on lock acquisition, and use of shared global state (disk registry, lock client, GLOBAL_*). Serialized cross-disk commits exceed small lock timeouts under full-suite CI disk load. If the test shares global state or saturates IO, it must join the `ecstore-serial-flaky` nextest test-group in .config/nextest.toml (note: serial_test's #[serial] does NOT work — nextest runs each test in its own process). Require readiness polling, never fixed sleeps.
+  - Where: crates/ecstore tests, crates/e2e_test, .config/nextest.toml
+  - Evidence: Commit 2dfa3d3c3 (#4370): concurrent_resend test flaked with Lock(Timeout 5s) on CI — six legitimate serialized cross-disk commits under IO pressure needed 30s. Commit 7c701d9f2 (#4558) created the nextest test-group after bucket_delete_* raced make_bucket into InsufficientWriteQuorum. 65849740f (#4213) deflaked global-state contamination. crates/e2e_test/AGENTS.md: 'readiness checks and explicit polling over fixed sleep-based timing'.
+- If the diff writes internal object metadata, run the dual-key mutation: delete the `x-minio-internal-<suffix>` write (keeping only `x-rustfs-internal-`) and check whether any test fails. Because `get_bytes` prefers the RustFS key, every read-back test stays green while MinIO interop is silently broken — coverage must include an assertion that BOTH keys are present in the stored metadata map.
+  - Where: crates/utils/src/http/metadata_compat.rs and all its callers in crates/ecstore and rustfs/src/storage
+  - Evidence: CLAUDE.md domain convention: metadata must be written under both x-rustfs-internal- and x-minio-internal- keys for MinIO interop; get_bytes prefers the RustFS key, making the MinIO-key half of the invariant invisible to read-back tests.
+- For changed quorum/version/UUID logic, name the tests covering the specific poison values: quorum−1 disks, nil UUID, absent vs empty vs nil-serialized UUID bytes, remote-tier version_id of None/"" (unversioned tier bucket → no versionId sent), and the same metadata read on both MetaObject and MetaDeleteMarker version types. Mutation check: remove a `.filter(|u| !u.is_nil())` guard from the diff and confirm a test fails; if none does, the nil-UUID class is uncovered.
+  - Where: crates/ecstore (tier recovery, heal, quorum paths), crates/filemeta, code reading UUIDs from xl.meta metadata
+  - Evidence: Commit 726f3dc18 (#4552) fixed rejection of empty remote version_id in tier recovery. CLAUDE.md invariant: absent/empty/nil UUID all mean 'no value', not Uuid::nil(). docs/operations/tier-ilm-debugging.md: None/"" tier version means unversioned bucket. df9cbc4ed (#4427): unvalidated distribution values caused shuffle index panic — edge values reached production untested.
+- For any pagination/limit/truncation change, construct the exact-boundary test: result count == max (page exactly full), max+1, and a delimiter re-fold that lands precisely on the page boundary — assert both the item count AND the is_truncated/continuation marker. Off-by-one at the page boundary is a recurring shipped bug here.
+  - Where: crates/ecstore listing paths (list_objects, ListMultipartUploads, metacache), S3 handlers in rustfs/src/storage
+  - Evidence: Two shipped boundary bugs: fefa70b31 (#4447) ListMultipartUploads returned one upload past max-uploads; d91f4d455 (#4538) delimiter re-fold of a full page lost the truncation flag. Both survived existing tests because no test pinned n == max exactly.
+- Green `cargo test -p <crate>` on the touched crate is not a coverage verdict for the diff's test code itself: run `cargo clippy --all-targets -p <crate>` and a workspace-wide test BUILD (`cargo check --workspace --all-targets` at minimum) before accepting the tests as evidence. Test-only code that doesn't compile workspace-wide or fails clippy has repeatedly broken main and masked whether tests ran at all.
+  - Where: All crates; especially concurrent-branch merges into crates/ecstore
+  - Evidence: #4322 broke main because only cargo test ran (field_reassign_with_default is clippy-only). b06f3df6b (#4441) and 05890d6e2 (#4573): test code broke the workspace test build (E0061) on main after textually-clean merges, failing CI for every open PR.
+
+Null report example: "Attacked revert-detection for all 3 claimed behaviors (each has a named test that fails on revert), flag-inversion on the new fallback parameter (both branches covered in codec_streaming tests), full-body assertions on the changed GET path, and n==max pagination boundary — no coverage gap found."
+
+## Sources and maintenance
+
+Probes are distilled from shipped bugs in git history (commit/PR references
+above), GitHub security advisories (see the security-advisory-lessons
+skill), scoped `AGENTS.md` rules, and invariants under `docs/architecture/`
+and `docs/operations/`. Line numbers drift; when a cited location no longer
+matches, trust the invariant and re-locate the code. When a new bug class
+ships, add a probe with its evidence here rather than growing the policy
+section in `AGENTS.md`.

.agents/skills/adversarial-validation/references/compatibility.md

@@ -1,24 +0,0 @@
-# Compatibility Lens
-
-- Internal metadata uses `metadata_compat` helpers for dual RustFS/MinIO keys,
-  including mixed casing and removal of both twins.
-- Binary UUID metadata treats absent, empty, and nil as no value. Unversioned
-  remote tiers receive no `versionId`; versioned purge requests retain the real
-  version ID.
-- `xl.meta` changes preserve supported header/meta versions, recompute
-  signatures, decode legacy fixtures, and remain readable by old RustFS/MinIO.
-- Foreign/corrupt metadata validates parallel array lengths and missing fields;
-  it returns a decode error rather than indexing, panicking, or fabricating data.
-- Do not “correct” byte-for-byte MinIO ports without legacy fixture evidence.
-  Bitrot framing, shard math, distribution, and inline prefixes are contracts.
-- Client-visible metadata/events strip both internal prefixes
-  case-insensitively.
-- Proto fields are appended, never reused/renumbered; FlatBuffers tables extend
-  compatibly and absent new fields fail closed where authorization/quorum is
-  involved.
-- Replay real client request shapes and exact pagination boundaries for S3
-  handler changes.
-- Bucket metadata/IAM/config parsing remains compatible with pinned real MinIO
-  fixtures and encrypted migration data.
-- Compatibility shims use `RUSTFS_COMPAT_TODO(<task-id>)`, have a removal
-  condition, and default toward reading old data safely.

.agents/skills/adversarial-validation/references/concurrency-durability.md

@@ -1,23 +0,0 @@
-# Concurrency and Durability Lens
-
-- For every changed lock, enumerate overlapping lock sets and construct the
-  ABBA interleaving. Multiple-lock order must be documented and consistent.
-- Mark guard lifetimes and every `.await`, disk, and RPC call inside them.
-  Estimate contention and timeout behavior under concurrent requests.
-- Object commits remain fenced if the distributed lock is lost after shard
-  writes and before metadata rename.
-- For write/rename changes, trace `write tmp -> sync tmp -> rename -> sync parent
-  -> sync required ancestors`; simulate a crash after each step and honor the
-  configured durability gate.
-- Multi-disk fan-out counts every result. Quorum-minus-one cannot become success;
-  heal remains best-effort per target where that is the established contract.
-- At every new cancellable await between mutation and cleanup/commit, drop the
-  future and inspect leftover files, counters, permits, and replay state.
-- Multipart operations on the same upload ID are serialized where required;
-  abort/complete/list races cannot delete parts before durable commit.
-- Post-commit cleanup is best-effort, retry-safe, and cannot fail an already
-  committed write or delete the last surviving copy.
-- Persisted read-modify-write uses serialization/CAS. Queue replay is crash-safe
-  and duplicate delivery has an idempotency contract.
-- Streaming reconstruction failures after partial output surface as errors, not
-  successful EOF.

.agents/skills/adversarial-validation/references/correctness.md

@@ -1,29 +0,0 @@
-# Correctness Lens
-
-Attack the changed behavior, not every subsystem in the repository.
-
-- Trace new error paths to the caller. Inject the ignored/wildcard variants and
-  verify they cannot become success, not-found, or a plausible default.
-- Exercise zero/empty/missing, maximum, and exact-boundary inputs for every
-  changed count, size, index, page limit, or optional value.
-- For aggregation/quorum changes, test exactly quorum and quorum-minus-one with
-  mixed disk errors and nil/placeholder entries.
-- For listing/pagination, test `n == max`, `n == max + 1`, delimiter folding,
-  continuation markers, and object/prefix name collisions.
-- For EC/read/streaming changes, inject failure after partial output and verify
-  the client receives an error rather than a clean truncated body. Assert exact
-  bytes and length.
-- For multipart/object commits, fail before/after rename and cleanup; committed
-  data must remain readable and pre-commit cleanup must not destroy parts.
-- For version/index ordering, test `len - 1`, `len`, equal timestamps, missing
-  versions, and deterministic tie-breaking.
-- For directory-object behavior, trace `__XLDIR__` at the store layer; branches
-  below the layer that sees trailing slashes are dead.
-- For binary UUID metadata, absent, empty, and nil all mean no value. Never send
-  nil/empty `versionId` to an unversioned tier.
-- For agent rules/skill routers, test a trigger matrix covering ordinary
-  inquiry, low-risk implementation, explicit review, high-risk code, PR
-  creation, release, and post-PR monitoring. Each case must select only the
-  intended workflow and retain required safety/authorization boundaries.
-
-Null verdicts name only the probes relevant to the diff.

.agents/skills/adversarial-validation/references/performance.md

@@ -1,20 +0,0 @@
-# Performance Lens
-
-- For added clones/allocations on request/object/block paths, quantify copied
-  data and frequency. Recommend borrowing, move, `Bytes`/`Arc`, `Cow`, or
-  capacity reservation only for a concrete repeated cost.
-- Route every new sync/flush through the durability-mode and bucket override
-  gates; mode `none` must not pay the new fsync.
-- Keep blocking filesystem/CPU work off async runtime threads, but do not split
-  one small operation into many `spawn_blocking` round trips.
-- Measure lock hold time across I/O and compare acquisition order for ABBA.
-- Keep cleanup, extra stat/rename, and diagnostics out of the PUT commit critical
-  section when they need not be there.
-- Detect per-item serial I/O/RPC in batch APIs and accidental quadratic scans;
-  use a gate or bounded concurrency when the concrete fan-out warrants it.
-- Count buffer growth and byte copies in EC/bitrot paths; preserve pool gauge
-  balance and avoid repeated metadata decode/fetch per object.
-- Repetitive success logs stay at `trace`; metrics/instrumentation on hot paths
-  require an existing gate.
-- Claims of no impact on PUT/GET/commit/erasure paths need relevant benchmark or
-  A/B evidence, especially for 4 KiB objects.

.agents/skills/adversarial-validation/references/security.md

@@ -1,31 +0,0 @@
-# Security Lens
-
-Use `security-advisory-lessons` only for a dedicated advisory/security audit.
-For an ordinary matched diff, attack these boundaries:
-
-- Admin routes: route registration, whitelist, handler authn, and the exact
-  `AdminAction` must agree. Read-only diagnostics still require admin authz.
-- IAM/service accounts: treat parent, claims, keys, groups, status, and policy
-  names as attacker-controlled; prove ownership/root authority before writes.
-- Protocol frontends: every changed/sibling command authorizes the matching S3
-  action before reaching storage.
-- Secrets/signatures: use constant-time comparison, normalize public failures,
-  keep RPC/root/STS keys independent, and fail closed when secrets are absent.
-- RPC: bind signatures to the exact method/path and timestamp; reject replay,
-  stale, malformed, truncated, and invalid-enum payloads without panic.
-- Paths/object/archive entries: reject traversal, absolute/platform escapes,
-  and normalization differences between authz and storage.
-- Copy/multipart/presigned POST: enforce source, destination, version-aware
-  actions, copy-source conditions, and every signed policy condition.
-- Logging/errors: never expose credentials, tokens, expected signatures, raw
-  secret-bearing input, or merged configs—including via `Debug` and parse errors.
-- Untrusted serde: reject unknown fields where compatible and validate
-  security-critical defaults/ranges before numeric conversion.
-- SSE/browser/CORS/trusted proxy: inspect stored ciphertext and wrapper order;
-  isolate user content; never reflect credentialed arbitrary origins or trust
-  forwarded identity from direct clients.
-- Object Lock: unreadable/fabricated/unparsable metadata fails closed across
-  foreground, lifecycle, scanner, and force-delete paths.
-
-Security findings distinguish unauthenticated compromise from a
-low-privileged authenticated bypass.

.agents/skills/adversarial-validation/references/simplicity.md

@@ -1,22 +0,0 @@
-# Simplicity Lens
-
-- Compare the production diff with the smallest equivalent local edit. Fewer
-  lines alone are not evidence; the replacement must preserve correctness,
-  compatibility, readability, and real boundaries.
-- Search the touched crate, domain owner, `crates/utils`, `crates/common`, and
-  relevant dependencies for each new helper, constant, wrapper, or fixture.
-- Reject forced reuse when normalization, error, deadline, or durability
-  semantics differ.
-- Require a concrete trigger for every new defensive branch. Keep boundary
-  checks for disk/RPC/version data and checks immediately before destructive
-  actions.
-- Flag one-caller helpers only when they merely forward or split a short linear
-  flow without adding domain naming, invariant isolation, or useful context.
-- Ensure a replacement removes the superseded in-scope path or keeps one
-  canonical core behind a documented compatibility adapter.
-- Remove narration/change-history comments; preserve concise safety, lock,
-  durability, and compatibility invariants.
-- Treat tests, fixtures, generated code, and documentation separately from
-  production growth. Do not optimize away meaningful regression coverage.
-
-A finding must include a concrete smaller design, not a style preference.

.agents/skills/adversarial-validation/references/test-coverage.md

@@ -1,24 +0,0 @@
-# Test-Coverage Lens
-
-- For every behavior claim, name the focused test/check that fails if the
-  changed hunk is reverted. If none is practical, require the reason and
-  residual risk.
-- Confirm tests exercise the real production path and assert returned values,
-  exact bytes, stored state, or the specific error variant—not only success,
-  `is_err()`, or no panic.
-- For new flags/modes, verify each branch and ask which test fails if the branch
-  is inverted.
-- For new error propagation, inject the failure and assert the caller observes
-  it; mentally replacing `?`/`return Err` with success must break a test.
-- Streaming GET tests assert the complete body and length under degraded reads.
-- Disk/wire-format tests use pinned foreign/legacy fixtures; same-code
-  round-trips are insufficient for compatibility.
-- Concurrency tests use readiness polling, isolate global state, and avoid fixed
-  sleeps or unrealistically short timeouts. Use nextest groups when process-level
-  serialization is required.
-- Internal metadata tests assert both RustFS and MinIO keys, not only read-back
-  through a helper that prefers one key.
-- Boundary companions are distinct coverage: `n == max` vs `max + 1`, and
-  absent vs empty vs nil UUID.
-- A focused test proves only the targets/features it builds. Add compilation or
-  Clippy only for uncovered changed targets.

.agents/skills/code-change-verification/SKILL.md

@@ -1,12 +1,11 @@
 ---
 name: code-change-verification
-description: Review a commit, PR, or merged patch when the user requests ordinary code-change verification. Do not combine with adversarial-validation; use that skill instead for explicitly adversarial, substantial, or high-risk RustFS reviews.
+description: Verify code changes by identifying correctness, regression, security, and performance risks from diffs or patches, then produce prioritized findings with file/line evidence and concrete fixes. Use when reviewing commits, PRs, and merged patches before/after release.
 ---
 
 # Code Change Verification
 
-Use this skill for an ordinary requested review. If the root policy or user calls
-for adversarial validation, use `adversarial-validation` instead of running both.
+Use this skill to review code changes consistently before merge, before release, and during incident follow-up.
 
 ## Quick Start
 
@@ -80,3 +79,4 @@ Run the full checklist in [rust-code-quality](../rust-code-quality/SKILL.md) —
   - Impact: ...
   - Fix suggestion: ...
   - Validation: ...
+

.agents/skills/code-change-verification/agents/openai.yaml

@@ -1,4 +1,4 @@
 interface:
   display_name: "Code Change Verification"
   short_description: "Prioritize risks and verify code changes before merge."
-  default_prompt: "Use $code-change-verification for an ordinary requested diff review with prioritized findings."
+  default_prompt: "Inspect a patch or diff, identify correctness/security/regression risks, and return prioritized findings with file/line evidence and fixes."

.agents/skills/pr-creation-checker/SKILL.md

@@ -1,46 +1,94 @@
 ---
 name: pr-creation-checker
-description: Perform the final RustFS PR preflight and draft compliant English title/body metadata immediately before creating or updating a PR. Do not use during implementation or as a second general code review.
+description: Prepare PR-ready diffs by validating scope, checking required verification steps, drafting a compliant English PR title/body, and surfacing blockers before opening or updating a pull request in RustFS.
 ---
 
 # PR Creation Checker
 
-Use this skill only at the PR boundary. Reuse completed diff review and
-verification evidence; do not reread the repository or rerun equivalent checks.
+Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whether a branch is ready for PR.
 
-## Preflight
+## Read sources of truth first
 
-1. Confirm the branch is based on current `origin/main` and contains only the
-   intended task diff.
-2. Inspect `git diff --stat`, `git diff --check`, and changed file names for
-   secrets, logs, generated artifacts, or unrelated edits.
-3. Confirm the checks selected by root `AGENTS.md` passed on the final diff.
-   Do not replace focused behavioral tests with a generic gate or rerun checks
-   already covered by an unchanged umbrella run.
-4. Read `.github/pull_request_template.md`. Consult `Makefile`, `.config/make/`,
-   or CI only when the required command/current gate is uncertain.
-5. Return `BLOCKED` for an unclean scope, missing required evidence, failed
-   required checks, or non-compliant metadata.
+- Read `AGENTS.md`.
+- Read `.github/pull_request_template.md`.
+- Use `Makefile` and `.config/make/` for local quality commands.
+- Use `.github/workflows/ci.yml` for CI expectations.
+- Do not restate long command matrices or template sections from memory when the files exist.
 
-## Metadata
+## Workflow
 
-- Title: English Conventional Commit, at most 72 characters, with no tool
-  prefix.
-- Body: English, exact template headings, `N/A` where needed, concise rationale,
-  actual verification commands, and material risks/rollback notes.
-- Use repository-relative paths; never include local absolute paths.
-- Keep prose paragraphs on one logical line and never include the literal
-  sequence `\n`.
-- Use a temporary body file with `gh pr create --body-file` or
-  `gh pr edit --body-file`; never pass multiline Markdown inline.
+1. Collect PR context
+- Confirm base branch, current branch, change goal, and scope.
+- Confirm whether the task is: draft a new PR, update an existing PR, or preflight-check readiness.
+- Confirm whether the branch includes only intended changes.
 
-## Output
+2. Inspect change scope
+- Review the diff and summarize what changed.
+- Call out unrelated edits, generated artifacts, logs, or secrets as blockers.
+- Mark risky areas explicitly: auth, storage, config, network, migrations, breaking changes.
+- Scan the diff for newly added string literals and confirm whether they duplicate values already defined as constants/enums/typed wrappers in the same module or shared modules.
+- Treat introducing a new hardcoded literal where a project constant already exists as a likely regression risk; require either a refactor to reuse the constant or an explicit exception explanation in the PR body.
 
-- Status: `READY` or `BLOCKED`.
-- Title.
-- Complete PR body.
-- Verification commands and results.
-- Risks or `N/A`.
+3. Verify readiness requirements
+- Require `make pre-commit` before marking PRs ready when the diff changes Rust code, product behavior, CI behavior, runtime configuration, security-sensitive logic, migrations, storage, auth, networking, or other high-risk paths.
+- For documentation-only, agent-instruction-only, or local developer-tooling-only changes, allow focused verification instead of `make pre-commit` when it directly validates the changed surface.
+- For focused verification, explain why the full gate was not run and list the scope-specific commands in the PR body.
+- If `make` is unavailable, use the equivalent commands from `.config/make/`.
+- Add scope-specific verification commands when the changed area needs more than the baseline.
+- If required checks fail, stop and return `BLOCKED`.
 
-Immediately before the GitHub write, repeat only the five preflight checks above
-against the final head.
+4. Draft PR metadata
+- Write the PR title in English using Conventional Commits and keep it within 72 characters.
+- If a generic PR workflow suggests a different title format, ignore it and follow the repository rule instead.
+- In RustFS, do not use tool-specific prefixes such as `[codex]` when the repository requires Conventional Commits.
+- Keep the PR body in English.
+- Use the exact section headings from `.github/pull_request_template.md`.
+- Fill non-applicable sections with `N/A`.
+- Include verification commands in the PR description.
+- Do not include local filesystem paths in the PR body unless the user explicitly asks for them.
+- Prefer repo-relative paths, command names, and concise summaries over machine-specific paths such as `/Users/...`.
+
+5. Prepare reviewer context
+- Summarize why the change exists.
+- Summarize what was verified.
+- Call out risks, rollout notes, config impact, and rollback notes when applicable.
+- Mention assumptions or missing context instead of guessing.
+
+6. Prepare CLI-safe output
+- When proposing `gh pr create` or `gh pr edit`, use `--body-file`, never inline `--body` for multiline markdown.
+- Return a ready-to-save PR body plus a short title.
+- If not ready, return blockers first and list the minimum steps needed to unblock.
+
+## Output format
+
+### Status
+- `READY` or `BLOCKED`
+
+### Title
+- `<type>(<scope>): <summary>`
+
+### PR Body
+- Reproduce the repository template headings exactly.
+- Fill every section.
+- Omit local absolute paths unless explicitly required.
+
+### Verification
+- List each command run.
+- State pass/fail.
+
+### Risks
+- List breaking changes, config changes, migration impact, or `N/A`.
+
+## Blocker rules
+
+- Return `BLOCKED` if a code, behavior, CI, runtime configuration, security-sensitive, migration, storage, auth, networking, or other high-risk change has not passed `make pre-commit`.
+- Return `BLOCKED` if a documentation-only, agent-instruction-only, or local developer-tooling-only change lacks focused verification for the changed surface.
+- Return `BLOCKED` if the diff contains unrelated changes that are not acknowledged.
+- Return `BLOCKED` if required template sections are missing.
+- Return `BLOCKED` if the title/body is not in English.
+- Return `BLOCKED` if the title does not follow the repository's Conventional Commit rule.
+- Return `BLOCKED` if the diff introduces string literals that should use existing constants but did not.
+
+## Reference
+
+- Use [pr-readiness-checklist.md](references/pr-readiness-checklist.md) for a short final pass before opening or editing the PR.

.agents/skills/pr-creation-checker/agents/openai.yaml

@@ -1,4 +1,4 @@
 interface:
   display_name: "PR Creation Checker"
   short_description: "Draft RustFS-ready PRs with checks, template, and blockers."
-  default_prompt: "Use $pr-creation-checker for final PR preflight and compliant English title/body metadata."
+  default_prompt: "Inspect a branch or diff, verify required PR checks, and produce a compliant English PR title/body plus blockers or readiness status."

.agents/skills/pr-creation-checker/references/pr-readiness-checklist.md

@@ -0,0 +1,16 @@
+# PR Readiness Checklist
+
+- Confirm the branch is based on current `main`.
+- Confirm the diff matches the stated scope.
+- Confirm no secrets, logs, temp files, or unrelated refactors are included.
+- Confirm `make pre-commit` passed for code, behavior, CI, runtime configuration, security-sensitive, migration, storage, auth, networking, or other high-risk changes.
+- For documentation-only, agent-instruction-only, or local developer-tooling-only changes, confirm focused verification covered the changed surface and the PR body explains why the full gate was not run.
+- Confirm extra verification commands are listed for risky changes.
+- Confirm the PR title uses Conventional Commits and stays within 72 characters.
+- Confirm the PR title does not use tool-specific prefixes such as `[codex]`.
+- Confirm the PR body is in English.
+- Confirm the PR body keeps the exact headings from `.github/pull_request_template.md`.
+- Confirm non-applicable sections are filled with `N/A`.
+- Confirm the PR body does not include local absolute paths unless explicitly required.
+- Confirm multiline GitHub CLI commands use `--body-file`.
+- Confirm new hardcoded string literals were not introduced for values already represented by existing constants/enums (including protocol labels, error identifiers, headers, and metric names), or record a justified exception.

.agents/skills/rust-code-quality/SKILL.md

@@ -1,41 +1,38 @@
 ---
 name: rust-code-quality
-description: Run a focused Rust quality review when the user requests one, when reviewing a Rust PR/commit, or when another selected review workflow delegates Rust-specific checks. Do not auto-load for every implementation edit.
+description: Enforce Rust-specific code quality rules on every code change. Use before merge to catch unwrap abuse, silent truncation, unnecessary cloning, lock ordering violations, recursion risks, and error type anti-patterns.
 ---
 
 # Rust Code Quality Gate
 
-Use this skill for a dedicated Rust review to cover rules that `cargo clippy`
-does not catch.
+Use this skill on every Rust code change to enforce quality rules that `cargo clippy` does not catch.
 
 ## Quick Start
 
 1. Identify changed `.rs` files.
 2. Run automated checks on changed files.
 3. Run manual review checklist on the diff.
-4. Resolve or rebut every finding with evidence; P0/P1 findings cannot be deferred.
+4. Report findings; block merge if P0/P1 issues exist.
 
 ## Automated Checks
 
-Use these searches to find candidates in changed `.rs` files. Inspect syntax,
-`#[cfg(test)]` scope, and the changed hunk before reporting a finding; text
-filters do not reliably distinguish production code from tests.
+Run these on every changed `.rs` file (excluding test modules):
 
 ```bash
-# 1. unwrap/expect candidates
-rg -n '\.unwrap\(\)|\.expect\(' <changed-files>
+# 1. unwrap/expect in production code
+rg -n '\.unwrap\(\)|\.expect\(' <changed-files> | grep -v '#\[cfg(test)\]' | grep -v 'test' | grep -v 'bench'
 
 # 2. Silent type truncation via `as` cast
 rg -n ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' <changed-files>
 
 # 3. String as error type
-rg -n 'Result<.*String>' <changed-files>
+rg -n 'Result<.*String>' <changed-files> | grep -v test
 
 # 4. Box<dyn Error> in public APIs
-rg -n 'Box<dyn.*Error' <changed-files>
+rg -n 'Box<dyn.*Error' <changed-files> | grep -v test
 
 # 5. println/eprintln in production
-rg -n 'println!\|eprintln!' <changed-files>
+rg -n 'println!\|eprintln!' <changed-files> | grep -v test
 
 # 6. Ordering::Relaxed usage (verify each is intentional)
 rg -n 'Ordering::Relaxed' <changed-files>
@@ -46,38 +43,40 @@ rg -n 'unwrap_or_default\(\)|unwrap_or\(' <changed-files>
 
 ## Manual Review Checklist
 
-For the Rust diff under review, verify:
+For every Rust code change, verify:
 
 ### Error Handling
-- [ ] Every production `unwrap()` or `expect()` is infallible by type or a checked invariant; explain only non-obvious invariants, using an existing type, a useful `expect` message, or a concise comment
+- [ ] No `unwrap()` or `expect()` in production code without justification comment
 - [ ] No `Result<_, String>` in public API signatures
-- [ ] Public library APIs use domain errors unless deliberate error erasure at a boundary is part of the contract
+- [ ] No `Box<dyn Error>` in public trait/struct methods
 - [ ] `Error::source()` is overridden when inner error is stored
-- [ ] Error messages are actionable without exposing secret input
+- [ ] Error messages are actionable (what failed, with what input)
 
 ### Type Safety
 - [ ] No silent `as` truncation (negative→unsigned, large→small)
-- [ ] Fallible numeric conversions use `TryFrom`/`try_into()` and return a typed error; clamp or saturate only when the domain explicitly requires it
-- [ ] Floating-point to integer conversion validates finiteness, sign, and range before conversion
+- [ ] `try_into()` or explicit clamping used for numeric conversions
+- [ ] No `f64 as usize` without prior clamping
 
 ### Concurrency
 - [ ] Lock acquisition order is documented when multiple locks are used, and matches every other call site taking any overlapping subset (ABBA check)
 - [ ] No `tokio::sync` lock guard (read or write) held across `.await` without bounded hold time — long-lived read guards wedge writers (#4195)
-- [ ] Atomic read-modify-write uses the direct `fetch_*` operation when possible; use `compare_exchange` only for conditional updates
+- [ ] Concurrent counters use `compare_exchange` loops, not load-then-store
 - [ ] `std::sync::Mutex` in async context is held only briefly, never across `.await`
 
 ### Memory and Performance
-- [ ] On an identified hot path, report cloning or allocation only with a concrete per-request/per-object cost or benchmark signal
-- [ ] Prefer borrowing, moving, `Bytes`/`Arc`, or capacity reservation only when it reduces that cost without obscuring ownership or APIs
+- [ ] No `.clone()` on structs with >5 heap-allocated fields in hot paths
+- [ ] `HashMap::with_capacity()` / `Vec::with_capacity()` used when size is known
+- [ ] Large buffers wrapped in `Arc` rather than cloned
+- [ ] Temporary string computations use `&str` or `Cow<str>` instead of `String`
 
 ### Recursion Safety
-- [ ] Recursion over untrusted, persisted, or otherwise unbounded input has a depth limit or uses iterative traversal
+- [ ] Recursive functions have a depth limit or use iterative traversal
 - [ ] Tree/cache traversals handle corrupted/cyclic input safely
 
 ### Testing
-- [ ] Tests have an observable failure criterion; delegated assertions, `#[should_panic]`, snapshot/property checks, and meaningful `Result` failures do not need a redundant `assert!`
-- [ ] Use `expect` only when its message improves failure diagnosis; do not add boilerplate to self-evident test setup
-- [ ] Test volume and line count are never treated as production-code growth
+- [ ] Every test function has at least one `assert!`
+- [ ] Tests use `.expect("context")` not bare `.unwrap()`
+- [ ] No `println!`/`eprintln!` in production code (use `tracing`)
 
 ### Serde
 - [ ] Structs from untrusted input have `#[serde(deny_unknown_fields)]`
@@ -89,18 +88,18 @@ For the Rust diff under review, verify:
 - [ ] New string literals don't duplicate existing constants
 
 ### Reuse and Necessity
-- [ ] No new helper duplicates `crates/utils`, `crates/common`, the touched crate, the likely domain-owning crate, a relevant direct dependency, or plain std/tokio behavior; reused helpers match the call site's semantics
+- [ ] No new helper duplicating an existing workspace utility (`crates/utils`, `crates/common`, the touched crate) or plain std/tokio behavior no wrapper refines; reused helpers match the call site's semantics (normalization, error type, backoff, durability gating)
 - [ ] No branch without a nameable concrete trigger; no re-validation of what a validated upstream layer on the same path already guarantees (Cross-Cutting Domain Invariant patterns and pre-destructive-action re-checks are load-bearing — keep them)
 - [ ] Error context attached once where actionable, not re-wrapped at every hop; no typed→generic error conversion below aggregation/quorum layers
-- [ ] Comments avoid narration and change history while completely stating non-obvious lock, `SAFETY`, durability, compatibility, and unwrap invariants
+- [ ] No comments narrating the next line, restating a signature, or describing the change itself (invariant comments — lock ordering, `SAFETY`, unwrap justification — are not narration)
 - [ ] No near-duplicate test pinning the same code path and poison-value class as an existing test (boundary companions — n==max vs max+1, absent/empty/nil UUID — are never near-duplicates)
 
 ## Severity Classification
 
-- **P0 (Block merge)**: demonstrated data loss, security breach, remote crash, or deadlock
-- **P1 (Must fix)**: concrete correctness, compatibility, or material hot-path regression
-- **P2 (Should fix)**: avoidable duplication or maintainability issue with a concrete simpler replacement
-- **P3 (Nice to fix)**: local style or clarity issue with no behavioral risk
+- **P0 (Block merge)**: `unwrap()` in request hot path, silent truncation on user input, lock ordering violation, recursion without depth limit
+- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method, `unwrap_or_default()` on a domain-required value (metadata, quorum, version id)
+- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`, new helper duplicating an existing workspace utility, defensive branch with no nameable trigger (corrupt or stale persisted/peer data is always a nameable trigger for boundary-crossing values), near-duplicate test, redundant error re-wrapping
+- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`, narrating comment
 
 ## Output Template
 
@@ -108,10 +107,10 @@ For the Rust diff under review, verify:
 ## Rust Code Quality Report
 
 ### Automated Scan
-- unwrap/expect candidates inspected: N
-- numeric-cast candidates inspected: N
-- error-type candidates inspected: N
-- output-macro candidates inspected: N
+- unwrap/expect in production: N found
+- as casts: N found
+- String errors: N found
+- println/eprintln: N found
 
 ### Findings
 - [P1] `path:line` — description

.agents/skills/rust-code-quality/references/checklist.md

@@ -0,0 +1,52 @@
+# Rust Code Quality Checklist
+
+Use this as a quick pre-merge checklist for every Rust code change.
+
+## Critical (P0 — block merge)
+
+| Check | Command |
+|-------|---------|
+| No `unwrap()` in request/storage hot path | `rg '\.unwrap\(\)' <files> \| grep -v test` |
+| No `as` truncation on user input | `rg ' as (u32\|usize\|i32)' <files>` |
+| Lock order consistent across call sites | Manual: trace all lock acquisitions |
+| Recursive functions have depth limit | Manual: check for `max_depth` or iterative pattern |
+| No `panic!`/`unwrap_or_else(panic!)` in production | `rg 'panic!\|unwrap_or_else.*panic' <files> \| grep -v test` |
+
+## High (P1 — must fix)
+
+| Check | Command |
+|-------|---------|
+| No `Result<_, String>` in public API | `rg 'Result<.*String>' <files> \| grep -v test` |
+| No `Box<dyn Error>` in public trait | `rg 'Box<dyn.*Error' <files> \| grep -v test` |
+| No unnecessary `.clone()` in hot path | Manual: check loops and per-request paths |
+| `Error::source()` implemented when inner error stored | Manual: check `impl Error` |
+| No `eprintln!`/`println!` in production | `rg 'println!\|eprintln!' <files> \| grep -v test` |
+
+## Medium (P2 — should fix)
+
+| Check | Command |
+|-------|---------|
+| Tests have assertions | Manual: check for `assert` in test functions |
+| `HashMap`/`Vec` use `with_capacity` when size known | Manual: check `::new()` in loops |
+| No `#![allow(dead_code)]` at crate root | `rg 'allow.dead_code' <files> \| grep 'lib.rs'` |
+| Serde structs from untrusted input have `deny_unknown_fields` | Manual: check `#[derive(Deserialize)]` |
+
+## Low (P3 — nice to fix)
+
+| Check | Command |
+|-------|---------|
+| No camelCase statics | `rg 'static ref [a-z]' <files>` |
+| `Arc::ptr_eq` instead of `as_ptr + ptr::eq` | `rg 'as_ptr\|ptr::eq' <files>` |
+| Public functions have doc comments | `rg 'pub fn' <files> \| grep -v '///'` |
+
+## Quick One-Liner
+
+```bash
+# Run all automated checks on changed files
+CHANGED=$(git diff --name-only HEAD~1 -- '*.rs' | grep -v test | grep -v bench)
+echo "=== unwrap/expect ===" && rg -c '\.unwrap\(\)|\.expect\(' $CHANGED 2>/dev/null
+echo "=== as casts ===" && rg -c ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' $CHANGED 2>/dev/null
+echo "=== String errors ===" && rg -c 'Result<.*String>' $CHANGED 2>/dev/null
+echo "=== println ===" && rg -c 'println!|eprintln!' $CHANGED 2>/dev/null
+echo "=== Ordering::Relaxed ===" && rg -c 'Ordering::Relaxed' $CHANGED 2>/dev/null
+```

.agents/skills/rustfs-logging-governance/SKILL.md

@@ -1,34 +1,107 @@
 ---
 name: rustfs-logging-governance
-description: Add or review RustFS `tracing` events with the repository field shape, level policy, privacy boundaries, and guardrails. Use when a change adds or edits a tracing macro/instrumentation site or the logging guardrail script.
+description: Standardize and review RustFS logging with structured `tracing` events, lower noise on hot paths, preserve security-sensitive diagnostics, and extend guardrails to prevent legacy logging patterns from returning. Use when editing or reviewing RustFS logs, startup/config diagnostics, cloud metadata logs, request validation logs, or `scripts/check_logging_guardrails.sh`.
 ---
 
 # RustFS Logging Governance
 
-Apply this skill only to changed logging sites; do not turn a local log edit into
-a broad logging cleanup.
+Use this skill when RustFS logging needs to be added, cleaned up, reviewed, or protected against regressions.
 
-## Workflow
+## Quick Start
 
-1. Read the changed function/module context and classify the site as lifecycle,
-   request/hot path, fallback, external fetch, or summary.
-2. Match neighboring structured events and reuse existing `EVENT_*`,
-   `LOG_COMPONENT_*`, and `LOG_SUBSYSTEM_*` constants.
-3. Put stable fields first (`event`, `component`, `subsystem`, `state`/`result`,
-   then context) and a short label last.
-4. Select the level by operational meaning:
-   - `error`: behavior/security-affecting failure;
-   - `warn`: degraded/fallback/operator-actionable state;
-   - `info`: low-frequency lifecycle/mode change;
-   - `debug`: targeted diagnostics;
-   - `trace`: repetitive request/object/shard success paths.
-5. Never log secrets, tokens, auth headers, credential payloads, raw
-   attacker-controlled bodies, or merged config dumps. Error strings and
-   `Debug` output are log surfaces too.
-6. Prefer one aggregate summary over inventories or startup banners.
-7. Run `./scripts/check_logging_guardrails.sh` and the checks selected by root
-   `AGENTS.md`.
+1. Identify the files whose logs are changing.
+2. Scan current `tracing` or `log` macros before editing.
+3. Convert sentence-style logs to short event-style logs.
+4. Demote hot-path success logs unless operators truly need them at `info`.
+5. Preserve failure, fallback, and security-relevant diagnostics.
+6. Update `scripts/check_logging_guardrails.sh` when a broad cleanup removes a legacy pattern class.
+7. Validate with formatting, targeted checks/tests, and the logging guardrail script.
 
-Read [logging-governance.md](references/logging-governance.md) only for a broad
-logging audit, event-model migration, or guardrail expansion. Ordinary single-
-site edits do not require the full workspace scope map.
+## Core Workflow
+
+### 1. Scope the logging surface
+
+- Read the changed module in full before touching log lines.
+- Classify the log site:
+  - lifecycle/startup
+  - request or validation path
+  - background loop or hot path
+  - fallback/degraded behavior
+  - cloud metadata or external fetch path
+  - metrics/config summary
+- Do not rewrite business logic to make logging easier.
+
+### 2. Use the RustFS event shape
+
+- Prefer fields first, message second.
+- Use short labels, not prose paragraphs.
+- Default field shape:
+  - `event`
+  - `component`
+  - `subsystem`
+  - `state` or `result`
+  - key context fields
+- Reuse stable field names and avoid inventing near-duplicates.
+
+See `references/logging-governance.md` for the event model, level policy, and anti-pattern list.
+
+### 3. Choose the right level
+
+- `error`: operation failure that affects behavior or security guarantees.
+- `warn`: degraded path, fallback, suspicious input, or operator-actionable misconfiguration.
+- `info`: low-frequency lifecycle or mode selection.
+- `debug`: targeted diagnostics and low-volume detail.
+- `trace`: hot-path and repetitive success-path events.
+
+When in doubt, lower the verbosity of normal success paths and keep structured detail in fields.
+
+### 4. Preserve security and privacy boundaries
+
+- Do not log secrets, tokens, auth headers, raw credential payloads, or merged config dumps.
+- Avoid logging raw forwarded headers or full trusted network inventories above `debug`.
+- Keep warning/error logs useful without echoing attacker-controlled payloads unnecessarily.
+
+### 5. Keep summaries aggregated
+
+- Replace multi-line startup banners or checklist logs with one structured event.
+- If metrics already express a concept, avoid duplicating it with many `info!` lines.
+- Prefer counts, modes, and sources over inventories unless debug detail is truly needed.
+
+### 6. Update guardrails when needed
+
+- Broad logging cleanup should usually extend `scripts/check_logging_guardrails.sh`.
+- Add forbidden patterns only for styles the repo has intentionally retired:
+  - sentence-style lifecycle logs
+  - noisy hot-path `info!`
+  - checklist-style summary logs
+  - legacy fallback wording that has been replaced by structured fields
+- Keep guardrails concrete and grep-friendly.
+
+### 7. Validate manually
+
+Use the smallest relevant set:
+
+```bash
+cargo fmt --all --check
+./scripts/check_logging_guardrails.sh
+cargo check -p <affected-crate>
+cargo test -p <affected-crate>
+```
+
+For broader Rust changes, add:
+
+```bash
+./scripts/check_unsafe_code_allowances.sh
+./scripts/check_architecture_migration_rules.sh
+cargo clippy -p <affected-crates> --all-targets -- -D warnings
+```
+
+## RustFS-Specific Notes
+
+- The durable RustFS logging direction is `event + component + subsystem + state/result + key context fields`.
+- `crates/concurrency` and `crates/trusted-proxies` are examples of this style for lifecycle, fallback, and cloud metadata logs.
+- `scripts/check_logging_guardrails.sh` is the enforcement point for preventing removed log styles from returning.
+
+## References
+
+- Read `references/logging-governance.md` when you need the detailed field set, anti-pattern examples, or guardrail update checklist.

.agents/skills/rustfs-logging-governance/references/logging-governance.md

@@ -1,62 +1,285 @@
-# Logging Audit and Migration Reference
+# RustFS Logging Governance Reference
 
-Read this reference only for a broad logging audit, an event-model migration,
-or a change to `scripts/check_logging_guardrails.sh`. Use `Cargo.toml` for the
-current workspace/crate list instead of maintaining one here.
+## Workspace Scope Map
 
-## Audit by Operational Role
+Use `Cargo.toml` `[workspace].members` as the source of truth for crate membership. When doing a broad logging sweep, classify crates by operational role so logs stay consistent within each role.
 
-- Server/protocol/admin: lifecycle, authorization failures, request boundaries,
-  and degraded subsystems; avoid normal request success at `info`.
-- Storage/heal/scanner/capacity: integrity failures and aggregate lifecycle;
-  avoid per-object, per-shard, and folder iteration noise.
-- IAM/policy/credentials/KMS/crypto: safe identifiers and enforcement results;
-  never emit secrets, claims, payloads, or expected authenticators.
-- Notify/audit/targets: target lifecycle and batch/backpressure summaries; avoid
-  per-event success logs.
-- Locking/concurrency/I/O foundations: contention anomalies and state changes;
-  prefer metrics for high-frequency worker/permit signals.
-- Shared type/schema crates: log at the operational caller boundary unless the
-  crate itself owns the failure context.
+### Core Server And Request Handling
 
-## Event Shape
+- `rustfs`
+  - Role: top-level server, startup, auth, admin wiring, S3 request handling.
+  - Logging focus: startup lifecycle, config summaries, authn/authz failures, protocol entrypoints, degraded subsystems.
+- `crates/protocols`
+  - Role: protocol integrations such as FTP, SFTP, WebDAV, and related server-side protocol layers.
+  - Logging focus: listener lifecycle, per-protocol enablement/disablement, request bridge failures.
+- `crates/madmin`
+  - Role: admin API contracts and management interfaces.
+  - Logging focus: admin action boundaries, validation failures, compatibility warnings.
+- `crates/trusted-proxies`
+  - Role: forwarded IP trust, proxy chain validation, cloud metadata sources.
+  - Logging focus: direct/trusted/fallback decisions, degraded metadata fetches, aggregated config summaries.
+- `crates/keystone`
+  - Role: Keystone auth integration.
+  - Logging focus: integration enablement, upstream auth failures, config safety without credential leakage.
 
-Prefer stable fields in this order when available:
+### Storage, Healing, And Data Plane
 
-1. `event`
-2. `component`
-3. `subsystem`
-4. `state` or `result`
-5. stable context such as mode, duration, reason, counts, safe identifiers, or
-   capacity/permit values
-6. short message label
+- `crates/ecstore`
+  - Role: erasure-coded storage implementation and peer/store initialization.
+  - Logging focus: disk/peer lifecycle, storage fallback, object I/O failures, avoid per-object noise.
+- `crates/heal`
+  - Role: healing orchestration and repair workflows.
+  - Logging focus: scheduler lifecycle, repair decisions, backlog or skipped work summaries, avoid repetitive task spam at `info`.
+- `crates/scanner`
+  - Role: data integrity scanning and health monitoring.
+  - Logging focus: scan lifecycle, compaction/deep-heal transitions, lag/backlog, noisy folder iteration should stay at `debug/trace`.
+- `crates/object-capacity`
+  - Role: capacity scan and refresh core.
+  - Logging focus: refresh lifecycle, degraded capacity sources, aggregate stats rather than per-object chatter.
+- `crates/filemeta`
+  - Role: file metadata parsing and helpers.
+  - Logging focus: parse failures, schema/format mismatch, avoid dumping raw metadata payloads.
+- `crates/storage-api`
+  - Role: storage contracts and shared data plane interfaces.
+  - Logging focus: contract mismatch and boundary diagnostics, usually low-volume.
+- `crates/checksums`
+  - Role: checksum helpers and validation.
+  - Logging focus: integrity failures and compatibility mismatches, not per-chunk success logs.
+- `crates/zip`
+  - Role: ZIP handling and compression helpers.
+  - Logging focus: parse/extract failures, archive path safety issues, avoid verbose file-by-file success logs.
 
-Reuse the module's constants and neighboring field names. Do not create aliases
-for the same concept.
+### Security, Identity, And Policy
 
-## Patterns to Retire
+- `crates/iam`
+  - Role: identity and access management.
+  - Logging focus: authz decision boundaries, imported payload safety, do not leak principals, secrets, or claims.
+- `crates/policy`
+  - Role: policy modeling and evaluation.
+  - Logging focus: deny/allow decision context, parser/validation failures, no raw secret-bearing request dumps.
+- `crates/credentials`
+  - Role: credential handling.
+  - Logging focus: never log secrets or tokens; only safe identifiers and redacted states.
+- `crates/kms`
+  - Role: key management service integration.
+  - Logging focus: init/health/fallback, key-source availability, never log key material.
+- `crates/crypto`
+  - Role: cryptographic helpers and security primitives.
+  - Logging focus: only algorithm or mode state, not plaintext, ciphertext, or secret-derived material.
+- `crates/security-governance`
+  - Role: security governance contracts.
+  - Logging focus: policy/state transitions and enforcement diagnostics.
+- `crates/signer`
+  - Role: request signing helpers.
+  - Logging focus: signature validation failures without expected-signature leakage.
 
-- sentence-style lifecycle announcements;
-- startup banners and checklist lines;
-- repetitive success logs at `info`/`debug`;
-- raw inventories when an aggregate count is sufficient;
-- fallback prose with values embedded in the message;
-- `?value`/`Debug` output for credential-bearing or attacker-controlled data;
-- logging a parse input when the malformed input may itself be a secret.
+### Notifications, Audit, And Targets
 
-## Guardrail Changes
+- `crates/notify`
+  - Role: notification dispatch, runtime facade, notifier implementations.
+  - Logging focus: target lifecycle, dispatch summaries, stream lag/backpressure, avoid per-event success spam.
+- `crates/audit`
+  - Role: audit target fan-out and audit pipeline management.
+  - Logging focus: pipeline lifecycle, target availability, batch dispatch summaries, avoid noisy "started successfully" prose.
+- `crates/targets`
+  - Role: target-specific configuration and utilities used by fan-out style systems.
+  - Logging focus: target selection, config validation, per-target degraded state.
+- `crates/s3-types`
+  - Role: S3 event and type definitions.
+  - Logging focus: usually minimal; keep logging at integration boundaries rather than low-level type crates.
+- `crates/s3-ops`
+  - Role: S3 operation definitions and mapping.
+  - Logging focus: mapping/contract failures, unsupported combinations, not normal-path request spam.
 
-When expanding `scripts/check_logging_guardrails.sh`:
+### Concurrency, Locking, And Runtime Foundations
 
-1. Add only files/patterns intentionally migrated in the same change.
-2. Keep patterns concrete and grep-friendly.
-3. Do not encode a style that remains valid elsewhere as a global ban.
-4. Run the guardrail script and the root validation tier.
-5. Treat the script as a floor; manually verify level, field shape, and privacy.
+- `crates/concurrency`
+  - Role: timeout, locking, backpressure, and I/O scheduling facade.
+  - Logging focus: lifecycle transitions and degraded states, not high-frequency worker/permit churn at `info`.
+- `crates/lock`
+  - Role: distributed locking implementation.
+  - Logging focus: lock lifecycle, contention anomalies, lock ordering or timeout diagnostics.
+- `crates/tls-runtime`
+  - Role: shared TLS runtime foundation.
+  - Logging focus: certificate lifecycle, reload/fallback, validation failures without sensitive dumps.
+- `crates/obs`
+  - Role: observability helpers.
+  - Logging focus: this crate shapes other crates' telemetry conventions; avoid recursive or redundant summaries.
+- `crates/io-core`
+  - Role: zero-copy I/O core primitives.
+  - Logging focus: keep very sparse; prefer metrics unless failures are actionable.
+- `crates/io-metrics`
+  - Role: I/O metrics collection.
+  - Logging focus: typically minimal; metrics should carry the hot-path signal.
+- `crates/rio`
+  - Role: Rust I/O utility layer.
+  - Logging focus: compatibility or runtime boundary failures, not fast-path internals.
+- `crates/rio-v2`
+  - Role: next-generation I/O compatibility layer.
+  - Logging focus: migration/feature-mode differences and degraded fallback between I/O paths.
+- `crates/utils`
+  - Role: shared helpers.
+  - Logging focus: usually avoid direct logging in generic helpers unless the helper is itself an operational boundary.
+- `crates/common`
+  - Role: shared data structures and helpers.
+  - Logging focus: same principle as `utils`; prefer callers to log context-rich events.
+- `crates/config`
+  - Role: configuration management.
+  - Logging focus: config source, fallback, validation, and summary aggregation; avoid dumping merged configs.
+- `crates/data-usage`
+  - Role: shared data usage models and algorithms.
+  - Logging focus: refresh lifecycle, summary stats, and degraded reads.
 
-Useful search seeds for the changed surface:
+### Schema, Contracts, And API Support
+
+- `crates/protos`
+  - Role: protobuf definitions.
+  - Logging focus: usually none inside the crate; emit logs at decode/use boundaries.
+- `crates/extension-schema`
+  - Role: extension schema contracts.
+  - Logging focus: schema validation and compatibility mismatches.
+- `crates/s3select-api`
+  - Role: S3 Select API interfaces.
+  - Logging focus: request validation and unsupported feature boundaries.
+- `crates/s3select-query`
+  - Role: S3 Select query engine.
+  - Logging focus: query parse/planning/execution failures, avoid row-level spam.
+- `crates/protocols`
+  - Role: non-S3 protocol support.
+  - Logging focus: see core server section; keep per-request verbosity below `info`.
+
+### Testing And Non-Production Crates
+
+- `crates/e2e_test`
+  - Role: end-to-end tests.
+  - Logging focus: test clarity matters more than production governance, but avoid copying test-only logging style into production crates.
+
+## Current Guardrail Coverage Map
+
+`scripts/check_logging_guardrails.sh` currently enforces retired patterns in these high-signal areas:
+
+- `rustfs/src/main.rs`
+- `rustfs/src/startup_iam.rs`
+- `rustfs/src/auth.rs`
+- `rustfs/src/protocols/client.rs`
+- `crates/audit/src/pipeline.rs`
+- `crates/audit/src/system.rs`
+- `crates/audit/src/global.rs`
+- `crates/notify/src/config_manager.rs`
+- `crates/notify/src/runtime_facade.rs`
+- `crates/notify/src/notifier.rs`
+- `crates/ecstore/src/store/peer.rs`
+- `crates/ecstore/src/store/init.rs`
+- `crates/ecstore/src/tier/tier.rs`
+- `crates/concurrency/src/workers.rs`
+- `crates/concurrency/src/manager.rs`
+- `crates/concurrency/src/lock.rs`
+- `crates/concurrency/src/deadlock.rs`
+- `crates/trusted-proxies/src/global.rs`
+- `crates/trusted-proxies/src/config/loader.rs`
+- `crates/trusted-proxies/src/proxy/metrics.rs`
+- `crates/trusted-proxies/src/proxy/validator.rs`
+- `crates/trusted-proxies/src/proxy/chain.rs`
+- `crates/trusted-proxies/src/middleware/service.rs`
+- `crates/trusted-proxies/src/cloud/detector.rs`
+- `crates/trusted-proxies/src/cloud/ranges.rs`
+- `crates/trusted-proxies/src/cloud/metadata/aws.rs`
+- `crates/trusted-proxies/src/cloud/metadata/azure.rs`
+- `crates/trusted-proxies/src/cloud/metadata/gcp.rs`
+
+When expanding coverage, prefer crates with:
+
+- repeated sentence-style lifecycle logs
+- high-frequency success-path `info!`
+- startup/config checklist banners
+- security-sensitive fallback wording
+- external fetch/retry/fallback flows
+
+That typically means the next broad candidates are `rustfs`, `crates/notify`, `crates/audit`, `crates/targets`, `crates/heal`, and `crates/scanner`.
+
+## Event Model
+
+Prefer this structure when the fields are available:
+
+- `event`
+- `component`
+- `subsystem`
+- `state` or `result`
+- stable context fields such as:
+  - `enabled`
+  - `implementation`
+  - `validation_mode`
+  - `peer_ip`
+  - `client_ip`
+  - `proxy_hops`
+  - `duration_ms`
+  - `fallback`
+  - `reason`
+  - `range_count`
+  - `hold_time_ms`
+  - `available_slots`
+  - `total_slots`
+  - `permits_in_use`
+
+## Level Policy
+
+- `error`: the operation fails and callers or security guarantees are affected.
+- `warn`: a degraded path, fallback, suspicious request, or operator-actionable config issue occurs.
+- `info`: a low-frequency lifecycle or mode transition occurs.
+- `debug`: useful diagnostics exist but normal operators do not need them all the time.
+- `trace`: hot-path and repetitive success-path details occur.
+
+## Preferred Patterns
+
+- Use a short message label:
+  - `"trusted proxy validation failed"`
+  - `"concurrency manager state changed"`
+  - `"trusted proxy cloud metadata loaded"`
+- Put key meaning into fields, not only the message text.
+- Aggregate config or metrics summaries into one log event.
+
+## Retired Patterns
+
+These should usually be removed or replaced:
+
+- Sentence-style lifecycle logs:
+  - `info!("Concurrency manager stopped")`
+  - `info!("Trusted Proxies module initialized")`
+- Checklist or banner logs:
+  - `info!("=== Application Configuration ===")`
+  - `info!("Available metrics:")`
+- Hot-path noise:
+  - `info!("worker take, {}", *available)`
+  - `debug!("Proxy validation successful in {:?}", duration)`
+- Legacy fallback prose:
+  - `"Request from private network but not trusted: ..."`
+  - `"Cloud metadata fetching is disabled"`
+
+## Guardrail Update Checklist
+
+When extending `scripts/check_logging_guardrails.sh`:
+
+1. Add the touched files to `checked_files`.
+2. Add only legacy patterns that have been intentionally retired.
+3. Keep patterns literal and grep-friendly.
+4. Run the guardrail script after changes.
+5. Avoid adding patterns for logs that are still valid elsewhere in the repo.
+
+## Validation Checklist
+
+For logging-only changes:
 
 ```bash
-rg -n 'error!|warn!|info!|debug!|trace!|#\[instrument' <changed-paths>
-rg -n '\?[^,)]|secret|token|credential|authorization|merged_config' <changed-paths>
+cargo fmt --all --check
+./scripts/check_logging_guardrails.sh
+cargo check -p <affected-crate>
+cargo test -p <affected-crate>
+```
+
+For broader Rust changes:
+
+```bash
+./scripts/check_unsafe_code_allowances.sh
+./scripts/check_architecture_migration_rules.sh
+cargo clippy -p <affected-crates> --all-targets -- -D warnings
 ```

.agents/skills/rustfs-release-publish/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: rustfs-release-publish
-description: "Run the end-to-end RustFS console gate, version bump, preview validation, and final-tag publication pipeline. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)."
+description: "End-to-end RustFS release pipeline: first publish any merged-but-unreleased rustfs/console changes and wait for its latest Release asset, then bump RustFS version files on main directly to the final target, publish a visible GitHub prerelease from a preview tag without updating latest channels, validate it, and publish the final tag on the SAME commit. Use whenever the user wants to release/publish a RustFS version (发版/发布)."
 ---
 # RustFS Release Publish (preview-validated pipeline)
 

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: rustfs-release-version-bump
-description: "Prepare the version-file and release-asset bump for an exact RustFS alpha/beta/stable target, with verification and optional commit/push/PR delivery. Use for an explicit version bump or when invoked by the release-publish workflow."
+description: "Publish a RustFS alpha/beta/stable release with an auditable flow: confirm target version and scope, update workspace and release assets (including strict rustfs.spec changelog identity/date/version format), run required verification, and finish with commit, push, and GitHub PR creation."
 ---
 # RustFS Release Version Bump
 
@@ -81,7 +81,10 @@ Only drop a file when the current repository release process clearly no longer r
 
 4. Verify before shipping
 - Run:
+- `cargo fmt --all`
+- `cargo fmt --all --check`
 - `make pre-commit`
+- If verification passes, run `cargo clean`.
 - If `make pre-commit` fails, return `BLOCKED` with root cause and do not silently widen scope to fix unrelated issues unless user asks.
 
 5. Commit strategy
@@ -106,7 +109,10 @@ Only drop a file when the current repository release process clearly no longer r
 - `git diff --name-only origin/main...HEAD`
 - `git diff --stat origin/main...HEAD`
 - `rg -n "<old_version>|<new_version>" Cargo.toml Cargo.lock README.md README_ZH.md flake.nix helm/rustfs/Chart.yaml rustfs.spec`
+- `cargo fmt --all`
+- `cargo fmt --all --check`
 - `make pre-commit`
+- `cargo clean`
 
 ## Output contract
 

.agents/skills/rustfs-release-version-bump/agents/openai.yaml

@@ -1,4 +1,4 @@
 interface:
   display_name: "RustFS Release Bump"
   short_description: "Prepare RustFS release branches like PR #2957."
-  default_prompt: "Use $rustfs-release-version-bump to prepare and verify an exact RustFS release-version bump."
+  default_prompt: "Use $rustfs-release-version-bump to prepare a RustFS release version, ask about any unclear version policy, and finish the commit/push/PR flow."

.agents/skills/security-advisory-lessons/SKILL.md

@@ -1,40 +1,152 @@
 ---
 name: security-advisory-lessons
-description: Perform a dedicated RustFS security/advisory review for authn/authz, IAM, RPC trust, paths, secrets, browser isolation, encryption, Object Lock, or other security boundaries. Use only when the user requests a security/advisory review or an adversarial review explicitly escalates to the full advisory map; do not auto-load solely because code touches a sensitive path.
+description: Apply RustFS security lessons distilled from repository GitHub Security Advisories. Use when making or reviewing RustFS code changes, doing security checks, handling PR review for auth/authz, IAM, storage, RPC, logging, CORS, console/browser, encryption, policy, or endpoint changes, and when deciding which security regression tests are required.
 ---
 
 # RustFS Security Advisory Lessons
 
-Use this skill as the deep security lens. For a normal adversarial review with a
-matched security surface, the concise security reference under
-`adversarial-validation` is sufficient.
+Use this skill as a RustFS-specific security lens before changing or approving code. For the distilled advisory lessons and review patterns, read [advisory-patterns.md](references/advisory-patterns.md).
 
-## Workflow
-
-1. Freeze the exact diff/head and identify the changed trust boundaries.
-2. Read [advisory-patterns.md](references/advisory-patterns.md), then apply only
-   the matching sections. Useful headings are
-   auth/admin, IAM/STS/OIDC, policy/plugins, S3/copy/multipart, protocols, paths,
-   secrets/logging/RPC, browser/CORS/proxy, SSE, Object Lock, and serde.
-3. Trace unauthenticated, low-privilege, wrong-action/owner/bucket, malformed,
-   and default-config cases. Security decisions must fail closed.
-4. Require a focused negative regression test for the bypass/exploit form, not
-   only the intended success path. State residual risk when a test is impractical.
-5. Report proven vulnerabilities separately from defense-in-depth hardening.
-
-When advisory currentness matters, fetch the live inventory instead of treating
-the reference as a status mirror:
+When currentness matters, fetch the live advisory inventory instead of relying on this skill as a status mirror:
 
 ```bash
 gh api repos/rustfs/rustfs/security-advisories --paginate \
   --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
 ```
 
-Fetch an individual advisory only when the live summary indicates a new or
-changed lesson.
+Fetch full advisory details only when the live summary suggests a new or changed lesson:
 
-## Finding Standard
+```bash
+gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
+```
 
-Each finding includes severity, `file:line`, attacker prerequisites, concrete
-input/path, impact, smallest safe fix, and a regression check. Do not exaggerate
-unauthenticated impact when the actual issue requires authenticated low privilege.
+For the full pattern map, read [advisory-patterns.md](references/advisory-patterns.md).
+
+## Workflow
+
+### 1. Scope the change
+- Identify touched routes, protocol frontends, handlers, storage paths, credentials, logs, browser surfaces, CI/release code, and policy checks.
+- Treat these paths as security-sensitive by default: `rustfs/src/admin/`, `rustfs/src/storage/`, `rustfs/src/auth.rs`, `rustfs/src/server/layer.rs`, `crates/iam/`, `crates/policy/`, `crates/credentials/`, `crates/ecstore/src/rpc/`, `crates/protocols/`, `crates/rio/`, OIDC/STS federation code, and console preview/auth code.
+
+### 2. Map to advisory classes
+- Read [advisory-patterns.md](references/advisory-patterns.md) for matching GHSA lessons.
+- Do not rely on advisory titles alone. Confirm whether the issue is authentication, authorization, input validation, storage invariant, browser isolation, logging, or operational hardening.
+
+### 3. Verify fail-closed behavior
+- Check that unauthenticated, wrong-permission, cross-user, cross-bucket, malformed-input, and default-config cases fail explicitly.
+- Prefer exact action/permission checks over broad helper calls or inferred ownership.
+- Confirm lower storage/RPC layers do not bypass checks done in upper layers.
+
+### 4. Require regression evidence
+- For behavior changes, add focused negative tests that reproduce the advisory class.
+- For sensitive fixes, include tests for the bypass form, not only the happy path.
+- If a test is impractical, explain the residual risk and provide a manual verification command.
+
+### 5. Report clearly
+- Lead with concrete findings and file/line evidence.
+- Separate proven vulnerabilities from hardening risks.
+- Avoid exaggerating unauthenticated impact when the code actually rejects unauthenticated requests but allows a low-privileged authenticated bypass.
+
+## Advisory-Derived Guardrails
+
+### Auth and admin authorization
+- Every admin or diagnostic route needs an explicit authn and authz story. Route registration, router whitelist, and handler-level authorization must agree.
+- Match the admin action to the operation exactly. Copy-paste action constants are a known RustFS vulnerability class.
+- Avoid authentication-only helpers for state-changing admin APIs; use `validate_admin_request` or the established equivalent with the right `AdminAction`.
+- Read-only admin APIs such as metrics, server info, and diagnostics still require admin authorization; checking only that credentials exist is not enough.
+- Replication admin reads can expose remote target credentials; list/get target endpoints require replication/admin authorization and must not return secrets to low-privilege callers.
+- Do not assume admin-action `Resource` scoping constrains blast radius unless the policy engine actually enforces resources for that action.
+
+### IAM and service accounts
+- Treat imported IAM payload fields as attacker-controlled: `parent`, `claims`, `accessKey`, `secretKey`, status, policy names, and groups.
+- For service account create/update/import, prove parent ownership or root/admin authority before writing credentials or claims; an action permission alone must not allow choosing root or another user as `target_user`.
+- Treat IAM export packages as credential disclosure surfaces; never include plaintext user or service-account secret keys unless the caller is allowed to recover those secrets and the export format is intentionally sealed.
+- Do not let `deny_only` or "no explicit deny" become an allow decision that skips required allow checks.
+- Test cross-user list/update/import flows with wrong, correct, self, parent, and root identities.
+
+### STS, OIDC, and federation flows
+- Every STS endpoint must have an explicit authentication story: SigV4 where required, OIDC token verification for web identity, and role/session policy validation before issuing credentials.
+- JWT session tokens must be signed and verified by a trusted issuer/key path, not by service-account-controlled material or a reused root secret.
+- JWT verification must enforce required claims and expiration for every bearer token path; "allow missing exp" is never acceptable for user-presented credentials.
+- Public OIDC bootstrap and callback routes must treat `Host`, `X-Forwarded-Proto`, redirect targets, `state`, and callback parameters as untrusted; credential-bearing redirects require a configured, allowlisted origin.
+- OIDC discovery and validation URLs are SSRF sinks. Resolve and classify hostnames at connection time, reject rebinding to loopback/private/link-local ranges, and do not rely on literal string checks.
+
+### S3 copy, multipart, and presigned POST
+- Multipart copy must enforce source `GetObject` and destination `PutObject` semantics equivalent to `CopyObject`, including copy-source and policy conditions.
+- Do not let `CreateMultipartUpload`, `UploadPartCopy`, `CompleteMultipartUpload`, or `AbortMultipartUpload` return success without authorization.
+- Presigned POST policies are server-side contracts. Enforce `content-length-range`, key prefix, exact metadata/content-type, and all signed policy conditions.
+
+### Protocol frontends and IAM parity
+- FTP/FTPS, SFTP, gateway, and other protocol drivers must enforce IAM per operation before calling storage backends; authentication to a protocol listener is not authorization.
+- Match protocol commands to the same S3 actions as HTTP, such as `RETR` to `GetObject`, `SIZE`/`MDTM` to `HeadObject`, `MKD` to `CreateBucket`, and bucket probes to `ListBucket` or `HeadBucket`.
+- Review every handler in a protocol driver, not only the changed handler, because RustFS advisories show mixed guarded and unguarded siblings in the same driver.
+- Regression tests for protocol frontends should deny the shared authorization hook and prove the backend is not reached for the denied command.
+- Compare protocol secrets in constant time, normalize invalid-user and invalid-secret failures where practical, and add rate limiting before exposing password-style protocol endpoints.
+
+### Paths, object keys, and filesystem access
+- Never join untrusted bucket/object/RPC path strings onto filesystem roots without normalization and boundary checks.
+- Reject or safely handle `..`, absolute paths, URL-encoded traversal, platform separators, empty components, and paths that canonicalize outside the intended root.
+- Validate both S3 object-key paths and internode/RPC disk paths; storage helpers can bypass S3 authorization if they trust already-parsed paths.
+- Archive auto-extract paths are object keys too. Validate tar/zip entry names before IAM checks and before storage writes, and prove cleaned paths cannot cross bucket or prefix boundaries.
+
+### Secrets, default credentials, and crypto
+- Do not ship hard-coded shared tokens, HMAC secrets, private keys, or production test keys.
+- Defaults for root credentials and internode/RPC auth must fail closed for network-reachable deployments or generate per-install random secrets; warnings alone are not a security boundary.
+- Keep cryptographic roles separated: root S3 credentials, RPC HMAC keys, and STS/JWT signing keys must not be reused or deterministically derived from each other.
+- License or token validation must use signatures with embedded public/verifying keys only; do not use private-key decryption as authenticity.
+- Plan key rotation and key IDs when removing exposed keys.
+
+### Logging and debug output
+- Logs must never include access keys beyond safe identifiers, secret keys, session tokens, JWT claims, HMAC secrets, expected signatures, license secrets, or raw response bodies containing credentials.
+- Treat `Debug` implementations, `?value` tracing, merged config dumps, and dependency-level HTTP body logging as leak surfaces.
+- Error and panic messages are log content: they propagate through `?` and get printed by `error!`/startup logging far from where they were constructed. Never interpolate a raw config or credential value into an error string.
+- A value that fails secret-format parsing is usually the secret itself (e.g. a bare base64 key missing its `<name>:` prefix), so a parse-failure hint must name the env var or file and the expected format, never echo the input. Redacting `Debug` impls does not cover this channel.
+- Add log-capture tests or targeted unit tests for redaction wrappers when changing credential structs or response bodies.
+
+### RPC, parsing, and panic safety
+- Treat all RPC payload bytes as attacker-controlled. Replace `unwrap`, `expect`, and panic-prone deserialization with typed errors.
+- Malformed request tests should cover empty bytes, truncated MessagePack/protobuf, invalid enum values, stale timestamps, and invalid signatures.
+- RPC authentication must be independently strong; do not depend on S3 admin credentials unless the fallback is explicit and safe.
+- RPC signatures must bind the exact generated gRPC method path, timestamp, and request method. Service-prefix signatures must not authorize a different concrete NodeService call.
+
+### Browser, CORS, and console surfaces
+- Do not reflect arbitrary `Origin` while also allowing credentials. Default CORS should be no CORS unless explicitly configured.
+- Do not render user-controlled object content in a same-origin iframe with console credentials available to JavaScript.
+- Prefer origin separation for object preview/download, `nosniff`, CSP, strict content-type handling, and avoiding durable credentials in `localStorage`.
+- Preview safety must be based on trusted content type and sandboxing, not object names or extensions such as `.pdf`.
+- Console license/version-like metadata endpoints should expose only coarse public data unless authenticated, especially subject names and expiration timestamps.
+
+### Profiling, debug, and health endpoints
+- Profiling and debug endpoints are not health checks. They require admin auth, opt-in enablement, rate limiting, and safe responses.
+- Do not return absolute filesystem paths or other deployment layout in unauthenticated or low-privilege responses.
+- Ensure health endpoint allowlists cannot accidentally include expensive diagnostics.
+
+### Trusted proxy and network identity
+- Only honor `X-Forwarded-For` or `X-Real-IP` when the request came from a configured trusted proxy.
+- Apply the same trusted-proxy rule to scheme and host derivation; direct clients must not control security-sensitive redirects through `Host`, `X-Forwarded-Host`, or `X-Forwarded-Proto`.
+- Direct clients must use the socket peer address for `aws:SourceIp` and policy condition evaluation.
+- Add tests for direct spoofed headers and trusted-proxy headers.
+
+### SSE and storage invariants
+- Encryption metadata is not proof that bytes were encrypted on disk.
+- When touching reader/writer wrappers such as hashing, encryption, compression, or warp readers, verify wrapper order and inspect stored bytes in regression tests.
+- Avoid helper shortcuts that unwrap nested readers and accidentally bypass encryption or integrity layers.
+
+## Review Prompts
+
+Use these prompts while reviewing a diff:
+
+- Could a low-privileged authenticated user reach this path with the wrong action, parent, bucket, or source object?
+- Does a non-HTTP protocol path call the same authorization boundary as the S3 API before touching storage?
+- Does a public/default/empty config change security behavior from fail-closed to fail-open?
+- Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
+- Does this response contain stored replication, remote target, or service credentials that need redaction or stricter authorization?
+- Does any error constructor or `format!` interpolate a variable that can hold secret material, including a config parse error that echoes the raw input?
+- Does an IAM export/import path expose or trust plaintext credential secrets beyond the caller's intended authority?
+- Can this STS/OIDC path issue credentials without SigV4, trusted issuer validation, allowlisted redirects, or trusted-proxy host/scheme handling?
+- Can a service-account or STS token omit `exp`, forge `sessionPolicy`, or use a principal-controlled key as signing authority?
+- Does this outbound validation path resolve attacker-supplied hostnames and reject private, loopback, link-local, and rebound addresses at the actual connection boundary?
+- Is an archive entry, object key, or policy resource normalized differently between authorization and storage?
+- Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
+- Does a preview or browser-surface fix preserve the original security invariant when adding alternate viewers or file-type detection?
+- Does the test prove the exploit form is denied, or only that the intended form still works?

.agents/skills/security-advisory-lessons/agents/openai.yaml

@@ -1,4 +1,4 @@
 interface:
   display_name: "Security Advisory Lessons"
   short_description: "Apply advisory lessons in reviews."
-  default_prompt: "Use $security-advisory-lessons for a dedicated RustFS security review grounded in past advisories."
+  default_prompt: "Review code changes against past RustFS security advisory lessons and report concrete risks, missing tests, and recommended fixes."

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -35,21 +35,12 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 ### STS, OIDC, and federation flows
 
 - `GHSA-5qfg-mf7r-jp3w` and `GHSA-3473-5353-xhwh`: `AssumeRoleWithWebIdentity` was reachable through unauthenticated `POST /` routing and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption, and unauthenticated exemptions must be narrowed to the exact action with uniform failure responses.
-- `GHSA-jxrr-r6pv-h958`: unsigned JWT issuer data was decoded before verification to select an OIDC provider, and distinguishable failures could expose provider configuration. Lesson: web-identity routing may be unauthenticated, but pre-verification claims are untrusted routing hints; bound and rate-limit the request, normalize public errors, and verify signature, issuer, audience, and expiration before issuing credentials.
-- `GHSA-ccrv-v8v9-ch9q`, `GHSA-48rf-7j3q-3hfv`, and `GHSA-xvfh-7c9g-hpw2`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
+- `GHSA-ccrv-v8v9-ch9q` and `GHSA-48rf-7j3q-3hfv`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
 - `GHSA-9pjf-w3c2-m32r`, `GHSA-4x2q-cpx9-9h26`, and `GHSA-xvpm-p3f7-34c3`: public OIDC authorize/callback flows trusted request `Host` or forwarded scheme when building credential-bearing redirects. Lesson: OIDC redirects must use configured allowlisted origins and trusted-proxy handling; never derive the post-login credential destination from direct client headers.
 - `GHSA-m479-9x88-94w6`, `GHSA-frwq-mfqx-83p8`, `GHSA-q9q8-rf9r-fg9f`, and `GHSA-j5c2-hhf7-6gf5`: OIDC validation accepted attacker-controlled discovery URLs because hostname checks rejected only literal forbidden IPs, allowing DNS rebinding SSRF. Lesson: outbound federation URL validation must resolve and classify hostnames at the connection boundary and reject loopback, private, link-local, and rebound addresses.
 
-### IAM policy conditions and external policy plugins
+### S3 copy, multipart, and upload policy validation
 
-- `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
-- `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
-- `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
-
-### S3 object actions, copy, multipart, and upload policy validation
-
-- `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
-- `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
 - `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
 - `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
 - `GHSA-w5fh-f8xh-5x3p`: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
@@ -68,7 +59,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 ### Secrets, defaults, and cryptographic misuse
 
-- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, `GHSA-63xc-c3w3-m2cf`, and `GHSA-ch63-6q4v-hwp5`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
+- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, and `GHSA-63xc-c3w3-m2cf`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
 - `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
 - `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
 - `GHSA-75fx-qg6f-8rm7` and `GHSA-68cw-96m3-h2cf`: internode RPC secrets were derivable from known root credentials, making raw storage RPC signatures forgeable when explicit RPC secrets were unset. Lesson: RPC auth keys must be independent random secrets, never derived from S3 root credentials, and raw storage RPC should not share the public S3 listener without an internode-only boundary.
@@ -101,10 +92,6 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
 
-### Object Lock and retention invariants
-
-- `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
-
 ### Serde deserialization and input validation
 
 - No `#[serde(deny_unknown_fields)]` found across the entire codebase. Lesson: all structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs) should have `#[serde(deny_unknown_fields)]` to reject malformed or adversarial payloads.
@@ -120,13 +107,11 @@ Use these targeted searches when a diff touches security-sensitive code:
 rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
 rg -n "authorize_operation|FtpsDriver|SftpDriver|RETR|MKD|SIZE|MDTM|CreateBucket|GetObject|HeadObject" crates/protocols rustfs
 rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|content-length-range|starts-with" rustfs crates
-rg -n "ListBucketVersions|GetObjectVersion|versionId|VersionId|ExistingObjectTag|ForAllValues|ForAnyValue|POLICY_PLUGIN|opa" rustfs crates
 rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
 rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
 rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
 rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
 rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
-rg -n "ObjectLock|object_lock|retention|COMPLIANCE|GOVERNANCE|delete_prefix|lifecycle|scanner" rustfs crates
 rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 ```
 
@@ -136,12 +121,9 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 - Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases, and assert denied paths do not reach the storage backend.
 - IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
 - Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
-- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases.
-- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode.
 - Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
 - Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
 - IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
 - RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
 - Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
 - SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.
-- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths.

.config/e2e-full-selection.txt

@@ -1,2 +0,0 @@
-sha256-darwin=b4ae71aa894e5c7795ae3eb8116f1777a7601d0f5db3898be2e48faf3329bd9b
-sha256-linux=433debd9d9defa832986269abdf0f1d131597b2d7a417ce930e17c1fd47d85ba

.config/e2e-nightly-selection.txt

@@ -1 +0,0 @@
-sha256=9b9bc336b43b70d0e06e0adb5455bf035bb18945d85d60936eb6fe4d48e0e680

.config/e2e-protocols-selection.txt

@@ -1,2 +0,0 @@
-sha256-darwin=55534a97fbd376f64c8f6c341d319017d11ff77cad6da8629a1a7f6a874e0315
-sha256-linux=c06fb8c19aed6f388b9dc61cb8251b7a44f8561a9bf764ad2b9e635598f8dc17

.config/e2e-repl-nightly-selection.txt

@@ -1 +0,0 @@
-sha256=655a3f3c1d042e694339d15caba7580518320322d1bac0f09450b37e6c09e2e7

.config/e2e-smoke-selection.txt

@@ -1 +0,0 @@
-sha256=ec27cde6ce6400723c4b372bfbd2ac61709c744294e4810af765e8a808d8e31d

.config/make/lint-fmt.mak

@@ -60,26 +60,11 @@ body-cache-whitelist-check: ## Check the body-cache eligibility gate stays a fai
 	@echo "🧱 Checking body-cache whitelist guard..."
 	./scripts/check_body_cache_whitelist.sh
 
-.PHONY: s3s-footprint-check
-s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
-	@echo "📦 Checking s3s footprint ratchet..."
-	./scripts/check_s3s_footprint.sh
-
 .PHONY: fips-wording-check
-fips-wording-check: ## Check docs and crates/kms do not over-claim crypto capabilities
-	@echo "📣 Checking cryptographic capability wording guard..."
+fips-wording-check: ## Check outward docs do not make unsupported FIPS claims
+	@echo "📣 Checking FIPS wording guard..."
 	./scripts/check_fips_wording.sh
 
-.PHONY: embedded-secrets-check
-embedded-secrets-check: ## Check no private key material or credential literal is committed
-	@echo "🔑 Checking embedded secret material guard..."
-	./scripts/check_embedded_secrets.sh
-
-.PHONY: test-wiring-check
-test-wiring-check: ## Check tests stay registered and selected by their intended runners
-	@echo "🧪 Checking test wiring..."
-	python3 ./scripts/check_test_wiring.py
-
 .PHONY: log-analyzer-rules-check
 log-analyzer-rules-check: core-deps ## Check log-analyzer rule anchors still exist verbatim in source
 	@echo "🩺 Checking log-analyzer rule anchors..."

.config/make/pre-commit.mak

@@ -19,13 +19,13 @@ planning-docs-check: ## Check that no planning-type documents are committed
 	./scripts/check_no_planning_docs.sh
 
 .PHONY: pre-commit
-pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
+pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check fips-wording-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
 	@echo "✅ All pre-commit checks passed!"
 
 .PHONY: pre-pr
-pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests
+pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check fips-wording-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests
 	@echo "✅ All pre-PR checks passed!"
 
 .PHONY: dev-check
-dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
+dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check fips-wording-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
 	@echo "✅ Fast development checks passed!"

.config/make/tests.mak

@@ -25,7 +25,6 @@ TEST_THREADS ?= 1
 script-tests: ## Run shell script tests
 	@echo "Running script tests..."
 	./scripts/test_build_rustfs_options.sh
-	./scripts/test_docker_runtime_timezone.sh
 	./scripts/test_entrypoint_credentials.sh
 	./scripts/test_internode_grpc_ab_bench.sh
 	./scripts/test_object_batch_bench_enhanced.sh
@@ -34,9 +33,6 @@ script-tests: ## Run shell script tests
 	./scripts/test_exact_1mib_handoff_abba.sh
 	./scripts/test_pinned_paired_abba_bench.sh
 	./scripts/test_manual_transition_runbooks.sh
-	./scripts/check_embedded_secrets.sh --self-test
-	python3 ./scripts/check_test_wiring.py --self-test
-	python3 ./scripts/s3-tests/test_report_compat.py
 	bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
 	python3 ./scripts/check_object_data_cache_follower_samples.py --self-test
 	./scripts/validate_object_data_cache_cold_stampede.sh --self-test

.config/nextest.toml

@@ -29,20 +29,16 @@
 
 [test-groups]
 ecstore-serial-flaky = { max-threads = 1 }
-embedded-test-ports = { max-threads = 1 }
-e2e-vault = { max-threads = 1 }
 
 # Reliability / fault-injection e2e tests each spawn a single-node 4-disk RustFS
 # server and manipulate its disk directories at runtime (crates/e2e_test:
-# reliability_disk_fault_test, degraded_read_eof_regression_test / dist-13, and
-# replacement_privileged_e2e_test when explicitly run as root on Linux). They
+# reliability_disk_fault_test, degraded_read_eof_regression_test / dist-13). They
 # are correct in isolation but resource-heavy; serialize them under nextest's
 # process boundary (serial_test's #[serial] does not cross it) so several 4-disk
-# servers never run at once. The e2e-full merge/main lane picks these up;
+# servers never run at once. ci-7's nightly picks these up via the e2e suite;
 # they are deliberately NOT in the fast PR `e2e-smoke` filter.
 e2e-reliability = { max-threads = 1 }
 e2e-inline-boundaries = { max-threads = 1 }
-e2e-cluster-nightly = { max-threads = 1 }
 
 # --- default profile (local): serialize the flaky groups, never retry --------
 [[profile.default.overrides]]
@@ -58,20 +54,6 @@ test-group = 'ecstore-serial-flaky'
 filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
 test-group = 'ecstore-serial-flaky'
 
-# The production-handler relocation regression builds an isolated 8-disk,
-# 2-pool store and commits a 72 MiB multipart object. Keep that cross-disk IO
-# from overlapping the ecstore commit fixtures above.
-[[profile.default.overrides]]
-filter = 'package(rustfs) & test(execute_get_object_resumes_from_relocated_pool_without_splicing_body)'
-test-group = 'ecstore-serial-flaky'
-
-# Embedded integration-test binaries discover an ephemeral port and release
-# the probe listener before RustFS binds it. Serialize that cross-process
-# TOCTOU window; retries would only hide real startup failures.
-[[profile.default.overrides]]
-filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
-test-group = 'embedded-test-ports'
-
 # Serialize the durable manual-transition checkpoint test across nextest's
 # process boundary; it mutates bucket lifecycle metadata and is not quarantined.
 [[profile.default.overrides]]
@@ -88,30 +70,17 @@ test-group = 'ecstore-serial-flaky'
 filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
 test-group = 'ecstore-serial-flaky'
 
-# Serialize the default-off dst-dir fsync group-commit tests. They use
-# process-global test hooks/registry to deterministically freeze fsync batches;
-# no retries, just one at a time under nextest too.
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(dst_dir_fsync_group_commit)'
-test-group = 'ecstore-serial-flaky'
-
 # Serialize the 4-disk reliability / degraded-read e2e tests (see the
 # e2e-reliability test-group note above). The matching ci-profile override is at
 # the end of the file, after [profile.ci] is declared.
 [[profile.default.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
+filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
 test-group = 'e2e-reliability'
 
 [[profile.default.overrides]]
 filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
 test-group = 'e2e-inline-boundaries'
 
-# Vault KMS tests share the fixed dev-server port 8200. serial_test's #[serial]
-# does not cross nextest process boundaries, so keep these tests in one group.
-[[profile.default.overrides]]
-filter = 'package(e2e_test) & test(/^kms::kms_vault_test::/)'
-test-group = 'e2e-vault'
-
 # ---------------------------------------------------------------------------
 # ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
 # ---------------------------------------------------------------------------
@@ -162,9 +131,9 @@ retries = 2
 # Serialize the 4-disk reliability / degraded-read e2e tests under the ci
 # profile too (see the e2e-reliability test-group note near the top). Not a
 # quarantine: no retries, just single-threaded so several 4-disk servers never
-# run concurrently when e2e-full runs the suite.
+# run concurrently when ci-7's nightly runs the full e2e suite.
 [[profile.ci.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
+filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
 test-group = 'e2e-reliability'
 
 # Serialize the multipart crash-consistency scenarios under the ci profile too
@@ -174,16 +143,6 @@ test-group = 'e2e-reliability'
 filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
 test-group = 'ecstore-serial-flaky'
 
-[[profile.ci.overrides]]
-filter = 'package(rustfs) & test(execute_get_object_resumes_from_relocated_pool_without_splicing_body)'
-test-group = 'ecstore-serial-flaky'
-
-# Match the default-profile embedded test isolation without quarantining or
-# retrying failures in CI.
-[[profile.ci.overrides]]
-filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
-test-group = 'embedded-test-ports'
-
 # Serialize the durable manual-transition checkpoint test under the ci profile
 # too. No retries: failures stay visible.
 [[profile.ci.overrides]]
@@ -196,10 +155,6 @@ test-group = 'ecstore-serial-flaky'
 filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
 test-group = 'ecstore-serial-flaky'
 
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(dst_dir_fsync_group_commit)'
-test-group = 'ecstore-serial-flaky'
-
 # ---------------------------------------------------------------------------
 # e2e-smoke profile — PR smoke subset of the e2e_test crate (backlog#1149 ci-4)
 # ---------------------------------------------------------------------------
@@ -231,8 +186,8 @@ test-group = 'ecstore-serial-flaky'
 # the nightly profile derives its set as "the replication module MINUS this
 # allowlist", so any new replication test lands in nightly by default (never
 # silently unrun) until it is explicitly blessed as fast here. Keep the two
-# regexes byte-identical. The committed profile selection digests make changes
-# visible in CI; current counts live in docs/testing/e2e-suite-inventory.md.
+# regexes byte-identical. Count invariant: 20 here + 28 nightly = 48 total
+# (authority: `cargo nextest list`; docs/testing/e2e-suite-inventory.md).
 # HISTORY (2026-07-11): the 20 fast tests were briefly pulled out of this lane
 # (#4724) because they set a loopback (127.0.0.1) replication target that the
 # SSRF egress guard rejected on every PR after repl-1 (#4712). That is fixed —
@@ -264,16 +219,10 @@ test-group = 'ecstore-serial-flaky'
 # cluster, so it keeps the lane's parallel-safe / no-external-dependency
 # properties. The RustFS warm backend has no loopback guard (that guard is
 # replication-only), so it needs no opt-in env for its 127.0.0.1 tier target.
-#
-# Disk compression (backlog#1848): the `compression` module joins the smoke
-# lane so the multipart disk-compression roundtrips (restored after
-# rustfs/rustfs#5169 disabled them) have PR-lane signal, not just merge-gate.
-# Single-node servers on random ports with isolated temp dirs — meets the
-# admission criteria unchanged.
 [profile.e2e-smoke]
 default-filter = """
     package(e2e_test) & (
-        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|compression|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat)_test::|^fake_s3_target::/)
+        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat)_test::|^fake_s3_target::/)
         | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
         | test(/^reliant::lifecycle::/)
         | test(/^reliant::tiering::/)
@@ -299,12 +248,10 @@ slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
 # tests that are unfit for the per-PR e2e-smoke gate:
 #
 #   * 2 remote-target TLS validation tests.
-#   * 15 bucket-replication data-plane/helper tests — they PUT/delete objects
-#     and poll until source and target converge; two replicate over HTTPS,
-#     six pin SSE replication contracts (managed SSE-S3/SSE-KMS re-encrypt on
-#     the target incl. multipart and the resync path, SSE-C and
-#     target-without-KMS stay fail-closed), and one guards event/history
-#     observers.
+#   * 12 bucket-replication data-plane/helper tests — they PUT/delete objects
+#     and poll until source and target converge; two replicate over HTTPS, two
+#     pin active SSE failure contracts, and one guards event/history observers.
+#     The SSE-S3 contract remains ignored under backlog#1291.
 #   * 12 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
 #     servers and drives the cross-process site-replication control plane.
 #   * 1 `_real_three_node` site-replication test.
@@ -328,8 +275,9 @@ slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
 # the STS dual-node test actually exercises its path (it skips gracefully with
 # a visible log line when awscurl is absent), and routes scheduled failures
 # through .github/actions/schedule-failure-issue (ci-8). Explicit division of
-# labor with e2e-full: these tests run only in the consolidated nightly
-# workflow, not in the merge/main lane.
+# labor with ci-5's future e2e-full merge gate: these tests run ONLY here, not
+# double-run there. TODO(ci-7): fold this interim repl-owned lane into the ci
+# domain's consolidated scheduled e2e workflow once it exists.
 [profile.e2e-repl-nightly]
 default-filter = """
     package(e2e_test)
@@ -343,69 +291,35 @@ fail-fast = false
 # workflow as the failure-triage artifact.
 path = "junit.xml"
 
-# ---------------------------------------------------------------------------
-# e2e-nightly profile — destructive multi-process cluster fault domains
-# ---------------------------------------------------------------------------
-# These seven modules are deliberately outside e2e-full's merge budget. Each
-# starts a real multi-process or multi-disk topology and exercises node/disk
-# loss, quorum, cleanup, notification fan-in, or admin-timeout behavior. The
-# consolidated nightly workflow runs them serially to avoid resource
-# starvation; failures are never retried.
-[profile.e2e-nightly]
-default-filter = """
-    package(e2e_test)
-    & test(/^(admin_timeout_regression_test|cluster_concurrency_test|cluster_multidrive_pool_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
-"""
-fail-fast = false
-
-[profile.e2e-nightly.junit]
-path = "junit.xml"
-
-[[profile.e2e-nightly.overrides]]
-filter = 'package(e2e_test)'
-test-group = 'e2e-cluster-nightly'
-
-# ---------------------------------------------------------------------------
-# e2e-protocols profile — serial protocol lane
-# ---------------------------------------------------------------------------
-# The suite owns fixed ports, so the nightly workflow runs this exact profile
-# with one nextest worker.
-[profile.e2e-protocols]
-default-filter = 'package(e2e_test) & test(/^protocols::/)'
-fail-fast = false
-
-[profile.e2e-protocols.junit]
-path = "junit.xml"
-
 # ---------------------------------------------------------------------------
 # e2e-full profile — merge-gate full single-node e2e lane (backlog#1149 ci-5)
 # ---------------------------------------------------------------------------
 # The merge gate (ci.yml `e2e-full` job: push main + merge_group +
-# workflow_dispatch). Runs the user-visible KMS, object-lock, multipart-auth,
-# quota, checksum, encryption,
+# workflow_dispatch). Runs the never-automated user-visible suites — KMS (40),
+# object_lock (33), multipart_auth (109), quota, checksum, encryption,
 # security-boundary, ... — that the fast PR `e2e-smoke` subset deliberately
 # skips. Budget <= 45 min; authority for the suite count is `cargo nextest list
 # --profile e2e-full` (see docs/testing/e2e-suite-inventory.md).
 #
 # The filter is "the whole e2e_test crate MINUS the sets owned by other lanes":
-#   * protocols:: — FTPS/SFTP/WebDAV, run from the dedicated protocol profile
-#     with one worker because the suite owns fixed ports.
+#   * protocols:: — FTPS/SFTP/WebDAV, still pinned to --test-threads=1 by fixed
+#     ports; they join a scheduled lane once ci-6 randomises the ports (ci-7).
 #   * the 7 cluster suites that spin up a RustFSTestClusterEnvironment
 #     (cluster_concurrency, cluster_multidrive_pool, stale_multipart_cleanup_cluster,
 #     namespace_lock_quorum, heal_erasure_disk_rebuild, admin_timeout_regression,
-#     object_lambda) — too heavy for the merge budget; they run in the
-#     e2e-nightly serial cluster-fault lane.
+#     object_lambda) — too heavy for the merge budget; they run in ci-7's
+#     nightly 4-node lane.
 #   * replication_extension_test — repl-1 already splits it into the PR
-#     `e2e-smoke` (20 fast) and `e2e-repl-nightly` (55 slow) lanes and reserves
+#     `e2e-smoke` (20 fast) and `e2e-repl-nightly` (27 slow) lanes and reserves
 #     it for those, so e2e-full does not double-run it.
 #   * #[ignore]d tests — nextest skips them by default (no --run-ignored); the
 #     manual-localhost:9000 reliant/policy tests are ci-13's migration.
 #
 # Each e2e test spawns its own single-node rustfs server on a random port with
 # an isolated temp dir (crates/e2e_test/src/common.rs), so the set is
-# parallel-safe — the same property e2e-smoke relies on. The exceptions are the
-# 4-disk reliability / degraded-read fault-injection tests and the fixed-port
-# Vault tests, both serialized below.
+# parallel-safe — the same property e2e-smoke relies on. The exception is the
+# 4-disk reliability / degraded-read fault-injection tests, serialized below
+# (identical to the ci profile) so several 4-disk servers never run at once.
 # KNOWN-FAILURE EXCLUSIONS (characterization run 29381309848, 2026-07-15:
 # 341 ran / 32 failed on the suites' first automated run ever). Deterministic
 # product failures cannot be quarantined away with retries, so each family is
@@ -435,13 +349,9 @@ path = "junit.xml"
 # quarantine: no retries, just single-threaded so several 4-disk servers never
 # run concurrently.
 [[profile.e2e-full.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
+filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
 test-group = 'e2e-reliability'
 
 [[profile.e2e-full.overrides]]
 filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
 test-group = 'e2e-inline-boundaries'
-
-[[profile.e2e-full.overrides]]
-filter = 'package(e2e_test) & test(/^kms::kms_vault_test::/)'
-test-group = 'e2e-vault'

.docker/observability/README.md

@@ -170,10 +170,6 @@ Important behavior notes:
 
 - Logs and metrics usually appear during startup, so seeing those two signals
   first is expected.
-- The OpenTelemetry bridge sends `tracing` fields as log attributes. Loki stores
-  those attributes as structured metadata, and the Collector also mirrors the
-  common troubleshooting fields into the log line so simple line filters can
-  find them.
 - Visible trace data usually requires real HTTP/S3/gRPC request traffic after
   startup, because request-path spans are created on demand.
 - `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
@@ -199,17 +195,6 @@ curl -I http://127.0.0.1:9000/health/ready
 # Jaeger:  http://localhost:16686
 ```
 
-For a structured RustFS log such as an inter-node RPC authentication failure,
-the Loki line now includes fields such as `event`, `component`, `subsystem`,
-`failure_reason`, `rpc_service`, `rpc_method`, and `expected_audience`. Useful
-LogQL checks:
-
-```logql
-{service_name="RustFS"} |= "RPC signature verification failed"
-{service_name="RustFS"} |= "failure_reason="
-{service_name="RustFS"} | failure_reason != ""
-```
-
 If logs and metrics are present but traces are sparse, the most common cause is
 "no real request traffic yet" or "`info` level filtered nested spans", not an
 OTLP routing failure.

.docker/observability/README_ZH.md

@@ -169,7 +169,6 @@ RustFS 会自动在该基础 URL 后补全：
 需要注意：
 
 - 启动阶段通常会先看到日志和指标，因此“先有日志/指标、后有 trace”是正常现象。
-- OpenTelemetry bridge 会把 `tracing` 字段作为日志 attributes 发送。Loki 会将这些 attributes 存为 structured metadata，同时 Collector 会把常用排障字段镜像进日志行，方便用简单的行内容过滤直接查到。
 - 可见的 trace 数据通常依赖启动后的真实 HTTP/S3/gRPC 请求流量，因为请求路径上的 span 是按需创建的。
 - `RUSTFS_OBS_LOGGER_LEVEL=info` 会保留顶层请求 span，但会过滤掉很多 `debug` 级别的嵌套 span。
   如果 Tempo 或 Jaeger 中的 trace 看起来很稀疏，建议先改成 `RUSTFS_OBS_LOGGER_LEVEL=debug`，再判断是否是 collector 或 Tempo 问题。
@@ -193,14 +192,6 @@ curl -I http://127.0.0.1:9000/health/ready
 # Jaeger:  http://localhost:16686
 ```
 
-对于 RustFS 结构化日志，例如节点间 RPC 鉴权失败，Loki 日志行现在会包含 `event`、`component`、`subsystem`、`failure_reason`、`rpc_service`、`rpc_method`、`expected_audience` 等字段。常用 LogQL 检查：
-
-```logql
-{service_name="RustFS"} |= "RPC signature verification failed"
-{service_name="RustFS"} |= "failure_reason="
-{service_name="RustFS"} | failure_reason != ""
-```
-
 如果日志和指标已经正常，但 trace 仍然稀疏，最常见的原因通常是
 “还没有真实请求流量”或“`info` 级别过滤了嵌套 span”，而不是 OTLP 路由失败。
 

.docker/observability/otel-collector-config.yaml

@@ -29,27 +29,11 @@ processors:
     limit_mib: 1024
     spike_limit_mib: 256
   transform/logs:
-    error_mode: ignore
     log_statements:
       - context: log
         statements:
-          - set(attributes["message"], body.string) where IsString(body)
-          - set(attributes["log.body"], body.string) where IsString(body)
-          - set(body, Concat([body, " event=", attributes["event"]], "")) where IsString(body) and attributes["event"] != nil
-          - set(body, Concat([body, " component=", attributes["component"]], "")) where IsString(body) and attributes["component"] != nil
-          - set(body, Concat([body, " subsystem=", attributes["subsystem"]], "")) where IsString(body) and attributes["subsystem"] != nil
-          - set(body, Concat([body, " state=", attributes["state"]], "")) where IsString(body) and attributes["state"] != nil
-          - set(body, Concat([body, " result=", attributes["result"]], "")) where IsString(body) and attributes["result"] != nil
-          - set(body, Concat([body, " reason=", attributes["reason"]], "")) where IsString(body) and attributes["reason"] != nil
-          - set(body, Concat([body, " failure_reason=", attributes["failure_reason"]], "")) where IsString(body) and attributes["failure_reason"] != nil
-          - set(body, Concat([body, " rpc_path=", attributes["rpc_path"]], "")) where IsString(body) and attributes["rpc_path"] != nil
-          - set(body, Concat([body, " rpc_service=", attributes["rpc_service"]], "")) where IsString(body) and attributes["rpc_service"] != nil
-          - set(body, Concat([body, " rpc_method=", attributes["rpc_method"]], "")) where IsString(body) and attributes["rpc_method"] != nil
-          - set(body, Concat([body, " expected_audience=", attributes["expected_audience"]], "")) where IsString(body) and attributes["expected_audience"] != nil
-          - set(body, Concat([body, " peer_addr=", attributes["peer_addr"]], "")) where IsString(body) and attributes["peer_addr"] != nil
-          - set(body, Concat([body, " replay_scope_bootstrap_allowed=", attributes["replay_scope_bootstrap_allowed"]], "")) where IsString(body) and attributes["replay_scope_bootstrap_allowed"] != nil
-          - set(body, Concat([body, " error=", attributes["error"]], "")) where IsString(body) and attributes["error"] != nil
-          - set(body, Concat([body, " exception_message=", attributes["exception.message"]], "")) where IsString(body) and attributes["exception.message"] != nil
+          - set(attributes["message"], body.string)
+          - set(attributes["log.body"], body.string)
 
 exporters:
   otlp/tempo:

.docker/observability/prometheus-rules/rustfs-kms-alerts.yml

@@ -17,11 +17,9 @@
 # =============================================================================
 #
 # Metric source: the KMS operation-policy choke point in
-# crates/kms/src/policy.rs, except KmsKeyRotationOverdue, which reads the
-# label-less key-lifecycle gauge published by the deletion worker's sweep
-# (crates/kms/src/deletion_worker.rs). All label values are bounded static
-# strings (operation, op_class, outcome, error_class, backend, scope); key
-# identifiers, key material, and tokens never appear in labels.
+# crates/kms/src/policy.rs. All label values are bounded static strings
+# (operation, op_class, outcome, error_class, backend, scope); key identifiers,
+# key material, and tokens never appear in labels.
 #
 # Response procedures: docs/operations/kms-observability-runbook.md
 #
@@ -214,38 +212,3 @@ groups:
             circuit_open until the half-open probe succeeds or returns
             a non-retryable failure.
           runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendcircuitopen"
-
-      # ------------------------------------------------------------------
-      # 7. KmsKeyRotationOverdue
-      #    The least recently rotated usable key has gone more than 400
-      #    days without a rotation (measured from creation for keys with
-      #    no recorded rotation). Direct gauge state published by the
-      #    deletion worker's sweep, so no traffic guard applies; the
-      #    one-hour hold only bridges scrape gaps. The worker runs only
-      #    on backends with the schedule_deletion capability, so on the
-      #    Static backend the series never exists and this alert cannot
-      #    fire — that backend cannot rotate either; see the rotation
-      #    driver matrix in docs/operations/kms-backend-security.md.
-      #    Threshold: 400 days — conservative default sitting above a
-      #    one-year rotation policy. Align it with the rotation period
-      #    your compliance policy requires, and with
-      #    RUSTFS_KMS_ROTATION_MAX_AGE_SECS so the per-key rotation_due
-      #    verdict and this aggregate alert agree.
-      # ------------------------------------------------------------------
-      - alert: KmsKeyRotationOverdue
-        expr: |
-          rustfs_kms_oldest_key_rotation_age_seconds > (400 * 86400)
-        for: 1h
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "Oldest KMS key unrotated for more than 400 days"
-          description: >-
-            The least recently rotated usable KMS key was last rotated
-            {{ $value | humanizeDuration }} ago (measured from creation
-            for keys with no recorded rotation). List keys through the
-            admin API and read rotation_due / rotation_due_reason for
-            the per-key verdict; an "unsupported" reason means the
-            backend cannot rotate at all.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmskeyrotationoverdue"

.github/actions/setup/action.yml

@@ -57,7 +57,7 @@ runs:
   using: "composite"
   steps:
     # protobuf-compiler is deliberately absent: the setup-protoc step below
-    # installs 35.1 into the tool cache and prepends it to PATH, so the apt
+    # installs 34.1 into the tool cache and prepends it to PATH, so the apt
     # build (older, and never version-matched) was shadowed on every run and
     # simply never used.
     - name: Install system dependencies (Ubuntu)
@@ -81,11 +81,11 @@ runs:
     - name: Install protoc
       uses: rustfs/setup-protoc@a3705324d8f9bf5b6c3573fb6cf8ae421db55dd6 # v3.0.1
       with:
-        version: "35.1"
+        version: "34.1"
         repo-token: ${{ github.token }}
 
     - name: Install flatc
-      uses: Nugine/setup-flatc@698800de72a96bfb22cf60431dc21a2ff9a7e07b # v1
+      uses: Nugine/setup-flatc@e7855e994773ce90094a3f1626d4afc9080c23ae # v1
       with:
         version: "25.12.19"
 

.github/s3tests/README.md

@@ -46,9 +46,10 @@ lists when upstream changes.
   the PR.
 - **Weekly + manual**: `.github/workflows/e2e-s3tests.yml` runs the full
   upstream suite (`TEST_SCOPE=all`) against a Docker deployment (single node
-  or a 4-node distributed cluster behind HAProxy). The canonical gate policy
-  and compatibility-report behavior are documented in
-  [`scripts/s3-tests/README.md`](../../scripts/s3-tests/README.md).
+  or a 4-node distributed cluster behind HAProxy). It fails only on
+  regressions in the implemented whitelist and publishes a classification
+  report (`compat-report.md`, also shown in the job summary) listing promotion
+  candidates and unclassified tests.
 
 ## Running Tests Locally
 

.github/workflows/architecture-migration-rules.yml

@@ -53,9 +53,9 @@ jobs:
           persist-credentials: false
 
       - name: Install ripgrep
-        uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
-        with:
-          tool: ripgrep@15.2.0
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y ripgrep
 
       - name: Check architecture migration rules
         run: ./scripts/check_architecture_migration_rules.sh

.github/workflows/cache-warm.yml

@@ -182,12 +182,7 @@ jobs:
             echo '```'
           } >> "$GITHUB_STEP_SUMMARY"
 
-  # Readers: test-and-lint-rio-v2 (per-PR), build-rustfs-debug-binary-rio-v2
-  # (weekly schedule / manual dispatch only — dormant rio-v2 variant, see
-  # rustfs/backlog#1835 and docs/architecture/minio-file-format-compat.md).
-  # The second build below stays despite the reduced cadence: it warms the
-  # rio-v2,e2e-test-hooks feature resolution the scheduled build restores,
-  # which keeps that lane inside its 30-minute timeout.
+  # Readers: test-and-lint-rio-v2, build-rustfs-debug-binary-rio-v2.
   warm-ci-feat-rio:
     name: Warm ci-feat-rio
     runs-on: sm-standard-4

.github/workflows/ci-docs-only.yml

@@ -83,9 +83,7 @@ jobs:
           persist-credentials: false
 
       - name: Install ripgrep
-        uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
-        with:
-          tool: ripgrep@15.2.0
+        run: sudo apt-get update && sudo apt-get install -y ripgrep
 
       - name: Install Rust toolchain
         uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
@@ -104,9 +102,6 @@ jobs:
       - name: Check architecture migration rules
         run: ./scripts/check_architecture_migration_rules.sh
 
-      - name: Check logging guardrails
-        run: ./scripts/check_logging_guardrails.sh
-
       - name: Check tokio io-uring feature guard
         run: ./scripts/check_no_tokio_io_uring.sh
 
@@ -116,18 +111,6 @@ jobs:
       - name: Check body-cache whitelist guard
         run: ./scripts/check_body_cache_whitelist.sh
 
-      - name: Check s3s footprint ratchet
-        run: ./scripts/check_s3s_footprint.sh
-
-      - name: Check cryptographic capability wording
-        run: ./scripts/check_fips_wording.sh
-
-      - name: Check no embedded secret material
-        run: ./scripts/check_embedded_secrets.sh
-
-      - name: Check test wiring
-        run: python3 ./scripts/check_test_wiring.py
-
       - name: Check no planning docs committed
         run: ./scripts/check_no_planning_docs.sh
 

.github/workflows/ci.yml

@@ -118,9 +118,7 @@ jobs:
           persist-credentials: false
 
       - name: Install ripgrep
-        uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
-        with:
-          tool: ripgrep@15.2.0
+        run: sudo apt-get update && sudo apt-get install -y ripgrep
 
       - name: Install Rust toolchain
         uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
@@ -139,9 +137,6 @@ jobs:
       - name: Check architecture migration rules
         run: ./scripts/check_architecture_migration_rules.sh
 
-      - name: Check logging guardrails
-        run: ./scripts/check_logging_guardrails.sh
-
       - name: Check tokio io-uring feature guard
         run: ./scripts/check_no_tokio_io_uring.sh
 
@@ -151,18 +146,6 @@ jobs:
       - name: Check body-cache whitelist guard
         run: ./scripts/check_body_cache_whitelist.sh
 
-      - name: Check s3s footprint ratchet
-        run: ./scripts/check_s3s_footprint.sh
-
-      - name: Check cryptographic capability wording
-        run: ./scripts/check_fips_wording.sh
-
-      - name: Check no embedded secret material
-        run: ./scripts/check_embedded_secrets.sh
-
-      - name: Check test wiring
-        run: python3 ./scripts/check_test_wiring.py
-
       - name: Check no planning docs committed
         run: ./scripts/check_no_planning_docs.sh
 
@@ -400,7 +383,7 @@ jobs:
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
     needs: [ quick-checks ]
     runs-on: sm-standard-4
-    timeout-minutes: 90
+    timeout-minutes: 45
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
@@ -440,7 +423,7 @@ jobs:
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
     needs: [ quick-checks ]
     runs-on: sm-standard-4
-    timeout-minutes: 90
+    timeout-minutes: 60
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
@@ -470,7 +453,7 @@ jobs:
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
     needs: [ quick-checks ]
     runs-on: sm-standard-4
-    timeout-minutes: 90
+    timeout-minutes: 60
     strategy:
       # On a PR, one failing protocol leg is enough to know the PR is not ready,
       # so stop the sibling leg instead of paying another ~40 minutes for it.
@@ -544,12 +527,7 @@ jobs:
 
   build-rustfs-debug-binary-rio-v2:
     name: Build RustFS Debug Binary (rio-v2)
-    # Dormant rio-v2 variant (rustfs/backlog#1835): the feature ships in no
-    # default build, so this full-suite lane runs only on the weekly schedule
-    # and manual dispatch. Per-PR cfg-seam coverage stays with
-    # test-and-lint-rio-v2. Lifecycle and the promote-or-delete condition:
-    # docs/architecture/minio-file-format-compat.md ("rio-v2 variant lifecycle").
-    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
+    if: github.event_name != 'pull_request' || github.event.action != 'closed'
     needs: [ quick-checks ]
     runs-on: sm-standard-4
     timeout-minutes: 30
@@ -689,9 +667,9 @@ jobs:
       - name: Make binary executable
         run: chmod +x ./target/debug/rustfs
 
-      # Build the e2e test graph once. The archive is reused by the smoke
-      # selection guard, security exact-count check, and run below, avoiding a
-      # second compile of the same e2e_test target on cold runners (backlog#1645).
+      # Build the e2e test graph once. The archive is reused by the security
+      # count-floor check and the smoke run below, avoiding a second compile of
+      # the same e2e_test target on cold runners (backlog#1645).
       - name: Archive e2e smoke test binaries
         env:
           NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
@@ -699,7 +677,6 @@ jobs:
         run: |
           cargo nextest archive --profile e2e-smoke -p e2e_test --archive-file "${NEXTEST_ARCHIVE}"
           cargo nextest list --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" --message-format json > "${NEXTEST_LISTING}"
-          python3 ./scripts/check_test_wiring.py --check-profile e2e-smoke "${NEXTEST_LISTING}"
           ./scripts/check_security_smoke_count.sh check "${NEXTEST_LISTING}"
 
       # PR smoke subset of the in-repo e2e suite (backlog#1149 ci-4). The
@@ -764,7 +741,7 @@ jobs:
     # suites — KMS, object_lock, multipart_auth, quota, checksum, encryption,
     # security-boundary, ... — via the e2e-full nextest profile. Too heavy for
     # every PR, so it is gated to main pushes, the merge queue, and manual
-    # dispatch. protocols / the 7 cluster suites / replication / #[ignore] are
+    # dispatch. protocols / the 6 cluster suites / replication / #[ignore] are
     # owned by other lanes (see .config/nextest.toml profile.e2e-full).
     if: >-
       github.event_name == 'workflow_dispatch' ||
@@ -787,32 +764,6 @@ jobs:
           cache-save-if: 'false'
           install-build-packaging-tools: 'false'
 
-      - name: Set up Python
-        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
-        with:
-          python-version: "3.12"
-
-      - name: Install awscurl
-        run: |
-          python3 -m pip install --user --upgrade pip "awscurl==0.44"
-          echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
-
-      - name: Verify awscurl
-        run: test -x "$AWSCURL_PATH"
-
-      - name: Install Vault
-        run: |
-          VAULT_VERSION="1.17.6"
-          VAULT_ARCHIVE="vault_${VAULT_VERSION}_linux_amd64.zip"
-          curl -fsSLo "$RUNNER_TEMP/$VAULT_ARCHIVE" "https://releases.hashicorp.com/vault/${VAULT_VERSION}/${VAULT_ARCHIVE}"
-          echo "0cddc1fbbb88583b5ba5b845f9f8fae47c6fb39a6d48cd543c6ba6fd3ac1a669  $RUNNER_TEMP/$VAULT_ARCHIVE" | sha256sum --check --status
-          unzip -q "$RUNNER_TEMP/$VAULT_ARCHIVE" -d "$RUNNER_TEMP/vault-bin"
-          echo "RUSTFS_TEST_VAULT_BIN=$RUNNER_TEMP/vault-bin/vault" >> "$GITHUB_ENV"
-
-      - name: Verify Vault
-        run: |
-          "$RUSTFS_TEST_VAULT_BIN" version
-
       # Download after the cache restore so the freshly built binary from the
       # build job always wins over anything restored into target/debug.
       - name: Download debug binary
@@ -824,13 +775,6 @@ jobs:
       - name: Make binary executable
         run: chmod +x ./target/debug/rustfs
 
-      - name: Verify e2e full membership
-        env:
-          NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-full-list.json
-        run: |
-          cargo nextest list --profile e2e-full -p e2e_test --message-format json > "${NEXTEST_LISTING}"
-          python3 ./scripts/check_test_wiring.py --check-profile e2e-full "${NEXTEST_LISTING}"
-
       # Full single-node e2e lane (backlog#1149 ci-5). The e2e-full
       # default-filter in .config/nextest.toml is the single wiring mechanism —
       # extend that filter, never add ad-hoc e2e jobs here. Reuses the downloaded
@@ -843,16 +787,11 @@ jobs:
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
         with:
           name: e2e-full-junit-${{ github.run_number }}
-          path: |
-            target/nextest/e2e-full/junit.xml
-            ${{ runner.temp }}/rustfs-e2e-full-list.json
+          path: target/nextest/e2e-full/junit.xml
           retention-days: 7
 
   e2e-tests-rio-v2:
     name: End-to-End Tests (rio-v2)
-    # Inherits the schedule/dispatch-only gate through needs: on every other
-    # event build-rustfs-debug-binary-rio-v2 is skipped, so this job skips
-    # with it (see the dormant-variant comment on that job).
     needs: [ build-rustfs-debug-binary-rio-v2 ]
     runs-on: sm-standard-2
     timeout-minutes: 30

.github/workflows/docker.yml

@@ -94,7 +94,6 @@ jobs:
       short_sha: ${{ steps.check.outputs.short_sha }}
       is_prerelease: ${{ steps.check.outputs.is_prerelease }}
       create_latest: ${{ steps.check.outputs.create_latest }}
-      source_ref: ${{ steps.check.outputs.source_ref }}
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
@@ -119,7 +118,6 @@ jobs:
           short_sha=""
           is_prerelease=false
           create_latest=false
-          source_ref="$GITHUB_SHA"
 
           if [[ "${{ github.event_name }}" == "workflow_run" ]]; then
             # Triggered by build workflow completion
@@ -139,7 +137,6 @@ jobs:
             # Extract version info from commit message or use commit SHA
             # Use Git to generate consistent short SHA (ensures uniqueness like build.yml)
             short_sha=$(git rev-parse --short "$HEAD_SHA")
-            source_ref="$HEAD_SHA"
 
             # Determine build type based on triggering workflow event and ref
             triggering_event="$TRIGGERING_EVENT"
@@ -264,23 +261,6 @@ jobs:
                 echo "⚠️  Only release versions (latest, v1.0.0, 1.0.0) and prereleases (v1.0.0-alpha1, 1.0.0-beta2) are supported"
                 ;;
             esac
-
-            if [[ "$should_build" == true && "$input_version" != "latest" ]]; then
-              tag_ref="refs/tags/$input_version"
-              if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
-                if [[ "$input_version" == v* ]]; then
-                  tag_ref="refs/tags/${input_version#v}"
-                else
-                  tag_ref="refs/tags/v$input_version"
-                fi
-              fi
-
-              if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
-                echo "❌ Release tag not found for Docker build: $input_version"
-                exit 1
-              fi
-              source_ref="$tag_ref"
-            fi
           fi
 
           {
@@ -291,7 +271,6 @@ jobs:
             echo "short_sha=$short_sha"
             echo "is_prerelease=$is_prerelease"
             echo "create_latest=$create_latest"
-            echo "source_ref=$source_ref"
           } >> "$GITHUB_OUTPUT"
 
           echo "🐳 Docker Build Summary:"
@@ -302,7 +281,6 @@ jobs:
           echo "  - Short SHA: $short_sha"
           echo "  - Is prerelease: $is_prerelease"
           echo "  - Create latest: $create_latest"
-          echo "  - Source ref: $source_ref"
 
   # Build multi-arch Docker images
   # Strategy: Build images using pre-built binaries from dl.rustfs.com
@@ -330,7 +308,6 @@ jobs:
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
         with:
           persist-credentials: false
-          ref: ${{ needs.build-check.outputs.source_ref }}
 
       - name: Login to Docker Hub
         uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
@@ -420,8 +397,7 @@ jobs:
           LABELS="org.opencontainers.image.title=RustFS"
           LABELS="$LABELS,org.opencontainers.image.description=RustFS distributed object storage system"
           LABELS="$LABELS,org.opencontainers.image.version=$VERSION"
-          SOURCE_REVISION="$(git rev-parse HEAD)"
-          LABELS="$LABELS,org.opencontainers.image.revision=$SOURCE_REVISION"
+          LABELS="$LABELS,org.opencontainers.image.revision=${{ github.sha }}"
           LABELS="$LABELS,org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}"
           LABELS="$LABELS,org.opencontainers.image.created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
           LABELS="$LABELS,org.opencontainers.image.build-type=$BUILD_TYPE"

.github/workflows/e2e-replication-nightly.yml

@@ -12,28 +12,32 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# Consolidated nightly e2e lane for replication, cluster faults, and protocols.
+# Nightly full replication e2e lane (backlog#1147 repl-1, deps: ci-4).
 #
-# The per-PR gate (ci.yml `e2e-tests` job, `--profile e2e-smoke`) runs the
-# FAST replication tests. This scheduled lane runs the remaining heavier
-# replication e2e tests that are unfit for a per-PR gate: remote-target TLS
-# validation, bucket-replication data-plane/helper tests (PUT/delete + poll
-# for convergence, HTTPS targets, active SSE failure contracts, event/history
-# observers), and the `_real_dual_node` / `_real_three_node` /
-# `_real_single_node` site-replication tests that each spawn full rustfs
-# server processes.
+# The per-PR gate (ci.yml `e2e-tests` job, `--profile e2e-smoke`) runs the 20
+# FAST replication tests. This scheduled lane runs the remaining 27
+# heavier replication e2e tests that are unfit for a per-PR gate:
+#
+#   * 2 remote-target TLS validation tests.
+#   * 12 bucket-replication data-plane/helper tests (PUT/delete + poll for
+#     convergence; two replicate over HTTPS, two pin active SSE failure
+#     contracts, and one guards event/history observers). The SSE-S3 contract
+#     remains ignored under backlog#1291.
+#   * 11 `_real_dual_node` site-replication tests (each spawns TWO rustfs
+#     servers and drives the cross-process site-replication control plane).
+#   * 1 `_real_three_node` site-replication test.
+#   * 1 `_real_single_node` service-account round-trip test.
 #
 # The selection is the [profile.e2e-repl-nightly] default-filter in
 # .config/nextest.toml — the single wiring mechanism (repl-1 / ci-4). Do NOT
-# add ad-hoc cargo-test steps here; change the filterset instead. The
-# authoritative membership and count come from
-# `cargo nextest list -p e2e_test --profile e2e-repl-nightly`; the PR/nightly
-# selection digest is committed under .config/.
+# add ad-hoc cargo-test steps here; change the filterset instead.
 #
-# Explicit division of labor: these subsets run only here and never double-run
-# in the e2e-full merge gate.
+# Explicit division of labor: these 27 tests run ONLY here, never double-run
+# in ci-5's future e2e-full merge gate. TODO(ci-7): once the ci domain's
+# consolidated scheduled e2e workflow exists, fold this interim repl-owned lane
+# into it rather than growing a second scheduled entrypoint.
 
-name: e2e-nightly
+name: e2e-replication-nightly
 
 on:
   workflow_dispatch:
@@ -47,10 +51,6 @@ on:
 permissions:
   contents: read
 
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: false
-
 jobs:
   repl-nightly:
     name: Replication e2e (nightly)
@@ -98,20 +98,9 @@ jobs:
       # demand otherwise, but a single explicit build avoids several parallel
       # nextest test processes racing to build it at once.
       - name: Build rustfs binary
-        run: |
-          cargo build -p rustfs --bins
-          : > target/debug/rustfs.features
-
-      - name: Verify replication e2e membership
-        env:
-          NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-repl-nightly-list.json
-        run: |
-          cargo nextest list --profile e2e-repl-nightly -p e2e_test --message-format json > "${NEXTEST_LISTING}"
-          python3 ./scripts/check_test_wiring.py --check-profile e2e-repl-nightly "${NEXTEST_LISTING}"
+        run: cargo build -p rustfs --bins
 
       - name: Run replication e2e nightly suite
-        env:
-          RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-repl-nightly-logs
         run: cargo nextest run --profile e2e-repl-nightly -p e2e_test
 
       - name: Upload nextest junit report
@@ -119,112 +108,13 @@ jobs:
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
         with:
           name: e2e-replication-nightly-junit-${{ github.run_number }}
-          path: |
-            target/nextest/e2e-repl-nightly/junit.xml
-            ${{ runner.temp }}/rustfs-e2e-repl-nightly-list.json
-            ${{ runner.temp }}/rustfs-e2e-repl-nightly-logs/
+          path: target/nextest/e2e-repl-nightly/junit.xml
           retention-days: 7
           if-no-files-found: ignore
 
-  cluster-nightly:
-    name: Cluster fault e2e (nightly)
-    runs-on: sm-standard-4
-    timeout-minutes: 90
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-e2e-nightly
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
-
-      - name: Build rustfs binary
-        run: |
-          cargo build -p rustfs --bins --features e2e-test-hooks
-          : > target/debug/rustfs.features
-
-      - name: Verify cluster fault e2e membership
-        env:
-          NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-nightly-list.json
-        run: |
-          cargo nextest list --profile e2e-nightly -p e2e_test --message-format json > "${NEXTEST_LISTING}"
-          python3 ./scripts/check_test_wiring.py --check-profile e2e-nightly "${NEXTEST_LISTING}"
-
-      - name: Run cluster fault e2e nightly suite
-        env:
-          RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-nightly-logs
-        run: cargo nextest run --profile e2e-nightly -p e2e_test
-
-      - name: Upload cluster fault diagnostics
-        if: always()
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: e2e-cluster-nightly-${{ github.run_number }}
-          path: |
-            target/nextest/e2e-nightly/junit.xml
-            ${{ runner.temp }}/rustfs-e2e-nightly-list.json
-            ${{ runner.temp }}/rustfs-e2e-nightly-logs/
-          retention-days: 7
-          if-no-files-found: warn
-
-  protocols-nightly:
-    name: Protocol e2e (nightly)
-    runs-on: sm-standard-4
-    timeout-minutes: 90
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-      RUSTFS_BUILD_FEATURES: ftps,webdav,sftp
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-e2e-protocols
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
-
-      # The suite owns fixed protocol ports and serializes its internal cases.
-      - name: Verify protocol e2e membership
-        env:
-          NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-protocols-list.json
-        run: |
-          cargo nextest list --profile e2e-protocols -p e2e_test --message-format json > "${NEXTEST_LISTING}"
-          python3 ./scripts/check_test_wiring.py --check-profile e2e-protocols "${NEXTEST_LISTING}"
-
-      - name: Run protocol e2e nightly suite
-        env:
-          RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-protocol-e2e-logs
-        run: >-
-          cargo nextest run -j 1 --profile e2e-protocols -p e2e_test --no-capture
-
-      - name: Upload protocol diagnostics
-        if: always()
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: e2e-protocol-nightly-${{ github.run_number }}
-          path: |
-            target/nextest/e2e-protocols/junit.xml
-            ${{ runner.temp }}/rustfs-e2e-protocols-list.json
-            ${{ runner.temp }}/rustfs-protocol-e2e-logs/
-          retention-days: 7
-          if-no-files-found: warn
-
   alert-on-failure:
     name: Alert on scheduled failure
-    needs: [repl-nightly, cluster-nightly, protocols-nightly]
+    needs: [repl-nightly]
     # Only scheduled runs open/append the tracking issue (backlog#1149 ci-8);
     # manual workflow_dispatch runs stay quiet so a debugging run never files a
     # spurious alert.

.github/workflows/e2e-s3tests.yml

@@ -18,9 +18,10 @@
 # runs only the implemented_tests.txt whitelist. This workflow complements it:
 #
 #   - Scheduled weekly full sweep (TEST_SCOPE=all): runs the ENTIRE upstream
-#     suite and reports promotion candidates. Regressions, unclassified tests,
-#     incomplete execution, and infrastructure errors fail the job; classified
-#     failures for not-yet-implemented features remain informational.
+#     suite and reports promotion candidates (tests that newly pass) and
+#     unclassified tests. The job fails only on regressions in the implemented
+#     whitelist or on infrastructure errors — expected failures from
+#     not-yet-implemented features do not turn the run red.
 #   - Manual runs (workflow_dispatch): same, with configurable mode/scope.
 #
 # All test execution is delegated to scripts/s3-tests/run.sh (single source of
@@ -44,6 +45,13 @@
 # The PR gate (ci.yml s3-implemented-tests) is unaffected: it avoids Docker
 # via DEPLOY_MODE=binary and defers all pip setup to run.sh's self-bootstrap.
 
+# DISABLED. This workflow is switched off in the repository's Actions settings
+# (state: disabled_manually) and does not run on any trigger, including its cron
+# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
+# reading this file, which has already misled at least one audit — hence this
+# banner. Re-enabling is a UI action; anyone doing so should first check that the
+# workflow still matches the current CI layout. See rustfs/backlog#1603.
+#
 name: e2e-s3tests
 
 on:
@@ -73,27 +81,10 @@ on:
         description: "Stop after N failures. '0' to run everything."
         required: false
         default: "0"
-      shard-count:
-        description: "Exact-node-ID shard count for a targeted manual run"
-        required: false
-        default: "1"
-        type: choice
-        options:
-          - "1"
-          - "2"
-          - "4"
-      shard-index:
-        description: "Zero-based shard index for a targeted manual run"
-        required: false
-        default: "0"
       markexpr:
         description: "Optional pytest -m expression"
         required: false
         default: ""
-      testexpr:
-        description: "Optional pytest -k expression"
-        required: false
-        default: ""
   schedule:
     # Weekly full sweep (Sunday 02:00 UTC): full suite, run against BOTH the
     # single-node and the 4-node distributed topologies (matrix below).
@@ -120,9 +111,6 @@ env:
   XDIST: ${{ github.event.inputs.xdist || '4' }}
   MAXFAIL: ${{ github.event.inputs.maxfail || '0' }}
   MARKEXPR: ${{ github.event.inputs.markexpr || '' }}
-  TESTEXPR: ${{ github.event.inputs.testexpr || '' }}
-  S3_SHARD_COUNT: ${{ github.event_name == 'schedule' && '4' || github.event.inputs.shard-count || '1' }}
-  TEST_TIMEOUT: "300"
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event.inputs['test-mode'] || 'single' }}
@@ -139,22 +127,19 @@ defaults:
 
 jobs:
   s3tests:
-    name: s3tests (${{ matrix.test-mode }}, shard ${{ matrix.shard-index }})
     # GitHub-hosted: reliably provides Docker + docker compose + python3/pip.
     # See the header note (ci-1) for why the self-hosted sm-standard-4 label
-    # was abandoned. Scheduled failures are handled by alert-on-failure below.
+    # was abandoned. TODO(ci-8): scheduled-failure alerting (auto-open issue)
+    # is added by the ci-8 composite action; do not implement it here.
     runs-on: ubuntu-latest
     timeout-minutes: 180
     strategy:
       fail-fast: false
-      max-parallel: 2
       matrix:
         # Scheduled sweeps cover both topologies; manual runs use the input.
         test-mode: ${{ github.event_name == 'schedule' && fromJSON('["single", "multi"]') || fromJSON(format('["{0}"]', github.event.inputs.test-mode || 'single')) }}
-        shard-index: ${{ github.event_name == 'schedule' && fromJSON('[0, 1, 2, 3]') || fromJSON(format('[{0}]', github.event.inputs.shard-index || '0')) }}
     env:
       TEST_MODE: ${{ matrix.test-mode }}
-      S3_SHARD_INDEX: ${{ matrix.shard-index }}
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
         with:
@@ -196,7 +181,6 @@ jobs:
       - name: Start single RustFS
         if: env.TEST_MODE == 'single'
         run: |
-          SSE_KEY="$(head -c 32 /dev/zero | base64 -w0)"
           docker network inspect rustfs-net >/dev/null 2>&1 || docker network create rustfs-net
           docker rm -f rustfs-single >/dev/null 2>&1 || true
           # The four disks share one physical device on the runner (a single
@@ -209,7 +193,6 @@ jobs:
             -e RUSTFS_ADDRESS=0.0.0.0:9000 \
             -e RUSTFS_ACCESS_KEY="${S3_ACCESS_KEY}" \
             -e RUSTFS_SECRET_KEY="${S3_SECRET_KEY}" \
-            -e RUSTFS_SSE_S3_MASTER_KEY="${SSE_KEY}" \
             -e RUSTFS_VOLUMES="/data/rustfs{0...3}" \
             -e RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true \
             -v /tmp/rustfs-single:/data \
@@ -218,7 +201,6 @@ jobs:
       - name: Start 4-node distributed cluster
         if: env.TEST_MODE == 'multi'
         run: |
-          SSE_KEY="$(head -c 32 /dev/zero | base64 -w0)"
           # A real distributed deployment: every node lists all endpoints in
           # RUSTFS_VOLUMES so data is erasure-coded ACROSS nodes. Do not use
           # node-local volume paths here — that would create four independent
@@ -231,7 +213,6 @@ jobs:
               RUSTFS_ADDRESS: "0.0.0.0:9000"
               RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
               RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
-              RUSTFS_SSE_S3_MASTER_KEY: "${SSE_KEY}"
               RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
               # Each node's four disks share one physical device inside its
               # container, so bypass the local physical-disk-independence guard
@@ -274,13 +255,8 @@ jobs:
           EOF
 
           cat > haproxy.cfg <<'EOF'
-          global
-            log stdout format raw local0 info
-
           defaults
             mode http
-            log global
-            log-format '%ci:%cp [%tr] %ft %b/%s %TR/%Tw/%Tc/%Tr/%Ta %ST %B %tsc %HM %HP'
             timeout connect 5s
             timeout client  30s
             timeout server  30s
@@ -318,14 +294,34 @@ jobs:
 
       - name: Run ceph s3-tests
         run: |
+          set +e
           DEPLOY_MODE=existing \
           TEST_MODE="${TEST_MODE}" \
           TEST_SCOPE="${TEST_SCOPE}" \
           XDIST="${XDIST}" \
           MAXFAIL="${MAXFAIL}" \
           MARKEXPR="${MARKEXPR}" \
-          TESTEXPR="${TESTEXPR}" \
           ./scripts/s3-tests/run.sh
+          RC=$?
+          set -e
+
+          if [ "${TEST_SCOPE}" = "implemented" ]; then
+            # Whitelist run: every failure is a regression.
+            exit "${RC}"
+          fi
+
+          # Full sweep: failures outside the implemented whitelist are
+          # inventory (promotion candidates / unimplemented features), not a
+          # gate. Fail only on whitelist regressions or infrastructure errors.
+          JUNIT="artifacts/s3tests-${TEST_MODE}/junit.xml"
+          if [ ! -f "${JUNIT}" ]; then
+            echo "No junit.xml produced — infrastructure failure (exit ${RC})" >&2
+            exit "${RC}"
+          fi
+          python3 scripts/s3-tests/report_compat.py \
+            --junit "${JUNIT}" \
+            --lists-dir scripts/s3-tests \
+            --fail-on-regression
 
       - name: Publish compatibility report
         if: always()
@@ -350,7 +346,7 @@ jobs:
         if: always() && env.ACT != 'true'
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
         with:
-          name: s3tests-${{ env.TEST_MODE }}-shard-${{ matrix.shard-index }}
+          name: s3tests-${{ env.TEST_MODE }}
           path: artifacts/**
 
   alert-on-failure:

.github/workflows/fuzz.yml

@@ -12,22 +12,27 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
+# DISABLED. This workflow is switched off in the repository's Actions settings
+# (state: disabled_manually) and does not run on any trigger, including its cron
+# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
+# reading this file, which has already misled at least one audit — hence this
+# banner. Re-enabling is a UI action; anyone doing so should first check that the
+# workflow still matches the current CI layout. See rustfs/backlog#1603.
+#
 name: Fuzz
 
 on:
   pull_request:
     types: [ opened, synchronize, reopened, closed ]
-    # Run when the harness or any directly fuzzed production crate changes.
+    # PR trigger is intentionally narrow: only changes to the fuzz harness
+    # itself gate a PR. Broad crate paths (ecstore/filemeta/utils/policy/…)
+    # are covered by the nightly `schedule` run below, which fuzzes against
+    # whatever landed on main. Widening these paths previously queued a
+    # ~45min fuzz-build on nearly every PR and is why this workflow was
+    # disabled; do not re-add crate paths here.
     paths:
       - "fuzz/**"
       - "scripts/fuzz/**"
-      - "crates/ecstore/**"
-      - "crates/filemeta/**"
-      - "crates/policy/**"
-      - "crates/security-governance/**"
-      - "crates/utils/**"
-      - "Cargo.toml"
-      - "Cargo.lock"
       - ".github/workflows/fuzz.yml"
   schedule:
     - cron: "0 2 * * *"
@@ -76,7 +81,7 @@ jobs:
       github.event_name == 'schedule' ||
       github.event_name == 'workflow_dispatch'
     runs-on: sm-standard-4
-    timeout-minutes: 60
+    timeout-minutes: 45
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
@@ -116,7 +121,12 @@ jobs:
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
         with:
           name: fuzz-prebuilt-binaries-${{ github.run_number }}
-          path: fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/
+          path: |
+            fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/archive_extract
+            fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/bucket_validation
+            fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/local_metadata
+            fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/path_containment
+            fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/policy_ingress
           if-no-files-found: error
           retention-days: 1
           compression-level: 0
@@ -182,7 +192,10 @@ jobs:
   nightly-fuzz-corpus:
     name: "Nightly / ${{ matrix.target }}"
     needs: fuzz-build
-    # Scheduled failures are handled by alert-on-failure below.
+    # TODO(ci-8): when the schedule-failure-issue composite action lands,
+    # add a step here (or a dependent job) that opens/updates a GitHub issue
+    # on nightly failure. ci-8 is the single alerting mechanism for all
+    # scheduled workflows; do not self-roll alerting in this workflow.
     if: >
       github.event_name == 'schedule' ||
       (github.event_name == 'workflow_dispatch' &&

.github/workflows/nightly-gnu.yml

@@ -55,142 +55,3 @@ jobs:
 
       - name: Build RustFS
         run: cargo build --release --locked --target x86_64-unknown-linux-gnu -p rustfs --bins
-
-  # Live-Vault lane for the rustfs-kms suite (rustfs/backlog#1774).
-  #
-  # RUSTFS_KMS_VAULT_TOKEN is the single switch that adds the Vault KV2 and
-  # Vault Transit backends to every for_each_backend spec in
-  # crates/kms/tests/behavior_*.rs (see crates/kms/AGENTS.md). rotate and
-  # versioning are advertised only by the Vault backends, so without this lane
-  # no CI run ever asserts the working half of behavior_rotation.rs — a
-  # rotation that silently dropped historical key versions would stay green.
-  # The same lane runs the dev-Vault #[ignore] tests and the two self-hosting
-  # live scripts (AppRole login, three-node Raft leader failover).
-  #
-  # GitHub-hosted ubuntu-latest, deliberately not the self-hosted sm-standard
-  # fleet: the HA failover script needs a working Docker daemon, and the
-  # self-hosted fleet is heterogeneous — a docker-dependent workflow has been
-  # burned by it before (see the banner in e2e-s3tests.yml, rustfs/backlog#1149).
-  kms-vault-lane:
-    name: KMS live Vault lane
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-      # Root token of the ephemeral loopback dev server. Not a secret: the
-      # server lives only for this job, listens on 127.0.0.1, and holds only
-      # keys the tests create. The literal value matters — the dev-Vault
-      # #[ignore] fixtures in crates/kms/src/backends/vault.rs hardcode it.
-      VAULT_LANE_TOKEN: dev-only-token
-      VAULT_LANE_ADDR: http://127.0.0.1:8200
-      # Keeps a runner-level proxy from swallowing the loopback dev-server
-      # traffic (see crates/kms/AGENTS.md). Actions env keys are
-      # case-insensitive, so only the uppercase form is set; reqwest reads
-      # either casing.
-      NO_PROXY: 127.0.0.1,localhost
-    steps:
-      - name: Checkout main branch
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-          ref: main
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          # Dedicated key: rust-cache cannot tell runner images apart, so
-          # sharing a key with an sm-standard lane would let two different
-          # system images overwrite each other's artifacts (same reasoning as
-          # ci.yml's ci-uring lane). Saved from this nightly job itself so the
-          # next night starts warm.
-          cache-shared-key: kms-vault-lane
-          cache-save-if: 'true'
-          install-build-packaging-tools: 'false'
-          install-test-tools: 'false'
-
-      - name: Install Vault CLI
-        run: |
-          set -euo pipefail
-          wget -qO- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
-          echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list >/dev/null
-          sudo apt-get update -qq
-          sudo apt-get install -y -qq vault
-          vault version
-
-      - name: Start Vault dev server with KV2 and Transit engines
-        run: |
-          set -euo pipefail
-          nohup vault server -dev \
-            -dev-root-token-id="${VAULT_LANE_TOKEN}" \
-            -dev-listen-address=127.0.0.1:8200 >/tmp/vault-dev.log 2>&1 &
-          for _ in $(seq 1 60); do
-            if curl -fsS "${VAULT_LANE_ADDR}/v1/sys/health" >/dev/null 2>&1; then
-              break
-            fi
-            sleep 1
-          done
-          curl -fsS "${VAULT_LANE_ADDR}/v1/sys/health"
-          export VAULT_ADDR="${VAULT_LANE_ADDR}" VAULT_TOKEN="${VAULT_LANE_TOKEN}"
-          # Dev mode mounts KV v2 at secret/ by default; Transit is explicit.
-          # Prove both engines actually work rather than assuming the defaults.
-          vault secrets enable transit
-          vault kv put secret/rustfs-ci-lane-probe value=ok >/dev/null
-          vault kv get secret/rustfs-ci-lane-probe >/dev/null
-          vault write -f transit/keys/rustfs-ci-lane-probe >/dev/null
-
-      - name: Run rustfs-kms suite with the Vault lane on
-        env:
-          RUSTFS_KMS_VAULT_TOKEN: ${{ env.VAULT_LANE_TOKEN }}
-          RUSTFS_KMS_VAULT_ADDR: ${{ env.VAULT_LANE_ADDR }}
-        run: cargo test -p rustfs-kms --locked
-
-      - name: Run dev-Vault ignored tests
-        env:
-          RUSTFS_KMS_VAULT_TOKEN: ${{ env.VAULT_LANE_TOKEN }}
-          RUSTFS_KMS_VAULT_ADDR: ${{ env.VAULT_LANE_ADDR }}
-        # Filters select the dev-Vault-only #[ignore] tests. The AWS #[ignore]
-        # tests (backends::aws, service_manager) stay excluded — they need real
-        # AWS credentials and create billable keys. The AppRole and HA #[ignore]
-        # tests are excluded here because their own scripts below provision the
-        # Vault topology they need.
-        run: |
-          set -euo pipefail
-          cargo test -p rustfs-kms --locked --lib backends::contract_tests -- --ignored
-          cargo test -p rustfs-kms --locked --lib backends::vault -- --ignored
-          cargo test -p rustfs-kms --locked --test vault_fault_injection -- --ignored
-
-      - name: Run AppRole live checks (self-hosting ephemeral Vault)
-        run: bash scripts/test/vault_approle_kms_live.sh
-
-      - name: Show Vault dev server log on failure
-        if: failure()
-        run: tail -n 200 /tmp/vault-dev.log || true
-
-  # Three-node Raft leader failover (crates/kms/tests/vault_ha_failover_live.rs,
-  # first validated by rustfs/rustfs#5653). Its own job so an election-timing
-  # flake cannot mask the main lane's verdict, and vice versa. The script
-  # provisions and tears down its own Docker cluster.
-  kms-vault-ha-failover:
-    name: KMS Vault HA failover lane
-    runs-on: ubuntu-latest
-    timeout-minutes: 60
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-      NO_PROXY: 127.0.0.1,localhost
-    steps:
-      - name: Checkout main branch
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-          ref: main
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          cache-shared-key: kms-vault-lane
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
-          install-test-tools: 'false'
-
-      - name: Run HA leader failover live checks (three-node Raft cluster in Docker)
-        run: bash scripts/test/vault_ha_kms_live.sh

.github/workflows/package.yml

@@ -1,562 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Package Workflow - Build DEB/RPM packages
-#
-# This workflow builds DEB and RPM packages from pre-built Linux binaries
-# and uploads them to Cloudflare R2 and the GitHub release.
-#
-# Trigger:
-# - workflow_run: automatically package after "Build and Release" completes
-#   for a release tag (the mac/windows/linux binaries are already uploaded
-#   to the GitHub release before packaging starts)
-# - workflow_dispatch: manual fallback (backfill / re-run) with optional tag/run_id
-#
-# Flow:
-# 1. Resolve the triggering Build workflow run for the release tag
-# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu) from build artifacts
-# 3. Build DEB packages for amd64 and arm64
-# 4. Build RPM packages for x86_64 and aarch64
-# 5. Upload all packages to Cloudflare R2 and the GitHub release
-
-name: Package DEB/RPM
-
-permissions:
-  # contents: write is required to upload packages to the GitHub release
-  contents: write
-  actions: read
-
-on:
-  # Follows the same pattern as docker.yml: run after the release build
-  # workflow completes, so packaging is triggered only by release tags
-  # (e.g. 1.0.0-rc.2, 1.0.0-rc.3), never by development builds.
-  workflow_run:
-    workflows: [ "Build and Release" ]
-    types: [ completed ]
-  workflow_dispatch:
-    inputs:
-      tag:
-        description: "Release tag to package (e.g. 1.0.0-beta.12). Leave empty for latest main build."
-        required: false
-        type: string
-      build_run_id:
-        description: "Build workflow run ID (overrides tag lookup)"
-        required: false
-        type: string
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.event.workflow_run.head_branch || github.event.inputs.tag || github.run_id }}
-  cancel-in-progress: true
-
-env:
-  HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
-  WORKFLOW_RUN_ID: ${{ github.event.workflow_run.id }}
-
-jobs:
-  # Resolve which build run to use and extract version info
-  resolve:
-    name: Resolve Build
-    # Auto-trigger only from successful tag builds of "Build and Release".
-    # Tag pushes arrive as event == push with head_branch != main (a
-    # non-main push head_branch is the release tag name). Manual dispatch
-    # stays available as a fallback for backfills and re-runs.
-    if: >-
-      github.event_name == 'workflow_dispatch' ||
-      (github.event.workflow_run.conclusion == 'success' &&
-       github.event.workflow_run.event == 'push' &&
-       github.event.workflow_run.head_branch != 'main')
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    outputs:
-      version: ${{ steps.resolve.outputs.version }}
-      build_type: ${{ steps.resolve.outputs.build_type }}
-      build_run_id: ${{ steps.resolve.outputs.build_run_id }}
-      tag: ${{ steps.resolve.outputs.tag }}
-    steps:
-      - name: Resolve build run
-        id: resolve
-        shell: bash
-        env:
-          GH_TOKEN: ${{ github.token }}
-          INPUT_TAG: ${{ github.event.inputs.tag }}
-          INPUT_RUN_ID: ${{ github.event.inputs.build_run_id }}
-        run: |
-          set -euo pipefail
-
-          # Determine tag
-          if [[ "${{ github.event_name }}" == "workflow_run" ]]; then
-            TAG="${HEAD_BRANCH}"
-          elif [[ -n "$INPUT_TAG" ]]; then
-            TAG="$INPUT_TAG"
-          else
-            TAG=""
-          fi
-
-          echo "Tag: ${TAG:-<none>}"
-
-          # Determine build run ID
-          BUILD_RUN_ID=""
-
-          if [[ -n "$INPUT_RUN_ID" ]]; then
-            # Explicit run ID takes priority
-            BUILD_RUN_ID="$INPUT_RUN_ID"
-            echo "Using explicit build run ID: $BUILD_RUN_ID"
-
-          elif [[ "${{ github.event_name }}" == "workflow_run" ]]; then
-            # Use the Build and Release run that triggered this workflow
-            BUILD_RUN_ID="${WORKFLOW_RUN_ID}"
-            echo "Using triggering workflow run: $BUILD_RUN_ID"
-
-          elif [[ -n "$TAG" ]]; then
-            # Find the build run that produced this tag
-            echo "Looking for build run for tag: $TAG"
-            BUILD_RUN_ID=$(gh api \
-              "repos/${{ github.repository }}/actions/workflows/build.yml/runs?branch=${TAG}&status=success&per_page=1" \
-              --jq '.workflow_runs[0].id' 2>/dev/null || echo "")
-
-            if [[ -z "$BUILD_RUN_ID" || "$BUILD_RUN_ID" == "null" ]]; then
-              # Tag might not be a branch; try event=push with head_branch matching
-              BUILD_RUN_ID=$(gh api \
-                "repos/${{ github.repository }}/actions/workflows/build.yml/runs?event=push&status=success&per_page=100" \
-                --jq ".workflow_runs[] | select(.head_branch == \"$TAG\") | .id" 2>/dev/null | head -1 || echo "")
-            fi
-
-            if [[ -z "$BUILD_RUN_ID" || "$BUILD_RUN_ID" == "null" ]]; then
-              echo "❌ No successful build run found for tag: $TAG"
-              exit 1
-            fi
-            echo "Found build run: $BUILD_RUN_ID"
-
-          else
-            # No tag — latest successful main build
-            echo "No tag specified, looking for latest main build"
-            BUILD_RUN_ID=$(gh api \
-              "repos/${{ github.repository }}/actions/workflows/build.yml/runs?branch=main&status=success&per_page=1" \
-              --jq '.workflow_runs[0].id' 2>/dev/null || echo "")
-
-            if [[ -z "$BUILD_RUN_ID" || "$BUILD_RUN_ID" == "null" ]]; then
-              echo "❌ No successful main build found"
-              exit 1
-            fi
-            echo "Latest main build: $BUILD_RUN_ID"
-          fi
-
-          # Determine version and build type
-          if [[ -n "$TAG" ]]; then
-            VERSION="$TAG"
-            if [[ "$TAG" == *"-preview"* ]]; then
-              BUILD_TYPE="preview"
-            elif [[ "$TAG" == *"alpha"* || "$TAG" == *"beta"* || "$TAG" == *"rc"* ]]; then
-              BUILD_TYPE="prerelease"
-            else
-              BUILD_TYPE="release"
-            fi
-          else
-            SHORT_SHA=$(gh api "repos/${{ github.repository }}/actions/runs/${BUILD_RUN_ID}" \
-              --jq '.head_sha' 2>/dev/null | head -c 7)
-            VERSION="dev-${SHORT_SHA}"
-            BUILD_TYPE="development"
-          fi
-
-          {
-            echo "version=$VERSION"
-            echo "build_type=$BUILD_TYPE"
-            echo "build_run_id=$BUILD_RUN_ID"
-            echo "tag=${TAG}"
-          } >> "$GITHUB_OUTPUT"
-
-          echo "📊 Resolved:"
-          echo "  Version: $VERSION"
-          echo "  Build type: $BUILD_TYPE"
-          echo "  Build run ID: $BUILD_RUN_ID"
-
-  # Build DEB and RPM packages for each architecture
-  package:
-    name: Package (${{ matrix.arch }})
-    needs: resolve
-    runs-on: ubuntu-latest
-    timeout-minutes: 30
-    strategy:
-      fail-fast: false
-      max-parallel: 1
-      matrix:
-        include:
-          - arch: x86_64
-            deb_arch: amd64
-            rpm_arch: x86_64
-            artifact_name: "rustfs-linux-x86_64-gnu"
-          - arch: aarch64
-            deb_arch: arm64
-            rpm_arch: aarch64
-            artifact_name: "rustfs-linux-aarch64-gnu"
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Download binary artifact from build run
-        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
-        with:
-          pattern: ${{ matrix.artifact_name }}*
-          path: ./binary-artifact
-          run-id: ${{ needs.resolve.outputs.build_run_id }}
-          github-token: ${{ github.token }}
-          merge-multiple: true
-
-      - name: Extract binary
-        id: binary
-        shell: bash
-        run: |
-          set -euo pipefail
-
-          ZIP_FILE=$(find ./binary-artifact -name "*.zip" -type f | head -1)
-          if [[ -z "$ZIP_FILE" ]]; then
-            echo "❌ No binary artifact found"
-            ls -la ./binary-artifact/ || true
-            exit 1
-          fi
-
-          echo "Found artifact: $ZIP_FILE"
-
-          mkdir -p ./bin
-          unzip -o "$ZIP_FILE" -d ./bin
-
-          if [[ ! -f ./bin/rustfs ]]; then
-            echo "❌ rustfs binary not found in archive"
-            exit 1
-          fi
-
-          chmod +x ./bin/rustfs
-          ls -lh ./bin/rustfs
-          echo "✅ Binary extracted"
-
-      - name: Build DEB package
-        id: deb
-        shell: bash
-        run: |
-          set -euo pipefail
-
-          VERSION="${{ needs.resolve.outputs.version }}"
-          DEB_ARCH="${{ matrix.deb_arch }}"
-          # DEB version: replace - with ~ (1.0.0-beta.12 -> 1.0.0~beta.12)
-          # Use a variable for ~ to prevent tilde expansion by bash
-          TILDE='~'
-          DEB_VERSION="${VERSION/-/$TILDE}"
-          PKG_DIR="rustfs_${DEB_VERSION}_${DEB_ARCH}"
-
-          echo "Building DEB: ${PKG_DIR}.deb"
-
-          mkdir -p "${PKG_DIR}/DEBIAN"
-          mkdir -p "${PKG_DIR}/usr/bin"
-          mkdir -p "${PKG_DIR}/etc/default"
-          mkdir -p "${PKG_DIR}/lib/systemd/system"
-          mkdir -p "${PKG_DIR}/usr/share/doc/rustfs"
-
-          cp ./bin/rustfs "${PKG_DIR}/usr/bin/"
-          chmod 755 "${PKG_DIR}/usr/bin/rustfs"
-
-          cp deploy/build/rustfs.service "${PKG_DIR}/lib/systemd/system/"
-
-          cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
-          # RustFS Environment Configuration
-          # See https://rustfs.com/docs/ for more information
-          # RUSTFS_VOLUMES=""
-          # RUSTFS_ROOT_USER=""
-          # RUSTFS_ROOT_PASSWORD=""
-          ENVEOF
-
-          cat > "${PKG_DIR}/DEBIAN/control" << EOF
-          Package: rustfs
-          Version: ${DEB_VERSION}
-          Section: utils
-          Priority: optional
-          Architecture: ${DEB_ARCH}
-          Depends: libc6 (>= 2.31)
-          Maintainer: RustFS Team <support@rustfs.com>
-          Description: High-performance distributed object storage
-           RustFS is a high-performance distributed object storage software
-           built using Rust. It is compatible with MinIO and S3 API.
-          Homepage: https://rustfs.com
-          EOF
-
-          # Declare /etc/default/rustfs as a conffile so dpkg preserves user
-          # modifications on upgrade instead of silently overwriting them.
-          cat > "${PKG_DIR}/DEBIAN/conffiles" << 'CONFFILES'
-          /etc/default/rustfs
-          CONFFILES
-
-          cat > "${PKG_DIR}/DEBIAN/postinst" << 'POSTINST'
-          #!/bin/bash
-          set -e
-          if ! getent passwd rustfs > /dev/null 2>&1; then
-              useradd -r -s /bin/false -d /opt/rustfs rustfs
-          fi
-          mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
-          chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
-          if [ -d /run/systemd/system ]; then
-              systemctl daemon-reload
-          fi
-          echo "RustFS installed. Configure /etc/default/rustfs then: systemctl start rustfs"
-          POSTINST
-          chmod 755 "${PKG_DIR}/DEBIAN/postinst"
-
-          cat > "${PKG_DIR}/DEBIAN/prerm" << 'PRERM'
-          #!/bin/bash
-          set -e
-          if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
-              systemctl stop rustfs
-          fi
-          PRERM
-          chmod 755 "${PKG_DIR}/DEBIAN/prerm"
-
-          cat > "${PKG_DIR}/DEBIAN/postrm" << 'POSTRM'
-          #!/bin/bash
-          set -e
-          if [ -d /run/systemd/system ]; then
-              systemctl daemon-reload
-          fi
-          POSTRM
-          chmod 755 "${PKG_DIR}/DEBIAN/postrm"
-
-          cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
-          cp README.md "${PKG_DIR}/usr/share/doc/rustfs/"
-
-          fakeroot dpkg-deb --build "${PKG_DIR}"
-
-          DEB_FILE="${PKG_DIR}.deb"
-          ls -lh "$DEB_FILE"
-          echo "deb_file=$DEB_FILE" >> "$GITHUB_OUTPUT"
-          echo "✅ DEB built: $DEB_FILE"
-
-      - name: Build RPM package
-        id: rpm
-        shell: bash
-        run: |
-          set -euo pipefail
-
-          VERSION="${{ needs.resolve.outputs.version }}"
-          RPM_ARCH="${{ matrix.rpm_arch }}"
-
-          echo "Building RPM for ${RPM_ARCH}"
-
-          sudo apt-get update && sudo apt-get install -y ruby ruby-dev build-essential
-          sudo gem install fpm
-
-          # Create config file for fpm (DEB build creates it in its package dir structure,
-          # but fpm needs the file to exist before packaging)
-          mkdir -p ./tmp-pkg/etc/default
-          cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
-          # RustFS Environment Configuration
-          # See https://rustfs.com/docs/ for more information
-          # RUSTFS_VOLUMES=""
-          # RUSTFS_ROOT_USER=""
-          # RUSTFS_ROOT_PASSWORD=""
-          ENVEOF
-
-          fpm -s dir -t rpm \
-            --name rustfs \
-            --version "$VERSION" \
-            --architecture "$RPM_ARCH" \
-            --depends "glibc >= 2.31" \
-            --maintainer "RustFS Team <support@rustfs.com>" \
-            --description "High-performance distributed object storage" \
-            --url "https://rustfs.com" \
-            --license "Apache-2.0" \
-            --after-install <(cat <<'POSTINST'
-          #!/bin/bash
-          set -e
-          if ! getent passwd rustfs > /dev/null 2>&1; then
-              useradd -r -s /bin/false -d /opt/rustfs rustfs
-          fi
-          mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
-          chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
-          if [ -d /run/systemd/system ]; then
-              systemctl daemon-reload
-          fi
-          POSTINST
-            ) \
-            --before-remove <(cat <<'PRERM'
-          #!/bin/bash
-          set -e
-          if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
-              systemctl stop rustfs
-          fi
-          PRERM
-            ) \
-            --after-remove <(cat <<'POSTRM'
-          #!/bin/bash
-          set -e
-          if [ -d /run/systemd/system ]; then
-              systemctl daemon-reload
-          fi
-          POSTRM
-            ) \
-            --config-files /etc/default/rustfs \
-            ./bin/rustfs=/usr/bin/rustfs \
-            ./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
-            deploy/build/rustfs.service=/lib/systemd/system/rustfs.service \
-            LICENSE=/usr/share/doc/rustfs/LICENSE \
-            README.md=/usr/share/doc/rustfs/README.md
-
-          RPM_FILE=$(ls -1 rustfs-*.rpm 2>/dev/null | head -1)
-          if [[ -z "$RPM_FILE" ]]; then
-            echo "❌ RPM build failed"
-            exit 1
-          fi
-
-          ls -lh "$RPM_FILE"
-          echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
-          echo "✅ RPM built: $RPM_FILE"
-
-      - name: Upload packages to artifacts
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: packages-${{ matrix.arch }}
-          path: |
-            *.deb
-            *.rpm
-          retention-days: 30
-
-      - name: Upload packages to Cloudflare R2
-        if: env.R2_ACCESS_KEY_ID != ''
-        env:
-          R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
-          R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
-          R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
-          R2_BUCKET: ${{ secrets.R2_BUCKET }}
-          AWS_EC2_METADATA_DISABLED: true
-        shell: bash
-        run: |
-          set -euo pipefail
-
-          if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
-            echo "⚠️ R2 credentials missing, skipping upload"
-            exit 0
-          fi
-
-          if ! command -v aws >/dev/null 2>&1; then
-            sudo apt-get update && sudo apt-get install -y awscli
-          fi
-
-          export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
-          export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
-          export AWS_DEFAULT_REGION="auto"
-
-          BUILD_TYPE="${{ needs.resolve.outputs.build_type }}"
-          if [[ "$BUILD_TYPE" == "development" ]]; then
-            R2_PREFIX="artifacts/rustfs/packages/dev"
-          else
-            R2_PREFIX="artifacts/rustfs/packages/release"
-          fi
-          R2_PATH="s3://${R2_BUCKET}/${R2_PREFIX}/"
-
-          echo "📤 Uploading to $R2_PATH"
-
-          DEB_FILE="${{ steps.deb.outputs.deb_file }}"
-          RPM_FILE="${{ steps.rpm.outputs.rpm_file }}"
-
-          for f in "$DEB_FILE" "$RPM_FILE"; do
-            if [[ -n "$f" && -f "$f" ]]; then
-              echo "Uploading: $f"
-              aws s3 cp "$f" "$R2_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
-            fi
-          done
-
-          echo "✅ Upload complete"
-
-          # Also upload as latest for release/prerelease
-          if [[ "$BUILD_TYPE" == "release" || "$BUILD_TYPE" == "prerelease" ]]; then
-            LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
-            for f in "$DEB_FILE" "$RPM_FILE"; do
-              if [[ -n "$f" && -f "$f" ]]; then
-                echo "Uploading latest: $(basename "$f")"
-                aws s3 cp "$f" "$LATEST_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
-              fi
-            done
-            echo "✅ Latest packages updated"
-          fi
-
-      - name: Upload packages to GitHub Release
-        if: needs.resolve.outputs.tag != ''
-        env:
-          GH_TOKEN: ${{ github.token }}
-        shell: bash
-        run: |
-          set -euo pipefail
-
-          TAG="${{ needs.resolve.outputs.tag }}"
-          DEB_FILE="${{ steps.deb.outputs.deb_file }}"
-          RPM_FILE="${{ steps.rpm.outputs.rpm_file }}"
-
-          # Upload the packages, then refresh the release checksums so the new
-          # assets are covered, matching the binary release flow.
-          for f in "$DEB_FILE" "$RPM_FILE"; do
-            if [[ -n "$f" && -f "$f" ]]; then
-              echo "📤 Uploading $(basename "$f") to GitHub release ${TAG}..."
-              gh release upload "$TAG" "$f" --clobber
-            fi
-          done
-
-          CHECKSUM_DIR="$(mktemp -d)"
-          gh release download "$TAG" -p 'SHA256SUMS' -p 'SHA512SUMS' \
-            -D "$CHECKSUM_DIR" --clobber
-
-          for spec in "SHA256SUMS:sha256sum" "SHA512SUMS:sha512sum"; do
-            asset="${spec%%:*}"
-            checksum_cmd="${spec##*:}"
-            checksum_file="${CHECKSUM_DIR}/${asset}"
-
-            for f in "$DEB_FILE" "$RPM_FILE"; do
-              if [[ -n "$f" && -f "$f" ]]; then
-                base="$(basename "$f")"
-                # GitHub stores release asset names with '~' normalized to '.'
-                # (e.g. rustfs_1.0.0~rc.2_amd64.deb is stored as
-                # rustfs_1.0.0.rc.2_amd64.deb), so checksum entries must
-                # reference the name as stored on the release.
-                github_base="${base//\~/.}"
-                # Remove any stale entry (both naming variants), then append
-                grep -Fv -- "$base" "$checksum_file" > "${checksum_file}.tmp" || true
-                grep -Fv -- "$github_base" "${checksum_file}.tmp" > "${checksum_file}.tmp2" || true
-                mv "${checksum_file}.tmp2" "$checksum_file"
-                digest=$("$checksum_cmd" -- "$f" | awk '{print $1}')
-                printf '%s  %s\n' "$digest" "$github_base" >> "$checksum_file"
-              fi
-            done
-
-            echo "📤 Updating ${asset} for release ${TAG}..."
-            gh release upload "$TAG" "$checksum_file" --clobber
-          done
-
-          echo "✅ GitHub release assets updated"
-
-  # Summary
-  summary:
-    name: Summary
-    needs: [ resolve, package ]
-    if: always()
-    runs-on: ubuntu-latest
-    timeout-minutes: 5
-    steps:
-      - name: Print summary
-        shell: bash
-        run: |
-          echo "## 📦 Package Summary" >> "$GITHUB_STEP_SUMMARY"
-          echo "" >> "$GITHUB_STEP_SUMMARY"
-          echo "| Item | Value |" >> "$GITHUB_STEP_SUMMARY"
-          echo "|------|-------|" >> "$GITHUB_STEP_SUMMARY"
-          echo "| Version | \`${{ needs.resolve.outputs.version }}\` |" >> "$GITHUB_STEP_SUMMARY"
-          echo "| Build Type | ${{ needs.resolve.outputs.build_type }} |" >> "$GITHUB_STEP_SUMMARY"
-          echo "| Build Run | #${{ needs.resolve.outputs.build_run_id }} |" >> "$GITHUB_STEP_SUMMARY"
-          echo "| Package Status | ${{ needs.package.result }} |" >> "$GITHUB_STEP_SUMMARY"

.github/workflows/windows-filesystem.yml

@@ -69,10 +69,6 @@ jobs:
           install-build-packaging-tools: 'false'
           install-test-tools: 'false'
 
-      - name: Check production Windows dependencies
-        shell: pwsh
-        run: cargo check -p rustfs-ecstore --lib
-
       - name: Test guarded rename publication
         shell: pwsh
         run: cargo test -p rustfs-ecstore --lib rename_all_ -- --nocapture

AGENTS.md

@@ -1,255 +1,373 @@
-# RustFS Agent Instructions
+# RustFS Agent Instructions (Global)
 
-This file contains repository-wide rules. Use the nearest subdirectory
-`AGENTS.md` for path-specific invariants.
+This root file keeps repository-wide rules only.
+Use the nearest subdirectory `AGENTS.md` for path-specific guidance.
 
-## Precedence
+## Rule Precedence
 
 1. System/developer instructions.
-2. The current user request.
-3. The nearest `AGENTS.md`.
-4. This file.
+2. Current user/task instructions.
+3. The nearest `AGENTS.md` in the current path.
+4. This file (global defaults).
 
-## Operating Model
+If repo-level instructions conflict, follow the nearest file and keep behavior aligned with CI.
 
-- Inquiry, diagnosis, review, and planning tasks are read-only unless the user
-  explicitly requests changes.
-- For implementation, read the relevant code, tests, and local guidance, then
-  make the smallest change that satisfies the request.
-- State assumptions only when they affect behavior or verification. Ask only
-  when a wrong assumption would materially change the result.
-- Do not load every skill or inspect unrelated modules preemptively. Select a
-  skill only when its description directly matches the request or changed
-  surface.
-- Avoid repeated reads and equivalent verification commands once enough
-  evidence exists.
+## Execution Discipline
+
+- Read the relevant existing code, tests, and local guidance before changing behavior. For new helpers or test setup, that read includes `crates/utils`, `crates/common`, and the touched crate's own `test_util`/fixtures (see Reuse Before You Write).
+- State assumptions when they affect the implementation or verification path.
+- If a task has multiple plausible interpretations, list the options briefly and choose the narrowest reasonable path; ask when the ambiguity would make the change risky.
+- For multi-step work, keep the plan minimal and tied to verifiable outcomes.
+- Avoid redundant file reads, repeated commands, and unnecessary exploratory work once enough context is available.
+- A good result is a minimal diff with clear assumptions, no over-engineering, and independent verification that survives Adversarial Validation (below).
 
 ## Worktree and Disk Hygiene
 
-- Start implementation from the latest `origin/main` and confirm the requested
-  change is not already present.
-- An existing clean, isolated task worktree is sufficient. Create another
-  worktree only when the current checkout is shared, dirty with unrelated work,
-  or belongs to another task.
-- Never commit from a shared checkout. Use an `overtrue/` feature branch unless
-  the user requests another name.
-- Check free space before artifact-heavy builds, tests, coverage, or downloads.
-  Re-check before a broad gate when space is tight.
-- Remove only task-owned temporary/build artifacts. Never delete another task's
-  worktree or uncommitted data.
-- At handoff, mention disk or cleanup details only when they affected execution
-  or artifacts/worktrees remain intentionally.
+- Unless the requester explicitly says otherwise, treat every new implementation task as isolated work: fetch the latest `origin/main`, confirm the requested change is not already present there, and create a dedicated feature branch and worktree from that exact upstream commit before editing. Do not implement new work directly in the primary checkout or reuse a worktree from another task.
+- Check available disk space before creating the worktree or starting dependency downloads, builds, tests, coverage, or other artifact-heavy commands. For long-running or artifact-heavy work, re-check disk usage at natural phase boundaries and before broad validation; if remaining space may not safely accommodate the next command, stop and reclaim task-owned artifacts before continuing.
+- Keep cleanup scoped and safe: remove generated build/test/coverage artifacts and temporary files created by the task when they are no longer needed, and never delete another task's worktree or uncommitted files. Prefer shared dependency caches where supported instead of duplicating large artifacts across worktrees.
+- At handoff, report the disk-space checks, cleanup performed, and any retained worktree or artifacts with the reason they are still needed.
 
-## Change Style
+## PR Lifecycle Monitoring
 
-- Preserve existing control flow unless changing it is required for correctness.
-- Prefer a direct local edit over new files, wrappers, managers, or speculative
-  abstractions.
-- Add a helper only when it removes current duplication, names a real domain
-  boundary, or isolates a non-trivial invariant.
-- Remove an in-scope path superseded by the change. If compatibility requires it,
-  adapt at the boundary to one canonical core and use the repository's
-  `RUSTFS_COMPAT_TODO` policy.
-- Comments explain non-obvious invariants or reasons. Do not narrate code or
-  record change history.
-- Mention unrelated problems when useful; do not fix them in a narrow task.
+- Creating or updating a PR is not the terminal state. Unless the requester explicitly limits the task to PR creation, monitor the PR through its terminal state: merged, closed, or explicitly handed off because progress requires user or maintainer action.
+- While the task is active, monitor CI/check runs, review decisions and unresolved threads, mergeability and conflicts, and unexpected head/base changes. Prefer event-driven or bounded waits provided by the current environment over frequent polling; report only state changes, actionable failures, or meaningful prolonged delays.
+- Investigate every failing check and review comment before changing code. Fix failures attributable to the task, run the verification required for the new diff, push the update, respond to or resolve the corresponding review threads, and resume monitoring. Do not weaken checks, dismiss valid feedback, or retry flaky failures merely to obtain a green result.
+- Treat opening, green CI, approval, and mergeability as intermediate states. Never merge without the required reviewer approval or explicit authority. If progress depends on credentials, infrastructure, a maintainer decision, or another external action, report the exact blocker and the evidence already collected.
+- If the current execution environment cannot remain active until the next PR event, use a supported automation, monitor, or thread wakeup when available and within scope. Otherwise leave an explicit handoff containing the PR, current state, next event to observe, and pending cleanup; do not imply that background monitoring exists when none is scheduled.
+- After observing a merge, verify the commits are preserved on the upstream base, ensure the worktree is clean, remove the dedicated worktree, prune stale worktree metadata, and delete the local task branch when it is no longer in use. For a closed or abandoned PR, preserve any unmerged work unless deletion was explicitly authorized. Do not delete remote branches unless explicitly requested or repository automation owns that cleanup.
 
-## Reuse and Boundary Rules
+## Autonomy and Approval Boundaries
 
-- Before adding helpers, constants, fixtures, or wrappers, search the touched
-  crate, the domain-owning crate, `crates/utils`, `crates/common`, and relevant
-  direct dependencies.
-- Reuse requires matching semantics: normalization, error types, deadlines,
-  durability, and compatibility must fit the call site. A narrowly named local
-  helper is better than forced reuse with different semantics.
-- Validate untrusted input at its trust boundary, then trust the validated type.
-  Values crossing disk, RPC, persistence, or version boundaries remain
-  untrusted at every consumer.
-- Re-check boundary values immediately before destructive actions such as
-  delete, overwrite, or quorum decisions.
-- Every new branch needs a concrete triggering input/state. For decoded or peer
-  data, corruption and mixed-version input are valid triggers.
-- Required values must return a typed error when absent or corrupt; do not use a
-  default that converts corruption into a plausible result.
-- Attach error context once where it is actionable. Do not erase typed errors
-  below aggregation or quorum layers.
+- Inquiry tasks (answer, explain, review, diagnose, plan): report findings; do not change files unless a fix is explicitly requested.
+- Action tasks (change, build, fix): make in-scope local changes without asking for approval.
+- Ask for confirmation before destructive or hard-to-reverse operations (force-pushes, history rewrites, deleting data or branches), merging a PR (reviewer approval required), or any material expansion of the requested scope.
+
+## Communication and Language
+
+- Respond in the same language used by the requester.
+- Keep source code, comments, commit messages, and PR title/body in English.
+- Be concise. Avoid sycophantic openers, closing fluff, and verbose status reporting.
+
+## Change Style for Existing Logic
+
+- Prefer direct, local code over extracting one-off helpers.
+- Extract a helper only when logic is reused or the extraction materially clarifies a non-trivial flow.
+- Use Rust's default module file layout (`mod foo;` with `foo.rs` or `foo/mod.rs`/`foo/*.rs`).
+  Avoid `#[path = "..."]` for module inclusion; move files into the canonical module tree instead.
+  If an unavoidable generated-code, FFI, or test-fixture exception remains, keep it local and document why the canonical layout cannot work.
+- Solve only the requested problem; do not add speculative features, configurability, or adjacent improvements.
+- Prefer editing existing code over rewriting files or reshaping unrelated logic.
+- Modify only what is required and remove only artifacts introduced by your own changes.
+- Preserve the existing control-flow and logic shape when fixing bugs or addressing review comments, especially in init, distributed coordination, locking, metadata, and concurrency paths.
+- Do not refactor existing code only to make it easier to unit test.
+- Keep fixes narrowly aligned with the requested behavior; avoid semantic-adjacent rewrites while touching sensitive paths.
+- Keep code elegant, concise, and direct. Prefer minimal, readable implementations over over-engineering and excessive abstraction. Use comments to clarify non-obvious intent and invariants, not to compensate for unclear code.
+- Do not write comments that narrate what the next line does, restate a signature, or describe the change you just made — that commentary belongs in the PR description, not the code. Required invariant comments — lock ordering, `SAFETY`, unwrap justification, `#[allow(dead_code)]` rationale, `RUSTFS_COMPAT_TODO` — are never narration.
+- Mention unrelated issues when useful, but do not fix them as part of a narrow task.
+
+## Reuse Before You Write
+
+Search for an existing implementation before writing a new one; extend what exists instead of duplicating it:
+
+- **Helpers and utilities** (path/string handling, hashing, retry, env parsing, IO wrappers): check `ls crates/utils/src` first — file names map to operations (`retry.rs`, `envs.rs`, `hash.rs`, `path.rs`, `string.rs`, `io.rs`) — plus `crates/common` (shared structures/globals), then `rg -i 'fn \w*<term>' crates/utils/src crates/common/src <touched-crate>/src` for signatures. Helpers are snake_case: a full-text single-word grep over a large crate drowns you and a multi-word phrase returns nothing. Reimplementing an existing workspace helper — or hand-rolling what `std`, `tokio`, or an existing workspace dependency already provides — is a review finding, not a style preference.
+- **Reuse requires matching semantics, not a matching name**: before adopting a helper, check its normalization (`clean` resolves `.`/`..` — never apply it to raw S3 object keys), error type, backoff/deadline behavior, and durability gating against the call site. When semantics differ, a new narrowly-named helper with a comment naming the rejected lookalike is the correct outcome. The inverse also holds: workspace wrappers exist because raw `std`/`tokio` semantics were insufficient (durability gates, retries) — prefer the wrapper over the raw call.
+- **Constants and fixed tokens** (protocol labels, error identifiers, header keys, event names, metric names, command tags): search for existing constants/enums that already represent the same semantic value and reuse them. If a value is truly new, define one local constant near related logic; never scatter the literal across sites. When changing existing behavior, align naming and format with the established constants.
+- **Test scaffolding**: reuse existing test utilities and fixtures (the touched crate's own `test_util` module and `tests/fixtures`, or `crates/test-utils`) instead of writing new setup code — run `rg -l '<fn-under-test>' <crate>/src <crate>/tests` before writing a test. A new test must pin a failure mode no existing test covers. Near-duplicate means same code path AND same poison-value class: this repo's boundary companions (n==max vs max+1, absent vs empty vs nil UUID bytes, MetaObject vs MetaDeleteMarker) are distinct by definition and must all be written.
+
+## Necessary Code Only
+
+Net-new code — files, types, branches, comments — is cost to justify, not progress:
+
+- Validate at the trust boundary — untrusted client input, bytes read from disk, RPC payloads, config (see Serde Safety and Cross-Cutting Domain Invariants) — then trust the type: do not re-check what the type system or a validated upstream layer already guarantees, and cite the establishing check (`file:line`) when the guarantee is not obvious.
+- The exception is load-bearing: a value that crossed a persistence, RPC, or version boundary is never guaranteed by the code on the other side — a peer may be older or buggy, disk bytes may be corrupt — so the Cross-Cutting Domain Invariant patterns apply at every consumer, and re-checks immediately before a destructive action (delete, overwrite, quorum decision) stay. Deleting an existing guard is a behavior change requiring adversarial review, not cleanup.
+- Every new branch needs a nameable trigger: a concrete input, state, or failure that reaches it — for boundary-crossing values, corrupt or stale persisted/peer data is always nameable. If you cannot name one, do not write the branch. If the case is truly unreachable, encode the invariant in the type; where that is impossible, return a typed internal error (fail closed). `debug_assert!` is acceptable only for pure internal arithmetic on values that never crossed a disk/RPC/config boundary — never as the sole guard on decoded or peer-supplied data.
+- Never substitute a default where the value is required (e.g. `unwrap_or_default()` on metadata that must exist) — that converts corruption into a wrong answer. Return the typed error instead: explicit failure over implicit success.
+- Attach error context once, at the layer where it is actionable: re-wrapping equivalent context at every hop is noise, and expanding a fallible chain into nested `match` blocks where `?` or a combinator suffices is a finding. Never add context by converting a typed error into a generic variant below an error-aggregation or quorum layer (`reduce_errs` classifies by variant equality) — context there belongs in a `tracing` event, not the error value.
 
 ## Sources of Truth
 
-- Workspace membership: `Cargo.toml`.
-- Local gates: `Makefile` and `.config/make/`.
-- CI gates: `.github/workflows/ci.yml`.
-- PR format: `.github/pull_request_template.md`.
-- Architecture routing: `ARCHITECTURE.md` and `docs/architecture/README.md`.
-- Agent skills: `.agents/skills/*/SKILL.md`.
+- Workspace layout and crate membership: `Cargo.toml` (`[workspace].members`)
+- Local quality commands: `Makefile` and `.config/make/`
+- CI quality gates: `.github/workflows/ci.yml`
+- PR template: `.github/pull_request_template.md`
+- High-level architecture and crate map: `ARCHITECTURE.md`
+- Migration guardrails, readiness contracts, support matrices:
+  `docs/architecture/README.md` (routes by audience)
+- Shared agent skills (all tools): `.agents/skills/` — each `SKILL.md` carries
+  a frontmatter `description` stating when it applies. Scan the descriptions
+  before starting a task and follow any skill that matches, even if your tool
+  does not auto-load skills:
+  `grep -m1 '^description:' .agents/skills/*/SKILL.md`
+  Claude Code reads them through the `.claude/skills` symlink; add new skills
+  to `.agents/skills/` only, never as separate copies per tool
 
-Do not commit one-shot plans, trackers, migration ledgers, benchmark snapshots,
-or agent scratch notes. Durable architecture belongs under `docs/architecture/`,
-operations under `docs/operations/`, and testing references under
-`docs/testing/`. `scripts/check_no_planning_docs.sh` enforces this boundary.
+Avoid duplicating long crate lists or command matrices in instruction files.
+Reference the source files above instead.
 
-## Verification
+Do not commit planning-type documents — one-shot implementation/optimization
+plans, task trackers, migration-progress ledgers, phase/PR templates,
+issue-scoped benchmark-result snapshots or optimization conclusions, or
+agent-generated working notes (e.g. anything a `superpowers`/scratch workflow
+produces). Keep that work in the issue tracker or your local worktree, not in
+the repository. Only durable reference — the architecture set under
+`docs/architecture/`, repeatable operational runbooks under `docs/operations/`,
+and the test-suite references under `docs/testing/` — belongs in version
+control; `.gitignore` ignores everything else under `docs/` by default, so a new
+plan file will not be tracked unless someone force-adds it — don't.
+`scripts/check_no_planning_docs.sh` (wired into `make pre-commit`/`pre-pr` and
+CI) fails the build if anything is committed under `docs/superpowers/`, even via
+`git add -f`.
 
-Select checks from the final task-owned diff. Scoped `AGENTS.md` files may add a
-concrete path-specific check, but must not replace this tiering with a generic
-full-workspace gate.
+## Verification Before PR
 
-### Documentation and Instructions
+Convert changes into independently verifiable outcomes. This section controls
+agent-run local validation; preparing a commit or PR does not by itself require
+the broadest gate. Inspect only the final task-owned diff, classify it by
+behavioral impact rather than line count or path alone, and run the smallest
+set of checks that provides meaningful coverage. Do not let unrelated
+worktree changes or a generic contributor checklist expand the scope.
+Non-exempt changes must also pass Adversarial Validation (next section) before
+the checks below count as completion.
 
-For prose, comments, agent instructions, and skill metadata that cannot affect
-runtime/build output:
+### Validation floor
 
-- Run `git diff --check`.
-- Run the relevant documentation guard or skill validator when applicable.
-- Skip Cargo formatting, compilation, Clippy, tests, `make pre-commit`, and
-  `make pre-pr`.
+- Every change that is not documentation-only must finish with
+  `cargo fmt --all --check` passing. An umbrella gate that runs this exact
+  check satisfies the requirement; do not run it twice. Use `cargo fmt --all`
+  only when formatting needs to be fixed. Run the configured formatter or
+  validator for other changed languages when one exists.
+- Documentation-only or instruction-only means all task-owned changes are
+  prose or documentation assets and cannot affect runtime, builds, CI,
+  dependencies, generated code, or tests. Run `git diff --check` and any
+  relevant documentation guard, but skip Cargo formatting, compilation,
+  Clippy, tests, `make pre-commit`, and `make pre-pr`.
+- Behavior changes require relevant existing or new tests. Prefer the most
+  focused test or affected package. A passing targeted test can also provide
+  sufficient compilation coverage when it builds every changed target and
+  feature involved; do not add a redundant `cargo check` in that case.
+- `cargo check` supplements compilation coverage; it never substitutes for a
+  behavioral test. If a relevant test cannot reasonably be added or run, use
+  the narrowest compilation check and report the reason and remaining risk.
 
-### Non-Behavioral Source Changes
+### Validation tiers
 
-- Run the formatter/validator for the changed language.
-- Add compilation or doctests only when syntax or executable examples changed.
+1. **Documentation/instruction-only:** Apply the exemption above. Run a guard
+   such as `make doc-paths-check` only when it is relevant to the edited text.
+2. **Non-behavioral source change:** For comments, formatting, or another
+   demonstrably non-executable change, run the formatting floor. Compilation,
+   Clippy, and tests may be skipped only when the edit cannot affect
+   compilation or runtime behavior; run targeted doctests if executable
+   documentation examples changed.
+3. **Localized or bounded behavior change:** Run the formatting floor and the
+   narrowest relevant tests. Add package-scoped `cargo check` or Clippy only
+   for changed targets, features, APIs, error handling, async behavior, or
+   control flow not already covered. When several crates are affected but the
+   dependency set is identifiable, validate those packages and known
+   dependents instead of the whole workspace. Use `make pre-commit` only when
+   a repository-wide fast gate adds useful confidence beyond those checks.
+4. **Broad or high-risk change:** Run `make pre-pr` only when targeted coverage
+   cannot bound the impact, including:
+   - dependency, feature, build-script, procedural-macro, code-generation,
+     toolchain, or CI changes that alter compilation or the test matrix;
+   - cross-crate public APIs, shared foundational code, or broad refactors with
+     an unbounded dependent set;
+   - locking, storage durability or formats, erasure coding, replication,
+     RPC/protocol compatibility, IAM/KMS/auth, cryptography, or other
+     security-sensitive behavior;
+   - a targeted check that reveals wider impact, an explicit user request, or
+     a release policy that requires the full gate.
 
-### Localized Behavior Changes
+Documentation-only and non-behavioral classifications take precedence over
+path-based triggers. A small diff can still be high-risk, while a CI comment,
+manifest comment, or release-note edit does not require full validation.
 
-- Run `cargo fmt --all --check` for Rust changes.
-- Run the narrowest test that exercises the changed behavior.
-- Add package-scoped `cargo check` or Clippy only for targets, features, public
-  APIs, error handling, or control flow not compiled by the focused test.
-- Use `make pre-commit` only when its repository-wide fast checks add confidence
-  beyond the focused checks.
+`make pre-pr` includes `make pre-commit` coverage. Never run both for the same
+unchanged diff, and do not repeat equivalent checks during PR preparation or
+because a local hook already ran them. Rerun only checks whose scope is affected
+by later edits. Full workspace checks do not replace a relevant integration or
+E2E test for changed behavior; run that focused test when required and
+available, or report why it was not run and the remaining risk.
 
-### Broad or High-Risk Changes
+If `make` is unavailable, run the equivalent checks defined under
+`.config/make/`. At handoff, list the checks actually run, checks intentionally
+skipped, and the reason for the selected tier.
 
-After the required adversarial review, run `make pre-pr` when targeted coverage
-cannot bound the impact, including dependency/toolchain/build-matrix changes,
-unbounded cross-crate APIs, or locking, durability, erasure coding, replication,
-RPC, IAM/KMS/auth, cryptography, on-disk/on-wire, and S3-visible behavior.
+After build-based verification completes, clean generated build artifacts before wrapping up to avoid unnecessary disk usage.
+Do not open a PR with code changes when the required checks fail.
+Make a failing check pass by fixing the cause, never by weakening the gate:
+do not loosen or skip a guard script, add entries to a baseline or allowance
+list, suppress a lint with `#[allow]`, mark a failing test `#[ignore]`, or
+delete or relax a failing assertion to get green. If a check itself is wrong,
+change it deliberately and state the rationale in the PR.
 
-`make pre-pr` includes `make pre-commit`; never run both for the same unchanged
-diff. Do not repeat a check already covered by a successful umbrella gate.
-Rerun only checks affected by later edits.
+For flaky tests, do not paper over them with retries. Follow the flake policy
+in [docs/testing/README.md](docs/testing/README.md) (open an issue within 24h,
+quarantine with an issue link, fix or delete within 30 days); the local
+`default` nextest profile never retries.
 
-Never weaken a gate to get green: do not add baselines/allowances, suppress
-lints, ignore tests, or relax assertions unless changing that policy is itself
-the reviewed task. Follow `docs/testing/README.md` for flaky tests.
+## Adversarial Validation (Default On)
 
-## Adversarial Validation
+Every non-exempt output (see Risk tiers) — code change, bug fix, or
+design/solution proposal — passes multi-role adversarial review before it
+counts as done.
+Author confidence is not evidence: each role's job is to refute the change,
+not to bless it.
 
-Adversarial validation applies to final implementation diffs, explicitly
-requested adversarial/design reviews, and agent-instruction changes that alter
-execution. Ordinary questions, diagnoses, status reports, non-adversarial code
-reviews, and low-risk planning do not trigger it.
+### Risk tiers
 
-Risk and review shape:
+Pick the tier from the riskiest file touched; when in doubt, pick the higher.
 
-- **Exempt:** documentation, comments, formatting, or typos with no runtime,
-  build, test, or agent-execution effect.
-- **Mechanical:** renames, moves, test/tooling-only changes, and agent-rule
-  changes. Run correctness and simplicity lenses.
-- **Standard:** localized behavior changes. Run one integrated final-diff pass
-  covering correctness, simplicity, and test coverage; add only domain lenses
-  matched by the diff.
-- **High risk / substantial PR review:** high risk includes locking,
-  erasure/quorum/heal, replication, multipart, RPC, lifecycle/tiering,
-  persistence/fsync, IAM/KMS/auth, cryptography, on-disk/on-wire formats, and
-  S3-visible semantics. Cover all applicable lenses using exactly two
-  independent reviewers when delegation is explicitly authorized. Split the
-  lenses between them. Otherwise perform two fresh sequential passes.
+- **Exempt:** docs/comments/instruction-only changes, formatting, typos with
+  no runtime surface. Skip this section.
+- **Mechanical:** pure renames, file moves, test-only or tooling changes —
+  correctness and simplicity adversaries only.
+- **Standard (the default):** any change that affects behavior.
+- **High risk:** touches locking, erasure coding, quorum/heal, replication,
+  multipart, RPC, lifecycle/tiering, metadata formats (`xl.meta`),
+  persistence/fsync, IAM/KMS/auth, on-disk or on-wire formats, or
+  S3 API-visible behavior.
 
-Available domain lenses are security, concurrency/durability, compatibility,
-and performance. Select `.agents/skills/adversarial-validation/SKILL.md` for an
-explicit adversarial request, a high-risk change, or a substantial PR review;
-then read only its matching role references. A routine standard pass does not
-load the playbook unless the reviewer needs a RustFS-specific probe.
+### Roles
 
-A finding must name a concrete input/state/interleaving and wrong outcome, or a
-specific missing regression check, with `file:line`. Resolve it by fixing the
-diff or rebutting it with code-path/test/invariant evidence. After a non-trivial
-fix, rerun only affected lenses.
+Run each applicable role as an independent pass over the final diff (or
+proposal text) — parallel reviewer agents where the tooling supports them,
+otherwise sequential passes that each start fresh from the diff and the
+nearest scoped `AGENTS.md`, discarding the writing session's assumptions.
+Each role either produces findings or reports "attacked X, Y, Z — no break
+found"; a bare pass is not a result. Repo-specific attack probes for every
+role live in `.agents/skills/adversarial-validation/` — run them, they
+encode this repo's shipped bugs.
 
-For high-risk PRs, record one concise verdict per covered lens in the PR body.
+- **Correctness adversary** — construct a concrete input/state/interleaving
+  that yields wrong output, data loss, or a crash. Probe error paths and edge
+  values (empty, nil UUID, zero-length, quorum−1, missing version).
+- **Simplicity adversary** — same behavior, less code. Hunt the materially smaller or more idiomatic diff (see Change Style for Existing Logic, Reuse Before You Write, and Necessary Code Only): reimplemented workspace helpers, one-caller extractions, rewrites where an in-place edit suffices, defensive branches with no nameable trigger, redundant error wrapping, near-duplicate tests, narration comments. A smaller diff achieving identical behavior is a finding, reported with the concrete replacement; forced reuse of a helper with mismatched semantics is equally a finding.
+- **Security reviewer** — authn/authz bypass, injection, secret leakage,
+  untrusted deserialization (see Serde Safety), path traversal, timing leaks.
+- **Concurrency/durability reviewer** — lock ordering, races, cancellation,
+  partial failure, retry/idempotency, crash and power-loss ordering.
+- **Compatibility reviewer** — S3 API surface, MinIO interop, on-disk and
+  on-wire formats, mixed-version upgrade/downgrade paths.
+- **Performance reviewer** — allocation and cloning on hot paths, lock hold
+  time across IO, sync or CPU-heavy work on async runtime threads, added
+  fsync/flush outside the durability gate, hot-path logging noise. A
+  measurable regression on a per-request or per-object path is a finding.
+- **Test-coverage skeptic** — for each claimed behavior, name the test that
+  fails if the change is reverted; then name a changed line that could be
+  wrong while all tests stay green — if one exists, coverage is insufficient.
+  A missing test is a finding, not a note.
 
-## Pull Request Lifecycle
+Standard tier: correctness adversary + simplicity adversary + test-coverage
+skeptic, plus every role whose domain the diff touches (async or
+shared-state code → concurrency; parsing of untrusted input → security;
+public crate API shape → compatibility; per-request or per-object hot paths
+→ performance).
+High risk: all seven roles.
 
-- Creating or updating a PR includes one immediate snapshot of checks,
-  mergeability, reviews, and unresolved threads.
-- Unless the user explicitly requests monitoring, a release workflow requires
-  it, or an automation already owns it, hand off after the PR is open with the
-  current state and next event to watch. Do not delay ordinary handoff with
-  fixed quiet-period sleeps.
-- For requested monitoring, use event-driven or bounded waits. Report only state
-  changes, actionable failures, or a meaningful prolonged delay.
-- Investigate failures/comments before changing code. Fix task-attributable
-  issues, rerun affected verification, push, reply or resolve the thread, then
-  resume the requested monitor.
-- Never merge without required reviewer approval or explicit authority.
-- After an observed merge, verify the commit reached the base, then clean the
-  task worktree/branch when safe. Preserve unmerged work for closed PRs unless
-  deletion was explicitly authorized.
+### Protocol
+
+1. A finding states a concrete failure scenario (input/state → wrong
+   outcome) or names a missing test, with severity and file:line. "Looks
+   risky" is not a finding.
+2. Resolve every finding: fix it, or rebut it with evidence — a test, a
+   traced code path, or a cited invariant. Restated intent and "unlikely"
+   are not rebuttals.
+3. After non-trivial fixes, re-run the roles whose domain the fix touched.
+4. For proposals with no diff, roles attack assumptions, failure modes,
+   migration/rollback, and testability instead — including the simplest
+   rejected alternative and the blast radius when the design fails.
+
+### Exit criteria
+
+- Every applicable role has run; every finding is fixed or rebutted with
+  evidence.
+- Every behavior change has a test that fails without it.
+- The Verification Before PR gates pass — adversarial review supplements
+  those gates, never replaces them.
+- High risk only: record a one-line verdict per role in the PR description.
 
 ## Git and PR Baseline
 
-- Follow Conventional Commits; keep the subject at most 72 characters.
-- Source comments, commits, PR titles, and PR bodies are in English.
-- Keep every heading from `.github/pull_request_template.md`; use `N/A` where
-  needed and include commands actually run.
-- Use `--body-file` for multiline `gh pr create`/`gh pr edit` content.
-- PR/issue/discussion content must not contain the literal sequence `\n` or
-  hard-wrapped prose paragraphs.
-- Do not include local absolute paths or tool-specific labels/prefixes in GitHub
-  content.
-- Resolve review threads after the underlying issue is fixed. If declining a
-  suggestion, reply with a short evidence-based reason.
+- Use feature branches based on the latest `main`.
+- Assume other agent sessions work this repository concurrently. Never commit
+  in a shared checkout; do all work on a dedicated feature branch, preferably
+  in a dedicated worktree.
+- Immediately before branching, fetch `origin/main` and branch from it;
+  confirm the target issue is not already fixed there before writing code.
+- Follow Conventional Commits, with subject length <= 72 characters.
+- Keep PR title and description in English.
+- Use `.github/pull_request_template.md` and keep all section headings.
+- Use `N/A` for non-applicable template sections.
+- Include verification commands in the PR description.
+- When using `gh pr create`/`gh pr edit`, write the markdown body to a file
+  and pass `--body-file`; multiline inline `--body` is unsafe — backticks and
+  shell expansion can corrupt content or trigger unintended commands.
+  Pattern: `cat > /tmp/pr_body.md <<'EOF' ... EOF`, then
+  `--body-file /tmp/pr_body.md` (keep the file outside the checkout).
+- Do not include the literal sequence `\n` in any GitHub issue, pull request, or discussion comment.
+- Do not hard-wrap prose in PR/issue/discussion bodies; write each paragraph as a
+  single line and let it reflow. GitHub renders single newlines inside a paragraph
+  as line breaks, so mid-sentence wrapping shows up as ugly breaks. Only break lines
+  for list items, code blocks, and deliberate separators.
+- After fixing code review comments or CI findings, always mark corresponding review
+  comments/threads as resolved before returning to the user.
+- In handling review comments, confirm the underlying issue before changing code.
+  If a suggested change is not appropriate for behavior or risk, reply with a
+  concise rationale instead of blindly applying it.
 
 ## Security Baseline
 
 - Never commit secrets, credentials, or key material.
 - Use environment variables or vault tooling for sensitive configuration.
-- For localhost-sensitive tests, bypass proxies explicitly.
-- Untrusted S3 XML/JSON, lifecycle, policy, replication, and RPC structures use
-  strict deserialization where compatibility permits. Security-critical
-  defaults require explicit validation.
+- For localhost-sensitive tests, verify proxy settings to avoid traffic leakage.
 
-## Logging
+## Tools
 
-For every added or edited `tracing` call:
+### xl.meta decode tool Quick Use
 
-- Reuse the module's `EVENT_*`, `LOG_COMPONENT_*`, and `LOG_SUBSYSTEM_*`
-  constants and field shape.
-- Put fields first and a short label last.
-- Use `error` for behavior/security failure, `warn` for degradation/fallback,
-  `info` for low-frequency lifecycle, `debug` for diagnostics, and `trace` for
-  repetitive request/object success paths.
-- Never log secrets, credential payloads, or merged configs.
+```
+cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/file/xl.meta"
+```
 
-Use `.agents/skills/rustfs-logging-governance/SKILL.md` for logging changes.
+## Serde Safety
 
-## Cross-Cutting Storage Invariants
+- Add `#[serde(deny_unknown_fields)]` to structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs).
+- When `deny_unknown_fields` is impractical (backward compatibility), at minimum log unknown fields at `warn` level.
+- Never use `#[serde(default)]` on security-critical fields without explicit validation of the resulting value.
 
-- Write internal object metadata under both `x-rustfs-internal-<suffix>` and
-  `x-minio-internal-<suffix>` using
-  `crates/utils/src/http/metadata_compat.rs` helpers.
-- Read binary UUID metadata with
-  `.and_then(|v| Uuid::from_slice(&v).ok()).filter(|u| !u.is_nil())`; absent,
-  empty, and nil all mean no value.
-- Remote-tier version `None` or `""` means an unversioned bucket; send no
-  `versionId` on tier GET/DELETE.
-- `DataUsageCacheInfo` and `DataUsageEntry` keep their hand-written map
-  serialization and new fields remain `#[serde(default)]` for older readers.
+## Cross-Cutting Domain Invariants
 
-## Naming
+- Write internal object metadata under **both** `x-rustfs-internal-<suffix>`
+  and `x-minio-internal-<suffix>` keys (MinIO interop). Use the helpers in
+  `crates/utils/src/http/metadata_compat.rs` (`get_bytes` prefers the RustFS
+  key); never write only one of the two.
+- Read binary UUID metadata defensively:
+  `.and_then(|v| Uuid::from_slice(&v).ok()).filter(|u| !u.is_nil())` —
+  absent, empty, and nil all mean "no value", never `Uuid::nil()`.
+- A remote-tier version of `None`/`""` means the tier bucket is unversioned:
+  send **no** `versionId` on tier GET/DELETE.
+- Structs persisted in the scanner data-usage cache (`DataUsageCacheInfo`,
+  `DataUsageEntry`) carry a hand-written map-encoded `Serialize`. MessagePack
+  encodes derived structs as arrays, where an appended field makes the whole
+  cache a decode error for older readers — keep new fields `#[serde(default)]`
+  and keep the map encoding rather than reverting to `derive(Serialize)`.
 
-Use Rust API naming: `SCREAMING_SNAKE_CASE` constants/statics, `snake_case`
-functions/variables, and `PascalCase` types. Do not rename unrelated existing
-violations.
+## Naming Conventions
 
-## Scoped Guidance
+- Follow Rust API Guidelines for naming: `SCREAMING_SNAKE_CASE` for statics and constants, `snake_case` for functions and variables, `PascalCase` for types.
+- Do not use camelCase or Hungarian notation (e.g., `globalDeploymentIDPtr` → `GLOBAL_DEPLOYMENT_ID`).
+- If existing code violates naming conventions, do not widen the violation in new code. Do not rename existing symbols as part of an unrelated task; mention the violation instead (see Change Style for Existing Logic).
 
-Before editing, locate the nearest instructions with:
+## Scoped Guidance in This Repository
+
+Many crates and modules carry their own `AGENTS.md` with path-specific rules
+(security boundaries, lock ordering, domain invariants). Before editing a
+path, check for the nearest one:
 
 ```bash
 git ls-files '*AGENTS.md'
 ```
 
-The nearest file wins for domain invariants. Keep generic workflow and
-validation policy in this root file.
+The nearest file wins. Do not maintain a hand-written index of these files
+here — it goes stale.

ARCHITECTURE.md

@@ -1,6 +1,6 @@
 # ARCHITECTURE.md
 
-> Last updated: 2026-08-12 · Revision: 3
+> Last updated: 2026-07-02 · Revision: 2
 >
 > This document describes the high-level architecture of RustFS.
 > If you want to familiarize yourself with the code base, you are in the right place!
@@ -31,7 +31,7 @@ HTTP request
       → storage/ecfs (erasure coding, encryption, checksums)
         → ecstore (disk pool selection, data distribution)
           → rio (reader pipeline: encrypt → compress → hash → write)
-            → io-core (buffer pool, storage profiling, admission control)
+            → io-core (zero-copy I/O, buffer pool, direct I/O)
               → local disk / remote disk via RPC
 ```
 
@@ -55,7 +55,7 @@ rustfs/                      # Workspace root (virtual manifest)
 ├── crates/                  # library crates (authoritative list: Cargo.toml [workspace].members)
 │   ├── ecstore/             # Erasure-coded storage engine
 │   ├── rio/                 # Reader I/O pipeline (encrypt, compress, hash)
-│   ├── io-core/             # Buffer pool, storage profiling, backpressure/deadlock policy, lock optimizer, operation progress
+│   ├── io-core/             # Zero-copy I/O, scheduling, buffer pool
 │   ├── io-metrics/          # I/O metrics collection
 │   ├── common/              # Shared runtime state, globals, data usage types
 │   ├── config/              # Configuration types and parsing
@@ -101,10 +101,7 @@ refactors.
 
 The `rustfs` binary crate composes these libraries into the running server.
 `ecstore` remains the storage engine at the architectural center; its internal
-module split is tracked under `docs/architecture/`. `rio-v2` is the
-feature-gated MinIO on-disk format compatibility I/O layer; it ships in no
-default build (lifecycle:
-[docs/architecture/minio-file-format-compat.md](docs/architecture/minio-file-format-compat.md)).
+module split is tracked under `docs/architecture/`.
 
 ## Architecture Invariants
 
@@ -122,44 +119,19 @@ default build (lifecycle:
 
 3. **Each type has exactly one definition.** Types shared across crates must be defined
    in one crate and re-exported or imported by others.
-   - ⚠️ VIOLATED: `ReplicationStats` names three unrelated types
-     (`crates/data-usage/src/data_usage.rs`,
-     `crates/obs/src/metrics/collectors/replication.rs`,
-     `crates/ecstore/src/bucket/replication/replication_state.rs`) — a naming
-     collision, not copies; renaming is tracked in rustfs/backlog#1847.
-   - `LastMinuteLatency` has two deliberately different implementations: the
-     per-second bucketed accumulator in `crates/common/src/last_minute.rs` and
-     the in-memory endpoint-health sample tracker in
-     `crates/ecstore/src/bucket/bucket_target_sys.rs` (its doc comment explains
-     why it stays local).
-   - ✅ RESOLVED: `BackpressureConfig` and `DataUsageInfo` each have exactly one
-     definition (`crates/io-core/src/backpressure.rs`,
-     `crates/data-usage/src/data_usage.rs`). The zero-consumer
-     `BackpressureSettings` copy that lingered in io-metrics was removed
-     (rustfs/backlog#1833).
+   - ⚠️ VIOLATED: `ReplicationStats` (4 copies), `LastMinuteLatency` (3 copies),
+     `BackpressureConfig` (3 copies), `DataUsageInfo` (2 copies).
 
 4. **ecstore does not know about HTTP or S3 protocol details.** It operates on
    storage-level abstractions (objects, buckets, disks, pools).
-   - ⚠️ VIOLATED: 58 files under `crates/ecstore/src` reference `s3s`
-     (`rg -l 's3s' crates/ecstore/src | wc -l`), `crates/ecstore/src/client/`
-     is a ~9.4K-line embedded S3 HTTP client, and `crates/ecstore/Cargo.toml`
-     depends on `s3s`, `http`, `hyper`/`hyper-util`/`hyper-rustls`, and
-     `reqwest`. Target state: the engine's need to act as an S3 client
-     (tiering, replication targets) is served by an extracted client crate,
-     and ecstore holds no wire or DTO types.
 
 5. **The `rustfs` binary crate is the only place that wires everything together.**
    Individual crates should be testable in isolation.
 
 6. **Error types use `thiserror` with descriptive names** (e.g., `StorageError`,
    not bare `Error`).
-   - ✅ RESOLVED (strategy): `snafu` is gone from source
-     (`rg -l snafu crates/ rustfs/` is empty) and library code no longer uses
-     `anyhow` (remaining hits are test code and the `e2e_test` crate; `heal`
-     uses `thiserror`).
-   - ⚠️ VIOLATED (naming): 6 crates still export a bare `pub enum Error`:
-     `crypto`, `filemeta`, `heal`, `iam`, `policy`, and `replication`
-     (`src/resync.rs`) — all `thiserror`-derived.
+   - ⚠️ VIOLATED: 6 crates use `pub enum Error`; 2 crates use `snafu`;
+      `heal` use `anyhow` in library code.
 
 ## Known Structural Issues
 
@@ -168,25 +140,13 @@ default build (lifecycle:
 
 ### Critical
 
-- **scanner/data-usage duplicate `.usage-cache.bin` serialization types.** The
-  original finding ("common/scanner code duplication, ~3K lines") is resolved:
-  `scanner` imports the shared data-usage types from `rustfs-data-usage` (see
-  the `pub use rustfs_data_usage::…` re-exports at the top of
-  `crates/scanner/src/data_usage_define.rs`). What remains: `scanner` and
-  `data-usage` each hold their own serialization types for the scanner cache
-  file (`DataUsageCacheInfo`/`DataUsageEntryInfo` in
-  `crates/scanner/src/data_usage_define.rs` vs
-  `DataUsageCacheInfo`/`DataUsageEntry` in
-  `crates/data-usage/src/data_usage.rs`); convergence is tracked in
-  rustfs/backlog#1828.
+- **common/scanner code duplication (~3K lines).** `scanner` depends on `common`
+  but maintains its own copies of `DataUsageInfo`, `LastMinuteLatency`, and related
+  types instead of importing them.
 
-- **ecstore is a monolith (265 files, ~288K lines — roughly half is inline
-  `#[cfg(test)]` code).** Measured with
-  `find crates/ecstore/src -name '*.rs' | xargs wc -l`. It contains disk
-  management, bucket management, erasure coding, replication, lifecycle, RPC,
-  and configuration — all in one crate. It should be decomposed along its
-  existing subdirectories; the split plan lives in
-  [docs/architecture/ecstore-module-split-plan.md](docs/architecture/ecstore-module-split-plan.md).
+- **ecstore is a monolith (87K lines, 163 files).** It contains disk management,
+  bucket management, erasure coding, replication, lifecycle, RPC, and configuration
+  — all in one crate. It should be decomposed along its existing subdirectories.
 
 ### High
 
@@ -194,26 +154,19 @@ default build (lifecycle:
   `common → filemeta/madmin` edges must stay removed so leaf/helper crates do
   not regain upward dependencies.
 
-- **Three-layer backpressure/deadlock policy bridging** across io-core,
-  concurrency, and `rustfs/src/storage`. The config types are no longer
-  duplicated (`BackpressureConfig` and `DeadlockDetectorConfig` are each
-  defined once, in io-core). Storage policies expose and consume explicit
-  projections into the concurrency/io-core policy shapes, and workload
+- **Three-layer BackpressureConfig/DeadlockConfig duplication** across io-core,
+  concurrency, and `rustfs/src/storage`. Storage policies now expose and consume
+  explicit projections into the concurrency/io-core policy shapes, and workload
   admission snapshots are composed through provider registries; later work
   should use those bridges before deleting compatibility wrappers.
 
 ### Medium
 
-- **Bare `Error` naming.** Error-handling strategy has converged on `thiserror`
-  (no `snafu`, no `anyhow` in library code); the remaining inconsistency is the
-  bare `pub enum Error` naming in the 6 crates listed under Invariant 6.
+- **Inconsistent error handling.** Three strategies (thiserror/snafu/anyhow) and
+  mixed naming (bare `Error` vs descriptive names).
 
-- **`common` is mostly parked domain code, not shared utilities.** Of its
-  6,724 lines, ~83% is scanner/heal domain code stranded there to break
-  dependency cycles (`metrics.rs`, ~4,810 lines of scanner-domain metrics;
-  `heal_channel.rs`, ~776 lines of heal-domain channel types). The
-  "common vs utils" naming ambiguity is secondary to moving that code to its
-  domain owners.
+- **Ambiguous common vs utils boundary.** Both described as "utilities and data
+  structures." Need clear ownership rules.
 
 ## Cross-Cutting Concerns
 
@@ -279,7 +232,7 @@ The binary (`main.rs`) boots in this order:
 
 ```
                             ┌─────────┐
-                            │  rustfs │  (binary + lib)
+                            │  rustfs │  (binary + lib, 75K lines)
                             │  main   │
                             └────┬────┘
                                  │
@@ -302,7 +255,7 @@ The binary (`main.rs`) boots in this order:
               │                  │                  │
         ┌─────▼──────┐    ┌──────▼──────┐    ┌──────▼──────┐
         │  ecstore   │    │     rio     │    │   io-core   │
-        │   (core)   │    │  (readers)  │    │ (buffers)   │
+        │ (87K,core) │    │  (readers)  │    │ (zero-copy) │
         └─────┬──────┘    └─────────────┘    └─────────────┘
               │
      ┌─────┬──┼──┬─────┬──────┐
@@ -314,7 +267,7 @@ The binary (`main.rs`) boots in this order:
 
 - **"Where does S3 PutObject go?"**
   `server/` routes → `app/object_usecase` validates → `storage/ecfs` encodes →
-  `ecstore` distributes → `rio` encrypts/compresses → `io-core` supplies buffers
+  `ecstore` distributes → `rio` encrypts/compresses → `io-core` writes
 
 - **"Where are bucket policies enforced?"**
   `app/bucket_usecase` calls into `crates/policy/`

CONTRIBUTING.md

@@ -91,9 +91,8 @@ A green `make pre-commit` is not enough to open a pull request.
 `make pre-pr` is the **full** gate: it runs all of the guard checks above,
 then `clippy-check` (`cargo clippy --all-targets --all-features -- -D warnings`)
 and `test` (shell script tests, workspace tests excluding `e2e_test`, and doc
-tests). Complete the applicable multi-role adversarial review described in
-`AGENTS.md` before running `make pre-pr`; then run the gate before opening or
-updating a pull request. This is what CI enforces.
+tests). Run `make pre-pr` before opening or updating a pull request — this is
+what CI enforces.
 
 ### 🔒 Git Pre-commit Hooks (optional)
 
@@ -151,9 +150,8 @@ Example output when formatting fails:
 2. **Format your code**: `make fmt` or `cargo fmt --all`
 3. **Run the fast gate**: `make pre-commit` (no clippy, no tests)
 4. **Commit your changes**: `git commit -m "your message"`
-5. **Complete the applicable multi-role adversarial review** for non-exempt changes (see `AGENTS.md`)
-6. **Run the full gate before opening/updating a PR**: `make pre-pr` (clippy + tests)
-7. **Push to your branch**: `git push`
+5. **Run the full gate before opening/updating a PR**: `make pre-pr` (clippy + tests)
+6. **Push to your branch**: `git push`
 
 ### 🛠️ IDE Integration
 

Cargo.toml

@@ -41,7 +41,7 @@ members = [
     "crates/protocols", # Protocol implementations (FTPS, SFTP, etc.)
     "crates/protos", # Protocol buffer definitions
     "crates/rio", # Rust I/O utilities and abstractions
-    "crates/rio-v2", # MinIO on-disk format compatibility I/O layer (feature-gated, ships in no default build)
+    "crates/rio-v2", # Next-generation Rust I/O compatibility layer
     "crates/replication", # Replication contracts and wire formats
     "crates/concurrency", # Concurrency management for RustFS - timeout, locking, backpressure, and I/O scheduling
     "crates/s3-types", # S3 event type definitions
@@ -69,7 +69,7 @@ edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
 rust-version = "1.97.1"
-version = "1.0.0-rc.3"
+version = "1.0.0-beta.12"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -86,52 +86,52 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "1.0.0-rc.3" }
-rustfs-heal = { path = "crates/heal", version = "1.0.0-rc.3" }
-rustfs-audit = { path = "crates/audit", version = "1.0.0-rc.3" }
-rustfs-checksums = { path = "crates/checksums", version = "1.0.0-rc.3" }
-rustfs-common = { path = "crates/common", version = "1.0.0-rc.3" }
-rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-rc.3" }
-rustfs-config = { path = "./crates/config", version = "1.0.0-rc.3" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-rc.3" }
-rustfs-credentials = { path = "crates/credentials", version = "1.0.0-rc.3" }
-rustfs-crypto = { path = "crates/crypto", version = "1.0.0-rc.3" }
-rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-rc.3" }
-rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-rc.3" }
-rustfs-iam = { path = "crates/iam", version = "1.0.0-rc.3" }
-rustfs-keystone = { path = "crates/keystone", version = "1.0.0-rc.3" }
-rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-rc.3" }
-rustfs-kms = { path = "crates/kms", version = "1.0.0-rc.3" }
-rustfs-lock = { path = "crates/lock", version = "1.0.0-rc.3" }
-rustfs-madmin = { path = "crates/madmin", version = "1.0.0-rc.3" }
-rustfs-notify = { path = "crates/notify", version = "1.0.0-rc.3" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-rc.3" }
-rustfs-io-core = { path = "crates/io-core", version = "1.0.0-rc.3" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-rc.3" }
-rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-rc.3", default-features = false }
-rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.0-rc.3" }
-rustfs-obs = { path = "crates/obs", version = "1.0.0-rc.3" }
-rustfs-policy = { path = "crates/policy", version = "1.0.0-rc.3" }
-rustfs-protos = { path = "crates/protos", version = "1.0.0-rc.3" }
-rustfs-protocols = { path = "crates/protocols", version = "1.0.0-rc.3" }
-rustfs-replication = { path = "crates/replication", version = "1.0.0-rc.3" }
-rustfs-rio = { path = "crates/rio", version = "1.0.0-rc.3" }
-rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-rc.3" }
-rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-rc.3" }
-rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-rc.3" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-rc.3" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-rc.3" }
-rustfs-scanner = { path = "crates/scanner", version = "1.0.0-rc.3" }
-rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-rc.3" }
-rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-rc.3" }
-rustfs-signer = { path = "crates/signer", version = "1.0.0-rc.3" }
-rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-rc.3" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-rc.3" }
-rustfs-targets = { path = "crates/targets", version = "1.0.0-rc.3" }
-rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-rc.3" }
-rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-rc.3" }
-rustfs-utils = { path = "crates/utils", version = "1.0.0-rc.3" }
-rustfs-zip = { path = "./crates/zip", version = "1.0.0-rc.3" }
+rustfs = { path = "./rustfs", version = "1.0.0-beta.12" }
+rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.12" }
+rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.12" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.12" }
+rustfs-common = { path = "crates/common", version = "1.0.0-beta.12" }
+rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.12" }
+rustfs-config = { path = "./crates/config", version = "1.0.0-beta.12" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.12" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.12" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.12" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.12" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.12" }
+rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.12" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.12" }
+rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-beta.12" }
+rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.12" }
+rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.12" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.12" }
+rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.12" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.12" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.12" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.12" }
+rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-beta.12" }
+rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.0-beta.12" }
+rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.12" }
+rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.12" }
+rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.12" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.12" }
+rustfs-replication = { path = "crates/replication", version = "1.0.0-beta.12" }
+rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.12" }
+rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-beta.12" }
+rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.12" }
+rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.12" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.12" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.12" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.12" }
+rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-beta.12" }
+rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-beta.12" }
+rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.12" }
+rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-beta.12" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.12" }
+rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.12" }
+rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-beta.12" }
+rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.12" }
+rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.12" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.12" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
@@ -139,13 +139,13 @@ async_zip = { default-features = false, version = "0.0.18" }
 mysql_async = { default-features = false, version = "0.37" }
 async-compression = { version = "0.4.43" }
 async-recursion = "1.1.1"
-async-trait = "0.1.92"
+async-trait = "0.1.91"
 async-nats = { version = "0.50.0", default-features = false }
 axum = "0.8.9"
-futures = "0.3.34"
-futures-core = "0.3.34"
+futures = "0.3.33"
+futures-core = "0.3.33"
 futures-lite = "2.6.1"
-futures-util = "0.3.34"
+futures-util = "0.3.33"
 pollster = "1.0.1"
 pulsar = { default-features = false, version = "6.8.0" }
 lapin = { default-features = false, version = "4.10.0" }
@@ -154,7 +154,7 @@ hyper-rustls = { default-features = false, version = "0.27.9" }
 hyper-util = { version = "0.1.20" }
 http = "1.5.0"
 http-body = "1.1.0"
-http-body-util = "0.1.5"
+http-body-util = "0.1.4"
 minlz = "1.2.3"
 reqwest = "0.13.4"
 rustfs-kafka-async = { version = "1.2.0" }
@@ -171,7 +171,7 @@ tower = { version = "0.5.3" }
 tower-http = { version = "0.7.0" }
 
 # Serialization and Data Formats
-apache-avro = { version = "0.22.0", features = ["snappy", "zstandard"] }
+apache-avro = "0.21.0"
 bytes = { version = "1.12.1" }
 bytesize = "2.7.0"
 byteorder = "1.5.0"
@@ -182,7 +182,6 @@ quick-xml = "0.41.0"
 rmp = { version = "0.8.15" }
 rmp-serde = { version = "1.3.1" }
 serde = { version = "1.0.229" }
-serde_ignored = { version = "0.1" }
 serde_json = { version = "1.0.151" }
 serde_urlencoded = "0.7.1"
 
@@ -204,7 +203,6 @@ rsa = { version = "=0.10.0-rc.18" }
 rustls = { default-features = false, version = "0.23.43" }
 rustls-native-certs = "0.8"
 rustls-pki-types = "1.15.1"
-x509-parser = "0.18.1"
 sha1 = "0.11.0"
 sha2 = "0.11.0"
 subtle = "2.6"
@@ -229,16 +227,16 @@ atoi = "3.1.0"
 atomic_enum = "0.3.0"
 aws-config = { version = "1.10.1" }
 aws-credential-types = { version = "1.3.0" }
-aws-sdk-kms = { default-features = false, version = "1.115.0" }
-aws-sdk-s3 = { default-features = false, version = "1.142.0" }
-aws-sdk-sts = { default-features = false, version = "1.111.0" }
-aws-smithy-http-client = { default-features = false, version = "1.4.0" }
-aws-smithy-runtime-api = { version = "1.15.0" }
-aws-smithy-types = { version = "1.6.2" }
-base64 = "0.23.1"
+aws-sdk-kms = { default-features = false, version = "1.114.0" }
+aws-sdk-s3 = { default-features = false, version = "1.140.0" }
+aws-sdk-sts = { default-features = false, version = "1.110.0" }
+aws-smithy-http-client = { default-features = false, version = "1.2.0" }
+aws-smithy-runtime-api = { version = "1.14.0" }
+aws-smithy-types = { version = "1.6.1" }
+base64 = "0.23.0"
 base64-simd = "0.8.0"
 brotli = "8.0.4"
-clap = { version = "4.6.6" }
+clap = { version = "4.6.5" }
 const-str = { version = "1.1.0" }
 convert_case = "0.11.0"
 criterion = { version = "0.8" }
@@ -246,7 +244,8 @@ crossbeam-queue = "0.3.13"
 crossbeam-channel = "0.5.16"
 crossbeam-deque = "0.8.7"
 crossbeam-utils = "0.8.22"
-datafusion = { default-features = false, version = "55.0.0" }
+datafusion = { default-features = false, git = "https://github.com/apache/datafusion.git", rev = "dae03ee062b2abf986de8df12ea82fb1578a2d99" }
+#datafusion = { default-features = false, version = "54.1.0" }
 derive_builder = "0.20.2"
 enumset = "1.1.14"
 faster-hex = "0.10.0"
@@ -264,13 +263,12 @@ lazy_static = "1.5.0"
 libc = "0.2.189"
 libsystemd = "0.7.2"
 local-ip-address = "0.6.13"
-log = "0.4"
 memmap2 = "0.9.11"
 lz4 = "1.28.1"
 matchit = "0.9.2"
 md-5 = "0.11.0"
 mime_guess = "2.0.5"
-moka = { version = "0.12.16" }
+moka = { version = "0.12.15" }
 netif = "0.1.6"
 num_cpus = { version = "1.17.0" }
 nvml-wrapper = "0.12.1"
@@ -280,23 +278,23 @@ percent-encoding = "2.3.2"
 pin-project-lite = "0.2.17"
 pretty_assertions = "1.4.1"
 rand = { version = "0.10.2" }
-ratelimit = "2.0.0"
+ratelimit = "0.10.1"
 rayon = "1.12.0"
 reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "8.0.2" }
 reed-solomon-simd = "3.1.0"
 regex = { version = "1.13.1" }
-rumqttc = { package = "rumqttc-next", version = "0.34.0" }
-redis = { version = "1.6.0" }
+rumqttc = { package = "rumqttc-next", version = "0.33.3" }
+redis = { version = "1.5.0" }
 rustify = { version = "0.7", default-features = false }
 rustix = { version = "1.1.4" }
 rust-embed = { version = "8.12.0" }
 rustc-hash = { version = "2.1.3" }
-s3s = { git = "https://github.com/rustfs/s3s.git", rev = "ed70cb048cc4be168419d461cb9ac3c2c7fa6d5a" }
+s3s = { git = "https://github.com/cxymds/s3s.git", rev = "fe3941d91fa1c69956f209a9145995c9f0235bff" }
 serial_test = "4.0.1"
 shadow-rs = { default-features = false, version = "2.0.0" }
 siphasher = "1.0.3"
 smallvec = { version = "1.15.2" }
-compact_str = "0.10.0"
+smartstring = "1.0.1"
 snap = "1.1.2"
 starshard = { version = "2.2.2" }
 strum = { version = "0.28.0" }
@@ -304,7 +302,7 @@ sysinfo = "0.39.6"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
 test-case = "3.3.1"
-thiserror = "2.0.20"
+thiserror = "2.0.19"
 tracing = { version = "0.1.44" }
 tracing-appender = "0.2.5"
 tracing-core = "0.1.36"
@@ -314,7 +312,7 @@ tracing-subscriber = { version = "0.3.23" }
 transform-stream = "0.3.1"
 url = "2.5.8"
 urlencoding = "2.1.3"
-uuid = { version = "1.24.1" }
+uuid = { version = "1.24.0" }
 vaultrs = { version = "0.8.0" }
 tar = "0.4.46"
 walkdir = "2.5.0"
@@ -327,7 +325,6 @@ zstd = "0.13.3"
 
 # Observability and Metrics
 metrics = "0.24.6"
-metrics-util = "0.20"
 dial9-tokio-telemetry = "0.3"
 opentelemetry = { version = "0.32.0" }
 opentelemetry-appender-tracing = { version = "0.32.0" }
@@ -341,23 +338,23 @@ pyroscope = { version = "2.1.1" }
 # FTP and SFTP
 libunftp = { version = "0.23.0" }
 unftp-core = "0.1.0"
-suppaftp = { version = "10.0.2" }
-rcgen = { version = "0.14.9", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
-russh = { version = "0.62.7" }
-russh-sftp = "2.4.0"
+suppaftp = { version = "10.0.1" }
+rcgen = { version = "0.14.8", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
+russh = { version = "0.62.5" }
+russh-sftp = "2.3.0"
 
 # WebDAV
 dav-server = "0.11.0"
 
 # Performance Analysis and Memory Profiling
-mimalloc = { version = "0.1.52", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11" }
-libmimalloc-sys = { version = "0.1.49", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11", features = ["extended"] }
-hotpath = { version = "0.23.3", default-features = false }
+mimalloc = { version = "0.1.52", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "1cdadea43e9c5a0f054b65be21200ce580e4eb13" }
+libmimalloc-sys = { version = "0.1.49", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "1cdadea43e9c5a0f054b65be21200ce580e4eb13", features = ["extended"] }
+hotpath = { version = "0.23.0", default-features = false }
 # Snapshot testing for output format regression detection
 insta = { version = "1.48" }
 
 [workspace.metadata.cargo-shear]
-ignored = ["hotpath", "rustfs"]
+ignored = ["rustfs"]
 
 [profile.dev]
 # Full debuginfo roughly doubles compile+link time and produces multi-GB

Dockerfile

@@ -91,12 +91,7 @@ LABEL name="RustFS" \
 # Upgrade base-image packages so published images pick up security fixes
 # (e.g. openssl/libssl3 CVEs) without waiting for a new Alpine point release.
 RUN apk upgrade --no-cache && \
-    apk add --no-cache \
-    ca-certificates \
-    coreutils \
-    curl \
-    tzdata \
-    && test "$(TZ=Asia/Kolkata date +%z)" = "+0530"
+    apk add --no-cache ca-certificates coreutils curl
 
 COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
 COPY --from=build /build/rustfs /usr/bin/rustfs

Dockerfile.glibc

@@ -96,11 +96,9 @@ LABEL name="RustFS" \
 # Upgrade base-image packages so published images pick up security fixes
 # (e.g. tar/gzip/perl CVEs) without waiting for a new Ubuntu point release.
 RUN apt-get update && apt-get upgrade -y \
-    && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+    && apt-get install -y --no-install-recommends \
     ca-certificates \
     curl \
-    tzdata \
-    && test "$(TZ=Asia/Kolkata date +%z)" = "+0530" \
     && rm -rf /var/lib/apt/lists/*
 
 COPY --from=build /build/rustfs /usr/bin/rustfs

MINIO_REPLICATION_COMPAT_REVIEW.md

@@ -0,0 +1,314 @@
+# RustFS 站点复制 / 桶复制 — MinIO 兼容性审查报告
+
+> 审查日期:2026-08-05
+> 审查对象:RustFS(worktree `reatang/minio-compatibility-review-03a7fb`)vs MinIO(`/Users/tang/Documents/GitHub/minio`)
+> 审查方式:白盒代码对比(5 个维度并行审查)+ P0 问题对抗性复核
+> 审查维度:站点复制白盒对比、桶复制白盒对比、mc 工具兼容性、S3 标准协议兼容性、代码结构与分层
+
+---
+
+## 一、总体结论
+
+| 领域 | 兼容性评价 |
+|---|---|
+| **站点复制(RustFS↔RustFS + mc 管理)** | 良好。admin 端点全覆盖、JSON 结构对齐 madmin-go、请求体 DARE 加密兼容,mc admin replicate 全家桶基本可用 |
+| **站点复制(RustFS↔MinIO 混合组网)** | **断裂**。4 个 P0:出站 join 路径 404、metainfo 大小写解析失败、STS item 类型名不一致、policy-mapping userType 数值错位 |
+| **桶复制(控制面,S3 标准 API)** | 良好。Put/Get/DeleteBucketReplication、错误码、状态机字符串、xl.meta 内部键均对齐 |
+| **桶复制(数据面,RustFS→MinIO)** | **断裂**。复制 PUT 缺 `?versionId=` 导致目标端版本漂移(P0);CopyObject 完全不复制(P0) |
+| **mc 桶复制命令** | **部分断裂**。`mc replicate add` 默认参数即失败(P0);status/resync/backlog 响应结构不匹配导致静默空输出(P1) |
+| **代码结构** | 桶复制侧迁移架构有纪律但成本高;**站点复制侧无领域层,约 9500 行业务逻辑堆在 admin handler,且存在 3 处反向依赖违反项目分层不变量(P0)** |
+
+**做得好的地方**(已确认兼容,无需整改):复制状态机字符串(PENDING/COMPLETED/FAILED/REPLICA 含 legacy COMPLETE)、xl.meta 内部键双前缀(x-rustfs-internal- + x-minio-internal-)读写、ReplicateDecision 内部状态串格式、复制内部头主链路双前缀、Delete/VersionPurge 语义、Resync reset-id 判定、admin 路由 `/minio/admin/v3` 前缀别名、madmin DARE 加密流解密、站点复制 gob netperf 编码、`site-repl-<deploymentID>` 规则模板。
+
+---
+
+## 二、P0 问题清单(8 项)
+
+| # | 问题 | 来源维度 | 断裂方向 |
+|---|---|---|---|
+| P0-1 | 出站 peer join 使用 MinIO 已移除的遗留路径 `/site-replication/join` → 404 | 站点复制 | RustFS→MinIO |
+| P0-2 | 解析 MinIO metainfo(SRInfo)字段大小写不匹配 → add preflight 失败 | 站点复制 | RustFS→MinIO |
+| P0-3 | STS 凭证复制 item 类型名 `sts-credential` vs `sts-account` | 站点复制 | 双向 |
+| P0-4 | policy-mapping `userType` 数值语义错位(RustFS: None=0/Svc=1/Sts=2/Reg=3;MinIO: reg=0/sts=1/svc=2)→ 权限静默漂移 | 站点复制 | 双向 |
+| P0-5 | 复制 PUT/CompleteMultipart 不携带 `?versionId=` query → MinIO 端版本号漂移、版本删除永久 no-op、双端静默发散(**功能视角复核:定级调整为 P1**,问题重述为"普通复制对象缺少可靠的源→目标版本身份策略";versionId query 是可行修复之一而非唯一正确方案) | 桶复制 | RustFS→MinIO |
+| P0-6 | CopyObject(含 metadata-replace 自拷贝)完全不触发复制调度,对象静默不复制(**功能视角复核:定级调整为 P1**;scanner 在 ExistingObjectReplication 启用+状态为空时可最终补齐,但同步复制语义失效,且继承 stale COMPLETED / 显式 Disabled 场景长期漏复制) | 桶复制 + S3 协议 | 所有方向 |
+| P0-7 | `mc replicate add` 默认参数(healthcheck-seconds=60)被硬拒 400;且字段单位按秒解析而 wire 为纳秒 | mc 兼容 | mc→RustFS |
+| P0-8 | 架构:站点复制约 9500 行业务逻辑堆在 admin handler 单文件;app/storage 层 3 处反向导入 admin 层,违反 ARCHITECTURE.md 分层不变量 #1(**对抗复核后降级为 P1**:反向边已被 arch 守卫棘轮基线锁死,属受控技术债) | 代码结构 | — |
+
+每项 P0 的对抗性复核结论、验证方案与解决方案见 **第五节**。
+
+**修复状态(2026-08-05)**:7 项确认 P0 已全部修复并创建 PR(红灯→绿灯 TDD):P0-1 [#5748](https://github.com/rustfs/rustfs/pull/5748)、P0-2 [#5749](https://github.com/rustfs/rustfs/pull/5749)、P0-3 [#5750](https://github.com/rustfs/rustfs/pull/5750)、P0-4 [#5751](https://github.com/rustfs/rustfs/pull/5751)、P0-5 [#5752](https://github.com/rustfs/rustfs/pull/5752)、P0-6+P1-10 [#5753](https://github.com/rustfs/rustfs/pull/5753)、P0-7 [#5754](https://github.com/rustfs/rustfs/pull/5754)。合并顺序:#5748+#5749 同批;#5752 先于 #5753。
+
+---
+
+## 三、P1 问题清单
+
+### 站点复制
+
+| # | 问题 | 证据 | 影响 |
+|---|---|---|---|
+| P1-1 | ILM(lc-config)复制语义:对外开关限定 `replicateILMExpiry`,但发送端把**完整** lifecycle.xml 放入 `expiry_lc_config`,接收端整体覆盖/删除本地配置(功能视角复核:**确认,维持 P1**;更新时间检查只能拒旧,不能修复整体覆盖语义) | RustFS `bucket_meta.rs:948-951`、`site_replication.rs:7590-7683` vs MinIO `site-replication.go:1784-1810,6138` | lifecycle 同时含 expiry 与本地 transition 时,非 expiry 规则被错误传播或本地 transition 被覆盖。**缺"同步 expiry 后保留本地 transition"测试** |
+| ~~P1-2~~→**P2-25** | `SRInfo.ilmExpiryRules` 从不填充,ILM 一致性状态恒为空(功能视角复核:**降级 P2**——仅影响管理面可观测性,不改变对象数据) | `site_replication.rs:4152-4266,4855-4868` | `mc admin replicate status --ilm-expiry-rules` 恒空,ILM 漂移不可见 |
+| P1-3 | 无自动跨站元数据 heal(MinIO 有周期 heal 协程) | RustFS 仅 600s 本地 wiring 修复(`site_replication_reconcile.rs:34,59-81`)+ 手动 repair 端点 vs MinIO `site-replication.go:4257-4288` | 错过的 IAM/bucket 元数据更新持续漂移,须手工 repair |
+| P1-4(拆分) | ①`sync` 同步复制指控:功能视角复核**不成立/证据不足**——RustFS 自身契约明确将 `sync_state` 定义为站点可达性/配置完整性健康状态且有测试,不能以他家同名字段判其错误(属"RustFS 独特设计保持不变"项,撤销);②`defaultbandwidth`:**确认,降级 P2**——公共 API 接受并持久化,但建 site replication bucket target 时不应用,reconcile 只保留既有 `bandwidth_limit`,配置成功但不生效 | `site_replication.rs:6303-6357,5004-5027` | ②为用户可见的"配置成功但无效"能力缺口 |
+
+### 桶复制 / S3 协议
+
+| # | 问题 | 证据 | 影响 |
+|---|---|---|---|
+| P1-5 | 未复制完成对象的 GET/HEAD 远端 proxy 未实现;也不识别 MinIO 的 `X-Minio-Source-Proxy-Request` 防环头 | 仅指标占位(`storage_api.rs:799-804`);`SUFFIX_SOURCE_PROXY_REQUEST` 定义后无人使用 vs MinIO `bucket-replication.go:2334,2409,2534` | active-active 复制滞后窗口内 RustFS 端 404 |
+| P1-6 | `X-Minio-Source-Replication-{Tagging,Retention,LegalHold}-Timestamp` 三个时间戳头收发均缺失 | `replication_target_boundary.rs:251-297` 填了 options 但 `PutObjectOptions::header()` 不序列化;接收端不解析 vs MinIO `object-api-options.go:377-399` | active-active 下标签/retention/legal-hold 并发修改的 LWW 冲突解析退化,可能元数据回滚 |
+| P1-7 | ARN 前缀 `arn:rustfs:` 与 `arn:minio:` 不互认(解析侧强制 `arn:rustfs:`) | `crates/ecstore/src/bucket/target/arn.rs:43,51` vs MinIO `bucket-targets.go:709` | 存量 MinIO 复制配置迁移被 StaleTarget 拒;原生 madmin SDK 解析 RustFS ARN 失败 |
+| P1-8 | PutBucketReplication 校验缺口(规则数/Priority 唯一/ID 长度/Filter 互斥/2MB 上限全缺)+ 主动拒绝 `Destination.StorageClass` 等 MinIO/AWS 合法字段 | `bucket_usecase.rs:582-616`、`config.rs:143-232` vs MinIO `internal/bucket/replication/replication.go:29-90` | 非法配置被接受、优先级冲突行为不可预测;存量 AWS/Terraform 配置(含 StorageClass)直接 400 |
+| ~~P1-9~~→**P2-26** | GetObject 响应缺 `x-amz-replication-status` 头(HEAD 有 GET 无),且 GET 专门把它从 metadata 过滤掉(功能视角复核:**降级 P2**;GET/HEAD 不一致确认,缺 GET replication-status 回归测试) | `object_usecase.rs:5696-5735`、`options.rs:702` vs MinIO `api-headers.go:236-238` | 依赖 GET 判断复制状态的客户端/监控失效;修复约一行 |
+| P1-10 | Snowball auto-extract 解包对象不触发复制(功能视角复核:**确认,维持 P1**,但"全部永不复制"不准确——scanner 在状态空+ExistingObjectReplication 启用时可补齐;显式 Disabled 等场景长期遗漏,即时复制始终失效。带 REPLICA 状态的入站成员须继续避免回环)。**已随 [#5753](https://github.com/rustfs/rustfs/pull/5753) 修复**(含入站复制 PUT 不再被误派发 extract 的次生缺陷) | `object_usecase.rs:8201` vs MinIO `object-handlers.go:2452,2510-2511` | 批量导入对象不即时复制;缺普通解包成员复制结果的测试(已在 #5753 补充 e2e) |
+
+### mc 响应结构(静默空输出类)
+
+| # | 问题 | 证据 | 影响 |
+|---|---|---|---|
+| P1-11 | `?replication-metrics[=2]` 响应为 Rust snake_case,minio-go MetricsV2 期望 camelCase(`currStats`/`queueStats`/…) | `stats.rs:617-770`、`admin/router.rs:1583-1592` vs MinIO `bucket-stats.go:154-188` | `mc replicate status` 不报错但全零(静默错误) |
+| P1-12 | replication-reset(resync)响应壳不匹配:`{"Targets":[{"Arn","ResetID",...}]}` vs `{"target":[{"arn","resetid","resyncStatus",...}]}` | `router.rs:126-198,1735-1803` vs MinIO `bucket-replication-utils.go:613-636` | `mc replicate resync start/status` 输出空;仅响应壳问题,修复成本低 |
+| P1-13 | `/v3/replication/mrf` 与 `/v3/replication/diff` 返回单个聚合对象而非条目流(代码自述 deliberate) | `replication.rs:695-725,879-911,998-1047` vs madmin-go `replication-api.go:104-176` | `mc replicate backlog` 输出空;`node`/`arn`/`verbose` 参数被忽略 |
+| P1-14 | set-remote-target 请求体 `deny_unknown_fields` + 字段名偏差(期望 `bandwidth_limit`,madmin 发 `bandwidthlimit`;`session_token` vs `sessionToken` 等) | `handlers/replication.rs:88-95,108-163` vs madmin-go `bucket-targets.go:76` | `mc replicate add/update --bandwidth` 整请求失败;凡 omitempty 字段一旦出现即 400 |
+
+### 代码结构
+
+| # | 问题 | 证据 | 影响 |
+|---|---|---|---|
+| P1-15 | 站点复制状态两套归一化实现(handler 类型化 vs service 无类型 JSON),且 reload 的 read→normalize→save 全程无共同分布式对象锁,存在 lost-update 竞争;repair state 已用 `with_config_object_write_lock` 包住完整 RMW,主 state 未采用同等保护(功能视角复核:**确认,维持 P1**;进程内 `SITE_REPLICATION_STATE_LOCK` 与单次 read/save 各自的对象锁均不能保护跨调用 RMW:A 读旧→B 另节点写入→A 用旧快照覆盖,B 丢失) | `handlers/site_replication.rs:114,347,1039-1130` vs `service/site_replication.rs:26-135` | 归一化语义可 drift;多节点/RPC 并发写状态互相覆盖。**缺多节点/双写者 lost-update 回归测试** |
+| P1-16 | 复制状态机类型双份定义:`rustfs-filemeta` 与 `rustfs-replication` 各持一份(ReplicationStatusType/VersionPurgeStatusType/ReplicationState/MrfReplicateEntry/ReplicateObjectInfo),靠 boundary 双向转换 | `crates/filemeta/src/replication.rs` vs `crates/replication/src/filemeta.rs` | 状态机语义修改须同步两处+转换层,漏一处即静默数据语义错误;建议加 enum 对账测试 |
+| P1-17 | 桶复制逻辑分裂:`crates/replication` 仅契约,执行引擎(pool 5947 行、resyncer 4090 行)仍在 ecstore,中间 20+ 个 boundary/bridge 微文件;迁移无完成判据,脚手架有固化风险 | `crates/ecstore/src/bucket/replication/README.md`、`mod.rs:15-45` | 可读性/可维护性成本;需设定迁移里程碑 |
+| P1-18 | 超长函数集中在复制热路径:`resync_bucket` 536 行、`replicate_all` 403 行、`start_mrf_processor` 305 行、`apply_iam_item` 248 行 | `replication_resyncer.rs:546`、`replication_pool.rs`、`site_replication.rs:7806` | 正确性审查与修改风险高 |
+
+### 第三方复审新增与调整项(功能视角二次复核后)
+
+| # | 问题 | 来源 | 影响 |
+|---|---|---|---|
+| P1-19 | 普通复制对象缺少可靠的源→目标版本身份策略:PUT 响应的目标版本 ID 未捕获/持久化,对不支持 versionId query 的目标(原生 AWS S3 等),后续版本删除复制落空;MRF 只会重试同一个错误身份,HEAD ETag fallback 不能修复删除 | P0-5 复审 | 非 MinIO 系目标的版本化复制双端发散。缺"目标自行分配版本 ID"场景测试 |
+| P1-20 | 缺少 scanner 补偿边界的 e2e:ExistingObjectReplication Enabled/Disabled × 空状态/继承状态 组合下的补齐与不补齐行为无回归覆盖(Copy 与 Snowball 两路径) | P0-6/P1-10 复审 | scanner 兜底语义变化不可见 |
+| P1-21 | delete-marker 延迟 purge 失败静默丢弃(由 P2-20① 升级):目标删除失败无日志/状态/MRF,目标端 marker/版本可能永久残留 | P2-20 复核升级 | 数据一致性;缺失败注入测试 |
+| P1-22 | 桶复制整体 SSE 支持能力缺口(替代原 P2-23):SSE-S3/SSE-KMS 所有复制模式统一 fail closed,SSE-C 失败被 e2e 钉为当前行为,无 encrypted-object resync e2e | P2-23 复核改写 | 加密对象跨站不复制;需覆盖普通复制/Heal/Resync/Multipart 四模式 |
+
+### 功能视角二次复核采纳记录(backlog#1675,基于 main f0c4fbd28)
+
+复核共 10 项,判定依据为 RustFS 自身功能契约与实际调用链,不以对齐 MinIO 为正确性标准。采纳结果:
+
+| 原编号 | 复核结论 | 采纳动作 |
+|---|---|---|
+| P0-5 | 确认,P0→P1,问题重述为"源→目标版本身份策略缺失" | 定级调整;修复已合 [#5752](https://github.com/rustfs/rustfs/pull/5752);残留缺口 P1-19 |
+| P0-6 | 确认,P0→P1,scanner 描述纠正 | 定级调整;修复已合 [#5753](https://github.com/rustfs/rustfs/pull/5753);测试缺口 P1-20 |
+| P1-1 | 确认,维持 P1 | 补记"expiry 同步后保留本地 transition"测试缺口 |
+| P1-2 | 确认,P1→P2(仅管理面可观测性) | 改编号 P2-25 |
+| P1-4 | 拆分:`sync` 指控不成立(RustFS 自身契约定义为健康状态,有测试);`defaultbandwidth` 确认为 P2 能力缺口 | `sync` 撤销并归入"独特设计保持不变";`defaultbandwidth` 降 P2 |
+| P1-9 | 确认,P1→P2 | 改编号 P2-26;补记缺 GET 回归测试 |
+| P1-10 | 确认,维持 P1,"全部永不复制"改为"即时复制失效+部分场景长期遗漏" | 已随 [#5753](https://github.com/rustfs/rustfs/pull/5753) 修复(含回环防护) |
+| P1-15 | 确认,维持 P1(竞争机理精确化:跨调用 RMW 无共同分布式锁) | 补记缺双写者 lost-update 测试 |
+| P2-20① | 确认,P2→P1(延迟 purge 失败静默丢弃部分) | 升级为 P1-21;②③维持 P2 |
+| P2-23 | resync 专属指控不成立;暴露桶复制整体 SSE 能力缺口 | 撤销原表述,改立 P1-22 |
+
+**复核指出的测试补齐清单**(均未运行跨实例集成验证,需落地):目标自行分配版本 ID、Copy/Snowball scanner 补偿边界、lifecycle expiry/transition 保留、site state 双写竞争、delayed purge 失败注入、encrypted-object resync。
+
+---
+
+## 四、P2 问题清单
+
+### 站点复制
+- **P2-1** `showDeleted` 选项与 `bucketDeletedTimestamp` 未实现(`site_replication.rs:1364-1381`)
+- **P2-2** 错误码泛化:统一 `InvalidRequest`/`InternalError`,无 MinIO 的 9 个 `XMinioSiteReplication*` 专用码(400/503 语义丢失)
+- **P2-3** `make-with-versioning` 忽略 `versioningEnabled`/`forceCreate` 参数,恒 true(`site_replication.rs:8597-8627`)
+- **P2-4** netperf 返回"不支持"占位(gob 格式兼容不会崩);devnull 有请求体大小上限(MinIO 无限 discard)
+- **P2-5** Metrics 摘要仅含本站,无 per-peer 链路统计(downtime/latency/失败窗口)
+- **P2-6** `external-user`/`credential` IAM item 未实现——与本仓 MinIO 版本等价缺失,结构已预留;对接新版 MinIO 时会成缺口
+- **P2-7** 本地 deploymentID 缺失时回退 endpoint 哈希(16 位 hex,非 UUID 形态)
+
+### 桶复制 / S3 协议
+- **P2-8** 遗留内部 client 头名错误:`X-Source-DeleteMarker`/`X-Check-Replication-Ready` 缺 `X-Minio-` 前缀(`client/api_stat.rs:191-231`,当前路径未激活,潜伏缺陷)
+- **P2-9** Remote target admin 错误码扁平化(MinIO 有 404/503 专用码,RustFS 统一 400/500)
+- **P2-10** Remote target 拒绝 `disableProxy`/`edge`/`edgeSyncBeforeExpiry` 等 madmin 字段(非默认参数,影响小)
+- **P2-11** `list-remote-targets` 序列化偏差:`bandwidth_limit`/`storage_class`/`deployment_id`/`reset_id`/`session_token` vs madmin 的 `bandwidthlimit`/`storageclass`/`deploymentID`/`resetID`/`sessionToken`;`healthCheckDuration`/`totalDowntime` 按秒序列化而 Go 按纳秒解;`type` 过滤参数被忽略
+- **P2-12** set-remote-target?update=true 忽略 madmin 的 op 标志(creds/sync/proxy/…),固定整体覆盖
+- **P2-13** XML 反序列化:Rule 内未知元素严格报 MalformedXML(顶层却跳过,行为不一致);缺 `<Role>` 报 MalformedXML(Go 容忍)——向前兼容性差,当前主流客户端不受影响
+- **P2-14** `ReplicaModifications` 默认 Disabled(与 AWS 一致、与 MinIO 的注入 Enabled 分歧);PUT 时不像 MinIO 那样注入默认元素回写
+- **P2-15** PutBucketReplication 要求预先注册 remote target(与 MinIO 同构、与纯 AWS 流程分歧),报错未指引先建 target
+- **P2-16** GetBucketReplication 响应无 xmlns(与 MinIO 一致,极少数严格 SDK 可能拒收)
+- **P2-17** 站点复制启用时不阻止普通用户直接改桶复制配置(MinIO 非 root 报 `ErrReplicationDenyEditError`)
+- **P2-18** Prometheus 指标名对齐 metrics-v3 但注册前缀为 rustfs 体系;versioning 错误文案与 MinIO 不同(code 一致)
+
+### 代码结构
+- **P2-19** `apply_iam_item` / bucket-ops 用裸字符串 match 分发,无法穷尽检查;建议改 `#[serde(tag)]` 枚举
+- **P2-20(拆分)** 静默吞错:①`replication_resyncer.rs:1693` delete-marker 延迟 purge 失败被 `let _ =` 丢弃,target client 缺失时直接跳过——**功能视角复核:升级为 P1-21**(失败后无日志、无状态更新、不入 MRF,目标 delete marker/版本可能永久残留;启动前的 5 次循环只是等源 marker 消失,不是对目标删除失败的重试。缺注入目标删除失败并验证重试/状态/MRF 的测试);②`site_replication.rs:8661` purge-deleted-bucket 吞掉非 NotFound 错误、`:9227` cancel resync 失败无痕迹——维持 P2
+- **P2-21** `MrfV2` 全套机制(Error/Capabilities/Readiness/Reader/Envelope)未接线,生产只用 v1,属投机代码
+- **P2-22** `persist_site_replication_state` 双重 clone + 双重 normalize(`site_replication.rs:1143-1152` → `:1116-1122`)
+- **P2-23(撤销并改写)** 原"resync 不处理 SSE"指控不成立——`ReplicationType::Resync` 与普通复制/Heal 最终走同一 `replication_put_object_options`,`// TODO: SSE` 不构成 resync 独立行为差异。真实状态:SSE-S3/SSE-KMS 在**所有复制模式**下统一 fail closed,SSE-C 普通桶复制失败已被现有 e2e 钉为当前行为,且无 encrypted-object resync e2e → 改立能力项 **P1-22"桶复制整体 SSE 支持"**(需分别覆盖普通复制、Heal、手动 Resync、Multipart)
+- **P2-24** `crates/replication` 命名误导(名为复制引擎实为契约库),建议 lib.rs 顶部文档说明
+- 正面确认:生产代码 unwrap/expect 纪律良好(几乎全在测试模块);MinIO 概念映射(ReplicationPool/Resyncer/MRF/TargetClient)桶复制侧清晰,站点复制侧缺 `SiteReplicationSys` 聚合体
+
+---
+
+## 五、P0 问题对抗性分析(复核结论 + 验证方案 + 解决方案)
+
+### P0-1 出站 peer join 路径 — **CONFIRMED(比原指控更严重)**
+
+**复核结论**:指控全部成立,且加重三点:
+1. `/minio/admin/v3/site-replication/join` 在 MinIO 历史上**从未存在过**(`git log -S` 追到功能诞生的 2021 年首个提交,注册的就是 `peer/join`)。RustFS 实现者疑似被 MinIO `admin-handlers-site-replication.go:76` 一条过时的文档注释误导。
+2. 无任何 404 回退、版本探测或 feature flag;唯一的重试逻辑只针对 secret 不匹配(`site_replication.rs:3036-3082`),404 直接失败。
+3. 现有单测 `:13683-13696` 正在**固化错误行为**(测试名声称匹配 MinIO 路由,断言的却是不存在的路由)。RustFS↔RustFS 之所以不暴雷,是因为 RustFS 入站自己注册了该错误路径的兼容别名,掩盖了 bug。
+
+**影响面**:RustFS 发起的 add(含 MinIO 站点)、服务账号轮换通知 MinIO peer 均断;MinIO→RustFS 与 RustFS↔RustFS 不受影响;其余 peer/* 端点走通用前缀改写,路径正确。
+
+**修路径还不够,还有三处 join 协议分歧须同批修**:①加密判定 `site_replication_peer_payload_encrypted`(:2899-2901)只对旧路径加密,MinIO `SRPeerJoin` 强制解密,须跟随路径改;②MinIO join 成功返回**空 body**,RustFS `:8163` 强制解析 `SRPeerJoinResponse` 会失败,须容忍空 body(peer 身份回退用 preflight 已取得的数据合成);③`deferSyncStateEnable`/`bootstrapToken` 对 MinIO 无效但不阻断(行为差异,建议日志标注)。
+
+**验证方案**:
+- 单测:翻转 `:13683`/`:13699` 两个测试断言为 `peer/join`(把固化 bug 的测试变成回归防护)。
+- 集成测:测试内起 axum stub 精确复刻 `admin-router.go` 路由(仅注册 `PUT .../peer/join`,其余 404),handler 内用 `decrypt_stream_io` 验证 body 是 madmin 兼容密文,返回 200 空 body;断言修复前 404、修复后全链路成功。
+- e2e:docker compose(rustfs+minio),RustFS 侧 `mc admin replicate add`,MinIO 侧 `mc admin trace -a` 断言 `PUT .../peer/join` 200。注意:**e2e 会先被 P0-2 的 preflight 挡住,两问题必须同批修复才能全链路验证**。
+
+**解决方案**(均在 `handlers/site_replication.rs`):删除 :2885-2886 的 join 特判使其落入通用前缀改写;:2899-2901 加密判定改为对 `peer/join` 返回 true;:8163 响应解析容忍空 body;更新两个单测。
+**滚动升级风险**:必须保留入站的 `/v3/site-replication/join` 旧路径路由(旧版 RustFS 出站仍发它);发版前对最近 release tag 复核旧版入站已注册 `peer/join`。
+
+### P0-2 SRInfo 大小写不匹配 — **CONFIRMED(范围精确化)**
+
+**复核结论**:成立。madmin-go v3.0.109(minio go.mod 锁定版)`SRInfo` 除 `APIVersion` 外 12 个顶层字段**全部无 json tag**,Go 按 PascalCase 序列化;RustFS `SRInfo` serde 大小写敏感、全字段 `#[serde(default)]` → 解析 MinIO 输出**不报错而是静默全空**。精确化:**不兼容仅限 SRInfo 顶层 12 个字段**,嵌套结构(SRBucketInfo/SRStateInfo/SRIAMPolicy 等)madmin 本就带小写 tag,不受影响。`:5581` 的 `"buckets"|"Buckets"` 手写双读证明作者已知 MinIO 输出 PascalCase,只是未系统化修复。
+
+**影响面**:RustFS 发起 add 时 preflight 硬失败("site did not report deploymentID")——**触发顺序先于 P0-1 的 join**;`mc admin replicate status` 对 MinIO peer 静默显示全空/全 mismatch(HTTP 200,无报错)。MinIO 读 RustFS 方向因 Go unmarshal 大小写不敏感而无恙。
+
+**验证方案**:
+- 单测(crates/madmin):用 Go `json.Marshal(madmin.SRInfo{...})` 真实生成的 PascalCase JSON 作 fixture,断言反序列化后字段非空;再加序列化回归断言输出仍为 camelCase(保证 RustFS↔RustFS 不回归)。
+- 集成测:stub 在 metainfo 端点返回 PascalCase body,走 `remote_add_preflight_info`,断言不再报错。
+- e2e:与 P0-1 同批,`mc admin replicate status --json` 断言 MinIO 站点条目完整。
+
+**解决方案**:`crates/madmin/src/site_replication.rs:642-670` 为 12 个顶层字段逐一加 `#[serde(alias = "...")]`(精确取 Go 字段名,注意是 `ILMExpiryRules` 不是 `IlmExpiryRules`)。alias 只影响反序列化,出站格式零变化,风险几乎为零。**只加顶层、不扩散到嵌套结构**,并留注释说明原因。回归防护关键是把 Go 真实输出固化为测试 fixture。
+
+### P0-7 `mc replicate add` 默认参数被拒 + 单位错误 — **CONFIRMED**
+
+**复核结论**:全部反驳方向反向坐实(本地有 mc 源码,非推断):
+- mc `replicate-add.go:93-95` 默认 `healthcheck-seconds=60`,`:301-303` 无条件调用 `SetRemoteTarget`,失败即终止,无跳过路径;
+- madmin `bucket-targets.go:79` `HealthCheckDuration time.Duration` 无自定义 Marshal → wire 上是纳秒整数 `60000000000`;
+- RustFS `handlers/replication.rs:213-225` 对非零值必拒 400;`mc replicate update` 同样失败;无老端点绕过。
+- **单位错误独立成立且双向**:请求侧按 `Duration::from_secs` 解析(60e9 ns 会被当 60e9 秒 ≈ 1900 年);响应/持久化侧 `bucket_target.rs:195-197` 按秒序列化,mc 按纳秒解(60s 显示为 60ns),同时构成与 MinIO `bucket-targets.json` 的持久化格式偏差。
+- **为何没被发现**:这是刻意的"能力契约式拒绝"策略,且有单测 `replication.rs:1353-1379` 固化拒绝行为;e2e 全部自行构造 JSON、不含该字段,测的是"RustFS 自己的请求形态"而非"mc 默认请求形态"。缓解:`--healthcheck-seconds 0` 时字段 omitempty 被省略可通过,但默认路径必失败,P0 成立。
+
+**验证方案**:复现——`mc replicate add rustfs/src --remote-bucket http://ak:sk@target/dst` 预期 400;修复后——madmin 形态 payload(60e9 ns)单测断言内部 Duration==60s;set→list 往返断言响应为纳秒;e2e 增加"mc 默认 payload"用例;持久化防御性读回归(旧秒格式升级后读取不变)。
+
+**解决方案(分阶段)**:
+1. **解阻塞**:从不支持清单移除 `healthCheckDuration`(能力契约版本号递增);请求按 `Duration::from_nanos` 解析(`total_downtime` 同步核查);调度上显式忽略并在契约/文档标注"接受但暂不生效";响应侧新增 DTO 按纳秒序列化(**勿直接改 `bucket_target.rs` 的 `duration_seconds`,它同时是持久化格式**);持久化读取加防御(≥10^7 视为纳秒),写入统一新格式。
+2. **落地语义**:`bucket_target_sys.rs:332-441` heartbeat 循环改为按 target 取值,对齐 MinIO(默认 5s、有下限)。
+3. **防复发**:建立容器内跑真 mc 命令的兼容 e2e 通道,覆盖 `replicate add/update/status`。
+
+### P0-8 站点复制架构 — **事实 CONFIRMED,定性部分 REFUTED,降级为 P1**
+
+**复核结论**:巨型文件(14614 行,非测试约 9533 行,24 个 handler)与三处反向导入全部属实;但"失察"定性被推翻:
+- `scripts/check_layer_dependencies.sh` **已建模并拦截**这些边,`layer-dependency-baseline.txt` 棘轮基线逐条列出全部 46 条存量反向边,**新增反向边 CI 必炸**;
+- `ecfs.rs` 被脚本刻意归类为 interface 层(有意的建模决策);
+- ARCHITECTURE.md 自己声明部分不变量 "currently violated... documenting them makes violations explicit and trackable";git 历史显示这是已知、受控、正在偿还的过渡态。
+- **结论:不构成正确性风险,从 P0 降为 P1(可维护性债务)**。真实成本:9.5k 行单文件的评审/合并冲突/增量编译负担,hook 直连使 app/storage 单测无法脱离 admin 层。
+
+**验证方案**:每阶段跑 `make pre-pr`;每消除一条反向边即**删除基线对应行**(而非重生成),使回归必炸;行为回归靠 site replication e2e + 路由快照测试 + `git diff --color-moved` 评审纯移动。
+
+**解决方案(分阶段)**:
+1. **解反向依赖(低风险,先做)**:复用 `site_replication_reconcile.rs` 已验证的 OnceLock 注册模式——bucket 三个 hook 在 app 层定义 fn-pointer 契约、admin 构建路由时注册;`node_service.rs` 的 reload 走 infra 层"运行时重载注册表"。注册缺失时显式降级(warn + no-op)。
+2. **文件拆分(纯移动)**:`site_replication.rs` → 模块目录:`transport`(peer client/DNS/TLS)、`gob`、`state`(注意 config key 路径不可变)、`iam_sync`、`heal`、`handlers`(24 个薄 handler)。
+3. **领域下沉(风险最高,最后做)**:hook 解耦后把 gob/transport/状态机移入独立 crate,注意全局状态清单(`docs/architecture/global-state-inventory.md:114`)。
+
+### P0-3 STS item 类型名不一致 — **CONFIRMED(双向硬断)**
+
+**复核结论**:成立,且两端都是**报错而非静默忽略**:MinIO 收到 `"sts-credential"` 走 default 分支返回 400 `errSRInvalidRequest`;RustFS 收到 `"sts-account"` 返回 NotImplemented。两端 heal/重试机制都会永久重试失败(MinIO 日志持续 "Unable to heal temporary credentials")。MinIO 当前版本 STS 复制发送面很广(AssumeRole/WebIdentity/ClientGrants/LDAPIdentity/Certificate 全系 + sftp/ftp + heal 路径)。除类型串外 `SRSTSCredential` 字段双方完全对齐——**只差这一个字符串**(推测 RustFS 实现时把 madmin 的 JSON 字段名 `stsCredential` 误当成了类型常量)。
+
+**影响面**:跨厂商 STS 临时凭证双向不复制(客户端在对端站点 `InvalidAccessKeyId`),纯可用性问题,无权限漂移;RustFS↔RustFS 自洽。
+
+**验证方案**:单测——出站产物断言 `type == "sts-account"`(改 `federated_identity.rs:497` 现有快照测试);入站构造 `"sts-account"` item 断言不落 NotImplemented。e2e——compose(RustFS+MinIO,root 凭证必须一致,否则 token 验签失败会误判修复无效):对 MinIO assume-role 拿临时凭证访问 RustFS,修复前 InvalidAccessKeyId、修复后成功;反向同测。
+
+**解决方案**:出站(`sts.rs:248`、`federated_identity.rs:241`)改发 `"sts-account"`(提常量集中定义);入站(`site_replication.rs:7857`)match 臂改 `"sts-account" | "sts-credential"`(**永久保留旧别名**兼容旧 RustFS peer)。滚动升级窗口内新→旧 RustFS 会降级(warn+重试,peer 升级后收敛);STS 凭证短生命周期,不建议为此拆两阶段发布。
+
+### P0-4 policy-mapping userType 数值错位 — **CONFIRMED(比指控更严重)**
+
+**复核结论**:数值表属实(RustFS: None=0/Svc=1/Sts=2/Reg=3;MinIO: unknown=-1/reg=0/sts=1/svc=2),wire 上确为数值、无翻译层。对抗复核修正与加重:
+- **RustFS→MinIO 方向今天"侥幸能用"**:RustFS 当前只出站 Reg=3 与组的 0,MinIO 对超范围值静默落 default 分支,恰好落对位置;
+- **MinIO→RustFS 方向三类断裂**:①**组映射硬失败(新发现)**——MinIO 组映射发 `UserType: -1`,RustFS `user_type: u64` 反序列化直接报错,整个 item 被拒,组→策略映射完全无法同步;②STS 用户映射(MinIO 发 1)被 RustFS 解释为 Svc,落错前缀/缓存,联邦用户在 RustFS 站点**静默丢权限**;③svc=2 被解释为 Sts,同类错位;
+- **低概率提权路径**:LDAP DN/OIDC 主体的映射被误存入常规用户缓存后,若本地恰有同名静态用户则继承本不属于它的策略——名字碰撞概率低但非零,这是保 P0 的理由。
+
+**验证方案**:单测——wire 编解码全矩阵(-1/0/1/2/3/非法值);e2e——MinIO 侧 `mc admin policy attach --group` 修复前 RustFS 查不到组实体、修复后可见;`mc idp ldap policy attach` 修复前落 `policydb/service-accounts/` 且访问被拒、修复后落 `sts-users/` 且放行;反向回归守住"侥幸兼容";混版本(旧+新 RustFS)双向 attach 互通。
+
+**解决方案(核心原则:不改 `UserType::to_u64/from_u64`)**——该编码被集群内部节点 RPC 使用(`node_service.rs:1513`),改动会破坏同集群滚动重启。只在站点复制 wire 边界加 MinIO 语义编解码:
+1. `SRPolicyMapping.user_type` 由 `u64` 改 `i64`(必须,才能收下 -1);
+2. 出站 `sr_wire_user_type`:Reg→0/Sts→1/Svc→2,组一律发 0(对 MinIO 与旧 RustFS 同时兼容);入站 `user_type_from_sr_wire`:-1→None/0→Reg/1→Sts/2→Svc/**3→Reg(旧 RustFS 别名,永久保留)**;
+3. 兼容矩阵已逐格验证:新↔旧 RustFS、MinIO↔新 RustFS 全通;唯一残余窗口(未来出站 Sts/Svc 映射对旧 RustFS 错读)当前不可达,在 doc comment 写明约束;
+4. 回归防护:编解码矩阵单测 + "wire 常量契约"字面值断言测试(防止将来被"顺手统一"回内部编码)+ e2e 进 P0 套件;顺带把 `SRCredInfo.iam_user_type` 一并改 `i64` 复用同一编解码,消除同族隐患。
+
+### P0-5 复制 PUT 缺 `?versionId=` query — **CONFIRMED**
+
+**复核结论**:所有反驳方向均失败,指控成立:
+- minio-go 官方复制端(v7.0.91)`api-put-object-streaming.go:767-776` 等三处全部是 `urlValues.Set("versionId", ...)`——**query,不是 header**;`x-minio-source-version-id` 这个 header 在 MinIO 全仓不存在,被静默忽略;
+- multipart 的版本在 **initiate 时**决定(`erasure-multipart.go:458-460`,为空即生成新 UUID),complete 不读 versionId;
+- aws-sdk-s3 `PutObjectInput` 无 versionId 成员属实,但 DELETE 路径已用 `.set_version_id()` 正确落 query,证明是遗漏而非不可行;
+- RustFS↔RustFS 不受影响的原因:RustFS 接收端有私有 header fallback(`options.rs:296-301`),恰好掩盖了 bug。
+
+**影响加重**:除版本漂移与按版本删除永久 no-op 外,目标校验/heal 用源 versionId `head_object` 永远 miss → **反复重传,目标端版本无限膨胀**。另有边缘缺陷:RustFS 内部 null 版本是 nil-UUID,直接发 query 会被 MinIO 当真实版本;minio-go 约定发字面 `"null"`。
+
+**验证方案**:L1 e2e(本仓可落地,红→绿)——复用 `crates/e2e_test/src/fake_s3_target/`(已解析 versionId query 并写 journal),断言 PutObject/CreateMultipartUpload 请求的 query == 源版本;L2 互操作(docker + 真 MinIO)`mc ls --versions` 断言目标 versionId == 源、删源版本目标同步消失;L3 单测 nil-UUID→`"null"` 映射。
+
+**解决方案**(`bucket_target_sys.rs`):`put_object`/`create_multipart_upload` 在 `map_request` 闭包内改写 URI 追加 `versionId` query(nil-UUID 映射 `"null"`);保留双 header 兼容旧版 RustFS 接收端;顺带核对 delete 路径的 nil-UUID 映射。**签名安全性已验证**:`map_request` 挂在 `modify_before_signing`,query 会进 canonical request,不会 SignatureDoesNotMatch。非版本化目标桶沿用"空则不发",`"null"` 值 MinIO 免检。
+
+### P0-6 CopyObject 不触发复制 — **CONFIRMED(附带加重发现)**
+
+**复核结论**:三个反驳方向全部不成立:
+- copy 直接调 `store.copy_object`,不经 put 路径;ecstore 层 copy 实现无任何调度;
+- **scanner 兜底不存在(关键)**:heal 入队条件是状态为 Pending/Failed 或手动 resync;而 copy 路径不 stamp PENDING(对照 put 路径 `object_usecase.rs:5255-5266`),状态为空 → heal 判定 Skip。
+- **加重发现**:copy 路径没有 MinIO `filterReplicationStatusMetadata` 的等价清理——COPY 指令下源对象的旧复制状态可能原样带到目的对象,**伪造 COMPLETED 假状态**。
+- 附带 P1(snowball `execute_put_object_extract`)同样确认:无 stamp 无 schedule。
+
+**影响面**:配复制规则的桶上,CopyObject 写入的对象(跨桶复制、rename 工作流、REPLACE 元数据更新)永不复制、scanner 不捞、仅手动 resync 可补;还可能带 stale 假状态。
+
+**验证方案**:e2e(参照 `replication_extension_test.rs` 双实例)——copy 后断言目的对象在目标桶超时内出现、源 COMPLETED、目标 REPLICA、无 stale 状态;snowball 参照 `snowball_auto_extract_test.rs` 加成员对象复制断言;usecase 单测用 `storage_api.rs:641` 现有 test-only 调用计数断言 copy/extract 触发决策与调度。
+
+**解决方案**(`object_usecase.rs`):
+1. `execute_copy_object` 在 `store.copy_object` 之前算一次 `dsc = must_replicate_object(...)`,`replicate_any` 时向 `dst_opts.user_defined` stamp pending + timestamp(严格镜像 put 路径,单一 dsc 决策贯穿两阶段);
+2. 同处清理源带来的复制状态 reserved 元数据;
+3. copy 成功、锁释放后 `schedule_object_replication`;
+4. `execute_put_object_extract` 对每个解出对象同样处理。
+风险已排除:replica 判定内置于 `must_replicate_object` 不会回环;self-copy 调度与 MinIO 一致。
+**落地顺序约束:先修 P0-5 再修 P0-6**——否则 copy 的失败重试经 heal 兜底后,只会在 MinIO 端制造更多漂移版本。
+
+### 第三方复审修正(2026-08-05,修复分支均已完成 review)
+
+**P0-5 修正**:问题的准确表述应为"**普通复制对象缺少可靠的源→目标版本身份策略**"——复制 PUT 只返回成功/失败,未捕获目标实际分配的版本 ID(已核实 `bucket_target_sys.rs` put 路径无 `res.version_id()` 捕获,delete 路径 :2030 有);multipart 只保留 upload ID。`fix/p0-5` 的 versionId query 方案对 MinIO/RustFS 目标成立(目标端沿用源版本 ID,身份问题消解),但对**忽略该私有 query 的目标(如原生 AWS S3)**身份问题仍在:目标自行生成版本 ID → 后续按源版本 ID 的删除复制落空。第三方建议定级 P1(修复已完成,残留缺口另行跟进):可选方案包括捕获 PUT 响应的 `x-amz-version-id` 并持久化源→目标映射。→ 记为 **P1-19(新增)**。
+
+**P0-6 修正**:scanner"兜底不存在"的表述过度。已核实 `crates/replication/src/operation.rs` `resync_target_for_object`:无 reset 记录且复制状态为 Empty 时返回 `replicate=true`,即 ExistingObjectReplication 启用时 scanner **可能最终补齐**空状态对象,无需手动 resync。准确结论:即时/同步复制语义失效(P0 定级依据),且以下场景**长期**漏复制——①源对象 COMPLETED 等复制元数据被 Copy 继承致误判(`fix/p0-6` 已修,清理先于决策);②显式 ExistingObjectReplication=Disabled;③其他无法进入 existing-object 补偿的场景。`fix/p0-6` 分支已含 copy 调度 e2e 与 stale 元数据白盒断言;**scanner 补偿边界的 e2e 仍缺** → 记为 **P1-20(新增)**。
+
+### 对抗性复核总览
+
+| 问题 | 复核结论 | 关键修正/加重 |
+|---|---|---|
+| P0-1 join 路径 | CONFIRMED,加重 | 路径在 MinIO 从未存在;现有单测固化错误;修复需同批改加密判定与空响应容忍 |
+| P0-2 SRInfo 大小写 | CONFIRMED,精确化 | 仅顶层 12 个无 tag 字段;preflight 失败先于 P0-1 触发 |
+| P0-3 STS 类型名 | CONFIRMED | 双向硬断、两端 heal 永久重试;只差一个字符串 |
+| P0-4 userType 错位 | CONFIRMED,加重 | MinIO 组映射发 -1 → RustFS u64 解析硬失败;存在低概率名字碰撞提权路径;修复不得触碰内部 RPC 编码 |
+| P0-5 versionId query | CONFIRMED,加重 | heal 反复重传致目标版本膨胀;nil-UUID 需映射 "null" |
+| P0-6 CopyObject | CONFIRMED,加重 | scanner 兜底不存在;stale COMPLETED 假状态;须在 P0-5 之后落地 |
+| P0-7 healthCheckDuration | CONFIRMED | 单位错误双向独立成立;有单测固化拒绝行为 |
+| P0-8 架构 | 事实 CONFIRMED,定性 REFUTED | 反向边被棘轮基线锁死,降级 P1(受控技术债) |
+
+---
+
+## 六、修复路线图(2026-08-05 更新)
+
+**✅ 第一批已完成**:全部 7 项 P0 已修复并创建 PR(见第二节修复状态;P1-10 snowball 随 #5753 一并修复)。待合并,注意顺序约束:#5748+#5749 同批、#5752 先于 #5753。
+
+**第二批(数据一致性优先,采纳功能视角复核定级)**
+1. **P1-21** delete-marker 延迟 purge 失败静默丢弃(复核升级,数据一致性,建议单独小 PR + 失败注入测试)
+2. **P1-19** 源→目标版本身份策略(捕获 PUT 响应 `x-amz-version-id` / 持久化映射,覆盖非 MinIO 系目标)
+3. **P1-1** ILM expiry 同步语义(只传播 expiry、保留接收端本地 transition + 对应测试)
+4. **P1-15** site state RMW 分布式锁统一(对齐 repair state 的 `with_config_object_write_lock` 模式)+ 双写者回归测试
+5. **P1-22** 桶复制 SSE 能力(普通复制/Heal/Resync/Multipart 四模式,先补 encrypted-object e2e 钉现状)
+
+**第三批(mc 可观测性与互操作补齐)**
+6. P1-11/12/14 mc 响应结构 serde rename(改动小、消除静默空输出)
+7. P1-7 ARN 解析侧兼容 `arn:minio:` 前缀
+8. P1-5 GET/HEAD proxy、P1-6 时间戳头、P1-3 自动跨站 heal
+9. P1-20 scanner 补偿边界 e2e;P0-7 阶段 2(per-target 心跳 + healthcheck update op)
+10. P2-26 GET 补 `x-amz-replication-status`(约一行)+ 回归测试;P2 清单其余项
+
+**第四批(架构与长期)**
+11. P0-8(降级 P1)架构:先解 3 处反向依赖(复用 reconcile 注册模式),再拆分/下沉站点复制领域模块
+12. P1-16 类型对账测试、P1-17 迁移完成判据、P1-8 配置校验补齐

P1_REVIEW_AND_PLAN.md

@@ -0,0 +1,195 @@
+# P1 逐条复审订正与方案计划
+
+> 复审基线:main @ `77f2b948c`(7 个 P0 修复 #5748~#5754 已全部合入)
+> 复审方式:5 组对抗性复审 agent 并行,先怀疑后确认;以 RustFS 自身功能契约为正确性标准,不以"未对齐 MinIO"为根因;RustFS 更优/独特设计标注"保持不变"
+> 参照:MinIO 源码、mc@cf909e1063a9、madmin-go v3.0.109、minio-go v7.0.91
+> 日期:2026-08-06
+
+---
+
+## 〇、复审总裁定表
+
+| 项 | 主题 | 复审结论 | 关键订正 | 工作量 |
+|---|---|---|---|---|
+| P1-1 | ILM expiry 复制语义 | CONFIRMED(范围扩大) | 发送点共 4 处非 1 处;接收端无门禁;修复重心移到接收端 merge | M |
+| P1-3 | 自动跨站元数据 heal | CONFIRMED(范围收窄) | 真实缺口="retry queue 有账本无消费者";不移植 MinIO 全量 heal | M |
+| P1-5 | GET/HEAD 远端 proxy | CONFIRMED | 同步复制模式是已实现的部分缓解(保持不变);proxy 指标语义被出站 HEAD 污染 | L(P0 段 M) |
+| P1-6 | 三类时间戳头收发 | CONFIRMED(缺口扩大) | 实为三段缺失:tagging 无本地写入方 + 不发头 + 接收端无 LWW 合并点 | M |
+| P1-7 | ARN 前缀不互认 | CONFIRMED+(加重) | 新发现 FromStr id/region 互换 bug;madmin ParseARN 硬校验实锤 → 生成侧必须改 | M |
+| P1-8 | 配置校验缺口 + StorageClass | 部分 CONFIRMED | 2MB 子项 REFUTED(MinIO 亦无);StorageClass 属刻意设计成立(MinIO 也不消费 rule 级,target 级 RustFS 已生效)| S |
+| P1-11 | replication-metrics snake_case | CONFIRMED | BucketStats 复用内部 RPC 线格式实锤 → 必须独立响应 DTO | M |
+| P1-12 | replication-reset 响应壳 | CONFIRMED(面缩小) | 致命键仅 5 个(壳 `Targets`≠`target` + 4 个字段名);其余靠 Go 大小写不敏感能对上 | S |
+| P1-13 | mrf/diff 聚合响应 | CONFIRMED(症状加重) | 实际输出**伪数据行**而非空;diff/mrf 数据源均可支撑逐条流 | diff S / mrf M |
+| P1-14 | set-remote-target 请求体 | 原缺口已缓解;**新 CONFIRMED 阻断** | #5754 后 26 字段已全覆盖;但**零值 `expiration` 恒被拒 → mc replicate add 仍 100% 失败**;latency 单位 round-trip 污染 | S(**建议立即修**) |
+| P1-15 | site state RMW 竞争 | CONFIRMED(加重) | hook 路径 enqueue/dequeue 同进程内绕过既有 Mutex → 单节点即可触发 | M-L |
+| P1-16 | 状态机类型双份定义 | CONFIRMED(加重+收窄) | drift 已发生(MrfOpKind 两侧不一致);但 filemeta 侧 worker DTO 是死代码,活跃双份仅 3 个 wire 类型;"抽公共 crate"否决 | S+M |
+| P1-17 | 桶复制逻辑分裂 | CONFIRMED;微文件合并子项 REFUTED | boundary 微文件是棘轮机制的机械接缝(守护脚本按文件名锚定),合并负收益;缺的是完成判据 | M0=S,整体 L |
+| P1-18 | 超长函数 | 行数 CONFIRMED;apply_iam_item 降级 | apply_iam_item 长而不复杂(6 臂 dispatch),不拆降 P2;其余 4 个给纯移动拆分草案 | M |
+| P1-19 | 源→目标版本身份策略 | CONFIRMED(范围收窄) | delete-marker 的"捕获+持久化映射"模式已落地(保持不变);推荐能力探测+显式拒绝而非全量映射 | M |
+| P1-20 | scanner 补偿边界 e2e | CONFIRMED(缺口收窄) | 决策函数单测与 Failed-heal e2e 已存在;缺 existing-object 矩阵与 Replica 防环 e2e;附完整入队真值表 | M |
+| P1-21 | delayed purge 静默丢弃 | CONFIRMED | 映射损坏防护已加固(保持不变);`let _ =` 与无 MRF 通道仍在;附带发现 MRF outcome 恒 false 滞留问题 | M |
+| P1-22 | 桶复制 SSE 能力 | CONFIRMED(前提订正) | SSE-S3 自 #5633 已 fail closed,被 ignore 的 e2e 理由过期(先摘 ignore);SSE-C 缺的是目标侧头摄取 | L(4 阶段) |
+
+**"保持不变"清单(复审确认的 RustFS 更优/刻意设计,不纳入修复)**:per-PUT 即时元数据传播 hook(优于 MinIO 纯周期 heal)、单向推送+stale 守卫收敛模型、delete 走 merge-with-empty(优于 MinIO 整删)、delete-marker 版本映射持久化+损坏拒猜、同步复制模式(partition_by_sync)、能力契约式显式拒绝+`deny_unknown_fields`(字段清单已与 madmin v3.0.109 同步)、StorageClass 显式拒绝非 STANDARD(target 级已真正生效)、replication-check 真实探针写删、响应中的 RustFS 增强字段(ResetBeforeDate/Error/可观测性键,Go 忽略未知键可共存)。
+
+---
+
+## 一、紧急项(建议立即处理)
+
+### ⚡ P1-14 新阻断:零值 `expiration` 拒绝 → mc replicate add 仍 100% 失败
+
+- **证据**:Go `omitempty` 不省略零值 `time.Time`(已用 Go 程序按 madmin 逐字 tag 实测),mc/madmin marshal 恒输出 `"credentials":{"expiration":"0001-01-01T00:00:00Z"}` 与 `"resetBeforeDate":"0001-01-01T00:00:00Z"`;RustFS `handlers/replication.rs:286-291` 对 `expiration.is_some()` 一律 400。#5754 的测试全部用手写 payload(`expiration: None`),未被现网形状打中。
+- **修复(S)**:①`expiration` 改"非 Go 零值时间才拒"(与 `sessionToken` trim-empty 判断对称);②`latency` 请求字段直接忽略(消除 #5754 后纳秒响应 ↔ 毫秒请求的 round-trip 1e6 倍污染);③把"Go 真实 marshal 形状 payload"固化为测试夹具惯例。
+- **红灯测试**:用实测 Go marshal 全形状 body(含零值 expiration/resetBeforeDate/latency{0,0,0}/edge:false/healthCheckDuration:60000000000)打 set-remote-target,期望 200;非零 expiration 仍 400(能力契约保持)。
+
+### ⚡ P1-7 附带 bug:ARN FromStr 字段互换
+
+`arn.rs` Display 输出 `{type}:{region}:{id}:{bucket}`,FromStr 却读 `id=parts[3], region=parts[4]`——id 与 region 互换。当前仅因消费方只用 arn_type 而潜伏。随 P1-7 一并修。
+
+---
+
+## 二、逐项方案计划
+
+### P1-1 ILM expiry 复制语义(M)
+
+**订正后事实**:发送完整 lifecycle XML 的路径 4 处——PUT hook(`bucket_usecase.rs:2177-2180`)、DELETE hook(`:1512-1514`,触发接收端**整删**)、import(`bucket_meta.rs:948-951`)、build_sr_info/bootstrap(`site_replication.rs:4190,2241-2249`);接收端 `apply_bucket_meta_item`(`:7669-7683`)整体覆盖/删除,且**无 `replicate_ilm_expiry` 门禁**。P0 后已有缓解(发送开关、bootstrap 跳过、stale 判定)只解决"发不发/新旧",不解决"发什么/怎么合"。
+
+**方案**:接收端 merge 为主(信任边界),发送端 expiry-only 提取为辅:
+1. 新增纯函数 `extract_expiry_only(cfg)` 与 `merge_expiry_rules(local, incoming)`——语义对齐 MinIO `mergeWithCurrentLCConfig`,两处 RustFS 改进:incoming 一律先剥 transition(防旧端);`None` 走 merge-with-empty 而非整删(**MinIO 整删连本地 transition 一起删是缺陷,不照抄**);
+2. 接收端 lc-config 分支改 读→merge→条件写/删,保留 stale 判定与 incarnation 守卫;补 `replicate_ilm_expiry` 门禁;
+3. 4 个发送点接 `extract_expiry_only`;expiry 判定用 RustFS 口径(含 `del_marker_expiration`)。
+
+**红灯测试**:L1 单测 5 例(提取剥离/合并保留 T/防御剥离/merge-with-empty/import 无 transition);L3 e2e——B 配本地 transition,A PUT expiry → B 两者共存;A DELETE lifecycle → B transition 仍在。
+**兼容**:旧端发完整 XML → 新接收端剥后 merge 正确;新端 expiry-only → 旧接收端仍整覆盖(不劣于现状)。规则按 ID 对齐,`rule-{idx}` 撞名同 MinIO 语义,文档注明。
+
+### P1-3 自动跨站 heal → 改为"retry queue 自动 drain"(M)
+
+**订正后事实**:retry queue 是现成增量账本(失败即入队 `:3243-3262`,持久化于 state,`retry_count` 字段存在)但**全库无消费者**;手动 repair 是本地快照单向推送,收敛方向依赖运维判断。即时 hook + 显式 repair 模型保持不变。
+
+**方案**:
+- 阶段 1(核心):周期任务挂进现有 reconcile ticker,per-event 重发(body 从本地当前元数据重建,复用 `SiteReplicationRepairTask::send`,天然发"当前值"+对端 stale 守卫幂等);指数退避(`retry_count`+上限转 failed);drain 全程包分布式锁去抖(先用 `with_config_object_write_lock` 专用对象,P1-15 落地后并入统一 state store);结构化 tracing 汇总一条。
+- 阶段 2(可选,默认关闭):每 N tick 比对 repair plan token,不同才自动 dry-run→execute。**不移植** MinIO 跨站取最新 pull 语义(各站各自 drain 即双向收敛)。
+
+**红灯测试**:L2——state 带 retry event,调 `drain_site_replication_retry_queue()`(现不存在),fake peer 成功后断言队列清空;退避断言。L3——停 B→A PUT policy 失败入队→起 B→drain 后 B 收到且 SRRetryStats 归零。
+
+### P1-5 GET/HEAD 远端 proxy(L;P0 段 M)
+
+**订正后事实**:`SUFFIX_SOURCE_PROXY_REQUEST` 零消费者;`ProxyMetric` 字段与 admin 汇总通路已就位,但 resyncer 把**出站** HEAD 计入 `head_total` 污染语义;`disable_proxy` 管道存在无人消费;同步复制模式(`partition_by_sync`,`replication_pool.rs:2667-2689`)是部分缓解但不等价(手动 per-target、失败仍 404、不覆盖兜底窗口)。防环头当前仅潜在问题,但 proxy 实现与防环识别**必须同 PR**(否则 RustFS↔RustFS 成环)。
+
+**方案**(P0 段):新增 `replication_proxy_boundary.rs`——`proxy_targets`(version_suspended/入站 proxy 头/disable_proxy 三重 gate)+ `proxy_get/head_to_replication_target`(走现有 TargetClient,range/条件头透传);触发点在 usecase 层 NotFound/VersionNotFound 分支;接收侧 options.rs 解析防环头,出站双前缀发送;`tokio::timeout`(~3s env 可调)、仅 2xx 采纳其余回落本地 404、复用离线标记短路;指标接 `record_replication_proxy` 并纠正 resyncer 计数语义。P1 段:tagging 三操作 proxy(依赖 P1-6)。
+**红灯测试**:e2e 双站断复制链路后从对端 GET/HEAD 应 200(现 404);防环负例(带头请求不转发、计数不增);降级负例(target 全离线时限时 404);disable_proxy 负例。
+
+### P1-6 时间戳头收发(M;三段修复)
+
+**订正后事实**:①`SUFFIX_TAGGING_TIMESTAMP` 全仓无写入方(retention/legalhold 已有双前缀写入);②`PutObjectOptions::header()` 只序列化 4 个内部头,三类时间戳被丢弃,multipart 同;③接收端不解析,且 replica PUT 是 verbatim 覆盖——解析后必须在写盘前与本地版本做 per-类别 LWW 合并才有效;④`AdvancedPutOptions` 默认 `now_utc()` 无法当"未设置"哨兵,需 Option 化。
+
+**方案**:阶段 0——`put/delete_object_tagging` 落 `SUFFIX_TAGGING_TIMESTAMP`(双前缀);阶段 1——新增三个 suffix 常量(对齐 MinIO headers.go:239-243),三字段 Option 化,`header()` 与 multipart 条件序列化;阶段 2——接收端解析(仅授权复制请求)+ PUT 路径 LWW 合并并持久化赢家时间戳(合并仅限三类元数据,不触碰数据与其余元数据,与 verbatim-replica 不变式共存)。
+**红灯测试**:单测 header 双前缀序列化断言/未设置缺席断言;接收端解析单测;e2e active-active tagging 并发收敛(晚者胜,现 main 旧值覆盖新值为红)。
+
+### P1-7 ARN 前缀(M)
+
+**订正后事实**:madmin `ParseARN` 硬校验 `arn:minio:` 前缀 + ID/bucket 非空(v3.0.109 remote-target-commands.go:50-63);mc 爆炸点仅 `replicate update`(fatalIf)与 `replicate ls`(软降级);`replicate add` 把 ARN 当不透明串不受影响——解释了"add 通 update 挂"。RustFS ARN 结构(`type::id:bucket`)与 madmin 兼容,仅 vendor token 障碍;另有 FromStr id/region 互换 bug(见紧急项)。
+
+**方案(推荐路线 A)**:生成侧默认改 `arn:minio:`(留常量可品牌化);解析侧接受双前缀(存量 `arn:rustfs:` 靠双前缀解析 + 现有字符串等值匹配继续工作);修字段序;改 `generate_arn`、`site_replication.rs:6329` 与相关测试断言。混合版本集群前缀不一致靠双前缀解析吸收;不做存量数据前缀归一化改写。
+**红灯测试**:单测 `from_str("arn:minio:replication:us-east-1:depl:bucket")` 成功且 id/region 正确(现双重红灯);round-trip 属性测试;e2e set-remote-target 返回 ARN 可被 madmin 语义解析、预置 `arn:minio:` 目标可 remove。
+
+### P1-8 配置校验(S)
+
+**订正后事实**:2MB 上限 REFUTED(MinIO 亦无显式检查,剔除);StorageClass 已缓解且刻意设计成立——MinIO 自己也不消费 rule 级 `Destination.StorageClass`(复制 PUT 用 target 级 `tgt.StorageClass`),RustFS target 级 storage_class 已真正生效(`bucket_target_sys.rs:1633-1634`),容忍显式 STANDARD 已实现。仍缺:规则数≤1000、≥1 条、Priority 唯一非负、ID≤255、Filter 互斥、Tag×DeleteMarkerReplication 互斥、sameTarget 拒绝。
+
+**方案**:`config.rs` 新增 `validate_replication_config_structure` 纯函数,`bucket_usecase.rs:2418` 接入;StorageClass 保持现状+契约文档化("rule 级请改用 remote target 的 storageclass 字段")。
+**红灯测试**:单测逐格(1001 规则/重复 Priority/256 字符 ID/Filter 并存/Tag+DMR)期望特定错误;e2e aws-sdk 形状 XML 断言 InvalidRequest。
+
+### P1-11 replication-metrics DTO(M)
+
+**订正后事实**:`BucketStats` 走内部 peer RPC 线格式(`rmp_serde::to_vec_named` 字段名入线,node_service.rs:1401 / peer_rest_client.rs:88-104)——**改原结构 serde 名会破坏混合版本集群 RPC,禁止**;必须走 #5754 的响应 DTO 模式(同文件先例 `remote_target_admin_json`)。
+
+**方案**:新增仅 Serialize 的 `MetricsV2Dto{uptime,currStats,queueStats,downtimeInfo}`/`MetricsDto`/`TargetMetricsDto`,显式映射(`q_stat`→`queued`、`bandwidth_limit_bytes_per_sec`→`limitInBits`、failed→TimedErrStats total-only);`queueStats.nodes` 先填本机一条;RustFS 可观测性扩展键保留(Go 忽略未知键,双栖零成本)。
+**红灯测试**:e2e 用镜像 minio-go MetricsV2 tag 的结构反序列化断言 `currStats.completedReplicationSize > 0`(现全零);DTO 键名 snapshot 单测。
+
+### P1-12 replication-reset 响应壳(S)
+
+**订正后事实**:致命键仅 5 个——壳 `Targets`≠`target`、`Status`≠`resyncStatus`、`ReplicatedSize`≠`completedReplicationSize`、`ReplicatedCount`≠`replicationCount`、`FailedSize/FailedCount`≠`failedReplicationSize/failedReplicationCount`;其余(Arn/ResetID/StartTime/...)靠 Go 大小写不敏感能对上;`ResetBeforeDate`/`Error` 是增强字段可保留。响应结构是 router.rs 独立 DTO 无内部复用,改名零风险。
+
+**方案**:纯 serde rename(建议全字段精确对齐 madmin 小写形态),保留增强键+文档标注。
+**红灯测试**:e2e 断言响应含 `target` 数组且 `target[0].resetid` 非空、status 侧 `resyncStatus`/`completedReplicationSize` 键存在。
+
+### P1-13 mrf/diff 流式响应(diff S / mrf M)
+
+**订正后事实**:症状比"输出空"更糟——聚合对象会被 madmin `json.Decoder` 成功解码一次,`mc replicate backlog` 输出一条 object 为空的**伪行**(静默伪数据);路线 A(保持聚合+文档化)无法消除伪行且与 madmin 同 path 无内容协商,**不可行**。数据源评估:diff 已逐条扫描只需去壳;mrf 的 durable backlog(`MrfReplicateEntry` 字段恰好覆盖 `ReplicationMRF` 所需)已可枚举。
+
+**方案(路线 B)**:diff 去壳输出 NDJSON `DiffInfo` 形状(仅 `IsDeleteMarker`/`ReplicationStatus` 需 rename;truncation 信息入日志不入流);mrf 遍历 durable entries 逐条输出 `ReplicationMRF` 形状(nodeName 填本机);聚合响应保留在 `?aggregate=true`(RustFS 扩展,deliberate 注释随迁)。条目量有 `REPLICATION_DIFF_MAX_SCAN` 封顶,内存拼 NDJSON 即可不必真流式。
+**红灯测试**:e2e 制造失败复制后逐行反序列化断言至少一条 `object` 非空(现为伪空行);diff 断言无 `Entries` 壳。
+
+### P1-14 set-remote-target(S,含紧急项)
+
+见"一、紧急项"。另:`deny_unknown_fields` **保留**(推荐)——字段清单已与 madmin v3.0.109 全同步,严格模式+显式清单兼得契约哲学与防静默;代价写进维护清单:"madmin 版本升级时同步字段清单"(加对照 madmin tag 列表的常量测试防漂移)。
+
+### P1-15 site state 统一 store(M-L,两 PR)
+
+**订正后事实**:主 state 有进程内 Mutex(`:347`)但两处不完备——①无分布式锁(多节点 RMW 丢更新);②**retry event enqueue/dequeue 不持锁**(挂在所有 hook 广播路径上,同进程即可丢更新);reload 路径完全无锁(稳态不写盘收窄窗口,迁移期可覆盖并发写)。repair state 的 `with_config_object_write_lock` + no-lock IO 是正确样板(`:1097-1114`);两套归一化的语义差异(JSON-level 容忍畸形 peer)是**有意的**,统一时必须保留。锁序注释 `:346` 可挂靠。
+
+**方案**:PR1——新建 `admin/site_replication_state.rs`:两阶段归一化合一(JSON 宽容清洗→类型化)、`read_state()/update_state(F)`(分布式锁包完整 RMW,锁内禁网络调用与嵌套配置锁)、常量收敛;service reload 接入;迁移 service 侧 5 个归一化测试保语义。PR2——迁移全部 ~30 个 RMW 调用点(含 enqueue/dequeue),**移除**进程内 Mutex(避免双锁新顺序约束);dequeue 热路径保留"先无锁读、命中才进 update_state"两段式;更新锁序注释。每个调用点做重入审查(现有 drop-reacquire 模式保持)。
+**红灯测试**:L2 单进程并发——持锁 RMW(mark_pending_rotation_peer_acked)×绕锁写者(enqueue_retry_event)注入交错,断言最终 state 两者共存(现必丢其一,确定性红灯);L1 归一化等价性测试迁移;L3 双节点并发(nice-to-have)。
+**风险**:盘上格式不变;锁超时从"静默丢更新"变"显式报错",hook 路径保持 warn 不阻断 S3 主路径。
+
+### P1-16 类型对账护栏(S)+ 死代码清理(M)
+
+**订正后事实**:drift 已发生(filemeta 侧 `MrfOpKind` 缺 Metadata/Heal/ExistingObject 三 variant、`MrfReplicateEntry` 缺 force_delete/target_arns)——但 filemeta 侧 8 个 worker DTO 全是**死代码**(零消费者);活跃双份仅 `ReplicationStatusType/VersionPurgeStatusType/ReplicationState` 三个 wire 类型(filemeta 绑 xl.meta 磁盘格式,replication 绑 MRF/resync 持久化格式);boundary 枚举转换 `as_str()` 兜底 `_ => Empty` 会静默降级。"抽公共 leaf crate"否决(两 wire 格式演进节奏不同,迁移规则 #12 本意是所有权独立)。
+
+**方案**:Step 1(S,即刻)——boundary 加对账测试:两侧枚举穷尽 match(新增 variant 即编译失败)+ as_str 双向 round-trip + ReplicationState 全字段往返;Step 2(M)——清理 filemeta 侧 ~600 行死代码 DTO,注意 crates.io semver(先 `#[deprecated]` 一版再删);Step 3(S)——replication 侧注释指向对账测试。
+
+### P1-17 迁移完成判据(M0=S;整体 L)
+
+**订正后事实**:"合并 boundary 微文件"REFUTED——守护脚本按具体文件名锚定每个 boundary,合并要同步改脚本+mod+导入点而功能收益为零;微文件是棘轮机制的机械接缝。唯一可退役:`datatypes.rs`(消费者迁完即删)。README 建议的第一步(event sink/runtime boundary)实际已部分落地,文档滞后。
+
+**方案**:M0(S)文档 PR——完成判据 = Required Contracts 表 "Current dependency to remove" 列清空;终态 = pool/resyncer/state 移入 crates/replication,boundary 随 crate 移动自然消解;更新 split-plan "Proposal only" 状态。M2(M)resyncer 纯决策逻辑下沉;M3(L)trait 稳定后移 worker 运行时(全计划唯一高危段,最后做);M4(S)统一退役 boundary 与守护条目。**不做**批量合并微文件。
+
+### P1-18 超长函数拆分(M;4 个 PR)
+
+**订正后事实**:行数确认(resync_bucket 537 / start_mrf_processor 306 / replicate_all 409 / delete 路径 replicate_object 299 / apply_iam_item 255);`apply_iam_item` **降级 P2 不拆**(6 臂 dispatch,每臂线性短小,拆分违反 "Prefer direct, local code");`replicate_object` 有两个同名体,原清单指 delete 路径 trait impl。
+
+**方案**(每函数独立 PR,纯移动,`git diff --color-moved=dimmed-zebra` 验证):
+1. `resync_bucket`(最优先,三处历史并发 bug 注释所在):acquire_resync_leadership / load_resync_replication_config / spawn workers+collector 三段抽出,并发 bug 注释随代码移动,每个 return 前的 mark_status 逐一保持;
+2. `start_mrf_processor`:抽 `reconstruct_mrf_delete/object` 纯函数(主循环 -150 行,重建逻辑可单测);
+3. `replicate_all` + delete 路径 `replicate_object`:各拆 3-4 个阶段 helper;**明确不合并两函数**(delete-marker 404/405 校验语义是刻意差异)。
+**排序依赖**:先 P1-18 拆分、后 P1-17 M2/M3 迁移(小函数降低搬运风险)。
+
+### P1-19 版本身份策略(M,推荐方案 B)
+
+**订正后事实**:#5752 已合入(PUT/multipart initiate 带 query,RustFS 目标侧也支持);PUT 响应 `x-amz-version-id` 仍被丢弃(`:1891 Ok(_)`);**delete-marker 子案已系统性缓解**——`remove_object` 捕获目标版本号→`target_delete_marker_version_ids` 持久化进 xl.meta(含上限与损坏标记)→延迟 purge 优先用映射、损坏拒猜(**保持不变**);RustFS 无"仅支持 MinIO 目标"契约声明;replication-check 探针已捕获响应版本号但不比对。MinIO 同样丢弃响应版本号(平价),RustFS 已有两点增强。
+
+**方案对比**:A 全量映射持久化(完整但 xl.meta 膨胀、全链路改造,L);**B(推荐)**:契约=仅支持"沿用源版本 ID"的目标,在 replication-check 增加 VersionFidelity phase(探针 PUT 带 versionId query,比对响应版本号)+ `validate_target` 复用同一探测,不镜像则新错误 `BucketRemoteTargetVersionMismatch` 显式拒绝/告警(M);C 混合(无需求支撑)。探针是主动写,进 validate_target 会扩 set-target 副作用面——可先只做 check phase + 运行期首次 PUT 抽查告警。
+**红灯测试**:FakeS3Target 加 `assign_own_version_ids` 开关模拟原生 S3,断言版本删除复制落空(现红)与探测后显式拒绝(修后绿)。
+
+### P1-20 scanner 补偿边界 e2e(M,纯测试)
+
+**订正后事实**:决策函数单测(queue.rs 7 例等)与 scanner 驱动的 Failed-heal e2e(target 断电恢复/源重启重放,FAST_SCANNER_ENV)已存在;真实缺口=无任何"先写对象→后配复制"的 existing-object 用例。完整入队真值表已梳理(见复审记录):Enabled×Empty 补齐、Pending/Failed 恒补(不受 existing 开关影响)、Disabled×Empty 永不补、Replica 恒不补(防环)、null-version 永不入队、reset_id 重置补齐。
+
+**方案**:e2e 矩阵 1-2 个用例(先 PUT 四种来源对象含 Copy/Snowball 产物→后配 Enabled/Disabled 规则→正例 wait_for_replicated_object / 负例 assert_failed_replication_stays_absent_for ≥3 周期,**"永不补齐"是契约必须显式断言**)+ Replica 防环变体 + queue.rs 补 2 格单测;null-version 跳过行为先写"记录现状"断言并注明出处。不改产品代码。
+
+### P1-21 delayed purge 失败处理(M)
+
+**订正后事实**:静默点两处——target client 缺失 `continue` 无日志(`:1673-1675`)、`let _ = remove_object`(`:1693-1700`);5 次循环是等源 marker 消失非重试;purge 调用后无条件 break;MRF 入队接口(`queue_replica_delete_task`,队满自动落盘)同 crate 可用无分层障碍;映射优先/损坏拒猜是已加固项保持不变。**附带发现**(建议单独跟进):`requires_delayed_purge` 恒真使 delete-marker 类 MRF 条目 outcome 恒 false → 重放永远 Missed 保留,可能永久滞留。
+
+**方案**:①purge 函数返回 per-target 成败,失败 warn(带 event 常量)+ metrics,client 缺失同样 warn(S);②循环内失败重试、轮次耗尽入 MRF、入队失败 warn+metric 兜底(S/M);③两层失败注入测试(mock 503 断言重试/状态/MRF;FakeS3Target inject 断言故障清除后最终收敛)(M)。风险:MRF 重放重发 DELETE marker 创建——mtime 幂等,风险低。
+
+### P1-22 SSE 能力(L,4 阶段)
+
+**订正后事实**:fail-closed 由 #5633 引入(`replication_target_boundary.rs:101-174`),普通/Heal/Resync/Multipart 全走同一函数;SSE-C 发送半边已建(内部头→`X-Rustfs-Replication-*` 映射+CRC),**目标侧摄取代码完全缺失**(链路必断,e2e 已钉 FAILED);SSE-S3 契约 e2e 的 `#[ignore]` 理由(backlog#1291 silently drops)已被 #5633 过期;直传托管 SSE 不可行(封存密钥绑本站 KMS),MinIO 是源解密+目标重加密;ecstore 已有 `ObjectEncryptionResolver` trait seam,解密不破分层。
+
+**方案**:阶段 0(S)摘 ignore + 补 encrypted resync/heal e2e 钉全矩阵 fail-closed 现状;阶段 1(M)SSE-C 目标侧头摄取+加密尺寸/CRC(MinIO :1670-1740 参照);阶段 2(M/L)SSE-S3 经 resolver 解密+目标 AES256 重加密(resolver 未注册必须继续 fail closed;multipart 按明文尺寸分片);阶段 3(L)SSE-KMS + key id 随行开关(目标站无同名 key 显式失败,禁止回退 SSE-S3)。过渡期全矩阵维持 fail closed,禁止明文降级。
+
+---
+
+## 三、执行批次建议
+
+| 批次 | 内容 | 性质 |
+|---|---|---|
+| **B0 立即** | P1-14 零值 expiration + latency 忽略(S);P1-7 FromStr 字段互换(并入 P1-7 或先行) | mc 阻断修复 |
+| **B1 小改动高收益** | P1-12 响应壳 rename(S)、P1-13 diff 去壳(S)、P1-8 结构校验(S)、P1-16 Step1 对账测试(S)、P1-17 M0 文档判据(S)、P1-22 阶段 0 摘 ignore(S) | serde/校验/测试护栏 |
+| **B2 数据一致性** | P1-21 purge 失败处理(M)→ P1-20 scanner 矩阵 e2e(M,纯测试)→ P1-19 方案 B 能力探测(M)→ P1-15 state store PR1+PR2(M-L) | 一致性核心 |
+| **B3 互操作补齐** | P1-7 ARN 路线 A(M)、P1-11 MetricsV2 DTO(M)、P1-13 mrf 流(M)、P1-6 时间戳三段(M)、P1-1 ILM merge(M)、P1-3 retry drain(M) | mc/跨站语义 |
+| **B4 大功能与架构** | P1-5 proxy P0 段(M→L)、P1-22 阶段 1-3(L)、P1-18 四函数拆分(M)→ P1-17 M2-M4(L)、P1-16 Step2 死代码(M) | 长期 |
+
+**批内依赖**:P1-6 先于 P1-5 的 tagging proxy;P1-18 先于 P1-17 M2/M3;P1-15 PR1 的锁对象可先供 P1-3 drain 使用。

README.md

@@ -116,7 +116,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-rc.3
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.12
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command

README_ZH.md

@@ -113,7 +113,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # 使用指定版本运行
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-rc.3
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.12
 ```
 
 如果您通过绑定挂载启用 TLS 证书目录，也请用同样方式准备该目录：

_typos.toml

@@ -40,7 +40,6 @@ mak = "mak"
 gae = "gae"
 GAE = "GAE"
 thr = "thr"
-mis = "mis"
 # s3-tests original test names (cannot be changed)
 nonexisted = "nonexisted"
 consts = "consts"

crates/AGENTS.md

@@ -19,9 +19,7 @@ Applies to all paths under `crates/`.
 
 - Document lock acquisition order when a module uses multiple locks. Never acquire the same set of locks in different orders across code paths.
 - Never hold a `tokio::sync::RwLock`/`Mutex` write guard across `.await` points unless the critical section is unavoidably async and the hold time is bounded.
-- Prefer direct atomic `fetch_*` operations for unconditional updates and
-  `compare_exchange` loops only for conditional updates such as peaks or
-  adaptive state.
+- Prefer `compare_exchange` loops over load-then-store for concurrent counters (peak values, adaptive heuristics).
 - When resetting multi-field atomic statistics, use a version/sequence counter or accept that concurrent readers may see partial snapshots; document the tradeoff.
 - `std::sync::Mutex` is acceptable in async context only when held for a brief, non-`await`-containing critical section. If in doubt, use `tokio::sync::Mutex`.
 
@@ -42,9 +40,7 @@ Applies to all paths under `crates/`.
 - Keep unit tests close to the module they test.
 - Keep integration tests under each crate's `tests/` directory.
 - Add regression tests for bug fixes and behavior changes.
-- Every test needs an observable failure criterion. Direct assertions,
-  delegated assertions, snapshots/properties, `#[should_panic]`, and meaningful
-  `Result` failures are all valid; a call that can silently succeed is not.
+- Every test function must contain at least one `assert!`/`assert_eq!`/`assert_matches!`. A test that only calls code without asserting is not a test.
 - In tests, prefer `.expect("context: what was being tested")` over bare `.unwrap()`. A test failure should tell you which operation failed and with what input.
 
 ## Async and Performance

crates/audit/AGENTS.md

@@ -50,3 +50,4 @@ crate.
   - `cargo test -p rustfs-audit`
   - Focused: `cargo test -p rustfs-audit --test pipeline_layer_test`
   - Focused: `cargo test -p rustfs-audit pipeline`
+- Full gate before commit: `make pre-commit`

crates/audit/tests/pipeline_layer_test.rs

@@ -236,19 +236,12 @@ async fn audit_pipeline_reports_empty_runtime_snapshots() {
 }
 
 #[tokio::test]
-async fn stopping_audit_replay_workers_is_a_no_op_when_there_are_none() {
+async fn audit_runtime_facade_stops_empty_replay_workers() {
     let registry = Arc::new(Mutex::new(AuditRegistry::new()));
     let replay_workers = Arc::new(RwLock::new(rustfs_targets::ReplayWorkerManager::new()));
-    let facade = AuditRuntimeFacade::new(registry, Arc::clone(&replay_workers));
+    let facade = AuditRuntimeFacade::new(registry, replay_workers);
 
     facade.stop_replay_workers().await;
-
-    // The stop path takes the manager's workers and hands them to the adapter,
-    // so an empty facade must leave it empty rather than wedge it, and a second
-    // call — which shutdown paths make — must stay harmless (rustfs/backlog#1836).
-    assert!(replay_workers.read().await.is_empty());
-    facade.stop_replay_workers().await;
-    assert!(replay_workers.read().await.is_empty());
 }
 
 #[tokio::test]

crates/checksums/src/base64.rs

@@ -11,6 +11,7 @@
 // WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
 // See the License for the specific language governing permissions and
 // limitations under the License.
+#![allow(dead_code)]
 
 use base64_simd::STANDARD;
 

crates/checksums/src/http.rs

@@ -21,13 +21,6 @@ use crate::{
     Xxhash3, Xxhash64, Xxhash128,
 };
 
-// DELIBERATE DUPLICATION of the x-amz-checksum-* names that also exist as
-// AMZ_CHECKSUM_* in rustfs-utils' headers module (crates/utils/src/http/
-// headers.rs): this crate is a zero-internal-dependency leaf, so it cannot
-// import them, and it additionally owns the RustFS extension names
-// (sha512/xxhash*) that utils does not carry. Values are pinned by the S3
-// wire protocol; do not merge without a maintainer decision on the leaf
-// boundary (backlog#1833).
 pub const CRC_32_HEADER_NAME: &str = "x-amz-checksum-crc32";
 pub const CRC_32_C_HEADER_NAME: &str = "x-amz-checksum-crc32c";
 pub const SHA_1_HEADER_NAME: &str = "x-amz-checksum-sha1";
@@ -38,10 +31,7 @@ pub const XXHASH_3_HEADER_NAME: &str = "x-amz-checksum-xxhash3";
 pub const XXHASH_64_HEADER_NAME: &str = "x-amz-checksum-xxhash64";
 pub const XXHASH_128_HEADER_NAME: &str = "x-amz-checksum-xxhash128";
 
-#[allow(
-    dead_code,
-    reason = "Content-MD5 wire name, resolved by header_name() below and asserted by this crate's tests (backlog#1823)"
-)]
+#[allow(dead_code)]
 pub(crate) static MD5_HEADER_NAME: &str = "content-md5";
 
 pub const CHECKSUM_ALGORITHMS_IN_PRIORITY_ORDER: [&str; 5] =

crates/checksums/src/lib.rs

@@ -41,14 +41,6 @@ pub const XXHASH_64_NAME: &str = "xxhash64";
 pub const XXHASH_128_NAME: &str = "xxhash128";
 pub const MD5_NAME: &str = "md5";
 
-/// One of three deliberately separate checksum registries (backlog#1833):
-/// this enum owns the **streaming-hash algorithm registry**, including the
-/// RustFS extensions (sha512, xxhash3/64/128). The on-disk xl.meta bitset
-/// lives in `rustfs_rio::ChecksumType` (crates/rio/src/checksum.rs, varint
-/// bits are append-only), and the MinIO-port client keeps its own
-/// `ChecksumMode` (crates/ecstore/src/client/checksum.rs). When adding an
-/// algorithm, extend all three (or record why not) — they do not derive from
-/// each other.
 #[derive(Debug, Clone, Copy, PartialEq, Eq, Default)]
 #[non_exhaustive]
 pub enum ChecksumAlgorithm {
@@ -476,19 +468,13 @@ impl Checksum for Xxhash64 {
     }
 }
 
+#[allow(dead_code)]
 #[derive(Debug, Default)]
-#[allow(
-    dead_code,
-    reason = "Content-MD5 is not a ChecksumAlgorithm variant and has no arm in into_impl: S3 carries it as its own header, separate from the x-amz-checksum-* family. This impl exists so the two paths share the Checksum trait, and is asserted by this crate's tests (backlog#1823)"
-)]
 struct Md5 {
     hasher: md5::Md5,
 }
 
-#[allow(
-    dead_code,
-    reason = "Content-MD5 is not a ChecksumAlgorithm variant and has no arm in into_impl: S3 carries it as its own header, separate from the x-amz-checksum-* family. This impl exists so the two paths share the Checksum trait, and is asserted by this crate's tests (backlog#1823)"
-)]
+#[allow(dead_code)]
 impl Md5 {
     fn update(&mut self, bytes: &[u8]) {
         use md5::Digest;

crates/common/Cargo.toml

@@ -39,15 +39,11 @@ tokio = { workspace = true, features = ["fs", "rt-multi-thread"] }
 tonic = { workspace = true, features = ["gzip", "deflate"] }
 uuid = { workspace = true, features = ["v4", "fast-rng", "macro-diagnostics"] }
 chrono = { workspace = true, features = ["serde"] }
-jiff = { workspace = true, features = ["serde"] }
 metrics = { workspace = true }
 serde = { workspace = true, features = ["derive"] }
-smallvec = { workspace = true }
 rmp-serde = { workspace = true }
+s3s = { workspace = true, features = ["minio"] }
 tracing = { workspace = true }
 
-[dev-dependencies]
-serde_json = { workspace = true }
-
 [lib]
 doctest = false

crates/common/src/bucket_stats.rs

@@ -0,0 +1,87 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use crate::last_minute::{self};
+use std::collections::HashMap;
+
+pub struct ReplicationLatency {
+    // Delays for single and multipart PUT requests
+    upload_histogram: last_minute::LastMinuteHistogram,
+}
+
+impl ReplicationLatency {
+    // Merge two ReplicationLatency
+    pub fn merge(&mut self, other: &mut ReplicationLatency) -> &ReplicationLatency {
+        self.upload_histogram.merge(&other.upload_histogram);
+        self
+    }
+
+    // Get upload delay (categorized by object size interval)
+    pub fn get_upload_latency(&mut self) -> HashMap<String, u64> {
+        let mut ret = HashMap::new();
+        let avg = self.upload_histogram.get_avg_data();
+        for (i, v) in avg.iter().enumerate() {
+            let avg_duration = v.avg();
+            ret.insert(self.size_tag_to_string(i), avg_duration.as_millis() as u64);
+        }
+        ret
+    }
+    pub fn update(&mut self, size: i64, during: std::time::Duration) {
+        self.upload_histogram.add(size, during);
+    }
+
+    // Simulate the conversion from size tag to string
+    fn size_tag_to_string(&self, tag: usize) -> String {
+        match tag {
+            0 => String::from("Size < 1 KiB"),
+            1 => String::from("Size < 1 MiB"),
+            2 => String::from("Size < 10 MiB"),
+            3 => String::from("Size < 100 MiB"),
+            4 => String::from("Size < 1 GiB"),
+            _ => String::from("Size > 1 GiB"),
+        }
+    }
+}
+
+// #[derive(Debug, Clone, Default)]
+// pub struct ReplicationLastMinute {
+//     pub last_minute: LastMinuteLatency,
+// }
+
+// impl ReplicationLastMinute {
+//     pub fn merge(&mut self, other: ReplicationLastMinute) -> ReplicationLastMinute {
+//         let mut nl = ReplicationLastMinute::default();
+//         nl.last_minute = self.last_minute.merge(&mut other.last_minute);
+//         nl
+//     }
+
+//     pub fn add_size(&mut self, n: i64) {
+//         let t = SystemTime::now()
+//             .duration_since(UNIX_EPOCH)
+//             .expect("Time went backwards")
+//             .as_secs();
+//         self.last_minute.add_all(t - 1, &AccElem { total: t - 1, size: n as u64, n: 1 });
+//     }
+
+//     pub fn get_total(&self) -> AccElem {
+//         self.last_minute.get_total()
+//     }
+// }
+
+// impl fmt::Display for ReplicationLastMinute {
+//     fn fmt(&self, f: &mut fmt::Formatter) -> fmt::Result {
+//         let t = self.last_minute.get_total();
+//         write!(f, "ReplicationLastMinute sz= {}, n= {}, dur= {}", t.size, t.n, t.total)
+//     }
+// }

crates/common/src/heal_channel.rs

@@ -12,6 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use s3s::dto::{BucketLifecycleConfiguration, ExpirationStatus, LifecycleRule, ReplicationConfiguration, ReplicationRuleStatus};
 use serde::{Deserialize, Serialize};
 use std::{
     fmt::{self, Display},
@@ -223,13 +224,6 @@ pub struct HealOpts {
 pub enum HealAdmissionDropReason {
     QueueFull,
     PolicyDropped,
-    /// HS-06: an admin heal start overlaps (same bucket with mutually
-    /// containing prefixes, or the same erasure set) an already running or
-    /// queued task. Only produced when RUSTFS_HEAL_OVERLAP_POLICY=minio_error.
-    AlreadyRunning,
-    /// HS-06: same as [`Self::AlreadyRunning`] but for paths that merely
-    /// contain (or are contained by) the active task's path.
-    OverlappingPaths,
 }
 
 impl HealAdmissionDropReason {
@@ -237,8 +231,6 @@ impl HealAdmissionDropReason {
         match self {
             Self::QueueFull => "queue_full",
             Self::PolicyDropped => "policy_dropped",
-            Self::AlreadyRunning => "already_running",
-            Self::OverlappingPaths => "overlapping_paths",
         }
     }
 }
@@ -295,9 +287,6 @@ pub enum HealRequestSource {
     Scanner,
     AutoHeal,
     ReadRepair,
-    /// Mission Repair Feed: intents delivered by error paths and replayed
-    /// from the durable MRF journal.
-    Mrf,
 }
 
 impl HealRequestSource {
@@ -308,7 +297,6 @@ impl HealRequestSource {
             Self::Scanner => "scanner",
             Self::AutoHeal => "auto_heal",
             Self::ReadRepair => "read_repair",
-            Self::Mrf => "mrf",
         }
     }
 }
@@ -325,9 +313,6 @@ pub enum HealChannelCommand {
     Query {
         heal_path: String,
         client_token: String,
-        /// Incremental result cursor (HS-06): only items with a sequence
-        /// greater than this are returned; `None` keeps the full snapshot.
-        since_seq: Option<u64>,
         response_tx: oneshot::Sender<Result<HealChannelResponse, String>>,
     },
     /// Cancel heal task
@@ -533,21 +518,10 @@ async fn receive_heal_channel_response(
 
 /// Send heal query request
 pub async fn query_heal_status(heal_path: String, client_token: String) -> Result<HealChannelResponse, String> {
-    query_heal_status_since(heal_path, client_token, None).await
-}
-
-/// Incremental heal query (HS-06): pass the client's last seen sequence
-/// number to receive only newer result items.
-pub async fn query_heal_status_since(
-    heal_path: String,
-    client_token: String,
-    since_seq: Option<u64>,
-) -> Result<HealChannelResponse, String> {
     let (response_tx, response_rx) = oneshot::channel();
     send_heal_command(HealChannelCommand::Query {
         heal_path,
         client_token,
-        since_seq,
         response_tx,
     })
     .await?;
@@ -632,6 +606,104 @@ pub fn create_heal_response(
     }
 }
 
+fn lc_get_prefix(rule: &LifecycleRule) -> String {
+    if let Some(p) = &rule.prefix {
+        return p.to_string();
+    } else if let Some(filter) = &rule.filter {
+        if let Some(p) = &filter.prefix {
+            return p.to_string();
+        } else if let Some(and) = &filter.and
+            && let Some(p) = &and.prefix
+        {
+            return p.to_string();
+        }
+    }
+
+    "".into()
+}
+
+pub fn lc_has_active_rules(config: &BucketLifecycleConfiguration, prefix: &str) -> bool {
+    if config.rules.is_empty() {
+        return false;
+    }
+
+    for rule in config.rules.iter() {
+        if rule.status == ExpirationStatus::from_static(ExpirationStatus::DISABLED) {
+            continue;
+        }
+        let rule_prefix = lc_get_prefix(rule);
+        if !prefix.is_empty() && !rule_prefix.is_empty() && !prefix.starts_with(&rule_prefix) && !rule_prefix.starts_with(prefix)
+        {
+            continue;
+        }
+
+        if let Some(e) = &rule.noncurrent_version_expiration {
+            if e.noncurrent_days.is_some() {
+                return true;
+            }
+            if let Some(true) = e.newer_noncurrent_versions.map(|d| d > 0) {
+                return true;
+            }
+        }
+
+        if rule.noncurrent_version_transitions.is_some() {
+            return true;
+        }
+        if let Some(true) = rule.expiration.as_ref().map(|e| e.date.is_some()) {
+            return true;
+        }
+
+        if let Some(true) = rule.expiration.as_ref().map(|e| e.days.is_some()) {
+            return true;
+        }
+
+        if let Some(Some(true)) = rule.expiration.as_ref().map(|e| e.expired_object_delete_marker) {
+            return true;
+        }
+
+        if let Some(true) = rule.transitions.as_ref().map(|t| !t.is_empty()) {
+            return true;
+        }
+
+        if rule.transitions.is_some() {
+            return true;
+        }
+    }
+    false
+}
+
+pub fn rep_has_active_rules(config: &ReplicationConfiguration, prefix: &str, recursive: bool) -> bool {
+    if config.rules.is_empty() {
+        return false;
+    }
+
+    for rule in config.rules.iter() {
+        if rule
+            .status
+            .eq(&ReplicationRuleStatus::from_static(ReplicationRuleStatus::DISABLED))
+        {
+            continue;
+        }
+        if !prefix.is_empty()
+            && let Some(filter) = &rule.filter
+            && let Some(r_prefix) = &filter.prefix
+            && !r_prefix.is_empty()
+        {
+            // incoming prefix must be in rule prefix
+            if !recursive && !prefix.starts_with(r_prefix) {
+                continue;
+            }
+            // If recursive, we can skip this rule if it doesn't match the tested prefix or level below prefix
+            // does not match
+            if recursive && !r_prefix.starts_with(prefix) && !prefix.starts_with(r_prefix) {
+                continue;
+            }
+        }
+        return true;
+    }
+    false
+}
+
 pub async fn send_heal_disk(set_disk_id: String, priority: Option<HealChannelPriority>) -> Result<(), String> {
     let req = HealChannelRequest {
         id: Uuid::new_v4().to_string(),

crates/common/src/last_minute.rs

@@ -13,6 +13,82 @@
 // limitations under the License.
 
 use std::time::{Duration, SystemTime, UNIX_EPOCH};
+
+#[allow(dead_code)]
+#[derive(Debug, Default)]
+struct TimedAction {
+    count: u64,
+    acc_time: u64,
+    min_time: Option<u64>,
+    max_time: Option<u64>,
+    bytes: u64,
+}
+
+#[allow(dead_code)]
+impl TimedAction {
+    // Avg returns the average time spent on the action.
+    pub fn avg(&self) -> Option<Duration> {
+        if self.count == 0 {
+            return None;
+        }
+        Some(Duration::from_nanos(self.acc_time / self.count))
+    }
+
+    // AvgBytes returns the average bytes processed.
+    pub fn avg_bytes(&self) -> u64 {
+        if self.count == 0 {
+            return 0;
+        }
+        self.bytes / self.count
+    }
+
+    // Merge other into t.
+    pub fn merge(&mut self, other: TimedAction) {
+        self.count += other.count;
+        self.acc_time += other.acc_time;
+        self.bytes += other.bytes;
+
+        if self.count == 0 {
+            self.min_time = other.min_time;
+        }
+        if let Some(other_min) = other.min_time {
+            self.min_time = self.min_time.map_or(Some(other_min), |min| Some(min.min(other_min)));
+        }
+
+        self.max_time = self
+            .max_time
+            .map_or(other.max_time, |max| Some(max.max(other.max_time.unwrap_or(0))));
+    }
+}
+
+#[allow(dead_code)]
+#[derive(Debug)]
+enum SizeCategory {
+    SizeLessThan1KiB = 0,
+    SizeLessThan1MiB,
+    SizeLessThan10MiB,
+    SizeLessThan100MiB,
+    SizeLessThan1GiB,
+    SizeGreaterThan1GiB,
+    // Add new entries here
+    SizeLastElemMarker,
+}
+
+impl std::fmt::Display for SizeCategory {
+    fn fmt(&self, f: &mut std::fmt::Formatter<'_>) -> std::fmt::Result {
+        let s = match *self {
+            SizeCategory::SizeLessThan1KiB => "SizeLessThan1KiB",
+            SizeCategory::SizeLessThan1MiB => "SizeLessThan1MiB",
+            SizeCategory::SizeLessThan10MiB => "SizeLessThan10MiB",
+            SizeCategory::SizeLessThan100MiB => "SizeLessThan100MiB",
+            SizeCategory::SizeLessThan1GiB => "SizeLessThan1GiB",
+            SizeCategory::SizeGreaterThan1GiB => "SizeGreaterThan1GiB",
+            SizeCategory::SizeLastElemMarker => "SizeLastElemMarker",
+        };
+        write!(f, "{s}")
+    }
+}
+
 #[derive(Clone, Debug, Default, Copy)]
 pub struct AccElem {
     pub total: u64,
@@ -496,3 +572,44 @@ mod tests {
         assert_eq!(total.n, 6);
     }
 }
+
+const SIZE_LAST_ELEM_MARKER: usize = 10; // Assumed marker size is 10, modify according to actual situation
+
+#[allow(dead_code)]
+#[derive(Debug, Default)]
+pub struct LastMinuteHistogram {
+    histogram: Vec<LastMinuteLatency>,
+    size: u32,
+}
+
+impl LastMinuteHistogram {
+    pub fn merge(&mut self, other: &LastMinuteHistogram) {
+        for i in 0..self.histogram.len() {
+            self.histogram[i].merge(&other.histogram[i]);
+        }
+    }
+
+    pub fn add(&mut self, size: i64, t: Duration) {
+        let index = size_to_tag(size);
+        self.histogram[index].add(&t);
+    }
+
+    pub fn get_avg_data(&mut self) -> [AccElem; SIZE_LAST_ELEM_MARKER] {
+        let mut res = [AccElem::default(); SIZE_LAST_ELEM_MARKER];
+        for (i, elem) in self.histogram.iter_mut().enumerate() {
+            res[i] = elem.get_total();
+        }
+        res
+    }
+}
+
+fn size_to_tag(size: i64) -> usize {
+    match size {
+        _ if size < 1024 => 0,               // sizeLessThan1KiB
+        _ if size < 1024 * 1024 => 1,        // sizeLessThan1MiB
+        _ if size < 10 * 1024 * 1024 => 2,   // sizeLessThan10MiB
+        _ if size < 100 * 1024 * 1024 => 3,  // sizeLessThan100MiB
+        _ if size < 1024 * 1024 * 1024 => 4, // sizeLessThan1GiB
+        _ => 5,                              // sizeGreaterThan1GiB
+    }
+}

crates/common/src/lib.rs

@@ -12,15 +12,13 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+pub mod bucket_stats;
 // pub mod error;
 pub mod globals;
 pub mod heal_channel;
 pub mod last_minute;
 pub mod metrics;
-pub mod mrf_channel;
 mod readiness;
-pub mod table_catalog;
-pub mod trace_bus;
 
 pub use globals::*;
 pub use readiness::{GlobalReadiness, SystemStage};

crates/common/src/metrics.rs

@@ -15,7 +15,6 @@
 use crate::heal_channel::HealScanMode;
 use crate::last_minute::{AccElem, LastMinuteLatency};
 use chrono::{DateTime, Utc};
-use jiff::Timestamp;
 use serde::{Deserialize, Serialize};
 use std::{
     collections::{BTreeSet, HashMap},
@@ -670,7 +669,7 @@ impl LockedLastMinuteLatency {
 #[derive(Clone, Debug)]
 struct CurrentPathState {
     path: String,
-    updated_at: Timestamp,
+    updated_at: DateTime<Utc>,
 }
 
 struct CurrentPathTracker {
@@ -679,10 +678,10 @@ struct CurrentPathTracker {
 
 impl CurrentPathTracker {
     fn new(initial_path: String) -> Self {
-        Self::new_at(initial_path, Timestamp::now())
+        Self::new_at(initial_path, Utc::now())
     }
 
-    fn new_at(initial_path: String, updated_at: Timestamp) -> Self {
+    fn new_at(initial_path: String, updated_at: DateTime<Utc>) -> Self {
         Self {
             state: Arc::new(RwLock::new(CurrentPathState {
                 path: initial_path,
@@ -694,7 +693,7 @@ impl CurrentPathTracker {
     async fn update_path(&self, path: String) {
         let mut state = self.state.write().await;
         state.path = path;
-        state.updated_at = Timestamp::now();
+        state.updated_at = Utc::now();
     }
 
     async fn get_state(&self) -> CurrentPathState {
@@ -702,36 +701,6 @@ impl CurrentPathTracker {
     }
 }
 
-fn chrono_to_jiff_timestamp(dt: DateTime<Utc>) -> Timestamp {
-    let seconds = dt.timestamp();
-    let nanoseconds = match i32::try_from(dt.timestamp_subsec_nanos()) {
-        Ok(nanoseconds) => nanoseconds,
-        Err(_) => {
-            return if seconds < 0 { Timestamp::MIN } else { Timestamp::MAX };
-        }
-    };
-
-    match Timestamp::new(seconds, nanoseconds) {
-        Ok(timestamp) => timestamp,
-        Err(_) => {
-            if seconds < 0 {
-                Timestamp::MIN
-            } else {
-                Timestamp::MAX
-            }
-        }
-    }
-}
-
-fn timestamp_elapsed_seconds_since(now: Timestamp, earlier: Timestamp) -> u64 {
-    let duration = now.duration_since(earlier);
-    if duration.is_negative() {
-        return 0;
-    }
-
-    u64::try_from(duration.as_secs()).unwrap_or(u64::MAX)
-}
-
 #[derive(Clone, Copy, Debug, Default)]
 struct ScannerDiskBucketScanState {
     concurrency_limit: u64,
@@ -781,19 +750,6 @@ struct ScannerBucketDriveResultValue {
     last_seen: u64,
 }
 
-#[derive(Clone, Debug, Eq, Hash, PartialEq)]
-struct ScannerActiveBucketDriveKey {
-    source: String,
-    bucket: String,
-    drive: String,
-}
-
-#[derive(Clone, Copy, Debug)]
-struct ScannerActiveBucketDriveValue {
-    count: u64,
-    started_at: Timestamp,
-}
-
 // ---------------------------------------------------------------------------
 // Metrics
 // ---------------------------------------------------------------------------
@@ -826,7 +782,6 @@ pub struct Metrics {
     scanner_set_scans_active: AtomicU64,
     scanner_disk_bucket_scan_states: Mutex<HashMap<ScannerDiskBucketScanKey, ScannerDiskBucketScanState>>,
     scanner_bucket_drive_results: Mutex<ScannerBucketDriveResults>,
-    scanner_active_bucket_drive_scans: Mutex<HashMap<ScannerActiveBucketDriveKey, ScannerActiveBucketDriveValue>>,
     scanner_bucket_drive_result_clock: AtomicU64,
     current_scan_cycle_bucket_drive_results_start: Mutex<HashMap<ScannerBucketDriveResultKey, u64>>,
     last_scan_cycle_bucket_drive_results: Mutex<Vec<ScannerBucketDriveResultSnapshot>>,
@@ -929,13 +884,11 @@ const SCAN_CYCLE_RESULT_SUCCESS: u8 = 1;
 const SCAN_CYCLE_RESULT_ERROR: u8 = 2;
 const SCAN_CYCLE_RESULT_PARTIAL: u8 = 3;
 const SCAN_CYCLE_RESULT_SUPERSEDED: u8 = 4;
-const SCAN_CYCLE_RESULT_DEFERRED: u8 = 5;
 const SCAN_CYCLE_RESULT_UNKNOWN_LABEL: &str = "unknown";
 const SCAN_CYCLE_RESULT_SUCCESS_LABEL: &str = "success";
 const SCAN_CYCLE_RESULT_ERROR_LABEL: &str = "error";
 const SCAN_CYCLE_RESULT_PARTIAL_LABEL: &str = "partial";
 const SCAN_CYCLE_RESULT_SUPERSEDED_LABEL: &str = "superseded";
-const SCAN_CYCLE_RESULT_DEFERRED_LABEL: &str = "deferred";
 
 #[derive(Clone, Copy, Debug, Default, PartialEq, Eq)]
 pub enum ScanCyclePartialReason {
@@ -1059,15 +1012,6 @@ pub struct ScannerBucketDriveResultSnapshot {
     pub count: u64,
 }
 
-#[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
-pub struct ScannerActiveBucketDriveSnapshot {
-    pub source: String,
-    pub bucket: String,
-    pub drive: String,
-    pub count: u64,
-    pub age_seconds: u64,
-}
-
 #[derive(Clone, Debug, Default, Serialize, Deserialize, PartialEq, Eq)]
 pub struct ScannerReplicationRepairSnapshot {
     pub source: String,
@@ -1222,12 +1166,12 @@ pub struct ScannerLastMinute {
 
 #[derive(Clone, Debug, Default, Serialize, Deserialize)]
 pub struct ScannerMetricsReport {
-    pub collected_at: Timestamp,
+    pub collected_at: DateTime<Utc>,
     pub current_cycle: u64,
     #[serde(default)]
     pub current_cycle_active: bool,
-    pub current_started: Timestamp,
-    pub cycles_completed_at: Vec<Timestamp>,
+    pub current_started: DateTime<Utc>,
+    pub cycles_completed_at: Vec<DateTime<Utc>>,
     pub ongoing_buckets: usize,
     #[serde(default)]
     pub active_scan_paths: usize,
@@ -1410,8 +1354,6 @@ pub struct ScannerRuntimeDetailsReport {
     pub current_cycle_bucket_drive_results: Vec<ScannerBucketDriveResultSnapshot>,
     #[serde(default)]
     pub last_cycle_bucket_drive_results: Vec<ScannerBucketDriveResultSnapshot>,
-    #[serde(default)]
-    pub active_bucket_drive_scans: Vec<ScannerActiveBucketDriveSnapshot>,
 }
 
 impl CurrentCycle {
@@ -1426,18 +1368,31 @@ impl CurrentCycle {
 }
 
 /// OTEL metric name constants for scanner metrics
+const OTEL_SCANNER_OBJECTS_SCANNED: &str = "rustfs_scanner_objects_scanned_total";
+const OTEL_SCANNER_DIRECTORIES_SCANNED: &str = "rustfs_scanner_directories_scanned_total";
 const OTEL_SCANNER_BUCKETS_SCANNED: &str = "rustfs_scanner_buckets_scanned_total";
 const OTEL_SCANNER_CYCLES: &str = "rustfs_scanner_cycles_total";
 const OTEL_SCANNER_CYCLE_DURATION_SECONDS: &str = "rustfs_scanner_cycle_duration_seconds";
 const OTEL_SCANNER_BUCKET_DRIVE_DURATION_SECONDS: &str = "rustfs_scanner_bucket_drive_duration_seconds";
 
+fn emit_otel_counter(metric: usize, count: u64) {
+    match Metric::from_index(metric) {
+        Some(Metric::ScanObject) => {
+            metrics::counter!(OTEL_SCANNER_OBJECTS_SCANNED).increment(count);
+        }
+        Some(Metric::ScanFolder) => {
+            metrics::counter!(OTEL_SCANNER_DIRECTORIES_SCANNED).increment(count);
+        }
+        _ => {}
+    }
+}
+
 fn scan_cycle_result_label(result: u8) -> &'static str {
     match result {
         SCAN_CYCLE_RESULT_SUCCESS => SCAN_CYCLE_RESULT_SUCCESS_LABEL,
         SCAN_CYCLE_RESULT_ERROR => SCAN_CYCLE_RESULT_ERROR_LABEL,
         SCAN_CYCLE_RESULT_PARTIAL => SCAN_CYCLE_RESULT_PARTIAL_LABEL,
         SCAN_CYCLE_RESULT_SUPERSEDED => SCAN_CYCLE_RESULT_SUPERSEDED_LABEL,
-        SCAN_CYCLE_RESULT_DEFERRED => SCAN_CYCLE_RESULT_DEFERRED_LABEL,
         _ => SCAN_CYCLE_RESULT_UNKNOWN_LABEL,
     }
 }
@@ -1766,12 +1721,7 @@ pub fn emit_scan_cycle_superseded(duration: Duration) {
     metrics::counter!(OTEL_SCANNER_CYCLES, "result" => SCAN_CYCLE_RESULT_SUPERSEDED_LABEL).increment(1);
 }
 
-pub fn emit_scan_cycle_deferred(duration: Duration) {
-    global_metrics().record_scan_cycle_deferred(duration);
-    metrics::counter!(OTEL_SCANNER_CYCLES, "result" => SCAN_CYCLE_RESULT_DEFERRED_LABEL).increment(1);
-}
-
-pub fn emit_scan_bucket_drive_complete(_source: ScannerWorkSource, success: bool, bucket: &str, disk: &str, duration: Duration) {
+pub fn emit_scan_bucket_drive_complete(success: bool, bucket: &str, disk: &str, duration: Duration) {
     let result = if success { "success" } else { "error" };
     global_metrics().record_scanner_bucket_drive_result(bucket, disk, result);
     metrics::counter!(
@@ -1789,7 +1739,7 @@ pub fn emit_scan_bucket_drive_complete(_source: ScannerWorkSource, success: bool
     .record(duration.as_secs_f64());
 }
 
-pub fn emit_scan_bucket_drive_partial(_source: ScannerWorkSource, bucket: &str, disk: &str, duration: Duration) {
+pub fn emit_scan_bucket_drive_partial(bucket: &str, disk: &str, duration: Duration) {
     global_metrics().record_scanner_bucket_drive_result(bucket, disk, SCAN_CYCLE_RESULT_PARTIAL_LABEL);
     metrics::counter!(
         OTEL_SCANNER_BUCKETS_SCANNED,
@@ -1842,7 +1792,6 @@ impl Metrics {
             scanner_set_scans_active: AtomicU64::new(0),
             scanner_disk_bucket_scan_states: Mutex::new(HashMap::new()),
             scanner_bucket_drive_results: Mutex::new(ScannerBucketDriveResults::default()),
-            scanner_active_bucket_drive_scans: Mutex::new(HashMap::new()),
             scanner_bucket_drive_result_clock: AtomicU64::new(0),
             current_scan_cycle_bucket_drive_results_start: Mutex::new(HashMap::new()),
             last_scan_cycle_bucket_drive_results: Mutex::new(Vec::new()),
@@ -1972,6 +1921,7 @@ impl Metrics {
             let duration = SystemTime::now().duration_since(start).unwrap_or_default();
             global_metrics().operations[metric_idx].fetch_add(1, Ordering::Relaxed);
             global_metrics().record_source_work_for_metric(metric, 1);
+            emit_otel_counter(metric_idx, 1);
             if metric_idx < Metric::LastRealtime as usize {
                 global_metrics().latency[metric_idx].add(duration);
             }
@@ -1987,6 +1937,7 @@ impl Metrics {
             let duration = SystemTime::now().duration_since(start).unwrap_or_default();
             global_metrics().operations[metric_idx].fetch_add(1, Ordering::Relaxed);
             global_metrics().record_source_work_for_metric(metric, 1);
+            emit_otel_counter(metric_idx, 1);
             if metric_idx < Metric::LastRealtime as usize {
                 global_metrics().latency[metric_idx].add_size(duration, size);
             }
@@ -2002,6 +1953,7 @@ impl Metrics {
             let duration = SystemTime::now().duration_since(start).unwrap_or_default();
             global_metrics().operations[metric_idx].fetch_add(1, Ordering::Relaxed);
             global_metrics().record_source_work_for_metric(metric, 1);
+            emit_otel_counter(metric_idx, 1);
             if metric_idx < Metric::LastRealtime as usize {
                 global_metrics().latency[metric_idx].add(duration);
             }
@@ -2019,6 +1971,7 @@ impl Metrics {
                 let count = usize_to_u64_saturated(count);
                 global_metrics().operations[metric_idx].fetch_add(count, Ordering::Relaxed);
                 global_metrics().record_source_work_for_metric(metric, count);
+                emit_otel_counter(metric_idx, count);
                 if metric_idx < Metric::LastRealtime as usize {
                     global_metrics().latency[metric_idx].add(duration);
                 }
@@ -2039,6 +1992,7 @@ impl Metrics {
                 let duration = SystemTime::now().duration_since(start).unwrap_or_default();
                 let metric_idx = Metric::Ilm as usize;
                 global_metrics().operations[metric_idx].fetch_add(versions, Ordering::Relaxed);
+                emit_otel_counter(metric_idx, versions);
                 global_metrics().actions[a_idx].fetch_add(versions, Ordering::Relaxed);
                 global_metrics().actions_latency[a_idx].add(duration);
             })
@@ -2051,6 +2005,7 @@ impl Metrics {
         let metric_idx = metric as usize;
         global_metrics().operations[metric_idx].fetch_add(1, Ordering::Relaxed);
         global_metrics().record_source_work_for_metric(metric, 1);
+        emit_otel_counter(metric_idx, 1);
         if metric_idx < Metric::LastRealtime as usize {
             global_metrics().latency[metric_idx].add(duration);
         }
@@ -2334,45 +2289,8 @@ impl Metrics {
         }
     }
 
-    pub fn record_scan_bucket_drive_start(&self, source: ScannerWorkSource, bucket: &str, drive: &str) {
+    pub fn record_scan_bucket_drive_start(&self) {
         self.operations[Metric::ScanBucketDriveStart as usize].fetch_add(1, Ordering::Relaxed);
-        if bucket.is_empty() || drive.is_empty() {
-            return;
-        }
-        let key = ScannerActiveBucketDriveKey {
-            source: source.as_str().to_string(),
-            bucket: bucket.to_string(),
-            drive: drive.to_string(),
-        };
-        let mut active = self
-            .scanner_active_bucket_drive_scans
-            .lock()
-            .unwrap_or_else(|poisoned| poisoned.into_inner());
-        active
-            .entry(key)
-            .and_modify(|value| value.count = value.count.saturating_add(1))
-            .or_insert(ScannerActiveBucketDriveValue {
-                count: 1,
-                started_at: Timestamp::now(),
-            });
-    }
-
-    pub fn record_scan_bucket_drive_end(&self, source: ScannerWorkSource, bucket: &str, drive: &str) {
-        let key = ScannerActiveBucketDriveKey {
-            source: source.as_str().to_string(),
-            bucket: bucket.to_string(),
-            drive: drive.to_string(),
-        };
-        let mut active = self
-            .scanner_active_bucket_drive_scans
-            .lock()
-            .unwrap_or_else(|poisoned| poisoned.into_inner());
-        if let Some(value) = active.get_mut(&key) {
-            value.count = value.count.saturating_sub(1);
-            if value.count == 0 {
-                active.remove(&key);
-            }
-        }
     }
 
     pub fn record_scan_bucket_drive_failure(&self) {
@@ -2600,17 +2518,6 @@ impl Metrics {
             .store(duration_millis_saturated(duration), Ordering::Relaxed);
     }
 
-    pub fn record_scan_cycle_deferred(&self, duration: Duration) {
-        self.record_scanner_cycle_end_time();
-        self.last_scan_cycle_result
-            .store(SCAN_CYCLE_RESULT_DEFERRED, Ordering::Relaxed);
-        self.last_scan_cycle_partial_reason
-            .store(ScanCyclePartialReason::Unknown as u8, Ordering::Relaxed);
-        self.last_scan_cycle_partial_source.store(0, Ordering::Relaxed);
-        self.last_scan_cycle_duration_millis
-            .store(duration_millis_saturated(duration), Ordering::Relaxed);
-    }
-
     pub fn record_scan_cycle_partial(&self, duration: Duration, reason: ScanCyclePartialReason) {
         self.record_scan_cycle_partial_with_source(duration, reason, None);
     }
@@ -2845,26 +2752,6 @@ impl Metrics {
         } else {
             Vec::new()
         };
-        let now = Timestamp::now();
-        let mut active_bucket_drive_scans = self
-            .scanner_active_bucket_drive_scans
-            .lock()
-            .unwrap_or_else(|poisoned| poisoned.into_inner())
-            .iter()
-            .map(|(key, value)| ScannerActiveBucketDriveSnapshot {
-                source: key.source.clone(),
-                bucket: key.bucket.clone(),
-                drive: key.drive.clone(),
-                count: value.count,
-                age_seconds: timestamp_elapsed_seconds_since(now, value.started_at),
-            })
-            .collect::<Vec<_>>();
-        active_bucket_drive_scans.sort_by(|left, right| {
-            left.source
-                .cmp(&right.source)
-                .then_with(|| left.bucket.cmp(&right.bucket))
-                .then_with(|| left.drive.cmp(&right.drive))
-        });
         ScannerRuntimeDetailsReport {
             disk_bucket_scan_states: self.scanner_disk_bucket_scan_state_snapshots(),
             bucket_drive_results: self.scanner_bucket_drive_result_counter_snapshots(),
@@ -2874,7 +2761,6 @@ impl Metrics {
                 .lock()
                 .unwrap_or_else(|poisoned| poisoned.into_inner())
                 .clone(),
-            active_bucket_drive_scans,
         }
     }
 
@@ -3102,8 +2988,8 @@ impl Metrics {
             let cycle = self.cycle_info.read().await;
             let has_cycle = if let Some(cycle) = cycle.as_ref() {
                 m.current_cycle = cycle.current;
-                m.cycles_completed_at = cycle.cycle_completed.iter().copied().map(chrono_to_jiff_timestamp).collect();
-                m.current_started = chrono_to_jiff_timestamp(cycle.started);
+                m.cycles_completed_at = cycle.cycle_completed.clone();
+                m.current_started = cycle.started;
                 true
             } else {
                 false
@@ -3138,15 +3024,15 @@ impl Metrics {
         };
 
         if !has_cycle && let Some(init_time) = crate::get_global_init_time().await {
-            m.current_started = chrono_to_jiff_timestamp(init_time);
+            m.current_started = init_time;
         }
 
-        m.collected_at = Timestamp::now();
+        m.collected_at = Utc::now();
         let current_path_snapshots = self.current_path_snapshots().await;
         m.active_scan_paths = current_path_snapshots.len();
         m.oldest_active_path_age_seconds = current_path_snapshots
             .iter()
-            .map(|(_, state)| timestamp_elapsed_seconds_since(m.collected_at, state.updated_at))
+            .map(|(_, state)| m.collected_at.signed_duration_since(state.updated_at).num_seconds().max(0) as u64)
             .max()
             .unwrap_or_default();
         m.active_paths = current_path_snapshots
@@ -3422,22 +3308,6 @@ impl Drop for CloseDiskGuard {
 mod tests {
     use super::*;
 
-    #[test]
-    fn scanner_metrics_report_timestamps_serialize_as_rfc3339_utc() {
-        let report = ScannerMetricsReport {
-            collected_at: Timestamp::constant(1_700_000_000, 123_456_000),
-            current_started: Timestamp::constant(1_699_999_940, 0),
-            cycles_completed_at: vec![Timestamp::constant(1_700_000_060, 987_654_000)],
-            ..Default::default()
-        };
-
-        let value = serde_json::to_value(&report).expect("scanner metrics report should serialize");
-
-        assert_eq!(value["collected_at"].as_str(), Some("2023-11-14T22:13:20.123456Z"));
-        assert_eq!(value["current_started"].as_str(), Some("2023-11-14T22:12:20Z"));
-        assert_eq!(value["cycles_completed_at"][0].as_str(), Some("2023-11-14T22:14:20.987654Z"));
-    }
-
     #[tokio::test]
     async fn close_disk_guard_runs_cleanup_when_an_early_return_drops_it() {
         let (closed_tx, closed_rx) = tokio::sync::oneshot::channel();
@@ -3496,7 +3366,7 @@ mod tests {
     #[tokio::test]
     async fn report_counts_active_scan_paths() {
         let metrics = Metrics::new();
-        let updated_at = Timestamp::now() - jiff::SignedDuration::from_secs(12);
+        let updated_at = Utc::now() - chrono::Duration::seconds(12);
         metrics.current_paths.write().await.insert(
             "disk-a".to_string(),
             Arc::new(CurrentPathTracker::new_at("bucket-a".to_string(), updated_at)),
@@ -3518,7 +3388,7 @@ mod tests {
         let metrics = Metrics::new();
         let tracker = Arc::new(CurrentPathTracker::new_at(
             "bucket-a".to_string(),
-            Timestamp::now() - jiff::SignedDuration::from_secs(60 * 60),
+            Utc::now() - chrono::Duration::hours(1),
         ));
         metrics
             .current_paths
@@ -4291,7 +4161,7 @@ mod tests {
         let report = metrics.report().await;
         *crate::globals::GLOBAL_INIT_TIME.write().await = previous_init_time;
 
-        assert_eq!(report.current_started, chrono_to_jiff_timestamp(cycle_started));
+        assert_eq!(report.current_started, cycle_started);
     }
 
     #[tokio::test]
@@ -4347,21 +4217,6 @@ mod tests {
         assert_eq!(report.partial_cycles, 0);
     }
 
-    #[tokio::test]
-    async fn report_tracks_deferred_cycle_without_failed_increment() {
-        let metrics = Metrics::new();
-        metrics.record_scan_cycle_deferred(Duration::from_millis(250));
-
-        let report = metrics.report().await;
-
-        assert_eq!(report.last_cycle_result, SCAN_CYCLE_RESULT_DEFERRED_LABEL);
-        assert_eq!(report.last_cycle_result_code, u64::from(SCAN_CYCLE_RESULT_DEFERRED));
-        assert_eq!(report.last_cycle_duration_seconds, 0.25);
-        assert_eq!(report.failed_cycles, 0);
-        assert_eq!(report.superseded_cycles, 0);
-        assert_eq!(report.partial_cycles, 0);
-    }
-
     #[tokio::test]
     async fn report_tracks_successful_scan_cycle_without_failed_increment() {
         let metrics = Metrics::new();
@@ -4455,7 +4310,7 @@ mod tests {
     #[tokio::test]
     async fn report_includes_bucket_drive_scan_starts() {
         let metrics = Metrics::new();
-        metrics.record_scan_bucket_drive_start(ScannerWorkSource::Usage, "bucket-a", "/mnt/data/1");
+        metrics.record_scan_bucket_drive_start();
         metrics.record_scan_bucket_drive_failure();
 
         let report = metrics.report().await;
@@ -4464,27 +4319,6 @@ mod tests {
         assert_eq!(report.life_time_ops.get("scan_bucket_drive_failure"), Some(&1));
     }
 
-    #[tokio::test]
-    async fn active_bucket_drive_snapshot_is_structured_and_retired_on_end() {
-        let metrics = Metrics::new();
-        metrics.record_scan_bucket_drive_start(ScannerWorkSource::Usage, "bucket-a", "/mnt/data/1");
-        metrics.record_scan_bucket_drive_start(ScannerWorkSource::Usage, "bucket-a", "/mnt/data/1");
-        let active = metrics.scanner_runtime_details_report().active_bucket_drive_scans;
-        assert_eq!(active.len(), 1);
-        assert_eq!(active[0].source, ScannerWorkSource::Usage.as_str());
-        assert_eq!(active[0].bucket, "bucket-a");
-        assert_eq!(active[0].drive, "/mnt/data/1");
-        assert_eq!(active[0].count, 2);
-
-        metrics.record_scan_bucket_drive_end(ScannerWorkSource::Usage, "bucket-a", "/mnt/data/1");
-        assert_eq!(metrics.scanner_runtime_details_report().active_bucket_drive_scans[0].count, 1);
-        metrics.record_scan_bucket_drive_end(ScannerWorkSource::Usage, "bucket-a", "/mnt/data/1");
-        assert!(metrics.scanner_runtime_details_report().active_bucket_drive_scans.is_empty());
-
-        metrics.record_scan_bucket_drive_start(ScannerWorkSource::Usage, "", "/mnt/data/1");
-        assert!(metrics.scanner_runtime_details_report().active_bucket_drive_scans.is_empty());
-    }
-
     #[tokio::test]
     async fn report_includes_structured_bucket_drive_results() {
         let metrics = Metrics::new();
@@ -4750,7 +4584,7 @@ mod tests {
         let active = metrics.report().await;
         assert!(active.current_cycle_active);
         assert_eq!(active.current_cycle, 12);
-        assert_eq!(active.current_started, chrono_to_jiff_timestamp(cycle_started));
+        assert_eq!(active.current_started, cycle_started);
 
         let idle_cycle = CurrentCycle {
             current: 0,

crates/common/src/mrf_channel.rs

@@ -1,287 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Mission Repair Feed (MRF) intent channel.
-//!
-//! Producers on error paths (read decode failure, scanner metadata
-//! corruption, partial-write recovery) hand a lightweight [`MrfIntent`] to the
-//! heal crate through a global bounded channel. Delivery is strictly
-//! non-blocking: `try_send_mrf_intent` never awaits and drops the intent
-//! (counting it) when the channel is full or uninitialized — losing one heal
-//! hint is always preferred over stalling an IO path. Durable replay of
-//! unconsumed intents is the consumer's job (see `rustfs-heal`
-//! `heal::mrf_queue`), mirroring MinIO's `.heal/mrf/list.bin`.
-
-use std::sync::{
-    Arc, OnceLock,
-    atomic::{AtomicBool, Ordering},
-};
-use tokio::sync::mpsc;
-use uuid::Uuid;
-
-/// Bounded capacity of the global MRF channel. Backpressure is resolved by
-/// dropping (and counting) intents, never by blocking the producer.
-const MRF_CHANNEL_CAPACITY: usize = 8192;
-
-/// Why an intent was produced. Drives the heal priority mapping on the
-/// consumer side (DecodeFailure -> Urgent, MetadataCorruption -> High,
-/// PartialWrite -> Normal).
-#[derive(Clone, Copy, Debug, PartialEq, Eq)]
-pub enum MrfKind {
-    /// Erasure decode failed while serving a read (read path).
-    DecodeFailure,
-    /// Scanner classified object metadata as corrupt.
-    MetadataCorruption,
-    /// A write left the object with fewer committed shards than the set size.
-    PartialWrite,
-}
-
-impl MrfKind {
-    pub const fn as_str(self) -> &'static str {
-        match self {
-            MrfKind::DecodeFailure => "decode-failure",
-            MrfKind::MetadataCorruption => "metadata-corruption",
-            MrfKind::PartialWrite => "partial-write",
-        }
-    }
-}
-
-/// One repair intent. Kept deliberately small so the in-memory queue and the
-/// journal stay bounded; `bucket`/`object` are `Arc<str>` so re-arming an
-/// intent never re-allocates the strings.
-#[derive(Clone, Debug)]
-pub struct MrfIntent {
-    pub bucket: Arc<str>,
-    pub object: Arc<str>,
-    /// Version the intent targets, as raw UUID bytes.
-    pub version_id: Option<[u8; 16]>,
-    pub kind: MrfKind,
-    pub enqueued_at_ms: u64,
-    /// Times this intent has already been offered to the heal manager.
-    /// Dropped by the consumer once it reaches `MRF_MAX_ATTEMPTS`.
-    pub attempts: u8,
-}
-
-/// Consumer-side retry ceiling before an intent is given up on.
-pub const MRF_MAX_ATTEMPTS: u8 = 3;
-
-impl MrfIntent {
-    /// Rough in-memory footprint used by the queue's byte budget.
-    pub fn estimated_bytes(&self) -> usize {
-        // Struct + strings + version bytes; buckets and objects are usually
-        // far below this bound, so rounding up keeps the budget conservative.
-        64 + self.bucket.len() + self.object.len()
-    }
-}
-
-static GLOBAL_MRF_SENDER: OnceLock<mpsc::Sender<MrfIntent>> = OnceLock::new();
-
-/// Delivery kill-switch, set from `RUSTFS_HEAL_MRF_ENABLE`. Producers check
-/// this before touching the channel so the disabled path stays allocation- and
-/// sync-free.
-static MRF_DELIVERY_ENABLED: AtomicBool = AtomicBool::new(true);
-
-/// Override delivery (used at heal-runtime startup from configuration).
-pub fn set_mrf_delivery_enabled(enabled: bool) {
-    MRF_DELIVERY_ENABLED.store(enabled, Ordering::Relaxed);
-}
-
-/// Whether producers currently deliver intents.
-pub fn mrf_delivery_enabled() -> bool {
-    MRF_DELIVERY_ENABLED.load(Ordering::Relaxed)
-}
-
-/// Create the global MRF channel and return the consumer half. Fails if the
-/// channel is already initialized (the heal runtime is a singleton).
-pub fn init_mrf_channel() -> Result<mpsc::Receiver<MrfIntent>, &'static str> {
-    let (sender, receiver) = mpsc::channel(MRF_CHANNEL_CAPACITY);
-    GLOBAL_MRF_SENDER
-        .set(sender)
-        .map_err(|_| "MRF channel sender already initialized")?;
-    Ok(receiver)
-}
-
-/// Best-effort, non-blocking intent delivery from an error path.
-///
-/// Returns `true` when the intent was accepted into the channel. `false`
-/// means the intent was dropped (feature disabled, channel not yet
-/// initialized, or channel full) — callers must not retry or await; the
-/// existing read-repair / scanner heal paths remain the safety net.
-///
-/// This runs on IO error paths, so it stays synchronous and cheap: one
-/// bounded allocation for the two `Arc<str>` handles plus the channel slot.
-pub fn try_send_mrf_intent(kind: MrfKind, bucket: &str, object: &str, version_id: Option<Uuid>) -> bool {
-    if !mrf_delivery_enabled() {
-        return false;
-    }
-    let Some(sender) = GLOBAL_MRF_SENDER.get() else {
-        return false;
-    };
-    let intent = MrfIntent {
-        bucket: Arc::from(bucket),
-        object: Arc::from(object),
-        version_id: version_id.map(|vid| *vid.as_bytes()),
-        kind,
-        enqueued_at_ms: unix_now_ms(),
-        attempts: 0,
-    };
-    sender.try_send(intent).is_ok()
-}
-
-fn unix_now_ms() -> u64 {
-    // Kept trivial: the timestamp is diagnostic metadata only; wall-clock
-    // failure would be a bug rather than something to handle here.
-    std::time::SystemTime::now()
-        .duration_since(std::time::UNIX_EPOCH)
-        .map(|d| d.as_millis() as u64)
-        .unwrap_or(0)
-}
-
-/// A repair the MRF consumer landed, fanned out so retry ledgers can drop
-/// entries the journal no longer tracks (backlog#1894 axis B). The payload
-/// mirrors the intent identity so consumers match without re-parsing.
-#[derive(Debug, Clone, PartialEq, Eq)]
-pub struct MrfRepairedEvent {
-    pub bucket: Arc<str>,
-    pub object: Arc<str>,
-    pub version_id: Option<[u8; 16]>,
-}
-
-/// Bound on the repaired-event backlog. Notices are best-effort hints; when
-/// the ring is full the oldest are dropped and the affected ledger entries
-/// simply expire through their own attempts/age limits.
-const MRF_REPAIRED_EVENT_CAP: usize = 4096;
-
-static MRF_REPAIRED_EVENTS: OnceLock<std::sync::Mutex<std::collections::VecDeque<MrfRepairedEvent>>> = OnceLock::new();
-
-/// Record that the MRF consumer landed a repair. Never blocks: the critical
-/// section is a deque push under a std mutex.
-pub fn note_mrf_repaired(bucket: &str, object: &str, version_id: Option<[u8; 16]>) {
-    let registry = MRF_REPAIRED_EVENTS.get_or_init(|| std::sync::Mutex::new(std::collections::VecDeque::new()));
-    let Ok(mut events) = registry.lock() else {
-        return;
-    };
-    if events.len() >= MRF_REPAIRED_EVENT_CAP {
-        events.pop_front();
-    }
-    events.push_back(MrfRepairedEvent {
-        bucket: Arc::from(bucket),
-        object: Arc::from(object),
-        version_id,
-    });
-}
-
-/// Take the repair notices recorded for `bucket`, leaving other buckets'
-/// notices in place for their own scanners.
-pub fn take_mrf_repaired_events_for(bucket: &str) -> Vec<MrfRepairedEvent> {
-    let Some(registry) = MRF_REPAIRED_EVENTS.get() else {
-        return Vec::new();
-    };
-    let Ok(mut events) = registry.lock() else {
-        return Vec::new();
-    };
-    let mut taken = Vec::new();
-    let mut retained = std::collections::VecDeque::with_capacity(events.len());
-    while let Some(event) = events.pop_front() {
-        if event.bucket.as_ref() == bucket {
-            taken.push(event);
-        } else {
-            retained.push_back(event);
-        }
-    }
-    *events = retained;
-    taken
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn intents_estimate_is_conservative() {
-        let intent = MrfIntent {
-            bucket: Arc::from("bucket"),
-            object: Arc::from("object"),
-            version_id: Some([0u8; 16]),
-            kind: MrfKind::DecodeFailure,
-            enqueued_at_ms: 0,
-            attempts: 0,
-        };
-        assert!(intent.estimated_bytes() >= intent.bucket.len() + intent.object.len());
-    }
-
-    #[tokio::test]
-    async fn try_send_delivers_and_respects_capacity() {
-        let mut receiver = init_mrf_channel().expect("first initialization should succeed");
-        assert!(init_mrf_channel().is_err(), "double initialization must fail");
-
-        assert!(try_send_mrf_intent(MrfKind::DecodeFailure, "b", "o", Some(Uuid::nil())));
-        let intent = receiver.recv().await.expect("intent should arrive");
-        assert_eq!(intent.kind, MrfKind::DecodeFailure);
-        assert_eq!(intent.bucket.as_ref(), "b");
-
-        // Disable delivery: producers become no-ops.
-        set_mrf_delivery_enabled(false);
-        assert!(!try_send_mrf_intent(MrfKind::PartialWrite, "b", "o", None));
-        set_mrf_delivery_enabled(true);
-
-        // Fill the bounded channel past capacity: excess intents are dropped,
-        // never blocking.
-        let mut accepted = 0;
-        for _ in 0..(MRF_CHANNEL_CAPACITY + 64) {
-            if try_send_mrf_intent(MrfKind::PartialWrite, "b", "o", None) {
-                accepted += 1;
-            }
-        }
-        assert_eq!(accepted, MRF_CHANNEL_CAPACITY);
-    }
-
-    #[test]
-    fn try_send_without_channel_is_false() {
-        // This test may run after the tokio test above in the same process;
-        // the singleton semantics make a clean "uninitialized" case hard, so
-        // assert the flag-off behavior only.
-        set_mrf_delivery_enabled(false);
-        assert!(!try_send_mrf_intent(MrfKind::MetadataCorruption, "b", "o", None));
-        set_mrf_delivery_enabled(true);
-    }
-
-    #[test]
-    fn repaired_events_take_is_bucket_scoped_and_cap_bounded() {
-        // Distinct buckets keep their notices until their own scanner takes
-        // them; a take for one bucket leaves the others' notices in place.
-        note_mrf_repaired("bucket-a", "object-1", None);
-        note_mrf_repaired("bucket-b", "object-2", None);
-        note_mrf_repaired("bucket-a", "object-3", None);
-
-        let taken_a = take_mrf_repaired_events_for("bucket-a");
-        assert_eq!(taken_a.len(), 2);
-        assert_eq!(taken_a[0].object.as_ref(), "object-1");
-        assert_eq!(taken_a[1].object.as_ref(), "object-3");
-        assert!(take_mrf_repaired_events_for("bucket-a").is_empty(), "take is destructive per bucket");
-
-        let taken_b = take_mrf_repaired_events_for("bucket-b");
-        assert_eq!(taken_b.len(), 1);
-        assert_eq!(taken_b[0].object.as_ref(), "object-2");
-
-        // Cap bound: flooding the ring drops the oldest notices rather than
-        // growing unbounded.
-        for i in 0..=(MRF_REPAIRED_EVENT_CAP + 8) {
-            note_mrf_repaired("flood-bucket", &format!("object-{i}"), None);
-        }
-        let flooded = take_mrf_repaired_events_for("flood-bucket");
-        assert_eq!(flooded.len(), MRF_REPAIRED_EVENT_CAP);
-        assert_eq!(flooded[0].object.as_ref(), "object-9", "the oldest notices past the cap are dropped");
-    }
-}

crates/common/src/trace_bus.rs

@@ -1,333 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-use smallvec::SmallVec;
-use std::{
-    sync::{
-        Arc, OnceLock,
-        atomic::{AtomicUsize, Ordering},
-    },
-    time::{Duration, SystemTime},
-};
-use tokio::sync::broadcast;
-
-const DEFAULT_TRACE_BUS_CAPACITY: usize = 1024;
-const TRACE_ATTR_INLINE_CAPACITY: usize = 8;
-
-static GLOBAL_TRACE_BUS: OnceLock<TraceBus> = OnceLock::new();
-
-#[derive(Debug, Clone, Copy, PartialEq, Eq)]
-pub enum TraceKind {
-    Heal,
-    Scanner,
-}
-
-impl TraceKind {
-    pub const fn as_str(self) -> &'static str {
-        match self {
-            Self::Heal => "heal",
-            Self::Scanner => "scanner",
-        }
-    }
-}
-
-#[derive(Debug, Clone, Copy, PartialEq, Eq)]
-pub enum TraceFunc {
-    HealTask,
-    HealBucket,
-    HealObject,
-    HealCheckAbandonedParts,
-    HealErasureSetPage,
-    ScannerFolder,
-    ScannerIlmAction,
-    ScannerHealCandidate,
-    Dropped,
-}
-
-impl TraceFunc {
-    pub const fn as_str(self) -> &'static str {
-        match self {
-            Self::HealTask => "heal.Task",
-            Self::HealBucket => "heal.Bucket",
-            Self::HealObject => "heal.Object",
-            Self::HealCheckAbandonedParts => "heal.CheckAbandonedParts",
-            Self::HealErasureSetPage => "heal.ErasureSetPage",
-            Self::ScannerFolder => "scanner.Folder",
-            Self::ScannerIlmAction => "scanner.IlmAction",
-            Self::ScannerHealCandidate => "scanner.HealCandidate",
-            Self::Dropped => "trace.Dropped",
-        }
-    }
-}
-
-#[derive(Debug, Clone, PartialEq, Eq)]
-pub enum TraceVal {
-    Bool(bool),
-    U64(u64),
-    I64(i64),
-    Str(Arc<str>),
-}
-
-impl From<bool> for TraceVal {
-    fn from(value: bool) -> Self {
-        Self::Bool(value)
-    }
-}
-
-impl From<u64> for TraceVal {
-    fn from(value: u64) -> Self {
-        Self::U64(value)
-    }
-}
-
-impl From<i64> for TraceVal {
-    fn from(value: i64) -> Self {
-        Self::I64(value)
-    }
-}
-
-impl From<&str> for TraceVal {
-    fn from(value: &str) -> Self {
-        Self::Str(Arc::from(value))
-    }
-}
-
-impl From<String> for TraceVal {
-    fn from(value: String) -> Self {
-        Self::Str(Arc::from(value))
-    }
-}
-
-#[derive(Debug, Clone, PartialEq, Eq)]
-pub struct TraceAttr {
-    pub key: &'static str,
-    pub value: TraceVal,
-}
-
-#[derive(Debug, Clone, PartialEq, Eq)]
-pub struct TraceEvent {
-    pub kind: TraceKind,
-    pub func: TraceFunc,
-    pub time: SystemTime,
-    pub bucket: Option<Arc<str>>,
-    pub object: Option<Arc<str>>,
-    pub duration: Duration,
-    pub bytes: u64,
-    pub attrs: SmallVec<[TraceAttr; TRACE_ATTR_INLINE_CAPACITY]>,
-}
-
-impl TraceEvent {
-    pub fn new(kind: TraceKind, func: TraceFunc) -> Self {
-        Self {
-            kind,
-            func,
-            time: SystemTime::now(),
-            bucket: None,
-            object: None,
-            duration: Duration::ZERO,
-            bytes: 0,
-            attrs: SmallVec::new(),
-        }
-    }
-
-    pub fn with_bucket(mut self, bucket: impl Into<Arc<str>>) -> Self {
-        self.bucket = Some(bucket.into());
-        self
-    }
-
-    pub fn with_object(mut self, object: impl Into<Arc<str>>) -> Self {
-        self.object = Some(object.into());
-        self
-    }
-
-    pub fn with_duration(mut self, duration: Duration) -> Self {
-        self.duration = duration;
-        self
-    }
-
-    pub fn with_bytes(mut self, bytes: u64) -> Self {
-        self.bytes = bytes;
-        self
-    }
-
-    pub fn with_attr(mut self, key: &'static str, value: impl Into<TraceVal>) -> Self {
-        self.attrs.push(TraceAttr {
-            key,
-            value: value.into(),
-        });
-        self
-    }
-}
-
-#[derive(Debug)]
-pub struct TraceBus {
-    sender: broadcast::Sender<Arc<TraceEvent>>,
-    subscriber_count: Arc<AtomicUsize>,
-}
-
-impl TraceBus {
-    pub fn new(capacity: usize) -> Self {
-        let capacity = capacity.max(1);
-        let (sender, _receiver) = broadcast::channel(capacity);
-        Self {
-            sender,
-            subscriber_count: Arc::new(AtomicUsize::new(0)),
-        }
-    }
-
-    pub fn subscriber_count(&self) -> usize {
-        self.subscriber_count.load(Ordering::Acquire)
-    }
-
-    pub fn subscribe(&self) -> TraceSubscription {
-        let receiver = self.sender.subscribe();
-        self.subscriber_count.fetch_add(1, Ordering::AcqRel);
-        TraceSubscription {
-            receiver,
-            subscriber_count: Arc::clone(&self.subscriber_count),
-        }
-    }
-
-    pub fn emit(&self, build: impl FnOnce() -> TraceEvent) -> bool {
-        if self.subscriber_count() == 0 {
-            return false;
-        }
-
-        self.sender.send(Arc::new(build())).is_ok()
-    }
-}
-
-impl Default for TraceBus {
-    fn default() -> Self {
-        Self::new(DEFAULT_TRACE_BUS_CAPACITY)
-    }
-}
-
-#[derive(Debug)]
-pub struct TraceSubscription {
-    receiver: broadcast::Receiver<Arc<TraceEvent>>,
-    subscriber_count: Arc<AtomicUsize>,
-}
-
-impl TraceSubscription {
-    pub async fn recv(&mut self) -> Result<Arc<TraceEvent>, broadcast::error::RecvError> {
-        self.receiver.recv().await
-    }
-
-    pub fn try_recv(&mut self) -> Result<Arc<TraceEvent>, broadcast::error::TryRecvError> {
-        self.receiver.try_recv()
-    }
-}
-
-impl Drop for TraceSubscription {
-    fn drop(&mut self) {
-        self.subscriber_count.fetch_sub(1, Ordering::AcqRel);
-    }
-}
-
-pub fn global_trace_bus() -> &'static TraceBus {
-    GLOBAL_TRACE_BUS.get_or_init(TraceBus::default)
-}
-
-pub fn subscribe_trace_events() -> TraceSubscription {
-    global_trace_bus().subscribe()
-}
-
-pub fn trace_emit(build: impl FnOnce() -> TraceEvent) -> bool {
-    global_trace_bus().emit(build)
-}
-
-pub fn trace_subscriber_count() -> usize {
-    global_trace_bus().subscriber_count()
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-    use std::sync::atomic::AtomicUsize;
-
-    #[test]
-    fn trace_emit_skips_builder_without_subscribers() {
-        let bus = TraceBus::new(4);
-        let built = AtomicUsize::new(0);
-
-        let sent = bus.emit(|| {
-            built.fetch_add(1, Ordering::Relaxed);
-            TraceEvent::new(TraceKind::Heal, TraceFunc::HealTask)
-        });
-
-        assert!(!sent);
-        assert_eq!(built.load(Ordering::Relaxed), 0);
-    }
-
-    #[tokio::test]
-    async fn trace_subscriber_receives_event() {
-        let bus = TraceBus::new(4);
-        let mut subscription = bus.subscribe();
-
-        assert!(bus.emit(|| {
-            TraceEvent::new(TraceKind::Heal, TraceFunc::HealObject)
-                .with_bucket("bucket")
-                .with_object("object")
-                .with_duration(Duration::from_millis(7))
-                .with_bytes(11)
-                .with_attr("dry", true)
-        }));
-
-        let event = subscription
-            .recv()
-            .await
-            .expect("subscriber should receive emitted trace event");
-
-        assert_eq!(event.kind, TraceKind::Heal);
-        assert_eq!(event.func, TraceFunc::HealObject);
-        assert_eq!(event.bucket.as_deref(), Some("bucket"));
-        assert_eq!(event.object.as_deref(), Some("object"));
-        assert_eq!(event.duration, Duration::from_millis(7));
-        assert_eq!(event.bytes, 11);
-        assert_eq!(
-            event.attrs.as_slice(),
-            &[TraceAttr {
-                key: "dry",
-                value: TraceVal::Bool(true)
-            }]
-        );
-    }
-
-    #[test]
-    fn trace_subscription_drop_decrements_count() {
-        let bus = TraceBus::new(4);
-        let subscription = bus.subscribe();
-
-        assert_eq!(bus.subscriber_count(), 1);
-        drop(subscription);
-        assert_eq!(bus.subscriber_count(), 0);
-    }
-
-    #[tokio::test]
-    async fn lagged_subscriber_drops_events_without_blocking_publishers() {
-        let bus = TraceBus::new(2);
-        let mut subscription = bus.subscribe();
-
-        for index in 0_u64..4 {
-            assert!(bus.emit(|| { TraceEvent::new(TraceKind::Scanner, TraceFunc::ScannerFolder).with_attr("index", index) }));
-        }
-
-        let err = subscription
-            .recv()
-            .await
-            .expect_err("receiver should observe lag instead of blocking publishers");
-        assert!(matches!(err, broadcast::error::RecvError::Lagged(_)));
-    }
-}

crates/concurrency/src/backpressure.rs

@@ -14,8 +14,9 @@
 
 //! Shared backpressure policy type.
 //!
-//! This module only carries the watermark policy; the admission primitive it
-//! projects into lives in `rustfs-io-core`.
+//! The runtime backpressure implementation (byte-watermark pipes and
+//! monitors) lives in `rustfs/src/storage/backpressure.rs`; this module only
+//! carries the watermark policy type that implementation shares.
 
 use rustfs_io_core::BackpressureConfig as CoreBackpressureConfig;
 

crates/config/README.md

@@ -97,14 +97,6 @@ Current guidance:
   - enables minimal payload mode for GET health responses (`status`, `ready` only).
 - `RUSTFS_HEALTH_READINESS_CACHE_TTL_MS`
   - TTL for readiness cache evaluation.
-- `RUSTFS_HEALTH_OBJECT_PROGRESS_ENABLE`
-  - withdraws readiness when bounded object read/write stages stop completing while requests remain active.
-  - default is `true`.
-- `RUSTFS_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS`
-  - maximum time without completion in a bounded object stage before readiness is withdrawn.
-  - default is `30000`; `0` uses the default.
-  - the effective value is at least 5 seconds longer than `RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT`.
-  - this readiness SLO is independent of disk read/write failure deadlines and may withdraw traffic before those deadlines expire.
 - `RUSTFS_HEALTH_COMPAT_BUSY_CHECK_ENABLE`
   - enables busy protection behavior for health probes.
   - default is `false`.
@@ -115,15 +107,6 @@ Current guidance:
   - enables KMS readiness enforcement for `/health/ready`.
   - default is `false`.
 
-## Object lock admission environment variables
-
-- `RUSTFS_PUT_COMMIT_NAMESPACE_LOCK_ACQUIRE_TIMEOUT_MS`
-  - experimental same-object PUT commit namespace-lock admission budget.
-  - default is `0`, which disables this override and keeps `RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT` behavior.
-  - when set, only `put_object_commit` write-lock acquisition is bounded by this millisecond budget; other namespace lock users keep the global object-lock timeout.
-  - timeout returns S3 `SlowDown`, so clients should use normal SDK retry handling.
-  - this is not a fdatasync or group-commit switch. Track fdatasync batching separately with `rustfs_s3_put_object_rename_fdatasync_batch_files`.
-
 ## Drive timeout environment variables
 
 - `RUSTFS_DRIVE_METADATA_TIMEOUT_SECS`

crates/config/src/constants/app.rs

@@ -353,11 +353,6 @@ pub const DEFAULT_OBS_TRACES_EXPORT_ENABLED: bool = true;
 /// Environment variable: RUSTFS_OBS_METRICS_EXPORT_ENABLED
 pub const DEFAULT_OBS_METRICS_EXPORT_ENABLED: bool = true;
 
-/// Default detailed PUT stage metrics enabled
-/// Default value: false
-/// Environment variable: RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED
-pub const DEFAULT_OBS_PUT_STAGE_METRICS_ENABLED: bool = false;
-
 /// Default logs export enabled
 /// It is used to enable or disable exporting logs
 /// Default value: true

crates/config/src/constants/heal.rs

@@ -177,40 +177,3 @@ pub const DEFAULT_HEAL_MAINLINE_WRITE_UTILIZATION_HIGH_PERCENT: usize = 80;
 
 /// Default foreground pressure recheck delay for heal scheduler, in milliseconds.
 pub const DEFAULT_HEAL_MAINLINE_MAX_SLEEP_MS: u64 = 250;
-
-/// Environment variable that toggles the MRF (mission repair feed) intent
-/// pipeline: error paths deliver repair intents to the heal runtime, and
-/// unconsumed intents are replayed from the durable journal after a restart.
-pub const ENV_HEAL_MRF_ENABLE: &str = "RUSTFS_HEAL_MRF_ENABLE";
-
-/// Environment variable for the MRF in-memory queue capacity (intent count).
-pub const ENV_HEAL_MRF_QUEUE_SIZE: &str = "RUSTFS_HEAL_MRF_QUEUE_SIZE";
-
-/// Environment variable for the MRF journal byte budget. The journal is
-/// compacted once its on-disk size crosses this bound.
-pub const ENV_HEAL_MRF_JOURNAL_MAX_BYTES: &str = "RUSTFS_HEAL_MRF_JOURNAL_MAX_BYTES";
-
-/// Environment variable for the MRF journal replay batch size (intents per
-/// replay push round).
-pub const ENV_HEAL_MRF_REPLAY_BATCH: &str = "RUSTFS_HEAL_MRF_REPLAY_BATCH";
-
-/// Default behavior keeps the MRF intent pipeline enabled.
-pub const DEFAULT_HEAL_MRF_ENABLE: bool = true;
-
-/// Default MRF queue capacity (matches MinIO's 100k MRF list ceiling).
-pub const DEFAULT_HEAL_MRF_QUEUE_SIZE: usize = 100_000;
-
-/// Default MRF journal byte budget (8 MiB), mirroring the channel payload cap.
-pub const DEFAULT_HEAL_MRF_JOURNAL_MAX_BYTES: usize = 8 * 1024 * 1024;
-
-/// Default MRF replay batch size.
-pub const DEFAULT_HEAL_MRF_REPLAY_BATCH: usize = 256;
-
-/// Environment variable selecting how admin heal starts behave when the
-/// requested path overlaps an already running or queued heal: `merge`
-/// (default, keep today's dedup/merge semantics) or `minio_error` (return a
-/// typed already-running / overlapping-paths rejection like madmin).
-pub const ENV_HEAL_OVERLAP_POLICY: &str = "RUSTFS_HEAL_OVERLAP_POLICY";
-
-/// Default overlap policy: merge duplicate/overlapping requests.
-pub const DEFAULT_HEAL_OVERLAP_POLICY: &str = "merge";

crates/config/src/constants/health.rs

@@ -22,19 +22,6 @@ pub const DEFAULT_HEALTH_ENDPOINT_ENABLE: bool = true;
 pub const ENV_HEALTH_READINESS_CACHE_TTL_MS: &str = "RUSTFS_HEALTH_READINESS_CACHE_TTL_MS";
 pub const DEFAULT_HEALTH_READINESS_CACHE_TTL_MS: u64 = 1000;
 
-/// Enable readiness withdrawal when bounded object read/write stages stop
-/// completing while requests remain active.
-pub const ENV_HEALTH_OBJECT_PROGRESS_ENABLE: &str = "RUSTFS_HEALTH_OBJECT_PROGRESS_ENABLE";
-pub const DEFAULT_HEALTH_OBJECT_PROGRESS_ENABLE: bool = true;
-
-/// Requested time without completion in a bounded object stage before local
-/// readiness is withdrawn (milliseconds). A value of `0` uses the default;
-/// runtime adds a safety floor based on the object-lock acquisition timeout.
-pub const ENV_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS: &str = "RUSTFS_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS";
-pub const DEFAULT_HEALTH_OBJECT_PROGRESS_TIMEOUT_MS: u64 = 30_000;
-/// Additional time beyond the configured object-lock acquisition deadline.
-pub const HEALTH_OBJECT_PROGRESS_LOCK_MARGIN_MS: u64 = 5_000;
-
 /// Timeout for cluster health readiness collectors (milliseconds).
 /// This bounds expensive storage and lock quorum checks used by cluster probes.
 pub const ENV_HEALTH_CLUSTER_TIMEOUT_MS: &str = "RUSTFS_HEALTH_CLUSTER_TIMEOUT_MS";

crates/config/src/constants/internode.rs

@@ -177,9 +177,10 @@ const _: () = assert!(!DEFAULT_INTERNODE_RPC_REPLAY_SCOPE_STRICT);
 ///
 /// The cache retains each nonce for the ~10-minute signature freshness envelope. Once peers use
 /// replay-scoped v3 authentication, every authenticated RPC consumes one entry, so the steady
-/// state holds roughly `authenticated RPC RPS x 601s` entries. This default is the minimum floor:
-/// explicit operator values and resource-aware auto sizing both clamp upward to at least this
-/// value. Overflow fails closed — legitimate signed traffic is the only thing that can fill the
+/// state holds roughly `authenticated RPC RPS x 601s` entries. The default sustains about 1,700
+/// authenticated RPCs per second (about 120 MiB worst case, allocated only under sustained load);
+/// operators must size it for the node's aggregate peak RPC rate before enabling strict replay
+/// scope. Overflow fails closed — legitimate signed traffic is the only thing that can fill the
 /// cache (replays are rejected before insertion, and an attacker cannot mint valid nonces without
 /// the shared secret) — and increments
 /// `rustfs_system_network_internode_replay_cache_overflow_total`, so a sustained non-zero overflow

crates/config/src/constants/object.rs

@@ -137,37 +137,6 @@ pub const DEFAULT_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED: bool = false;
 const _: () = assert!(!DEFAULT_TIER_REMOTE_VERSION_STATE_WRITE);
 const _: () = assert!(!DEFAULT_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED);
 
-/// Request the object-transaction fencing contract used by storage-owned
-/// cleanup receipts and lock-window optimizations.
-///
-/// This is fail-closed: enabling the writer without a live fleet proof rejects
-/// the commit rather than silently using a legacy-safe path.
-pub const ENV_OBJECT_TRANSACTION_FENCING_WRITE: &str = "RUSTFS_OBJECT_TRANSACTION_FENCING_WRITE";
-pub const DEFAULT_OBJECT_TRANSACTION_FENCING_WRITE: bool = false;
-
-/// Operator-attested confirmation that every serving node understands the
-/// object transaction fencing contract.
-pub const ENV_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED: &str = "RUSTFS_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED";
-pub const DEFAULT_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED: bool = false;
-
-const _: () = assert!(!DEFAULT_OBJECT_TRANSACTION_FENCING_WRITE);
-const _: () = assert!(!DEFAULT_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED);
-
-/// Request preserving legacy per-part checksum metadata during data movement.
-///
-/// This remains ineffective until
-/// [`ENV_DATA_MOVEMENT_PART_CHECKSUMS_FLEET_CONFIRMED`] is also enabled.
-pub const ENV_DATA_MOVEMENT_PART_CHECKSUMS_WRITE: &str = "RUSTFS_DATA_MOVEMENT_PART_CHECKSUMS_WRITE";
-pub const DEFAULT_DATA_MOVEMENT_PART_CHECKSUMS_WRITE: bool = false;
-
-/// Operator-attested confirmation that every serving node understands the
-/// data-movement per-part checksum sidecar.
-pub const ENV_DATA_MOVEMENT_PART_CHECKSUMS_FLEET_CONFIRMED: &str = "RUSTFS_DATA_MOVEMENT_PART_CHECKSUMS_FLEET_CONFIRMED";
-pub const DEFAULT_DATA_MOVEMENT_PART_CHECKSUMS_FLEET_CONFIRMED: bool = false;
-
-const _: () = assert!(!DEFAULT_DATA_MOVEMENT_PART_CHECKSUMS_WRITE);
-const _: () = assert!(!DEFAULT_DATA_MOVEMENT_PART_CHECKSUMS_FLEET_CONFIRMED);
-
 // =============================================================================
 // Concurrent Request Fix - Timeout and Backpressure Configuration
 // =============================================================================
@@ -234,31 +203,6 @@ pub const ENV_OBJECT_DISK_WRITE_ABSOLUTE_CAP: &str = "RUSTFS_OBJECT_DISK_WRITE_A
 /// Default absolute per-object erasure write cap in seconds (`0` = disabled).
 pub const DEFAULT_OBJECT_DISK_WRITE_ABSOLUTE_CAP: u64 = 0;
 
-/// Enable foreground PutObject request admission.
-///
-/// This is an experimental, default-off foreground write backpressure gate for
-/// strict commit tail investigations. When disabled, PUTs follow the legacy
-/// path and only the existing request counters are updated.
-pub const ENV_PUT_FOREGROUND_ADMISSION_ENABLE: &str = "RUSTFS_PUT_FOREGROUND_ADMISSION_ENABLE";
-pub const DEFAULT_PUT_FOREGROUND_ADMISSION_ENABLE: bool = false;
-
-/// Maximum foreground PutObject requests admitted concurrently per process.
-///
-/// The limit is used only when [`ENV_PUT_FOREGROUND_ADMISSION_ENABLE`] is true.
-/// A value of `0` disables the gate even when the enable flag is present, so a
-/// partially configured rollout cannot reject every PUT.
-pub const ENV_PUT_FOREGROUND_ADMISSION_LIMIT: &str = "RUSTFS_PUT_FOREGROUND_ADMISSION_LIMIT";
-pub const DEFAULT_PUT_FOREGROUND_ADMISSION_LIMIT: usize = 0;
-
-/// Time in milliseconds a foreground PutObject waits for an admission permit.
-///
-/// Once this timeout expires the request fails before body ingest/storage
-/// mutation with S3 `SlowDown`/503. `0` means fail fast when the limit is full.
-pub const ENV_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: &str = "RUSTFS_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS";
-pub const DEFAULT_PUT_FOREGROUND_ADMISSION_WAIT_TIMEOUT_MS: u64 = 0;
-
-const _: () = assert!(!DEFAULT_PUT_FOREGROUND_ADMISSION_ENABLE);
-
 /// Environment variable for minimum GetObject timeout in seconds.
 ///
 /// When dynamic timeout calculation is enabled, this is the minimum timeout
@@ -427,19 +371,6 @@ pub const ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT: &str = "RUSTFS_OBJECT_LOCK_ACQUIRE_TI
 /// Default lock acquisition timeout: 5 seconds.
 pub const DEFAULT_OBJECT_LOCK_ACQUIRE_TIMEOUT: u64 = 5;
 
-/// Environment variable for the experimental PUT commit namespace lock acquire timeout in milliseconds.
-///
-/// A value of `0` disables the experiment and keeps
-/// `RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT` as the timeout. This only bounds the
-/// `put_object_commit` namespace write-lock wait and is intended for #925
-/// tail-drain admission experiments.
-///
-/// Default: 0 milliseconds (disabled).
-pub const ENV_PUT_COMMIT_NAMESPACE_LOCK_ACQUIRE_TIMEOUT_MS: &str = "RUSTFS_PUT_COMMIT_NAMESPACE_LOCK_ACQUIRE_TIMEOUT_MS";
-
-/// Default: PUT commit namespace lock acquire timeout override is disabled.
-pub const DEFAULT_PUT_COMMIT_NAMESPACE_LOCK_ACQUIRE_TIMEOUT_MS: u64 = 0;
-
 /// Environment variable for remote namespace lock RPC transport timeout in milliseconds.
 ///
 /// This timeout bounds the internode RPC call itself. It is intentionally
@@ -718,22 +649,4 @@ mod remote_version_state_tests {
             "RUSTFS_TIER_REMOTE_VERSION_STATE_FLEET_CONFIRMED"
         );
     }
-
-    #[test]
-    fn data_movement_part_checksum_gate_uses_stable_environment_names() {
-        assert_eq!(super::ENV_DATA_MOVEMENT_PART_CHECKSUMS_WRITE, "RUSTFS_DATA_MOVEMENT_PART_CHECKSUMS_WRITE");
-        assert_eq!(
-            super::ENV_DATA_MOVEMENT_PART_CHECKSUMS_FLEET_CONFIRMED,
-            "RUSTFS_DATA_MOVEMENT_PART_CHECKSUMS_FLEET_CONFIRMED"
-        );
-    }
-
-    #[test]
-    fn object_transaction_fencing_gate_uses_stable_environment_names() {
-        assert_eq!(super::ENV_OBJECT_TRANSACTION_FENCING_WRITE, "RUSTFS_OBJECT_TRANSACTION_FENCING_WRITE");
-        assert_eq!(
-            super::ENV_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED,
-            "RUSTFS_OBJECT_TRANSACTION_FENCING_FLEET_CONFIRMED"
-        );
-    }
 }

crates/config/src/constants/runtime.rs

@@ -57,13 +57,6 @@ pub const DEFAULT_MAX_IO_EVENTS_PER_TICK: usize = 1024;
 pub const DEFAULT_EVENT_INTERVAL: u32 = 61;
 pub const DEFAULT_RNG_SEED: Option<u64> = None; // None means random
 
-/// Dedicated blocking thread pool for fsync/fdatasync operations.
-/// When > 1, fsync operations are isolated from the main blocking pool to
-/// prevent device-bound fsync from starving read operations (pread/stat/open).
-/// Default 0 means auto (no isolation, use main runtime).
-pub const ENV_FSYNC_BLOCKING_THREADS: &str = "RUSTFS_RUNTIME_FSYNC_BLOCKING_THREADS";
-pub const DEFAULT_FSYNC_BLOCKING_THREADS: usize = 0;
-
 // Dial9 Tokio Telemetry Default values
 pub const DEFAULT_RUNTIME_DIAL9_ENABLED: bool = false; // Disabled by default
 pub const DEFAULT_RUNTIME_DIAL9_OUTPUT_DIR: &str = "/var/log/rustfs/telemetry";
@@ -88,9 +81,6 @@ pub const ENV_TEST_IAM_FAIL_INIT_ATTEMPTS: &str = "RUSTFS_TEST_IAM_FAIL_INIT_ATT
 pub const ENV_TEST_IAM_RETRY_INTERVAL_MS: &str = "RUSTFS_TEST_IAM_RETRY_INTERVAL_MS";
 /// Runtime env var controlling the transition worker count.
 pub const ENV_TRANSITION_WORKERS: &str = "RUSTFS_MAX_TRANSITION_WORKERS";
-/// Runtime env var controlling the ILM expiry worker count. A set, parsable,
-/// non-zero value wins; anything else falls back to `min(cpus, 16)`.
-pub const ENV_MAX_EXPIRY_WORKERS: &str = "RUSTFS_MAX_EXPIRY_WORKERS";
 /// Runtime env var controlling the absolute maximum transition workers.
 pub const ENV_TRANSITION_WORKERS_ABSOLUTE_MAX: &str = "RUSTFS_ABSOLUTE_MAX_WORKERS";
 /// Runtime env var controlling the transition queue capacity.

crates/config/src/constants/scanner.rs

@@ -228,6 +228,15 @@ pub const DEFAULT_SCANNER_MAX_CONCURRENT_DISK_SCANS: usize = 4;
 /// Default object interval for cooperative scanner yields.
 pub const DEFAULT_SCANNER_YIELD_EVERY_N_OBJECTS: u64 = 128;
 
+/// Compatibility flag kept for Patch 3 rollback windows.
+///
+/// Inline scanner heal execution has been removed in favor of heal-candidate enqueue.
+/// When this flag is enabled, RustFS logs a warning and continues to use enqueue-based heal.
+pub const ENV_SCANNER_INLINE_HEAL_ENABLE: &str = "RUSTFS_SCANNER_INLINE_HEAL_ENABLE";
+
+/// Default inline scanner heal compatibility mode.
+pub const DEFAULT_SCANNER_INLINE_HEAL_ENABLE: bool = false;
+
 /// Scanner speed preset controlling throttling behavior.
 ///
 /// Each preset defines three parameters:

crates/config/src/constants/tls.rs

@@ -36,11 +36,6 @@ pub const ENV_TRUST_SYSTEM_CA: &str = "RUSTFS_TRUST_SYSTEM_CA";
 /// To change this behavior, set the environment variable RUSTFS_TRUST_SYSTEM_CA=1
 pub const DEFAULT_TRUST_SYSTEM_CA: bool = false;
 
-/// Environment variable for an extra outbound root CA certificate bundle.
-/// Use this to trust an internal CA for outbound HTTPS clients without replacing
-/// the default operating-system/web PKI roots via SSL_CERT_FILE.
-pub const ENV_RUSTFS_EXTRA_CA_CERT: &str = "RUSTFS_EXTRA_CA_CERT";
-
 /// Environment variable to trust leaf certificates as CA
 /// When set to "1", RustFS will treat leaf certificates as CA certificates for trust validation.
 /// By default, this is disabled.

crates/config/src/notify/mod.rs

@@ -92,11 +92,15 @@ pub const NOTIFY_SUB_SYSTEMS: &[&str] = &[
 pub const NOTIFY_KAFKA_SUB_SYS: &str = "notify_kafka";
 pub const NOTIFY_MQTT_SUB_SYS: &str = "notify_mqtt";
 pub const NOTIFY_MYSQL_SUB_SYS: &str = "notify_mysql";
+#[allow(dead_code)]
 pub const NOTIFY_NATS_SUB_SYS: &str = "notify_nats";
+#[allow(dead_code)]
 pub const NOTIFY_NSQ_SUB_SYS: &str = "notify_nsq";
+#[allow(dead_code)]
 pub const NOTIFY_ES_SUB_SYS: &str = "notify_elasticsearch";
 pub const NOTIFY_AMQP_SUB_SYS: &str = "notify_amqp";
 pub const NOTIFY_POSTGRES_SUB_SYS: &str = "notify_postgres";
+#[allow(dead_code)]
 pub const NOTIFY_REDIS_SUB_SYS: &str = "notify_redis";
 pub const NOTIFY_REDIS_DEFAULT_CHANNEL: &str = "rustfs_notify_channel";
 pub const NOTIFY_PULSAR_SUB_SYS: &str = "notify_pulsar";

crates/config/src/observability/mod.rs

@@ -44,10 +44,6 @@ pub const ENV_OBS_METRICS_EXPORT_ENABLED: &str = "RUSTFS_OBS_METRICS_EXPORT_ENAB
 pub const ENV_OBS_LOGS_EXPORT_ENABLED: &str = "RUSTFS_OBS_LOGS_EXPORT_ENABLED";
 pub const ENV_OBS_PROFILING_EXPORT_ENABLED: &str = "RUSTFS_OBS_PROFILING_EXPORT_ENABLED";
 
-/// Enables detailed per-stage PUT metrics. Disabled by default because each
-/// PUT records multiple timers and histograms when attribution is active.
-pub const ENV_OBS_PUT_STAGE_METRICS_ENABLED: &str = "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED";
-
 pub const ENV_OBS_LOGGER_LEVEL: &str = "RUSTFS_OBS_LOGGER_LEVEL";
 pub const ENV_OBS_LOG_STDOUT_ENABLED: &str = "RUSTFS_OBS_LOG_STDOUT_ENABLED";
 pub const ENV_OBS_LOG_DIRECTORY: &str = "RUSTFS_OBS_LOG_DIRECTORY";
@@ -145,7 +141,6 @@ mod tests {
         assert_eq!(ENV_OBS_METRICS_EXPORT_ENABLED, "RUSTFS_OBS_METRICS_EXPORT_ENABLED");
         assert_eq!(ENV_OBS_LOGS_EXPORT_ENABLED, "RUSTFS_OBS_LOGS_EXPORT_ENABLED");
         assert_eq!(ENV_OBS_PROFILING_EXPORT_ENABLED, "RUSTFS_OBS_PROFILING_EXPORT_ENABLED");
-        assert_eq!(ENV_OBS_PUT_STAGE_METRICS_ENABLED, "RUSTFS_OBS_PUT_STAGE_METRICS_ENABLED");
         // Test log cleanup related env keys
         assert_eq!(ENV_OBS_LOG_MAX_TOTAL_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_TOTAL_SIZE_BYTES");
         assert_eq!(ENV_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES, "RUSTFS_OBS_LOG_MAX_SINGLE_FILE_SIZE_BYTES");

crates/crypto/src/license_token.rs

@@ -203,6 +203,14 @@ mod tests {
         assert!(result.is_err());
     }
 
+    #[test]
+    fn test_source_does_not_embed_private_key() {
+        let source = include_str!("license_token.rs");
+        let forbidden = ["BEGIN", "PRIVATE KEY"].join(" ");
+
+        assert!(!source.contains(&forbidden));
+    }
+
     #[test]
     fn test_parse_signed_license_token_rejects_invalid_token() {
         let mut rng = rand::rng();

crates/data-usage/Cargo.toml

@@ -37,6 +37,7 @@ hotpath-cpu = ["hotpath", "hotpath/hotpath-cpu", "rustfs-filemeta/hotpath-cpu"]
 hotpath.workspace = true
 serde = { workspace = true, features = ["derive"] }
 rmp-serde = { workspace = true }
+async-trait = { workspace = true }
 rustfs-filemeta = { workspace = true }
 
 [lib]

crates/data-usage/src/data_usage.rs

@@ -37,10 +37,6 @@ pub const USAGE_LAST_UPDATE_FUTURE_TOLERANCE: Duration = Duration::from_secs(5 *
 /// Keeping the existing object name preserves rolling-upgrade and rollback
 /// compatibility without allowing an ambiguous snapshot to become authoritative.
 pub const DATA_USAGE_OBJECT_NAME: &str = ".usage.v2.json";
-/// Latest structurally complete scanner observation. Unlike
-/// [`DATA_USAGE_OBJECT_NAME`], this object is never authoritative for quota
-/// admission because namespace activity may have raced the scan.
-pub const DATA_USAGE_OBSERVED_OBJECT_NAME: &str = ".usage.observed.json";
 
 /// Usage snapshot written by scanner implementations predating distributed
 /// leadership fencing. It is read only when neither authoritative snapshot
@@ -222,20 +218,6 @@ pub struct DataUsageInfo {
     /// explicit entry for every bucket, including confirmed-empty buckets.
     #[serde(default)]
     pub usage_snapshot_complete: bool,
-    /// Whether no namespace activity or dirty-usage generation changed while
-    /// the coordinated snapshot was being produced.
-    ///
-    /// `false` still describes a structurally complete, useful point-in-time
-    /// usage view, but follow-up scanner work remains pending. `None` is kept
-    /// for snapshots written before this status became observable.
-    #[serde(default, skip_serializing_if = "Option::is_none")]
-    pub usage_snapshot_converged: Option<bool>,
-    /// Identity of the authoritative snapshot from which a nonconverged
-    /// observation started. Admin readers require an exact match before using
-    /// the observation, so bucket namespace mutations fence old observations
-    /// without relying on synchronized clocks.
-    #[serde(default, skip_serializing_if = "Option::is_none")]
-    pub usage_snapshot_authoritative_baseline: Option<DataUsageSnapshotIdentity>,
     /// Deprecated kept here for backward compatibility reasons
     pub bucket_sizes: HashMap<String, u64>,
     /// Per-disk snapshot information when available
@@ -243,59 +225,6 @@ pub struct DataUsageInfo {
     pub disk_usage_status: Vec<DiskUsageStatus>,
 }
 
-/// Stable identity fields changed by both coordinated scanner publication and
-/// backward-compatible bucket namespace cleanup.
-#[derive(Debug, Default, Clone, Copy, Serialize, Deserialize, PartialEq, Eq)]
-pub struct DataUsageSnapshotIdentity {
-    pub last_update: Option<SystemTime>,
-    pub scanner_cycle: Option<u64>,
-    pub scanner_epoch: Option<u64>,
-}
-
-impl DataUsageInfo {
-    pub fn snapshot_identity(&self) -> DataUsageSnapshotIdentity {
-        DataUsageSnapshotIdentity {
-            last_update: self.last_update,
-            scanner_cycle: self.scanner_cycle,
-            scanner_epoch: self.scanner_epoch,
-        }
-    }
-}
-
-/// Return whether `candidate` was produced after `baseline`.
-///
-/// New coordinated snapshots are ordered by leadership epoch and scanner
-/// cycle. The timestamp fallback preserves ordering for legacy snapshots that
-/// predate those fields.
-pub fn data_usage_snapshot_is_newer(candidate: &DataUsageInfo, baseline: &DataUsageInfo) -> bool {
-    match (
-        candidate.scanner_epoch.zip(candidate.scanner_cycle),
-        baseline.scanner_epoch.zip(baseline.scanner_cycle),
-    ) {
-        (Some(candidate), Some(baseline)) => candidate > baseline,
-        (Some(_), None) => true,
-        (None, Some(_)) => false,
-        (None, None) => match (candidate.last_update, baseline.last_update) {
-            (Some(candidate), Some(baseline)) => candidate > baseline,
-            (Some(_), None) => true,
-            (None, Some(_) | None) => false,
-        },
-    }
-}
-
-/// Return whether a nonconverged observation may safely supersede the admin
-/// view of `authoritative`.
-///
-/// The exact baseline identity is independent of clock ordering. Older binaries
-/// already advance the authoritative timestamp when deleting a bucket, so a
-/// rollback delete/recreate fences the previous bucket incarnation too.
-pub fn observed_data_usage_is_newer(observed: &DataUsageInfo, authoritative: &DataUsageInfo) -> bool {
-    observed.usage_snapshot_converged == Some(false)
-        && observed.is_complete_bucket_usage_snapshot()
-        && observed.usage_snapshot_authoritative_baseline.as_ref() == Some(&authoritative.snapshot_identity())
-        && data_usage_snapshot_is_newer(observed, authoritative)
-}
-
 /// Metadata describing the status of a disk-level data usage snapshot.
 #[derive(Debug, Default, Clone, Serialize, Deserialize, PartialEq, Eq)]
 pub struct DiskUsageStatus {
@@ -317,15 +246,15 @@ pub struct SizeSummary {
     /// Number of delete markers
     pub delete_markers: usize,
     /// Replicated size
-    pub replicated_size: i64,
+    pub replicated_size: usize,
     /// Replicated count
     pub replicated_count: usize,
     /// Pending size
-    pub pending_size: i64,
+    pub pending_size: usize,
     /// Failed size
-    pub failed_size: i64,
+    pub failed_size: usize,
     /// Replica size
-    pub replica_size: i64,
+    pub replica_size: usize,
     /// Replica count
     pub replica_count: usize,
     /// Pending count
@@ -334,21 +263,19 @@ pub struct SizeSummary {
     pub failed_count: usize,
     /// Replication target stats
     pub repl_target_stats: HashMap<String, ReplTargetSizeSummary>,
-    /// Per-tier accounting, keyed by storage class or remote tier name
-    pub tier_stats: HashMap<String, TierStats>,
 }
 
 /// Replication target size summary
 #[derive(Debug, Default, Clone)]
 pub struct ReplTargetSizeSummary {
     /// Replicated size
-    pub replicated_size: i64,
+    pub replicated_size: usize,
     /// Replicated count
     pub replicated_count: usize,
     /// Pending size
-    pub pending_size: i64,
+    pub pending_size: usize,
     /// Failed size
-    pub failed_size: i64,
+    pub failed_size: usize,
     /// Pending count
     pub pending_count: usize,
     /// Failed count
@@ -712,6 +639,28 @@ impl DataUsageEntry {
         self.children.insert(hash.key());
     }
 
+    pub fn add_sizes(&mut self, summary: &SizeSummary) {
+        self.size += summary.total_size;
+        self.versions += summary.versions;
+        self.delete_markers += summary.delete_markers;
+        self.obj_sizes.add(summary.total_size as u64);
+        self.obj_versions.add(summary.versions as u64);
+
+        let replication_stats = self.replication_stats.get_or_insert_with(ReplicationAllStats::default);
+        replication_stats.replica_size += summary.replica_size as u64;
+        replication_stats.replica_count += summary.replica_count as u64;
+
+        for (arn, st) in &summary.repl_target_stats {
+            let tgt_stat = replication_stats.targets.entry(arn.to_string()).or_default();
+            tgt_stat.pending_size += st.pending_size as u64;
+            tgt_stat.failed_size += st.failed_size as u64;
+            tgt_stat.replicated_size += st.replicated_size as u64;
+            tgt_stat.replicated_count += st.replicated_count as u64;
+            tgt_stat.failed_count += st.failed_count as u64;
+            tgt_stat.pending_count += st.pending_count as u64;
+        }
+    }
+
     pub fn merge(&mut self, other: &DataUsageEntry) {
         self.objects += other.objects;
         self.versions += other.versions;
@@ -826,15 +775,8 @@ impl DataUsageEntry {
     }
 }
 
-/// Read-only projection of the scanner's `.usage-cache.bin` info block.
-///
-/// The canonical wire format is written by the hand-written map-encoded
-/// `Serialize` on the scanner-side `DataUsageCacheInfo`
-/// (`crates/scanner/src/data_usage_define.rs`), which carries 16 fields.
-/// This type decodes only the shared subset and is deliberately not
-/// `Serialize`: a derived (array) encoding of this 6-field subset would
-/// corrupt the cache for scanner readers, so no write path may exist here.
-#[derive(Clone, Debug, Default, Deserialize)]
+/// Data usage cache info
+#[derive(Clone, Debug, Default, Serialize, Deserialize)]
 pub struct DataUsageCacheInfo {
     pub name: String,
     pub next_cycle: u64,
@@ -850,163 +792,8 @@ pub struct DataUsageCacheInfo {
     pub snapshot_complete: bool,
 }
 
-/// Prefix-level usage over a raw entry map — the shared core behind
-/// [`DataUsageCache::prefix_usage`], usable by any cache-shaped reader (the
-/// scanner's writer-side cache has the same map type).
-///
-/// Cache keys are cleaned literal paths (`bucket/pre/fix`), so sub-prefix
-/// names come straight off the child keys — no reverse mapping exists or is
-/// needed. A compacted prefix carries its aggregate but no children, which
-/// the `compacted` flag reports so callers can say why the breakdown is
-/// empty. `truncated` is set when the breakdown exceeded `max_entries` and
-/// was cut (largest first).
-pub fn prefix_usage_in_cache(
-    cache: &HashMap<String, DataUsageEntry>,
-    bucket: &str,
-    prefix: &str,
-    max_entries: usize,
-) -> Option<PrefixUsageQuery> {
-    let prefix = prefix.trim_matches('/');
-    let root = if prefix.is_empty() {
-        bucket.to_string()
-    } else {
-        format!("{bucket}/{prefix}")
-    };
-    let entry = cache.get(&hash_path(&root).key())?.clone();
-
-    let usage = PrefixUsageSummary::from_entry(&flatten_entry(cache, &entry, 0)?);
-
-    let child_prefix = format!("{root}/");
-    let mut sub_prefixes: Vec<PrefixUsageEntry> = entry
-        .children
-        .iter()
-        .filter_map(|child_key| {
-            let child = cache.get(child_key)?;
-            let child_flat = flatten_entry(cache, child, 1)?;
-            // Child keys are literal `bucket/pre/name` paths; a trailing
-            // slash marks a directory object and is display-only here.
-            let name = child_key
-                .strip_prefix(child_prefix.as_str())
-                .unwrap_or(child_key.as_str())
-                .trim_end_matches('/')
-                .to_string();
-            Some(PrefixUsageEntry {
-                prefix: name,
-                usage: PrefixUsageSummary::from_entry(&child_flat),
-            })
-        })
-        .collect();
-    sub_prefixes.sort_by(|left, right| {
-        right
-            .usage
-            .size
-            .cmp(&left.usage.size)
-            .then_with(|| left.prefix.cmp(&right.prefix))
-    });
-    let truncated = sub_prefixes.len() > max_entries;
-    sub_prefixes.truncate(max_entries);
-
-    Some(PrefixUsageQuery {
-        usage,
-        compacted: entry.compacted,
-        truncated,
-        sub_prefixes,
-    })
-}
-
-/// Maximum subtree depth [`flatten_entry`] will walk before declaring the
-/// cache corrupt — the same bound the scanner's checked flatten uses.
-const PREFIX_USAGE_MAX_DEPTH: usize = 1024;
-
-/// Flatten one entry's subtree into an aggregate: the free-function twin of
-/// [`DataUsageCache::flatten`], carrying the scanner checked-flatten
-/// hardening so a corrupt cache (cycles, over-deep trees, overflowing
-/// counters) yields `None` instead of unbounded recursion or wrapped totals.
-fn flatten_entry(cache: &HashMap<String, DataUsageEntry>, root: &DataUsageEntry, depth: usize) -> Option<DataUsageEntry> {
-    if depth > PREFIX_USAGE_MAX_DEPTH {
-        return None;
-    }
-    let mut flattened = DataUsageEntry::default();
-    if !flattened.checked_merge(root) {
-        return None;
-    }
-    flattened.compacted = root.compacted;
-    // The root itself is not pre-seeded: it is merged above, and a corrupt
-    // child edge pointing back at the root's own key is still terminated by
-    // the visited set on first encounter.
-    let mut visited: HashSet<&str> = HashSet::new();
-    let mut pending: Vec<(&String, usize)> = root.children.iter().map(|child| (child, depth + 1)).collect();
-    while let Some((key, child_depth)) = pending.pop() {
-        if child_depth > PREFIX_USAGE_MAX_DEPTH || !visited.insert(key.as_str()) {
-            return None;
-        }
-        let entry = cache.get(key)?;
-        if !flattened.checked_merge(entry) {
-            return None;
-        }
-        pending.extend(entry.children.iter().map(|child| (child, child_depth + 1)));
-    }
-    flattened.children.clear();
-    Some(flattened)
-}
-
-/// Flattened counters of one prefix subtree, as returned by
-/// [`DataUsageCache::prefix_usage`].
-#[derive(Clone, Copy, Debug, Default, PartialEq, Eq, serde::Serialize)]
-#[serde(rename_all = "camelCase")]
-pub struct PrefixUsageSummary {
-    pub size: u64,
-    pub objects: u64,
-    pub versions: u64,
-    pub delete_markers: u64,
-}
-
-impl PrefixUsageSummary {
-    fn from_entry(entry: &DataUsageEntry) -> Self {
-        Self {
-            size: entry.size as u64,
-            objects: entry.objects as u64,
-            versions: entry.versions as u64,
-            delete_markers: entry.delete_markers as u64,
-        }
-    }
-
-    /// Add another set's counters into this one (entries are partitioned by
-    /// set, so per-set results sum).
-    pub fn merge(&mut self, other: &Self) {
-        self.size = self.size.saturating_add(other.size);
-        self.objects = self.objects.saturating_add(other.objects);
-        self.versions = self.versions.saturating_add(other.versions);
-        self.delete_markers = self.delete_markers.saturating_add(other.delete_markers);
-    }
-}
-
-/// One first-level sub-prefix row of a [`PrefixUsageQuery`].
-#[derive(Clone, Debug, PartialEq, Eq, serde::Serialize)]
-pub struct PrefixUsageEntry {
-    pub prefix: String,
-    pub usage: PrefixUsageSummary,
-}
-
-/// Result of [`DataUsageCache::prefix_usage`].
-#[derive(Clone, Debug, Default, PartialEq, Eq, serde::Serialize)]
-#[serde(rename_all = "camelCase")]
-pub struct PrefixUsageQuery {
-    pub usage: PrefixUsageSummary,
-    /// The prefix entry was compacted by the scanner: its aggregate is valid
-    /// but no sub-prefix breakdown exists on disk.
-    pub compacted: bool,
-    /// The breakdown had more entries than `max_entries`; the largest remain.
-    pub truncated: bool,
-    pub sub_prefixes: Vec<PrefixUsageEntry>,
-}
-
-/// Read-only projection of a scanner-written `.usage-cache.bin` file.
-///
-/// The scanner-side `DataUsageCache` (`crates/scanner/src/data_usage_define.rs`)
-/// owns the persisted format; this type only decodes it (see
-/// [`DataUsageCacheInfo`]) and must never grow a serialization path.
-#[derive(Clone, Debug, Default, Deserialize)]
+/// Data usage cache
+#[derive(Clone, Debug, Default, Serialize, Deserialize)]
 pub struct DataUsageCache {
     pub info: DataUsageCacheInfo,
     pub cache: HashMap<String, DataUsageEntry>,
@@ -1128,21 +915,6 @@ impl DataUsageCache {
         }
     }
 
-    /// Prefix-level usage for one bucket subtree, plus the one-level
-    /// breakdown below it (rustfs/backlog#1872, MinIO
-    /// `loadPrefixUsageFromBackend` parity and beyond: arbitrary prefixes and
-    /// full counters instead of first-level sizes only).
-    ///
-    /// Cache keys are cleaned literal paths (`bucket/pre/fix`), so sub-prefix
-    /// names come straight off the child keys — no reverse mapping exists or
-    /// is needed. A compacted prefix carries its aggregate but no children,
-    /// which the `compacted` flag reports so callers can say why the
-    /// breakdown is empty. `truncated` is set when the breakdown exceeded
-    /// `max_entries` and was cut (largest first).
-    pub fn prefix_usage(&self, bucket: &str, prefix: &str, max_entries: usize) -> Option<PrefixUsageQuery> {
-        prefix_usage_in_cache(&self.cache, bucket, prefix, max_entries)
-    }
-
     pub fn force_compact(&mut self, limit: usize) {
         if self.cache.len() < limit {
             return;
@@ -1343,10 +1115,31 @@ impl DataUsageCache {
         }
     }
 
+    pub fn marshal_msg(&self) -> Result<Vec<u8>, Box<dyn std::error::Error + Send + Sync>> {
+        let mut buf = Vec::new();
+        self.serialize(&mut rmp_serde::Serializer::new(&mut buf))?;
+        Ok(buf)
+    }
+
     pub fn unmarshal(buf: &[u8]) -> Result<Self, Box<dyn std::error::Error + Send + Sync>> {
         let t: Self = rmp_serde::from_slice(buf)?;
         Ok(t)
     }
+
+    // Note: load and save methods are storage-specific and should be implemented
+    // in the ecstore crate where storage access is available
+}
+
+/// Trait for storage-specific operations on DataUsageCache
+#[async_trait::async_trait]
+pub trait DataUsageCacheStorage {
+    /// Load data usage cache from backend storage
+    async fn load(store: &dyn std::any::Any, name: &str) -> Result<Self, Box<dyn std::error::Error + Send + Sync>>
+    where
+        Self: Sized;
+
+    /// Save data usage cache to backend storage
+    async fn save(&self, name: &str) -> Result<(), Box<dyn std::error::Error + Send + Sync>>;
 }
 
 // Helper structs and functions for cache operations
@@ -1702,6 +1495,14 @@ impl BucketUsageInfo {
     }
 
     /// Add size summary to this bucket usage
+    pub fn add_size_summary(&mut self, summary: &SizeSummary) {
+        self.size += summary.total_size as u64;
+        self.versions_count += summary.versions as u64;
+        self.delete_markers_count += summary.delete_markers as u64;
+        self.replica_size += summary.replica_size as u64;
+        self.replica_count += summary.replica_count as u64;
+    }
+
     /// Merge another BucketUsageInfo into this one
     pub fn merge(&mut self, other: &BucketUsageInfo) {
         self.size += other.size;
@@ -1747,32 +1548,29 @@ impl SizeSummary {
         Self::default()
     }
 
-    /// Add another SizeSummary to this one.
-    ///
-    /// Saturating throughout: a scan that overflows a counter should report the
-    /// ceiling rather than panic in a debug build or wrap in a release one.
+    /// Add another SizeSummary to this one
     pub fn add(&mut self, other: &SizeSummary) {
-        self.total_size = self.total_size.saturating_add(other.total_size);
-        self.versions = self.versions.saturating_add(other.versions);
-        self.delete_markers = self.delete_markers.saturating_add(other.delete_markers);
-        self.replicated_size = self.replicated_size.saturating_add(other.replicated_size);
-        self.replicated_count = self.replicated_count.saturating_add(other.replicated_count);
-        self.pending_size = self.pending_size.saturating_add(other.pending_size);
-        self.failed_size = self.failed_size.saturating_add(other.failed_size);
-        self.replica_size = self.replica_size.saturating_add(other.replica_size);
-        self.replica_count = self.replica_count.saturating_add(other.replica_count);
-        self.pending_count = self.pending_count.saturating_add(other.pending_count);
-        self.failed_count = self.failed_count.saturating_add(other.failed_count);
+        self.total_size += other.total_size;
+        self.versions += other.versions;
+        self.delete_markers += other.delete_markers;
+        self.replicated_size += other.replicated_size;
+        self.replicated_count += other.replicated_count;
+        self.pending_size += other.pending_size;
+        self.failed_size += other.failed_size;
+        self.replica_size += other.replica_size;
+        self.replica_count += other.replica_count;
+        self.pending_count += other.pending_count;
+        self.failed_count += other.failed_count;
 
         // Merge replication target stats
         for (target, stats) in &other.repl_target_stats {
             let entry = self.repl_target_stats.entry(target.clone()).or_default();
-            entry.replicated_size = entry.replicated_size.saturating_add(stats.replicated_size);
-            entry.replicated_count = entry.replicated_count.saturating_add(stats.replicated_count);
-            entry.pending_size = entry.pending_size.saturating_add(stats.pending_size);
-            entry.failed_size = entry.failed_size.saturating_add(stats.failed_size);
-            entry.pending_count = entry.pending_count.saturating_add(stats.pending_count);
-            entry.failed_count = entry.failed_count.saturating_add(stats.failed_count);
+            entry.replicated_size += stats.replicated_size;
+            entry.replicated_count += stats.replicated_count;
+            entry.pending_size += stats.pending_size;
+            entry.failed_size += stats.failed_size;
+            entry.pending_count += stats.pending_count;
+            entry.failed_count += stats.failed_count;
         }
     }
 }
@@ -1963,202 +1761,6 @@ mod tests {
         assert!(decoded.all_tier_stats.is_none());
     }
 
-    /// Scanner-written `.usage-cache.bin` bytes: a 2-element array of the
-    /// canonical 16-field map-encoded info block and one map-encoded entry.
-    /// Captured from the canonical writer's `marshal_msg` — see
-    /// `usage_cache_wire_format_is_pinned` in
-    /// `crates/scanner/src/data_usage_define.rs`, which pins these exact
-    /// bytes and documents regeneration. Hardcoded here because a
-    /// dev-dependency on rustfs-scanner would pull the whole ecstore tree
-    /// into this crate's test build, and a fixture generated at test runtime
-    /// could not detect writer drift anyway.
-    const SCANNER_USAGE_CACHE_WIRE_FIXTURE: &[u8] = &[
-        0x92, 0xde, 0x00, 0x10, 0xa4, 0x6e, 0x61, 0x6d, 0x65, 0xab, 0x77, 0x69, 0x72, 0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65,
-        0x74, 0xaa, 0x6e, 0x65, 0x78, 0x74, 0x5f, 0x63, 0x79, 0x63, 0x6c, 0x65, 0x07, 0xac, 0x6c, 0x65, 0x61, 0x64, 0x65, 0x72,
-        0x5f, 0x65, 0x70, 0x6f, 0x63, 0x68, 0x09, 0xab, 0x6c, 0x61, 0x73, 0x74, 0x5f, 0x75, 0x70, 0x64, 0x61, 0x74, 0x65, 0x92,
-        0xce, 0x65, 0x53, 0xf1, 0x00, 0x00, 0xac, 0x73, 0x6b, 0x69, 0x70, 0x5f, 0x68, 0x65, 0x61, 0x6c, 0x69, 0x6e, 0x67, 0xc3,
-        0xa9, 0x6c, 0x69, 0x66, 0x65, 0x63, 0x79, 0x63, 0x6c, 0x65, 0xc0, 0xab, 0x72, 0x65, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74,
-        0x69, 0x6f, 0x6e, 0xc0, 0xae, 0x66, 0x61, 0x69, 0x6c, 0x65, 0x64, 0x5f, 0x6f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x81,
-        0xb0, 0x77, 0x69, 0x72, 0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0x2f, 0x6c, 0x6f, 0x73, 0x74, 0x0b, 0xb1, 0x73,
-        0x63, 0x61, 0x6e, 0x5f, 0x72, 0x65, 0x73, 0x75, 0x6d, 0x65, 0x5f, 0x61, 0x66, 0x74, 0x65, 0x72, 0xb2, 0x77, 0x69, 0x72,
-        0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0x2f, 0x72, 0x65, 0x73, 0x75, 0x6d, 0x65, 0xaf, 0x73, 0x63, 0x61, 0x6e,
-        0x5f, 0x63, 0x68, 0x65, 0x63, 0x6b, 0x70, 0x6f, 0x69, 0x6e, 0x74, 0xc0, 0xad, 0x70, 0x65, 0x6e, 0x64, 0x69, 0x6e, 0x67,
-        0x5f, 0x68, 0x65, 0x61, 0x6c, 0x73, 0x91, 0x9a, 0xa6, 0x6f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0xab, 0x77, 0x69, 0x72, 0x65,
-        0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65, 0x74, 0xa6, 0x62, 0x72, 0x6f, 0x6b, 0x65, 0x6e, 0xc0, 0x01, 0x64, 0xcc, 0xc8, 0x03,
-        0xa8, 0x64, 0x65, 0x66, 0x65, 0x72, 0x72, 0x65, 0x64, 0xa6, 0x62, 0x75, 0x64, 0x67, 0x65, 0x74, 0xab, 0x6f, 0x62, 0x6a,
-        0x65, 0x63, 0x74, 0x5f, 0x6c, 0x6f, 0x63, 0x6b, 0xc0, 0xa6, 0x73, 0x6f, 0x75, 0x72, 0x63, 0x65, 0x92, 0x01, 0x02, 0xb1,
-        0x73, 0x6e, 0x61, 0x70, 0x73, 0x68, 0x6f, 0x74, 0x5f, 0x63, 0x6f, 0x6d, 0x70, 0x6c, 0x65, 0x74, 0x65, 0xc3, 0xb0, 0x73,
-        0x63, 0x61, 0x6e, 0x5f, 0x70, 0x6c, 0x61, 0x6e, 0x5f, 0x64, 0x69, 0x67, 0x65, 0x73, 0x74, 0xdc, 0x00, 0x20, 0x03, 0x03,
-        0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03,
-        0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0x03, 0xb0, 0x63, 0x61, 0x63, 0x68, 0x65, 0x5f, 0x6b, 0x65, 0x79,
-        0x5f, 0x66, 0x6f, 0x72, 0x6d, 0x61, 0x74, 0x01, 0x81, 0xab, 0x77, 0x69, 0x72, 0x65, 0x2d, 0x62, 0x75, 0x63, 0x6b, 0x65,
-        0x74, 0x8b, 0xa8, 0x63, 0x68, 0x69, 0x6c, 0x64, 0x72, 0x65, 0x6e, 0x90, 0xa4, 0x73, 0x69, 0x7a, 0x65, 0xcd, 0x10, 0x00,
-        0xa7, 0x6f, 0x62, 0x6a, 0x65, 0x63, 0x74, 0x73, 0x03, 0xa8, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x73, 0x05, 0xae,
-        0x64, 0x65, 0x6c, 0x65, 0x74, 0x65, 0x5f, 0x6d, 0x61, 0x72, 0x6b, 0x65, 0x72, 0x73, 0x01, 0xa9, 0x6f, 0x62, 0x6a, 0x5f,
-        0x73, 0x69, 0x7a, 0x65, 0x73, 0x9b, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xac, 0x6f, 0x62,
-        0x6a, 0x5f, 0x76, 0x65, 0x72, 0x73, 0x69, 0x6f, 0x6e, 0x73, 0x97, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0x00, 0xb1, 0x72,
-        0x65, 0x70, 0x6c, 0x69, 0x63, 0x61, 0x74, 0x69, 0x6f, 0x6e, 0x5f, 0x73, 0x74, 0x61, 0x74, 0x73, 0xc0, 0xa9, 0x63, 0x6f,
-        0x6d, 0x70, 0x61, 0x63, 0x74, 0x65, 0x64, 0xc3, 0xae, 0x66, 0x61, 0x69, 0x6c, 0x65, 0x64, 0x5f, 0x6f, 0x62, 0x6a, 0x65,
-        0x63, 0x74, 0x73, 0x02, 0xae, 0x61, 0x6c, 0x6c, 0x5f, 0x74, 0x69, 0x65, 0x72, 0x5f, 0x73, 0x74, 0x61, 0x74, 0x73, 0x91,
-        0x81, 0xa4, 0x57, 0x41, 0x52, 0x4d, 0x93, 0xcd, 0x08, 0x00, 0x02, 0x01,
-    ];
-
-    #[test]
-    fn thin_usage_cache_decodes_scanner_wire_fixture() {
-        let decoded =
-            DataUsageCache::unmarshal(SCANNER_USAGE_CACHE_WIRE_FIXTURE).expect("thin projection decodes a scanner-written cache");
-
-        // The six fields shared with the scanner's 16-field info block; the
-        // remaining ten (lifecycle, replication, checkpoint, heals, ...) must
-        // be skipped, not error.
-        assert_eq!(decoded.info.name, "wire-bucket");
-        assert_eq!(decoded.info.next_cycle, 7);
-        assert_eq!(
-            decoded.info.last_update,
-            Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1_700_000_000))
-        );
-        assert!(decoded.info.skip_healing);
-        assert_eq!(decoded.info.failed_objects.get("wire-bucket/lost"), Some(&11));
-        assert!(decoded.info.snapshot_complete);
-
-        // Entries use the shared canonical map-encoded type end to end.
-        let entry = decoded.cache.get("wire-bucket").expect("fixture entry decodes");
-        assert_eq!(entry.size, 4096);
-        assert_eq!(entry.objects, 3);
-        assert_eq!(entry.versions, 5);
-        assert_eq!(entry.delete_markers, 1);
-        assert!(entry.compacted);
-        assert_eq!(entry.failed_objects, 2);
-        assert_eq!(
-            entry.all_tier_stats.as_ref().and_then(|tiers| tiers.tiers.get("WARM")),
-            Some(&TierStats {
-                total_size: 2048,
-                num_versions: 2,
-                num_objects: 1,
-            })
-        );
-    }
-
-    /// Build a cache shaped like `bucket/{a,b/{c,d}},bucket/loose` with
-    /// distinct counters so aggregation is observable.
-    fn prefix_usage_fixture_cache() -> DataUsageCache {
-        let mut cache = DataUsageCache::default();
-        let mut insert = |path: &str, parent: &str, size: usize, objects: usize, versions: usize, delete_markers: usize| {
-            cache.replace(
-                path,
-                parent,
-                DataUsageEntry {
-                    size,
-                    objects,
-                    versions,
-                    delete_markers,
-                    ..Default::default()
-                },
-            );
-        };
-        insert("bucket", "", 0, 0, 0, 0);
-        insert("bucket/a", "bucket", 100, 1, 1, 0);
-        insert("bucket/b", "bucket", 0, 0, 0, 0);
-        insert("bucket/b/c", "bucket/b", 200, 2, 2, 1);
-        insert("bucket/b/d", "bucket/b", 40, 1, 3, 0);
-        insert("bucket/loose", "bucket", 10, 1, 1, 1);
-        cache
-    }
-
-    #[test]
-    fn prefix_usage_aggregates_bucket_root_and_one_level_below() {
-        let cache = prefix_usage_fixture_cache();
-
-        let root = cache
-            .prefix_usage("bucket", "", 100)
-            .expect("root query must find the bucket entry");
-        assert_eq!(root.usage.size, 350, "root aggregate flattens the whole subtree");
-        assert_eq!(root.usage.objects, 5);
-        assert_eq!(root.usage.versions, 7);
-        assert_eq!(root.usage.delete_markers, 2);
-        assert!(!root.compacted);
-        assert!(!root.truncated);
-        // Breakdown is one level: b (240) before a (100) before loose (10),
-        // each flattened to its own subtree total.
-        let names: Vec<(&str, u64)> = root
-            .sub_prefixes
-            .iter()
-            .map(|entry| (entry.prefix.as_str(), entry.usage.size))
-            .collect();
-        assert_eq!(names, vec![("b", 240), ("a", 100), ("loose", 10)]);
-    }
-
-    #[test]
-    fn prefix_usage_drills_into_arbitrary_prefixes() {
-        let cache = prefix_usage_fixture_cache();
-
-        let b = cache.prefix_usage("bucket", "b", 100).expect("nested prefix must resolve");
-        assert_eq!(b.usage.size, 240);
-        assert_eq!(b.usage.versions, 5);
-        let names: Vec<&str> = b.sub_prefixes.iter().map(|entry| entry.prefix.as_str()).collect();
-        assert_eq!(names, vec!["c", "d"]);
-
-        // Prefix slashes are normalized away.
-        let slashed = cache.prefix_usage("bucket", "/b/", 100).expect("slash-insensitive lookup");
-        assert_eq!(slashed.usage.size, 240);
-
-        assert!(cache.prefix_usage("bucket", "absent", 100).is_none(), "unknown prefix must be a miss");
-        assert!(cache.prefix_usage("other", "", 100).is_none(), "unknown bucket must be a miss");
-    }
-
-    #[test]
-    fn prefix_usage_reports_and_respects_truncation() {
-        let cache = prefix_usage_fixture_cache();
-        let capped = cache.prefix_usage("bucket", "", 2).expect("root query");
-        assert!(capped.truncated, "three children capped to two must flag truncation");
-        let names: Vec<&str> = capped.sub_prefixes.iter().map(|entry| entry.prefix.as_str()).collect();
-        assert_eq!(names, vec!["b", "a"], "largest prefixes survive the cut");
-    }
-
-    #[test]
-    fn prefix_usage_marks_compacted_entries() {
-        let mut cache = DataUsageCache::default();
-        cache.replace(
-            "bucket",
-            "",
-            DataUsageEntry {
-                size: 999,
-                objects: 9,
-                compacted: true,
-                ..Default::default()
-            },
-        );
-
-        let compacted = cache.prefix_usage("bucket", "", 100).expect("compacted root resolves");
-        assert!(compacted.compacted, "compaction must be visible to callers");
-        assert_eq!(compacted.usage.size, 999);
-        assert!(compacted.sub_prefixes.is_empty(), "a compacted entry carries no children");
-    }
-
-    #[test]
-    fn prefix_usage_rejects_cyclic_and_dangling_caches() {
-        // A self-referencing child (corrupt cache) must yield a miss for the
-        // whole query, not unbounded recursion.
-        let mut cache = prefix_usage_fixture_cache();
-        if let Some(entry) = cache.cache.get_mut("bucket/b") {
-            entry.children.insert("bucket/b".to_string());
-        }
-        assert!(cache.prefix_usage("bucket", "b", 100).is_none(), "a cyclic subtree must be rejected");
-        // The unaffected sibling still answers.
-        assert!(cache.prefix_usage("bucket", "a", 100).is_some());
-
-        // A child key with no entry (dangling link) is rejected rather than
-        // silently dropped: half a tree would under-report usage.
-        let mut dangling = prefix_usage_fixture_cache();
-        if let Some(entry) = dangling.cache.get_mut("bucket/b") {
-            entry.children.insert("bucket/b/ghost".to_string());
-        }
-        assert!(
-            dangling.prefix_usage("bucket", "b", 100).is_none(),
-            "a dangling child link must be rejected"
-        );
-    }
-
     #[test]
     fn hash_path_uses_portable_slash_semantics() {
         for (input, expected) in [
@@ -2181,8 +1783,6 @@ mod tests {
         let current = DataUsageInfo {
             last_update: Some(SystemTime::UNIX_EPOCH),
             usage_snapshot_complete: true,
-            usage_snapshot_converged: Some(false),
-            usage_snapshot_authoritative_baseline: Some(DataUsageSnapshotIdentity::default()),
             ..Default::default()
         };
         let encoded = rmp_serde::to_vec_named(&current).expect("encode current data usage snapshot");
@@ -2190,76 +1790,6 @@ mod tests {
 
         assert_eq!(legacy.buckets_count, 0);
         assert!(current.is_complete_bucket_usage_snapshot());
-        assert_eq!(current.usage_snapshot_converged, Some(false));
-    }
-
-    #[test]
-    fn convergence_marker_defaults_to_unknown_for_older_snapshots() {
-        let encoded = rmp_serde::to_vec_named(&DataUsageInfo {
-            last_update: Some(SystemTime::UNIX_EPOCH),
-            usage_snapshot_complete: true,
-            ..Default::default()
-        })
-        .expect("encode pre-convergence data usage snapshot");
-        let decoded: DataUsageInfo = rmp_serde::from_slice(&encoded).expect("decode older data usage snapshot");
-
-        assert!(decoded.is_complete_bucket_usage_snapshot());
-        assert_eq!(decoded.usage_snapshot_converged, None);
-    }
-
-    #[test]
-    fn observation_selection_is_clock_independent_and_baseline_fenced() {
-        let mut authoritative = DataUsageInfo {
-            last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(600)),
-            scanner_epoch: Some(7),
-            scanner_cycle: Some(10),
-            usage_snapshot_complete: true,
-            ..Default::default()
-        };
-        let observed = DataUsageInfo {
-            // A newer leader may have a slower wall clock.
-            last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(300)),
-            scanner_epoch: Some(8),
-            scanner_cycle: Some(1),
-            usage_snapshot_complete: true,
-            usage_snapshot_converged: Some(false),
-            usage_snapshot_authoritative_baseline: Some(authoritative.snapshot_identity()),
-            ..Default::default()
-        };
-
-        assert!(observed_data_usage_is_newer(&observed, &authoritative));
-
-        authoritative.last_update = Some(SystemTime::UNIX_EPOCH + Duration::from_secs(601));
-        assert!(
-            !observed_data_usage_is_newer(&observed, &authoritative),
-            "an old-binary namespace mutation must fence the prior bucket incarnation regardless of clock skew"
-        );
-    }
-
-    #[test]
-    fn observation_selection_requires_nonconverged_complete_newer_data() {
-        let authoritative = DataUsageInfo {
-            last_update: Some(SystemTime::UNIX_EPOCH),
-            scanner_epoch: Some(2),
-            scanner_cycle: Some(10),
-            usage_snapshot_complete: true,
-            ..Default::default()
-        };
-        let baseline = Some(authoritative.snapshot_identity());
-        let candidate = |epoch, cycle, converged, complete| DataUsageInfo {
-            last_update: Some(SystemTime::UNIX_EPOCH + Duration::from_secs(1)),
-            scanner_epoch: Some(epoch),
-            scanner_cycle: Some(cycle),
-            usage_snapshot_complete: complete,
-            usage_snapshot_converged: converged,
-            usage_snapshot_authoritative_baseline: baseline,
-            ..Default::default()
-        };
-
-        assert!(observed_data_usage_is_newer(&candidate(2, 11, Some(false), true), &authoritative));
-        assert!(!observed_data_usage_is_newer(&candidate(2, 9, Some(false), true), &authoritative));
-        assert!(!observed_data_usage_is_newer(&candidate(2, 11, Some(true), true), &authoritative));
-        assert!(!observed_data_usage_is_newer(&candidate(2, 11, Some(false), false), &authoritative));
     }
 
     #[test]
@@ -2318,64 +1848,6 @@ mod tests {
         assert_eq!(usage1.versions_count, 15);
     }
 
-    #[test]
-    fn size_summary_add_saturates_instead_of_overflowing() {
-        // The scanner folds one summary per object into a per-prefix total, so a
-        // counter at its ceiling must stay there rather than panic in a debug
-        // build or wrap in a release one (backlog#1828).
-        let mut summary = SizeSummary {
-            total_size: usize::MAX,
-            versions: usize::MAX,
-            replicated_size: i64::MAX,
-            pending_size: i64::MAX,
-            failed_size: i64::MAX,
-            replica_size: i64::MAX,
-            ..Default::default()
-        };
-        summary.repl_target_stats.insert(
-            "arn".to_string(),
-            ReplTargetSizeSummary {
-                replicated_size: i64::MAX,
-                pending_size: i64::MAX,
-                failed_size: i64::MAX,
-                ..Default::default()
-            },
-        );
-
-        let mut increment = SizeSummary {
-            total_size: 1,
-            versions: 1,
-            replicated_size: 1,
-            pending_size: 1,
-            failed_size: 1,
-            replica_size: 1,
-            ..Default::default()
-        };
-        increment.repl_target_stats.insert(
-            "arn".to_string(),
-            ReplTargetSizeSummary {
-                replicated_size: 1,
-                pending_size: 1,
-                failed_size: 1,
-                ..Default::default()
-            },
-        );
-
-        summary.add(&increment);
-
-        assert_eq!(summary.total_size, usize::MAX);
-        assert_eq!(summary.versions, usize::MAX);
-        assert_eq!(summary.replicated_size, i64::MAX);
-        assert_eq!(summary.pending_size, i64::MAX);
-        assert_eq!(summary.failed_size, i64::MAX);
-        assert_eq!(summary.replica_size, i64::MAX);
-
-        let target = summary.repl_target_stats.get("arn").expect("target survives the merge");
-        assert_eq!(target.replicated_size, i64::MAX);
-        assert_eq!(target.pending_size, i64::MAX);
-        assert_eq!(target.failed_size, i64::MAX);
-    }
-
     #[test]
     fn test_size_summary_add() {
         let mut summary1 = SizeSummary::new();

crates/e2e_test/AGENTS.md

@@ -28,3 +28,4 @@ follow.
 ## Suggested Validation
 
 - `cargo test --package e2e_test`
+- Full gate before commit: `make pre-commit`

crates/e2e_test/Cargo.toml

@@ -96,6 +96,7 @@ tokio-stream = { workspace = true }
 rustfs-madmin.workspace = true
 rustfs-filemeta.workspace = true
 bytes = { workspace = true, features = ["serde"] }
+serial_test = { workspace = true }
 aws-sdk-s3 = { workspace = true, default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
 aws-sdk-sts = { workspace = true, default-features = false, features = ["default-https-client", "rt-tokio"] }
 aws-config = { workspace = true }

crates/e2e_test/README.md

@@ -48,14 +48,16 @@ cargo nextest run --profile e2e-smoke -p e2e_test
 cargo nextest run -j1 --run-ignored ignored-only -p rustfs-scanner -p rustfs \
   -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))'
 
+# Protocols suite — fixed ports, MUST be single-threaded, gated by build features
+RUSTFS_BUILD_FEATURES=ftps,webdav,sftp \
+  cargo test -p e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
 ```
 
 The protocols suite has its own contract (fixed bind ports 9022–9301,
-single-worker execution, feature-gated scheduling) documented in
+`--test-threads=1`, feature-gated scheduling) documented in
 [`src/protocols/README.md`](src/protocols/README.md). `RUSTFS_BUILD_FEATURES`
 selects which features the spawned binary is built with; leave it unset to run
-every protocol entry. Use the exact profile command under
-[Troubleshooting](#troubleshooting) for CI-equivalent execution.
+every protocol entry.
 
 ### `#[ignore]` semantics
 
@@ -157,26 +159,27 @@ construction (random port + isolated temp dir) and need no serialization.
 ## CI map
 
 `e2e_test` is **excluded** from the main `cargo nextest run --profile ci --all`
-pass (`--exclude e2e_test`) — the whole crate is too slow to gate every PR.
-Subsets join CI through nextest profiles; the fixed-port protocol suite uses
-the same profile for membership and execution with one nightly worker.
+pass ([`.github/workflows/ci.yml`](../../.github/workflows/ci.yml) line 158,
+`--exclude e2e_test`) — the whole crate is too slow to gate every PR. Subsets
+join CI through the nextest profile system only (never as ad-hoc jobs):
 
 | Suite | Runs where | Status |
 | --- | --- | --- |
 | Smoke subset (`e2e-smoke` profile) | `e2e-tests` job, every PR | **Active** (backlog#1149 ci-4) |
-| Full single-node suite (`e2e-full` profile) | `e2e-full` job, merge queue + main | **Active** (backlog#1149 ci-5) |
 | `s3s-e2e` black-box | `e2e-tests` + `e2e-tests-rio-v2` jobs | **Active** (external conformance tool) |
 | ILM / lifecycle (ignored) | `test-ilm-integration-serial` lane, `-j1` | **Active** (backlog#1148 ilm-1) |
-| KMS suite | `e2e-full` job, merge queue + main | **Active** |
-| Cluster faults (`e2e-nightly` profile) | consolidated nightly workflow | **Active** (backlog#1149 ci-7) |
-| Protocols (FTPS/WebDAV/SFTP) | consolidated nightly workflow, serial | **Active** (backlog#1149 ci-7) |
+| KMS suite | — | Not in CI yet (backlog#1149 ci-5) |
+| Protocols (FTPS/WebDAV/SFTP) | — | Not in CI yet (backlog#1149 ci-7) |
 | Replication (fast subset) | `e2e-smoke` profile, `e2e-tests` job, every PR | **Active** (backlog#1147 repl-1) |
-| Replication (slow + multi-node) | `e2e-repl-nightly` profile, consolidated nightly workflow | **Active** (backlog#1147 repl-1) |
-| `reliant/*` | 19 tests in PR smoke; remaining default tests in `e2e-full` | **Active** except `#[ignore]` |
+| Replication (slow + dual-node) | `e2e-repl-nightly` profile, scheduled workflow | **Active** (backlog#1147 repl-1) |
+| `reliant/*` (pre-started server) | — | Manual only |
 
-The profile filters in [`.config/nextest.toml`](../../.config/nextest.toml) are
-the wiring source of truth. Committed test-ID digests under
-`.config/e2e-*-selection.txt` make every membership change explicit.
+Links: [`ci.yml`](../../.github/workflows/ci.yml) `e2e-tests` (line 347),
+`test-ilm-integration-serial` (line 196). The `e2e-smoke` `default-filter` in
+[`.config/nextest.toml`](../../.config/nextest.toml) is the **single wiring
+mechanism** — extend that filter (or add a sibling profile) to admit more
+tests; do not add e2e jobs to `ci.yml`. repl-1 / ilm-3 are landing in parallel
+and may add lanes; keep the table above easy to extend.
 
 ## Troubleshooting
 
@@ -185,15 +188,9 @@ the wiring source of truth. Committed test-ID digests under
 ```bash
 # Smoke (e2e-tests job) — includes the 20 fast replication tests
 cargo nextest run --profile e2e-smoke -p e2e_test
-# Full single-node merge/main lane
-cargo nextest run --profile e2e-full -p e2e_test
-# Cluster fault nightly lane
-cargo nextest run --profile e2e-nightly -p e2e_test
-# Replication nightly lane; install awscurl so STS paths do not skip
+# Replication nightly lane (16 slow + dual-node tests; install awscurl for the
+# STS dual-node test, else it skips gracefully)
 cargo nextest run --profile e2e-repl-nightly -p e2e_test
-# Fixed-port protocol nightly lane
-RUSTFS_BUILD_FEATURES=ftps,webdav,sftp \
-  cargo nextest run -j 1 --profile e2e-protocols -p e2e_test --no-capture
 # ILM serial lane
 cargo nextest run -j1 --run-ignored ignored-only -p rustfs-scanner -p rustfs \
   -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))'
@@ -276,6 +273,4 @@ current subset is.
 `docs/testing/e2e-suite-inventory.md` records the per-module test counts as
 listed by `cargo nextest list -p e2e_test`. Regenerate it when adding or
 moving e2e tests so acceptance numbers in the test-strategy issues
-(backlog#1147–#1155) stay auditable. When a profile membership change is
-intentional, review its JSON listing before updating the matching
-`.config/e2e-*-selection.txt` test-ID digest.
+(backlog#1147–#1155) stay auditable.

crates/e2e_test/src/admin_auth_test.rs

@@ -38,6 +38,7 @@ mod tests {
     use rustfs_signer::constants::UNSIGNED_PAYLOAD;
     use rustfs_signer::sign_v4;
     use s3s::Body;
+    use serial_test::serial;
     use std::error::Error;
     use std::io::Read;
     use std::process::{Command, Stdio};
@@ -161,6 +162,7 @@ mod tests {
     /// A fully authenticated but non-admin credential must be rejected with
     /// `403 AccessDenied` on an admin API, while the root credential succeeds.
     #[tokio::test(flavor = "multi_thread")]
+    #[serial]
     async fn non_admin_credential_denied_on_admin_api() -> Result<(), Box<dyn Error + Send + Sync>> {
         init_logging();
         let mut env = RustFSTestEnvironment::new().await?;
@@ -196,6 +198,7 @@ mod tests {
     }
 
     #[tokio::test(flavor = "multi_thread")]
+    #[serial]
     async fn non_admin_credential_denied_on_manual_transition_run() -> Result<(), Box<dyn Error + Send + Sync>> {
         init_logging();
         let mut env = RustFSTestEnvironment::new().await?;
@@ -323,6 +326,7 @@ mod tests {
     /// credential is accepted and the old one is rejected, on both the S3 data
     /// plane and the admin plane.
     #[tokio::test(flavor = "multi_thread")]
+    #[serial]
     async fn root_credential_rotation_takes_effect() -> Result<(), Box<dyn Error + Send + Sync>> {
         init_logging();
         let mut env = RustFSTestEnvironment::new().await?;
@@ -385,6 +389,7 @@ mod tests {
     /// runtime. We capture the child's stdout/stderr directly (the shared
     /// harness inherits stdio) and poll for the warning until it appears.
     #[tokio::test(flavor = "multi_thread")]
+    #[serial]
     async fn default_credentials_emit_startup_warning() -> Result<(), Box<dyn Error + Send + Sync>> {
         init_logging();
 

crates/e2e_test/src/admin_iam_crud_test.rs

@@ -26,13 +26,12 @@
 //! Later batches tracked on backlog#1154: config get/set, info, pools status,
 //! group lifecycle, import/export IAM.
 
-use crate::common::{
-    RustFSTestEnvironment, admin_ok, admin_request, admin_request_with_session_token, build_test_sts_client, init_logging,
-};
+use crate::common::{RustFSTestEnvironment, admin_ok, admin_request, init_logging};
 use aws_sdk_s3::config::{Credentials, Region};
 use aws_sdk_s3::primitives::ByteStream;
 use aws_sdk_s3::{Client, Config};
 use reqwest::StatusCode;
+use serial_test::serial;
 use std::error::Error;
 use tokio::time::{Duration, sleep};
 
@@ -88,264 +87,10 @@ fn bucket_rw_policy(bucket: &str) -> String {
     .to_string()
 }
 
-async fn create_user_with_service_account_update_policy(
-    env: &RustFSTestEnvironment,
-    user: &str,
-    secret: &str,
-    policy: &str,
-) -> TestResult {
-    admin_ok(
-        env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/add-user?accessKey={user}"),
-        Some(serde_json::json!({ "secretKey": secret, "status": "enabled" }).to_string()),
-    )
-    .await?;
-    admin_ok(
-        env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/add-canned-policy?name={policy}"),
-        Some(
-            serde_json::json!({
-                "Version": "2012-10-17",
-                "Statement": [
-                    {
-                        "Effect": "Allow",
-                        "Action": ["admin:UpdateServiceAccount"]
-                    },
-                    {
-                        "Effect": "Allow",
-                        "Action": ["sts:AssumeRole"],
-                        "Resource": ["arn:aws:s3:::*"]
-                    }
-                ]
-            })
-            .to_string(),
-        ),
-    )
-    .await?;
-    admin_ok(
-        env,
-        http::Method::POST,
-        "/rustfs/admin/v3/idp/builtin/policy/attach",
-        Some(serde_json::json!({ "policies": [policy], "user": user }).to_string()),
-    )
-    .await?;
-    Ok(())
-}
-
-async fn create_service_account_for(
-    env: &RustFSTestEnvironment,
-    parent: &str,
-) -> Result<(String, String), Box<dyn Error + Send + Sync>> {
-    let response = admin_ok(
-        env,
-        http::Method::PUT,
-        "/rustfs/admin/v3/add-service-accounts",
-        Some(serde_json::json!({ "targetUser": parent }).to_string()),
-    )
-    .await?;
-    let response: serde_json::Value = serde_json::from_str(&response)?;
-    let access_key = response["credentials"]["accessKey"]
-        .as_str()
-        .ok_or("service account response should contain credentials.accessKey")?
-        .to_owned();
-    let secret_key = response["credentials"]["secretKey"]
-        .as_str()
-        .ok_or("service account response should contain credentials.secretKey")?
-        .to_owned();
-    Ok((access_key, secret_key))
-}
-
-async fn assert_admin_status(
-    env: &RustFSTestEnvironment,
-    credentials: (&str, &str, Option<&str>),
-    path: &str,
-    body: String,
-    expected: StatusCode,
-    context: &str,
-) -> TestResult {
-    let (access_key, secret_key, session_token) = credentials;
-    let (status, response) =
-        admin_request_with_session_token(&env.url, http::Method::POST, path, Some(body), access_key, secret_key, session_token)
-            .await?;
-    assert_eq!(status, expected, "{context}: got {status}: {response}");
-    if expected == StatusCode::FORBIDDEN {
-        assert!(response.contains("AccessDenied"), "{context}: expected AccessDenied body, got {response}");
-    }
-    Ok(())
-}
-
-#[tokio::test]
-async fn test_update_service_account_enforces_owner_and_parent_scope() -> TestResult {
-    init_logging();
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-
-    let parent = "updateparent";
-    let parent_secret = "updateparentsecret";
-    let outsider = "updateoutsider";
-    let outsider_secret = "updateoutsidersecret";
-    let ordinary = "updateordinary";
-    let ordinary_secret = "updateordinarysecret";
-    create_user_with_service_account_update_policy(&env, parent, parent_secret, "update-parent-policy").await?;
-    create_user_with_service_account_update_policy(&env, outsider, outsider_secret, "update-outsider-policy").await?;
-    admin_ok(
-        &env,
-        http::Method::POST,
-        "/rustfs/admin/v3/idp/builtin/policy/attach",
-        Some(serde_json::json!({ "policies": ["consoleAdmin"], "user": outsider }).to_string()),
-    )
-    .await?;
-    admin_ok(
-        &env,
-        http::Method::PUT,
-        &format!("/rustfs/admin/v3/add-user?accessKey={ordinary}"),
-        Some(serde_json::json!({ "secretKey": ordinary_secret, "status": "enabled" }).to_string()),
-    )
-    .await?;
-
-    let (target_access_key, _) = create_service_account_for(&env, parent).await?;
-    let target_path = format!("/rustfs/admin/v3/update-service-account?accessKey={target_access_key}");
-
-    assert_admin_status(
-        &env,
-        (&env.access_key, &env.secret_key, None),
-        &target_path,
-        serde_json::json!({}).to_string(),
-        StatusCode::NO_CONTENT,
-        "root no-op update across parents must succeed",
-    )
-    .await?;
-
-    let custom_policy = serde_json::json!({
-        "Version": "2012-10-17",
-        "Statement": [{
-            "Effect": "Allow",
-            "Action": ["s3:GetObject"],
-            "Resource": ["arn:aws:s3:::update-scope/*"]
-        }]
-    });
-    assert_admin_status(
-        &env,
-        (&env.access_key, &env.secret_key, None),
-        &target_path,
-        serde_json::json!({ "newPolicy": custom_policy }).to_string(),
-        StatusCode::NO_CONTENT,
-        "root implied-to-custom update across parents must succeed",
-    )
-    .await?;
-
-    assert_admin_status(
-        &env,
-        (parent, parent_secret, None),
-        &target_path,
-        serde_json::json!({ "newDescription": "updated by parent" }).to_string(),
-        StatusCode::NO_CONTENT,
-        "parent with UpdateServiceAccount may update its own service account",
-    )
-    .await?;
-
-    let takeover = serde_json::json!({
-        "newSecretKey": "cross-parent-takeover-secret",
-        "newDescription": "cross-parent takeover"
-    })
-    .to_string();
-    assert_admin_status(
-        &env,
-        (ordinary, ordinary_secret, None),
-        &target_path,
-        takeover.clone(),
-        StatusCode::FORBIDDEN,
-        "ordinary user must not update another parent's service account",
-    )
-    .await?;
-    assert_admin_status(
-        &env,
-        (outsider, outsider_secret, None),
-        &target_path,
-        takeover.clone(),
-        StatusCode::FORBIDDEN,
-        "non-owner consoleAdmin must not update across parents",
-    )
-    .await?;
-
-    let (derived_access_key, derived_secret_key) = create_service_account_for(&env, outsider).await?;
-    assert_admin_status(
-        &env,
-        (&derived_access_key, &derived_secret_key, None),
-        &target_path,
-        takeover.clone(),
-        StatusCode::FORBIDDEN,
-        "service-account credential must not update across parents",
-    )
-    .await?;
-
-    let assumed = build_test_sts_client(&env.url, outsider, outsider_secret, None, "e2e-admin-update-service-account")
-        .assume_role()
-        .role_arn("arn:aws:iam::123456789012:role/update-service-account")
-        .role_session_name("update-service-account-scope")
-        .send()
-        .await?;
-    let temporary = assumed
-        .credentials()
-        .ok_or("AssumeRole response should contain credentials")?;
-    assert_admin_status(
-        &env,
-        (temporary.access_key_id(), temporary.secret_access_key(), Some(temporary.session_token())),
-        &target_path,
-        takeover,
-        StatusCode::FORBIDDEN,
-        "temporary credential must not update across parents",
-    )
-    .await?;
-
-    let info = admin_ok(
-        &env,
-        http::Method::GET,
-        &format!("/rustfs/admin/v3/info-service-account?accessKey={target_access_key}"),
-        None,
-    )
-    .await?;
-    let info: serde_json::Value = serde_json::from_str(&info)?;
-    assert_eq!(
-        info["impliedPolicy"].as_bool(),
-        Some(false),
-        "root update must replace the implied policy with a custom policy"
-    );
-    assert!(
-        info["policy"].as_str().is_some_and(|policy| policy.contains("s3:GetObject")),
-        "custom policy must round-trip through the handler: {info}"
-    );
-    assert_eq!(
-        info["description"].as_str(),
-        Some("updated by parent"),
-        "denied takeover attempts must not mutate target"
-    );
-
-    let (missing_status, missing_body) = admin_request(
-        &env.url,
-        http::Method::POST,
-        "/rustfs/admin/v3/update-service-account?accessKey=missing-service-account",
-        Some(serde_json::json!({}).to_string()),
-        &env.access_key,
-        &env.secret_key,
-    )
-    .await?;
-    assert_eq!(missing_status, StatusCode::NOT_FOUND, "missing target must fail closed: {missing_body}");
-    assert!(
-        missing_body.contains("NoSuchResource"),
-        "missing target must preserve the lookup error: {missing_body}"
-    );
-
-    env.stop_server();
-    Ok(())
-}
-
 /// Full user -> policy -> service-account lifecycle, proving each management
 /// call takes effect on the data plane, not just that the endpoint answers 200.
 #[tokio::test]
+#[serial]
 async fn test_admin_user_policy_service_account_crud_lifecycle() -> TestResult {
     init_logging();
 
@@ -570,6 +315,7 @@ async fn test_admin_user_policy_service_account_crud_lifecycle() -> TestResult {
 /// non-admin credential with 403 AccessDenied (sec-4 assertion pattern; the
 /// gate implementation itself is owned by sec-4 / admin_auth_test).
 #[tokio::test]
+#[serial]
 async fn test_admin_iam_endpoints_deny_non_admin_credential() -> TestResult {
     init_logging();
 

crates/e2e_test/src/admin_timeout_regression_test.rs

@@ -21,6 +21,7 @@ use rustfs_signer::constants::UNSIGNED_PAYLOAD;
 use rustfs_signer::sign_v4;
 use s3s::Body;
 use serde::Deserialize;
+use serial_test::serial;
 use std::error::Error;
 use std::process::Command;
 use tokio::time::{Duration, sleep, timeout};
@@ -99,6 +100,7 @@ fn offline_server_count(info: &InfoMessage) -> usize {
 }
 
 #[tokio::test(flavor = "multi_thread")]
+#[serial]
 async fn test_single_admin_timeout_does_not_immediately_mark_peer_offline() -> Result<(), Box<dyn Error + Send + Sync>> {
     init_logging();
 

crates/e2e_test/src/anonymous_access_test.rs

@@ -18,6 +18,7 @@
 
 use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
 use aws_sdk_s3::types::PublicAccessBlockConfiguration;
+use serial_test::serial;
 use tracing::info;
 
 async fn setup_public_bucket(
@@ -72,6 +73,7 @@ async fn anonymous_get_object(
 /// Issue #2036: Anonymous GetObject should succeed when bucket policy allows it
 /// and no PublicAccessBlock configuration exists (ConfigNotFound).
 #[tokio::test]
+#[serial]
 async fn test_anonymous_access_allowed_when_public_access_block_missing() -> Result<(), Box<dyn std::error::Error + Send + Sync>>
 {
     init_logging();
@@ -98,6 +100,7 @@ async fn test_anonymous_access_allowed_when_public_access_block_missing() -> Res
 
 /// Anonymous GetObject should be denied when RestrictPublicBuckets is true.
 #[tokio::test]
+#[serial]
 async fn test_anonymous_access_denied_when_restrict_public_buckets_enabled()
 -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
     init_logging();
@@ -134,6 +137,7 @@ async fn test_anonymous_access_denied_when_restrict_public_buckets_enabled()
 /// Anonymous GetObject should succeed when PublicAccessBlock exists but
 /// RestrictPublicBuckets is explicitly false.
 #[tokio::test]
+#[serial]
 async fn test_anonymous_access_allowed_when_restrict_public_buckets_disabled()
 -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
     init_logging();
@@ -172,6 +176,7 @@ async fn test_anonymous_access_allowed_when_restrict_public_buckets_disabled()
 /// reaches authorization through a fallback branch, and that branch has to apply the
 /// same public-access gate as a direct grant.
 #[tokio::test]
+#[serial]
 async fn ghsa_x298_anonymous_list_object_versions_denied_when_restrict_public_buckets_enabled()
 -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
     init_logging();

crates/e2e_test/src/api_rate_limit_test.rs

@@ -18,11 +18,13 @@
 //! completely inert with default configuration.
 
 use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
+use serial_test::serial;
 use tracing::info;
 
 type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
 
 #[tokio::test]
+#[serial]
 async fn api_rate_limit_enforces_429_with_retry_after_when_enabled() -> TestResult {
     init_logging();
     let mut env = RustFSTestEnvironment::new().await?;
@@ -85,6 +87,7 @@ async fn api_rate_limit_enforces_429_with_retry_after_when_enabled() -> TestResu
 }
 
 #[tokio::test]
+#[serial]
 async fn api_rate_limit_bucket_dimension_throttles_per_bucket() -> TestResult {
     init_logging();
     let mut env = RustFSTestEnvironment::new().await?;
@@ -130,6 +133,7 @@ async fn api_rate_limit_bucket_dimension_throttles_per_bucket() -> TestResult {
 }
 
 #[tokio::test]
+#[serial]
 async fn api_rate_limit_stays_inert_by_default() -> TestResult {
     init_logging();
     let mut env = RustFSTestEnvironment::new().await?;

crates/e2e_test/src/archive_download_integrity_test.rs

@@ -24,6 +24,7 @@ mod tests {
     use rustfs_signer::constants::UNSIGNED_PAYLOAD;
     use rustfs_signer::{pre_sign_v4, sign_v4};
     use s3s::Body;
+    use serial_test::serial;
     use sha2::{Digest, Sha256};
     use std::error::Error;
     use std::io::{Cursor, Write};
@@ -338,6 +339,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_archive_put_allows_content_encoding_by_default() -> Result<(), Box<dyn Error + Send + Sync>> {
         init_logging();
         let mut env = RustFSTestEnvironment::new().await?;
@@ -365,6 +367,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_archive_put_rejects_content_encoding_when_strict_mode_enabled() -> Result<(), Box<dyn Error + Send + Sync>> {
         init_logging();
         let mut env = RustFSTestEnvironment::new().await?;
@@ -388,6 +391,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_archive_put_with_aws_chunked_does_not_persist_content_encoding_by_default()
     -> Result<(), Box<dyn Error + Send + Sync>> {
         init_logging();
@@ -423,6 +427,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_archive_put_with_aws_chunked_and_effective_encoding_roundtrips_by_default()
     -> Result<(), Box<dyn Error + Send + Sync>> {
         init_logging();
@@ -458,6 +463,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_archive_put_with_aws_chunked_allowed_when_strict_mode_enabled() -> Result<(), Box<dyn Error + Send + Sync>> {
         init_logging();
         let mut env = RustFSTestEnvironment::new().await?;
@@ -492,6 +498,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_archive_put_with_aws_chunked_and_effective_encoding_rejects_when_strict_mode_enabled()
     -> Result<(), Box<dyn Error + Send + Sync>> {
         init_logging();
@@ -522,6 +529,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_archive_download_roundtrip_with_http_compression_enabled() -> Result<(), Box<dyn Error + Send + Sync>> {
         init_logging();
         let mut env = RustFSTestEnvironment::new().await?;
@@ -583,6 +591,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_archive_multipart_roundtrip_preserves_bytes() -> Result<(), Box<dyn Error + Send + Sync>> {
         init_logging();
         let mut env = RustFSTestEnvironment::new().await?;
@@ -678,6 +687,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_multipart_get_ignores_empty_conditional_etag_headers() -> Result<(), Box<dyn Error + Send + Sync>> {
         init_logging();
         let mut env = RustFSTestEnvironment::new().await?;
@@ -713,6 +723,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_archive_multipart_with_aws_chunked_and_effective_encoding_roundtrips_by_default()
     -> Result<(), Box<dyn Error + Send + Sync>> {
         init_logging();
@@ -742,6 +753,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_archive_multipart_with_aws_chunked_allowed_when_strict_mode_enabled() -> Result<(), Box<dyn Error + Send + Sync>>
     {
         init_logging();
@@ -771,6 +783,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_archive_multipart_with_aws_chunked_and_effective_encoding_rejects_when_strict_mode_enabled()
     -> Result<(), Box<dyn Error + Send + Sync>> {
         init_logging();
@@ -803,6 +816,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_presigned_get_and_reverse_proxy_preserve_multipart_bytes() -> Result<(), Box<dyn Error + Send + Sync>> {
         init_logging();
         let mut env = RustFSTestEnvironment::new().await?;

crates/e2e_test/src/bucket_logging_test.rs

@@ -16,17 +16,93 @@
 
 #[cfg(test)]
 mod tests {
-    use crate::common::{RustFSTestEnvironment, init_logging, signed_s3_request};
+    use crate::common::{RustFSTestEnvironment, init_logging};
     use aws_sdk_s3::error::ProvideErrorMetadata;
     use aws_sdk_s3::types::{
         AccelerateConfiguration, BucketAccelerateStatus, BucketLoggingStatus, IndexDocument, LoggingEnabled, Payer,
         RequestPaymentConfiguration, WebsiteConfiguration,
     };
-    use http::Method;
-    use http::header::CONTENT_TYPE;
+    use serial_test::serial;
+    use std::path::PathBuf;
+    use std::process::Command;
     use tracing::info;
 
+    fn awscurl_binary_path() -> PathBuf {
+        std::env::var_os("AWSCURL_PATH")
+            .map(PathBuf::from)
+            .unwrap_or_else(|| PathBuf::from("awscurl"))
+    }
+
+    fn awscurl_available() -> bool {
+        Command::new(awscurl_binary_path()).arg("--version").output().is_ok()
+    }
+
+    fn execute_s3_awscurl(
+        method: &str,
+        url: &str,
+        access_key: &str,
+        secret_key: &str,
+    ) -> Result<String, Box<dyn std::error::Error + Send + Sync>> {
+        let output = Command::new(awscurl_binary_path())
+            .args([
+                "--service",
+                "s3",
+                "--region",
+                "us-east-1",
+                "--access_key",
+                access_key,
+                "--secret_key",
+                secret_key,
+                "-i",
+                "-X",
+                method,
+                url,
+            ])
+            .output()?;
+        if !output.status.success() {
+            let stderr = String::from_utf8_lossy(&output.stderr);
+            let stdout = String::from_utf8_lossy(&output.stdout);
+            return Err(format!("awscurl failed: stderr='{stderr}', stdout='{stdout}'").into());
+        }
+        Ok(String::from_utf8_lossy(&output.stdout).to_string())
+    }
+
+    fn parse_status(raw: &str) -> Option<u16> {
+        raw.lines()
+            .filter_map(|line| {
+                if line.starts_with("HTTP/") {
+                    line.split_whitespace().nth(1)?.parse::<u16>().ok()
+                } else {
+                    None
+                }
+            })
+            .next_back()
+    }
+
+    fn parse_body(raw: &str) -> String {
+        if let Some(pos) = raw.rfind("\r\n\r\n") {
+            return raw[pos + 4..].to_string();
+        }
+        if let Some(pos) = raw.rfind("\n\n") {
+            return raw[pos + 2..].to_string();
+        }
+        String::new()
+    }
+
+    fn parse_headers(raw: &str) -> String {
+        let start = raw.rfind("HTTP/").unwrap_or(0);
+        let tail = &raw[start..];
+        if let Some(pos) = tail.find("\r\n\r\n") {
+            return tail[..pos].to_string();
+        }
+        if let Some(pos) = tail.find("\n\n") {
+            return tail[..pos].to_string();
+        }
+        tail.to_string()
+    }
+
     #[tokio::test]
+    #[serial]
     async fn test_dummy_bucket_compatibility_endpoints() {
         init_logging();
         info!("Starting test: dummy-compat bucket APIs should match S3-compatible behavior");
@@ -234,6 +310,7 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_dummy_bucket_compatibility_endpoints_no_such_bucket() {
         init_logging();
         info!("Starting test: dummy-compat bucket APIs should return NoSuchBucket for missing bucket");
@@ -389,9 +466,14 @@ mod tests {
     }
 
     #[tokio::test]
+    #[serial]
     async fn test_dummy_bucket_endpoints_http_contracts() {
         init_logging();
         info!("Starting test: dummy-compat bucket API HTTP contracts");
+        if !awscurl_available() {
+            info!("Skipping test_dummy_bucket_endpoints_http_contracts: awscurl binary not found");
+            return;
+        }
 
         let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
         env.start_rustfs_server(vec![]).await.expect("Failed to start RustFS");
@@ -406,112 +488,56 @@ mod tests {
             .await
             .expect("Failed to create bucket");
 
-        let logging_response = signed_s3_request(
-            Method::GET,
-            &format!("{}/{bucket}?logging=", env.url),
-            None,
-            None,
-            &env.access_key,
-            &env.secret_key,
-        )
-        .await
-        .expect("GetBucketLogging HTTP request failed");
-        assert_eq!(logging_response.status(), 200, "GetBucketLogging should return 200");
-        let logging_body = logging_response
-            .text()
-            .await
-            .expect("Failed to read GetBucketLogging response body");
+        let logging_raw = execute_s3_awscurl("GET", &format!("{}/{bucket}?logging=", env.url), &env.access_key, &env.secret_key)
+            .expect("GetBucketLogging HTTP request failed");
+        assert_eq!(parse_status(&logging_raw), Some(200), "GetBucketLogging should return 200");
+        let logging_body = parse_body(&logging_raw);
         assert!(
             logging_body.contains("<BucketLoggingStatus"),
             "GetBucketLogging response should contain BucketLoggingStatus XML, got: {logging_body}"
         );
 
-        let accel_response = signed_s3_request(
-            Method::GET,
-            &format!("{}/{bucket}?accelerate=", env.url),
-            None,
-            None,
-            &env.access_key,
-            &env.secret_key,
-        )
-        .await
-        .expect("GetBucketAccelerateConfiguration HTTP request failed");
-        assert_eq!(accel_response.status(), 200, "GetBucketAccelerateConfiguration should return 200");
-        let accel_body = accel_response
-            .text()
-            .await
-            .expect("Failed to read GetBucketAccelerateConfiguration response body");
+        let accel_raw = execute_s3_awscurl("GET", &format!("{}/{bucket}?accelerate=", env.url), &env.access_key, &env.secret_key)
+            .expect("GetBucketAccelerateConfiguration HTTP request failed");
+        assert_eq!(parse_status(&accel_raw), Some(200), "GetBucketAccelerateConfiguration should return 200");
+        let accel_body = parse_body(&accel_raw);
         assert!(
             accel_body.contains("<AccelerateConfiguration"),
             "GetBucketAccelerateConfiguration response should contain AccelerateConfiguration XML, got: {accel_body}"
         );
 
-        let payment_response = signed_s3_request(
-            Method::GET,
-            &format!("{}/{bucket}?requestPayment=", env.url),
-            None,
-            None,
-            &env.access_key,
-            &env.secret_key,
-        )
-        .await
-        .expect("GetBucketRequestPayment HTTP request failed");
-        assert_eq!(payment_response.status(), 200, "GetBucketRequestPayment should return 200");
-        let payment_body = payment_response
-            .text()
-            .await
-            .expect("Failed to read GetBucketRequestPayment response body");
+        let payment_raw =
+            execute_s3_awscurl("GET", &format!("{}/{bucket}?requestPayment=", env.url), &env.access_key, &env.secret_key)
+                .expect("GetBucketRequestPayment HTTP request failed");
+        assert_eq!(parse_status(&payment_raw), Some(200), "GetBucketRequestPayment should return 200");
+        let payment_body = parse_body(&payment_raw);
         assert!(
             payment_body.contains("<Payer>BucketOwner</Payer>"),
             "GetBucketRequestPayment should return BucketOwner payer, got: {payment_body}"
         );
 
-        let website_response = signed_s3_request(
-            Method::GET,
-            &format!("{}/{bucket}?website=", env.url),
-            None,
-            None,
-            &env.access_key,
-            &env.secret_key,
-        )
-        .await
-        .expect("GetBucketWebsite HTTP request failed");
+        let website_raw = execute_s3_awscurl("GET", &format!("{}/{bucket}?website=", env.url), &env.access_key, &env.secret_key)
+            .expect("GetBucketWebsite HTTP request failed");
         assert_eq!(
-            website_response.status(),
-            404,
+            parse_status(&website_raw),
+            Some(404),
             "GetBucketWebsite should return 404 when website config is absent"
         );
-        let website_content_type = website_response
-            .headers()
-            .get(CONTENT_TYPE)
-            .expect("GetBucketWebsite response should include Content-Type")
-            .to_str()
-            .expect("GetBucketWebsite Content-Type should be valid ASCII")
-            .to_ascii_lowercase();
+        let website_content_type = parse_headers(&website_raw).to_ascii_lowercase();
         assert!(
-            website_content_type.contains("xml"),
+            website_content_type.contains("content-type:") && website_content_type.contains("xml"),
             "GetBucketWebsite error response should be XML, got content-type: {website_content_type}"
         );
-        let website_body = website_response
-            .text()
-            .await
-            .expect("Failed to read GetBucketWebsite response body");
+        let website_body = parse_body(&website_raw);
         assert!(
             website_body.contains("<Code>NoSuchWebsiteConfiguration</Code>"),
             "GetBucketWebsite should return NoSuchWebsiteConfiguration code, got: {website_body}"
         );
 
-        let delete_response = signed_s3_request(
-            Method::DELETE,
-            &format!("{}/{bucket}?website=", env.url),
-            None,
-            None,
-            &env.access_key,
-            &env.secret_key,
-        )
-        .await
-        .expect("DeleteBucketWebsite HTTP request failed");
-        assert_eq!(delete_response.status(), 204, "DeleteBucketWebsite should return 204");
+        let delete_raw =
+            execute_s3_awscurl("DELETE", &format!("{}/{bucket}?website=", env.url), &env.access_key, &env.secret_key)
+                .expect("DeleteBucketWebsite HTTP request failed");
+        assert_eq!(parse_status(&delete_raw), Some(204), "DeleteBucketWebsite should return 204");
 
         env.stop_server();
     }