Refactor code structure for improved readability and maintainability

fix(sse): separate SSE-S3 and KMS key providers
chore(deps): refresh cargo dependencies (#5132 )
2026-07-29 17:48:58 +00:00 · 2026-07-24 14:01:40 +08:00 · 2026-07-22 21:05:23 +08:00 · 2026-07-22 21:02:59 +08:00 · 2026-07-22 11:52:56 +00:00 · 2026-07-22 11:31:03 +00:00
1683 changed files with 593500 additions and 81122 deletions
@@ -0,0 +1,274 @@
+---
+name: adversarial-validation
+description: Execute the Adversarial Validation policy from the root AGENTS.md — run the seven reviewer roles (correctness, simplicity, security, concurrency/durability, compatibility, performance, test coverage) with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, or design proposal before declaring it done.
+---
+
+# Adversarial Validation Playbooks
+
+The policy — risk tiers, role list, protocol, exit criteria — lives in the
+root `AGENTS.md` under "Adversarial Validation (Default On)". Read it first;
+this skill does not restate it. This file adds the RustFS-specific probe
+playbook for each role: concrete attacks, where they apply, and the real
+shipped bug or rule that earns each probe its place.
+
+## How to run a role
+
+1. Pick the tier and the applicable roles per the root `AGENTS.md`.
+2. Run each role as an independent pass over the final diff — a parallel
+   reviewer agent where the tooling supports it, otherwise a fresh
+   sequential pass that starts from the diff and the nearest scoped
+   `AGENTS.md`, discarding the writing session's assumptions.
+3. Within a role, execute the probes whose domain the diff touches, plus any
+   attack the diff obviously invites that no probe lists — the playbook is a
+   floor, not a ceiling.
+4. Report findings (concrete failure scenario or named missing test, with
+   file:line) or the role's null report: "attacked X, Y, Z — no break
+   found". A bare pass is not a result.
+
+## Role playbooks
+
+### Correctness adversary
+
+- For any change touching error aggregation or quorum decisions, build the exact disk-error slice at the quorum boundary: N disks where successes == quorum, then flip one success to an error (quorum-1) and separately inject None/nil placeholder entries into the slice. Trace whether reduce_errs (or the new equivalent) picks the placeholder as the dominant error or lets quorum-1 pass as success. Also check heal/write paths: does a per-target failure at quorum-1 return an explicit error, or silently degrade to success?
+  - Where: crates/ecstore/src/disk/error_reduce.rs; crates/ecstore/src/set_disk/{core,ops}; crates/heal
+  - Evidence: Commit 20d61c73b 'stop reduce_errs leaking nil placeholder as dominant error' (#4551) and 47c1e730c 'make erasure heal write quorum best-effort per target' (#4545); crates/ecstore/AGENTS.md: 'Do not weaken quorum checks... Prefer explicit failure over silent data corruption or implicit success.'
+- For any change in EC read/reconstruct/streaming code, trace the mid-stream error path: the first K shards read fine, then a shard turns out bitrot-corrupt or inconsistent after N bytes have already been sent to the client. Verify the error propagates as a stream error (client sees failure), not a clean end-of-body — a silently truncated GET body is data corruption. Also re-check byte accounting: sum of per-part bytes vs object size, and partNumber-to-offset routing for the first and last part.
+  - Where: crates/ecstore/src/set_disk/read.rs (reconstruct-read, inconsistent_source_indexes handling ~line 3827); codec streaming paths in crates/ecstore/src/erasure_coding
+  - Evidence: Known live bug: EC reconstruct-read failing on inconsistent shards mid-stream silently truncates GET body → client 'unexpected EOF'; commit 15808254d 'correct codec-streaming byte accounting and partNumber routing' (#4535).
+- For any listing/pagination change, construct the exact-boundary inputs: (a) exactly max_keys/max_uploads matching entries — assert the response contains max and is_truncated=false, then max+1 entries — assert exactly max returned with is_truncated=true and a correct continuation marker; (b) a delimiter listing where folding into CommonPrefixes re-fills a full page; (c) an object 'a' coexisting with prefix dir 'a/'. Off-by-one and dropped-truncation bugs live exactly at these boundaries.
+  - Where: crates/ecstore listing/merge paths (metacache, list_objects, list_multipart_uploads); rustfs/src/storage
+  - Evidence: Three recent real bugs: fefa70b31 'stop ListMultipartUploads from returning one upload past max-uploads' (#4447), d91f4d455 'report truncation when delimiter list re-folds a full page' (#4538), 7e1f7f242 'preserve CommonPrefixes when an object and same-named prefix dir coexist' (#4563).
+- Run the diff's logic with a directory object key (trailing slash, e.g. 'pre/dir/') as input. Check which layer encodes/decodes __XLDIR__ — set_disk never sees the trailing slash, so any trailing-slash branch added below the store layer is dead code and a wrong-layer bug. For delete paths, check whether options force a nil versionId onto directory keys: the resulting miss surfaces as version-not-found, not object-not-found, so callers matching only ObjectNotFound leak ghost directory entries.
+  - Where: crates/ecstore/src/store*.rs (store layer) vs crates/ecstore/src/set_disk/*; delete option construction (del_opts) and its callers
+  - Evidence: trailing-slash branches must live at store layer; del_opts injects nil version for dir keys, PR#4220 ghost-directory cleanup never fired on the real path (rustfs#4307, backlog#798 still OPEN).
+- Feed every new binary-UUID metadata read the three degenerate values: key absent, zero-length bytes, and 16 zero bytes (nil UUID). All three must mean 'no value' — any path that produces Uuid::nil() and then acts on it (e.g. sends it as a versionId) is a finding. For tier code specifically: with remote-tier version None or "", assert the tier GET/DELETE request carries no versionId parameter at all — sending versionId="" or nil yields NoSuchVersion against unversioned tier buckets.
+  - Where: crates/filemeta/src/filemeta/version.rs; crates/ecstore/src/bucket/lifecycle/; crates/ecstore/src/services/tier/; any new consumer of crates/utils/src/http/metadata_compat.rs
+  - Evidence: AGENTS.md Cross-Cutting Domain Invariants (defensive Uuid read pattern + unversioned-tier rule); docs/operations/tier-ilm-debugging.md ('nil transition_ver_id = corrupt legacy write-back, readers must filter'); commit 726f3dc18 'accept empty remote version_id in tier recovery paths' (#4552).
+- For each match/if-let on an error or algorithm enum touched by the diff, enumerate what the wildcard/else arm swallows. Inject the variants the author didn't think of — DiskNotFound during listing, an unsupported checksum algorithm, an Err from a cleanup rename/delete — and trace whether they degrade into 'not found', a wrong-but-plausible value, or silent success. Any error path that converges with the success path without logging and propagating is a finding.
+  - Where: crates/ecstore (listing, delete/rename cleanup); crates/checksums; error-mapping layers in rustfs/src/storage
+  - Evidence: Three recent real bugs of this exact shape: e0619e355 'stop treating DiskNotFound as object not-found in listing' (#4536), afaf8c681 'reject md5 instead of silently returning crc32' (#4513), f7d2b2563 'propagate disk delete/rename failures instead of swallowing them' (#4546).
+- For any change to version ordering, index lookup, or shard/part indexing: (a) call the accessor with index == len() and len()-1 — a get_idx-style bound must reject, not panic or wrap; (b) construct two versions with identical mod-times and check the sort tie-break is total and deterministic (equal keys must not compare as both before each other); (c) for EC shard math, compute shard size for object sizes 0, 1, blockSize-1, blockSize, blockSize+1 and cross-check total reconstructed length against the object size.
+  - Where: crates/filemeta/src/filemeta/*.rs (version sort, get_idx); crates/ecstore/src/erasure_coding shard-size math
+  - Evidence: Commit 8bfb00bc0 'guard get_idx bound and fix sorts_before tie-break' (#4509) — both bug classes shipped before; ecstore AGENTS.md high-risk designation for read/write/repair correctness.
+- Exercise the zero/empty end of every new size or count parameter: zero-length object PUT then GET (body must be empty, not error), part count 0, empty Vec of disks/entries into aggregation functions, and env/config values of 0 (must clamp or reject, never divide-by-zero or 'scan nothing and report zero usage'). Anywhere the diff computes a ratio, capacity, or progress percentage, plug in 0 and the max value.
+  - Where: crates/ecstore aggregation and scanner paths; crates/object-capacity; config/env parsing in touched crates
+  - Evidence: Commits 787cc77a7 'clamp zero capacity env values to safe defaults' (#4559) and 32b1094ec 'resolve a symlinked scan root instead of silently counting zero' (#4564) — zero-as-silent-wrong-answer is a recurring repo bug class.
+- For any diff touching multipart or object commit paths, order the operations on paper and attack the failure point between them: kill the process (or return Err) after the commit rename but before cleanup, and after cleanup but before commit. Verify the earlier-failure case leaves the object readable and the later-failure case leaves no half-visible object; part meta files must never be deleted before the commit is durable.
+  - Where: crates/ecstore multipart commit/cleanup (set_disk/ops); rustfs/src/storage multipart handlers
+  - Evidence: Commit c77c5f047 'defer multipart part.N.meta cleanup until after commit' (#4548) — cleanup-before-commit ordering already caused a real data-loss window; the #4221 durability work shows fsync/ordering bugs are endemic here.
+
+Null report example: "Attacked quorum-1 error reduction, exact max-keys listing boundary, trailing-slash dir keys, nil-UUID tier versionId, and mid-stream reconstruct error propagation — no break found."
+
+### Simplicity adversary
+
+- Smaller-diff attack: rewrite the diff's change mentally (or actually, in scratch) as the minimal in-place edit and compare. Flag as findings: a helper function with exactly one caller introduced by this diff; a file rewrite where a 3-line edit inside the existing control flow suffices; reshaped control flow in init/locking/metadata/quorum paths beyond what the fix requires; new string literals duplicating existing constants (grep the token first); #[path] module inclusion. If the smaller diff achieves identical behavior, report it with the concrete replacement.
+  - Where: Any diff; extra scrutiny for crates/ecstore, crates/lock, rustfs/src/storage where 'preserve the existing control-flow shape' is an explicit rule
+  - Evidence: AGENTS.md 'Change Style for Existing Logic' (one-off helper ban, preserve control-flow shape in distributed/locking/metadata paths, no #[path]) and 'Reuse Before You Write' (constants clause); the Adversarial Validation roles list charters the simplicity adversary with exactly this attack.
+- Reuse-and-necessity attack: for each new helper the diff introduces, run `ls crates/utils/src crates/common/src` and `rg -i 'fn \w*<term>'` over those dirs plus the touched crate (snake_case signatures — a full-text single-word grep drowns, a multi-word phrase returns nothing). A reimplementation of an existing workspace utility, or of plain std/tokio behavior no wrapper refines, is a finding — but so is forced reuse with mismatched semantics (normalization such as `clean` resolving `.`/`..` against raw S3 keys, error type, backoff, durability gating). For each new defensive branch, demand the nameable trigger and flag re-validation of what a validated upstream layer on the SAME path already guarantees — excluding the Cross-Cutting Domain Invariant patterns (nil/empty/absent UUID, dual metadata keys, unversioned-tier versionId) and re-checks before destructive actions, which are load-bearing even when redundant on the happy path. For each new test, flag near-duplicates pinning the same code path AND poison-value class as an existing test — boundary companions (n==max vs max+1, absent vs empty vs nil UUID, MetaObject vs MetaDeleteMarker) are never near-duplicates; the test-coverage skeptic playbook below mandates them.
+  - Where: Any diff adding helpers, branches on decoded/peer data, or tests; helper checks against crates/utils, crates/common, and the touched crate
+  - Evidence: AGENTS.md 'Reuse Before You Write' and 'Necessary Code Only'; GHSA-f4vq-9ffr-m8m3 (normalization-asymmetry traversal — why forced reuse of normalizing helpers on raw keys is itself an attack); docs/operations/tier-ilm-debugging.md nil-versionId incident (why boundary re-checks are load-bearing).
+
+Null report example: "Rewrote the diff as an in-place edit (no smaller equivalent exists), grepped both new helpers against crates/utils, crates/common, and the touched crate (no existing equivalent; call-site semantics checked), verified the two new defensive branches name concrete corrupt-input triggers, and checked the added tests against the existing suite (each pins a distinct poison-value class) — no break found."
+
+### Security reviewer
+
+- For every admin handler in the diff, grep the exact AdminAction constant it passes to validate_admin_request and confirm it names the operation the handler actually performs. Construct the escalation: a low-privileged user whose policy grants the wrong-but-adjacent action (e.g. Export while the handler Imports, or Update while it Lists) — if the mismatched constant lets them through, that is the bug. Also confirm read-only endpoints (metrics, list, server-info, diagnostics) still call an operation-specific admin authz path and not a mere 'credentials exist' check.
+  - Where: rustfs/src/admin/handlers/**, rustfs/src/admin/router registration; check_permissions / validate_admin_request / AdminAction::* call sites
+  - Evidence: GHSA-vcwh-pff9-64cc (ImportIam checked ExportIAMAction), GHSA-mm2q-qcmx-gw4w (ListServiceAccount used UpdateServiceAccountAdminAction), GHSA-f5cv-v44x-2xgf (/admin/v3/metrics accepted any authenticated IAM user). advisory-patterns.md 'Admin authorization and route exposure'; rustfs/src/admin/AGENTS.md 'route registration, whitelist, handler authz must agree'.
+- If the diff touches service-account or IAM import/update, treat parent, claims, accessKey, secretKey, status, policy names, and groups as attacker-controlled. Construct an ImportIam/create payload where parent points at root (or another user) and prove the code writes credentials without proving caller ownership or root authority. Separately, set deny_only=true (or 'no explicit deny') on a restricted account and check it does not skip the required allow check, letting it mint an unrestricted child.
+  - Where: crates/iam/, rustfs/src/admin/handlers (service account / import IAM), rustfs/src/auth.rs
+  - Evidence: GHSA-566f-q62r-wcr8 (attacker-controlled parent/claims/accessKey/secretKey → persistent backdoor under root), GHSA-xgr5-qc6w-vcg9 (deny_only=true skipped allow checks → privilege creation). crates/iam/AGENTS.md security boundaries; advisory-patterns.md 'IAM import, service accounts'.
+- For a changed protocol-frontend handler (FTP/FTPS/SFTP/WebDAV/gateway), enumerate ALL sibling command handlers in the same driver — not just the changed one. For each, confirm it calls the per-operation IAM authorize hook mapped to the correct S3 action (RETR→GetObject, SIZE/MDTM→HeadObject, MKD→CreateBucket, bucket probe→ListBucket/HeadBucket) BEFORE touching storage. Construct a denied-authz case and prove the storage backend is never reached.
+  - Where: crates/protocols/ (FtpsDriver, SftpDriver, WebDAV), authorize_operation call sites
+  - Evidence: GHSA-3g29-xff2-92vp (FTP RETR/SIZE/MDTM authenticated but skipped IAM), GHSA-g3vq-vv42-f647 (FTPS MKD called create_bucket without s3:CreateBucket). advisory-patterns.md: 'RustFS advisories show mixed guarded and unguarded siblings in the same driver.'
+- For any secret/token/signature/password comparison in the diff, check it uses a constant-time compare (e.g. subtle/constant_time_eq), not == or early-return byte loops. Then check the failure-response paths: construct an invalid-user request and an invalid-secret request and confirm they are indistinguishable (same error, no early length short-circuit) so an attacker cannot enumerate valid users or time-side-channel the secret.
+  - Where: crates/protocols/ (FTPS/WebDAV/FormPost auth), crates/credentials/, rustfs/src/auth.rs, RPC signature verification
+  - Evidence: GHSA-3p3x-734c-h5vx (FTPS/WebDAV early-return string equality + distinguishable invalid-user vs invalid-password). Fix commits 3c3113619 (constant-time FTPS/WebDAV) and c41062f27 (constant-time FormPost signature). 3p3x was fixed by PR #4403.
+- If the diff touches internode/RPC auth secret handling, trace whether the RPC HMAC secret can fall back to a public default (e.g. 'rustfsadmin', 'rustfs rpc') or be derived deterministically from the S3 root credentials. Construct the case where RUSTFS_RPC_SECRET is unset and confirm the code fails closed rather than silently using a default or a root-derived key. Verify RPC signing keys are independent random secrets, not reused across S3-root/RPC-HMAC/STS-JWT roles.
+  - Where: crates/credentials/, crates/ecstore/src/rpc/, internode auth setup
+  - Evidence: GHSA-r5qv-rc46-hv8q (fell back to 'rustfsadmin'), GHSA-75fx/68cw (RPC secret derivable from root creds → forgeable signatures), GHSA-h956 (hard-coded 'rustfs rpc'), GHSA-m77q (STS JWT reused root secret). Fix commit 7b2055405 (fail closed when deriving RPC secret from default credentials, PR#4402).
+- If the diff touches RPC/NodeService authentication, verify the HMAC payload binds the EXACT concrete gRPC method path (not a service prefix), the HTTP method surrogate, and a fresh timestamp. Construct captured valid metadata for method A and replay it to method B within the timestamp window — if it authorizes, the signature is under-bound. Also test stale timestamp, wrong path, wrong method, wrong secret.
+  - Where: crates/ecstore/src/rpc/, verify_rpc_signature / NodeServiceServer, x-rustfs-signature handling
+  - Evidence: GHSA-c667-rgrv-99vj (signed service prefix instead of concrete method path → cross-method replay in timestamp window). advisory-patterns.md 'RPC input validation and panic safety'; Minimum Regression Test Expectations lists replay across two methods.
+- For any RPC/gRPC handler or deserialization touched, feed empty bytes, truncated MessagePack/protobuf, invalid enum discriminants, and stale timestamps. Grep the deserialization path for unwrap()/expect()/panic-prone decode and prove malformed attacker payloads return a typed error, not a panic (remote DoS). Weak internode auth makes reachability worse, so combine with the RPC-secret probe.
+  - Where: crates/ecstore/src/rpc/, any #[derive(Deserialize)] decoded from wire bytes, RPC handler bodies
+  - Evidence: GHSA-gw2x-q739-qhcr (malformed GetMetrics reached unwrap() → remote DoS). advisory-patterns.md 'Treat all RPC payload bytes as attacker-controlled.' rust-code-quality skill: unwrap abuse.
+- Take any object key, RPC disk path, or archive/tar/zip entry name introduced or handled in the diff and construct traversal payloads: '../', URL-encoded '%2e%2e%2f', absolute paths, platform separators, empty components. Trace the value through parse → authz check → final storage path and prove (a) authz and storage normalize the SAME way, and (b) the canonicalized path cannot escape the bucket/prefix root. Attack the case where authz sees the raw attacker bucket but storage cleaning crosses into a victim bucket.
+  - Where: crates/ecstore (path join/canonicalize), rustfs/src/storage/, Snowball auto-extract / normalize_extract_entry_key, rpc read_file_stream
+  - Evidence: GHSA-pq29-69jg-9mxc (read_file_stream joined untrusted paths, no canonical boundary check), GHSA-8r6f-hmq2-28rg (traversal object keys bypassed authz), GHSA-f4vq-9ffr-m8m3 (Snowball '../victim-bucket/object' authorized raw path then storage crossed boundary). Note __XLDIR__ trailing-slash encoding is store-layer only.
+- If the diff touches multipart/copy or presigned POST, verify UploadPartCopy enforces source GetObject AND destination PutObject semantics equivalent to CopyObject, including copy-source policy conditions (not just independent source-read + dest-write). Construct a cross-bucket UploadPartCopy from a bucket the caller cannot read. For presigned POST, submit an upload that violates content-length-range, key prefix, or exact content-type and prove the server rejects it.
+  - Where: rustfs/src/storage / S3 API handlers: upload_part_copy, CompleteMultipartUpload, PostObject policy enforcement
+  - Evidence: GHSA-mx42-j6wv-px98 (UploadPartCopy missed source authz → cross-bucket exfil), GHSA-wfxj-ph3v-7mjf (missed destination copy-source policy constraint), GHSA-w5fh-f8xh-5x3p (presigned POST didn't enforce signed policy conditions). advisory-patterns.md 'S3 copy, multipart, and upload policy validation'.
+- Grep the diff for debug!/trace!/info!/error! and any ?value / {:?} on structs or response bodies that can carry secret_key, session_token, JWT claims, HMAC secrets, expected signatures, access keys beyond safe identifiers, or raw credential-bearing responses. Check custom Debug impls and merged-config dumps too. Construct the error path (invalid signature, failed auth) and confirm it does not log the secret or the derived authenticator. Verify audit/notify entries redact credential request headers.
+  - Where: rustfs/src/**, crates/iam/, crates/audit/, crates/notify/, crates/targets/, RPC signature error paths
+  - Evidence: GHSA-r54g (STS creds logged at info), GHSA-8cm2 (debug logs leaked tokens/secrets/JWT claims/raw STS bodies), GHSA-333v (invalid RPC signature log included HMAC secret + expected signature). Fix commit ee6f79110 (redact credential request headers from audit/notify, backlog#963). rustfs-logging-governance skill.
+- For any struct in the diff deserialized from untrusted input (S3 XML/JSON, lifecycle rules, bucket policy, replication config, RPC payload), check for #[serde(deny_unknown_fields)]. Construct a payload with a typo'd field (e.g. 'NoncurentDays') or an extra field and prove it is rejected, not silently ignored. Flag #[serde(default)] on security-critical fields (retention days, limits, permissions) lacking explicit post-deserialize validation, and any user-controlled integer cast with `as` (i32 as u32) — feed a negative value and check it doesn't wrap to a huge positive.
+  - Where: crates/policy/ (bucket policy), crates/ecstore lifecycle/ILM config, replication config structs, crates/protocols XML parsing
+  - Evidence: AGENTS.md 'Serde Safety'; advisory-patterns.md 'Serde deserialization' (no deny_unknown_fields found repo-wide; 'NoncurentDays' typo silently accepted; i32 as u32 wrap). Fix commit 1acd47f15 (SSE crash-loop DoS + credential reserved-char bypass, backlog#806).
+- If the diff touches SSE / encryption reader-writer composition, do not trust API metadata claiming encryption. Trace the reader wrapper order (HashReader / EncryptReader / compression / warp) and confirm EncryptReader is actually in the chain that writes to disk — construct the case where a helper unwraps a nested reader and bypasses encryption, storing plaintext. Require a regression test that inspects the ACTUAL stored bytes on disk, not just read-back. Also check encrypted-object checksums are not exposed.
+  - Where: rustfs/src/storage/ecfs.rs, crates/rio/ (reader wrappers), crates/kms/, SSE-C replication
+  - Evidence: GHSA-xrrf-67jm-3c2r (SSE metadata reported encryption while composition bypassed EncryptReader, stored plaintext). Fix commits a7b9659e7 (hide encrypted object checksums, #4529), 80cc3b1fc (preserve SSE-C checksum state, #4410). advisory-patterns.md 'SSE and on-disk storage invariants'.
+- If the diff touches CORS or the console/browser/object-preview surface: confirm default CORS does not reflect an arbitrary Origin while also sending Access-Control-Allow-Credentials: true — construct a request with a spoofed Origin and check the response. For preview, confirm attacker-controlled object content is origin-isolated (not rendered in a same-origin iframe with console creds), served with nosniff/CSP, and that preview trust derives from validated content-type + sandboxing, NOT from object name/extension (.pdf, .html). Separately, if aws:SourceIp is evaluated, spoof X-Forwarded-For / X-Real-IP as a direct (non-trusted-proxy) client and confirm the socket peer IP is used instead.
+  - Where: rustfs/src/server/layer.rs (CORS), console preview/auth code, aws:SourceIp / policy condition evaluation, X-Forwarded-For handling
+  - Evidence: GHSA-x5xv-223c-8vm7 (default CORS reflected arbitrary origins with credentials), GHSA-v9fg-3cr2-277j (preview rendered attacker HTML same-origin, exposed localStorage creds), GHSA-7gcx-wg4x-q9x6 (extension-based PDF detection bypassed sandbox), GHSA-fc6g-2gcp-2qrq (aws:SourceIp trusted client XFF). advisory-patterns.md 'Browser, CORS' + 'Trusted proxy'.
+
+Null report example: "Attacked admin action-constant matching in the two changed handlers (both call validate_admin_request with the exact AdminAction), the FTPS RETR/SIZE authz parity, and the new lifecycle struct's serde surface (has deny_unknown_fields) — no break found."
+
+### Concurrency/durability reviewer
+
+- For every new or moved lock acquisition, enumerate all other code paths that take any overlapping subset of those locks and construct the concrete ABBA interleaving (thread 1 holds A wants B, thread 2 holds B wants A). If the diff acquires 2+ locks without a comment documenting acquisition order, that alone is a finding.
+  - Where: crates/ecstore/** (namespace locks, set_disk, disk registry), crates/audit/**, crates/lock/**, any Mutex/RwLock pair in a diff
+  - Evidence: crates/ecstore/AGENTS.md 'Lock Ordering' (document order; same set in different orders = deadlock); real ABBA deadlock fixed in c0d5f938f (#4421, audit registry vs stream_cancellers)
+- If the diff touches the object write/commit path or a lock guard's lifetime, construct the timeline where the distributed lock is lost (heartbeat refresh fails / expiry) after shard writes but before the xl.meta rename commit — verify the commit is fenced on guard.is_lock_lost() (set_disk/ops/object.rs:874-880) and the diff does not move the commit outside the fenced region or drop the guard early.
+  - Where: crates/ecstore/src/set_disk/ops/object.rs, ops/multipart.rs, crates/lock/**
+  - Evidence: 1e6207c08 (#4406) fence write commit on lock loss; ddf197ba5 (#4388) heartbeat lock refresh; backlog#899 fencing comment in object.rs
+- For any change to file creation or write-then-rename: write out the exact syscall order (write tmp -> fdatasync tmp -> rename -> fsync parent dir -> fsync ancestor dirs on first object under a prefix) and simulate a power cut after each step. Flag any dropped/reordered sync, and check the change honors the durability gate (RUSTFS_DRIVE_SYNC_ENABLE, strict/relaxed/none modes, per-bucket overrides) instead of hardcoding one mode. The 'skip tmp parent fsync' optimization is only sound when the file is renamed out of tmp — verify that precondition still holds.
+  - Where: crates/ecstore/src/disk/local.rs, disk/os.rs, disk/fs.rs, crates/ecstore/src/bucket/durability.rs, set_disk/core/io_primitives.rs
+  - Evidence: PR #4221 (the repo previously had no fsync anywhere); 2df315baf/c081586e7 (#4493) fsync ancestor dirs; 062a68d15 (#4387) tmp-parent-fsync skip is rename-conditional; eaff17cad (#4397) durability modes; 54872d52d (#4478) rename_data crash harness exists — extend it for the diff
+- If the diff touches quorum counting or per-disk error aggregation, construct adversarial error vectors for reduce_errs: nil/placeholder entries, DiskNotFound mixed with FileNotFound, exactly quorum-1 agreeing errors — and show which dominant error wins. Specifically attack the case where offline-disk errors get counted as 'object does not exist', flipping a read/heal decision into data loss. Also check quorum monotonicity: a retry or heal pass must never conclude with a LOWER quorum than the original write.
+  - Where: crates/ecstore/src/disk/error_reduce.rs, crates/ecstore/src/api/mod.rs, set_disk read/heal paths
+  - Evidence: 20d61c73b (#4551) reduce_errs leaked nil placeholder as dominant error; e0619e355 (#4536) DiskNotFound treated as object-not-found in listing; quorum monotonicity flagged as open follow-up to #4221 (#4221 follow-up list); crates/ecstore/AGENTS.md 'Do not weaken quorum checks'
+- For any multi-disk fan-out (delete, rename, heal write, cleanup), trace each per-disk Result: find any `let _ =`, `.ok()`, or best-effort collapse that keeps a failed disk out of the quorum math. Construct the run where exactly write_quorum-1 disks succeed and prove the op still returns success — that is the bug. Conversely, for heal writes, check one bad target cannot fail the whole heal (best-effort per target).
+  - Where: crates/ecstore/src/set_disk/ops/*.rs, disk/disk_store.rs, crates/heal/**
+  - Evidence: f7d2b2563 (#4546) disk delete/rename failures were swallowed; 47c1e730c (#4545) heal write quorum made best-effort per target; 2b063b0c4 (#4400) disk-replacement heal missed versions
+- For every new .await placed between a state mutation and its cleanup/commit (or inside select!/timeout/spawned task that can be aborted), construct the cancellation point: client disconnects and the future is dropped exactly there. Enumerate what is left behind — tmp files, incremented counters never decremented, half-written xl.meta, a held permit/waiter — and verify cleanup runs in Drop or the state is re-entrant. Background loops the diff adds must have a hard outer timeout so a wedged awaitee cannot pin them forever.
+  - Where: crates/ecstore/** write paths, crates/object-capacity/** scanners, crates/audit/**, anything using tokio::select! or spawn+abort
+  - Evidence: d608e320f io_uring cancel-safety spike (cancellation known-hard here); 5a372557e (#4533) wedged scans needed hard outer timeout; 7b87d4d13 (#4520) waiter-count leak; e44bece00 (#4497) audit start race + paused drops
+- If the diff touches metacache/list producers or cursor resume, construct the interleaving where the producer task completes (or errors) while a reader with a saved cursor comes back for the next page — verify a completed producer is tolerated (no error, no hang) and that a re-folded/full page reports truncation instead of silently ending the listing early. Also feed a corrupt/oversized length prefix into any metacache decode the diff touches.
+  - Where: crates/ecstore/src/cache_value/metacache_set.rs, crates/ecstore/src/store/list_objects.rs, crates/filemeta/**
+  - Evidence: 91a23361e (#4531) tolerate completed metacache producers; d91f4d455 (#4538) delimiter re-fold dropped truncation flag; d2c100fd3 (#4226) corrupt length-prefix guard
+- For multipart changes, construct concurrent operations on the SAME uploadId: put_object_part racing put_object_part (same part number), abort racing complete between the parts listing and the commit rename, and list-parts racing cleanup. Verify every metadata read/list/abort holds the per-uploadId lock, and that part.N.meta cleanup is deferred until AFTER the commit rename — cleanup before commit loses parts on a crash between the two.
+  - Where: crates/ecstore/src/set_disk/ops/multipart.rs, rustfs/src/storage/
+  - Evidence: 3bc8d79fe (#4329) serialize put_object_part per uploadId; 7fb95d4fc (#4428) unlocked upload metadata reads/aborts; 93ffbdb9b (#4437) unlocked part listings; c77c5f047 (#4548) part.N.meta cleanup moved after commit
+- Any cleanup/rollback logic added near a commit: verify it runs strictly AFTER the commit is durable, is best-effort (its failure must not fail an already-committed write), and is safe under retry — i.e., re-running it after a partial first attempt must never delete the newly-committed data dir or the last surviving copy.
+  - Where: crates/ecstore/src/set_disk/ops/object.rs (rename_data tail), ops/multipart.rs, disk cleanup helpers
+  - Evidence: afc7f1d6f/d908243e6 (#4386, backlog#898) post-commit old-data-dir cleanup had to be made best-effort; e7cc719c1 (#4389) speculative tmp cleanup moved off hot path
+- If the diff does read-modify-write on any persisted shared state (bucket metadata, notify/target config, queue store), construct two concurrent writers: show whether the second write silently discards the first (lost update) — RMW must be serialized or CAS-guarded. For persisted queues/replay, construct crash-mid-replay and prove entries are neither lost nor delivered twice without an idempotency key.
+  - Where: crates/notify/**, crates/targets/** (queue store, SQL backends), crates/ecstore/src/bucket/metadata_sys.rs
+  - Evidence: 2490d4ee2 (#4425) persisted config RMW lost updates; 08e44b95f (#4505) queue store crash-safety + replay lifecycle; e008cc5da (#4500) SQL backend idempotency
+- If the diff touches erasure decode/reconstruct or streaming GET, construct the failure mid-stream: shards become inconsistent (or a disk read fails) after N bytes of the body have already been sent — verify the stream surfaces an error to the client instead of ending cleanly at a truncated length. Silent truncation on a 200 response is the known failure mode.
+  - Where: crates/ecstore/src/set_disk/read.rs (reconstruct-read validation, historically ~line 3117), crates/ecstore/src/erasure/coding/decode.rs
+  - Evidence: Known open bug: EC reconstruct-read 'inconsistent shards' mid-GET silently truncates body -> client unexpected EOF; crates/ecstore/AGENTS.md 'explicit failure over silent corruption'
+
+Null report example: "Attacked lock ordering on the new disk-registry mutex pair, lock-loss fencing across the moved commit, power-cut points around the added rename, and cancellation at the two new awaits — no break found; quorum math and metacache paths untouched by this diff."
+
+### Compatibility reviewer
+
+- Grep the diff for raw 'x-rustfs-internal-' or 'x-minio-internal-' string literals used with map.insert/remove/get instead of the metadata_compat helpers. If found, construct the MinIO-written object case: a metadata map containing ONLY 'X-Minio-Internal-<suffix>' (mixed case, no RustFS key) and trace the diff's read path — does it miss the value? Then construct the removal case: does remove leave the twin key behind so a stale MinIO-key value resurrects on next read? Also check the value-type trap: get_bytes has NO case-insensitive fallback (unlike get_str), so a diff that moves a suffix from FileInfo.metadata (String) to meta_sys (Vec<u8>) silently loses mixed-case MinIO keys.
+  - Where: Any code touching FileInfo.metadata / user_defined / meta_sys: crates/ecstore/, crates/filemeta/, rustfs/src/storage/, crates/utils/src/http/metadata_compat.rs
+  - Evidence: Repo-wide invariant in AGENTS.md 'Cross-Cutting Domain Invariants' and CLAUDE.md; helpers and the asymmetry are pinned by tests test_str_lookup_accepts_minio_metadata_case and test_get_bytes_no_case_insensitive_fallback in crates/utils/src/http/metadata_compat.rs
+- For any diff reading a binary UUID from internal metadata (transitioned-versionID, tier-free-versionID, data_dir), trace the three degenerate inputs — key absent, value empty (b""), value nil UUID — through to the outgoing tier/S3 request. The bug shape to hunt: unwrap_or_default() or Uuid::from_slice(..).unwrap_or(Uuid::nil()) turning 'no value' into Uuid::nil(), which then gets serialized as ?versionId=00000000-... and the remote tier returns NoSuchVersion. The required pattern is .and_then(|v| Uuid::from_slice(&v).ok()).filter(|u| !u.is_nil()).
+  - Where: crates/ecstore/src/bucket/lifecycle/ (bucket_lifecycle_ops.rs, tier_sweeper.rs), crates/ecstore/src/services/tier/warm_backend_*.rs, crates/filemeta/src/filemeta/version.rs
+  - Evidence: Historical production bug documented in docs/operations/tier-ilm-debugging.md ('Nil-UUID versionId sent to tier'); regression tests live in crates/filemeta/src/filemeta/version.rs; follow-up fix 726f3dc18 'accept empty remote version_id in tier recovery paths' (#4552)
+- For any diff touching tier or replication GET/DELETE against a remote S3 target, enumerate BOTH directions of the versionId contract and trace each: (a) remote version None/"" means the tier bucket is unversioned — the request must carry NO versionId parameter at all (not an empty one, not nil); (b) remote version Some(v) on a versioned target — the versionId MUST be sent, especially on version-purge deletes, or the delete lands on the wrong version / creates a delete marker instead of purging. Check whether the diff collapses these cases through a single Option/String conversion that loses the distinction.
+  - Where: crates/ecstore/src/services/tier/warm_backend*.rs, crates/ecstore/src/bucket/lifecycle/tier_sweeper.rs, replication code under crates/ecstore/src/bucket/
+  - Evidence: Invariant in AGENTS.md and docs/operations/tier-ilm-debugging.md; real bug fixed by 0fad35645 'send versionId on version-purge deletes to generic S3 targets' (#4401) — the versioned direction, and #4552 — the unversioned direction
+- If the diff changes xl.meta encoding (adds/reorders msgpack header fields, touches FileMeta::marshal_msg or codec.rs encode paths), attack downgrade and cross-vendor parse: encode an object with the new code and decode it with (a) the meta_ver<=3 read path and (b) the real-MinIO fixture tests from #4377. Then check the header signature: is it recomputed over the new bytes, or copied/hardcoded? MinIO validates it; a stale or zero signature makes MinIO reject the file. Finally verify XL_META_VERSION was not silently bumped — old RustFS/MinIO nodes reject meta_ver > 3 during a rolling upgrade.
+  - Where: crates/filemeta/src/filemeta.rs (XL_HEADER_VERSION/XL_META_VERSION, lines ~46-54), crates/filemeta/src/filemeta/codec.rs (check_xl2_v1, decode_xl_headers)
+  - Evidence: Real bug 073bc9675 'compute header signature instead of hardcoding zero' (#4343); format contract pinned in docs/architecture/minio-file-format-compat.md (write meta_ver 3, read <=3, XL2 magic); parity fixtures from a91d9cefc (#4377)
+- If the diff touches xl.meta / FileInfo decode (into_fileinfo, version parsing, part arrays), construct hostile foreign input: a MinIO- or corruption-shaped msgpack with a parts-count that disagrees with the etags/sizes array lengths, missing optional fields, and a meta_ver 2 object with legacy checksum. Trace whether the new code indexes past an array, panics, or fabricates default values instead of returning a decode error. Run the pinned legacy fixtures (test_issue_2265_legacy_meta_v2_object_compatibility, test_issue_2288) plus the #4377 real-MinIO xl.meta parse tests against the diff.
+  - Where: crates/filemeta/src/ (fileinfo.rs, filemeta.rs, filemeta/codec.rs, filemeta/version.rs)
+  - Evidence: Real bug 7efacbdf9 'validate part array lengths in into_fileinfo' (#4382); legacy meta_ver 2 regression fixtures at crates/filemeta/src/filemeta.rs (~:1130-:1174) cited by docs/architecture/minio-file-format-compat.md
+- If the diff 'corrects' a formula, constant, or layout that is a byte-for-byte MinIO port (shard-size math, bitrot hash interleaving, erasure distribution, inline-data prefix), treat the correction itself as the bug: verify against legacy on-disk data before accepting. Concretely: run crates/ecstore/tests/legacy_bitrot_read_test.rs and the ECA-18 pinning tests; check whether existing objects written by old RustFS or MinIO still verify byte-for-byte. Known trap examples: bitrot_shard_file_size's bare return for non-streaming algorithms is CORRECT MinIO whole-file behavior, and the 32-byte prefix on inline data is the HighwayHash256 bitrot hash, not corruption.
+  - Where: crates/ecstore/src/erasure/coding/bitrot.rs, crates/ecstore/src/io_support/bitrot.rs, crates/filemeta/src/fileinfo.rs
+  - Evidence: f96314a1d 'pin streaming-only bitrot layout invariant (ECA-18)' (#4553) — audit explicitly decided NOT to change the formula because it breaks legacy interop; #4377 proved the inline-data prefix is the bitrot hash
+- For any diff that copies object metadata into an S3-client-visible surface (GET/HEAD response headers, notification event userMetadata, ListObjects/replication payloads, copy-object metadata directives), construct an object carrying internal keys under BOTH prefixes and in non-canonical casing ('X-Minio-Internal-Compression') and confirm every one is stripped via is_internal_key (which is case-insensitive) — not by an exact-match filter on one prefix. Leaked internal keys are an API-semantics break and an information leak.
+  - Where: rustfs/src/storage/, crates/notify/, replication and copy_object paths in crates/ecstore/
+  - Evidence: Real bug cf8929189 'strip rustfs/minio internal metadata from event userMetadata' (#4419); is_internal_key contract in crates/utils/src/http/metadata_compat.rs
+- If the diff touches crates/protos (node.proto, models.fbs) or internode RPC request/response structs, attack the rolling-upgrade interleaving: an old node sends a message without the new field to a new node, and a new node sends the extended message to an old node. Verify proto field numbers are only appended (never reused/renumbered), FlatBuffers tables are only extended at the end, and that an absent new field decodes to a safe default on the receiving side — 'safe' meaning it must not be interpreted as success/authorization (RPC errors fail closed) and must not flip a quorum decision.
+  - Where: crates/protos/ (node.proto, models.fbs, generated/), gRPC transport and dispatch in the internode layer
+  - Evidence: 6f613317f 'optimize gRPC transport' (#4337) shows the wire layer churns; security advisory 68cw fixed by PR#4402 established RPC fail-closed as a repo rule (see .agents/skills/security-advisory-lessons)
+- For S3 handler diffs, replay the request shapes real clients actually send, not just the canonical one: mc and aws-sdk differ on path normalization (root '//' ListBuckets), virtual-host vs path style, and header casing. Then attack every pagination boundary the diff touches: request exactly max-keys/max-uploads/max-parts items and verify the response returns exactly N (not N+1), sets IsTruncated correctly, and yields a NextMarker/KeyMarker that resumes without skipping or duplicating — construct the N+1st-item case explicitly.
+  - Where: rustfs/src/storage/ S3 handlers, listing paths in crates/ecstore/src/store/ and set_disk/
+  - Evidence: Real bugs 511ad31ba 'normalize root double-slash ListBuckets requests' (#4336) and fefa70b31 'stop ListMultipartUploads from returning one upload past max-uploads' (#4447); docs/architecture/s3-compatibility-matrix.md is the compat source of truth
+- If the diff changes bucket-metadata (.metadata.bin) or IAM/config parsing structs, run it against the real MinIO RELEASE.2025-07-23 fixtures: the msgpack blob uses PascalCase field names, so any serde rename, field-type change, or derive tweak silently drops MinIO-written fields instead of erroring. Verify parse_all_configs still loads all ten config types from the fixture without loss, and that drop-in migration still decrypts MinIO-encrypted IAM/server config rather than treating ciphertext as corrupt.
+  - Where: crates/ecstore/src/bucket/metadata*, crates/ecstore/src/bucket/migration.rs, IAM/config load paths in crates/iam/ and crates/config/
+  - Evidence: Fixture parity test parses_real_minio_bucket_metadata_blob_without_loss from a91d9cefc (#4377); real bug 717cdd2ab 'decrypt MinIO IAM & server config on drop-in migration' (#4358); format matrix in docs/architecture/minio-file-format-compat.md
+- If the diff adds a compatibility shim, legacy fallback, wrapper, or old-endpoint alias (grep the diff for 'legacy', 'fallback', 'compat', 'deprecated'), verify two things: (1) it carries a RUSTFS_COMPAT_TODO(<task-id>) marker with an exact removal condition and a matching entry in the register — an unmarked shim becomes permanent dead weight; (2) the fallback's default direction is safe for old data: e.g. a new decode path must fall back to the legacy decode for old objects by default, not gate legacy reads behind an opt-in flag that makes existing data unreadable after upgrade.
+  - Where: Anywhere in the diff; register at docs/architecture/compat-cleanup-register.md; recent example: allow_inplace_legacy_fallback flag in the ecstore erasure codec streaming path
+  - Evidence: docs/architecture/compat-cleanup-register.md review checklist; d232a46b4 wired legacy decode prefetch behind a default-OFF gate while keeping legacy reads working (#4542), with arity fallout fixed in 05890d6e2 (#4573)
+
+Null report example: "Attacked dual-key metadata writes/removals against MinIO-only-key objects, nil/empty transitioned-versionID paths to the tier, xl.meta encode against meta_ver<=3 decoders and the #4377 real-MinIO fixtures, and proto field-number evolution for old-node/new-node RPC — no compatibility break found."
+
+### Performance reviewer
+
+- For every `.clone()` the diff adds or moves onto a per-request/per-object path, open the cloned type and count heap fields (String, Vec, HashMap, Bytes). If >5 heap fields or it contains an EC block buffer, construct the cost: N concurrent PUTs x M objects -> N*M deep copies per second. Demand Arc-wrapping of heavy fields or pass-by-reference; also flag new `String` allocations in header/path/signature parsing where `&str`/`Cow<str>` suffices.
+  - Where: crates/ecstore/src/set_disk/**, crates/ecstore/src/store*.rs, rustfs/src/storage/, crates/filemeta/, request handlers in rustfs/src/
+  - Evidence: crates/ecstore/AGENTS.md 'Allocation Discipline in Hot Paths' (no Clone on >5-heap-field structs, Arc for large buffers, &str/Cow for temporary computations); .agents/skills/rust-code-quality/SKILL.md ranks 'unnecessary clone in hot path' as P1 must-fix
+- For every new sync_all/sync_data/fdatasync/flush/File::sync call in the diff, trace the call chain to DurabilityMode / RUSTFS_DRIVE_SYNC_ENABLE resolution (crates/ecstore/src/disk/local.rs:291 DurabilityMode, :347 resolve_durability_mode) and to per-bucket durability overrides. Construct the run where the operator sets mode=none (or legacy RUSTFS_DRIVE_SYNC_ENABLE=false) and the new fsync still fires — that is an ungated durability cost and a regression on 4KiB writes.
+  - Where: crates/ecstore/src/disk/local.rs, crates/ecstore/src/bucket/durability.rs, crates/ecstore/src/set_disk/** (rename_data/commit paths), any crate doing tokio::fs or std::fs writes
+  - Evidence: #4221 fsync work caused a measured -10% 4KiB write regression (#814 investigation), later gated; durability modes added in eaff17cad (#4397), per-bucket tier overrides in 13e48d93a (#4407); 2df315baf (#4493) shows even ancestor-dir fsyncs are routed through the gate
+- Attack blocking-work placement from both directions: (a) find new synchronous fs calls, hashing, or EC encode/decode executed directly on an async runtime thread without spawn_blocking/block_in_place — construct the stall (a slow disk blocks a worker thread and every task queued on it); (b) find new code that splits one logical disk operation into multiple spawn_blocking hops per object — each hop is a threadpool round-trip, so K hops x N objects multiplies latency. Demand the author justify the placement with the size of the work, not habit.
+  - Where: crates/ecstore/src/disk/local.rs, crates/ecstore/src/erasure_coding/, crates/ecstore/src/bitrot/, crates/rio/
+  - Evidence: 608ab14d7 (#4554, HP-12) folded metadata open+fstat+read into a single spawn_blocking because per-op hops were measurably slow; 8fc637fb1 (#4484) moved the short EC encode inline because block_in_place cost exceeded the work — direction depends on measured work size
+- For each lock acquisition the diff adds or relocates, mark the guard's live range and list every .await and disk/RPC call inside it. Construct the contention interleaving: N concurrent requests serialize on the guard while the holder waits on IO; for namespace/multipart commit locks, compute worst-case hold time (fsync + rename per disk) against the lock's timeout. Also diff the acquisition order against other paths taking the same locks (ABBA).
+  - Where: crates/ecstore/src/set_disk/** (commit/rename paths), crates/lock/, crates/audit/ registry, any RwLock/Mutex in per-request paths
+  - Evidence: crates/ecstore/AGENTS.md 'Lock Ordering'; c0d5f938f (#4421) fixed a real ABBA deadlock between registry and stream_cancellers; #4370 history: fsync-heavy serial cross-disk commits held a lock long enough to blow test timeouts (#4370)
+- Trace exactly what executes inside the PUT commit critical section (under the object write lock, between tmp write and rename_data completion) before vs after the diff. Any newly added work there — cleanup, extra stat, additional rename, O_DIRECT write, logging — is an attack target: construct the per-PUT latency delta and demand it be moved off the critical section or parallelized across disks.
+  - Where: crates/ecstore/src/set_disk/ops/*, crates/ecstore/src/disk/local.rs rename_data path
+  - Evidence: Three real optimizations removed exactly this class of regression: e7cc719c1 (#4389) moved speculative PUT-tail tmp cleanup off the hot path, 92c8c6db7 (#4411) moved O_DIRECT shard-writes off the commit critical section, 651ccac13 (#4487) parallelized tmp xl.meta write and shard fdatasync on commit
+- Find any new loop in a batch API that performs a per-item stat/read/RPC sequentially. Construct the concrete blowup: a 1000-key DeleteObjects or a full listing page -> 1000 serial round-trips added by the diff. Demand either a gate (skip when not needed) or bounded parallelism; for startup/load paths, check for accidental O(n^2) (re-scanning the full list per item).
+  - Where: crates/ecstore/src/store_delete*.rs / batch object APIs, listing/metacache paths, crates/iam/ store loading, crates/heal/
+  - Evidence: a413729b1 (#4398) had to gate and parallelize the DeleteObjects per-object stat fanout after it shipped serial; 16a91c35e (#4537) fixed O(n^2) IAM startup load by chunking — both were diff-introduced fanouts of this exact shape
+- For every buffer the diff allocates on the encode/decode/shard path, check: is it sized with with_capacity to the EC-expanded block (not the logical size, not default-grown)? Does it copy into a fresh Vec where Bytes::slice/clone (refcount) or the io-core buffer pool would avoid the copy? Does the diff read hash and data in separate passes where one pass suffices? Construct the per-block byte-copy count before vs after. If the diff touches the io-core pool, verify gauge accounting still balances.
+  - Where: crates/ecstore/src/erasure_coding/, crates/ecstore/src/bitrot/, crates/io-core/src/pool.rs, crates/rio/
+  - Evidence: 92bf55ce6 (#4396) fixed a real regression by right-sizing BytesMut encode ingest capacity to the EC-expanded block; 47bee8b31 (#4475) merged bitrot hash+data into one read pass; 7fa3d0d4b (#4534) shows pool gauge accounting is easy to drift when touching buffer reuse
+- For every logging or instrumentation statement the diff adds, classify the call site frequency: per-request, per-object, per-shard, or per-block. Anything info!/warn!/error! at per-object frequency or higher is a finding — construct the flood (one listing under client cancellation, one 10k-object heal) and count emitted lines. New metrics/timers on the data path must be feature-gated, not always-on. Run scripts/check_logging_guardrails.sh on the diff.
+  - Where: any per-request/per-object code, especially crates/ecstore listing and heal loops, rustfs/src/storage/ handlers; scripts/check_logging_guardrails.sh
+  - Evidence: .agents/skills/rustfs-logging-governance/SKILL.md (trace level for hot-path/repetitive success events); d25ddb0e1 (#4372) fixed real listing-cancellation error-log noise; hotpath instrumentation is deliberately feature-gated (3f13d098b #4394, f262fcfce #4541 HP-14)
+- Count how many times the diff's request path parses or fetches the same metadata: xl.meta/FileMeta decoded more than once per object, bucket metadata (metadata_sys) re-fetched inside a per-object loop, or the dual x-rustfs-internal/x-minio-internal key lookup re-run repeatedly on the same map. Construct the per-request parse count before vs after; a second full FileMeta decode per GET is a finding.
+  - Where: crates/filemeta/, crates/ecstore/src/set_disk/** read paths, crates/ecstore/src/bucket/metadata_sys.rs, crates/utils/src/http/metadata_compat.rs
+  - Evidence: 608ab14d7 (#4554) exists because redundant metadata-read syscall sequences per object were measurable; CLAUDE.md dual-key metadata convention makes repeated get_bytes lookups an easy hidden double-parse
+- If the diff touches PUT/GET/commit/erasure paths and claims 'no perf impact', demand numbers, not assertion: run the criterion benches (cargo bench -p ecstore — comparison_benchmark, erasure_benchmark, rename_data_meta_benchmark, single_block_non_inline_benchmark per crates/ecstore/benches/) against origin/main, and for end-to-end paths the warp A/B relative-budget gate (scripts/run_hotpath_warp_ab.sh --baseline-ref origin/main, as .github/workflows/performance-ab.yml runs it). Probe specifically at 4KiB object size — that is where the last real regression hid.
+  - Where: crates/ecstore/benches/, .github/workflows/performance-ab.yml, scripts/run_hotpath_warp_ab.sh
+  - Evidence: crates/ecstore/AGENTS.md: 'Benchmark-sensitive changes should include measurable rationale'; performance-ab.yml (215747022 #4480) is the repo's own relative-budget gate; the #4221 regression was only visible at 4KiB writes (#814 bisect)
+
+Null report example: "Attacked the new rename_data commit-section work, durability-gate routing of the added fdatasync, guard live-range across the shard-write awaits, and per-object clone count in the PUT path; ran comparison_benchmark + rename_data_meta_benchmark vs origin/main (4KiB delta within noise) — no break found."
+
+### Test-coverage skeptic
+
+- For every behavior claim in the PR description, revert that hunk (git stash / manual undo of the changed lines) and name the exact test (`cargo test -p <crate> <test_name>`) that fails. If no test fails on revert, the behavior is untested — file a finding, not a note. Especially verify the test exercises the REAL production call path, not a lookalike helper.
+  - Where: All crates; highest value in crates/ecstore, rustfs/src/storage, crates/heal
+  - Evidence: AGENTS.md exit criterion 'Every behavior change has a test that fails without it'. Real bug: PR #4220 (ghost-directory cleanup) merged with green tests but its fix never executed on the real delete path — required follow-up rustfs#4307, backlog#798 stayed OPEN. The tests exercised a path the production flow never took.
+- Read each added/modified test and confirm it asserts the real outcome (returned value, stored bytes, error variant), not merely 'call succeeded' or 'no panic'. Flag any test whose only observable is that the function returned, and any `assert!(result.is_err())` that never checks WHICH error. Then check: does the test prove the exploit/failure form is denied, or only that the intended form still works?
+  - Where: crates/e2e_test (security_boundary_test.rs pattern), and every #[cfg(test)] module in the diff
+  - Evidence: Commit dee8e4e63 (#4466) had to rewrite 277 lines of crates/e2e_test/src/security_boundary_test.rs because 'security boundary tests' passed without asserting real outcomes. .agents/skills/rust-code-quality/SKILL.md checklist: 'Every test function has at least one assert!'; .agents/skills/security-advisory-lessons/SKILL.md: 'Does the test prove the exploit form is denied, or only that the intended form still works?'
+- When the diff adds a boolean/mode parameter or config flag, find the test that fails if the flag's effect is INVERTED inside the changed function. Tests that were mechanically updated to pass `false`/default at every call site assert nothing about the new behavior. Execute the check: flip the flag's branch in the source and confirm at least one test goes red for each branch.
+  - Where: crates/ecstore/src/set_disk/ (e.g. build_codec_streaming_part_reader), any function gaining a parameter
+  - Evidence: Commit 05890d6e2 (#4573): PR #4560 added a 15th param allow_inplace_legacy_fallback; the arity tests were fixed by passing `false` everywhere — they assert Err outcomes independent of the flag, so the fallback behavior itself has no revert-detecting test at those sites.
+- Mutation spot-check on error propagation: for each newly added `?`, `return Err`, or error-mapping line, mentally replace it with `Ok(default)`/ignore and ask which test fails. The swallowed-error bug class recurs in this repo and always ships with green tests — a fix that propagates errors needs a test that injects the failure (faulty disk, failed rename, dispatch error) and asserts the caller sees Err.
+  - Where: crates/ecstore (disk delete/rename, reduce_errs), crates/audit, crates/notify, crates/targets
+  - Evidence: Three recent fixes for the same class: f7d2b2563 (#4546, disk delete/rename failures swallowed), dbc628f16 (#4424, audit dispatch failures swallowed), 20d61c73b (#4551, reduce_errs leaking nil placeholder as dominant error). All existed while tests were green.
+- Any test touching GET/read/reconstruct/stream paths must assert the FULL body content and exact length against a known value, not status-ok or first-bytes. Construct the degraded-read case (missing/inconsistent shards forcing EC reconstruction) and assert byte-for-byte equality; a mid-stream failure that truncates the body passes every test that only checks headers or the first chunk.
+  - Where: crates/ecstore/src/set_disk/read.rs and ops/, crates/rio, crates/e2e_test GET scenarios
+  - Evidence: Known live bug: EC reconstruct-read verification failure mid-GET at set_disk read path silently truncates the body → client 'unexpected EOF'; version-independent, undetected by existing suites because none assert full-body integrity under shard inconsistency.
+- For on-disk / on-wire format changes (xl.meta, .metadata.bin, bitrot framing), reject round-trip-only tests: a struct serialized and deserialized by the same code under test cannot catch format drift. Demand the test parse a REAL captured fixture from crates/filemeta/tests/fixtures, crates/ecstore/tests/fixtures, or crates/rio-v2/tests/minio_fixture_lab — or capture a new one from a single-disk MinIO instance (RELEASE.2025-07-23 procedure from #4377).
+  - Where: crates/filemeta, crates/ecstore (headers, msgpack bucket metadata), crates/rio-v2, migration code
+  - Evidence: Commit 073bc9675 (#4343): filemeta header signature was hardcoded to zero — round-trip tests passed for months. Commit a91d9cefc (#4377) established the real-MinIO fixture convention (inline/versioned/multipart xl.meta, HighwayHash256-prefixed inline bodies) precisely because synthetic fixtures proved nothing about interop.
+- Attack new concurrency tests for flakiness-by-construction: grep the added tests for `sleep(`, fixed timeouts under ~30s on lock acquisition, and use of shared global state (disk registry, lock client, GLOBAL_*). Serialized cross-disk commits exceed small lock timeouts under full-suite CI disk load. If the test shares global state or saturates IO, it must join the `ecstore-serial-flaky` nextest test-group in .config/nextest.toml (note: serial_test's #[serial] does NOT work — nextest runs each test in its own process). Require readiness polling, never fixed sleeps.
+  - Where: crates/ecstore tests, crates/e2e_test, .config/nextest.toml
+  - Evidence: Commit 2dfa3d3c3 (#4370): concurrent_resend test flaked with Lock(Timeout 5s) on CI — six legitimate serialized cross-disk commits under IO pressure needed 30s. Commit 7c701d9f2 (#4558) created the nextest test-group after bucket_delete_* raced make_bucket into InsufficientWriteQuorum. 65849740f (#4213) deflaked global-state contamination. crates/e2e_test/AGENTS.md: 'readiness checks and explicit polling over fixed sleep-based timing'.
+- If the diff writes internal object metadata, run the dual-key mutation: delete the `x-minio-internal-<suffix>` write (keeping only `x-rustfs-internal-`) and check whether any test fails. Because `get_bytes` prefers the RustFS key, every read-back test stays green while MinIO interop is silently broken — coverage must include an assertion that BOTH keys are present in the stored metadata map.
+  - Where: crates/utils/src/http/metadata_compat.rs and all its callers in crates/ecstore and rustfs/src/storage
+  - Evidence: CLAUDE.md domain convention: metadata must be written under both x-rustfs-internal- and x-minio-internal- keys for MinIO interop; get_bytes prefers the RustFS key, making the MinIO-key half of the invariant invisible to read-back tests.
+- For changed quorum/version/UUID logic, name the tests covering the specific poison values: quorum−1 disks, nil UUID, absent vs empty vs nil-serialized UUID bytes, remote-tier version_id of None/"" (unversioned tier bucket → no versionId sent), and the same metadata read on both MetaObject and MetaDeleteMarker version types. Mutation check: remove a `.filter(|u| !u.is_nil())` guard from the diff and confirm a test fails; if none does, the nil-UUID class is uncovered.
+  - Where: crates/ecstore (tier recovery, heal, quorum paths), crates/filemeta, code reading UUIDs from xl.meta metadata
+  - Evidence: Commit 726f3dc18 (#4552) fixed rejection of empty remote version_id in tier recovery. CLAUDE.md invariant: absent/empty/nil UUID all mean 'no value', not Uuid::nil(). docs/operations/tier-ilm-debugging.md: None/"" tier version means unversioned bucket. df9cbc4ed (#4427): unvalidated distribution values caused shuffle index panic — edge values reached production untested.
+- For any pagination/limit/truncation change, construct the exact-boundary test: result count == max (page exactly full), max+1, and a delimiter re-fold that lands precisely on the page boundary — assert both the item count AND the is_truncated/continuation marker. Off-by-one at the page boundary is a recurring shipped bug here.
+  - Where: crates/ecstore listing paths (list_objects, ListMultipartUploads, metacache), S3 handlers in rustfs/src/storage
+  - Evidence: Two shipped boundary bugs: fefa70b31 (#4447) ListMultipartUploads returned one upload past max-uploads; d91f4d455 (#4538) delimiter re-fold of a full page lost the truncation flag. Both survived existing tests because no test pinned n == max exactly.
+- Green `cargo test -p <crate>` on the touched crate is not a coverage verdict for the diff's test code itself: run `cargo clippy --all-targets -p <crate>` and a workspace-wide test BUILD (`cargo check --workspace --all-targets` at minimum) before accepting the tests as evidence. Test-only code that doesn't compile workspace-wide or fails clippy has repeatedly broken main and masked whether tests ran at all.
+  - Where: All crates; especially concurrent-branch merges into crates/ecstore
+  - Evidence: #4322 broke main because only cargo test ran (field_reassign_with_default is clippy-only). b06f3df6b (#4441) and 05890d6e2 (#4573): test code broke the workspace test build (E0061) on main after textually-clean merges, failing CI for every open PR.
+
+Null report example: "Attacked revert-detection for all 3 claimed behaviors (each has a named test that fails on revert), flag-inversion on the new fallback parameter (both branches covered in codec_streaming tests), full-body assertions on the changed GET path, and n==max pagination boundary — no coverage gap found."
+
+## Sources and maintenance
+
+Probes are distilled from shipped bugs in git history (commit/PR references
+above), GitHub security advisories (see the security-advisory-lessons
+skill), scoped `AGENTS.md` rules, and invariants under `docs/architecture/`
+and `docs/operations/`. Line numbers drift; when a cited location no longer
+matches, trust the invariant and re-locate the code. When a new bug class
+ships, add a probe with its evidence here rather than growing the policy
+section in `AGENTS.md`.
@@ -0,0 +1,58 @@
+---
+name: arch-checks
+description: Resolve failures from the repository's architecture guard scripts — check_layer_dependencies.sh, check_architecture_migration_rules.sh, check_unsafe_code_allowances.sh, check_logging_guardrails.sh, check_doc_paths.sh. Use when make pre-commit / pre-pr or CI fails on one of these checks.
+---
+
+# Architecture Guard Checks
+
+All five run in `make pre-commit` / `make pre-pr` and in CI. Fix the cause;
+never weaken a check to get green.
+
+## `check_layer_dependencies.sh` — layer DAG in `rustfs/src`
+
+Enforces `interface (admin, storage/ecfs, storage/s3_api) → app → infra`; no
+upward imports. Known legacy violations live in
+`scripts/layer-dependency-baseline.txt`.
+
+- **New violation**: restructure your change so the dependency points
+  downward (move the shared type/function to the lower layer).
+- **You legitimately removed a baseline entry**: run
+  `./scripts/check_layer_dependencies.sh --update-baseline` and commit the
+  shrunken baseline. Never add new entries to the baseline to make a new
+  violation pass.
+
+## `check_architecture_migration_rules.sh` — required doc sections
+
+Asserts that the core docs under `docs/architecture/` (overview,
+crate-boundaries, runtime-lifecycle, readiness-matrix,
+storage-control-data-plane, global-state-crate-split-plan,
+ecstore-module-split-plan, …) still contain specific headings and exact
+source lines. If it fails after a doc edit, you reworded or removed a
+guarded line — restore the wording or update the script deliberately in the
+same PR, with rationale.
+
+## `check_unsafe_code_allowances.sh`
+
+Every `#[allow(unsafe_code)]` needs a `SAFETY:` comment within a few lines.
+Write the actual safety argument; don't add a placeholder.
+
+## `check_logging_guardrails.sh`
+
+A fixed list of security-sensitive files (auth, IAM, KMS, admin handlers…)
+is scanned for logging violations. If you created a new sensitive file,
+consider adding it to the script's `checked_files` list.
+
+## `check_doc_paths.sh`
+
+Instruction/architecture docs (`AGENTS.md`, `CLAUDE.md`, `ARCHITECTURE.md`,
+`docs/architecture/*.md`) must not reference repo file paths that no longer
+exist. If your refactor moved code, update the docs that point at it — the
+error message lists `doc -> stale-path` pairs.
+
+## `check_no_planning_docs.sh`
+
+Planning-type documents must not be committed (see AGENTS.md "Sources of
+Truth"). The guard fails if anything is tracked under `docs/superpowers/` —
+`.gitignore` already ignores it, but `git add -f` bypasses that, so this closes
+the hole. Fix by removing the listed file(s) with `git rm`; keep the plan or
+spec in the issue tracker or a local worktree instead.
@@ -41,6 +41,10 @@ Use this skill to review code changes consistently before merge, before release,
  - hidden coupling to shared helpers/constants/features
 - If a point is uncertain, mark it as an open question instead of guessing.

+#### Rust-specific checks (apply to all Rust changes)
+
+Run the full checklist in [rust-code-quality](../rust-code-quality/SKILL.md) — the canonical Rust review checklist for the unwrap/casting/cloning/locking/recursion/error-type/serde/test rules and the reuse-and-necessity checks (duplicated helpers, defensive branches without a nameable trigger, redundant error wrapping). Do not restate those rules here; carry its P0–P3 ratings over unchanged and use this skill's output format.
+
 ### 4) Findings-first output
 - Order findings by severity:
  - P0: critical failure, security breach, or data loss risk
@@ -0,0 +1,83 @@
+---
+name: plugin-contract-guard
+description: Invariants and change procedure for the target-plugin / extension system — plugin manifests, admin plugin/extension catalog and instance APIs, secret redaction, external-plugin install policy. Use when editing crates/targets (manifest, plugin, control_plane, catalog, runtime), crates/extension-schema, or rustfs/src/admin plugin_contract.rs / plugins_*.rs / extensions.rs / target_descriptor.rs.
+---
+
+# Plugin & Extension Contract Guard
+
+The "plugin system" spans four surfaces that must stay consistent:
+
+| Surface | Location |
+|---|---|
+| Manifests & registry | `crates/targets/src/{manifest,plugin}.rs` |
+| Install/enable planning (control plane) | `crates/targets/src/control_plane.rs` |
+| Extension schemas | `crates/extension-schema/src/lib.rs`, `crates/targets/src/catalog/extension.rs` |
+| Admin API contract | `rustfs/src/admin/plugin_contract.rs`, `handlers/{plugins_catalog,plugins_instances,extensions,target_descriptor}.rs` |
+
+## Hard invariants (verify before merging)
+
+1. **Secrets have one source of truth.** Secret config keys are declared only
+   in the plugin manifest (`TargetPluginManifest.secret_fields`,
+   `crates/targets/src/manifest.rs`) and flow to admin via
+   `AdminTargetSpec.secret_fields`. Never add a hand-maintained per-service
+   secret table in a handler; if redaction misses a field, fix the manifest.
+
+2. **Redaction must round-trip.** Instance GET responses replace secret values
+   with `***redacted***` (`REDACTED_SECRET_VALUE` in `plugins_instances.rs`).
+   Instance PUT restores the stored secret when it receives that placeholder
+   back (`restore_redacted_secret_values`). Any new read or write path for
+   target config must keep both halves: redact on the way out, restore the
+   placeholder on the way in. The placeholder literal must never be persisted.
+
+3. **Fixtures never reach production responses.**
+   `example_external_webhook_plugin()` (`crates/targets/src/catalog/mod.rs`)
+   is a test/demo fixture for control-plane planning tests. Production
+   catalog/extension handlers must not include it; regression tests
+   (`plugin_catalog_never_exposes_example_or_external_fixtures`,
+   `extension_catalog_never_exposes_example_or_external_fixtures`) enforce it.
+
+4. **External plugin flow is planning-only and deny-by-default.**
+   `plan_external_target_plugin_action` returns decisions, it executes
+   nothing. `TargetPluginExternalFlowGate::default()` is fully closed and
+   `TargetPluginInstallPolicy::default().allowed_download_hosts` is empty —
+   keep it that way; tests opt in via explicit policies. Install validation
+   requires https, an allowlisted host, a full 64-hex-char sha256 digest,
+   signature and provenance URIs, and an artifact matching the host
+   `target_triple`.
+
+5. **Custom target types must not collide.** Unknown target types get an
+   interned unique `custom:<type>` plugin id (`custom_plugin_id` in
+   `manifest.rs`). Custom plugins with secrets must register via
+   `TargetPluginDescriptor::with_manifest` and declare `secret_fields`;
+   `::new` derives a manifest with no secrets.
+
+## Changing the admin JSON contract
+
+- Shapes are locked twice in `plugin_contract.rs` tests: insta snapshots
+  (`rustfs/src/admin/snapshots/`) plus literal `json!` assertions. Update
+  both deliberately; a shape change is a console-facing API change.
+- Field naming is `snake_case`, except discovery blocks
+  (`runtimeCapabilities`, `clusterSnapshot`, `extensionsCatalog`) which are
+  camelCase **by cross-endpoint convention** (same shape in `system.rs`,
+  `console.rs`, `pools.rs`). Do not "fix" that inconsistency locally.
+- Contract types deliberately duplicate `rustfs_targets` types
+  (anti-corruption layer). Add a `From` impl; do not serialize internal
+  types directly.
+
+## Handler conventions
+
+- Every new admin plugin/extension route needs authorization at the top of
+  `call` and an `include_str!` guard test asserting it (repo-wide pattern —
+  see `plugin_instance_handlers_require_admin_authorization_contract`).
+- Reads use `GetBucketTargetAction` (instances) or `ServerInfoAdminAction`
+  (catalogs); writes use `SetBucketTargetAction`.
+- Refresh persisted module switches once per request
+  (`refresh_persisted_module_switches`), then evaluate the sync
+  `module_disabled_block_reason` per domain — do not re-read the store per
+  domain or per instance.
+
+## Generic bounds
+
+Event-payload generics use the `PluginEvent` blanket trait
+(`crates/targets/src/plugin.rs`). Do not respell
+`Send + Sync + 'static + Clone + Serialize + DeserializeOwned`.
@@ -30,7 +30,9 @@ Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whe
 - Treat introducing a new hardcoded literal where a project constant already exists as a likely regression risk; require either a refactor to reuse the constant or an explicit exception explanation in the PR body.

 3. Verify readiness requirements
- Require `make pre-commit` before marking the PR ready.
+- Require `make pre-commit` before marking PRs ready when the diff changes Rust code, product behavior, CI behavior, runtime configuration, security-sensitive logic, migrations, storage, auth, networking, or other high-risk paths.
+- For documentation-only, agent-instruction-only, or local developer-tooling-only changes, allow focused verification instead of `make pre-commit` when it directly validates the changed surface.
+- For focused verification, explain why the full gate was not run and list the scope-specific commands in the PR body.
 - If `make` is unavailable, use the equivalent commands from `.config/make/`.
 - Add scope-specific verification commands when the changed area needs more than the baseline.
 - If required checks fail, stop and return `BLOCKED`.
@@ -79,7 +81,8 @@ Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whe

 ## Blocker rules

- Return `BLOCKED` if `make pre-commit` has not passed.
+- Return `BLOCKED` if a code, behavior, CI, runtime configuration, security-sensitive, migration, storage, auth, networking, or other high-risk change has not passed `make pre-commit`.
+- Return `BLOCKED` if a documentation-only, agent-instruction-only, or local developer-tooling-only change lacks focused verification for the changed surface.
 - Return `BLOCKED` if the diff contains unrelated changes that are not acknowledged.
 - Return `BLOCKED` if required template sections are missing.
 - Return `BLOCKED` if the title/body is not in English.
@@ -3,7 +3,8 @@
 - Confirm the branch is based on current `main`.
 - Confirm the diff matches the stated scope.
 - Confirm no secrets, logs, temp files, or unrelated refactors are included.
- Confirm `make pre-commit` passed, or document why it could not run.
+- Confirm `make pre-commit` passed for code, behavior, CI, runtime configuration, security-sensitive, migration, storage, auth, networking, or other high-risk changes.
+- For documentation-only, agent-instruction-only, or local developer-tooling-only changes, confirm focused verification covered the changed surface and the PR body explains why the full gate was not run.
 - Confirm extra verification commands are listed for risky changes.
 - Confirm the PR title uses Conventional Commits and stays within 72 characters.
 - Confirm the PR title does not use tool-specific prefixes such as `[codex]`.
@@ -0,0 +1,122 @@
+---
+name: rust-code-quality
+description: Enforce Rust-specific code quality rules on every code change. Use before merge to catch unwrap abuse, silent truncation, unnecessary cloning, lock ordering violations, recursion risks, and error type anti-patterns.
+---
+
+# Rust Code Quality Gate
+
+Use this skill on every Rust code change to enforce quality rules that `cargo clippy` does not catch.
+
+## Quick Start
+
+1. Identify changed `.rs` files.
+2. Run automated checks on changed files.
+3. Run manual review checklist on the diff.
+4. Report findings; block merge if P0/P1 issues exist.
+
+## Automated Checks
+
+Run these on every changed `.rs` file (excluding test modules):
+
+```bash
+# 1. unwrap/expect in production code
+rg -n '\.unwrap\(\)|\.expect\(' <changed-files> | grep -v '#\[cfg(test)\]' | grep -v 'test' | grep -v 'bench'
+
+# 2. Silent type truncation via `as` cast
+rg -n ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' <changed-files>
+
+# 3. String as error type
+rg -n 'Result<.*String>' <changed-files> | grep -v test
+
+# 4. Box<dyn Error> in public APIs
+rg -n 'Box<dyn.*Error' <changed-files> | grep -v test
+
+# 5. println/eprintln in production
+rg -n 'println!\|eprintln!' <changed-files> | grep -v test
+
+# 6. Ordering::Relaxed usage (verify each is intentional)
+rg -n 'Ordering::Relaxed' <changed-files>
+
+# 7. Default substituted for a possibly-required value (judge each: is the value optional by domain?)
+rg -n 'unwrap_or_default\(\)|unwrap_or\(' <changed-files>
+```
+
+## Manual Review Checklist
+
+For every Rust code change, verify:
+
+### Error Handling
+- [ ] No `unwrap()` or `expect()` in production code without justification comment
+- [ ] No `Result<_, String>` in public API signatures
+- [ ] No `Box<dyn Error>` in public trait/struct methods
+- [ ] `Error::source()` is overridden when inner error is stored
+- [ ] Error messages are actionable (what failed, with what input)
+
+### Type Safety
+- [ ] No silent `as` truncation (negative→unsigned, large→small)
+- [ ] `try_into()` or explicit clamping used for numeric conversions
+- [ ] No `f64 as usize` without prior clamping
+
+### Concurrency
+- [ ] Lock acquisition order is documented when multiple locks are used, and matches every other call site taking any overlapping subset (ABBA check)
+- [ ] No `tokio::sync` lock guard (read or write) held across `.await` without bounded hold time — long-lived read guards wedge writers (#4195)
+- [ ] Concurrent counters use `compare_exchange` loops, not load-then-store
+- [ ] `std::sync::Mutex` in async context is held only briefly, never across `.await`
+
+### Memory and Performance
+- [ ] No `.clone()` on structs with >5 heap-allocated fields in hot paths
+- [ ] `HashMap::with_capacity()` / `Vec::with_capacity()` used when size is known
+- [ ] Large buffers wrapped in `Arc` rather than cloned
+- [ ] Temporary string computations use `&str` or `Cow<str>` instead of `String`
+
+### Recursion Safety
+- [ ] Recursive functions have a depth limit or use iterative traversal
+- [ ] Tree/cache traversals handle corrupted/cyclic input safely
+
+### Testing
+- [ ] Every test function has at least one `assert!`
+- [ ] Tests use `.expect("context")` not bare `.unwrap()`
+- [ ] No `println!`/`eprintln!` in production code (use `tracing`)
+
+### Serde
+- [ ] Structs from untrusted input have `#[serde(deny_unknown_fields)]`
+- [ ] `#[serde(default)]` not used on security-critical fields without validation
+
+### Code Hygiene
+- [ ] No `#![allow(dead_code)]` at crate root
+- [ ] No camelCase statics or Hungarian notation
+- [ ] New string literals don't duplicate existing constants
+
+### Reuse and Necessity
+- [ ] No new helper duplicating an existing workspace utility (`crates/utils`, `crates/common`, the touched crate) or plain std/tokio behavior no wrapper refines; reused helpers match the call site's semantics (normalization, error type, backoff, durability gating)
+- [ ] No branch without a nameable concrete trigger; no re-validation of what a validated upstream layer on the same path already guarantees (Cross-Cutting Domain Invariant patterns and pre-destructive-action re-checks are load-bearing — keep them)
+- [ ] Error context attached once where actionable, not re-wrapped at every hop; no typed→generic error conversion below aggregation/quorum layers
+- [ ] No comments narrating the next line, restating a signature, or describing the change itself (invariant comments — lock ordering, `SAFETY`, unwrap justification — are not narration)
+- [ ] No near-duplicate test pinning the same code path and poison-value class as an existing test (boundary companions — n==max vs max+1, absent/empty/nil UUID — are never near-duplicates)
+
+## Severity Classification
+
+- **P0 (Block merge)**: `unwrap()` in request hot path, silent truncation on user input, lock ordering violation, recursion without depth limit
+- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method, `unwrap_or_default()` on a domain-required value (metadata, quorum, version id)
+- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`, new helper duplicating an existing workspace utility, defensive branch with no nameable trigger (corrupt or stale persisted/peer data is always a nameable trigger for boundary-crossing values), near-duplicate test, redundant error re-wrapping
+- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`, narrating comment
+
+## Output Template
+
+```
+## Rust Code Quality Report
+
+### Automated Scan
+- unwrap/expect in production: N found
+- as casts: N found
+- String errors: N found
+- println/eprintln: N found
+
+### Findings
+- [P1] `path:line` — description
+  - Fix: ...
+  - Validation: ...
+
+### Verdict
+PASS / BLOCKED (list blocking findings)
+```
@@ -0,0 +1,52 @@
+# Rust Code Quality Checklist
+
+Use this as a quick pre-merge checklist for every Rust code change.
+
+## Critical (P0 — block merge)
+
+| Check | Command |
+|-------|---------|
+| No `unwrap()` in request/storage hot path | `rg '\.unwrap\(\)' <files> \| grep -v test` |
+| No `as` truncation on user input | `rg ' as (u32\|usize\|i32)' <files>` |
+| Lock order consistent across call sites | Manual: trace all lock acquisitions |
+| Recursive functions have depth limit | Manual: check for `max_depth` or iterative pattern |
+| No `panic!`/`unwrap_or_else(panic!)` in production | `rg 'panic!\|unwrap_or_else.*panic' <files> \| grep -v test` |
+
+## High (P1 — must fix)
+
+| Check | Command |
+|-------|---------|
+| No `Result<_, String>` in public API | `rg 'Result<.*String>' <files> \| grep -v test` |
+| No `Box<dyn Error>` in public trait | `rg 'Box<dyn.*Error' <files> \| grep -v test` |
+| No unnecessary `.clone()` in hot path | Manual: check loops and per-request paths |
+| `Error::source()` implemented when inner error stored | Manual: check `impl Error` |
+| No `eprintln!`/`println!` in production | `rg 'println!\|eprintln!' <files> \| grep -v test` |
+
+## Medium (P2 — should fix)
+
+| Check | Command |
+|-------|---------|
+| Tests have assertions | Manual: check for `assert` in test functions |
+| `HashMap`/`Vec` use `with_capacity` when size known | Manual: check `::new()` in loops |
+| No `#![allow(dead_code)]` at crate root | `rg 'allow.dead_code' <files> \| grep 'lib.rs'` |
+| Serde structs from untrusted input have `deny_unknown_fields` | Manual: check `#[derive(Deserialize)]` |
+
+## Low (P3 — nice to fix)
+
+| Check | Command |
+|-------|---------|
+| No camelCase statics | `rg 'static ref [a-z]' <files>` |
+| `Arc::ptr_eq` instead of `as_ptr + ptr::eq` | `rg 'as_ptr\|ptr::eq' <files>` |
+| Public functions have doc comments | `rg 'pub fn' <files> \| grep -v '///'` |
+
+## Quick One-Liner
+
+```bash
+# Run all automated checks on changed files
+CHANGED=$(git diff --name-only HEAD~1 -- '*.rs' | grep -v test | grep -v bench)
+echo "=== unwrap/expect ===" && rg -c '\.unwrap\(\)|\.expect\(' $CHANGED 2>/dev/null
+echo "=== as casts ===" && rg -c ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' $CHANGED 2>/dev/null
+echo "=== String errors ===" && rg -c 'Result<.*String>' $CHANGED 2>/dev/null
+echo "=== println ===" && rg -c 'println!|eprintln!' $CHANGED 2>/dev/null
+echo "=== Ordering::Relaxed ===" && rg -c 'Ordering::Relaxed' $CHANGED 2>/dev/null
+```
@@ -0,0 +1,107 @@
+---
+name: rustfs-logging-governance
+description: Standardize and review RustFS logging with structured `tracing` events, lower noise on hot paths, preserve security-sensitive diagnostics, and extend guardrails to prevent legacy logging patterns from returning. Use when editing or reviewing RustFS logs, startup/config diagnostics, cloud metadata logs, request validation logs, or `scripts/check_logging_guardrails.sh`.
+---
+
+# RustFS Logging Governance
+
+Use this skill when RustFS logging needs to be added, cleaned up, reviewed, or protected against regressions.
+
+## Quick Start
+
+1. Identify the files whose logs are changing.
+2. Scan current `tracing` or `log` macros before editing.
+3. Convert sentence-style logs to short event-style logs.
+4. Demote hot-path success logs unless operators truly need them at `info`.
+5. Preserve failure, fallback, and security-relevant diagnostics.
+6. Update `scripts/check_logging_guardrails.sh` when a broad cleanup removes a legacy pattern class.
+7. Validate with formatting, targeted checks/tests, and the logging guardrail script.
+
+## Core Workflow
+
+### 1. Scope the logging surface
+
+- Read the changed module in full before touching log lines.
+- Classify the log site:
+  - lifecycle/startup
+  - request or validation path
+  - background loop or hot path
+  - fallback/degraded behavior
+  - cloud metadata or external fetch path
+  - metrics/config summary
+- Do not rewrite business logic to make logging easier.
+
+### 2. Use the RustFS event shape
+
+- Prefer fields first, message second.
+- Use short labels, not prose paragraphs.
+- Default field shape:
+  - `event`
+  - `component`
+  - `subsystem`
+  - `state` or `result`
+  - key context fields
+- Reuse stable field names and avoid inventing near-duplicates.
+
+See `references/logging-governance.md` for the event model, level policy, and anti-pattern list.
+
+### 3. Choose the right level
+
+- `error`: operation failure that affects behavior or security guarantees.
+- `warn`: degraded path, fallback, suspicious input, or operator-actionable misconfiguration.
+- `info`: low-frequency lifecycle or mode selection.
+- `debug`: targeted diagnostics and low-volume detail.
+- `trace`: hot-path and repetitive success-path events.
+
+When in doubt, lower the verbosity of normal success paths and keep structured detail in fields.
+
+### 4. Preserve security and privacy boundaries
+
+- Do not log secrets, tokens, auth headers, raw credential payloads, or merged config dumps.
+- Avoid logging raw forwarded headers or full trusted network inventories above `debug`.
+- Keep warning/error logs useful without echoing attacker-controlled payloads unnecessarily.
+
+### 5. Keep summaries aggregated
+
+- Replace multi-line startup banners or checklist logs with one structured event.
+- If metrics already express a concept, avoid duplicating it with many `info!` lines.
+- Prefer counts, modes, and sources over inventories unless debug detail is truly needed.
+
+### 6. Update guardrails when needed
+
+- Broad logging cleanup should usually extend `scripts/check_logging_guardrails.sh`.
+- Add forbidden patterns only for styles the repo has intentionally retired:
+  - sentence-style lifecycle logs
+  - noisy hot-path `info!`
+  - checklist-style summary logs
+  - legacy fallback wording that has been replaced by structured fields
+- Keep guardrails concrete and grep-friendly.
+
+### 7. Validate manually
+
+Use the smallest relevant set:
+
+```bash
+cargo fmt --all --check
+./scripts/check_logging_guardrails.sh
+cargo check -p <affected-crate>
+cargo test -p <affected-crate>
+```
+
+For broader Rust changes, add:
+
+```bash
+./scripts/check_unsafe_code_allowances.sh
+./scripts/check_architecture_migration_rules.sh
+cargo clippy -p <affected-crates> --all-targets -- -D warnings
+```
+
+## RustFS-Specific Notes
+
+- The durable RustFS logging direction is `event + component + subsystem + state/result + key context fields`.
+- `crates/concurrency` and `crates/trusted-proxies` are examples of this style for lifecycle, fallback, and cloud metadata logs.
+- `scripts/check_logging_guardrails.sh` is the enforcement point for preventing removed log styles from returning.
+
+## References
+
+- Read `references/logging-governance.md` when you need the detailed field set, anti-pattern examples, or guardrail update checklist.
@@ -0,0 +1,4 @@
+interface:
+  display_name: "RustFS Logging Governance"
+  short_description: "Standardize RustFS logs with structured events and guardrails."
+  default_prompt: "Use $rustfs-logging-governance to standardize or review RustFS logging, reduce noise, and update guardrails."
@@ -0,0 +1,285 @@
+# RustFS Logging Governance Reference
+
+## Workspace Scope Map
+
+Use `Cargo.toml` `[workspace].members` as the source of truth for crate membership. When doing a broad logging sweep, classify crates by operational role so logs stay consistent within each role.
+
+### Core Server And Request Handling
+
+- `rustfs`
+  - Role: top-level server, startup, auth, admin wiring, S3 request handling.
+  - Logging focus: startup lifecycle, config summaries, authn/authz failures, protocol entrypoints, degraded subsystems.
+- `crates/protocols`
+  - Role: protocol integrations such as FTP, SFTP, WebDAV, and related server-side protocol layers.
+  - Logging focus: listener lifecycle, per-protocol enablement/disablement, request bridge failures.
+- `crates/madmin`
+  - Role: admin API contracts and management interfaces.
+  - Logging focus: admin action boundaries, validation failures, compatibility warnings.
+- `crates/trusted-proxies`
+  - Role: forwarded IP trust, proxy chain validation, cloud metadata sources.
+  - Logging focus: direct/trusted/fallback decisions, degraded metadata fetches, aggregated config summaries.
+- `crates/keystone`
+  - Role: Keystone auth integration.
+  - Logging focus: integration enablement, upstream auth failures, config safety without credential leakage.
+
+### Storage, Healing, And Data Plane
+
+- `crates/ecstore`
+  - Role: erasure-coded storage implementation and peer/store initialization.
+  - Logging focus: disk/peer lifecycle, storage fallback, object I/O failures, avoid per-object noise.
+- `crates/heal`
+  - Role: healing orchestration and repair workflows.
+  - Logging focus: scheduler lifecycle, repair decisions, backlog or skipped work summaries, avoid repetitive task spam at `info`.
+- `crates/scanner`
+  - Role: data integrity scanning and health monitoring.
+  - Logging focus: scan lifecycle, compaction/deep-heal transitions, lag/backlog, noisy folder iteration should stay at `debug/trace`.
+- `crates/object-capacity`
+  - Role: capacity scan and refresh core.
+  - Logging focus: refresh lifecycle, degraded capacity sources, aggregate stats rather than per-object chatter.
+- `crates/filemeta`
+  - Role: file metadata parsing and helpers.
+  - Logging focus: parse failures, schema/format mismatch, avoid dumping raw metadata payloads.
+- `crates/storage-api`
+  - Role: storage contracts and shared data plane interfaces.
+  - Logging focus: contract mismatch and boundary diagnostics, usually low-volume.
+- `crates/checksums`
+  - Role: checksum helpers and validation.
+  - Logging focus: integrity failures and compatibility mismatches, not per-chunk success logs.
+- `crates/zip`
+  - Role: ZIP handling and compression helpers.
+  - Logging focus: parse/extract failures, archive path safety issues, avoid verbose file-by-file success logs.
+
+### Security, Identity, And Policy
+
+- `crates/iam`
+  - Role: identity and access management.
+  - Logging focus: authz decision boundaries, imported payload safety, do not leak principals, secrets, or claims.
+- `crates/policy`
+  - Role: policy modeling and evaluation.
+  - Logging focus: deny/allow decision context, parser/validation failures, no raw secret-bearing request dumps.
+- `crates/credentials`
+  - Role: credential handling.
+  - Logging focus: never log secrets or tokens; only safe identifiers and redacted states.
+- `crates/kms`
+  - Role: key management service integration.
+  - Logging focus: init/health/fallback, key-source availability, never log key material.
+- `crates/crypto`
+  - Role: cryptographic helpers and security primitives.
+  - Logging focus: only algorithm or mode state, not plaintext, ciphertext, or secret-derived material.
+- `crates/security-governance`
+  - Role: security governance contracts.
+  - Logging focus: policy/state transitions and enforcement diagnostics.
+- `crates/signer`
+  - Role: request signing helpers.
+  - Logging focus: signature validation failures without expected-signature leakage.
+
+### Notifications, Audit, And Targets
+
+- `crates/notify`
+  - Role: notification dispatch, runtime facade, notifier implementations.
+  - Logging focus: target lifecycle, dispatch summaries, stream lag/backpressure, avoid per-event success spam.
+- `crates/audit`
+  - Role: audit target fan-out and audit pipeline management.
+  - Logging focus: pipeline lifecycle, target availability, batch dispatch summaries, avoid noisy "started successfully" prose.
+- `crates/targets`
+  - Role: target-specific configuration and utilities used by fan-out style systems.
+  - Logging focus: target selection, config validation, per-target degraded state.
+- `crates/s3-types`
+  - Role: S3 event and type definitions.
+  - Logging focus: usually minimal; keep logging at integration boundaries rather than low-level type crates.
+- `crates/s3-ops`
+  - Role: S3 operation definitions and mapping.
+  - Logging focus: mapping/contract failures, unsupported combinations, not normal-path request spam.
+
+### Concurrency, Locking, And Runtime Foundations
+
+- `crates/concurrency`
+  - Role: timeout, locking, backpressure, and I/O scheduling facade.
+  - Logging focus: lifecycle transitions and degraded states, not high-frequency worker/permit churn at `info`.
+- `crates/lock`
+  - Role: distributed locking implementation.
+  - Logging focus: lock lifecycle, contention anomalies, lock ordering or timeout diagnostics.
+- `crates/tls-runtime`
+  - Role: shared TLS runtime foundation.
+  - Logging focus: certificate lifecycle, reload/fallback, validation failures without sensitive dumps.
+- `crates/obs`
+  - Role: observability helpers.
+  - Logging focus: this crate shapes other crates' telemetry conventions; avoid recursive or redundant summaries.
+- `crates/io-core`
+  - Role: zero-copy I/O core primitives.
+  - Logging focus: keep very sparse; prefer metrics unless failures are actionable.
+- `crates/io-metrics`
+  - Role: I/O metrics collection.
+  - Logging focus: typically minimal; metrics should carry the hot-path signal.
+- `crates/rio`
+  - Role: Rust I/O utility layer.
+  - Logging focus: compatibility or runtime boundary failures, not fast-path internals.
+- `crates/rio-v2`
+  - Role: next-generation I/O compatibility layer.
+  - Logging focus: migration/feature-mode differences and degraded fallback between I/O paths.
+- `crates/utils`
+  - Role: shared helpers.
+  - Logging focus: usually avoid direct logging in generic helpers unless the helper is itself an operational boundary.
+- `crates/common`
+  - Role: shared data structures and helpers.
+  - Logging focus: same principle as `utils`; prefer callers to log context-rich events.
+- `crates/config`
+  - Role: configuration management.
+  - Logging focus: config source, fallback, validation, and summary aggregation; avoid dumping merged configs.
+- `crates/data-usage`
+  - Role: shared data usage models and algorithms.
+  - Logging focus: refresh lifecycle, summary stats, and degraded reads.
+
+### Schema, Contracts, And API Support
+
+- `crates/protos`
+  - Role: protobuf definitions.
+  - Logging focus: usually none inside the crate; emit logs at decode/use boundaries.
+- `crates/extension-schema`
+  - Role: extension schema contracts.
+  - Logging focus: schema validation and compatibility mismatches.
+- `crates/s3select-api`
+  - Role: S3 Select API interfaces.
+  - Logging focus: request validation and unsupported feature boundaries.
+- `crates/s3select-query`
+  - Role: S3 Select query engine.
+  - Logging focus: query parse/planning/execution failures, avoid row-level spam.
+- `crates/protocols`
+  - Role: non-S3 protocol support.
+  - Logging focus: see core server section; keep per-request verbosity below `info`.
+
+### Testing And Non-Production Crates
+
+- `crates/e2e_test`
+  - Role: end-to-end tests.
+  - Logging focus: test clarity matters more than production governance, but avoid copying test-only logging style into production crates.
+
+## Current Guardrail Coverage Map
+
+`scripts/check_logging_guardrails.sh` currently enforces retired patterns in these high-signal areas:
+
+- `rustfs/src/main.rs`
+- `rustfs/src/startup_iam.rs`
+- `rustfs/src/auth.rs`
+- `rustfs/src/protocols/client.rs`
+- `crates/audit/src/pipeline.rs`
+- `crates/audit/src/system.rs`
+- `crates/audit/src/global.rs`
+- `crates/notify/src/config_manager.rs`
+- `crates/notify/src/runtime_facade.rs`
+- `crates/notify/src/notifier.rs`
+- `crates/ecstore/src/store/peer.rs`
+- `crates/ecstore/src/store/init.rs`
+- `crates/ecstore/src/tier/tier.rs`
+- `crates/concurrency/src/workers.rs`
+- `crates/concurrency/src/manager.rs`
+- `crates/concurrency/src/lock.rs`
+- `crates/concurrency/src/deadlock.rs`
+- `crates/trusted-proxies/src/global.rs`
+- `crates/trusted-proxies/src/config/loader.rs`
+- `crates/trusted-proxies/src/proxy/metrics.rs`
+- `crates/trusted-proxies/src/proxy/validator.rs`
+- `crates/trusted-proxies/src/proxy/chain.rs`
+- `crates/trusted-proxies/src/middleware/service.rs`
+- `crates/trusted-proxies/src/cloud/detector.rs`
+- `crates/trusted-proxies/src/cloud/ranges.rs`
+- `crates/trusted-proxies/src/cloud/metadata/aws.rs`
+- `crates/trusted-proxies/src/cloud/metadata/azure.rs`
+- `crates/trusted-proxies/src/cloud/metadata/gcp.rs`
+
+When expanding coverage, prefer crates with:
+
+- repeated sentence-style lifecycle logs
+- high-frequency success-path `info!`
+- startup/config checklist banners
+- security-sensitive fallback wording
+- external fetch/retry/fallback flows
+
+That typically means the next broad candidates are `rustfs`, `crates/notify`, `crates/audit`, `crates/targets`, `crates/heal`, and `crates/scanner`.
+
+## Event Model
+
+Prefer this structure when the fields are available:
+
+- `event`
+- `component`
+- `subsystem`
+- `state` or `result`
+- stable context fields such as:
+  - `enabled`
+  - `implementation`
+  - `validation_mode`
+  - `peer_ip`
+  - `client_ip`
+  - `proxy_hops`
+  - `duration_ms`
+  - `fallback`
+  - `reason`
+  - `range_count`
+  - `hold_time_ms`
+  - `available_slots`
+  - `total_slots`
+  - `permits_in_use`
+
+## Level Policy
+
+- `error`: the operation fails and callers or security guarantees are affected.
+- `warn`: a degraded path, fallback, suspicious request, or operator-actionable config issue occurs.
+- `info`: a low-frequency lifecycle or mode transition occurs.
+- `debug`: useful diagnostics exist but normal operators do not need them all the time.
+- `trace`: hot-path and repetitive success-path details occur.
+
+## Preferred Patterns
+
+- Use a short message label:
+  - `"trusted proxy validation failed"`
+  - `"concurrency manager state changed"`
+  - `"trusted proxy cloud metadata loaded"`
+- Put key meaning into fields, not only the message text.
+- Aggregate config or metrics summaries into one log event.
+
+## Retired Patterns
+
+These should usually be removed or replaced:
+
+- Sentence-style lifecycle logs:
+  - `info!("Concurrency manager stopped")`
+  - `info!("Trusted Proxies module initialized")`
+- Checklist or banner logs:
+  - `info!("=== Application Configuration ===")`
+  - `info!("Available metrics:")`
+- Hot-path noise:
+  - `info!("worker take, {}", *available)`
+  - `debug!("Proxy validation successful in {:?}", duration)`
+- Legacy fallback prose:
+  - `"Request from private network but not trusted: ..."`
+  - `"Cloud metadata fetching is disabled"`
+
+## Guardrail Update Checklist
+
+When extending `scripts/check_logging_guardrails.sh`:
+
+1. Add the touched files to `checked_files`.
+2. Add only legacy patterns that have been intentionally retired.
+3. Keep patterns literal and grep-friendly.
+4. Run the guardrail script after changes.
+5. Avoid adding patterns for logs that are still valid elsewhere in the repo.
+
+## Validation Checklist
+
+For logging-only changes:
+
+```bash
+cargo fmt --all --check
+./scripts/check_logging_guardrails.sh
+cargo check -p <affected-crate>
+cargo test -p <affected-crate>
+```
+
+For broader Rust changes:
+
+```bash
+./scripts/check_unsafe_code_allowances.sh
+./scripts/check_architecture_migration_rules.sh
+cargo clippy -p <affected-crates> --all-targets -- -D warnings
+```
@@ -0,0 +1,169 @@
+---
+name: rustfs-release-publish
+description: "End-to-end RustFS release pipeline: bump version files on main directly to the final target version, cut a preview tag on that commit, verify the CI build and release artifacts, run the downloaded binary locally and exercise the console, validate the server with the latest rc client, then publish the final tag on the SAME validated commit — never a new bump commit, never latest main. Use whenever the user wants to release/publish a RustFS version (发版/发布)."
+---
+# RustFS Release Publish (preview-validated pipeline)
+
+This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
+
+Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and prerelease classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
+
+Pipeline shape:
+
+```
+bump version files to <target> (final version, ONE commit) -> merge
+  -> tag <preview-tag> at that commit -> CI green
+  -> verify release artifacts -> run binary locally + console checks
+  -> validate with latest rc client
+  -> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
+```
+
+On validation failure: fix lands on main via normal PR (version files are already at `<target>`, no new bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2.
+
+## Required inputs
+
+- Final target version, for example `1.0.0-beta.10`.
+- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` — and for stable targets `git tag -l '<target>-rc.*'` — after `git fetch --tags`).
+
+If the target version is missing or ambiguous, stop and ask before doing anything (see the semver gate below).
+
+## Semver gate — confirm the target version before touching anything
+
+Versions follow [SemVer 2.0.0](https://semver.org/). Precedence reminder:
+
+```
+1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0 < 1.0.1 < 1.1.0 < 2.0.0
+```
+
+Numeric prerelease identifiers compare numerically (`beta.9 < beta.10`), not lexically — see [semver.org spec item 11](https://semver.org/#spec-item-11). Preview tags are internal validation tags layered on top of the target's prerelease channel — they are never themselves a deliverable version and never appear in version files.
+
+Rules:
+
+- A request like "发个版" / "release the next version" without an exact version string is ALWAYS ambiguous. Derive the current latest tag (`git tag --sort=-v:refname | head`), then ask the user to choose via AskUserQuestion with concrete candidates, e.g. from `1.0.0-beta.10`: next prerelease `1.0.0-beta.11`, promote to `1.0.0-rc.1`, promote to stable `1.0.0`. Never guess between these — they have very different meanings (channel promotion vs. iteration) and different CI classification consequences.
+- After a stable `X.Y.Z` exists, the next version must state which component bumps: patch `X.Y.(Z+1)` for fixes only, minor `X.(Y+1).0` for backward-compatible features, major `(X+1).0.0` for breaking changes. If the user names a bump type but not a number, compute it from the latest stable tag and echo the exact resulting version back for confirmation.
+- Echo the final confirmed version string verbatim in your first status report; every later phase must use exactly that string. If at any point the user's wording and the confirmed version diverge, stop and re-confirm.
+
+## Preview tag naming
+
+- Prerelease target (contains `alpha`/`beta`/`rc`): preview tag is `<target>-preview.N`, e.g. `1.0.0-beta.10-preview.3`. It contains `beta`, so `build.yml`'s substring-based classification marks it prerelease — safe.
+- **Stable** target (e.g. `1.1.0`): NEVER tag `1.1.0-preview.N` — `build.yml` marks a tag prerelease only if its name contains `alpha`, `beta`, or `rc`, so `1.1.0-preview.N` would be treated as a stable release and overwrite `latest.json` as stable. Use `1.1.0-rc.N` as the preview tag instead.
+
+## Hard rules
+
+- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
+- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
+- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
+- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
+- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
+- User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
+
+## Phase 0 — Preflight
+
+- `git status --short` clean; `git fetch origin main --tags`.
+- `gh auth status` works; confirm you can view `gh release list -L 3`.
+- Confirm the exact final target version with the user if not explicit.
+
+## Phase 1 — Version bump to the final target (once)
+
+- If main's version files already read `<target>` (e.g. this is a restart after a failed preview), verify with `rg -n "<target>" Cargo.toml rustfs.spec helm/rustfs/Chart.yaml` and skip to Phase 2.
+- Otherwise invoke the `rustfs-release-version-bump` skill with the final `<target>` (NOT a preview version), full GitHub flow (commit/push/PR).
+- Get the PR merged into main. Record the resulting main commit:
+
+```bash
+git fetch origin main
+PREVIEW_HASH=$(git rev-parse origin/main)   # must contain the bump PR
+```
+
+`PREVIEW_HASH` is the single source of truth for the rest of the pipeline — report it to the user and reuse it verbatim in Phases 2 and 6. Both the preview tag and the final tag will point at it.
+
+## Phase 2 — Publish the preview tag
+
+```bash
+git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
+git push origin "<preview-tag>"
+```
+
+Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
+
+On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
+
+## Phase 3 — CI and artifact verification
+
+- Watch the tag build: `gh run list --workflow build.yml --limit 5` then `gh run watch <run-id>`. Every matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64) plus the release and latest.json jobs.
+- Verify the GitHub release: `gh release view "<preview-tag>" --json isPrerelease,assets`
+  - `isPrerelease` must be `true`.
+  - Assets must include all 6 platform zips in both versioned (`rustfs-<platform>-v<tag>.zip`) and `-latest` forms, plus `SHA256SUMS`, `SHA512SUMS`, `rustfs-<tag>.sbom.cdx.json`, `rustfs-<tag>.provenance.json`.
+- Verify the chained Docker run succeeded: `gh run list --workflow docker.yml --limit 3`.
+- Checksum spot-check for the platform you will run locally: download the zip and `SHA256SUMS`, verify with `shasum -a 256 -c` (grep to one line).
+
+## Phase 4 — Run the artifact locally, verify the console
+
+Work inside the session scratchpad directory; never leave stray data dirs.
+
+```bash
+gh release download "<preview-tag>" -p "rustfs-macos-aarch64-v<preview-tag>.zip" -D "$SCRATCH"
+cd "$SCRATCH" && unzip -o rustfs-*.zip
+./rustfs --version        # must report the PREVIEW TAG (build::TAG), not the Cargo.toml version, plus expected short SHA
+mkdir -p data
+RUSTFS_ACCESS_KEY=rustfsadmin RUSTFS_SECRET_KEY=rustfsadmin ./rustfs ./data
+```
+
+Defaults: S3 endpoint `:9000`, embedded console `:9001`.
+
+Checks (all must pass):
+
+- `./rustfs --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
+- `curl -fsS http://localhost:9000/health/ready` returns ready.
+- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
+- Open `http://localhost:9001` in the browser: login with `rustfsadmin`/`rustfsadmin`; dashboard renders without JS console errors; create a bucket, upload a file, download it back (byte-identical), delete the object and bucket. Keep the server running for Phase 5.
+
+## Phase 5 — Validate with the latest rc client
+
+`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
+
+- Ensure the latest release is installed: compare `rc --version` against `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; update via `brew upgrade rustfs/tap/rc` (or download the release binary).
+- Point it at the preview server and run the command matrix, recording PASS/FAIL per command:
+
+```bash
+rc alias set preview http://localhost:9000 rustfsadmin rustfsadmin
+rc ls preview/
+rc mb preview/rel-check
+rc cp <local-file> preview/rel-check/
+rc stat preview/rel-check/<file>
+rc cat preview/rel-check/<file>          # matches source
+rc cp preview/rel-check/<file> ./out && cmp <local-file> ./out
+rc cp -r <local-dir>/ preview/rel-check/dir/
+rc find preview/rel-check --name "*"
+rc share download preview/rel-check/<file> --expire 1h   # presigned URL fetchable via curl
+rc rm preview/rel-check/<file> && rc rm -r --force preview/rel-check/dir
+rc rb preview/rel-check
+rc admin user list preview/
+rc admin user add preview/ relcheckuser relchecksecret12
+rc admin user remove preview/ relcheckuser
+rc alias remove preview
+```
+
+- Any FAIL blocks the release. Afterwards stop the server and delete the scratch data directory.
+
+## Phase 6 — Publish the final tag on the validated commit
+
+No second version bump, no release branch. The final tag goes on the exact commit the preview validated:
+
+```bash
+git fetch origin --tags
+git rev-parse "<preview-tag>^{commit}"          # must equal PREVIEW_HASH — abort if not
+git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
+git push origin "<target>"
+```
+
+- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
+- Re-run the Phase 3 verification against the final tag: all matrix jobs green; `gh release view "<target>"` shows the full asset set; for a prerelease target `isPrerelease` is `true`, for a stable target it must be `false` and `latest.json` must be updated.
+- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
+
+## Output contract
+
+Always report:
+
+- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
+- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: CI run URLs, release URLs, console check results, the rc command matrix.
+- Any deviation from this pipeline and why the user approved it.
@@ -0,0 +1,126 @@
+---
+name: rustfs-release-version-bump
+description: "Publish a RustFS alpha/beta/stable release with an auditable flow: confirm target version and scope, update workspace and release assets (including strict rustfs.spec changelog identity/date/version format), run required verification, and finish with commit, push, and GitHub PR creation."
+---
+# RustFS Release Version Bump
+
+Use this skill to publish a RustFS release (alpha, beta, or stable) with a minimal, auditable diff and a complete ship flow (`edit -> verify -> commit -> push -> PR`).
+
+Validated baseline: release pattern used in PR `#2957`.
+
+## Required inputs
+
+- Exact target version, for example `1.0.0-beta.4`.
+- Delivery scope:
+- Local only (`edit/verify`).
+- Local + git (`commit/push`).
+- Full GitHub flow (`commit/push/PR`).
+
+If target version is missing or ambiguous, stop and ask before editing.
+
+Reject any target version containing a `-preview.` suffix: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
+
+## Read before editing
+
+- `AGENTS.md` (root and nearest path-specific files).
+- `.github/pull_request_template.md`.
+- Current branch status and diff against `origin/main`.
+
+## Default release file scope
+
+Treat the following file list as the default checklist for each release bump:
+
+- `Cargo.toml`
+- `Cargo.lock`
+- `README.md`
+- `README_ZH.md`
+- `flake.nix`
+- `helm/rustfs/Chart.yaml`
+- `rustfs.spec`
+
+Only drop a file when the current repository release process clearly no longer requires it.
+
+## Hard release policy
+
+- Docker doc tags use `<version>` (for example `rustfs/rustfs:1.0.0-beta.4`), not `v<version>`.
+- Helm chart version mapping follows `beta.N -> 0.N.0`.
+- `rustfs.spec` `Release` uses prerelease suffix only (for example `beta.4`).
+- Do not change these rules without explicit confirmation.
+
+## Step-by-step workflow
+
+1. Confirm intent and isolate scope
+- Confirm target version string exactly.
+- Confirm whether user requested local-only or full GitHub flow.
+- Inspect current branch and ensure only release-related files are touched for this task.
+
+2. Update workspace versions
+- Bump `[workspace.package].version` in `Cargo.toml`.
+- Bump internal workspace crate dependency versions in `Cargo.toml`.
+- Update `Cargo.lock` so workspace package versions match target version.
+- Re-scan for partial leftovers.
+
+3. Update release assets
+- `README.md` and `README_ZH.md`: update versioned Docker examples to target version.
+- `flake.nix`: update package version to target version.
+- `helm/rustfs/Chart.yaml`:
+- `appVersion` = target version.
+- `version` follows chart mapping rule, for example:
+- `1.0.0-beta.3` -> `0.3.0`
+- `1.0.0-beta.4` -> `0.4.0`
+- `rustfs.spec`:
+- Set `Release` to prerelease suffix (example `beta.4`).
+- Add/update top changelog entry with exact format:
+- `* Thu May 20 2026 houseme <housemecn@gmail.com>`
+- `- Update RPM package to RustFS 1.0.0-beta.4`
+- Changelog identity and time must come from current environment:
+- `git config --get user.name`
+- `git config --get user.email`
+- `date '+%a %b %d %Y'`
+- Changelog version text must match target release version exactly.
+
+4. Verify before shipping
+- Run:
+- `cargo fmt --all`
+- `cargo fmt --all --check`
+- `make pre-commit`
+- If verification passes, run `cargo clean`.
+- If `make pre-commit` fails, return `BLOCKED` with root cause and do not silently widen scope to fix unrelated issues unless user asks.
+
+5. Commit strategy
+- Preferred split when both parts changed:
+- `chore(release): prepare <version>` for `Cargo.toml` and `Cargo.lock`.
+- `chore(release): align release assets for <version>` for docs and packaging files.
+- If user asks for one commit, use one commit.
+- Stage only intended release files; do not include unrelated working tree changes.
+
+6. Push and PR
+- Push branch:
+- `git push -u origin <branch>` (first push), or `git push` (tracking already exists).
+- Create PR with template headings unchanged:
+- `gh pr create --base main --head <branch> --title ... --body-file ...`
+- PR title/body must be English.
+- Use `N/A` for non-applicable template sections.
+- Include verification commands and any `BLOCKED` reason clearly.
+
+## Recommended check commands
+
+- `git status --short --branch`
+- `git diff --name-only origin/main...HEAD`
+- `git diff --stat origin/main...HEAD`
+- `rg -n "<old_version>|<new_version>" Cargo.toml Cargo.lock README.md README_ZH.md flake.nix helm/rustfs/Chart.yaml rustfs.spec`
+- `cargo fmt --all`
+- `cargo fmt --all --check`
+- `make pre-commit`
+- `cargo clean`
+
+## Output contract
+
+When using this skill, always report:
+
+- Target version.
+- Files changed.
+- Any assumptions or uncertainties requiring confirmation.
+- Verification result (`PASSED` or `BLOCKED`) with key evidence.
+- Commit message(s) used.
+- Push status and PR URL when GitHub flow is requested.
@@ -0,0 +1,4 @@
+interface:
+  display_name: "RustFS Release Bump"
+  short_description: "Prepare RustFS release branches like PR #2957."
+  default_prompt: "Use $rustfs-release-version-bump to prepare a RustFS release version, ask about any unclear version policy, and finish the commit/push/PR flow."
@@ -0,0 +1,149 @@
+---
+name: security-advisory-lessons
+description: Apply RustFS security lessons distilled from repository GitHub Security Advisories. Use when making or reviewing RustFS code changes, doing security checks, handling PR review for auth/authz, IAM, storage, RPC, logging, CORS, console/browser, encryption, policy, or endpoint changes, and when deciding which security regression tests are required.
+---
+
+# RustFS Security Advisory Lessons
+
+Use this skill as a RustFS-specific security lens before changing or approving code. For the distilled advisory lessons and review patterns, read [advisory-patterns.md](references/advisory-patterns.md).
+
+When currentness matters, fetch the live advisory inventory instead of relying on this skill as a status mirror:
+
+```bash
+gh api repos/rustfs/rustfs/security-advisories --paginate \
+  --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
+```
+
+Fetch full advisory details only when the live summary suggests a new or changed lesson:
+
+```bash
+gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
+```
+
+For the full pattern map, read [advisory-patterns.md](references/advisory-patterns.md).
+
+## Workflow
+
+### 1. Scope the change
+- Identify touched routes, protocol frontends, handlers, storage paths, credentials, logs, browser surfaces, CI/release code, and policy checks.
+- Treat these paths as security-sensitive by default: `rustfs/src/admin/`, `rustfs/src/storage/`, `rustfs/src/auth.rs`, `rustfs/src/server/layer.rs`, `crates/iam/`, `crates/policy/`, `crates/credentials/`, `crates/ecstore/src/rpc/`, `crates/protocols/`, `crates/rio/`, OIDC/STS federation code, and console preview/auth code.
+
+### 2. Map to advisory classes
+- Read [advisory-patterns.md](references/advisory-patterns.md) for matching GHSA lessons.
+- Do not rely on advisory titles alone. Confirm whether the issue is authentication, authorization, input validation, storage invariant, browser isolation, logging, or operational hardening.
+
+### 3. Verify fail-closed behavior
+- Check that unauthenticated, wrong-permission, cross-user, cross-bucket, malformed-input, and default-config cases fail explicitly.
+- Prefer exact action/permission checks over broad helper calls or inferred ownership.
+- Confirm lower storage/RPC layers do not bypass checks done in upper layers.
+
+### 4. Require regression evidence
+- For behavior changes, add focused negative tests that reproduce the advisory class.
+- For sensitive fixes, include tests for the bypass form, not only the happy path.
+- If a test is impractical, explain the residual risk and provide a manual verification command.
+
+### 5. Report clearly
+- Lead with concrete findings and file/line evidence.
+- Separate proven vulnerabilities from hardening risks.
+- Avoid exaggerating unauthenticated impact when the code actually rejects unauthenticated requests but allows a low-privileged authenticated bypass.
+
+## Advisory-Derived Guardrails
+
+### Auth and admin authorization
+- Every admin or diagnostic route needs an explicit authn and authz story. Route registration, router whitelist, and handler-level authorization must agree.
+- Match the admin action to the operation exactly. Copy-paste action constants are a known RustFS vulnerability class.
+- Avoid authentication-only helpers for state-changing admin APIs; use `validate_admin_request` or the established equivalent with the right `AdminAction`.
+- Read-only admin APIs such as metrics, server info, and diagnostics still require admin authorization; checking only that credentials exist is not enough.
+- Replication admin reads can expose remote target credentials; list/get target endpoints require replication/admin authorization and must not return secrets to low-privilege callers.
+- Do not assume admin-action `Resource` scoping constrains blast radius unless the policy engine actually enforces resources for that action.
+
+### IAM and service accounts
+- Treat imported IAM payload fields as attacker-controlled: `parent`, `claims`, `accessKey`, `secretKey`, status, policy names, and groups.
+- For service account create/update/import, prove parent ownership or root/admin authority before writing credentials or claims; an action permission alone must not allow choosing root or another user as `target_user`.
+- Treat IAM export packages as credential disclosure surfaces; never include plaintext user or service-account secret keys unless the caller is allowed to recover those secrets and the export format is intentionally sealed.
+- Do not let `deny_only` or "no explicit deny" become an allow decision that skips required allow checks.
+- Test cross-user list/update/import flows with wrong, correct, self, parent, and root identities.
+
+### STS, OIDC, and federation flows
+- Every STS endpoint must have an explicit authentication story: SigV4 where required, OIDC token verification for web identity, and role/session policy validation before issuing credentials.
+- JWT session tokens must be signed and verified by a trusted issuer/key path, not by service-account-controlled material or a reused root secret.
+- JWT verification must enforce required claims and expiration for every bearer token path; "allow missing exp" is never acceptable for user-presented credentials.
+- Public OIDC bootstrap and callback routes must treat `Host`, `X-Forwarded-Proto`, redirect targets, `state`, and callback parameters as untrusted; credential-bearing redirects require a configured, allowlisted origin.
+- OIDC discovery and validation URLs are SSRF sinks. Resolve and classify hostnames at connection time, reject rebinding to loopback/private/link-local ranges, and do not rely on literal string checks.
+
+### S3 copy, multipart, and presigned POST
+- Multipart copy must enforce source `GetObject` and destination `PutObject` semantics equivalent to `CopyObject`, including copy-source and policy conditions.
+- Do not let `CreateMultipartUpload`, `UploadPartCopy`, `CompleteMultipartUpload`, or `AbortMultipartUpload` return success without authorization.
+- Presigned POST policies are server-side contracts. Enforce `content-length-range`, key prefix, exact metadata/content-type, and all signed policy conditions.
+
+### Protocol frontends and IAM parity
+- FTP/FTPS, SFTP, gateway, and other protocol drivers must enforce IAM per operation before calling storage backends; authentication to a protocol listener is not authorization.
+- Match protocol commands to the same S3 actions as HTTP, such as `RETR` to `GetObject`, `SIZE`/`MDTM` to `HeadObject`, `MKD` to `CreateBucket`, and bucket probes to `ListBucket` or `HeadBucket`.
+- Review every handler in a protocol driver, not only the changed handler, because RustFS advisories show mixed guarded and unguarded siblings in the same driver.
+- Regression tests for protocol frontends should deny the shared authorization hook and prove the backend is not reached for the denied command.
+- Compare protocol secrets in constant time, normalize invalid-user and invalid-secret failures where practical, and add rate limiting before exposing password-style protocol endpoints.
+
+### Paths, object keys, and filesystem access
+- Never join untrusted bucket/object/RPC path strings onto filesystem roots without normalization and boundary checks.
+- Reject or safely handle `..`, absolute paths, URL-encoded traversal, platform separators, empty components, and paths that canonicalize outside the intended root.
+- Validate both S3 object-key paths and internode/RPC disk paths; storage helpers can bypass S3 authorization if they trust already-parsed paths.
+- Archive auto-extract paths are object keys too. Validate tar/zip entry names before IAM checks and before storage writes, and prove cleaned paths cannot cross bucket or prefix boundaries.
+
+### Secrets, default credentials, and crypto
+- Do not ship hard-coded shared tokens, HMAC secrets, private keys, or production test keys.
+- Defaults for root credentials and internode/RPC auth must fail closed for network-reachable deployments or generate per-install random secrets; warnings alone are not a security boundary.
+- Keep cryptographic roles separated: root S3 credentials, RPC HMAC keys, and STS/JWT signing keys must not be reused or deterministically derived from each other.
+- License or token validation must use signatures with embedded public/verifying keys only; do not use private-key decryption as authenticity.
+- Plan key rotation and key IDs when removing exposed keys.
+
+### Logging and debug output
+- Logs must never include access keys beyond safe identifiers, secret keys, session tokens, JWT claims, HMAC secrets, expected signatures, license secrets, or raw response bodies containing credentials.
+- Treat `Debug` implementations, `?value` tracing, merged config dumps, and dependency-level HTTP body logging as leak surfaces.
+- Add log-capture tests or targeted unit tests for redaction wrappers when changing credential structs or response bodies.
+
+### RPC, parsing, and panic safety
+- Treat all RPC payload bytes as attacker-controlled. Replace `unwrap`, `expect`, and panic-prone deserialization with typed errors.
+- Malformed request tests should cover empty bytes, truncated MessagePack/protobuf, invalid enum values, stale timestamps, and invalid signatures.
+- RPC authentication must be independently strong; do not depend on S3 admin credentials unless the fallback is explicit and safe.
+- RPC signatures must bind the exact generated gRPC method path, timestamp, and request method. Service-prefix signatures must not authorize a different concrete NodeService call.
+
+### Browser, CORS, and console surfaces
+- Do not reflect arbitrary `Origin` while also allowing credentials. Default CORS should be no CORS unless explicitly configured.
+- Do not render user-controlled object content in a same-origin iframe with console credentials available to JavaScript.
+- Prefer origin separation for object preview/download, `nosniff`, CSP, strict content-type handling, and avoiding durable credentials in `localStorage`.
+- Preview safety must be based on trusted content type and sandboxing, not object names or extensions such as `.pdf`.
+- Console license/version-like metadata endpoints should expose only coarse public data unless authenticated, especially subject names and expiration timestamps.
+
+### Profiling, debug, and health endpoints
+- Profiling and debug endpoints are not health checks. They require admin auth, opt-in enablement, rate limiting, and safe responses.
+- Do not return absolute filesystem paths or other deployment layout in unauthenticated or low-privilege responses.
+- Ensure health endpoint allowlists cannot accidentally include expensive diagnostics.
+
+### Trusted proxy and network identity
+- Only honor `X-Forwarded-For` or `X-Real-IP` when the request came from a configured trusted proxy.
+- Apply the same trusted-proxy rule to scheme and host derivation; direct clients must not control security-sensitive redirects through `Host`, `X-Forwarded-Host`, or `X-Forwarded-Proto`.
+- Direct clients must use the socket peer address for `aws:SourceIp` and policy condition evaluation.
+- Add tests for direct spoofed headers and trusted-proxy headers.
+
+### SSE and storage invariants
+- Encryption metadata is not proof that bytes were encrypted on disk.
+- When touching reader/writer wrappers such as hashing, encryption, compression, or warp readers, verify wrapper order and inspect stored bytes in regression tests.
+- Avoid helper shortcuts that unwrap nested readers and accidentally bypass encryption or integrity layers.
+
+## Review Prompts
+
+Use these prompts while reviewing a diff:
+
+- Could a low-privileged authenticated user reach this path with the wrong action, parent, bucket, or source object?
+- Does a non-HTTP protocol path call the same authorization boundary as the S3 API before touching storage?
+- Does a public/default/empty config change security behavior from fail-closed to fail-open?
+- Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
+- Does this response contain stored replication, remote target, or service credentials that need redaction or stricter authorization?
+- Does an IAM export/import path expose or trust plaintext credential secrets beyond the caller's intended authority?
+- Can this STS/OIDC path issue credentials without SigV4, trusted issuer validation, allowlisted redirects, or trusted-proxy host/scheme handling?
+- Can a service-account or STS token omit `exp`, forge `sessionPolicy`, or use a principal-controlled key as signing authority?
+- Does this outbound validation path resolve attacker-supplied hostnames and reject private, loopback, link-local, and rebound addresses at the actual connection boundary?
+- Is an archive entry, object key, or policy resource normalized differently between authorization and storage?
+- Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
+- Does a preview or browser-surface fix preserve the original security invariant when adding alternate viewers or file-type detection?
+- Does the test prove the exploit form is denied, or only that the intended form still works?
@@ -0,0 +1,4 @@
+interface:
+  display_name: "Security Advisory Lessons"
+  short_description: "Apply advisory lessons in reviews."
+  default_prompt: "Review code changes against past RustFS security advisory lessons and report concrete risks, missing tests, and recommended fixes."
@@ -0,0 +1,129 @@
+# RustFS Advisory Pattern Map
+
+This file is a lesson map, not an advisory inventory mirror. It keeps durable security patterns distilled from RustFS GitHub Security Advisories.
+
+When current advisory state, severity, URLs, or full text matters, fetch it live:
+
+```bash
+gh api repos/rustfs/rustfs/security-advisories --paginate \
+  --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
+gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
+```
+
+Update this file only when an advisory adds or changes a reusable lesson, affected surface, validation pattern, or regression-test expectation. Do not update it for state-only, URL-only, count-only, or timestamp-only changes.
+
+## Pattern Index
+
+### Admin authorization and route exposure
+
+- `GHSA-pfcq-4gjr-6gjm`: notification target endpoints accepted authenticated users but skipped admin authorization. Lesson: distinguish authn from authz; admin target CRUD must call the operation-specific admin authorization path.
+- `GHSA-mm2q-qcmx-gw4w`: `ListServiceAccount` used `UpdateServiceAccountAdminAction`, while update lacked target ownership checks. Lesson: exact action constants and ownership checks are both required; information disclosure can chain into secret rotation and takeover.
+- `GHSA-vcwh-pff9-64cc`: `ImportIam` checked `ExportIAMAction` for an import/write operation. Lesson: every admin handler must authorize the action it actually performs.
+- `GHSA-jqmc-mg33-v45g` and `GHSA-8784-9m7f-c6p6`: `/profile/cpu` and `/profile/memory` were whitelisted from auth and allowed expensive diagnostics plus path disclosure. Lesson: profiling/debug endpoints need admin auth, opt-in, rate limits, and non-sensitive responses.
+- `GHSA-f5cv-v44x-2xgf`: `/rustfs/admin/v3/metrics` accepted any authenticated IAM user and skipped admin authorization. Lesson: read-only metrics and diagnostic admin endpoints still require an operation-specific admin action check.
+- `GHSA-796f-j7xp-hwf4`: `/rustfs/admin/v3/list-remote-targets` checked only that credentials existed and leaked replication target credentials. Lesson: replication target reads are privileged admin operations, and stored remote credentials require strict authz plus response redaction review.
+- `GHSA-xp32-gxq2-3v52`: console license metadata endpoint was public and exposed subject and expiration fields. Lesson: management metadata endpoints should require admin auth or return only coarse public status.
+
+### IAM import, service accounts, and privilege boundaries
+
+- `GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
+- `GHSA-3495-h8r9-gfqg`: `ExportIAM` wrote regular-user and service-account secret keys into exported ZIP data. Lesson: IAM export is a credential-disclosure boundary; redact, seal, or strictly justify every exported secret before treating export permission as safe.
+- `GHSA-5354-r3w2-34m8`: `AddServiceAccount` checked `CreateServiceAccountAdminAction` but trusted caller-supplied `target_user`, allowing service accounts under the root parent. Lesson: service-account create paths must validate parent ownership or root/admin authority, not only the create action.
+- `GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
+- `GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
+
+### STS, OIDC, and federation flows
+
+- `GHSA-5qfg-mf7r-jp3w` and `GHSA-3473-5353-xhwh`: `AssumeRoleWithWebIdentity` was reachable through unauthenticated `POST /` routing and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption, and unauthenticated exemptions must be narrowed to the exact action with uniform failure responses.
+- `GHSA-ccrv-v8v9-ch9q` and `GHSA-48rf-7j3q-3hfv`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
+- `GHSA-9pjf-w3c2-m32r`, `GHSA-4x2q-cpx9-9h26`, and `GHSA-xvpm-p3f7-34c3`: public OIDC authorize/callback flows trusted request `Host` or forwarded scheme when building credential-bearing redirects. Lesson: OIDC redirects must use configured allowlisted origins and trusted-proxy handling; never derive the post-login credential destination from direct client headers.
+- `GHSA-m479-9x88-94w6`, `GHSA-frwq-mfqx-83p8`, `GHSA-q9q8-rf9r-fg9f`, and `GHSA-j5c2-hhf7-6gf5`: OIDC validation accepted attacker-controlled discovery URLs because hostname checks rejected only literal forbidden IPs, allowing DNS rebinding SSRF. Lesson: outbound federation URL validation must resolve and classify hostnames at the connection boundary and reject loopback, private, link-local, and rebound addresses.
+
+### S3 copy, multipart, and upload policy validation
+
+- `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
+- `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
+- `GHSA-w5fh-f8xh-5x3p`: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
+
+### Protocol frontends and IAM parity
+
+- `GHSA-3g29-xff2-92vp`: FTP `RETR` and `SIZE`/`MDTM` read paths authenticated the user but skipped IAM before calling storage. Lesson: non-HTTP protocol frontends must enforce the same per-operation authorization as the S3 API before backend access.
+- `GHSA-g3vq-vv42-f647`: FTPS `MKD` called `create_bucket` without checking `s3:CreateBucket`. Lesson: protocol command handlers need action-specific checks even when sibling handlers already authorize correctly.
+- `GHSA-3p3x-734c-h5vx`: FTPS and WebDAV compared secret keys with early-return string equality, while FTPS also returned distinguishable invalid-user and invalid-password failures. Lesson: password-style protocol auth needs constant-time secret comparison, indistinguishable failures where practical, and rate limiting.
+
+### Filesystem paths and object key traversal
+
+- `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
+- `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
+- `GHSA-f4vq-9ffr-m8m3`: Snowball auto-extract accepted archive entries such as `../victim-bucket/object`, authorized the raw attacker-bucket path, then storage path cleaning crossed bucket boundaries. Lesson: archive entries become object keys and need traversal rejection plus consistent authz/storage normalization before writes.
+
+### Secrets, defaults, and cryptographic misuse
+
+- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, and `GHSA-63xc-c3w3-m2cf`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
+- `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
+- `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
+- `GHSA-75fx-qg6f-8rm7` and `GHSA-68cw-96m3-h2cf`: internode RPC secrets were derivable from known root credentials, making raw storage RPC signatures forgeable when explicit RPC secrets were unset. Lesson: RPC auth keys must be independent random secrets, never derived from S3 root credentials, and raw storage RPC should not share the public S3 listener without an internode-only boundary.
+- `GHSA-m77q-r63m-pj89`: STS JWTs used the root secret key as the shared token signing key, allowing token forgery when the root secret was known. Lesson: STS signing keys need key separation, rotation, and key IDs; do not reuse root credentials for JWT/HMAC signing.
+- `GHSA-923g-jp7v-f97f`: license verification embedded a production RSA private key and used private-key decryption as authenticity. Lesson: ship verifying/public keys only and use real signature verification.
+
+### Sensitive logging and debug output
+
+- `GHSA-r54g-49rx-98cr`: STS credentials were logged at info level. Lesson: generated credentials must never be logged in plaintext.
+- `GHSA-8cm2-h255-v749`: debug logs leaked session tokens, secret keys, JWT claims, and raw STS response bodies. Lesson: redaction must cover custom `Debug` implementations and dependency response-body logging.
+- `GHSA-333v-68xh-8mmq`: invalid RPC signature logging included the shared HMAC secret and expected signature. Lesson: error paths often leak secrets; never log raw secrets or derived authenticators.
+
+### RPC input validation and panic safety
+
+- `GHSA-gw2x-q739-qhcr`: malformed gRPC `GetMetrics` payloads reached `unwrap()` on deserialization and caused remote DoS. Lesson: every network/RPC deserialization failure returns an error, not a panic.
+- `GHSA-h956-rh7x-ppgj` and `GHSA-r5qv-rc46-hv8q`: weak RPC auth increased reachability of otherwise internal handlers. Lesson: panic bugs become more severe when internode auth is weak or defaulted.
+- `GHSA-c667-rgrv-99vj`: NodeService authentication signed the service prefix instead of the concrete generated method path, so valid metadata for one RPC could be replayed to another method during the timestamp window. Lesson: RPC HMAC payloads must bind exact gRPC method path, HTTP method surrogate, timestamp, and secret.
+
+### Browser, CORS, and console isolation
+
+- `GHSA-v9fg-3cr2-277j`: object preview rendered attacker-controlled HTML in a same-origin iframe, exposing console credentials stored in `localStorage`. Lesson: user content must be origin-isolated from the console and protected with `nosniff`, CSP, and strict content-type handling.
+- `GHSA-7gcx-wg4x-q9x6`: an incomplete preview fix reintroduced extension-based PDF detection and bypassed the sandboxed fallback for attacker-controlled content. Lesson: browser-surface fixes need regression tests for alternate viewers and file-type branches, and preview trust must come from validated content type plus sandboxing rather than object names.
+- `GHSA-x5xv-223c-8vm7`: default CORS reflected arbitrary origins with credentials. Lesson: never combine reflected origins with `Access-Control-Allow-Credentials: true`; default should be fail-closed.
+
+### Trusted proxy and source IP conditions
+
+- `GHSA-fc6g-2gcp-2qrq`: `aws:SourceIp` trusted client-supplied `X-Forwarded-For` or `X-Real-IP`. Lesson: forwarded IP headers are valid only behind configured trusted proxies; direct clients use socket peer IP.
+
+### SSE and on-disk storage invariants
+
+- `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
+
+### Serde deserialization and input validation
+
+- No `#[serde(deny_unknown_fields)]` found across the entire codebase. Lesson: all structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs) should have `#[serde(deny_unknown_fields)]` to reject malformed or adversarial payloads.
+- `#[serde(default)]` on security-critical fields silently accepts missing values as zero/empty. Lesson: when a field has security implications (retention days, permissions, limits), validate the deserialized value explicitly rather than relying on defaults.
+- Integer fields deserialized from user input and cast with `as` (e.g., `i32 as u32`) can wrap negative values to large positives. Lesson: validate ranges before casting; use `try_into()` or clamp.
+- XML config typos (e.g., `"NoncurentDays"` instead of `"NoncurrentDays"`) are silently accepted when `deny_unknown_fields` is absent. Lesson: strict deserialization prevents silent misconfiguration that could cause data loss or unexpected retention behavior.
+
+## Useful Search Seeds
+
+Use these targeted searches when a diff touches security-sensitive code:
+
+```bash
+rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
+rg -n "authorize_operation|FtpsDriver|SftpDriver|RETR|MKD|SIZE|MDTM|CreateBucket|GetObject|HeadObject" crates/protocols rustfs
+rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|content-length-range|starts-with" rustfs crates
+rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
+rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
+rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
+rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
+rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
+rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
+```
+
+## Minimum Regression Test Expectations
+
+- Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
+- Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases, and assert denied paths do not reach the storage backend.
+- IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
+- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
+- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
+- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
+- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
+- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
+- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
+- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.
@@ -0,0 +1,34 @@
+---
+name: tier-debug
+description: Debug ILM tiering / lifecycle transition issues — NoSuchVersion on tier GET, restore failures, xl.meta inspection, remote-tier versionId tracing. Use when investigating tiered/transitioned objects, warm backends, or transition metadata.
+---
+
+# Tier / ILM Debugging
+
+Full playbook: [docs/operations/tier-ilm-debugging.md](../../../docs/operations/tier-ilm-debugging.md)
+— read it before changing tier code.
+
+Quick moves:
+
+```bash
+# Inspect transition metadata on disk (one xl.meta per erasure shard disk)
+cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/{bucket}/{object}/xl.meta"
+
+# Trace what versionId is sent to the remote tier
+RUST_LOG=rustfs_ecstore::bucket::lifecycle=debug ./target/debug/rustfs …
+```
+
+Interpretation:
+
+- `transition_ver_id: <none>` → correct for an unversioned tier bucket; no
+  `versionId` must be sent on tier GET/DELETE.
+- `transition_ver_id: 00000000-…` (nil) → corrupt legacy write-back; readers
+  must filter it out, never send it.
+- Empty-string `transitioned-versionID` metadata under both
+  `x-rustfs-internal-*` and `x-minio-internal-*` keys → object went to an
+  unversioned tier bucket.
+
+Code entry points: `crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs`
+(ILM actions), `crates/ecstore/src/services/tier/` (warm backends),
+`crates/filemeta/src/filemeta/version.rs` (metadata read/write + regression
+tests).
@@ -14,13 +14,20 @@

 # RustFS Cargo configuration

-# Enable tokio_unstable cfg for dial9-tokio-telemetry support
-# This allows dial9 to hook into Tokio's internal runtime events
-[build]
-# Enable Tokio unstable features required by dial9-tokio-telemetry for runtime tracing.
-# See: https://docs.rs/tokio/latest/tokio/#unstable-features
-rustflags = ["--cfg", "tokio_unstable"]
-
-# Enable frame pointers for CPU profiling (Linux only, optional but recommended)
-# Uncomment the following line for better CPU profiling data
-# rustflags = ["--cfg", "tokio_unstable", "-C", "force-frame-pointers=yes"]
+# NOTE: `--cfg tokio_unstable` is deliberately NOT set here.
+#
+# It used to be a global `[build] rustflags` entry so that the (default-off)
+# `dial9` telemetry feature could compile. That made every build depend on
+# Tokio's unstable, non-semver API, and it broke silently whenever a caller
+# exported their own RUSTFLAGS — an environment RUSTFLAGS replaces the value
+# from this file rather than appending to it.
+#
+# The flag is now scoped to telemetry builds, which opt in explicitly:
+#
+#     make build-profiling
+#     RUSTFLAGS="--cfg tokio_unstable" cargo build --features dial9
+#
+# `crates/obs/build.rs` fails the compile if the `dial9` feature is on without
+# the flag, so the two can no longer drift apart unnoticed.
+#
+# For CPU profiling, add `-C force-frame-pointers=yes` to that RUSTFLAGS value.
@@ -0,0 +1 @@
+../.agents/skills
@@ -9,7 +9,7 @@ build-docker: SOURCE_BUILD_CONTAINER_NAME = rustfs-$(BUILD_OS)-build
 build-docker: BUILD_CMD = /root/.cargo/bin/cargo build --release --bin rustfs --target-dir /root/s3-rustfs/target/$(BUILD_OS)
 build-docker: ## Build using Docker container # e.g (make build-docker BUILD_OS=ubuntu22.04)
 	@echo "🐳 Building RustFS using Docker ($(BUILD_OS))..."
-	$(DOCKER_CLI) buildx build -t $(SOURCE_BUILD_IMAGE_NAME) -f $(DOCKERFILE_SOURCE) .
+	$(DOCKER_CLI) buildx build -t $(SOURCE_BUILD_IMAGE_NAME) -f $(DOCKERFILE_SOURCE) --load .
 	$(DOCKER_CLI) run --rm --name $(SOURCE_BUILD_CONTAINER_NAME) -v $(shell pwd):/root/s3-rustfs -it $(SOURCE_BUILD_IMAGE_NAME) $(BUILD_CMD)

 .PHONY: docker-inspect-multiarch
@@ -19,4 +19,4 @@ docker-inspect-multiarch: ## Check image architecture support
 		exit 1; \
 	fi
 	@echo "🔍 Inspecting multi-architecture image: $(IMAGE)"
-	docker buildx imagetools inspect $(IMAGE)
+	docker buildx imagetools inspect $(IMAGE)
@@ -35,6 +35,25 @@ build-gnu-arm64: ## Build aarch64 GNU version
 	./build-rustfs.sh --platform aarch64-unknown-linux-gnu


+## —— Profiling build (dial9 Tokio runtime telemetry) ------------------------------------------
+
+# dial9 hooks Tokio's unstable runtime instrumentation, so it needs
+# `--cfg tokio_unstable`. That flag is deliberately absent from
+# .cargo/config.toml: it is not free, and release binaries do not carry it.
+# Setting RUSTFLAGS here replaces (never appends to) the config-file value, and
+# crates/obs/build.rs fails the build if the feature and the flag disagree.
+#
+# There are no task-dump or S3-upload features — see the notes in
+# crates/obs/Cargo.toml for why.
+DIAL9_FEATURES ?= dial9
+DIAL9_RUSTFLAGS ?= --cfg tokio_unstable
+
+.PHONY: build-profiling
+build-profiling: ## Build RustFS with dial9 Tokio runtime telemetry (diagnostic builds only)
+	@echo "🔬 Building RustFS with dial9 telemetry (features: $(DIAL9_FEATURES))..."
+	@echo "⚠️  Diagnostic build: telemetry writes trace segments to disk continuously."
+	RUSTFLAGS="$(DIAL9_RUSTFLAGS)" cargo build --release --bin rustfs --features $(DIAL9_FEATURES)
+
 .PHONY: build-cross-all
 build-cross-all: core-deps ## Build binaries for all architectures
 	@echo "🔧 Building all target architectures..."
@@ -11,14 +11,16 @@ check-%:

 # Warning-only check
 # Checks for optional dependencies and issues a warning if not found
-# (e.g., cargo-nextest for enhanced testing)
 warn-%:
 	@command -v $* >/dev/null 2>&1 || { \
 		echo >&2 "⚠️ '$*' is not installed."; \
 	}

 # For checking dependencies use check-<dep-name> or warn-<dep-name>
-.PHONY: core-deps fmt-deps test-deps
+#
+# NOTE: cargo-nextest is a HARD dependency of `make test`, gated inside the
+# test recipe itself (with a RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 escape hatch)
+# rather than a warn-only prerequisite here — see .config/make/tests.mak.
+.PHONY: core-deps fmt-deps
 core-deps: check-cargo ## Check core dependencies
 fmt-deps: check-rustfmt ## Check lint and formatting dependencies
-test-deps: warn-cargo-nextest ## Check tests dependencies
@@ -0,0 +1,26 @@
+## —— Coverage --------------------------------------------------------------------------------------
+
+# Local equivalent of the weekly coverage workflow (.github/workflows/coverage.yml,
+# backlog#1153 infra-5): same measurement scope (--workspace --exclude e2e_test,
+# nextest `ci` profile) and the same per-crate table. Slow — the instrumented
+# build cannot reuse your normal target cache and then runs the whole suite.
+# Doctests are not measured (needs nightly). Outputs land in target/llvm-cov/.
+.PHONY: coverage
+coverage: core-deps ## Workspace line coverage (cargo-llvm-cov + nextest; slow, writes target/llvm-cov/)
+	@if ! command -v cargo-llvm-cov >/dev/null 2>&1; then \
+		echo >&2 "❌ cargo-llvm-cov is required for 'make coverage' but was not found."; \
+		echo >&2 "   Install it with:"; \
+		echo >&2 "     cargo install cargo-llvm-cov --locked"; \
+		echo >&2 "     rustup component add llvm-tools-preview"; \
+		exit 1; \
+	fi
+	@if ! command -v cargo-nextest >/dev/null 2>&1; then \
+		echo >&2 "❌ cargo-nextest is required for 'make coverage' (see 'make test')."; \
+		echo >&2 "   Install it with: cargo install cargo-nextest --locked"; \
+		exit 1; \
+	fi
+	NEXTEST_PROFILE=ci cargo llvm-cov nextest --workspace --exclude e2e_test --no-report
+	@mkdir -p target/llvm-cov
+	cargo llvm-cov report --lcov --output-path target/llvm-cov/lcov.info
+	cargo llvm-cov report --json --output-path target/llvm-cov/coverage.json
+	python3 scripts/coverage_per_crate.py target/llvm-cov/coverage.json
@@ -4,7 +4,7 @@
 .PHONY: help
 help: ## Shows This Help Menu
 	echo -e "$$HEADER"
-	grep -E '(^[a-zA-Z0-9_-]+:.*?## .*$$)|(^## )' $(MAKEFILE_LIST) | sed 's/^[^:]*://g' | awk 'BEGIN {FS = ":.*?## | #"} ; {printf "${cyan}%-30s${reset} ${white}%s${reset} ${green}%s${reset}\n", $$1, $$2, $$3}' | sed -e 's/\[36m##/\n[32m##/'
+	grep -E '(^[a-zA-Z0-9_-]+:.*?## .*$$)|(^## )' $(MAKEFILE_LIST) | sed 's/^[^:]*://g' | awk 'BEGIN {FS = ":.*?## | #"} /^## / {printf "\n${green}%s${reset}\n", $$0; next} {printf "${cyan}%-30s${reset} ${white}%s${reset} ${green}%s${reset}\n", $$1, $$2, $$3}'

 .PHONY: help-build
 help-build: ## Shows RustFS build help
@@ -1,5 +1,7 @@
 ## —— Code quality and Formatting ------------------------------------------------------------------

+.NOTPARALLEL: fix
+
 .PHONY: fmt
 fmt: core-deps fmt-deps ## Format code
 	@echo "🔧 Formatting code..."
@@ -13,8 +15,55 @@ fmt-check: core-deps fmt-deps ## Check code formatting
 .PHONY: clippy-check
 clippy-check: core-deps ## Run clippy checks
 	@echo "🔍 Running clippy checks..."
+	cargo clippy --all-targets -- -D warnings
+
+.PHONY: clippy-fix
+clippy-fix: core-deps ## Apply clippy fixes
+	@echo "🔧 Applying clippy fixes..."
 	cargo clippy --fix --allow-dirty
-	cargo clippy --all-targets --all-features -- -D warnings
+
+.PHONY: fix
+fix: fmt clippy-fix ## Format code and apply clippy fixes
+
+.PHONY: quick-check
+quick-check: core-deps ## Run fast workspace compilation check
+	@echo "🔨 Running fast compilation check..."
+	cargo check --workspace --exclude e2e_test
+
+.PHONY: unsafe-code-check
+unsafe-code-check: ## Check unsafe_code allowances have SAFETY comments
+	@echo "🔒 Checking unsafe_code allowances..."
+	./scripts/check_unsafe_code_allowances.sh
+
+.PHONY: architecture-migration-check
+architecture-migration-check: ## Check architecture migration guardrails
+	@echo "🏗️ Checking architecture migration guardrails..."
+	./scripts/check_architecture_migration_rules.sh
+
+.PHONY: logging-guardrails-check
+logging-guardrails-check: ## Check logging guardrails for redaction and noise regressions
+	@echo "🪵 Checking logging guardrails..."
+	./scripts/check_logging_guardrails.sh
+
+.PHONY: tokio-io-uring-check
+tokio-io-uring-check: ## Check tokio io-uring runtime feature stays removed
+	@echo "🚫 Checking tokio io-uring feature guard..."
+	./scripts/check_no_tokio_io_uring.sh
+
+.PHONY: extension-schema-check
+extension-schema-check: ## Check extension-schema stays a lightweight contract crate
+	@echo "🧩 Checking extension schema boundaries..."
+	./scripts/check_extension_schema_boundaries.sh
+
+.PHONY: body-cache-whitelist-check
+body-cache-whitelist-check: ## Check the body-cache eligibility gate stays a fail-closed allow-list
+	@echo "🧱 Checking body-cache whitelist guard..."
+	./scripts/check_body_cache_whitelist.sh
+
+.PHONY: log-analyzer-rules-check
+log-analyzer-rules-check: core-deps ## Check log-analyzer rule anchors still exist verbatim in source
+	@echo "🩺 Checking log-analyzer rule anchors..."
+	./scripts/check_log_analyzer_rules.sh

 .PHONY: compilation-check
 compilation-check: core-deps ## Run compilation check
@@ -1,11 +1,31 @@
 ## —— Pre Commit Checks ----------------------------------------------------------------------------

+.NOTPARALLEL: pre-commit pre-pr dev-check
+
 .PHONY: setup-hooks
 setup-hooks: ## Set up git hooks
 	@echo "🔧 Setting up git hooks..."
 	chmod +x .git/hooks/pre-commit
 	@echo "✅ Git hooks setup complete!"

+.PHONY: doc-paths-check
+doc-paths-check: ## Check that instruction/architecture docs reference existing file paths
+	@echo "📄 Checking doc path references..."
+	./scripts/check_doc_paths.sh
+
+.PHONY: planning-docs-check
+planning-docs-check: ## Check that no planning-type documents are committed
+	@echo "📄 Checking for committed planning docs..."
+	./scripts/check_no_planning_docs.sh
+
 .PHONY: pre-commit
-pre-commit: fmt clippy-check compilation-check test ## Run pre-commit checks
-	@echo "✅ All pre-commit checks passed!"
+pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
+	@echo "✅ All pre-commit checks passed!"
+
+.PHONY: pre-pr
+pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests
+	@echo "✅ All pre-PR checks passed!"
+
+.PHONY: dev-check
+dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
+	@echo "✅ Fast development checks passed!"
@@ -2,14 +2,67 @@

 TEST_THREADS ?= 1

+# cargo-nextest is a HARD dependency of `make test`.
+#
+# nextest changes test semantics vs plain `cargo test`: it runs every test in
+# its own process (so serial_test's #[serial] mutex does not serialize across
+# tests) and it is the only runner that honours .config/nextest.toml
+# [test-groups] (e.g. the ecstore-serial-flaky serialization guard). CI runs
+# nextest (.github/actions/setup/action.yml installs it), so a silent fallback
+# to `cargo test` would run with different serialization behaviour than CI and
+# mask (or invent) flakes.
+#
+# Installing cargo-nextest:
+#   cargo install cargo-nextest --locked          # from source
+#   # or a prebuilt binary (faster) via the taiki-e installer / get.nexte.st:
+#   #   https://nexte.st/docs/installation/
+#
+# Escape hatch: set RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 to run the plain
+# `cargo test` fallback anyway. Results are NOT authoritative — semantics
+# differ from CI and .config/nextest.toml test-groups will NOT apply.
+
+.PHONY: script-tests
+script-tests: ## Run shell script tests
+	@echo "Running script tests..."
+	./scripts/test_build_rustfs_options.sh
+	./scripts/test_entrypoint_credentials.sh
+	bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
+	python3 ./scripts/check_object_data_cache_follower_samples.py --self-test
+	./scripts/validate_object_data_cache_cold_stampede.sh --self-test
+
 .PHONY: test
-test: core-deps test-deps ## Run all tests
+test: core-deps script-tests ## Run all tests (needs cargo-nextest; RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 to override)
 	@echo "🧪 Running tests..."
 	@if command -v cargo-nextest >/dev/null 2>&1; then \
 		cargo nextest run --all --exclude e2e_test; \
-	else \
-		echo "ℹ️ cargo-nextest not found; falling back to 'cargo test'"; \
+	elif [ "$${RUSTFS_ALLOW_CARGO_TEST_FALLBACK:-0}" = "1" ]; then \
+		echo >&2 "⚠️  ============================================================================"; \
+		echo >&2 "⚠️  cargo-nextest NOT found — running the 'cargo test' fallback (opt-in)."; \
+		echo >&2 "⚠️  TEST SEMANTICS DIFFER FROM CI; results are NOT authoritative:"; \
+		echo >&2 "⚠️    * nextest runs each test in its own process; 'cargo test' does not,"; \
+		echo >&2 "⚠️      so serial_test #[serial] serialization behaves differently."; \
+		echo >&2 "⚠️    * .config/nextest.toml [test-groups] will NOT apply (e.g. the"; \
+		echo >&2 "⚠️      ecstore-serial-flaky group), so load-sensitive tests may flake here"; \
+		echo >&2 "⚠️      but pass on CI (or vice versa)."; \
+		echo >&2 "⚠️  Install cargo-nextest and re-run before trusting these results."; \
+		echo >&2 "⚠️  ============================================================================"; \
 		cargo test --workspace --exclude e2e_test -- --nocapture --test-threads="$(TEST_THREADS)"; \
+	else \
+		echo >&2 "❌ cargo-nextest is required for 'make test' but was not found."; \
+		echo >&2 ""; \
+		echo >&2 "   RustFS tests run under cargo-nextest (process-per-test isolation)."; \
+		echo >&2 "   CI runs nextest and .config/nextest.toml [test-groups] only take effect"; \
+		echo >&2 "   under nextest. Plain 'cargo test' has different serialization semantics"; \
+		echo >&2 "   and is NOT a faithful substitute."; \
+		echo >&2 ""; \
+		echo >&2 "   Install it with either:"; \
+		echo >&2 "     cargo install cargo-nextest --locked"; \
+		echo >&2 "   or a prebuilt binary (faster) — see https://nexte.st/docs/installation/"; \
+		echo >&2 ""; \
+		echo >&2 "   To run the plain 'cargo test' fallback anyway (results NOT authoritative;"; \
+		echo >&2 "   serialization semantics differ from CI), re-run with:"; \
+		echo >&2 "     RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 make test"; \
+		exit 1; \
 	fi
 	cargo test --all --doc

@@ -0,0 +1,10 @@
+# Committed floor for the number of tests selected by the migration-critical
+# CI gate (see scripts/check_migration_gate_count.sh, backlog#1153 infra-12).
+#
+# The floor equals the exact count of rustfs-ecstore --lib tests matching the
+# gate filter (name substrings: data_movement, rebalance, decommission,
+# source_cleanup, delete_marker) at the time this file was last updated.
+# CI fails if the selected count drops below this number, so renames or
+# removals that thin the gate must update this file in the same PR.
+# Adding tests does not require a bump, but bumping keeps the guard tight.
+571
@@ -0,0 +1,316 @@
+# nextest configuration for RustFS.
+#
+# Serialize two known load-sensitive / global-state-sharing ecstore test groups
+# so the full parallel nextest suite stops producing spurious failures
+# (backlog #937). These tests pass in isolation but flake under the loaded
+# parallel run for two distinct reasons:
+#
+#   * store::bucket::tests::bucket_delete_* share process/global state (disk
+#     registry, lock client) and race make_bucket into InsufficientWriteQuorum
+#     when run concurrently with other ecstore tests.
+#   * bucket_lifecycle_ops::tests::concurrent_resend_same_part_commits_one_generation
+#     asserts a lock-acquire correctness property whose serialized cross-disk
+#     commits exceed the (already max'd, 60s) acquire deadline only when the
+#     suite saturates disk I/O.
+#
+# serial_test's #[serial] attribute does NOT serialize these across runs:
+# nextest executes each test in its own process, where the in-process
+# serial_test mutex has no effect. A nextest test-group with max-threads = 1 is
+# the mechanism that actually serializes across nextest's process boundary.
+#
+# ---------------------------------------------------------------------------
+# Profiles
+# ---------------------------------------------------------------------------
+# The `default` profile is what local `cargo nextest run` uses. It NEVER
+# retries: a red test locally means a real failure to investigate, not noise to
+# paper over. The `ci` profile (below) is the strict CI gate: global
+# retries = 0 so a new race's first occurrence is never masked, plus a
+# narrowly-scoped quarantine list (retries = 2) for tests with a tracked OPEN
+# flake issue. Flake policy lives in docs/testing/README.md.
+
+[test-groups]
+ecstore-serial-flaky = { max-threads = 1 }
+
+# Reliability / fault-injection e2e tests each spawn a single-node 4-disk RustFS
+# server and manipulate its disk directories at runtime (crates/e2e_test:
+# reliability_disk_fault_test, degraded_read_eof_regression_test / dist-13). They
+# are correct in isolation but resource-heavy; serialize them under nextest's
+# process boundary (serial_test's #[serial] does not cross it) so several 4-disk
+# servers never run at once. ci-7's nightly picks these up via the e2e suite;
+# they are deliberately NOT in the fast PR `e2e-smoke` filter.
+e2e-reliability = { max-threads = 1 }
+
+# --- default profile (local): serialize the flaky groups, never retry --------
+[[profile.default.overrides]]
+filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
+test-group = 'ecstore-serial-flaky'
+
+# Serialize the multipart crash-consistency scenarios (dist-2, backlog#1150):
+# each spawns a 4-disk hermetic erasure set and drives full staged-upload +
+# commit + GET cycles — the same cross-disk-commit IO shape that made
+# concurrent_resend load-sensitive. Preventive serialization only, no retries.
+# The matching ci-profile override is after [profile.ci].
+[[profile.default.overrides]]
+filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
+test-group = 'ecstore-serial-flaky'
+
+# Serialize the 4-disk reliability / degraded-read e2e tests (see the
+# e2e-reliability test-group note above). The matching ci-profile override is at
+# the end of the file, after [profile.ci] is declared.
+[[profile.default.overrides]]
+filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
+test-group = 'e2e-reliability'
+
+# ---------------------------------------------------------------------------
+# ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
+# ---------------------------------------------------------------------------
+[profile.ci]
+# Strict: a new race must fail on its first occurrence, never be retried away.
+retries = 0
+# Report every failure in one run instead of bailing on the first.
+fail-fast = false
+
+[profile.ci.junit]
+# Emitted to target/nextest/ci/junit.xml; uploaded as a CI artifact.
+# Tests that pass only after a quarantine retry are marked `flaky` here — that
+# marker is the observable signal the flake policy is built around.
+path = "junit.xml"
+
+# ===========================================================================
+# QUARANTINE — flaky tests granted retries = 2 under the ci profile ONLY.
+#
+# RULES (enforced by review, see docs/testing/README.md):
+#   * Every entry MUST link exactly one OPEN issue tracking the flake.
+#   * An entry stays until the issue is fixed (test made robust) or the test is
+#     deleted — 30-day policy. No entry may exist without a live issue link.
+#
+# Each entry also re-declares the `ecstore-serial-flaky` test-group so the
+# serialization holds under the ci profile (nextest evaluates a named
+# profile's own overrides list, not the default profile's).
+# ===========================================================================
+
+# QUARANTINE: OPEN backlog#937 — concurrent_resend lock-acquire deadline flakes
+# under saturated disk I/O in the full parallel suite.
+[[profile.ci.overrides]]
+filter = 'package(rustfs-ecstore) & test(concurrent_resend_same_part_commits_one_generation)'
+test-group = 'ecstore-serial-flaky'
+retries = 2
+
+# QUARANTINE: OPEN backlog#937 — store::bucket::tests::bucket_delete_* race
+# make_bucket into InsufficientWriteQuorum via shared global state under load.
+[[profile.ci.overrides]]
+filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/)'
+test-group = 'ecstore-serial-flaky'
+retries = 2
+
+# QUARANTINE: OPEN rustfs#4690 — walk_dir stall-budget accounting test depends
+# on producer/consumer timing windows that stretch past the budget on loaded
+# CI runners (regression test for rustfs#4644; failed on a zero-Rust-diff PR).
+[[profile.ci.overrides]]
+filter = 'package(rustfs-ecstore) & test(walk_dir_does_not_charge_consumer_backpressure_to_the_stall_budget)'
+retries = 2
+
+# Serialize the 4-disk reliability / degraded-read e2e tests under the ci
+# profile too (see the e2e-reliability test-group note near the top). Not a
+# quarantine: no retries, just single-threaded so several 4-disk servers never
+# run concurrently when ci-7's nightly runs the full e2e suite.
+[[profile.ci.overrides]]
+filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
+test-group = 'e2e-reliability'
+
+# Serialize the multipart crash-consistency scenarios under the ci profile too
+# (see the matching default-profile override near the top). Not a quarantine:
+# no retries, just serialized 4-disk cross-disk-commit IO.
+[[profile.ci.overrides]]
+filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
+test-group = 'ecstore-serial-flaky'
+
+# ---------------------------------------------------------------------------
+# e2e-smoke profile — PR smoke subset of the e2e_test crate (backlog#1149 ci-4)
+# ---------------------------------------------------------------------------
+# PR smoke subset of the e2e_test crate (backlog#1149 ci-4). This profile is
+# the single wiring mechanism for e2e tests in CI: other suites join by
+# extending this filter (or a sibling profile), never by adding ad-hoc e2e
+# jobs to ci.yml. Admission criteria (see crates/e2e_test/README.md): fast,
+# single-node topology, no external dependencies (no awscurl / Vault / fixed
+# ports / pre-started server), no #[ignore].
+#
+# Each e2e test spawns its own rustfs server on a random port with an isolated
+# temp dir (crates/e2e_test/src/common.rs), so the subset is parallel-safe.
+#
+# Replication failure harness (backlog#1147 repl-8): the first clause admits
+# its four in-process fake-target self-tests. They bind random loopback ports,
+# use no external service, and finish in under a second.
+#
+# Replication PR subset (backlog#1147 repl-1): the second clause admits the 20
+# FAST bucket-replication tests from replication_extension_test — the
+# target-registration / replication-check / list / remove / delete admin paths
+# that validate config synchronously and never wait for asynchronous
+# replication convergence. Each spawns its own single-node rustfs server(s) on
+# random ports (source, plus an independent single-node target for the pair
+# checks — NOT a cluster), so the subset stays parallel-safe and single-digit
+# seconds. The data-plane tests that poll for convergence and all
+# `_real_dual_node` / `_real_single_node`
+# site-replication tests run in the [profile.e2e-repl-nightly] lane below, NOT
+# here. This allowlist is the single source of truth for the PR/nightly split:
+# the nightly profile derives its set as "the replication module MINUS this
+# allowlist", so any new replication test lands in nightly by default (never
+# silently unrun) until it is explicitly blessed as fast here. Keep the two
+# regexes byte-identical. Count invariant: 20 here + 27 nightly = 47 total
+# (authority: `cargo nextest list`; docs/testing/e2e-suite-inventory.md).
+# HISTORY (2026-07-11): the 20 fast tests were briefly pulled out of this lane
+# (#4724) because they set a loopback (127.0.0.1) replication target that the
+# SSRF egress guard rejected on every PR after repl-1 (#4712). That is fixed —
+# the guard now honours an off-by-default opt-in and this suite's source servers
+# set it (RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) — so the allowlist below is
+# restored.
+#
+# Security negative-auth subset (backlog#1151 sec-5): the three attacker-facing
+# S3 auth-rejection suites join the first clause above by module name —
+# presigned_negative (sec-2), negative_sigv4 (sec-1, header SigV4), and
+# admin_auth (sec-4, admin gate + root-credential lifecycle). All three use
+# RustFSTestEnvironment on a random port and are parallel-safe, so they meet the
+# smoke admission criteria unchanged. This is the wiring step that makes those
+# merged suites actually execute on every PR (they were dead until listed here).
+# A rename that drops any of them out of this filter would silently thin the
+# security gate with no CI signal, so scripts/check_security_smoke_count.sh owns
+# a count-floor guard over exactly this subset (infra-12 mechanism, floor in
+# .config/security-smoke-floor.txt), invoked from the e2e-tests job in ci.yml.
+# NOT here by topology: the GHSA-3p3x FTPS/WebDAV constant-time e2e
+# (protocols::test_protocol_core_suite) binds fixed ports and needs the
+# ftps,webdav features, so it cannot join this random-port, default-feature
+# profile; its GHSA-r5qv sibling is a unit test that already runs in the
+# test-and-lint `--all --exclude e2e_test` pass. See
+# docs/testing/security-regressions.md for the full CI-execution map.
+#
+# ILM tiering main path (backlog#1148 ilm-7): the `reliant::tiering::` clause
+# admits the hermetic transition e2e. Like the fast replication pair checks it
+# spawns a second independent single-node server (the cold RustFS tier), not a
+# cluster, so it keeps the lane's parallel-safe / no-external-dependency
+# properties. The RustFS warm backend has no loopback guard (that guard is
+# replication-only), so it needs no opt-in env for its 127.0.0.1 tier target.
+[profile.e2e-smoke]
+default-filter = """
+    package(e2e_test) & (
+        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_source_invalid_date|content_encoding|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools)_test::|^fake_s3_target::/)
+        | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
+        | test(/^reliant::lifecycle::/)
+        | test(/^reliant::tiering::/)
+    )
+"""
+fail-fast = false
+
+# ---------------------------------------------------------------------------
+# e2e-repl-nightly profile — scheduled full replication e2e lane (repl-1)
+# ---------------------------------------------------------------------------
+# backlog#1147 repl-1 (deps: ci-4). Runs the SLOW / cross-process replication
+# tests that are unfit for the per-PR e2e-smoke gate:
+#
+#   * 2 remote-target TLS validation tests.
+#   * 12 bucket-replication data-plane/helper tests — they PUT/delete objects
+#     and poll until source and target converge; two replicate over HTTPS, two
+#     pin active SSE failure contracts, and one guards event/history observers.
+#     The SSE-S3 contract remains ignored under backlog#1291.
+#   * 11 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
+#     servers and drives the cross-process site-replication control plane.
+#   * 1 `_real_three_node` site-replication test.
+#   * 1 `_real_single_node` service-account round-trip test.
+#
+# The set is defined as "everything in replication_extension_test that is NOT
+# in the e2e-smoke PR allowlist above" (the negated clause is byte-identical to
+# the allowlist), so a newly added replication test automatically runs here
+# until it is explicitly promoted to the fast PR subset — no replication test
+# is ever silently left out of CI.
+#
+# #[serial] does NOT serialize under nextest (process-per-test; see the file
+# header). These tests need no cross-test serialization: each spawns its own
+# server(s) on random ports with isolated temp dirs, so they are parallel-safe
+# by construction — the same property the e2e-smoke subset relies on. If load
+# on the runner surfaces a real flake, quarantine the specific test with an
+# OPEN issue link (ci-10 / backlog#937 policy), never blanket-retry or exclude.
+#
+# Wired by .github/workflows/e2e-replication-nightly.yml (schedule +
+# workflow_dispatch), which builds the rustfs binary once, installs awscurl so
+# the STS dual-node test actually exercises its path (it skips gracefully with
+# a visible log line when awscurl is absent), and routes scheduled failures
+# through .github/actions/schedule-failure-issue (ci-8). Explicit division of
+# labor with ci-5's future e2e-full merge gate: these tests run ONLY here, not
+# double-run there. TODO(ci-7): fold this interim repl-owned lane into the ci
+# domain's consolidated scheduled e2e workflow once it exists.
+[profile.e2e-repl-nightly]
+default-filter = """
+    package(e2e_test)
+    & test(/^replication_extension_test::/)
+    & !test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
+"""
+fail-fast = false
+
+[profile.e2e-repl-nightly.junit]
+# Emitted to target/nextest/e2e-repl-nightly/junit.xml; uploaded by the nightly
+# workflow as the failure-triage artifact.
+path = "junit.xml"
+
+# ---------------------------------------------------------------------------
+# e2e-full profile — merge-gate full single-node e2e lane (backlog#1149 ci-5)
+# ---------------------------------------------------------------------------
+# The merge gate (ci.yml `e2e-full` job: push main + merge_group +
+# workflow_dispatch). Runs the never-automated user-visible suites — KMS (40),
+# object_lock (33), multipart_auth (109), quota, checksum, encryption,
+# security-boundary, ... — that the fast PR `e2e-smoke` subset deliberately
+# skips. Budget <= 45 min; authority for the suite count is `cargo nextest list
+# --profile e2e-full` (see docs/testing/e2e-suite-inventory.md).
+#
+# The filter is "the whole e2e_test crate MINUS the sets owned by other lanes":
+#   * protocols:: — FTPS/SFTP/WebDAV, still pinned to --test-threads=1 by fixed
+#     ports; they join a scheduled lane once ci-6 randomises the ports (ci-7).
+#   * the 7 cluster suites that spin up a RustFSTestClusterEnvironment
+#     (cluster_concurrency, cluster_multidrive_pool, stale_multipart_cleanup_cluster,
+#     namespace_lock_quorum, heal_erasure_disk_rebuild, admin_timeout_regression,
+#     object_lambda) — too heavy for the merge budget; they run in ci-7's
+#     nightly 4-node lane.
+#   * replication_extension_test — repl-1 already splits it into the PR
+#     `e2e-smoke` (20 fast) and `e2e-repl-nightly` (27 slow) lanes and reserves
+#     it for those, so e2e-full does not double-run it.
+#   * #[ignore]d tests — nextest skips them by default (no --run-ignored); the
+#     manual-localhost:9000 reliant/policy tests are ci-13's migration.
+#
+# Each e2e test spawns its own single-node rustfs server on a random port with
+# an isolated temp dir (crates/e2e_test/src/common.rs), so the set is
+# parallel-safe — the same property e2e-smoke relies on. The exception is the
+# 4-disk reliability / degraded-read fault-injection tests, serialized below
+# (identical to the ci profile) so several 4-disk servers never run at once.
+# KNOWN-FAILURE EXCLUSIONS (characterization run 29381309848, 2026-07-15:
+# 341 ran / 32 failed on the suites' first automated run ever). Deterministic
+# product failures cannot be quarantined away with retries, so each family is
+# excluded here with its tracking issue, under the same discipline as the
+# ci-profile quarantine (docs/testing/README.md): every entry MUST cite one
+# OPEN issue, and the fixing PR MUST delete the exclusion. The passing
+# negative-path siblings of each family stay in as regression guards.
+#   * rustfs#4843 — over-limit archive entry paths hard-reject the whole
+#     archive even under ignore-errors semantics.
+#   * rustfs#4846 — distributed-lock quorum tests misclassify as timeout
+#     under parallel load (multi-node in-process clusters; natural home is
+#     ci-7's nightly cluster lane).
+[profile.e2e-full]
+default-filter = """
+    package(e2e_test)
+    & !test(/^protocols::/)
+    & !test(/^(admin_timeout_regression_test|cluster_concurrency_test|cluster_multidrive_pool_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
+    & !test(/^replication_extension_test::/)
+    & !test(/^multipart_auth_test::test_signed_put_object_extract_skips_invalid_entry_when_ignore_errors_enabled$/)
+    & !test(/^snowball_auto_extract_test::tests::snowball_auto_extract_(ignores_invalid_entries_when_requested|supports_standard_headers_with_combined_extract_options)$/)
+    & !test(/^reliant::lock::test_distributed_lock_(2_nodes_grpc_read_survives_failed_node|4_nodes_grpc_read_write_quorum_split_with_two_failed_nodes)$/)
+"""
+fail-fast = false
+
+[profile.e2e-full.junit]
+# Emitted to target/nextest/e2e-full/junit.xml; uploaded by the e2e-full job.
+path = "junit.xml"
+
+# Serialize the 4-disk reliability / degraded-read e2e tests under e2e-full too
+# (see the e2e-reliability test-group note near the top of this file). Not a
+# quarantine: no retries, just single-threaded so several 4-disk servers never
+# run concurrently.
+[[profile.e2e-full.overrides]]
+filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
+test-group = 'e2e-reliability'
@@ -0,0 +1,12 @@
+# Committed floor for the number of security negative-auth tests selected by the
+# e2e-smoke PR profile (see scripts/check_security_smoke_count.sh, backlog#1151
+# sec-5).
+#
+# The floor equals the exact count of e2e_test cases whose name starts with a
+# security module prefix (negative_sigv4_test, presigned_negative_test,
+# admin_auth_test) that the [profile.e2e-smoke] default-filter in
+# .config/nextest.toml selects, at the time this file was last updated. CI fails
+# if the selected count drops below this number, so a rename or removal that
+# thins the security smoke gate must update this file in the same PR.
+# Adding tests does not require a bump, but bumping keeps the guard tight.
+16
@@ -36,6 +36,64 @@ To start a 4-node cluster for distributed testing:
 docker compose -f .docker/compose/docker-compose.cluster.yaml up -d
 ```

+### Script-Based 4-Node Validation (Recommended)
+
+Use the local validation script when you need local-source image build, failover checks,
+and benchmark workflow in one command:
+
+```bash
+# Default mode: WAIT_PROBE_MODE=service
+# This avoids false negatives where /health/ready remains 503 locally
+# while the service path is already available.
+./scripts/run_four_node_cluster_failover_bench.sh
+```
+
+Strict mode is available when you explicitly want `/health/ready == 200` as the gate:
+
+```bash
+WAIT_PROBE_MODE=ready ./scripts/run_four_node_cluster_failover_bench.sh
+```
+
+### Profiling + Trace Validation
+
+The profiling-focused 4-node compose keeps profiling enabled and points RustFS
+to an OTLP/HTTP collector endpoint:
+
+```bash
+docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
+```
+
+Important behavior notes:
+
+- `RUSTFS_OBS_ENDPOINT` is the OTLP/HTTP base URL. RustFS automatically sends
+  traces to `/v1/traces`, metrics to `/v1/metrics`, and logs to `/v1/logs`.
+- Startup usually produces logs and metrics first. That does not guarantee
+  visible traces yet.
+- Trace data becomes obvious only after real HTTP/S3/gRPC requests hit RustFS.
+- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
+  many nested `debug` spans. If Tempo/Jaeger looks sparse, retry with
+  `RUSTFS_OBS_LOGGER_LEVEL=debug` before suspecting the collector.
+
+Minimal trace verification flow:
+
+```bash
+# 1. Start the profiling compose with richer span visibility.
+RUSTFS_OBS_LOGGER_LEVEL=debug \
+docker compose -f .docker/compose/docker-compose.cluster.local-build.profiling-amd64.yml up -d
+
+# 2. Generate real request traffic after startup.
+curl -I http://127.0.0.1:9000/health
+curl -I http://127.0.0.1:9000/health/ready
+
+# 3. Then inspect Tempo or Jaeger.
+# Grafana: http://localhost:3000
+# Jaeger:  http://localhost:16686
+```
+
+If logs and metrics are present but traces are sparse, the most common cause is
+"no real request traffic yet" or "`info` level filtered nested spans", not an
+OTLP routing failure.
+
 ### (Deprecated) Minimal Observability

 ```bash
@@ -0,0 +1,236 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Profiling-first 4-node local-build compose.
+#
+# Goals:
+# - force linux/amd64 runtime/build on Apple Silicon hosts;
+# - enable RustFS built-in CPU profiling;
+# - keep all tuning knobs host-overridable via env.
+#
+# Observability notes:
+# - `RUSTFS_OBS_ENDPOINT` is the OTLP/HTTP base URL. RustFS appends
+#   `/v1/traces`, `/v1/metrics`, and `/v1/logs` automatically.
+# - Logs and metrics usually appear during startup. Traces mainly appear after
+#   real HTTP/S3/gRPC requests create spans.
+# - `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request trace span but
+#   filters many `debug`-level nested spans. Use `debug` when validating trace
+#   richness rather than collector reachability.
+
+services:
+  node1:
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node1
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
+      # should show richer nested spans during request-path verification.
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
+      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
+      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
+      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
+      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
+      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
+      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node1_data_0:/data/rustfs0
+      - node1_data_1:/data/rustfs1
+      - node1_data_2:/data/rustfs2
+      - node1_data_3:/data/rustfs3
+    ports:
+      - "9000:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node2:
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node2
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
+      # should show richer nested spans during request-path verification.
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
+      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
+      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
+      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
+      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
+      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
+      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node2_data_0:/data/rustfs0
+      - node2_data_1:/data/rustfs1
+      - node2_data_2:/data/rustfs2
+      - node2_data_3:/data/rustfs3
+    ports:
+      - "9001:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node3:
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node3
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
+      # should show richer nested spans during request-path verification.
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
+      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
+      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
+      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
+      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
+      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
+      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node3_data_0:/data/rustfs0
+      - node3_data_1:/data/rustfs1
+      - node3_data_2:/data/rustfs2
+      - node3_data_3:/data/rustfs3
+    ports:
+      - "9002:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node4:
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/amd64}
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node4
+    environment:
+      - RUSTFS_VOLUMES=http://node{1...4}:9000/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfs-cluster-admin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfs-cluster-secret}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      # `info` is enough for startup logs/metrics. Use `debug` if Tempo/Jaeger
+      # should show richer nested spans during request-path verification.
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_PROFILING_ENDPOINT=${RUSTFS_OBS_PROFILING_ENDPOINT:-http://host.docker.internal:4040}
+      - RUSTFS_OBS_PROFILING_EXPORT_ENABLED=${RUSTFS_OBS_PROFILING_EXPORT_ENABLED:-true}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+      - RUSTFS_ENABLE_PROFILING=${RUSTFS_ENABLE_PROFILING:-true}
+      - RUSTFS_PROF_CPU_MODE=${RUSTFS_PROF_CPU_MODE:-continuous}
+      - RUSTFS_PROF_CPU_FREQ=${RUSTFS_PROF_CPU_FREQ:-99}
+      - RUSTFS_PROF_OUTPUT_DIR=${RUSTFS_PROF_OUTPUT_DIR:-/tmp/rustfs-profiles}
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=${RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS:-48}
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=${RUSTFS_OBJECT_IO_BUFFER_SIZE:-262144}
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD:-6}
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=${RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD:-12}
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=${RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY:-8}
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=${RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE:-8388608}
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=${RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES:-25165824}
+      - RUSTFS_RUNTIME_WORKER_THREADS=${RUSTFS_RUNTIME_WORKER_THREADS:-12}
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=${RUSTFS_RUNTIME_MAX_BLOCKING_THREADS:-512}
+      - RUSTFS_ALLOCATOR_RECLAIM_ENABLED=${RUSTFS_ALLOCATOR_RECLAIM_ENABLED:-false}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node4_data_0:/data/rustfs0
+      - node4_data_1:/data/rustfs1
+      - node4_data_2:/data/rustfs2
+      - node4_data_3:/data/rustfs3
+    ports:
+      - "9003:9000"
+    networks:
+      - rustfs-cluster-net
+
+volumes:
+  node1_data_0:
+  node1_data_1:
+  node1_data_2:
+  node1_data_3:
+  node2_data_0:
+  node2_data_1:
+  node2_data_2:
+  node2_data_3:
+  node3_data_0:
+  node3_data_1:
+  node3_data_2:
+  node3_data_3:
+  node4_data_0:
+  node4_data_1:
+  node4_data_2:
+  node4_data_3:
+
+networks:
+  rustfs-cluster-net:
+    driver: bridge
@@ -0,0 +1,220 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+services:
+  node1:
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node1
+    environment:
+      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
+      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
+      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
+      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+      # Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
+      # defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
+      - RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
+      - RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
+      - RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
+      - RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
+      - RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
+      - RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
+      - RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
+      - RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
+      - RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
+      - RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
+      - RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
+      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node1_data_0:/data/rustfs0
+      - node1_data_1:/data/rustfs1
+      - node1_data_2:/data/rustfs2
+      - node1_data_3:/data/rustfs3
+    ports:
+      - "9000:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node2:
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node2
+    environment:
+      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
+      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
+      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
+      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+      # Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
+      # defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
+      - RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
+      - RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
+      - RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
+      - RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
+      - RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
+      - RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
+      - RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
+      - RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
+      - RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
+      - RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
+      - RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
+      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node2_data_0:/data/rustfs0
+      - node2_data_1:/data/rustfs1
+      - node2_data_2:/data/rustfs2
+      - node2_data_3:/data/rustfs3
+    ports:
+      - "9001:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node3:
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node3
+    environment:
+      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
+      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
+      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
+      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+      # Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
+      # defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
+      - RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
+      - RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
+      - RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
+      - RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
+      - RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
+      - RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
+      - RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
+      - RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
+      - RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
+      - RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
+      - RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
+      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node3_data_0:/data/rustfs0
+      - node3_data_1:/data/rustfs1
+      - node3_data_2:/data/rustfs2
+      - node3_data_3:/data/rustfs3
+    ports:
+      - "9002:9000"
+    networks:
+      - rustfs-cluster-net
+
+  node4:
+    image: ${RUSTFS_IMAGE:-rustfs/rustfs:local-4node}
+    build:
+      context: ../..
+      dockerfile: Dockerfile.source
+    hostname: node4
+    environment:
+      - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
+      - RUSTFS_ADDRESS=:9000
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
+      - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
+      - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
+      - RUSTFS_OBS_LOG_STDOUT_ENABLED=${RUSTFS_OBS_LOG_STDOUT_ENABLED:-false}
+      - RUSTFS_ISSUE3031_DIAG_ENABLE=${RUSTFS_ISSUE3031_DIAG_ENABLE:-false}
+      - RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_OBJECT_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_LOCK_ACQUIRE_TIMEOUT=${RUSTFS_LOCK_ACQUIRE_TIMEOUT:-5}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+      # Internode gRPC optimization knobs (grpc-optimization P0-P3). Defaults match the binary
+      # defaults, so leaving these unset is a no-op; the A/B driver exports them to toggle a stage.
+      - RUSTFS_INTERNODE_RPC_TCP_NODELAY=${RUSTFS_INTERNODE_RPC_TCP_NODELAY:-true}
+      - RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_STREAM_WINDOW_SIZE:-1048576}
+      - RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE=${RUSTFS_INTERNODE_RPC_HTTP2_CONN_WINDOW_SIZE:-2097152}
+      - RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE=${RUSTFS_INTERNODE_RPC_MAX_MESSAGE_SIZE:-104857600}
+      - RUSTFS_INTERNODE_CHANNEL_ISOLATION=${RUSTFS_INTERNODE_CHANNEL_ISOLATION:-false}
+      - RUSTFS_INTERNODE_BULK_CHANNELS=${RUSTFS_INTERNODE_BULK_CHANNELS:-2}
+      - RUSTFS_INTERNODE_RPC_MSGPACK_ONLY=${RUSTFS_INTERNODE_RPC_MSGPACK_ONLY:-false}
+      - RUSTFS_INTERNODE_PREWARM=${RUSTFS_INTERNODE_PREWARM:-false}
+      - RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
+      - RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
+      - RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
+      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
+    extra_hosts:
+      - "host.docker.internal:host-gateway"
+    volumes:
+      - node4_data_0:/data/rustfs0
+      - node4_data_1:/data/rustfs1
+      - node4_data_2:/data/rustfs2
+      - node4_data_3:/data/rustfs3
+    ports:
+      - "9003:9000"
+    networks:
+      - rustfs-cluster-net
+
+volumes:
+  node1_data_0:
+  node1_data_1:
+  node1_data_2:
+  node1_data_3:
+  node2_data_0:
+  node2_data_1:
+  node2_data_2:
+  node2_data_3:
+  node3_data_0:
+  node3_data_1:
+  node3_data_2:
+  node3_data_3:
+  node4_data_0:
+  node4_data_1:
+  node4_data_2:
+  node4_data_3:
+
+networks:
+  rustfs-cluster-net:
+    driver: bridge
@@ -0,0 +1,56 @@
+services:
+  node1:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=16
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
+
+  node2:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=16
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
+
+  node3:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=16
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
+
+  node4:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=56
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=131072
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=10
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=20
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=10
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=16
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=768
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=420
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=50
@@ -0,0 +1,56 @@
+services:
+  node1:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
+      - RUSTFS_RUNTIME_WORKER_THREADS=24
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
+
+  node2:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
+      - RUSTFS_RUNTIME_WORKER_THREADS=24
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
+
+  node3:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
+      - RUSTFS_RUNTIME_WORKER_THREADS=24
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
+
+  node4:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=96
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=524288
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=32
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=24
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=33554432
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=50331648
+      - RUSTFS_RUNTIME_WORKER_THREADS=24
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1536
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=600
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=80
@@ -0,0 +1,56 @@
+services:
+  node1:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
+      - RUSTFS_RUNTIME_WORKER_THREADS=20
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
+
+  node2:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
+      - RUSTFS_RUNTIME_WORKER_THREADS=20
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
+
+  node3:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
+      - RUSTFS_RUNTIME_WORKER_THREADS=20
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
+
+  node4:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=64
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=24
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=16
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=16777216
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=33554432
+      - RUSTFS_RUNTIME_WORKER_THREADS=20
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=1024
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=60
@@ -0,0 +1,56 @@
+services:
+  node1:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=12
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
+
+  node2:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=12
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
+
+  node3:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=12
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
+
+  node4:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=48
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=8
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=16
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=12
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=25165824
+      - RUSTFS_RUNTIME_WORKER_THREADS=12
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=512
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=40
@@ -0,0 +1,56 @@
+services:
+  node1:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
+      - RUSTFS_RUNTIME_WORKER_THREADS=6
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
+
+  node2:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
+      - RUSTFS_RUNTIME_WORKER_THREADS=6
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
+
+  node3:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
+      - RUSTFS_RUNTIME_WORKER_THREADS=6
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
+
+  node4:
+    environment:
+      - RUSTFS_OBJECT_MAX_CONCURRENT_DISK_READS=32
+      - RUSTFS_OBJECT_IO_BUFFER_SIZE=262144
+      - RUSTFS_OBJECT_MEDIUM_CONCURRENCY_THRESHOLD=6
+      - RUSTFS_OBJECT_HIGH_CONCURRENCY_THRESHOLD=12
+      - RUSTFS_OBJECT_IO_RANDOM_READAHEAD_DISABLE_CONCURRENCY=8
+      - RUSTFS_OBJECT_DUPLEX_BUFFER_SIZE=8388608
+      - RUSTFS_ERASURE_ENCODE_MAX_INFLIGHT_BYTES=16777216
+      - RUSTFS_RUNTIME_WORKER_THREADS=6
+      - RUSTFS_RUNTIME_MAX_BLOCKING_THREADS=256
+      - RUSTFS_CAPACITY_SCHEDULED_INTERVAL=300
+      - RUSTFS_CAPACITY_WRITE_FREQUENCY_THRESHOLD=30
@@ -21,8 +21,8 @@ services:
      - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
      - RUSTFS_ADDRESS=0.0.0.0:9000
      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfsadmin
-      - RUSTFS_SECRET_KEY=rustfsadmin
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
    platform: linux/amd64
    ports:
      - "9000:9000" # Map port 9001 of the host to port 9000 of the container
@@ -38,8 +38,8 @@ services:
      - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
      - RUSTFS_ADDRESS=0.0.0.0:9000
      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfsadmin
-      - RUSTFS_SECRET_KEY=rustfsadmin
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
    platform: linux/amd64
    ports:
      - "9001:9000" # Map port 9002 of the host to port 9000 of the container
@@ -55,8 +55,8 @@ services:
      - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
      - RUSTFS_ADDRESS=0.0.0.0:9000
      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfsadmin
-      - RUSTFS_SECRET_KEY=rustfsadmin
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
    platform: linux/amd64
    ports:
      - "9002:9000" # Map port 9003 of the host to port 9000 of the container
@@ -72,8 +72,8 @@ services:
      - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
      - RUSTFS_ADDRESS=0.0.0.0:9000
      - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=rustfsadmin
-      - RUSTFS_SECRET_KEY=rustfsadmin
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
    platform: linux/amd64
    ports:
      - "9003:9000" # Map port 9004 of the host to port 9000 of the container
@@ -26,7 +26,7 @@ services:
    restart: "no"

  tempo:
-    image: grafana/tempo:latest
+    image: grafana/tempo:2.10.5
    user: "10001"
    command: [ "-config.file=/etc/tempo.yaml" ]
    volumes:
@@ -36,9 +36,19 @@ services:
      - "3200:3200" # tempo
      - "4317"      # otlp grpc
      - "4318"      # otlp http
+      - "7946"      # memberlist
    restart: unless-stopped
    networks:
      - rustfs-network
+    depends_on:
+      tempo-init:
+        condition: service_completed_successfully
+    healthcheck:
+      test: [ "CMD", "/tempo", "-version" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 15s

  otel-collector:
    image: otel/opentelemetry-collector-contrib:latest
@@ -61,6 +71,12 @@ services:
      - jaeger
      - prometheus
      - loki
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "/otelcol-contrib", "--version" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3

  jaeger:
    image: jaegertracing/jaeger:latest
@@ -72,12 +88,23 @@ services:
      - BADGER_DIRECTORY_KEY=/badger/key
      - COLLECTOR_OTLP_ENABLED=true
    volumes:
+      - ../../.docker/observability/jaeger.yaml:/etc/jaeger/config.yml:ro
      - jaeger-data:/badger
    ports:
      - "16686:16686" # Web UI
      - "14269:14269" # Admin/Metrics
+      - "4317"        # otlp grpc
+      - "4318"        # otlp http
+    command: [ "--config", "/etc/jaeger/config.yml" ]
    networks:
      - rustfs-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3
+      start_period: 15s

  prometheus:
    image: prom/prometheus:latest
@@ -85,6 +112,7 @@ services:
      - TZ=Asia/Shanghai
    volumes:
      - ../../.docker/observability/prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - ../../.docker/observability/prometheus-rules:/etc/prometheus/rules:ro
      - prometheus-data:/prometheus
    ports:
      - "9090:9090"
@@ -94,23 +122,45 @@ services:
      - '--web.enable-remote-write-receiver'
      - '--enable-feature=promql-experimental-functions'
      - '--storage.tsdb.path=/prometheus'
-      - '--web.console.libraries=/usr/share/prometheus/console_libraries'
-      - '--web.console.templates=/usr/share/prometheus/consoles'
+      - '--storage.tsdb.retention.time=30d'
    networks:
      - rustfs-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:9090/-/healthy" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3

  loki:
    image: grafana/loki:latest
    environment:
      - TZ=Asia/Shanghai
    volumes:
-      - ../../.docker/observability/loki.yaml:/etc/loki/local-config.yaml:ro
+      - ../../.docker/observability/loki.yaml:/etc/loki/loki.yaml:ro
      - loki-data:/loki
    ports:
      - "3100:3100"
-    command: -config.file=/etc/loki/local-config.yaml
+    command: -config.file=/etc/loki/loki.yaml
    networks:
      - rustfs-network
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
+      interval: 15s
+      timeout: 10s
+      retries: 5
+      start_period: 60s
+
+  pyroscope:
+    image: grafana/pyroscope:latest
+    ports:
+      - "4040:4040"
+    command:
+      - -self-profiling.disable-push=true
+    networks:
+      - rustfs-network
+    restart: unless-stopped

  grafana:
    image: grafana/grafana:latest
@@ -127,10 +177,17 @@ services:
    volumes:
      - ../../.docker/observability/grafana/provisioning:/etc/grafana/provisioning:ro
      - ../../.docker/observability/grafana/dashboards:/var/lib/grafana/dashboards:ro
+      - grafana-data:/var/lib/grafana
    depends_on:
      - prometheus
      - tempo
      - loki
+    restart: unless-stopped
+    healthcheck:
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3000/api/health" ]
+      interval: 10s
+      timeout: 5s
+      retries: 3

  # --- RustFS Cluster ---

@@ -215,6 +272,7 @@ volumes:
  tempo-data:
  loki-data:
  jaeger-data:
+  grafana-data:

 networks:
  rustfs-network:
@@ -74,6 +74,17 @@ http {
    #     ssl_certificate /etc/nginx/ssl/server.crt;
    #     ssl_certificate_key /etc/nginx/ssl/server.key;
    #
+    #     # Restrict to modern TLS versions and ciphers. Operators copying this
+    #     # example must keep at least these directives — without them, nginx
+    #     # may negotiate older protocol versions that have known weaknesses.
+    #     ssl_protocols TLSv1.2 TLSv1.3;
+    #     ssl_prefer_server_ciphers on;
+    #     ssl_ciphers 'ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-CHACHA20-POLY1305:ECDHE-RSA-CHACHA20-POLY1305';
+    #     ssl_session_timeout 1d;
+    #     ssl_session_cache shared:SSL:10m;
+    #     ssl_session_tickets off;
+    #     # add_header Strict-Transport-Security "max-age=63072000" always;
+    #
    #     location / {
    #         proxy_pass http://rustfs:9000;
    #         ...
@@ -13,6 +13,9 @@ The stack is composed of the following best-in-class open-source components:
 - **Jaeger** (v1.59.0): Distributed tracing system (configured as a secondary UI/storage).
 - **OpenTelemetry Collector** (v0.104.0): A vendor-agnostic implementation for receiving, processing, and exporting telemetry data.

+By default, this stack uses Tempo in single-binary mode and does not require Kafka/Redpanda.
+If you want the Kafka-backed HA Tempo path, use `docker-compose-example-for-rustfs.yml` together with `docker-compose-tempo-ha-override.yml`.
+
 ## Architecture

 1. **Telemetry Collection**: Applications send OTLP (OpenTelemetry Protocol) data (Metrics, Logs, Traces) to the **OpenTelemetry Collector**.
@@ -31,6 +34,57 @@ The stack is composed of the following best-in-class open-source components:
 - **High Performance**: Optimized configurations for batching, compression, and memory management.
 - **Standardized Protocols**: Built entirely on OpenTelemetry standards.

+## GET Performance Optimization Dashboards
+
+Three pre-built Grafana dashboards are included for monitoring RustFS GET performance optimization rollout:
+
+### Available Dashboards
+
+| Dashboard | File | Description |
+|-----------|------|-------------|
+| **GET Rollout Health** | `grafana-get-rollout-health.json` | Monitors optimization rollout: latency by reader path, early-stop hit rate, codec streaming usage, pipeline failures |
+| **GET Data Integrity** | `grafana-get-data-integrity.json` | Monitors data safety: bitrot verify failures, decode errors, short reads, shard read outcomes |
+| **GET Resource Impact** | `grafana-get-resource-impact.json` | Monitors resource usage: concurrent requests, IO queue utilization, disk permit wait, RSS trend |
+| **Object Data Cache** | `grafana-object-data-cache.json` | Monitors the GET body cache (`rustfs_object_data_cache_*`): hit ratio, lookup/plan/fill outcomes, fill duration quantiles, hit vs fill throughput, entries/weighted bytes, inflight fills, memory-pressure skips, invalidations, and size-class breakdowns |
+
+### Prometheus Alert Rules
+
+The file `prometheus-rules/rustfs-get-optimization-alerts.yaml` contains pre-configured alerting rules:
+
+| Alert | Severity | Condition |
+|-------|----------|-----------|
+| `GetP99Regression` | Critical | GET p99 latency > 2x baseline for 10m |
+| `PipelineFailureSpike` | Critical | Pipeline failure rate > 5x baseline for 5m |
+| `BitrotMismatchSpike` | Critical | Bitrot mismatch rate > 3x baseline for 5m |
+| `EarlyStopInsufficientQuorum` | Warning | Early-stop insufficient quorum rate > 0.1/s for 5m |
+| `CodecStreamingFallbackSpike` | Warning | Codec streaming fallback > 10x baseline for 10m |
+| `IoQueueSaturation` | Warning | IO queue utilization > 90% for 5m |
+
+### Enabling Alert Rules
+
+Add the alert rules file to your Prometheus configuration:
+
+```yaml
+# prometheus.yml
+rule_files:
+  - "/etc/prometheus/rules/*.yml"
+
+# Or mount the file in docker-compose.yml:
+# volumes:
+#   - ./prometheus-rules:/etc/prometheus/rules
+```
+
+### Dashboard Usage
+
+The dashboards are automatically provisioned when Grafana starts. They use the `${DS_PROMETHEUS}` datasource variable, so you need a Prometheus datasource configured in Grafana.
+
+Key panels to monitor during optimization rollout:
+
+1. **GET Latency by Reader Path** - Compare `codec_streaming` vs `legacy_duplex` latency
+2. **Early-Stop Hit Rate** - Verify early-stop is triggering effectively
+3. **Pipeline Failure Rate** - Detect any new failure modes introduced by optimizations
+4. **Bitrot Verify Failures** - Ensure data integrity is maintained
+
 ## Quick Start

 ### Prerequisites
@@ -46,6 +100,15 @@ Run the following command to start the entire stack:
 docker compose up -d
 ```

+### High Availability Tempo
+
+The default `docker-compose.yml` is the single-node stack.
+If you need the Kafka-backed HA Tempo configuration, start it with:
+
+```bash
+docker compose -f docker-compose-example-for-rustfs.yml -f docker-compose-tempo-ha-override.yml up -d
+```
+
 ### Access Dashboards

 | Service        | URL                                              | Credentials       | Description                    |
@@ -78,6 +141,54 @@ docker compose down -v
 - **Grafana**: Dashboards and datasources are provisioned from the `grafana/` directory.
 - **Collector**: Edit `otel-collector-config.yaml` to modify pipelines, processors, or exporters.

+### Verifying RustFS Traces
+
+When RustFS points `RUSTFS_OBS_ENDPOINT` at this stack, treat the value as the
+OTLP/HTTP base URL, for example:
+
+```bash
+export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
+```
+
+RustFS automatically expands that base URL to:
+
+- `/v1/traces`
+- `/v1/metrics`
+- `/v1/logs`
+
+Important behavior notes:
+
+- Logs and metrics usually appear during startup, so seeing those two signals
+  first is expected.
+- Visible trace data usually requires real HTTP/S3/gRPC request traffic after
+  startup, because request-path spans are created on demand.
+- `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
+  many nested `debug` spans. If Tempo or Jaeger looks sparse, retry with
+  `RUSTFS_OBS_LOGGER_LEVEL=debug` before suspecting collector or Tempo issues.
+
+Minimal validation flow:
+
+```bash
+# 1. Start this observability stack.
+docker compose up -d
+
+# 2. Start RustFS with OTLP/HTTP export and richer span visibility.
+export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
+export RUSTFS_OBS_LOGGER_LEVEL=debug
+
+# 3. Generate real request traffic.
+curl -I http://127.0.0.1:9000/health
+curl -I http://127.0.0.1:9000/health/ready
+
+# 4. Inspect Grafana or Jaeger.
+# Grafana: http://localhost:3000
+# Jaeger:  http://localhost:16686
+```
+
+If logs and metrics are present but traces are sparse, the most common cause is
+"no real request traffic yet" or "`info` level filtered nested spans", not an
+OTLP routing failure.
+
 ## Troubleshooting

 - **Service Health**: Check the health of services using `docker compose ps`.
@@ -13,6 +13,9 @@
 - **Jaeger** (v1.59.0): 分布式追踪系统（配置为辅助 UI/存储）。
 - **OpenTelemetry Collector** (v0.104.0): 接收、处理和导出遥测数据的供应商无关实现。

+默认情况下，这套技术栈使用 Tempo 单二进制模式，不依赖 Kafka/Redpanda。
+如果需要基于 Kafka 的 HA Tempo 路径，请使用 `docker-compose-example-for-rustfs.yml` 配合 `docker-compose-tempo-ha-override.yml`。
+
 ## 架构

 1.  **遥测收集**: 应用程序将 OTLP (OpenTelemetry Protocol) 数据（指标、日志、追踪）发送到 **OpenTelemetry Collector**。
@@ -31,6 +34,57 @@
 -   **高性能**: 针对批处理、压缩和内存管理进行了优化配置。
 -   **标准化协议**: 完全基于 OpenTelemetry 标准构建。

+## GET 性能优化仪表盘
+
+包含三个预构建的 Grafana 仪表盘，用于监控 RustFS GET 性能优化发布：
+
+### 可用仪表盘
+
+| 仪表盘 | 文件 | 描述 |
+|--------|------|------|
+| **GET 发布健康度** | `grafana-get-rollout-health.json` | 监控优化发布：按 reader path 的延迟、early-stop 命中率、codec streaming 使用率、pipeline 失败率 |
+| **GET 数据完整性** | `grafana-get-data-integrity.json` | 监控数据安全：bitrot 校验失败、decode 错误、short read、shard 读取结果 |
+| **GET 资源影响** | `grafana-get-resource-impact.json` | 监控资源使用：并发请求数、IO 队列利用率、disk permit 等待、RSS 趋势 |
+| **对象数据缓存** | `grafana-object-data-cache.json` | 监控 GET body 缓存（`rustfs_object_data_cache_*`）：命中率、查找/规划/填充结果、填充耗时分位、命中 vs 填充吞吐、条目数/加权字节、在途填充、内存压力拒绝、失效、按尺寸档拆分 |
+
+### Prometheus 告警规则
+
+文件 `prometheus-rules/rustfs-get-optimization-alerts.yaml` 包含预配置的告警规则：
+
+| 告警 | 级别 | 条件 |
+|------|------|------|
+| `GetP99Regression` | 严重 | GET p99 延迟 > 2x 基线，持续 10 分钟 |
+| `PipelineFailureSpike` | 严重 | Pipeline 失败率 > 5x 基线，持续 5 分钟 |
+| `BitrotMismatchSpike` | 严重 | Bitrot 不匹配率 > 3x 基线，持续 5 分钟 |
+| `EarlyStopInsufficientQuorum` | 警告 | Early-stop quorum 不足率 > 0.1/s，持续 5 分钟 |
+| `CodecStreamingFallbackSpike` | 警告 | Codec streaming 回退 > 10x 基线，持续 10 分钟 |
+| `IoQueueSaturation` | 警告 | IO 队列利用率 > 90%，持续 5 分钟 |
+
+### 启用告警规则
+
+在 Prometheus 配置中添加告警规则文件：
+
+```yaml
+# prometheus.yml
+rule_files:
+  - "/etc/prometheus/rules/*.yml"
+
+# 或在 docker-compose.yml 中挂载文件：
+# volumes:
+#   - ./prometheus-rules:/etc/prometheus/rules
+```
+
+### 仪表盘使用
+
+仪表盘在 Grafana 启动时自动预置。它们使用 `${DS_PROMETHEUS}` 数据源变量，因此需要在 Grafana 中配置 Prometheus 数据源。
+
+优化发布期间需要关注的关键面板：
+
+1. **GET 延迟按 Reader Path** - 对比 `codec_streaming` vs `legacy_duplex` 延迟
+2. **Early-Stop 命中率** - 验证 early-stop 是否有效触发
+3. **Pipeline 失败率** - 检测优化引入的新故障模式
+4. **Bitrot 校验失败** - 确保数据完整性
+
 ## 快速开始

 ### 前置条件
@@ -46,6 +100,15 @@
 docker compose up -d
 ```

+### Tempo 高可用模式
+
+默认的 `docker-compose.yml` 对应单机栈。
+如果需要基于 Kafka 的 HA Tempo 配置，请使用：
+
+```bash
+docker compose -f docker-compose-example-for-rustfs.yml -f docker-compose-tempo-ha-override.yml up -d
+```
+
 ### 访问仪表盘

 | 服务 | URL | 凭据 | 描述 |
@@ -78,6 +141,50 @@ docker compose down -v
 -   **Grafana**: 仪表盘和数据源从 `grafana/` 目录预置。
 -   **Collector**: 编辑 `otel-collector-config.yaml` 以修改管道、处理器或导出器。

+### 验证 RustFS Trace
+
+当 RustFS 将 `RUSTFS_OBS_ENDPOINT` 指向这套技术栈时，应将该值视为
+OTLP/HTTP 的基础 URL，例如：
+
+```bash
+export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
+```
+
+RustFS 会自动在该基础 URL 后补全：
+
+- `/v1/traces`
+- `/v1/metrics`
+- `/v1/logs`
+
+需要注意：
+
+- 启动阶段通常会先看到日志和指标，因此“先有日志/指标、后有 trace”是正常现象。
+- 可见的 trace 数据通常依赖启动后的真实 HTTP/S3/gRPC 请求流量，因为请求路径上的 span 是按需创建的。
+- `RUSTFS_OBS_LOGGER_LEVEL=info` 会保留顶层请求 span，但会过滤掉很多 `debug` 级别的嵌套 span。
+  如果 Tempo 或 Jaeger 中的 trace 看起来很稀疏，建议先改成 `RUSTFS_OBS_LOGGER_LEVEL=debug`，再判断是否是 collector 或 Tempo 问题。
+
+最小验证流程：
+
+```bash
+# 1. 启动本目录下的可观测性技术栈。
+docker compose up -d
+
+# 2. 以 OTLP/HTTP 导出方式启动 RustFS，并提高 span 可见性。
+export RUSTFS_OBS_ENDPOINT=http://host.docker.internal:4318
+export RUSTFS_OBS_LOGGER_LEVEL=debug
+
+# 3. 产生真实请求流量。
+curl -I http://127.0.0.1:9000/health
+curl -I http://127.0.0.1:9000/health/ready
+
+# 4. 到 Grafana 或 Jaeger 中检查。
+# Grafana: http://localhost:3000
+# Jaeger:  http://localhost:16686
+```
+
+如果日志和指标已经正常，但 trace 仍然稀疏，最常见的原因通常是
+“还没有真实请求流量”或“`info` 级别过滤了嵌套 span”，而不是 OTLP 路由失败。
+
 ## 故障排除

 -   **服务健康**: 使用 `docker compose ps` 检查服务健康状况。
@@ -85,10 +85,8 @@ services:
    networks:
      - otel-network
    restart: unless-stopped
-    depends_on:
-      - redpanda
    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
+      test: [ "CMD", "/tempo", "-version" ]
      interval: 10s
      timeout: 5s
      retries: 3
@@ -205,7 +203,7 @@ services:
      - prometheus
      - loki
    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:13133" ]
+      test: [ "CMD", "/otelcol-contrib", "--version" ]
      interval: 10s
      timeout: 5s
      retries: 3
@@ -16,10 +16,8 @@ services:

  # --- Tracing ---
  tempo:
-    image: grafana/tempo:2.10.3
+    image: grafana/tempo:2.10.5
    container_name: tempo
-    depends_on:
-      - redpanda
    command: [ "-config.file=/etc/tempo.yaml" ]
    volumes:
      - ./tempo.yaml:/etc/tempo.yaml:ro
@@ -33,31 +31,11 @@ services:
      - otel-network
    restart: unless-stopped
    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3200/ready" ]
+      test: [ "CMD", "/tempo", "-version" ]
      interval: 10s
      timeout: 5s
      retries: 3
      start_period: 15s
-  redpanda:
-    image: redpandadata/redpanda:latest
-    ports:
-      - "9092:9092" # Kafka API for clients
-    command: >
-      redpanda start --overprovisioned
-      --mode=dev-container
-      --kafka-addr=PLAINTEXT://0.0.0.0:9092
-      --advertise-kafka-addr=PLAINTEXT://redpanda:9092
-
-  redpanda-console:
-    image: docker.redpanda.com/redpandadata/console:latest
-    environment:
-      - CONFIG_FILEPATH=/etc/redpanda/redpanda-console-config.yaml
-    volumes:
-      - ./redpanda-console.yaml:/etc/redpanda/redpanda-console-config.yaml
-    ports:
-      - "8080:8080"
-    depends_on:
-      - redpanda

  vulture:
    image: grafana/tempo-vulture:latest
@@ -85,7 +63,7 @@ services:
      - jaeger-data:/badger
    ports:
      - "16686:16686" # Web UI
-      - "14269:14269" # Admin/Metrics
+      - "18888:8888" # Metrics
      - "4317" # otlp grpc
      - "4318" # otlp http
    command: [ "--config", "/etc/jaeger/config.yml" ]
@@ -93,7 +71,7 @@ services:
      - otel-network
    restart: unless-stopped
    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:14269" ]
+      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:8888/metrics" ]
      interval: 10s
      timeout: 5s
      retries: 3
@@ -106,6 +84,7 @@ services:
    container_name: prometheus
    volumes:
      - ./prometheus.yml:/etc/prometheus/prometheus.yml:ro
+      - ./prometheus-rules:/etc/prometheus/rules:ro
      - prometheus-data:/prometheus
    ports:
      - "9090:9090"
@@ -139,7 +118,7 @@ services:
      - otel-network
    restart: unless-stopped
    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:3100/ready" ]
+      test: [ "CMD", "/usr/bin/loki", "--version" ]
      interval: 15s
      timeout: 10s
      retries: 5
@@ -168,7 +147,7 @@ services:
      - prometheus
      - loki
    healthcheck:
-      test: [ "CMD", "wget", "--spider", "-q", "http://localhost:13133" ]
+      test: [ "CMD", "/otelcol-contrib", "--version" ]
      interval: 10s
      timeout: 5s
      retries: 3
@@ -0,0 +1,500 @@
+{
+  "annotations": {
+    "list": []
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "id": null,
+  "links": [],
+  "liveNow": false,
+  "panels": [
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "ops",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 20,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "line+area"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "transparent",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 0
+      },
+      "id": 1,
+      "options": {
+        "reduceOptions": {
+          "values": false,
+          "calcs": ["lastNotNull"],
+          "fields": ""
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        },
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom",
+          "calcs": ["mean", "max", "sum"]
+        }
+      },
+      "title": "Bitrot Verify Failures",
+      "type": "timeseries",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{stage=\"bitrot_verify\"}[$__rate_interval])) by (path, reason)",
+          "legendFormat": "bitrot_fail {{path}} / {{reason}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{reason=\"bitrot_verify_failed\"}[$__rate_interval])) by (path, stage)",
+          "legendFormat": "verify_failed {{path}} / {{stage}}",
+          "refId": "B"
+        }
+      ]
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 0
+      },
+      "id": 2,
+      "options": {
+        "reduceOptions": {
+          "values": false,
+          "calcs": ["lastNotNull"],
+          "fields": ""
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        },
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom",
+          "calcs": ["mean", "max"]
+        }
+      },
+      "title": "Bitrot Verify Duration (p50 / p95 / p99)",
+      "type": "timeseries",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "histogram_quantile(0.50, sum(rate(rustfs_io_get_object_shard_bitrot_verify_duration_seconds_bucket[$__rate_interval])) by (le))",
+          "legendFormat": "p50",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "histogram_quantile(0.95, sum(rate(rustfs_io_get_object_shard_bitrot_verify_duration_seconds_bucket[$__rate_interval])) by (le))",
+          "legendFormat": "p95",
+          "refId": "B"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "histogram_quantile(0.99, sum(rate(rustfs_io_get_object_shard_bitrot_verify_duration_seconds_bucket[$__rate_interval])) by (le))",
+          "legendFormat": "p99",
+          "refId": "C"
+        }
+      ]
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "ops",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 20,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "line+area"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "transparent",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 8
+      },
+      "id": 3,
+      "options": {
+        "reduceOptions": {
+          "values": false,
+          "calcs": ["lastNotNull"],
+          "fields": ""
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        },
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom",
+          "calcs": ["mean", "max", "sum"]
+        }
+      },
+      "title": "Decode Errors",
+      "type": "timeseries",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{reason=\"decode_error\"}[$__rate_interval])) by (path, stage)",
+          "legendFormat": "decode_error {{path}} / {{stage}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "ops",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 20,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "line+area"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "transparent",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 8
+      },
+      "id": 4,
+      "options": {
+        "reduceOptions": {
+          "values": false,
+          "calcs": ["lastNotNull"],
+          "fields": ""
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        },
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom",
+          "calcs": ["mean", "max", "sum"]
+        }
+      },
+      "title": "Short Reads",
+      "type": "timeseries",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total{reason=\"short_read\"}[$__rate_interval])) by (path, stage)",
+          "legendFormat": "short_read {{path}} / {{stage}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "ops",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 16
+      },
+      "id": 5,
+      "options": {
+        "reduceOptions": {
+          "values": false,
+          "calcs": ["lastNotNull"],
+          "fields": ""
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        },
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom",
+          "calcs": ["mean", "max"]
+        }
+      },
+      "title": "Shard Read Outcomes by Role",
+      "type": "timeseries",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(rustfs_io_get_object_shard_read_total[$__rate_interval])) by (path, role, outcome)",
+          "legendFormat": "{{path}} {{role}}/{{outcome}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "ops",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 16
+      },
+      "id": 6,
+      "options": {
+        "reduceOptions": {
+          "values": false,
+          "calcs": ["lastNotNull"],
+          "fields": ""
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        },
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom",
+          "calcs": ["mean", "max"]
+        }
+      },
+      "title": "Request Result Status Rate",
+      "type": "timeseries",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(rustfs_io_get_object_request_results_total[$__rate_interval])) by (status)",
+          "legendFormat": "{{status}}",
+          "refId": "A"
+        }
+      ]
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 38,
+  "style": "dark",
+  "tags": ["rustfs", "get-optimization", "data-integrity"],
+  "templating": {
+    "list": [
+      {
+        "current": {
+          "selected": false,
+          "text": "Prometheus",
+          "value": "Prometheus"
+        },
+        "hide": 0,
+        "includeAll": false,
+        "multi": false,
+        "name": "DS_PROMETHEUS",
+        "options": [],
+        "query": "prometheus",
+        "refresh": 1,
+        "regex": "",
+        "skipUrlSync": false,
+        "type": "datasource"
+      }
+    ]
+  },
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "timepicker": {},
+  "timezone": "",
+  "title": "RustFS GET Data Integrity",
+  "uid": "rustfs-get-data-integrity",
+  "version": 1
+}
@@ -0,0 +1,716 @@
+{
+  "annotations": {
+    "list": []
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "id": null,
+  "links": [],
+  "liveNow": false,
+  "panels": [
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "ops",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 0
+      },
+      "id": 1,
+      "options": {
+        "legend": {
+          "calcs": [
+            "mean",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_strategy_total[$__rate_interval])) or vector(0)",
+          "legendFormat": "{{strategy}} / {{mode}}",
+          "refId": "A"
+        }
+      ],
+      "title": "GET Reader Setup Strategy Rate",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "ops",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 0
+      },
+      "id": 2,
+      "options": {
+        "legend": {
+          "calcs": [
+            "mean",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_strategy_by_size_total[$__rate_interval])) or vector(0)",
+          "legendFormat": "{{path}} / {{strategy}} / {{size_bucket}}",
+          "refId": "A"
+        }
+      ],
+      "title": "GET Reader Setup Strategy by Size",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "none",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 8
+      },
+      "id": 3,
+      "options": {
+        "legend": {
+          "calcs": [
+            "mean",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_scheduled_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_scheduled_count[$__rate_interval])), 1) or vector(0)",
+          "legendFormat": "scheduled {{strategy}} / {{mode}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_attempted_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_attempted_count[$__rate_interval])), 1) or vector(0)",
+          "legendFormat": "attempted {{strategy}} / {{mode}}",
+          "refId": "B"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_ready_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_ready_count[$__rate_interval])), 1) or vector(0)",
+          "legendFormat": "ready {{strategy}} / {{mode}}",
+          "refId": "C"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_deferred_sum[$__rate_interval])) / clamp_min(sum by (strategy, mode) (rate(rustfs_io_get_object_reader_setup_deferred_count[$__rate_interval])), 1) or vector(0)",
+          "legendFormat": "deferred {{strategy}} / {{mode}}",
+          "refId": "D"
+        }
+      ],
+      "title": "GET Reader Setup Fanout Average",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "none",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 8
+      },
+      "id": 4,
+      "options": {
+        "legend": {
+          "calcs": [
+            "mean",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_scheduled_by_size_sum[$__rate_interval])) / clamp_min(sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_scheduled_by_size_count[$__rate_interval])), 1) or vector(0)",
+          "legendFormat": "scheduled {{path}} / {{strategy}} / {{size_bucket}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_ready_by_size_sum[$__rate_interval])) / clamp_min(sum by (path, strategy, size_bucket) (rate(rustfs_io_get_object_reader_setup_ready_by_size_count[$__rate_interval])), 1) or vector(0)",
+          "legendFormat": "ready {{path}} / {{strategy}} / {{size_bucket}}",
+          "refId": "B"
+        }
+      ],
+      "title": "GET Reader Setup Fanout by Size",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 16
+      },
+      "id": 5,
+      "options": {
+        "legend": {
+          "calcs": [
+            "mean",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "histogram_quantile(0.95, sum by (path, stage, size_bucket, le) (rate(rustfs_io_get_object_stage_duration_seconds_by_size_bucket{stage=~\"reader_setup|reader_setup_schedule|reader_setup_wait_quorum|reader_setup_drop_pending|reader_task_file_open|reader_task_reader_construction|reader_task_bitrot_reader_init\"}[$__rate_interval]))) or vector(0)",
+          "legendFormat": "p95 {{path}} / {{stage}} / {{size_bucket}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "histogram_quantile(0.99, sum by (path, stage, size_bucket, le) (rate(rustfs_io_get_object_stage_duration_seconds_by_size_bucket{stage=~\"reader_setup|reader_setup_schedule|reader_setup_wait_quorum|reader_setup_drop_pending|reader_task_file_open|reader_task_reader_construction|reader_task_bitrot_reader_init\"}[$__rate_interval]))) or vector(0)",
+          "legendFormat": "p99 {{path}} / {{stage}} / {{size_bucket}}",
+          "refId": "B"
+        }
+      ],
+      "title": "GET Hot Stage Latency by Size",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 16
+      },
+      "id": 6,
+      "options": {
+        "legend": {
+          "calcs": [
+            "mean",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "histogram_quantile(0.95, sum by (path, stage, le) (rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=~\"request_context|first_byte|response_handoff|full_body|body_build|output_poll|output_lock_wait|reader_open_mmap_copy_success|reader_open_mmap_copy_fallback|reader_open_stream|reader_stream_first_read|reader_mmap_blocking_wait|reader_mmap_blocking_task|reader_mmap_file_open|reader_mmap_map|reader_mmap_copy_buffer|reader_mmap_direct_read_copy\"}[$__rate_interval]))) or vector(0)",
+          "legendFormat": "p95 {{path}} / {{stage}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "histogram_quantile(0.99, sum by (path, stage, le) (rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=~\"request_context|first_byte|response_handoff|full_body|body_build|output_poll|output_lock_wait|reader_open_mmap_copy_success|reader_open_mmap_copy_fallback|reader_open_stream|reader_stream_first_read|reader_mmap_blocking_wait|reader_mmap_blocking_task|reader_mmap_file_open|reader_mmap_map|reader_mmap_copy_buffer|reader_mmap_direct_read_copy\"}[$__rate_interval]))) or vector(0)",
+          "legendFormat": "p99 {{path}} / {{stage}}",
+          "refId": "B"
+        }
+      ],
+      "title": "GET Reader, Handler and Body Latency",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "ops",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 24
+      },
+      "id": 7,
+      "options": {
+        "legend": {
+          "calcs": [
+            "mean",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum by (subpath) (rate(rustfs_io_get_object_direct_memory_subpath_total[$__rate_interval])) or vector(0)",
+          "legendFormat": "{{subpath}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum by (subpath, size_bucket) (rate(rustfs_io_get_object_direct_memory_subpath_by_size_total[$__rate_interval])) or vector(0)",
+          "legendFormat": "{{subpath}} / {{size_bucket}}",
+          "refId": "B"
+        }
+      ],
+      "title": "GET Direct-Memory Subpath Rate",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "ops",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 24
+      },
+      "id": 8,
+      "options": {
+        "legend": {
+          "calcs": [
+            "mean",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum by (path, stage, kind) (rate(rustfs_io_get_object_mmap_page_faults_total[$__rate_interval])) or vector(0)",
+          "legendFormat": "mmap {{path}} / {{stage}} / {{kind}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum by (path, stage, kind) (rate(rustfs_io_get_object_direct_read_page_faults_total[$__rate_interval])) or vector(0)",
+          "legendFormat": "direct-read {{path}} / {{stage}} / {{kind}}",
+          "refId": "B"
+        }
+      ],
+      "title": "GET mmap vs Direct-Read Page Fault Rate",
+      "type": "timeseries"
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 38,
+  "style": "dark",
+  "tags": [
+    "rustfs",
+    "get",
+    "performance",
+    "attribution"
+  ],
+  "templating": {
+    "list": [
+      {
+        "current": {
+          "selected": false,
+          "text": "Prometheus",
+          "value": "Prometheus"
+        },
+        "hide": 0,
+        "includeAll": false,
+        "multi": false,
+        "name": "DS_PROMETHEUS",
+        "options": [],
+        "query": "prometheus",
+        "refresh": 1,
+        "regex": "",
+        "skipUrlSync": false,
+        "type": "datasource"
+      }
+    ]
+  },
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "timezone": "",
+  "title": "RustFS GET Performance Attribution",
+  "uid": "rustfs-get-performance-attribution",
+  "version": 1,
+  "weekStart": ""
+}
@@ -0,0 +1,525 @@
+{
+  "annotations": {
+    "list": []
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "id": null,
+  "links": [],
+  "liveNow": false,
+  "panels": [
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "none",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 20,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 0
+      },
+      "id": 1,
+      "options": {
+        "reduceOptions": {
+          "values": false,
+          "calcs": ["lastNotNull"],
+          "fields": ""
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        },
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom",
+          "calcs": ["mean", "max"]
+        }
+      },
+      "title": "Concurrent GET Requests",
+      "type": "timeseries",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rustfs_io_get_object_concurrent_requests",
+          "legendFormat": "concurrent GETs",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percent",
+          "min": 0,
+          "max": 100,
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 20,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "line+area"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "transparent",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 70
+              },
+              {
+                "color": "red",
+                "value": 90
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 0
+      },
+      "id": 2,
+      "options": {
+        "reduceOptions": {
+          "values": false,
+          "calcs": ["lastNotNull"],
+          "fields": ""
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        },
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom",
+          "calcs": ["mean", "max"]
+        }
+      },
+      "title": "IO Queue Utilization",
+      "type": "timeseries",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rustfs_io_queue_utilization_percent",
+          "legendFormat": "queue utilization %",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rustfs_io_queue_permits_in_use",
+          "legendFormat": "permits in use",
+          "refId": "B"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rustfs_io_queue_permits_available",
+          "legendFormat": "permits available",
+          "refId": "C"
+        }
+      ]
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 8
+      },
+      "id": 3,
+      "options": {
+        "reduceOptions": {
+          "values": false,
+          "calcs": ["lastNotNull"],
+          "fields": ""
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        },
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom",
+          "calcs": ["mean", "max"]
+        }
+      },
+      "title": "Disk Permit Wait Duration (p50 / p95 / p99)",
+      "type": "timeseries",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "histogram_quantile(0.50, sum(rate(rustfs_io_disk_permit_wait_duration_seconds_bucket[$__rate_interval])) by (le))",
+          "legendFormat": "p50",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "histogram_quantile(0.95, sum(rate(rustfs_io_disk_permit_wait_duration_seconds_bucket[$__rate_interval])) by (le))",
+          "legendFormat": "p95",
+          "refId": "B"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "histogram_quantile(0.99, sum(rate(rustfs_io_disk_permit_wait_duration_seconds_bucket[$__rate_interval])) by (le))",
+          "legendFormat": "p99",
+          "refId": "C"
+        }
+      ]
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "bytes",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 15,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 8
+      },
+      "id": 4,
+      "options": {
+        "reduceOptions": {
+          "values": false,
+          "calcs": ["lastNotNull"],
+          "fields": ""
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        },
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom",
+          "calcs": ["lastNotNull"]
+        }
+      },
+      "title": "RSS Trend (Process Memory)",
+      "type": "timeseries",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "process_resident_memory_bytes{job=~\"rustfs.*\"}",
+          "legendFormat": "RSS {{instance}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "rustfs_process_memory_bytes",
+          "legendFormat": "RustFS memory {{instance}}",
+          "refId": "B"
+        }
+      ]
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "ops",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 16
+      },
+      "id": 5,
+      "options": {
+        "reduceOptions": {
+          "values": false,
+          "calcs": ["lastNotNull"],
+          "fields": ""
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        },
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom",
+          "calcs": ["mean", "max"]
+        }
+      },
+      "title": "IO Queue Operations by Priority",
+      "type": "timeseries",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(rustfs_io_queue_operations[$__rate_interval])) by (operation, priority)",
+          "legendFormat": "{{operation}} / {{priority}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "ops",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 15,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 16
+      },
+      "id": 6,
+      "options": {
+        "reduceOptions": {
+          "values": false,
+          "calcs": ["lastNotNull"],
+          "fields": ""
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        },
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom",
+          "calcs": ["sum"]
+        }
+      },
+      "title": "IO Queue Congestion Events",
+      "type": "timeseries",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(rustfs_io_queue_congestion_total[$__rate_interval]))",
+          "legendFormat": "congestion events/s",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(rustfs_io_get_object_timeout_total[$__rate_interval])) by (stage)",
+          "legendFormat": "timeout {{stage}}",
+          "refId": "B"
+        }
+      ]
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 38,
+  "style": "dark",
+  "tags": ["rustfs", "get-optimization", "resource-impact"],
+  "templating": {
+    "list": [
+      {
+        "current": {
+          "selected": false,
+          "text": "Prometheus",
+          "value": "Prometheus"
+        },
+        "hide": 0,
+        "includeAll": false,
+        "multi": false,
+        "name": "DS_PROMETHEUS",
+        "options": [],
+        "query": "prometheus",
+        "refresh": 1,
+        "regex": "",
+        "skipUrlSync": false,
+        "type": "datasource"
+      }
+    ]
+  },
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "timepicker": {},
+  "timezone": "",
+  "title": "RustFS GET Resource Impact",
+  "uid": "rustfs-get-resource-impact",
+  "version": 1
+}
@@ -0,0 +1,530 @@
+{
+  "annotations": {
+    "list": []
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "id": null,
+  "links": [],
+  "liveNow": false,
+  "panels": [
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "s",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "axisLabel": "",
+            "axisColorMode": "text",
+            "scaleDistribution": {
+              "type": "linear"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 0
+      },
+      "id": 1,
+      "options": {
+        "reduceOptions": {
+          "values": false,
+          "calcs": ["lastNotNull"],
+          "fields": ""
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        },
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom",
+          "calcs": ["mean", "max"]
+        }
+      },
+      "title": "GET Latency by Reader Path (p50 / p95 / p99)",
+      "type": "timeseries",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "histogram_quantile(0.50, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
+          "legendFormat": "p50 {{path}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "histogram_quantile(0.95, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
+          "legendFormat": "p95 {{path}}",
+          "refId": "B"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "histogram_quantile(0.99, sum(rate(rustfs_io_get_object_stage_duration_seconds_bucket{stage=\"total\"}[$__rate_interval])) by (le, path))",
+          "legendFormat": "p99 {{path}}",
+          "refId": "C"
+        }
+      ]
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit",
+          "min": 0,
+          "max": 1,
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 15,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "yellow",
+                "value": 0.3
+              },
+              {
+                "color": "red",
+                "value": 0.7
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 0
+      },
+      "id": 2,
+      "options": {
+        "reduceOptions": {
+          "values": false,
+          "calcs": ["lastNotNull"],
+          "fields": ""
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        },
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom",
+          "calcs": ["mean", "max"]
+        }
+      },
+      "title": "Early-Stop Hit Rate",
+      "type": "timeseries",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(rustfs_io_get_object_metadata_early_stop_total{decision=\"hit\"}[$__rate_interval])) by (path) / clamp_min(sum(rate(rustfs_io_get_object_metadata_early_stop_total[$__rate_interval])) by (path), 1)",
+          "legendFormat": "hit rate {{path}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit",
+          "min": 0,
+          "max": 1,
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 15,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 8
+      },
+      "id": 3,
+      "options": {
+        "reduceOptions": {
+          "values": false,
+          "calcs": ["lastNotNull"],
+          "fields": ""
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        },
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom",
+          "calcs": ["mean", "max"]
+        }
+      },
+      "title": "Codec Streaming Usage Rate",
+      "type": "timeseries",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(rustfs_io_get_object_codec_streaming_decision_total[$__rate_interval])) by (decision) / clamp_min(sum(rate(rustfs_io_get_object_codec_streaming_decision_total[$__rate_interval])), 1)",
+          "legendFormat": "codec_streaming {{decision}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(rustfs_io_get_object_stream_strategy_total[$__rate_interval])) by (strategy) / clamp_min(sum(rate(rustfs_io_get_object_stream_strategy_total[$__rate_interval])), 1)",
+          "legendFormat": "stream_strategy {{strategy}}",
+          "refId": "B"
+        }
+      ]
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "ops",
+          "custom": {
+            "drawStyle": "bars",
+            "fillOpacity": 80,
+            "lineWidth": 0,
+            "pointSize": 5,
+            "showPoints": "never",
+            "spanNulls": false,
+            "stacking": {
+              "mode": "normal",
+              "group": "A"
+            },
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              },
+              {
+                "color": "red",
+                "value": 1
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 8
+      },
+      "id": 4,
+      "options": {
+        "reduceOptions": {
+          "values": false,
+          "calcs": ["lastNotNull"],
+          "fields": ""
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        },
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom",
+          "calcs": ["sum"]
+        }
+      },
+      "title": "Pipeline Failure Rate by Stage",
+      "type": "timeseries",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(rustfs_io_get_object_pipeline_failures_total[$__rate_interval])) by (stage, reason)",
+          "legendFormat": "{{stage}} / {{reason}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "reqps",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 16
+      },
+      "id": 5,
+      "options": {
+        "reduceOptions": {
+          "values": false,
+          "calcs": ["lastNotNull"],
+          "fields": ""
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        },
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom",
+          "calcs": ["mean", "max"]
+        }
+      },
+      "title": "GET Request Rate by Path",
+      "type": "timeseries",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(rustfs_io_get_object_reader_path_total[$__rate_interval])) by (path)",
+          "legendFormat": "{{path}}",
+          "refId": "A"
+        }
+      ]
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "ops",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 16
+      },
+      "id": 6,
+      "options": {
+        "reduceOptions": {
+          "values": false,
+          "calcs": ["lastNotNull"],
+          "fields": ""
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        },
+        "legend": {
+          "displayMode": "table",
+          "placement": "bottom",
+          "calcs": ["mean", "max"]
+        }
+      },
+      "title": "GET Total Latency (p50 / p95 / p99)",
+      "type": "timeseries",
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "histogram_quantile(0.50, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
+          "legendFormat": "p50",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "histogram_quantile(0.95, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
+          "legendFormat": "p95",
+          "refId": "B"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "histogram_quantile(0.99, sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[$__rate_interval])) by (le))",
+          "legendFormat": "p99",
+          "refId": "C"
+        }
+      ]
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 38,
+  "style": "dark",
+  "tags": ["rustfs", "get-optimization"],
+  "templating": {
+    "list": [
+      {
+        "current": {
+          "selected": false,
+          "text": "Prometheus",
+          "value": "Prometheus"
+        },
+        "hide": 0,
+        "includeAll": false,
+        "multi": false,
+        "name": "DS_PROMETHEUS",
+        "options": [],
+        "query": "prometheus",
+        "refresh": 1,
+        "regex": "",
+        "skipUrlSync": false,
+        "type": "datasource"
+      }
+    ]
+  },
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "timepicker": {},
+  "timezone": "",
+  "title": "RustFS GET Rollout Health",
+  "uid": "rustfs-get-rollout-health",
+  "version": 1
+}
@@ -0,0 +1,534 @@
+{
+  "annotations": {
+    "list": []
+  },
+  "editable": true,
+  "fiscalYearStartMonth": 0,
+  "graphTooltip": 1,
+  "id": null,
+  "links": [],
+  "liveNow": false,
+  "panels": [
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "ops",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 0
+      },
+      "id": 1,
+      "options": {
+        "legend": {
+          "calcs": [
+            "mean",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum by (path, eager_status, size_bucket, buffer_bucket, large_concurrency_tuning) (rate(rustfs_s3_put_object_diagnostics_total[$__rate_interval])) or vector(0)",
+          "legendFormat": "{{path}} / {{eager_status}} / {{size_bucket}} / {{buffer_bucket}} / large={{large_concurrency_tuning}}",
+          "refId": "A"
+        }
+      ],
+      "title": "PUT Diagnostics Decision Rate",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "ops",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 0
+      },
+      "id": 2,
+      "options": {
+        "legend": {
+          "calcs": [
+            "mean",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum by (path) (rate(rustfs_s3_put_object_path_total[$__rate_interval])) or vector(0)",
+          "legendFormat": "{{path}}",
+          "refId": "A"
+        }
+      ],
+      "title": "PUT Path Rate",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "ms",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 8
+      },
+      "id": 3,
+      "options": {
+        "legend": {
+          "calcs": [
+            "mean",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "histogram_quantile(0.95, sum by (stage, le) (rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage=~\"ingress_prepare|set_disk_writer_setup|set_disk_encode|set_disk_rename|set_disk_old_data_cleanup|multipart_ingress_prepare|multipart_set_disk_writer_setup|multipart_set_disk_encode|multipart_complete_tail\"}[$__rate_interval]))) or vector(0)",
+          "legendFormat": "p95 {{stage}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "histogram_quantile(0.99, sum by (stage, le) (rate(rustfs_s3_put_object_stage_duration_ms_bucket{stage=~\"ingress_prepare|set_disk_writer_setup|set_disk_encode|set_disk_rename|set_disk_old_data_cleanup|multipart_ingress_prepare|multipart_set_disk_writer_setup|multipart_set_disk_encode|multipart_complete_tail\"}[$__rate_interval]))) or vector(0)",
+          "legendFormat": "p99 {{stage}}",
+          "refId": "B"
+        }
+      ],
+      "title": "PUT Hot Stage Latency",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "bytes",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 8
+      },
+      "id": 4,
+      "options": {
+        "legend": {
+          "calcs": [
+            "mean",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "histogram_quantile(0.50, sum by (path, size_bucket, le) (rate(rustfs_s3_put_object_selected_buffer_size_bytes_bucket[$__rate_interval]))) or vector(0)",
+          "legendFormat": "p50 {{path}} / {{size_bucket}}",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "histogram_quantile(0.95, sum by (path, size_bucket, le) (rate(rustfs_s3_put_object_selected_buffer_size_bytes_bucket[$__rate_interval]))) or vector(0)",
+          "legendFormat": "p95 {{path}} / {{size_bucket}}",
+          "refId": "B"
+        }
+      ],
+      "title": "PUT Selected Buffer Size",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "percentunit",
+          "min": 0,
+          "max": 1,
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 15,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 0,
+        "y": 16
+      },
+      "id": 5,
+      "options": {
+        "legend": {
+          "calcs": [
+            "mean",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(rustfs_s3_put_object_zero_copy_eligible_total[$__rate_interval])) / clamp_min(sum(rate(rustfs_s3_put_object_total[$__rate_interval])), 1) or vector(0)",
+          "legendFormat": "eligible / total",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(rustfs_s3_put_object_zero_copy_enabled_total[$__rate_interval])) / clamp_min(sum(rate(rustfs_s3_put_object_total[$__rate_interval])), 1) or vector(0)",
+          "legendFormat": "enabled / total",
+          "refId": "B"
+        }
+      ],
+      "title": "PUT Zero-Copy Eligibility Ratio",
+      "type": "timeseries"
+    },
+    {
+      "datasource": {
+        "type": "prometheus",
+        "uid": "${DS_PROMETHEUS}"
+      },
+      "fieldConfig": {
+        "defaults": {
+          "unit": "ops",
+          "custom": {
+            "drawStyle": "line",
+            "lineInterpolation": "smooth",
+            "fillOpacity": 10,
+            "pointSize": 5,
+            "lineWidth": 1,
+            "showPoints": "auto",
+            "spanNulls": false,
+            "thresholdsStyle": {
+              "mode": "off"
+            }
+          },
+          "thresholds": {
+            "mode": "absolute",
+            "steps": [
+              {
+                "color": "green",
+                "value": null
+              }
+            ]
+          }
+        },
+        "overrides": []
+      },
+      "gridPos": {
+        "h": 8,
+        "w": 12,
+        "x": 12,
+        "y": 16
+      },
+      "id": 6,
+      "options": {
+        "legend": {
+          "calcs": [
+            "mean",
+            "max"
+          ],
+          "displayMode": "table",
+          "placement": "bottom"
+        },
+        "reduceOptions": {
+          "calcs": [
+            "lastNotNull"
+          ],
+          "fields": "",
+          "values": false
+        },
+        "tooltip": {
+          "mode": "multi",
+          "sort": "desc"
+        }
+      },
+      "targets": [
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum(rate(rustfs_s3_put_object_total[$__rate_interval])) or vector(0)",
+          "legendFormat": "PUT total",
+          "refId": "A"
+        },
+        {
+          "datasource": {
+            "type": "prometheus",
+            "uid": "${DS_PROMETHEUS}"
+          },
+          "expr": "sum by (stage) (rate(rustfs_system_storage_erasure_write_quorum_failures_total[$__rate_interval])) or vector(0)",
+          "legendFormat": "write quorum failure {{stage}}",
+          "refId": "B"
+        }
+      ],
+      "title": "PUT Throughput and Quorum Failure Signals",
+      "type": "timeseries"
+    }
+  ],
+  "refresh": "30s",
+  "schemaVersion": 38,
+  "style": "dark",
+  "tags": [
+    "rustfs",
+    "put",
+    "performance",
+    "attribution"
+  ],
+  "templating": {
+    "list": [
+      {
+        "current": {
+          "selected": false,
+          "text": "Prometheus",
+          "value": "Prometheus"
+        },
+        "hide": 0,
+        "includeAll": false,
+        "multi": false,
+        "name": "DS_PROMETHEUS",
+        "options": [],
+        "query": "prometheus",
+        "refresh": 1,
+        "regex": "",
+        "skipUrlSync": false,
+        "type": "datasource"
+      }
+    ]
+  },
+  "time": {
+    "from": "now-1h",
+    "to": "now"
+  },
+  "timezone": "",
+  "title": "RustFS PUT Performance Attribution",
+  "uid": "rustfs-put-performance-attribution",
+  "version": 1,
+  "weekStart": ""
+}
@@ -1,97 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-apiVersion: 1
-
-datasources:
-  - name: Prometheus
-    type: prometheus
-    uid: prometheus
-    access: proxy
-    orgId: 1
-    url: http://prometheus:9090
-    isDefault: true
-    version: 1
-    editable: false
-    jsonData:
-      httpMethod: GET
-      exemplarTraceIdDestinations:
-        - name: trace_id
-          datasourceUid: tempo
-
-  - name: Tempo
-    type: tempo
-    uid: tempo
-    access: proxy
-    orgId: 1
-    url: http://tempo:3200
-    isDefault: false
-    version: 1
-    editable: false
-    jsonData:
-      httpMethod: GET
-      serviceMap:
-        datasourceUid: prometheus
-      tracesToLogs:
-        datasourceUid: loki
-        tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
-        mappedTags: [ { key: 'service.name', value: 'app' } ]
-        spanStartTimeShift: '1s'
-        spanEndTimeShift: '-1s'
-        filterByTraceID: true
-        filterBySpanID: false
-      tracesToMetrics:
-        datasourceUid: prometheus
-        tags: [ { key: 'service.name' }, { key: 'job' } ]
-        queries:
-          - name: 'Service-Level Latency'
-            query: 'sum(rate(traces_spanmetrics_latency_bucket{$$__tags}[5m])) by (le)'
-          - name: 'Service-Level Calls'
-            query: 'sum(rate(traces_spanmetrics_calls_total{$$__tags}[5m]))'
-          - name: 'Service-Level Errors'
-            query: 'sum(rate(traces_spanmetrics_calls_total{status_code="ERROR", $$__tags}[5m]))'
-      nodeGraph:
-        enabled: true
-
-  - name: Loki
-    type: loki
-    uid: loki
-    orgId: 1
-    url: http://loki:3100
-    isDefault: false
-    version: 1
-    editable: false
-    jsonData:
-      derivedFields:
-        - datasourceUid: tempo
-          matcherRegex: 'trace_id=(\w+)'
-          name: 'TraceID'
-          url: '$${__value.raw}'
-
-  - name: Jaeger
-    type: jaeger
-    uid: jaeger
-    url: http://jaeger:16686
-    access: proxy
-    isDefault: false
-    editable: false
-    jsonData:
-      tracesToLogs:
-        datasourceUid: loki
-        tags: [ 'job', 'instance', 'pod', 'namespace', 'service.name' ]
-        mappedTags: [ { key: 'service.name', value: 'app' } ]
-        spanStartTimeShift: '1s'
-        spanEndTimeShift: '-1s'
-        filterByTraceID: true
-        filterBySpanID: false
@@ -0,0 +1,53 @@
+groups:
+  - name: rustfs-dashboard
+    interval: 30s
+    rules:
+      - record: rustfs:http_server_requests:rate5m
+        expr: sum by (job) (rate(rustfs_http_server_requests_total[5m]))
+
+      - record: rustfs:http_server_request_duration_seconds:p50_5m
+        expr: histogram_quantile(0.50, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
+      - record: rustfs:http_server_request_duration_seconds:p95_5m
+        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
+      - record: rustfs:http_server_request_duration_seconds:p99_5m
+        expr: histogram_quantile(0.99, sum by (le, job) (rate(rustfs_http_server_request_duration_seconds_bucket[5m])))
+
+      - record: rustfs:http_server_response_body_size_bytes:p50_5m
+        expr: histogram_quantile(0.50, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
+      - record: rustfs:http_server_response_body_size_bytes:p95_5m
+        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
+      - record: rustfs:http_server_response_body_size_bytes:p99_5m
+        expr: histogram_quantile(0.99, sum by (le, job) (rate(rustfs_http_server_response_body_size_bytes_bucket[5m])))
+
+      - record: rustfs:log_cleaner_runs:rate15m
+        expr: sum by (job) (rate(rustfs_log_cleaner_runs_total[15m]))
+      - record: rustfs:log_cleaner_failure_ratio:rate5m
+        expr: sum by (job) (rate(rustfs_log_cleaner_run_failures_total[5m])) / clamp_min(sum by (job) (rate(rustfs_log_cleaner_runs_total[5m])), 1e-9)
+      - record: rustfs:log_cleaner_rotation_failure_ratio:rate5m
+        expr: sum by (job) (rate(rustfs_log_cleaner_rotation_failures_total[5m])) / clamp_min(sum by (job) (rate(rustfs_log_cleaner_rotation_total[5m])), 1e-9)
+      - record: rustfs:log_cleaner_rotation_duration_seconds:p95_5m
+        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_log_cleaner_rotation_duration_seconds_bucket[5m])))
+      - record: rustfs:log_cleaner_compress_duration_seconds:p95_5m
+        expr: histogram_quantile(0.95, sum by (le, job) (rate(rustfs_log_cleaner_compress_duration_seconds_bucket[5m])))
+
+      - record: rustfs:scanner_objects_scanned:rate5m
+        expr: sum by (job) (rate(rustfs_scanner_objects_scanned_total[5m]))
+      - record: rustfs:scanner_directories_scanned:rate5m
+        expr: sum by (job) (rate(rustfs_scanner_directories_scanned_total[5m]))
+      - record: rustfs:scanner_buckets_scanned:rate5m
+        expr: sum by (job) (rate(rustfs_scanner_buckets_scanned_total[5m]))
+      - record: rustfs:scanner_cycles_success:rate5m
+        expr: sum by (job) (rate(rustfs_scanner_cycles_total{result="success"}[5m]))
+
+      - record: rustfs:log_chain_op_event_mismatch:rate5m
+        expr: sum by (job) (rate(rustfs_log_chain_op_event_mismatch_total[5m]))
+
+      - alert: RustFSLogChainOpEventMismatchDetected
+        expr: rustfs:log_chain_op_event_mismatch:rate5m > 0
+        for: 10m
+        labels:
+          severity: warning
+          component: s3-log-chain
+        annotations:
+          summary: "RustFS log-chain op/event mismatch detected"
+          description: "job={{ $labels.job }} has non-zero rustfs_log_chain_op_event_mismatch_total rate for more than 10m. Check s3 op/event mapping changes."
@@ -0,0 +1,260 @@
+# =============================================================================
+# RustFS GET Optimization — Prometheus Alerting Rules
+# =============================================================================
+#
+# Import into Prometheus:
+#   1. Copy this file to your Prometheus rules directory
+#   2. Add to prometheus.yml:
+#        rule_files:
+#          - "prometheus-alert-rules.yaml"
+#   3. Validate: promtool check rules prometheus-alert-rules.yaml
+#   4. Reload:   curl -X POST http://localhost:9090/-/reload
+#
+# All metric names match those registered in crates/io-metrics/src/lib.rs
+# and documented in crates/ecstore/src/diagnostics/get.rs.
+#
+# Baseline comparison uses "offset 1d" — adjust to "offset 7d" for weekly
+# seasonality if your traffic pattern varies by day of week.
+# =============================================================================
+
+groups:
+  # ==========================================================================
+  # Critical alerts — immediate action required
+  # ==========================================================================
+  - name: rustfs-get-optimization-critical
+    interval: 30s
+    rules:
+      # ------------------------------------------------------------------
+      # 1. GetP99Regression
+      #    GET p99 latency exceeds 2x the baseline (same time yesterday)
+      #    sustained for 10 minutes.
+      #    Action: Roll back the GET optimization immediately.
+      # ------------------------------------------------------------------
+      - alert: GetP99Regression
+        expr: |
+          histogram_quantile(0.99,
+            sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[5m])) by (le)
+          )
+          >
+          2
+          *
+          histogram_quantile(0.99,
+            sum(rate(rustfs_io_get_object_total_duration_seconds_bucket[5m] offset 1d)) by (le)
+          )
+        for: 10m
+        labels:
+          severity: critical
+          team: rustfs-storage
+          area: get-optimization
+        annotations:
+          summary: "GET p99 latency regression detected (>2x baseline for 10m)"
+          description: >-
+            The 99th-percentile GET object latency is {{ $value | humanizeDuration }}
+            which is more than double the baseline measured 24 hours ago.
+            This indicates a severe performance regression introduced by
+            a recent GET optimization change.
+          runbook_url: "https://internal.wiki/runbooks/rustfs/get-p99-regression"
+          action: >
+            1. Verify the regression is not caused by external factors
+               (disk health, network, load spike).
+            2. If confirmed optimization-related, roll back:
+               - Set RUSTFS_GET_CODEC_STREAMING=0
+               - Set RUSTFS_GET_METADATA_EARLY_STOP=0
+               - Restart affected nodes.
+            3. Collect flamegraphs and open a P0 incident.
+
+      # ------------------------------------------------------------------
+      # 2. PipelineFailureSpike
+      #    Pipeline failure rate exceeds 5x the baseline sustained for
+      #    5 minutes.  Covers all failure reasons: bitrot_mismatch,
+      #    decode_error, downstream_closed, io, read_quorum, timeout, etc.
+      #    Action: Investigate pipeline health and roll back if needed.
+      # ------------------------------------------------------------------
+      - alert: PipelineFailureSpike
+        expr: |
+          sum(rate(rustfs_io_get_object_pipeline_failures_total[5m]))
+          >
+          5
+          *
+          sum(rate(rustfs_io_get_object_pipeline_failures_total[5m] offset 1d))
+        for: 5m
+        labels:
+          severity: critical
+          team: rustfs-storage
+          area: get-optimization
+        annotations:
+          summary: "GET pipeline failure rate spike (>5x baseline for 5m)"
+          description: >-
+            The GET pipeline failure rate is {{ $value | printf "%.2f" }}/s,
+            more than 5x the baseline from 24 hours ago.
+            Failure reasons may include: bitrot_mismatch, decode_error,
+            downstream_closed, io, range_or_length_invalid, read_quorum,
+            short_read, timeout, unknown.
+          runbook_url: "https://internal.wiki/runbooks/rustfs/pipeline-failure-spike"
+          action: >
+            1. Check Grafana "GET Data Integrity" dashboard for failure
+               breakdown by reason label.
+            2. If decode_error or bitrot_mismatch dominates, stop
+               optimization and investigate data integrity.
+            3. If io or timeout dominates, check disk and network health.
+            4. Roll back optimization if failures persist.
+
+      # ------------------------------------------------------------------
+      # 3. BitrotMismatchSpike
+      #    Bitrot verification mismatch rate exceeds 3x baseline for
+      #    5 minutes.  This is a data-integrity signal — shard checksums
+      #    do not match after read.
+      #
+      #    The "bitrot_mismatch" reason is recorded on the
+      #    rustfs_io_get_object_pipeline_failures_total counter when a
+      #    StorageError::FileCorrupt or DiskError::FileCorrupt /
+      #    DiskError::PartMissingOrCorrupt is classified during the GET
+      #    pipeline (see classify_storage_error / classify_disk_error in
+      #    crates/ecstore/src/diagnostics/get.rs).
+      #
+      #    Action: Stop optimization, investigate data integrity urgently.
+      # ------------------------------------------------------------------
+      - alert: BitrotMismatchSpike
+        expr: |
+          sum(rate(rustfs_io_get_object_pipeline_failures_total{reason="bitrot_mismatch"}[5m]))
+          >
+          3
+          *
+          sum(rate(rustfs_io_get_object_pipeline_failures_total{reason="bitrot_mismatch"}[5m] offset 1d))
+        for: 5m
+        labels:
+          severity: critical
+          team: rustfs-storage
+          area: get-optimization
+        annotations:
+          summary: "Bitrot mismatch rate spike (>3x baseline for 5m)"
+          description: >-
+            The rate of pipeline failures classified as bitrot_mismatch is
+            {{ $value | printf "%.2f" }}/s, more than 3x the baseline from
+            24 hours ago.  This indicates shard checksum verification
+            failures (FileCorrupt / PartMissingOrCorrupt) which may point
+            to data corruption introduced by the GET optimization pipeline
+            (e.g., incorrect decode, buffer reuse bug).
+          runbook_url: "https://internal.wiki/runbooks/rustfs/bitrot-mismatch-spike"
+          action: >
+            1. Immediately disable codec streaming:
+               RUSTFS_GET_CODEC_STREAMING=0
+            2. Run "mc admin scan" on affected buckets to verify on-disk
+               integrity independent of the GET path.
+            3. Compare xl.meta checksums across erasure shards.
+            4. If corruption confirmed, initiate data recovery from parity.
+            5. Do NOT re-enable optimization until root cause is identified.
+
+  # ==========================================================================
+  # Warning alerts — investigation needed
+  # ==========================================================================
+  - name: rustfs-get-optimization-warning
+    interval: 30s
+    rules:
+      # ------------------------------------------------------------------
+      # 4. EarlyStopInsufficientQuorum
+      #    The metadata early-stop path is hitting "insufficient_quorum"
+      #    at a rate above 0.1/s for 5 minutes.  This means too many
+      #    disks are failing to return valid metadata in time.
+      #    Action: Check disk health and metadata fanout latency.
+      # ------------------------------------------------------------------
+      - alert: EarlyStopInsufficientQuorum
+        expr: |
+          sum(rate(rustfs_io_get_object_metadata_early_stop_total{reason="insufficient_quorum"}[5m]))
+          > 0.1
+        for: 5m
+        labels:
+          severity: warning
+          team: rustfs-storage
+          area: get-optimization
+        annotations:
+          summary: "Early-stop insufficient quorum rate elevated (>0.1/s for 5m)"
+          description: >-
+            The metadata early-stop path is returning "insufficient_quorum"
+            at {{ $value | printf "%.3f" }}/s.  This means the bounded
+            metadata fanout cannot gather enough valid responses before
+            the quorum deadline, suggesting disk or network issues.
+          runbook_url: "https://internal.wiki/runbooks/rustfs/early-stop-quorum"
+          action: >
+            1. Check disk health: mc admin info --json | jq '.disks'
+            2. Review rustfs_io_get_object_metadata_response_total by
+               outcome (error, timeout, disk_not_found) in Grafana.
+            3. Check rustfs_io_disk_permit_wait_duration_seconds for
+               I/O scheduler saturation.
+            4. If disks are healthy, consider increasing the early-stop
+               timeout or temporarily disabling early-stop.
+
+      # ------------------------------------------------------------------
+      # 5. CodecStreamingFallbackSpike
+      #    The codec streaming fallback rate is >10x baseline for 10
+      #    minutes.  This means the optimized codec streaming path is
+      #    being bypassed much more often than expected.
+      #    Action: Check fallback reasons and object eligibility.
+      # ------------------------------------------------------------------
+      - alert: CodecStreamingFallbackSpike
+        expr: |
+          sum(rate(rustfs_io_get_object_codec_streaming_fallback_total[5m]))
+          >
+          10
+          *
+          sum(rate(rustfs_io_get_object_codec_streaming_fallback_total[5m] offset 1d))
+        for: 10m
+        labels:
+          severity: warning
+          team: rustfs-storage
+          area: get-optimization
+        annotations:
+          summary: "Codec streaming fallback rate spike (>10x baseline for 10m)"
+          description: >-
+            The codec streaming fallback rate is {{ $value | printf "%.2f" }}/s,
+            more than 10x the baseline from 24 hours ago.  Fallback reasons
+            are labeled by "reason" — check Grafana for breakdown.
+            Common reasons: object too small, multipart not supported,
+            unsupported erasure layout, feature flag disabled.
+          runbook_url: "https://internal.wiki/runbooks/rustfs/codec-fallback-spike"
+          action: >
+            1. Query by reason label:
+               sum by (reason) (rate(rustfs_io_get_object_codec_streaming_fallback_total[5m]))
+            2. If dominated by a single reason, investigate why that
+               condition became more frequent (e.g., workload change,
+               configuration drift).
+            3. Cross-reference with rustfs_io_get_object_reader_path_total
+               to verify the fallback path (legacy_duplex) is healthy.
+            4. If fallback is expected (e.g., workload shifted to small
+               objects), update the alert baseline.
+
+      # ------------------------------------------------------------------
+      # 6. IoQueueSaturation
+      #    I/O queue utilization exceeds 90% for 5 minutes.  High
+      #    utilization causes disk permit wait latency to increase and
+      #    can cascade into pipeline timeouts.
+      #    Action: Check disk load and consider reducing concurrency.
+      # ------------------------------------------------------------------
+      - alert: IoQueueSaturation
+        expr: |
+          rustfs_io_queue_utilization_percent > 90
+        for: 5m
+        labels:
+          severity: warning
+          team: rustfs-storage
+          area: get-optimization
+        annotations:
+          summary: "I/O queue utilization >90% for 5m"
+          description: >-
+            The I/O queue utilization is {{ $value | printf "%.1f" }}%,
+            sustained above 90% for 5 minutes.  This indicates the disk
+            I/O scheduler is near saturation, which will increase
+            rustfs_io_disk_permit_wait_duration_seconds and may trigger
+            pipeline timeouts.
+          runbook_url: "https://internal.wiki/runbooks/rustfs/io-queue-saturation"
+          action: >
+            1. Check disk I/O metrics (iostat, node_exporter) for
+               individual disk saturation.
+            2. Review rustfs_io_queue_permits_in_use vs
+               rustfs_io_queue_permits_available for permit exhaustion.
+            3. Check rustfs_io_starvation_events for priority starvation.
+            4. If GET optimization increased concurrency, consider:
+               - Reducing RUSTFS_GET_PIPELINE_PARALLELISM
+               - Lowering RUSTFS_IO_QUEUE_PERMITS
+            5. If caused by background operations (ILM, healing), throttle
+               those before adjusting GET concurrency.
@@ -19,6 +19,9 @@ global:
    cluster: 'rustfs-dev'  # Label to identify the cluster
    replica: '1'  # Replica identifier

+rule_files:
+  - /etc/prometheus/rules/*.yml
+
 scrape_configs:
  - job_name: 'otel-collector'
    static_configs:
@@ -12,7 +12,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

-partition_ring_live_store: true
 stream_over_http_enabled: true

 server:
@@ -33,33 +32,17 @@ distributor:
          endpoint: "0.0.0.0:4317"
        http:
          endpoint: "0.0.0.0:4318"
-  #log_received_spans:
-  #    enabled: true
-  #  log_discarded_spans:
-  #    enabled: true

-backend_scheduler:
-  provider:
-    compaction:
-      compaction:
-        block_retention: 1h
-
-backend_worker:
-  backend_scheduler_addr: localhost:3200
-  compaction:
-    block_retention: 1h
-  ring:
-    kvstore:
-      store: memberlist
-
-querier:
-  query_live_store: true
+ingester:
+  max_block_duration: 5m

 metrics_generator:
  registry:
    external_labels:
      source: tempo
      cluster: docker-compose
+  traces_storage:
+    path: /var/tempo/generator/traces
  storage:
    path: /var/tempo/generator/wal
    remote_write:
@@ -85,15 +68,5 @@ overrides:
      processors: [ "span-metrics", "service-graphs", "local-blocks" ]
      generate_native_histograms: both

-ingest:
-  enabled: true
-  kafka:
-    address: redpanda:9092
-    topic: tempo-ingest
-
-block_builder:
-  consume_cycle_duration: 30s
-
 usage_report:
  reporting_enabled: false
-
@@ -0,0 +1,91 @@
+# Rio compatibility compose files
+
+These compose files prepare 4-node, 4-disk clusters for rio/rio-v2 storage format compatibility checks. All disks are bind-mounted under `.docker/compat/data` so the on-disk files remain available on the host.
+
+## Clusters
+
+```bash
+docker compose -f .docker/compat/docker-compose.rustfs-beta5.yml up -d --build
+docker compose -f .docker/compat/docker-compose.minio.yml up -d
+docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
+```
+
+Default API endpoints:
+
+- RustFS `1.0.0-beta.5`: `http://127.0.0.1:9100`
+- MinIO: `http://127.0.0.1:9200`
+- current main with `rio-v2`: `http://127.0.0.1:9300`
+
+## Reading old datasets with rio-v2
+
+Stop the writer cluster before mounting its disks into the rio-v2 cluster.
+
+```bash
+docker compose -f .docker/compat/docker-compose.rustfs-beta5.yml down
+RUSTFS_RIO_V2_DATASET=./data/rustfs-beta5 \
+  docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
+
+docker compose -f .docker/compat/docker-compose.minio.yml down
+RUSTFS_RIO_V2_DATASET=./data/minio \
+  docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
+```
+
+## 200G object mix
+
+Use the same bucket/object matrix against the beta5 and MinIO endpoints, then read it back through the rio-v2 endpoint. A practical 200G mix is:
+
+- 1 KiB x 1024
+- 1 MiB x 1024
+- 64 MiB x 512
+- 1 GiB x 64
+- 8 GiB x 12
+- 6 GiB x 1
+
+Compression is enabled by default for RustFS and MinIO. Server-side KMS/SSE settings are intentionally left to environment variables or mounted key directories so real key material is not committed. For SSE-C cases, run the clusters with TLS because MinIO requires HTTPS for SSE-C.
+
+## High-concurrency write/read stress
+
+Use `run_rw_compat_stress.sh` to generate a manifest on an old endpoint, then verify the same objects through the rio-v2 endpoint after mounting the old disks.
+
+```bash
+.docker/compat/run_rw_compat_stress.sh \
+  --mode write \
+  --endpoint http://127.0.0.1:9100 \
+  --access-key rustfsadmin \
+  --secret-key rustfsadmin \
+  --bucket compat-beta5 \
+  --concurrency 96
+
+RUSTFS_RIO_V2_DATASET=./data/rustfs-beta5 \
+  docker compose -f .docker/compat/docker-compose.rustfs-rio-v2.yml up -d --build
+
+.docker/compat/run_rw_compat_stress.sh \
+  --mode verify \
+  --endpoint http://127.0.0.1:9300 \
+  --access-key rustfsadmin \
+  --secret-key rustfsadmin \
+  --bucket compat-beta5 \
+  --concurrency 96 \
+  --manifest target/compat/rw-stress-YYYYmmdd-HHMMSS/manifest.csv
+```
+
+For encrypted datasets, add `--encryption sse-s3`, `--encryption sse-kms --sse-kms-key-id <key-id>`, or `--encryption sse-c --sse-c-key-file <raw-32-byte-key-file>` to both the write and verify commands.
+
+## 5 GiB encrypted compatibility run
+
+The `5g` profile covers 1 KiB, 1 MiB, 16 MiB, 64 MiB, and 1 GiB objects and totals exactly 5 GiB. Generate `compat-key.key` under `.docker/compat/kms/rustfs-compat`, enable local KMS on both RustFS clusters, then use the same encryption arguments while writing with beta5 and verifying with rio-v2. Set non-default local test credentials first because distributed listeners reject the built-in default credentials.
+
+```bash
+export COMPAT_ACCESS_KEY='<non-default-access-key>'
+export COMPAT_SECRET_KEY='<non-default-secret-key>'
+
+RUSTFS_ACCESS_KEY="$COMPAT_ACCESS_KEY" RUSTFS_SECRET_KEY="$COMPAT_SECRET_KEY" \
+  RUSTFS_KMS_ENABLE=true RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS=true \
+  docker compose -f .docker/compat/docker-compose.rustfs-beta5.yml up -d
+
+.docker/compat/run_rw_compat_stress.sh \
+  --mode write --endpoint http://127.0.0.1:9100 \
+  --access-key "$COMPAT_ACCESS_KEY" --secret-key "$COMPAT_SECRET_KEY" \
+  --bucket compat-beta5-sse-s3 --profile 5g --concurrency 16 \
+  --encryption sse-s3
+```
@@ -0,0 +1,88 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0.
+
+x-minio-env: &minio-env
+  MINIO_ROOT_USER: "${MINIO_ROOT_USER:-minioadmin}"
+  MINIO_ROOT_PASSWORD: "${MINIO_ROOT_PASSWORD:-minioadmin}"
+  MINIO_COMPRESSION_ENABLE: "${MINIO_COMPRESSION_ENABLE:-on}"
+  MINIO_COMPRESSION_ALLOW_ENCRYPTION: "${MINIO_COMPRESSION_ALLOW_ENCRYPTION:-on}"
+  MINIO_COMPRESSION_EXTENSIONS: "${MINIO_COMPRESSION_EXTENSIONS:-.txt,.log,.csv,.json,.tar,.xml,.bin}"
+  MINIO_COMPRESSION_MIME_TYPES: "${MINIO_COMPRESSION_MIME_TYPES:-text/*,application/json,application/xml,binary/octet-stream}"
+
+x-minio-node: &minio-node
+  image: "${MINIO_IMAGE:-quay.io/minio/minio:latest}"
+  command: server --console-address ":9001" "http://minio{1...4}:9000/data/disk{1...4}"
+  environment: *minio-env
+  networks:
+    - minio-compat-net
+  restart: unless-stopped
+
+services:
+  minio-permission-helper:
+    image: alpine:3.23
+    command: sh -c "mkdir -p /compat-data && chown -R 1000:1000 /compat-data"
+    volumes:
+      - ./data/minio:/compat-data
+    restart: "no"
+
+  minio1:
+    <<: *minio-node
+    hostname: minio1
+    depends_on:
+      minio-permission-helper:
+        condition: service_completed_successfully
+    volumes:
+      - ./data/minio/node1/disk1:/data/disk1
+      - ./data/minio/node1/disk2:/data/disk2
+      - ./data/minio/node1/disk3:/data/disk3
+      - ./data/minio/node1/disk4:/data/disk4
+    ports:
+      - "${MINIO_API_PORT:-9200}:9000"
+      - "${MINIO_CONSOLE_PORT:-9201}:9001"
+
+  minio2:
+    <<: *minio-node
+    hostname: minio2
+    depends_on:
+      minio-permission-helper:
+        condition: service_completed_successfully
+    volumes:
+      - ./data/minio/node2/disk1:/data/disk1
+      - ./data/minio/node2/disk2:/data/disk2
+      - ./data/minio/node2/disk3:/data/disk3
+      - ./data/minio/node2/disk4:/data/disk4
+    ports:
+      - "${MINIO_NODE2_PORT:-9202}:9000"
+
+  minio3:
+    <<: *minio-node
+    hostname: minio3
+    depends_on:
+      minio-permission-helper:
+        condition: service_completed_successfully
+    volumes:
+      - ./data/minio/node3/disk1:/data/disk1
+      - ./data/minio/node3/disk2:/data/disk2
+      - ./data/minio/node3/disk3:/data/disk3
+      - ./data/minio/node3/disk4:/data/disk4
+    ports:
+      - "${MINIO_NODE3_PORT:-9203}:9000"
+
+  minio4:
+    <<: *minio-node
+    hostname: minio4
+    depends_on:
+      minio-permission-helper:
+        condition: service_completed_successfully
+    volumes:
+      - ./data/minio/node4/disk1:/data/disk1
+      - ./data/minio/node4/disk2:/data/disk2
+      - ./data/minio/node4/disk3:/data/disk3
+      - ./data/minio/node4/disk4:/data/disk4
+    ports:
+      - "${MINIO_NODE4_PORT:-9204}:9000"
+
+networks:
+  minio-compat-net:
+    driver: bridge
@@ -0,0 +1,104 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0.
+
+x-rustfs-env: &rustfs-env
+  RUSTFS_VOLUMES: "http://rustfs-beta5-node{1...4}:9000/data/disk{1...4}"
+  RUSTFS_ADDRESS: ":9000"
+  RUSTFS_CONSOLE_ADDRESS: ":9001"
+  RUSTFS_CONSOLE_ENABLE: "true"
+  RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS: "*"
+  RUSTFS_ACCESS_KEY: "${RUSTFS_ACCESS_KEY:-rustfsadmin}"
+  RUSTFS_SECRET_KEY: "${RUSTFS_SECRET_KEY:-rustfsadmin}"
+  RUSTFS_OBS_LOGGER_LEVEL: "${RUSTFS_OBS_LOGGER_LEVEL:-info}"
+  RUSTFS_OBS_LOG_DIRECTORY: "/logs"
+  RUSTFS_COMPRESSION_ENABLED: "${RUSTFS_COMPRESSION_ENABLED:-true}"
+  RUSTFS_COMPRESSION_EXTENSIONS: "${RUSTFS_COMPRESSION_EXTENSIONS:-.txt,.log,.csv,.json,.tar,.xml,.bin}"
+  RUSTFS_COMPRESSION_MIME_TYPES: "${RUSTFS_COMPRESSION_MIME_TYPES:-text/*,application/json,application/xml,binary/octet-stream}"
+  RUSTFS_KMS_ENABLE: "${RUSTFS_KMS_ENABLE:-false}"
+  RUSTFS_KMS_BACKEND: "${RUSTFS_KMS_BACKEND:-local}"
+  RUSTFS_KMS_KEY_DIR: "${RUSTFS_KMS_KEY_DIR:-/kms}"
+  RUSTFS_KMS_DEFAULT_KEY_ID: "${RUSTFS_KMS_DEFAULT_KEY_ID:-compat-key}"
+  RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS: "${RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS:-false}"
+  RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}"
+
+x-rustfs-node: &rustfs-node
+  image: "${RUSTFS_BETA5_IMAGE:-rustfs/rustfs:1.0.0-beta.5}"
+  build:
+    context: ../..
+    dockerfile: Dockerfile
+    args:
+      RELEASE: "1.0.0-beta.5"
+  environment: *rustfs-env
+  depends_on:
+    rustfs-beta5-permission-helper:
+      condition: service_completed_successfully
+  networks:
+    - rustfs-beta5-net
+  restart: unless-stopped
+
+services:
+  rustfs-beta5-permission-helper:
+    image: alpine:3.23
+    command: sh -c "mkdir -p /compat-data /kms && chown -R 10001:10001 /compat-data /kms"
+    volumes:
+      - ./data/rustfs-beta5:/compat-data
+      - ./kms/rustfs-compat:/kms
+    restart: "no"
+
+  rustfs-beta5-node1:
+    <<: *rustfs-node
+    hostname: rustfs-beta5-node1
+    volumes:
+      - ./data/rustfs-beta5/node1/disk1:/data/disk1
+      - ./data/rustfs-beta5/node1/disk2:/data/disk2
+      - ./data/rustfs-beta5/node1/disk3:/data/disk3
+      - ./data/rustfs-beta5/node1/disk4:/data/disk4
+      - ./data/rustfs-beta5/logs/node1:/logs
+      - ./kms/rustfs-compat:/kms
+    ports:
+      - "${RUSTFS_BETA5_API_PORT:-9100}:9000"
+      - "${RUSTFS_BETA5_CONSOLE_PORT:-9101}:9001"
+
+  rustfs-beta5-node2:
+    <<: *rustfs-node
+    hostname: rustfs-beta5-node2
+    volumes:
+      - ./data/rustfs-beta5/node2/disk1:/data/disk1
+      - ./data/rustfs-beta5/node2/disk2:/data/disk2
+      - ./data/rustfs-beta5/node2/disk3:/data/disk3
+      - ./data/rustfs-beta5/node2/disk4:/data/disk4
+      - ./data/rustfs-beta5/logs/node2:/logs
+      - ./kms/rustfs-compat:/kms
+    ports:
+      - "${RUSTFS_BETA5_NODE2_PORT:-9102}:9000"
+
+  rustfs-beta5-node3:
+    <<: *rustfs-node
+    hostname: rustfs-beta5-node3
+    volumes:
+      - ./data/rustfs-beta5/node3/disk1:/data/disk1
+      - ./data/rustfs-beta5/node3/disk2:/data/disk2
+      - ./data/rustfs-beta5/node3/disk3:/data/disk3
+      - ./data/rustfs-beta5/node3/disk4:/data/disk4
+      - ./data/rustfs-beta5/logs/node3:/logs
+      - ./kms/rustfs-compat:/kms
+    ports:
+      - "${RUSTFS_BETA5_NODE3_PORT:-9103}:9000"
+
+  rustfs-beta5-node4:
+    <<: *rustfs-node
+    hostname: rustfs-beta5-node4
+    volumes:
+      - ./data/rustfs-beta5/node4/disk1:/data/disk1
+      - ./data/rustfs-beta5/node4/disk2:/data/disk2
+      - ./data/rustfs-beta5/node4/disk3:/data/disk3
+      - ./data/rustfs-beta5/node4/disk4:/data/disk4
+      - ./data/rustfs-beta5/logs/node4:/logs
+      - ./kms/rustfs-compat:/kms
+    ports:
+      - "${RUSTFS_BETA5_NODE4_PORT:-9104}:9000"
+
+networks:
+  rustfs-beta5-net:
+    driver: bridge
@@ -0,0 +1,115 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0.
+
+x-rustfs-rio-v2-env: &rustfs-rio-v2-env
+  RUSTFS_VOLUMES: "http://rustfs-rio-v2-node{1...4}:9000/data/disk{1...4}"
+  RUSTFS_ADDRESS: ":9000"
+  RUSTFS_CONSOLE_ADDRESS: ":9001"
+  RUSTFS_CONSOLE_ENABLE: "true"
+  RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS: "*"
+  RUSTFS_ACCESS_KEY: "${RUSTFS_ACCESS_KEY:-rustfsadmin}"
+  RUSTFS_SECRET_KEY: "${RUSTFS_SECRET_KEY:-rustfsadmin}"
+  RUSTFS_OBS_LOGGER_LEVEL: "${RUSTFS_OBS_LOGGER_LEVEL:-info}"
+  RUSTFS_OBS_LOG_DIRECTORY: "/logs"
+  RUSTFS_COMPRESSION_ENABLED: "${RUSTFS_COMPRESSION_ENABLED:-true}"
+  RUSTFS_COMPRESSION_EXTENSIONS: "${RUSTFS_COMPRESSION_EXTENSIONS:-.txt,.log,.csv,.json,.tar,.xml,.bin}"
+  RUSTFS_COMPRESSION_MIME_TYPES: "${RUSTFS_COMPRESSION_MIME_TYPES:-text/*,application/json,application/xml,binary/octet-stream}"
+  RUSTFS_KMS_ENABLE: "${RUSTFS_KMS_ENABLE:-false}"
+  RUSTFS_KMS_BACKEND: "${RUSTFS_KMS_BACKEND:-local}"
+  RUSTFS_KMS_KEY_DIR: "${RUSTFS_KMS_KEY_DIR:-/kms}"
+  RUSTFS_KMS_DEFAULT_KEY_ID: "${RUSTFS_KMS_DEFAULT_KEY_ID:-compat-key}"
+  RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS: "${RUSTFS_KMS_ALLOW_INSECURE_DEV_DEFAULTS:-false}"
+  RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}"
+
+x-rustfs-rio-v2-node: &rustfs-rio-v2-node
+  image: "${RUSTFS_RIO_V2_IMAGE:-rustfs/rustfs:compat-rio-v2}"
+  entrypoint:
+    - /bin/sh
+    - -c
+    - |
+      until getent hosts rustfs-rio-v2-node1 >/dev/null &&
+        getent hosts rustfs-rio-v2-node2 >/dev/null &&
+        getent hosts rustfs-rio-v2-node3 >/dev/null &&
+        getent hosts rustfs-rio-v2-node4 >/dev/null; do
+        sleep 1
+      done
+      exec /entrypoint.sh /usr/bin/rustfs
+  build:
+    context: ../..
+    dockerfile: Dockerfile.source
+    args:
+      RUSTFS_BUILD_FEATURES: "rio-v2"
+  environment: *rustfs-rio-v2-env
+  depends_on:
+    rustfs-rio-v2-permission-helper:
+      condition: service_completed_successfully
+  networks:
+    - rustfs-rio-v2-net
+  restart: unless-stopped
+
+services:
+  rustfs-rio-v2-permission-helper:
+    image: alpine:3.23
+    command: sh -c "mkdir -p /compat-data /kms && chown -R 10001:10001 /compat-data /kms"
+    volumes:
+      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}:/compat-data
+      - ./kms/rustfs-compat:/kms
+    restart: "no"
+
+  rustfs-rio-v2-node1:
+    <<: *rustfs-rio-v2-node
+    hostname: rustfs-rio-v2-node1
+    volumes:
+      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node1/disk1:/data/disk1
+      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node1/disk2:/data/disk2
+      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node1/disk3:/data/disk3
+      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node1/disk4:/data/disk4
+      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/logs/node1:/logs
+      - ./kms/rustfs-compat:/kms
+    ports:
+      - "${RUSTFS_RIO_V2_API_PORT:-9300}:9000"
+      - "${RUSTFS_RIO_V2_CONSOLE_PORT:-9301}:9001"
+
+  rustfs-rio-v2-node2:
+    <<: *rustfs-rio-v2-node
+    hostname: rustfs-rio-v2-node2
+    volumes:
+      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node2/disk1:/data/disk1
+      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node2/disk2:/data/disk2
+      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node2/disk3:/data/disk3
+      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node2/disk4:/data/disk4
+      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/logs/node2:/logs
+      - ./kms/rustfs-compat:/kms
+    ports:
+      - "${RUSTFS_RIO_V2_NODE2_PORT:-9302}:9000"
+
+  rustfs-rio-v2-node3:
+    <<: *rustfs-rio-v2-node
+    hostname: rustfs-rio-v2-node3
+    volumes:
+      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node3/disk1:/data/disk1
+      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node3/disk2:/data/disk2
+      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node3/disk3:/data/disk3
+      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node3/disk4:/data/disk4
+      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/logs/node3:/logs
+      - ./kms/rustfs-compat:/kms
+    ports:
+      - "${RUSTFS_RIO_V2_NODE3_PORT:-9303}:9000"
+
+  rustfs-rio-v2-node4:
+    <<: *rustfs-rio-v2-node
+    hostname: rustfs-rio-v2-node4
+    volumes:
+      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node4/disk1:/data/disk1
+      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node4/disk2:/data/disk2
+      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node4/disk3:/data/disk3
+      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/node4/disk4:/data/disk4
+      - ${RUSTFS_RIO_V2_DATASET:-./data/rustfs-rio-v2}/logs/node4:/logs
+      - ./kms/rustfs-compat:/kms
+    ports:
+      - "${RUSTFS_RIO_V2_NODE4_PORT:-9304}:9000"
+
+networks:
+  rustfs-rio-v2-net:
+    driver: bridge
@@ -0,0 +1,639 @@
+#!/usr/bin/env bash
+set -euo pipefail
+
+# High-concurrency S3 read/write stress runner for rio/rio-v2 format compatibility.
+# Write a manifest on an old endpoint, then verify the same manifest through rio-v2.
+
+MODE="write"
+ENDPOINT=""
+ACCESS_KEY="${AWS_ACCESS_KEY_ID:-}"
+SECRET_KEY="${AWS_SECRET_ACCESS_KEY:-}"
+BUCKET="compat-rw-stress"
+REGION="us-east-1"
+CONCURRENCY=64
+OUT_DIR=""
+WORK_DIR=""
+MANIFEST=""
+PROFILE="200g"
+OBJECT_SPEC=""
+DATA_PATTERN="compressible"
+ENCRYPTION="none"
+SSE_KMS_KEY_ID=""
+SSE_C_KEY_FILE=""
+CLIENT="mc"
+AWS_BIN="${AWS_BIN:-aws}"
+MC_BIN="${MC_BIN:-}"
+KEEP_PAYLOADS=false
+DRY_RUN=false
+RESUME=false
+
+usage() {
+  cat <<'USAGE'
+Usage:
+  .docker/compat/run_rw_compat_stress.sh --mode <write|verify|mixed> \
+    --endpoint <url> --access-key <ak> --secret-key <sk> [options]
+
+Modes:
+  write     Create bucket, upload objects concurrently, and write manifest.csv.
+  verify    Read objects concurrently from --endpoint and verify against manifest.csv.
+  mixed     Write and verify against the same endpoint.
+
+Required:
+  --endpoint              S3 endpoint URL, for example http://127.0.0.1:9100
+  --access-key            S3 access key
+  --secret-key            S3 secret key
+
+Core options:
+  --bucket                Bucket name (default: compat-rw-stress)
+  --region                Region (default: us-east-1)
+  --concurrency           Parallel object operations (default: 64)
+  --out-dir               Output directory (default: target/compat/rw-stress-<timestamp>)
+  --work-dir              Payload scratch directory (default: <out-dir>/payloads)
+  --manifest              Manifest path (default: <out-dir>/manifest.csv for write/mixed)
+  --profile               compact | 5g | 200g (default: 200g)
+  --object-spec           Override profile. Format: size:count,size:count
+                         Example: 1KiB:1024,1MiB:1024,64MiB:512,1GiB:64
+  --data-pattern          compressible | random | mixed (default: compressible)
+  --keep-payloads         Do not delete local payload files after upload
+  --resume                Skip write tasks that already have rows in <out-dir>/tasks/write-rows
+  --dry-run               Print planned tasks and commands without executing S3 operations
+  --client                mc | aws (default: mc)
+  --mc-bin                Path to mc binary (default: first tmp/mc.* or mc in PATH)
+  --aws-bin               Path to aws binary (used with --client aws)
+
+Encryption options:
+  --encryption            none | sse-s3 | sse-kms | sse-c (default: none)
+  --sse-kms-key-id        KMS key id for --encryption sse-kms
+  --sse-c-key-file        Raw 32-byte SSE-C key file for --encryption sse-c
+
+Examples:
+  # Generate the old RustFS beta5 dataset.
+  .docker/compat/run_rw_compat_stress.sh \
+    --mode write --endpoint http://127.0.0.1:9100 \
+    --access-key rustfsadmin --secret-key rustfsadmin \
+    --bucket compat-beta5 --concurrency 96
+
+  # Verify that dataset after mounting beta5 disks into the rio-v2 compose.
+  .docker/compat/run_rw_compat_stress.sh \
+    --mode verify --endpoint http://127.0.0.1:9300 \
+    --access-key rustfsadmin --secret-key rustfsadmin \
+    --bucket compat-beta5 --concurrency 96 \
+    --manifest target/compat/rw-stress-YYYYmmdd-HHMMSS/manifest.csv
+
+  # Faster smoke run.
+  .docker/compat/run_rw_compat_stress.sh \
+    --mode mixed --endpoint http://127.0.0.1:9300 \
+    --access-key rustfsadmin --secret-key rustfsadmin \
+    --profile compact --concurrency 16
+USAGE
+}
+
+die() {
+  echo "ERROR: $*" >&2
+  exit 1
+}
+
+require_cmd() {
+  command -v "$1" >/dev/null 2>&1 || die "command not found: $1"
+}
+
+parse_args() {
+  while [[ $# -gt 0 ]]; do
+    case "$1" in
+      --mode) MODE="$2"; shift 2 ;;
+      --endpoint) ENDPOINT="$2"; shift 2 ;;
+      --access-key) ACCESS_KEY="$2"; shift 2 ;;
+      --secret-key) SECRET_KEY="$2"; shift 2 ;;
+      --bucket) BUCKET="$2"; shift 2 ;;
+      --region) REGION="$2"; shift 2 ;;
+      --concurrency) CONCURRENCY="$2"; shift 2 ;;
+      --out-dir) OUT_DIR="$2"; shift 2 ;;
+      --work-dir) WORK_DIR="$2"; shift 2 ;;
+      --manifest) MANIFEST="$2"; shift 2 ;;
+      --profile) PROFILE="$2"; shift 2 ;;
+      --object-spec) OBJECT_SPEC="$2"; shift 2 ;;
+      --data-pattern) DATA_PATTERN="$2"; shift 2 ;;
+      --encryption) ENCRYPTION="$2"; shift 2 ;;
+      --sse-kms-key-id) SSE_KMS_KEY_ID="$2"; shift 2 ;;
+      --sse-c-key-file) SSE_C_KEY_FILE="$2"; shift 2 ;;
+      --client) CLIENT="$2"; shift 2 ;;
+      --mc-bin) MC_BIN="$2"; shift 2 ;;
+      --aws-bin) AWS_BIN="$2"; shift 2 ;;
+      --keep-payloads) KEEP_PAYLOADS=true; shift ;;
+      --resume) RESUME=true; shift ;;
+      --dry-run) DRY_RUN=true; shift ;;
+      -h|--help) usage; exit 0 ;;
+      *) die "unknown arg: $1" ;;
+    esac
+  done
+}
+
+validate_args() {
+  [[ "$MODE" =~ ^(write|verify|mixed)$ ]] || die "--mode must be write, verify, or mixed"
+  [[ "$CLIENT" =~ ^(mc|aws)$ ]] || die "--client must be mc or aws"
+  [[ "$PROFILE" =~ ^(compact|5g|200g)$ ]] || die "--profile must be compact, 5g, or 200g"
+  [[ "$DATA_PATTERN" =~ ^(compressible|random|mixed)$ ]] || die "--data-pattern must be compressible, random, or mixed"
+  [[ "$ENCRYPTION" =~ ^(none|sse-s3|sse-kms|sse-c)$ ]] || die "--encryption must be none, sse-s3, sse-kms, or sse-c"
+  [[ -n "$ENDPOINT" && -n "$ACCESS_KEY" && -n "$SECRET_KEY" ]] || die "--endpoint/--access-key/--secret-key are required"
+  [[ "$CONCURRENCY" =~ ^[0-9]+$ && "$CONCURRENCY" -gt 0 ]] || die "--concurrency must be a positive integer"
+  if [[ "$ENCRYPTION" == "sse-kms" && -z "$SSE_KMS_KEY_ID" ]]; then
+    die "--sse-kms-key-id is required for --encryption sse-kms"
+  fi
+  if [[ "$ENCRYPTION" == "sse-c" ]]; then
+    [[ -n "$SSE_C_KEY_FILE" && -f "$SSE_C_KEY_FILE" ]] || die "--sse-c-key-file must point to an existing key file"
+  fi
+  if [[ "$MODE" == "verify" && -z "$MANIFEST" ]]; then
+    die "--manifest is required for --mode verify"
+  fi
+}
+
+setup_paths() {
+  if [[ -z "$OUT_DIR" ]]; then
+    OUT_DIR="target/compat/rw-stress-$(date +%Y%m%d-%H%M%S)"
+  fi
+  if [[ -z "$WORK_DIR" ]]; then
+    WORK_DIR="$OUT_DIR/payloads"
+  fi
+  if [[ -z "$MANIFEST" ]]; then
+    MANIFEST="$OUT_DIR/manifest.csv"
+  fi
+
+  mkdir -p "$OUT_DIR" "$WORK_DIR" "$OUT_DIR/tasks" "$OUT_DIR/logs"
+  TASKS_FILE="$OUT_DIR/tasks/tasks.tsv"
+  WRITE_ROWS_DIR="$OUT_DIR/tasks/write-rows"
+  VERIFY_ROWS_DIR="$OUT_DIR/tasks/verify-rows"
+  MC_CONFIG_DIR_LOCAL="$OUT_DIR/mc-config"
+  MC_ALIAS="compat"
+  mkdir -p "$WRITE_ROWS_DIR" "$VERIFY_ROWS_DIR"
+}
+
+resolve_mc_bin() {
+  if [[ -n "$MC_BIN" ]]; then
+    echo "$MC_BIN"
+    return
+  fi
+
+  local candidate
+  candidate="$(find tmp -maxdepth 1 -type f -name 'mc.*' -perm -111 2>/dev/null | sort | tail -n 1 || true)"
+  if [[ -n "$candidate" ]]; then
+    echo "$candidate"
+    return
+  fi
+
+  command -v mc 2>/dev/null || true
+}
+
+size_to_bytes() {
+  local raw="$1"
+  local num unit
+  if [[ "$raw" =~ ^([0-9]+)(B|KiB|MiB|GiB|KB|MB|GB)?$ ]]; then
+    num="${BASH_REMATCH[1]}"
+    unit="${BASH_REMATCH[2]:-B}"
+  else
+    die "invalid size: $raw"
+  fi
+
+  case "$unit" in
+    B) echo "$num" ;;
+    KiB) echo $((num * 1024)) ;;
+    MiB) echo $((num * 1024 * 1024)) ;;
+    GiB) echo $((num * 1024 * 1024 * 1024)) ;;
+    KB) echo $((num * 1000)) ;;
+    MB) echo $((num * 1000 * 1000)) ;;
+    GB) echo $((num * 1000 * 1000 * 1000)) ;;
+    *) die "invalid size unit: $unit" ;;
+  esac
+}
+
+profile_spec() {
+  if [[ -n "$OBJECT_SPEC" ]]; then
+    echo "$OBJECT_SPEC"
+    return
+  fi
+
+  case "$PROFILE" in
+    compact)
+      echo "1KiB:64,1MiB:64,16MiB:16,128MiB:4"
+      ;;
+    5g)
+      echo "1KiB:1024,1MiB:255,16MiB:64,64MiB:28,1GiB:2"
+      ;;
+    200g)
+      echo "1KiB:1024,1MiB:1024,64MiB:512,1GiB:64,8GiB:12,6GiB:1"
+      ;;
+  esac
+}
+
+content_for_index() {
+  local index="$1"
+  case $((index % 3)) in
+    0) echo "txt|text/plain" ;;
+    1) echo "json|application/json" ;;
+    2) echo "bin|binary/octet-stream" ;;
+  esac
+}
+
+generate_tasks() {
+  local spec item size count bytes i content ext mime key seed index=0
+  spec="$(profile_spec)"
+  : > "$TASKS_FILE"
+
+  IFS=',' read -r -a items <<< "$spec"
+  for item in "${items[@]}"; do
+    item="${item//[[:space:]]/}"
+    [[ -n "$item" ]] || continue
+    [[ "$item" =~ ^([^:]+):([0-9]+)$ ]] || die "invalid object spec item: $item"
+    size="${BASH_REMATCH[1]}"
+    count="${BASH_REMATCH[2]}"
+    bytes="$(size_to_bytes "$size")"
+
+    for ((i = 1; i <= count; i++)); do
+      content="$(content_for_index "$index")"
+      ext="${content%%|*}"
+      mime="${content#*|}"
+      key="rw-stress/${size}/obj-$(printf '%06d' "$i").${ext}"
+      seed="${BUCKET}:${key}:${bytes}"
+      printf '%s\t%s\t%s\t%s\t%s\n' "$key" "$size" "$bytes" "$mime" "$seed" >> "$TASKS_FILE"
+      index=$((index + 1))
+    done
+  done
+}
+
+write_row_file_for_key() {
+  local key="$1"
+  echo "$WRITE_ROWS_DIR/${key//\//_}.csv"
+}
+
+prepare_write_input() {
+  WRITE_INPUT="$TASKS_FILE"
+  if [[ "$RESUME" != "true" ]]; then
+    return
+  fi
+
+  WRITE_INPUT="$OUT_DIR/tasks/write-input.tsv"
+  : > "$WRITE_INPUT"
+
+  local line key _size _bytes _mime _seed row_file
+  while IFS= read -r line || [[ -n "$line" ]]; do
+    IFS=$'\t' read -r key _size _bytes _mime _seed <<< "$line"
+    row_file="$(write_row_file_for_key "$key")"
+    [[ -s "$row_file" ]] && continue
+    printf '%s\n' "$line" >> "$WRITE_INPUT"
+  done < "$TASKS_FILE"
+}
+
+print_plan() {
+  local total_objects total_bytes profile_label
+  if [[ -f "$TASKS_FILE" ]]; then
+    total_objects="$(wc -l < "$TASKS_FILE" | tr -d ' ')"
+    total_bytes="$(awk -F '\t' '{sum += $3} END {print sum + 0}' "$TASKS_FILE")"
+    profile_label="$(profile_spec)"
+  else
+    total_objects="$(awk 'END {count = NR - 1; if (count < 0) count = 0; print count}' "$MANIFEST")"
+    total_bytes="$(awk -F ',' 'NR > 1 {sum += $3} END {print sum + 0}' "$MANIFEST")"
+    profile_label="from manifest"
+  fi
+
+  cat <<PLAN
+Mode:        $MODE
+Endpoint:    $ENDPOINT
+Bucket:      $BUCKET
+Profile:     $profile_label
+Objects:     $total_objects
+Bytes:       $total_bytes
+Concurrency: $CONCURRENCY
+Encryption:  $ENCRYPTION
+Client:      $CLIENT
+Manifest:    $MANIFEST
+Out dir:     $OUT_DIR
+Work dir:    $WORK_DIR
+PLAN
+}
+
+aws_base() {
+  AWS_ACCESS_KEY_ID="$ACCESS_KEY" AWS_SECRET_ACCESS_KEY="$SECRET_KEY" AWS_DEFAULT_REGION="$REGION" \
+    "$AWS_BIN" --endpoint-url "$ENDPOINT" "$@"
+}
+
+mc_base() {
+  "$MC_BIN" --config-dir "$MC_CONFIG_DIR_LOCAL" "$@"
+}
+
+aws_cp_args() {
+  case "$ENCRYPTION" in
+    none) ;;
+    sse-s3) printf '%s\n' "--sse" "AES256" ;;
+    sse-kms) printf '%s\n' "--sse" "aws:kms" "--sse-kms-key-id" "$SSE_KMS_KEY_ID" ;;
+    sse-c) printf '%s\n' "--sse-c" "AES256" "--sse-c-key" "fileb://$SSE_C_KEY_FILE" ;;
+  esac
+}
+
+mc_enc_target() {
+  printf '%s/%s/rw-stress/' "$MC_ALIAS" "$BUCKET"
+}
+
+mc_cp_args() {
+  local target
+  target="$(mc_enc_target)"
+  case "$ENCRYPTION" in
+    none) ;;
+    sse-s3) printf '%s\n' "--enc-s3" "$target" ;;
+    sse-kms) printf '%s\n' "--enc-kms" "${target}=${SSE_KMS_KEY_ID}" ;;
+    sse-c)
+      local key_b64
+      key_b64="$(base64 < "$SSE_C_KEY_FILE" | tr -d '\n')"
+      printf '%s\n' "--enc-c" "${target}=${key_b64}"
+      ;;
+  esac
+}
+
+aws_get_args() {
+  if [[ "$ENCRYPTION" == "sse-c" ]]; then
+    printf '%s\n' "--sse-c" "AES256" "--sse-c-key" "fileb://$SSE_C_KEY_FILE"
+  fi
+}
+
+mc_get_args() {
+  if [[ "$ENCRYPTION" == "sse-c" ]]; then
+    local target key_b64
+    target="$(mc_enc_target)"
+    key_b64="$(base64 < "$SSE_C_KEY_FILE" | tr -d '\n')"
+    printf '%s\n' "--enc-c" "${target}=${key_b64}"
+  fi
+}
+
+setup_mc_alias() {
+  if [[ "$CLIENT" != "mc" || "$DRY_RUN" == "true" ]]; then
+    return
+  fi
+  mkdir -p "$MC_CONFIG_DIR_LOCAL"
+  mc_base alias set "$MC_ALIAS" "$ENDPOINT" "$ACCESS_KEY" "$SECRET_KEY" --api S3v4 --path auto >/dev/null
+}
+
+create_bucket_if_needed() {
+  if [[ "$DRY_RUN" == "true" ]]; then
+    echo "[DRY-RUN] create bucket if missing: $BUCKET"
+    return
+  fi
+
+  if [[ "$CLIENT" == "mc" ]]; then
+    mc_base mb --ignore-existing --region "$REGION" "$MC_ALIAS/$BUCKET" >/dev/null
+    return
+  fi
+
+  if aws_base s3api head-bucket --bucket "$BUCKET" >/dev/null 2>&1; then
+    return
+  fi
+
+  aws_base s3api create-bucket --bucket "$BUCKET" >/dev/null
+}
+
+payload_path_for_key() {
+  local key="$1"
+  echo "$WORK_DIR/${key//\//_}"
+}
+
+generate_payload() {
+  local file="$1"
+  local bytes="$2"
+  local seed="$3"
+  local pattern="$DATA_PATTERN"
+  mkdir -p "$(dirname "$file")"
+
+  if [[ "$pattern" == "mixed" ]]; then
+    if [[ $((bytes % 2)) -eq 0 ]]; then
+      pattern="compressible"
+    else
+      pattern="random"
+    fi
+  fi
+
+  if [[ "$pattern" == "random" ]]; then
+    head -c "$bytes" /dev/zero | openssl enc -aes-256-ctr -nosalt -pass "pass:$seed" -out "$file"
+  else
+    yes "$seed payload-for-rio-compatibility" | tr '\n' ' ' | head -c "$bytes" > "$file"
+  fi
+}
+
+sha256_file() {
+  shasum -a 256 "$1" | awk '{print $1}'
+}
+
+sha256_object() {
+  local key="$1"
+  shift
+  if [[ "$CLIENT" == "mc" ]]; then
+    mc_base cat "$@" "$MC_ALIAS/$BUCKET/$key" | shasum -a 256 | awk '{print $1}'
+  else
+    aws_base s3 cp "s3://$BUCKET/$key" - --no-progress "$@" | shasum -a 256 | awk '{print $1}'
+  fi
+}
+
+collect_args() {
+  local generator="$1"
+  extra_args=()
+  while IFS= read -r arg; do
+    extra_args+=("$arg")
+  done < <("$generator")
+}
+
+write_one() {
+  local line="$1"
+  local key size bytes mime seed file sha row_file
+  local -a extra_args
+  IFS=$'\t' read -r key size bytes mime seed <<< "$line"
+  file="$(payload_path_for_key "$key")"
+  row_file="$(write_row_file_for_key "$key")"
+
+  if [[ "$DRY_RUN" == "true" ]]; then
+    echo "[DRY-RUN] upload $bytes bytes to s3://$BUCKET/$key content-type=$mime"
+    printf '%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "DRY_RUN" "$ENCRYPTION" > "$row_file"
+    return
+  fi
+
+  generate_payload "$file" "$bytes" "$seed"
+  sha="$(sha256_file "$file")"
+
+  if [[ "$CLIENT" == "mc" ]]; then
+    collect_args mc_cp_args
+    mc_base cp --quiet --attr "Content-Type=$mime" ${extra_args[@]+"${extra_args[@]}"} "$file" "$MC_ALIAS/$BUCKET/$key" \
+      > "$OUT_DIR/logs/${key//\//_}.put.log" 2>&1
+  else
+    collect_args aws_cp_args
+    aws_base s3 cp "$file" "s3://$BUCKET/$key" --no-progress --content-type "$mime" ${extra_args[@]+"${extra_args[@]}"} \
+      > "$OUT_DIR/logs/${key//\//_}.put.log" 2>&1
+  fi
+
+  printf '%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "$sha" "$ENCRYPTION" > "$row_file"
+  if [[ "$KEEP_PAYLOADS" != "true" ]]; then
+    rm -f "$file"
+  fi
+}
+
+verify_one() {
+  local line="$1"
+  local key size bytes mime expected encryption actual row_file
+  local -a extra_args
+  IFS=',' read -r key size bytes mime expected encryption <<< "$line"
+  row_file="$VERIFY_ROWS_DIR/${key//\//_}.csv"
+
+  if [[ "$DRY_RUN" == "true" ]]; then
+    echo "[DRY-RUN] verify s3://$BUCKET/$key expected=$expected"
+    printf '%s,%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "$expected" "DRY_RUN" "dry-run" > "$row_file"
+    return
+  fi
+
+  if [[ "$CLIENT" == "mc" ]]; then
+    collect_args mc_get_args
+  else
+    collect_args aws_get_args
+  fi
+  if ! actual="$(sha256_object "$key" ${extra_args[@]+"${extra_args[@]}"})"; then
+    printf '%s,%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "$expected" "ERROR" "download failed" > "$row_file"
+    return 1
+  fi
+
+  if [[ "$actual" == "$expected" ]]; then
+    printf '%s,%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "$expected" "$actual" "ok" > "$row_file"
+  else
+    printf '%s,%s,%s,%s,%s,%s,%s\n' "$key" "$size" "$bytes" "$mime" "$expected" "$actual" "sha256-mismatch" > "$row_file"
+    return 1
+  fi
+}
+
+run_parallel_tasks() {
+  local action="$1"
+  local input_file="$2"
+  local failure_file="$OUT_DIR/tasks/${action}.failed"
+  local line
+  rm -f "$failure_file"
+
+  while IFS= read -r line || [[ -n "$line" ]]; do
+    while [[ "$(jobs -rp | wc -l | tr -d ' ')" -ge "$CONCURRENCY" ]]; do
+      sleep 0.2
+    done
+
+    ({
+      if [[ "$action" == "write" ]]; then
+        write_one "$line"
+      else
+        verify_one "$line"
+      fi
+    } || touch "$failure_file") &
+  done < "$input_file"
+
+  wait
+  [[ ! -f "$failure_file" ]]
+}
+
+combine_write_manifest() {
+  echo "key,size,bytes,content_type,sha256,encryption" > "$MANIFEST"
+  find "$WRITE_ROWS_DIR" -type f -name '*.csv' -print0 | sort -z | xargs -0 cat >> "$MANIFEST"
+
+  local expected actual
+  expected="$(wc -l < "$TASKS_FILE" | tr -d ' ')"
+  actual="$(( $(wc -l < "$MANIFEST" | tr -d ' ') - 1 ))"
+  if [[ "$actual" -ne "$expected" ]]; then
+    die "manifest row count mismatch: expected $expected, got $actual"
+  fi
+}
+
+prepare_verify_input() {
+  VERIFY_INPUT="$OUT_DIR/tasks/verify-input.csv"
+  tail -n +2 "$MANIFEST" > "$VERIFY_INPUT"
+}
+
+combine_verify_summary() {
+  VERIFY_SUMMARY="$OUT_DIR/verify-summary.csv"
+  echo "key,size,bytes,content_type,expected_sha256,actual_sha256,status" > "$VERIFY_SUMMARY"
+  find "$VERIFY_ROWS_DIR" -type f -name '*.csv' -print0 | sort -z | xargs -0 cat >> "$VERIFY_SUMMARY"
+
+  local failed
+  if [[ "$DRY_RUN" == "true" ]]; then
+    echo "Verification dry run complete. Summary: $VERIFY_SUMMARY"
+    return
+  fi
+
+  failed="$(awk -F ',' 'NR > 1 && $7 != "ok" {count++} END {print count + 0}' "$VERIFY_SUMMARY")"
+  local expected actual
+  expected="$(wc -l < "$VERIFY_INPUT" | tr -d ' ')"
+  actual="$(( $(wc -l < "$VERIFY_SUMMARY" | tr -d ' ') - 1 ))"
+  if [[ "$actual" -ne "$expected" ]]; then
+    echo "Verification row count mismatch: expected $expected, got $actual" >&2
+    return 1
+  fi
+  if [[ "$failed" -ne 0 ]]; then
+    echo "Verification failed: $failed object(s). See $VERIFY_SUMMARY" >&2
+    return 1
+  fi
+  echo "Verification passed. Summary: $VERIFY_SUMMARY"
+}
+
+export_functions() {
+  export ENDPOINT ACCESS_KEY SECRET_KEY BUCKET REGION ENCRYPTION SSE_KMS_KEY_ID SSE_C_KEY_FILE AWS_BIN
+  export CLIENT MC_BIN MC_CONFIG_DIR_LOCAL MC_ALIAS
+  export WORK_DIR OUT_DIR WRITE_ROWS_DIR VERIFY_ROWS_DIR DATA_PATTERN KEEP_PAYLOADS DRY_RUN
+  export -f aws_base mc_base aws_cp_args aws_get_args mc_enc_target mc_cp_args mc_get_args payload_path_for_key generate_payload sha256_file sha256_object write_one verify_one
+}
+
+main() {
+  parse_args "$@"
+  validate_args
+  setup_paths
+
+  if [[ "$DRY_RUN" != "true" ]]; then
+    if [[ "$CLIENT" == "mc" ]]; then
+      MC_BIN="$(resolve_mc_bin)"
+      [[ -n "$MC_BIN" ]] || die "mc binary not found; pass --mc-bin or put mc in PATH"
+      require_cmd "$MC_BIN"
+    else
+      require_cmd "$AWS_BIN"
+    fi
+    require_cmd shasum
+    require_cmd head
+    require_cmd yes
+    require_cmd openssl
+  elif [[ "$CLIENT" == "mc" ]]; then
+    MC_BIN="$(resolve_mc_bin)"
+  fi
+
+  setup_mc_alias
+
+  if [[ "$MODE" == "write" || "$MODE" == "mixed" ]]; then
+    local write_failed=false
+    generate_tasks
+    prepare_write_input
+    print_plan
+    create_bucket_if_needed
+    export_functions
+    if ! run_parallel_tasks write "$WRITE_INPUT"; then
+      write_failed=true
+    fi
+    combine_write_manifest
+    echo "Write manifest: $MANIFEST"
+    if [[ "$write_failed" == "true" ]]; then
+      die "one or more write tasks failed; see $OUT_DIR/logs"
+    fi
+  fi
+
+  if [[ "$MODE" == "verify" || "$MODE" == "mixed" ]]; then
+    local verify_failed=false
+    [[ -f "$MANIFEST" ]] || die "manifest not found: $MANIFEST"
+    if [[ "$MODE" == "verify" ]]; then
+      cp "$MANIFEST" "$OUT_DIR/manifest.csv"
+      MANIFEST="$OUT_DIR/manifest.csv"
+    fi
+    prepare_verify_input
+    print_plan
+    export_functions
+    if ! run_parallel_tasks verify "$VERIFY_INPUT"; then
+      verify_failed=true
+    fi
+    combine_verify_summary
+    if [[ "$verify_failed" == "true" ]]; then
+      die "one or more verify tasks failed; see $VERIFY_SUMMARY"
+    fi
+  fi
+}
+
+main "$@"
@@ -0,0 +1,52 @@
+services:
+  rustfs:
+    image: rustfs/rustfs:1.0.0-alpha.99-glibc
+    container_name: rustfs-issue-2715-test
+    security_opt:
+      - "no-new-privileges:true"
+    ports:
+      - "19000:9000"
+      - "19001:9001"
+    environment:
+      - RUSTFS_VOLUMES=/data/rustfs{0...8}
+      - RUSTFS_ADDRESS=0.0.0.0:9000
+      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_ACCESS_KEY=admin
+      - RUSTFS_SECRET_KEY=admin
+      - RUSTFS_OBS_LOGGER_LEVEL=info
+      - RUSTFS_OBS_ENDPOINT=http://otel-collector:4318
+      - RUSTFS_OBS_PROFILING_ENDPOINT=http://pyroscope:4040
+      - RUSTFS_STORAGE_CLASS_STANDARD=EC:2
+      - RUSTFS_STORAGE_CLASS_RRS=EC:1
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true
+      - RUSTFS_OBS_LOG_DIRECTORY=/opt/rustfs/logs
+    extra_hosts:
+      - "otel-collector:host-gateway"
+      - "pyroscope:host-gateway"
+    volumes:
+      - ./deploy/data/issue-2715/rustfs0:/data/rustfs0
+      - ./deploy/data/issue-2715/rustfs1:/data/rustfs1
+      - ./deploy/data/issue-2715/rustfs2:/data/rustfs2
+      - ./deploy/data/issue-2715/rustfs3:/data/rustfs3
+      - ./deploy/data/issue-2715/rustfs4:/data/rustfs4
+      - ./deploy/data/issue-2715/rustfs5:/data/rustfs5
+      - ./deploy/data/issue-2715/rustfs6:/data/rustfs6
+      - ./deploy/data/issue-2715/rustfs7:/data/rustfs7
+      - ./deploy/data/issue-2715/rustfs8:/data/rustfs8
+      - ./deploy/logs/issue-2715:/opt/rustfs/logs
+    restart: unless-stopped
+    healthcheck:
+      test:
+        [
+          "CMD",
+          "sh",
+          "-c",
+          "curl -f http://127.0.0.1:9000/health && curl -f http://127.0.0.1:9001/rustfs/console/health"
+        ]
+      interval: 30s
+      timeout: 10s
+      retries: 3
+      start_period: 40s
@@ -0,0 +1 @@
+data/
@@ -0,0 +1,106 @@
+# Issue 2815 Local Docker Verification
+
+## Purpose
+
+This directory contains the local distributed Docker verification assets used to validate issue `#2815` against the current source build.
+
+The target behavior is:
+
+- 4-node distributed cluster starts successfully
+- `/health/ready` becomes reachable on each node
+- logs no longer contain `storage_info failed: Io error: wrong msgpack marker FixArray(1)`
+- internode RPC authentication succeeds with an explicit non-default RPC secret
+
+## Files
+
+- `docker-compose.yml`: 4-node distributed cluster using a locally built image
+
+## Data Directories
+
+Create the bind-mount directories before `docker compose up`:
+
+```bash
+mkdir -p .docker/test/issues-2815/data/rustfs{1..4}-disk{0..3}
+```
+
+## Build
+
+Apple Silicon / arm64 host:
+
+```bash
+docker build --platform linux/arm64 -f Dockerfile.source -t rustfs-issue-2815-local .
+```
+
+If you intentionally want amd64 emulation:
+
+```bash
+docker build --platform linux/amd64 -f Dockerfile.source -t rustfs-issue-2815-local .
+```
+
+## Run
+
+```bash
+docker compose -f .docker/test/issues-2815/docker-compose.yml up -d
+```
+
+If the image platform is not `linux/arm64`, align compose explicitly:
+
+```bash
+RUSTFS_DOCKER_PLATFORM=linux/amd64 docker compose -f .docker/test/issues-2815/docker-compose.yml up -d
+```
+
+## Health Checks
+
+Container-level healthcheck is now included and probes:
+
+```bash
+curl -fsS http://127.0.0.1:9000/health
+```
+
+Manual checks:
+
+```bash
+curl -i http://127.0.0.1:9101/health/ready
+curl -i http://127.0.0.1:9102/health/ready
+curl -i http://127.0.0.1:9103/health/ready
+curl -i http://127.0.0.1:9104/health/ready
+```
+
+## RPC Secret Requirement
+
+The current source build no longer reproduces the original `FixArray(1)` decode error from issue `#2815`.
+
+Earlier local Docker attempts failed during erasure bootstrap with:
+
+```text
+No valid auth token
+store init failed to load formats after 10 retries: erasure read quorum
+```
+
+Root cause:
+
+- RPC authentication rejects the default secret `rustfsadmin`
+- distributed local Docker validation therefore needs an explicit non-default secret
+
+This compose now sets both:
+
+- `RUSTFS_SECRET_KEY=issue-2815-secret`
+- `RUSTFS_RPC_SECRET=issue-2815-rpc-secret`
+
+With those values in place, the current 4-node local Docker cluster reaches healthy state and `/health/ready` returns `200`.
+
+In other words:
+
+- `RUSTFS_ACCESS_KEY` may still be `rustfsadmin` for local service credentials if desired
+- `RUSTFS_SECRET_KEY` can still be used for service credentials
+- but RPC authentication must not resolve to the default secret value `rustfsadmin`
+- if `RUSTFS_RPC_SECRET` is unset, the code falls back to `RUSTFS_SECRET_KEY`
+- so at least one of them must provide a non-default shared secret for internode RPC signing
+
+## Suggested Debug Commands
+
+```bash
+docker compose -f .docker/test/issues-2815/docker-compose.yml ps
+docker compose -f .docker/test/issues-2815/docker-compose.yml logs --no-color --tail=200
+docker compose -f .docker/test/issues-2815/docker-compose.yml down -v
+```
@@ -0,0 +1,120 @@
+services:
+  rustfs1:
+    image: rustfs-issue-2815-local
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
+    hostname: rustfs1
+    container_name: rustfs-issue-2815-rustfs1
+    environment:
+      RUSTFS_ADDRESS: "0.0.0.0:9000"
+      RUSTFS_ACCESS_KEY: "rustfsadmin"
+      RUSTFS_SECRET_KEY: "issue-2815-secret"
+      RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
+      RUSTFS_CONSOLE_ENABLE: "false"
+      RUST_LOG: "info"
+      RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
+      RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
+    volumes:
+      - ./data/rustfs1-disk0:/data/rustfs0
+      - ./data/rustfs1-disk1:/data/rustfs1
+      - ./data/rustfs1-disk2:/data/rustfs2
+      - ./data/rustfs1-disk3:/data/rustfs3
+    networks: [rustfs-issue-2815-net]
+    ports:
+      - "9101:9000"
+    healthcheck:
+      test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 8
+      start_period: 30s
+
+  rustfs2:
+    image: rustfs-issue-2815-local
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
+    hostname: rustfs2
+    container_name: rustfs-issue-2815-rustfs2
+    environment:
+      RUSTFS_ADDRESS: "0.0.0.0:9000"
+      RUSTFS_ACCESS_KEY: "rustfsadmin"
+      RUSTFS_SECRET_KEY: "issue-2815-secret"
+      RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
+      RUSTFS_CONSOLE_ENABLE: "false"
+      RUST_LOG: "info"
+      RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
+      RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
+    volumes:
+      - ./data/rustfs2-disk0:/data/rustfs0
+      - ./data/rustfs2-disk1:/data/rustfs1
+      - ./data/rustfs2-disk2:/data/rustfs2
+      - ./data/rustfs2-disk3:/data/rustfs3
+    networks: [rustfs-issue-2815-net]
+    ports:
+      - "9102:9000"
+    healthcheck:
+      test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 8
+      start_period: 30s
+
+  rustfs3:
+    image: rustfs-issue-2815-local
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
+    hostname: rustfs3
+    container_name: rustfs-issue-2815-rustfs3
+    environment:
+      RUSTFS_ADDRESS: "0.0.0.0:9000"
+      RUSTFS_ACCESS_KEY: "rustfsadmin"
+      RUSTFS_SECRET_KEY: "issue-2815-secret"
+      RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
+      RUSTFS_CONSOLE_ENABLE: "false"
+      RUST_LOG: "info"
+      RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
+      RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
+    volumes:
+      - ./data/rustfs3-disk0:/data/rustfs0
+      - ./data/rustfs3-disk1:/data/rustfs1
+      - ./data/rustfs3-disk2:/data/rustfs2
+      - ./data/rustfs3-disk3:/data/rustfs3
+    networks: [rustfs-issue-2815-net]
+    ports:
+      - "9103:9000"
+    healthcheck:
+      test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 8
+      start_period: 30s
+
+  rustfs4:
+    image: rustfs-issue-2815-local
+    platform: ${RUSTFS_DOCKER_PLATFORM:-linux/arm64}
+    hostname: rustfs4
+    container_name: rustfs-issue-2815-rustfs4
+    environment:
+      RUSTFS_ADDRESS: "0.0.0.0:9000"
+      RUSTFS_ACCESS_KEY: "rustfsadmin"
+      RUSTFS_SECRET_KEY: "issue-2815-secret"
+      RUSTFS_RPC_SECRET: "issue-2815-rpc-secret"
+      RUSTFS_CONSOLE_ENABLE: "false"
+      RUST_LOG: "info"
+      RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
+      RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
+    volumes:
+      - ./data/rustfs4-disk0:/data/rustfs0
+      - ./data/rustfs4-disk1:/data/rustfs1
+      - ./data/rustfs4-disk2:/data/rustfs2
+      - ./data/rustfs4-disk3:/data/rustfs3
+    networks: [rustfs-issue-2815-net]
+    ports:
+      - "9104:9000"
+    healthcheck:
+      test: ["CMD", "sh", "-c", "curl -fsS http://127.0.0.1:9000/health || exit 1"]
+      interval: 15s
+      timeout: 5s
+      retries: 8
+      start_period: 30s
+
+networks:
+  rustfs-issue-2815-net:
+    name: rustfs-issue-2815-net
@@ -0,0 +1,183 @@
+# Site Replication Docker Compose Test
+
+## Purpose
+
+This directory contains a local three-site RustFS site replication check. It is intended to verify the admin site-replication flow against real containers:
+
+- three independent RustFS sites start successfully
+- the MinIO-compatible `mc admin replicate add` command configures all three sites
+- a bucket and object written to site 1 are replicated to site 2 and site 3
+- `mc admin replicate status` can read the resulting site-replication state
+
+The compose file uses named volumes so the test does not require preparing host bind-mount directories.
+Because Docker named volumes usually share the same physical device in local desktop environments, this test compose defaults `RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true`. Keep that setting limited to local test and CI environments.
+
+## Files
+
+- `docker-compose.yml`: three RustFS sites, a volume permission helper, and a one-shot setup/check container
+- `run-object-flow-check.sh`: host-side upload/download verification for replicated 10 MiB to 100 MiB objects
+
+## Ports
+
+Default host endpoints:
+
+- Site 1 API: `http://127.0.0.1:9000`
+- Site 1 Console: `http://127.0.0.1:9001`
+- Site 2 API: `http://127.0.0.1:9010`
+- Site 2 Console: `http://127.0.0.1:9011`
+- Site 3 API: `http://127.0.0.1:9020`
+- Site 3 Console: `http://127.0.0.1:9021`
+
+Default credentials are `rustfsadmin` / `rustfsadmin`. These are for local testing only.
+
+## Run
+
+From the repository root:
+
+```bash
+docker compose -f .docker/test/site-replication/docker-compose.yml up
+```
+
+To test a locally built image instead of Docker Hub `rustfs/rustfs:latest`, set `RUSTFS_SITE_REPL_IMAGE`:
+
+```bash
+docker build -f Dockerfile.source -t rustfs-site-repl-local:latest .
+RUSTFS_SITE_REPL_IMAGE=rustfs-site-repl-local:latest \
+docker compose -f .docker/test/site-replication/docker-compose.yml up
+```
+
+For a detached run:
+
+```bash
+docker compose -f .docker/test/site-replication/docker-compose.yml up -d
+docker compose -f .docker/test/site-replication/docker-compose.yml logs -f site-replication-setup
+```
+
+## Test Flow
+
+The compose stack performs these steps:
+
+1. `site-replication-volume-permission-helper` fixes ownership on all named volumes for the RustFS runtime user.
+2. `rustfs-site1`, `rustfs-site2`, and `rustfs-site3` start as separate RustFS sites.
+3. Each site exposes its S3 API and Console on a unique host port.
+4. Health checks wait for `/health/ready` on each RustFS container.
+5. `site-replication-setup` configures `mc` aliases for all three sites.
+6. The setup container waits until `mc admin info` succeeds for all sites.
+7. It runs:
+
+```bash
+mc admin replicate add site1 site2 site3
+```
+
+8. It creates the test bucket on site 1 and uploads `from-site1.txt`.
+9. It polls site 2 and site 3 until the replicated object is visible.
+10. It prints `mc admin replicate status site1`.
+
+The setup container exits with status `0` only after the object replication check passes.
+
+## Object Flow Check
+
+After the compose setup succeeds, run the larger object flow check from the repository root:
+
+```bash
+.docker/test/site-replication/run-object-flow-check.sh
+```
+
+The script creates five local files and uploads them from different sites:
+
+- 10 MiB from site 1
+- 25 MiB from site 2
+- 50 MiB from site 3
+- 75 MiB from site 1
+- 100 MiB from site 2
+
+For each uploaded object, the script waits for replication to the other two sites, downloads the object from those sites, and verifies both byte size and SHA-256 checksum. It uses a temporary `mc` config directory, so it does not overwrite existing host aliases.
+
+The default bucket is `site-repl-flow-check`. Override it when needed:
+
+```bash
+RUSTFS_SITE_REPL_FLOW_BUCKET='site-repl-large-flow' \
+.docker/test/site-replication/run-object-flow-check.sh
+```
+
+The script keeps uploaded objects under a timestamped prefix. Override the prefix for repeatable runs:
+
+```bash
+RUSTFS_SITE_REPL_FLOW_PREFIX='manual-check-001' \
+.docker/test/site-replication/run-object-flow-check.sh
+```
+
+If replication is slow on the local machine, increase polling:
+
+```bash
+RUSTFS_SITE_REPL_WAIT_ATTEMPTS=180 \
+RUSTFS_SITE_REPL_WAIT_SLEEP_SECONDS=2 \
+.docker/test/site-replication/run-object-flow-check.sh
+```
+
+## Optional Settings
+
+Override local test credentials:
+
+```bash
+RUSTFS_SITE_REPL_ACCESS_KEY='localadmin' \
+RUSTFS_SITE_REPL_SECRET_KEY='localadmin-secret' \
+docker compose -f .docker/test/site-replication/docker-compose.yml up
+```
+
+Use a different test bucket:
+
+```bash
+RUSTFS_SITE_REPL_BUCKET='site-repl-check' \
+docker compose -f .docker/test/site-replication/docker-compose.yml up
+```
+
+Enable ILM expiry rule replication during site setup:
+
+```bash
+RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY=true \
+docker compose -f .docker/test/site-replication/docker-compose.yml up
+```
+
+Use this only when the test needs lifecycle expiry metadata included in site replication.
+
+## Manual Checks
+
+After the setup container succeeds, you can inspect the sites with `mc` from the host:
+
+```bash
+mc alias set site1 http://127.0.0.1:9000 rustfsadmin rustfsadmin
+mc alias set site2 http://127.0.0.1:9010 rustfsadmin rustfsadmin
+mc alias set site3 http://127.0.0.1:9020 rustfsadmin rustfsadmin
+
+mc admin replicate info site1
+mc admin replicate status site1
+mc stat site2/site-repl-demo/from-site1.txt
+mc stat site3/site-repl-demo/from-site1.txt
+```
+
+After larger object flow checks, replication should converge without a growing queue:
+
+```bash
+mc admin replicate status site1
+mc admin replicate status site2
+mc admin replicate status site3
+```
+
+Useful Docker commands:
+
+```bash
+docker compose -f .docker/test/site-replication/docker-compose.yml ps
+docker compose -f .docker/test/site-replication/docker-compose.yml logs --no-color --tail=200
+docker compose -f .docker/test/site-replication/docker-compose.yml logs --no-color site-replication-setup
+```
+
+## Cleanup
+
+Remove containers and named volumes:
+
+```bash
+docker compose -f .docker/test/site-replication/docker-compose.yml down -v
+```
+
+Use `down -v` before rerunning the full setup from scratch. Site replication state is persisted in the named volumes, so rerunning without deleting volumes may attempt to add an already-configured replication topology.
@@ -0,0 +1,243 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+services:
+  rustfs-site1:
+    image: ${RUSTFS_SITE_REPL_IMAGE:-rustfs/rustfs:latest}
+    container_name: rustfs-site-repl-1
+    security_opt:
+      - "no-new-privileges:true"
+    ports:
+      - "9000:9000"
+      - "9001:9001"
+    environment:
+      - RUSTFS_VOLUMES=/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=0.0.0.0:9000
+      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_SITE_REPL_LOG_LEVEL:-info}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+    volumes:
+      - site1_data_0:/data/rustfs0
+      - site1_data_1:/data/rustfs1
+      - site1_data_2:/data/rustfs2
+      - site1_data_3:/data/rustfs3
+      - site1_logs:/app/logs
+    networks:
+      - rustfs-site-replication
+    restart: unless-stopped
+    healthcheck:
+      test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health/ready"]
+      interval: 10s
+      timeout: 5s
+      retries: 30
+      start_period: 20s
+    depends_on:
+      site-replication-volume-permission-helper:
+        condition: service_completed_successfully
+
+  rustfs-site2:
+    image: ${RUSTFS_SITE_REPL_IMAGE:-rustfs/rustfs:latest}
+    container_name: rustfs-site-repl-2
+    security_opt:
+      - "no-new-privileges:true"
+    ports:
+      - "9010:9000"
+      - "9011:9001"
+    environment:
+      - RUSTFS_VOLUMES=/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=0.0.0.0:9000
+      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_SITE_REPL_LOG_LEVEL:-info}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+    volumes:
+      - site2_data_0:/data/rustfs0
+      - site2_data_1:/data/rustfs1
+      - site2_data_2:/data/rustfs2
+      - site2_data_3:/data/rustfs3
+      - site2_logs:/app/logs
+    networks:
+      - rustfs-site-replication
+    restart: unless-stopped
+    healthcheck:
+      test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health/ready"]
+      interval: 10s
+      timeout: 5s
+      retries: 30
+      start_period: 20s
+    depends_on:
+      site-replication-volume-permission-helper:
+        condition: service_completed_successfully
+
+  rustfs-site3:
+    image: ${RUSTFS_SITE_REPL_IMAGE:-rustfs/rustfs:latest}
+    container_name: rustfs-site-repl-3
+    security_opt:
+      - "no-new-privileges:true"
+    ports:
+      - "9020:9000"
+      - "9021:9001"
+    environment:
+      - RUSTFS_VOLUMES=/data/rustfs{0...3}
+      - RUSTFS_ADDRESS=0.0.0.0:9000
+      - RUSTFS_CONSOLE_ADDRESS=0.0.0.0:9001
+      - RUSTFS_CONSOLE_ENABLE=true
+      - RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS=*
+      - RUSTFS_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_SITE_REPL_LOG_LEVEL:-info}
+      - RUSTFS_UNSAFE_BYPASS_DISK_CHECK=${RUSTFS_UNSAFE_BYPASS_DISK_CHECK:-true}
+    volumes:
+      - site3_data_0:/data/rustfs0
+      - site3_data_1:/data/rustfs1
+      - site3_data_2:/data/rustfs2
+      - site3_data_3:/data/rustfs3
+      - site3_logs:/app/logs
+    networks:
+      - rustfs-site-replication
+    restart: unless-stopped
+    healthcheck:
+      test: ["CMD", "curl", "-fsS", "http://127.0.0.1:9000/health/ready"]
+      interval: 10s
+      timeout: 5s
+      retries: 30
+      start_period: 20s
+    depends_on:
+      site-replication-volume-permission-helper:
+        condition: service_completed_successfully
+
+  site-replication-setup:
+    image: minio/mc:latest
+    container_name: rustfs-site-repl-setup
+    depends_on:
+      rustfs-site1:
+        condition: service_healthy
+      rustfs-site2:
+        condition: service_healthy
+      rustfs-site3:
+        condition: service_healthy
+    environment:
+      - RUSTFS_SITE_REPL_ACCESS_KEY=${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SITE_REPL_SECRET_KEY=${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}
+      - RUSTFS_SITE_REPL_BUCKET=${RUSTFS_SITE_REPL_BUCKET:-site-repl-demo}
+      - RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY=${RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY:-false}
+    networks:
+      - rustfs-site-replication
+    entrypoint: ["/bin/sh", "-c"]
+    command:
+      - |
+        set -eu
+
+        access_key="$${RUSTFS_SITE_REPL_ACCESS_KEY}"
+        secret_key="$${RUSTFS_SITE_REPL_SECRET_KEY}"
+        bucket="$${RUSTFS_SITE_REPL_BUCKET}"
+
+        mc alias set site1 http://rustfs-site1:9000 "$${access_key}" "$${secret_key}"
+        mc alias set site2 http://rustfs-site2:9000 "$${access_key}" "$${secret_key}"
+        mc alias set site3 http://rustfs-site3:9000 "$${access_key}" "$${secret_key}"
+
+        for site in site1 site2 site3; do
+          until mc ls "$${site}" >/dev/null 2>&1; do
+            echo "waiting for $${site} S3 API"
+            sleep 2
+          done
+        done
+
+        ilm_flag=""
+        if [ "$${RUSTFS_SITE_REPL_ENABLE_ILM_EXPIRY}" = "true" ]; then
+          ilm_flag="--replicate-ilm-expiry"
+        fi
+
+        echo "configuring 3-site replication"
+        if ! mc admin replicate add site1 site2 site3 $${ilm_flag}; then
+          echo "replicate add failed; showing current replication info before exiting"
+          mc admin replicate info site1 || true
+          exit 1
+        fi
+
+        mc mb --ignore-existing "site1/$${bucket}"
+        printf 'rustfs 3-site replication check\n' > /tmp/site-replication-check.txt
+        mc cp /tmp/site-replication-check.txt "site1/$${bucket}/from-site1.txt"
+
+        for site in site2 site3; do
+          for attempt in $$(seq 1 60); do
+            if mc stat "$${site}/$${bucket}/from-site1.txt" >/dev/null 2>&1; then
+              echo "$${site} received replicated object"
+              break
+            fi
+            if [ "$${attempt}" = "60" ]; then
+              echo "$${site} did not receive replicated object in time"
+              exit 1
+            fi
+            sleep 2
+          done
+        done
+
+        mc admin replicate status site1
+        echo "3-site replication example is ready"
+    restart: "no"
+
+  site-replication-volume-permission-helper:
+    image: alpine:3.23.4
+    volumes:
+      - site1_data_0:/site1/data0
+      - site1_data_1:/site1/data1
+      - site1_data_2:/site1/data2
+      - site1_data_3:/site1/data3
+      - site1_logs:/site1/logs
+      - site2_data_0:/site2/data0
+      - site2_data_1:/site2/data1
+      - site2_data_2:/site2/data2
+      - site2_data_3:/site2/data3
+      - site2_logs:/site2/logs
+      - site3_data_0:/site3/data0
+      - site3_data_1:/site3/data1
+      - site3_data_2:/site3/data2
+      - site3_data_3:/site3/data3
+      - site3_logs:/site3/logs
+    command: >
+      sh -c "
+        chown -R 10001:10001 /site1 /site2 /site3 &&
+        echo 'site replication volume permissions fixed'
+      "
+    networks:
+      - rustfs-site-replication
+    restart: "no"
+
+networks:
+  rustfs-site-replication:
+
+volumes:
+  site1_data_0:
+  site1_data_1:
+  site1_data_2:
+  site1_data_3:
+  site1_logs:
+  site2_data_0:
+  site2_data_1:
+  site2_data_2:
+  site2_data_3:
+  site2_logs:
+  site3_data_0:
+  site3_data_1:
+  site3_data_2:
+  site3_data_3:
+  site3_logs:
@@ -0,0 +1,228 @@
+#!/bin/sh
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+set -eu
+
+ACCESS_KEY="${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}"
+SECRET_KEY="${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}"
+BUCKET="${RUSTFS_SITE_REPL_FLOW_BUCKET:-site-repl-flow-check}"
+DELETE_BUCKET="${RUSTFS_SITE_REPL_DELETE_BUCKET:-site-repl-delete-$(date +%Y%m%d-%H%M%S)-$$}"
+PREFIX="${RUSTFS_SITE_REPL_FLOW_PREFIX:-flow-$(date +%Y%m%d-%H%M%S)}"
+WAIT_ATTEMPTS="${RUSTFS_SITE_REPL_WAIT_ATTEMPTS:-90}"
+WAIT_SLEEP_SECONDS="${RUSTFS_SITE_REPL_WAIT_SLEEP_SECONDS:-2}"
+
+SITE1_ENDPOINT="${RUSTFS_SITE1_ENDPOINT:-http://127.0.0.1:9000}"
+SITE2_ENDPOINT="${RUSTFS_SITE2_ENDPOINT:-http://127.0.0.1:9010}"
+SITE3_ENDPOINT="${RUSTFS_SITE3_ENDPOINT:-http://127.0.0.1:9020}"
+
+command_exists() {
+  command -v "$1" >/dev/null 2>&1
+}
+
+require_command() {
+  if ! command_exists "$1"; then
+    echo "missing required command: $1" >&2
+    exit 1
+  fi
+}
+
+checksum_file() {
+  if command_exists sha256sum; then
+    sha256sum "$1" | awk '{print $1}'
+  elif command_exists shasum; then
+    shasum -a 256 "$1" | awk '{print $1}'
+  else
+    echo "missing required command: sha256sum or shasum" >&2
+    exit 1
+  fi
+}
+
+site_endpoint() {
+  case "$1" in
+    site1) printf '%s\n' "$SITE1_ENDPOINT" ;;
+    site2) printf '%s\n' "$SITE2_ENDPOINT" ;;
+    site3) printf '%s\n' "$SITE3_ENDPOINT" ;;
+    *) echo "unknown site alias: $1" >&2; exit 1 ;;
+  esac
+}
+
+other_sites() {
+  case "$1" in
+    site1) printf '%s\n' "site2 site3" ;;
+    site2) printf '%s\n' "site1 site3" ;;
+    site3) printf '%s\n' "site1 site2" ;;
+    *) echo "unknown source site: $1" >&2; exit 1 ;;
+  esac
+}
+
+wait_for_object() {
+  site="$1"
+  object="$2"
+  attempt=1
+
+  while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
+    if mc stat "$site/$BUCKET/$object" >/dev/null 2>&1; then
+      return 0
+    fi
+    sleep "$WAIT_SLEEP_SECONDS"
+    attempt=$((attempt + 1))
+  done
+
+  echo "object was not replicated in time: $site/$BUCKET/$object" >&2
+  return 1
+}
+
+wait_for_bucket() {
+  site="$1"
+  bucket="${2:-$BUCKET}"
+  attempt=1
+
+  while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
+    if mc stat "$site/$bucket" >/dev/null 2>&1; then
+      return 0
+    fi
+    sleep "$WAIT_SLEEP_SECONDS"
+    attempt=$((attempt + 1))
+  done
+
+  echo "bucket was not replicated in time: $site/$bucket" >&2
+  return 1
+}
+
+wait_for_bucket_delete() {
+  site="$1"
+  bucket="$2"
+  attempt=1
+
+  while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
+    if result="$(mc stat --json "$site/$bucket" 2>&1)"; then
+      :
+    else
+      case "$result" in
+        *NoSuchBucket*) return 0 ;;
+      esac
+    fi
+    sleep "$WAIT_SLEEP_SECONDS"
+    attempt=$((attempt + 1))
+  done
+
+  echo "bucket deletion was not replicated in time: $site/$bucket" >&2
+  return 1
+}
+
+require_command mc
+require_command dd
+require_command awk
+require_command date
+require_command mktemp
+require_command tr
+require_command wc
+
+WORK_DIR="$(mktemp -d "${TMPDIR:-/tmp}/rustfs-site-repl-flow.XXXXXX")"
+MC_CONFIG_DIR="$WORK_DIR/mc"
+export MC_CONFIG_DIR
+
+cleanup() {
+  rm -rf "$WORK_DIR"
+}
+trap cleanup EXIT INT TERM
+
+mkdir -p "$MC_CONFIG_DIR" "$WORK_DIR/src" "$WORK_DIR/downloads"
+
+for site in site1 site2 site3; do
+  mc alias set "$site" "$(site_endpoint "$site")" "$ACCESS_KEY" "$SECRET_KEY" >/dev/null
+done
+
+for site in site1 site2 site3; do
+  echo "checking S3 API for $site"
+  mc ls "$site" >/dev/null
+done
+
+echo "ensuring bucket exists on site1: $BUCKET"
+mc mb --ignore-existing "site1/$BUCKET" >/dev/null
+
+for site in site1 site2 site3; do
+  wait_for_bucket "$site"
+done
+
+cat <<'EOF' | while read -r size_mb source_site object_name; do
+10 site1 object-010m.bin
+25 site2 object-025m.bin
+50 site3 object-050m.bin
+75 site1 object-075m.bin
+100 site2 object-100m.bin
+EOF
+  src_file="$WORK_DIR/src/$object_name"
+  object="$PREFIX/$object_name"
+
+  echo "creating ${size_mb}MiB file: $object_name"
+  dd if=/dev/urandom of="$src_file" bs=1048576 count="$size_mb" >/dev/null 2>&1
+  src_checksum="$(checksum_file "$src_file")"
+  src_bytes="$(wc -c < "$src_file" | tr -d ' ')"
+
+  echo "uploading $object_name to $source_site ($src_bytes bytes)"
+  mc cp "$src_file" "$source_site/$BUCKET/$object" >/dev/null
+
+  for site in $(other_sites "$source_site"); do
+    wait_for_object "$site" "$object"
+
+    dst_file="$WORK_DIR/downloads/$site-$object_name"
+    verified=false
+    attempt=1
+    while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
+      rm -f "$dst_file"
+      echo "downloading $object_name from $site"
+      if mc cp "$site/$BUCKET/$object" "$dst_file" >/dev/null 2>&1; then
+        dst_checksum="$(checksum_file "$dst_file")"
+        dst_bytes="$(wc -c < "$dst_file" | tr -d ' ')"
+
+        if [ "$dst_checksum" = "$src_checksum" ] && [ "$dst_bytes" = "$src_bytes" ]; then
+          verified=true
+          break
+        fi
+      fi
+
+      sleep "$WAIT_SLEEP_SECONDS"
+      attempt=$((attempt + 1))
+    done
+
+    if [ "$verified" != "true" ]; then
+      echo "download verification failed for $site/$BUCKET/$object" >&2
+      echo "expected checksum: $src_checksum" >&2
+      echo "expected bytes:    $src_bytes" >&2
+      exit 1
+    fi
+  done
+
+  echo "verified replicated downloads for $object_name"
+done
+
+echo "creating empty bucket for replicated delete check: $DELETE_BUCKET"
+mc mb "site1/$DELETE_BUCKET" >/dev/null
+
+for site in site1 site2 site3; do
+  wait_for_bucket "$site" "$DELETE_BUCKET"
+done
+
+echo "deleting empty bucket on site1: $DELETE_BUCKET"
+mc rb "site1/$DELETE_BUCKET" >/dev/null
+
+for site in site1 site2 site3; do
+  wait_for_bucket_delete "$site" "$DELETE_BUCKET"
+done
+
+echo "site replication object flow check passed"
+echo "bucket: $BUCKET"
+echo "prefix: $PREFIX"
@@ -1 +1,4 @@
 target
+.docker/compat/data
+.docker/compat/kms
+target/compat
@@ -1,36 +1,32 @@
 # GitHub Workflow Instructions

-Applies to `.github/` and repository pull-request operations.
+Applies to `.github/`.

 ## Pull Requests

- PR titles and descriptions must be in English.
- Use `.github/pull_request_template.md` for every PR body.
- Keep all template section headings.
- Use `N/A` for non-applicable sections.
- Include verification commands in the PR details.
- For `gh pr create` and `gh pr edit`, always write markdown body to a file and pass `--body-file`.
- Do not use multiline inline `--body`; backticks and shell expansion can corrupt content or trigger unintended commands.
- Recommended pattern:
-  - `cat > /tmp/pr_body.md <<'EOF'`
-  - `...markdown...`
-  - `EOF`
-  - `gh pr create ... --body-file /tmp/pr_body.md`
+PR conventions (English title/body, template usage, `--body-file` with the
+heredoc pattern, review-thread etiquette) live in the root `AGENTS.md` under
+"Git and PR Baseline" — that section is the single normative home; do not
+duplicate its rules here.
+
+## Workflow Changes
+
+- Any change touching `.github/workflows/` must pass `actionlint` (run from
+  the repo root) with zero findings before commit/PR. Install via
+  `brew install actionlint`, or the download script in the actionlint repo
+  (rhysd/actionlint); `make pre-pr` does not cover workflow files, so this is
+  the required check for them.
+- Custom self-hosted runner labels (`sm-standard-*`, `dind-*`) are declared
+  in `.github/actionlint.yaml`. When introducing a new `runs-on:` label,
+  declare it there in the same change. Never use the config or `-ignore` to
+  silence real findings — fix the workflow (root `AGENTS.md`: make checks
+  pass by fixing the cause, not by weakening the gate).
+- If `shellcheck` is installed, actionlint also lints `run:` scripts; treat
+  those findings as part of the gate.

 ## CI Alignment

-When changing CI-sensitive behavior, keep local validation aligned with `.github/workflows/ci.yml`.
-
-Current `test-and-lint` gate includes:
-
- `cargo nextest run --all --exclude e2e_test`
- `cargo test --all --doc`
- `cargo test -p rustfs get_object_chunk_fast_path`
- `cargo test -p rustfs materialize_chunk_stream_before_commit`
- `touch rustfs/build.rs`
- `cargo build -p rustfs --bins --jobs 2`
- `cargo test -p e2e_test archive_multipart_roundtrip_preserves_bytes`
- `cargo test -p e2e_test presigned_get_and_reverse_proxy_preserve_multipart_bytes_with_fast_path`
- `cargo fmt --all --check`
- `cargo clippy --all-targets --all-features -- -D warnings`
- `./scripts/check_layer_dependencies.sh`
+When changing CI-sensitive behavior, keep local validation aligned with
+`.github/workflows/ci.yml`. Read the workflow file directly for the current
+gate steps — do not rely on (or add) a copied command list here; copies go
+stale. `make pre-pr` is the local equivalent of the main gate.
@@ -0,0 +1,7 @@
+self-hosted-runner:
+  # Custom labels of this repository's self-hosted runners. Keep in sync with
+  # the labels actually used in `runs-on:` across .github/workflows/.
+  labels:
+    - sm-standard-2
+    - sm-standard-4
+    - dind-sm-standard-2
@@ -0,0 +1,74 @@
+# schedule-failure-issue
+
+Opens (or updates) a GitHub issue when a **scheduled** workflow run fails.
+This is the single alerting mechanism for all timed pipelines
+(rustfs/backlog#1149 ci-8, arbitration G3): scheduled workflows must consume
+this action instead of inventing their own notification paths.
+
+## Behavior
+
+- Issue title: `[scheduled-failure] <workflow name>` — the workflow name is
+  the dedupe key.
+- If an **open** issue with that exact title exists, the failure is appended
+  as a comment; otherwise a new issue is created (labeled `infrastructure` by
+  default). Closing the issue resets the cycle: the next failure opens a
+  fresh one.
+- The issue body / comment includes the run URL, run attempt, event, ref and
+  the names of the failed (or timed-out/cancelled) jobs of the current run
+  attempt.
+- Requires `gh` and `jq` on the runner (both preinstalled on GitHub-hosted
+  runners such as `ubuntu-latest`).
+
+## Usage
+
+Add a final job to the workflow. `always()` is required — without it the job
+is skipped when a needed job fails; `contains(needs.*.result, 'failure')`
+makes it act only when something actually failed. Scope `issues: write` to
+this job only; no other job may gain permissions.
+
+```yaml
+  alert-on-failure:
+    name: Alert on scheduled failure
+    needs: [<the jobs to watch>]
+    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+      issues: write
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+      - name: Open or update failure-tracking issue
+        uses: ./.github/actions/schedule-failure-issue
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+```
+
+Notes:
+
+- Upstream jobs that are *cancelled* (e.g. a manually cancelled run) do not
+  trigger the alert — a human was already looking. Job `timeout-minutes`
+  expiry surfaces as `failure` and does alert.
+- Manual `workflow_dispatch` runs never alert; a human is watching those.
+
+## Current consumers
+
+- `.github/workflows/e2e-s3tests.yml` (weekly full s3-tests sweep)
+- `.github/workflows/mint.yml` (weekly multi-SDK mint run)
+- `.github/workflows/fuzz.yml` (nightly fuzz corpus jobs)
+- `.github/workflows/performance-ab.yml` (nightly warp A/B gate)
+- **Pending:** the ci-7 e2e nightly-full workflow does not exist yet
+  (backlog#1149 ci-7). When it lands, wire it up with the snippet above.
+
+## Drill
+
+`.github/workflows/schedule-failure-alert-drill.yml` is a manual-only
+workflow that forces a job failure and runs this action through the exact
+consumer wiring. Use it to verify the alert path end to end:
+
+```bash
+gh workflow run schedule-failure-alert-drill.yml -R rustfs/rustfs --ref <branch>
+```
+
+Run it twice to verify both paths (issue creation, then comment dedupe), and
+close the `[scheduled-failure] Schedule Failure Alert Drill` issue afterwards.
@@ -0,0 +1,104 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: "Schedule Failure Issue"
+description: >-
+  Open (or update) a tracking issue when a scheduled workflow run fails.
+  Dedupes by workflow name: if an open issue titled
+  "[scheduled-failure] <workflow name>" already exists, the failure is
+  appended as a comment; otherwise a new issue is created. This is the
+  single alerting mechanism for all scheduled pipelines (backlog#1149 ci-8).
+
+inputs:
+  github-token:
+    description: >-
+      Token with issues:write on the repository. Pass secrets.GITHUB_TOKEN
+      from a job that declares `permissions: issues: write` (scope the
+      permission to the alert job only, never workflow-wide).
+    required: true
+  workflow-name:
+    description: "Workflow name used for the issue title (the dedupe key)."
+    required: false
+    default: ${{ github.workflow }}
+  label:
+    description: >-
+      Label applied when a new issue is created. Must already exist in the
+      repository; if applying it fails, the issue is created without a label.
+      Set to an empty string to skip labeling.
+    required: false
+    default: "infrastructure"
+
+runs:
+  using: "composite"
+  steps:
+    - name: Open or update failure-tracking issue
+      shell: bash
+      env:
+        GH_TOKEN: ${{ inputs.github-token }}
+        WORKFLOW_NAME: ${{ inputs.workflow-name }}
+        ISSUE_LABEL: ${{ inputs.label }}
+      run: |
+        set -euo pipefail
+
+        title="[scheduled-failure] ${WORKFLOW_NAME}"
+        run_url="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
+
+        # Failed job names for this run attempt. The alert job runs while the
+        # run as a whole is still in progress, so inspect the jobs that have
+        # already completed with a non-success conclusion.
+        failed_jobs="$(gh api \
+          "repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/attempts/${GITHUB_RUN_ATTEMPT}/jobs" \
+          --paginate \
+          --jq '.jobs[]
+                | select(.conclusion == "failure" or .conclusion == "timed_out" or .conclusion == "cancelled")
+                | "- `\(.name)` (\(.conclusion))"')"
+        if [ -z "${failed_jobs}" ]; then
+          failed_jobs="- (failed job not recorded yet — see the run page)"
+        fi
+
+        body="$(cat <<EOF
+        Scheduled run of **${WORKFLOW_NAME}** failed.
+
+        - Run: ${run_url} (attempt ${GITHUB_RUN_ATTEMPT})
+        - Event: \`${GITHUB_EVENT_NAME}\`
+        - Ref: \`${GITHUB_REF_NAME}\` @ \`${GITHUB_SHA}\`
+
+        Failed jobs:
+        ${failed_jobs}
+        EOF
+        )"
+
+        # Dedupe by exact title among OPEN issues (a closed issue means the
+        # earlier breakage was resolved; a new failure gets a fresh issue).
+        # GitHub search strips the [] characters, so search loosely and match
+        # the exact title with jq.
+        existing="$(gh issue list --repo "${GITHUB_REPOSITORY}" --state open --limit 100 \
+          --search "\"scheduled-failure\" in:title" --json number,title \
+          | jq -r --arg title "${title}" 'map(select(.title == $title)) | (.[0].number // empty)')"
+
+        if [ -n "${existing}" ]; then
+          echo "Appending comment to existing open issue #${existing}"
+          gh issue comment "${existing}" --repo "${GITHUB_REPOSITORY}" --body "${body}"
+          exit 0
+        fi
+
+        echo "Creating new issue: ${title}"
+        if [ -n "${ISSUE_LABEL}" ]; then
+          if gh issue create --repo "${GITHUB_REPOSITORY}" \
+               --title "${title}" --body "${body}" --label "${ISSUE_LABEL}"; then
+            exit 0
+          fi
+          echo "::warning::issue creation with label '${ISSUE_LABEL}' failed (missing label?); retrying without a label"
+        fi
+        gh issue create --repo "${GITHUB_REPOSITORY}" --title "${title}" --body "${body}"
@@ -53,21 +53,25 @@ runs:
          musl-tools \
          build-essential \
          pkg-config \
-          libssl-dev
+          libssl-dev \
+          ripgrep \
+          unzip \
+          zip \
+          protobuf-compiler

    - name: Install protoc
-      uses: arduino/setup-protoc@v3
+      uses: rustfs/setup-protoc@a3705324d8f9bf5b6c3573fb6cf8ae421db55dd6 # v3.0.1
      with:
-        version: "33.1"
-        repo-token: ${{ inputs.github-token }}
+        version: "34.1"
+        repo-token: ${{ github.token }}

    - name: Install flatc
-      uses: Nugine/setup-flatc@v1.2.4
+      uses: Nugine/setup-flatc@e7855e994773ce90094a3f1626d4afc9080c23ae # v1
      with:
-        version: "25.9.23"
+        version: "25.12.19"

    - name: Install Rust toolchain
-      uses: dtolnay/rust-toolchain@stable
+      uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
      with:
        toolchain: ${{ inputs.rust-version }}
        targets: ${{ inputs.target }}
@@ -75,17 +79,17 @@ runs:

    - name: Install Zig
      if: inputs.install-cross-tools == 'true'
-      uses: mlugg/setup-zig@v2
+      uses: mlugg/setup-zig@d1434d08867e3ee9daa34448df10607b98908d29 # v2

    - name: Install cargo-zigbuild
      if: inputs.install-cross-tools == 'true'
-      uses: taiki-e/install-action@cargo-zigbuild
+      uses: taiki-e/install-action@a21ae4029b089b9ddc45704028756f51ab8abe48 # cargo-zigbuild

    - name: Install cargo-nextest
-      uses: taiki-e/install-action@cargo-nextest
+      uses: taiki-e/install-action@96c7780c1d8a2b8723e12031def873a434d39d8d # nextest

    - name: Setup Rust cache
-      uses: Swatinem/rust-cache@v2
+      uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
      with:
        cache-all-crates: true
        cache-on-failure: true
@@ -1,8 +1,22 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
 enabled: true

 document:
-  version: v1
-  url: https://github.com/rustfs/cla/blob/main/cla/v1.md
+  version: v2
+  url: https://github.com/rustfs/cla/blob/main/cla/v2.md

 signing:
  mode: comment
@@ -27,9 +27,8 @@ updates:
      timezone: "Asia/Shanghai"
      time: "08:00"
    assignees:
-      - "heihutu"
-    reviewers:
      - "houseme"
+    reviewers:
      - "overtrue"
      - "majinghe"
    ignore:
@@ -39,6 +38,10 @@ updates:
        versions: [ "0.23.x" ]
      - dependency-name: "ratelimit"
        versions: [ "1.x" ]
+      - dependency-name: "ratelimit"
+        versions: [ "2.x" ]
+      - dependency-name: "pyroscope"
+        versions: [ "2.x" ]
    groups:
      s3s:
        update-types:
@@ -2,36 +2,35 @@
 Pull Request Template for RustFS
 -->

-## Type of Change
- [ ] New Feature
- [ ] Bug Fix
- [ ] Documentation
- [ ] Performance Improvement
- [ ] Test/CI
- [ ] Refactor
- [ ] Other:
-
 ## Related Issues
-<!-- List related Issue numbers, e.g. #123 -->
+<!--
+List related issues, e.g. Fixes #123.
+Use N/A when there is no related issue.
+-->

 ## Summary of Changes
-<!-- Briefly describe the main changes and motivation for this PR -->
+<!--
+Briefly explain what changed and why reviewers should accept it.
+Focus on behavior, compatibility, and review-relevant context.
+-->

-## Checklist
- [ ] I have read and followed the [CONTRIBUTING.md](CONTRIBUTING.md) guidelines
- [ ] Passed `make pre-commit`
- [ ] Added/updated necessary tests
- [ ] Documentation updated (if needed)
- [ ] CI/CD passed (if applicable)
+## Verification
+<!--
+List the commands or checks you ran, for example:
+- `make pre-commit`
+
+Use N/A only when verification is not applicable.
+-->

 ## Impact
- [ ] Breaking change (compatibility)
- [ ] Requires doc/config/deployment update
- [ ] Other impact:
+<!--
+Describe user-facing, compatibility, API, deployment, configuration, or
+documentation impact. Use N/A when there is no expected impact.
+-->

 ## Additional Notes
-<!-- Any extra information for reviewers -->
+<!-- Any extra information for reviewers, or N/A. -->

 ---

-Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)). If this is your first contribution, review the [CLA document](https://github.com/rustfs/cla/blob/main/cla/v1.md) and sign it by commenting `I have read and agree to the CLA.` on the PR.
+Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)). If this is your first contribution, review the [CLA document](https://github.com/rustfs/cla/blob/main/cla/v2.md) and sign it by commenting `I have read and agree to the CLA.` on the PR.
@@ -1,103 +1,81 @@
 # S3 Compatibility Tests Configuration

-This directory contains the configuration for running [Ceph S3 compatibility tests](https://github.com/ceph/s3-tests) against RustFS.
+This directory contains the configuration template for running
+[Ceph S3 compatibility tests](https://github.com/ceph/s3-tests) against RustFS.
+
+The test runner itself lives in [`scripts/s3-tests/`](../../scripts/s3-tests/)
+— see its README for how tests are selected, executed, and reported. This
+directory only holds the shared config template.

 ## Configuration File

-The `s3tests.conf` file is based on the official `s3tests.conf.SAMPLE` from the ceph/s3-tests repository. It uses environment variable substitution via `envsubst` to configure the endpoint and credentials.
+`s3tests.conf` is based on the official `s3tests.conf.SAMPLE` from the
+ceph/s3-tests repository. It is a template: `scripts/s3-tests/run.sh` renders
+it with `envsubst` before every run.

-### Key Configuration Points
+Required environment variables (all have defaults in `run.sh`):

- **Host**: Set via `${S3_HOST}` environment variable (e.g., `rustfs-single` for single-node, `lb` for multi-node)
- **Port**: 9000 (standard RustFS port)
- **Credentials**: Uses `${S3_ACCESS_KEY}` and `${S3_SECRET_KEY}` from workflow environment
- **TLS**: Disabled (`is_secure = False`)
+| Variable | Meaning |
+|----------|---------|
+| `S3_HOST` | RustFS endpoint host |
+| `S3_PORT` | RustFS endpoint port |
+| `S3_ACCESS_KEY` / `S3_SECRET_KEY` | main user credentials |
+| `S3_ALT_ACCESS_KEY` / `S3_ALT_SECRET_KEY` | alt user credentials (must differ from main) |

-## Test Execution Strategy
+TLS is disabled (`is_secure = False`).

-### Network Connectivity Fix
+## Test Selection Strategy

-Tests run inside a Docker container on the `rustfs-net` network, which allows them to resolve and connect to the RustFS container hostnames. This fixes the "Temporary failure in name resolution" error that occurred when tests ran on the GitHub runner host.
+Tests are selected by exact pytest node ids from classification list files in
+`scripts/s3-tests/` — **not** by pytest markers:

-### Performance Optimizations
+- `implemented_tests.txt` — expected to pass; run as the per-PR gate
+  (`ci.yml`, job `s3-implemented-tests`)
+- `unimplemented_tests.txt` — standard S3 features not yet implemented
+- `excluded_tests.txt` — intentionally excluded (vendor-specific behavior or
+  product decisions)

-1. **Parallel Execution**: Uses `pytest-xdist` with `-n 4` to run tests in parallel across 4 workers
-2. **Load Distribution**: Uses `--dist=loadgroup` to distribute test groups across workers
-3. **Fail-Fast**: Uses `--maxfail=50` to stop after 50 failures, saving time on catastrophic failures
+The upstream s3-tests repository is pinned to a fixed commit (`S3TESTS_REV` in
+`run.sh`) so results are reproducible; bump it deliberately and reclassify the
+lists when upstream changes.

-### Feature Filtering
+## Where Tests Run

-Tests are filtered using pytest markers (`-m`) to skip features not yet supported by RustFS:
-
- `lifecycle` - Bucket lifecycle policies
- `versioning` - Object versioning
- `s3website` - Static website hosting
- `bucket_logging` - Bucket logging
- `encryption` / `sse_s3` - Server-side encryption
- `cloud_transition` / `cloud_restore` - Cloud storage transitions
- `lifecycle_expiration` / `lifecycle_transition` - Lifecycle operations
-
-This filtering:
-1. Reduces test execution time significantly (from 1+ hour to ~10-15 minutes)
-2. Focuses on features RustFS currently supports
-3. Avoids hundreds of expected failures
+- **Per PR**: `.github/workflows/ci.yml` job `s3-implemented-tests` runs the
+  implemented whitelist against a single-node debug binary. Any failure blocks
+  the PR.
+- **Weekly + manual**: `.github/workflows/e2e-s3tests.yml` runs the full
+  upstream suite (`TEST_SCOPE=all`) against a Docker deployment (single node
+  or a 4-node distributed cluster behind HAProxy). It fails only on
+  regressions in the implemented whitelist and publishes a classification
+  report (`compat-report.md`, also shown in the job summary) listing promotion
+  candidates and unclassified tests.

 ## Running Tests Locally

-### Single-Node Test
-
 ```bash
-# Set credentials
-export S3_ACCESS_KEY=rustfsadmin
-export S3_SECRET_KEY=rustfsadmin
+# Whitelist run against a locally built binary (the same thing the PR gate does)
+./scripts/s3-tests/run.sh

-# Start RustFS container
-docker run -d --name rustfs-single \
-  --network rustfs-net \
-  -e RUSTFS_ADDRESS=0.0.0.0:9000 \
-  -e RUSTFS_ACCESS_KEY=$S3_ACCESS_KEY \
-  -e RUSTFS_SECRET_KEY=$S3_SECRET_KEY \
-  -e RUSTFS_VOLUMES="/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3" \
-  rustfs-ci
+# Full sweep, never stop on failures, 4 pytest workers
+TEST_SCOPE=all MAXFAIL=0 XDIST=4 ./scripts/s3-tests/run.sh

-# Generate config
-export S3_HOST=rustfs-single
-envsubst < .github/s3tests/s3tests.conf > /tmp/s3tests.conf
-
-# Run tests
-docker run --rm \
-  --network rustfs-net \
-  -v /tmp/s3tests.conf:/etc/s3tests.conf:ro \
-  python:3.12-slim \
-  bash -c '
-    apt-get update -qq && apt-get install -y -qq git
-    git clone --depth 1 https://github.com/ceph/s3-tests.git /s3-tests
-    cd /s3-tests
-    pip install -q -r requirements.txt pytest-xdist
-    S3TEST_CONF=/etc/s3tests.conf pytest -v -n 4 \
-      s3tests/functional/test_s3.py \
-      -m "not lifecycle and not versioning and not s3website and not bucket_logging and not encryption and not sse_s3"
-  '
+# A specific test against an already-running server
+DEPLOY_MODE=existing TESTEXPR="test_bucket_list_empty" ./scripts/s3-tests/run.sh
 ```

-## Test Results Interpretation
-
- **PASSED**: Test succeeded, feature works correctly
- **FAILED**: Test failed, indicates a potential bug or incompatibility
- **ERROR**: Test setup failed (e.g., network issues, missing dependencies)
- **SKIPPED**: Test skipped due to marker filtering
+Results land in `artifacts/s3tests-single/` (`junit.xml`, `pytest.log`,
+`compat-report.md`).

 ## Adding New Feature Support

-When adding support for a new S3 feature to RustFS:
-
-1. Remove the corresponding marker from the filter in `.github/workflows/e2e-s3tests.yml`
-2. Run the tests to verify compatibility
-3. Fix any failing tests
-4. Update this README to reflect the newly supported feature
+Follow [`scripts/s3-tests/S3_COMPAT_WORKFLOW.md`](../../scripts/s3-tests/S3_COMPAT_WORKFLOW.md).
+In short: fix the behavior, verify with `TESTEXPR=...`, then move the test
+from `unimplemented_tests.txt` to `implemented_tests.txt` so the PR gate locks
+it in.

 ## References

 - [Ceph S3 Tests Repository](https://github.com/ceph/s3-tests)
 - [S3 API Compatibility](https://docs.aws.amazon.com/AmazonS3/latest/API/)
- [pytest-xdist Documentation](https://pytest-xdist.readthedocs.io/)
+- [RustFS test runner](../../scripts/s3-tests/README.md)
@@ -11,8 +11,8 @@
 # host set for RustFS - will be substituted via envsubst
 host = ${S3_HOST}

-# port for RustFS
-port = 9000
+# port for RustFS - will be substituted via envsubst
+port = ${S3_PORT}

 ## say "False" to disable TLS
 is_secure = False
@@ -0,0 +1,14 @@
+# GitHub secret scanning path exclusions.
+# Secrets detected under these paths are auto-closed as "ignored by
+# configuration" and are not blocked by push protection.
+#
+# Deliberately NOT excluded: production source files (e.g. rustfs/src/**,
+# crates/*/src/**) even when a hit sits inside a #[cfg(test)] module or a
+# doc-comment example — excluding the file would also stop scanning the
+# production code in it. Close those alerts manually as "used in tests".
+paths-ignore:
+  - "crates/e2e_test/**" # dedicated e2e test crate, test credentials throughout
+  - "**/tests/**" # Rust integration-test dirs (crates/*/tests, rustfs/tests)
+  - "**/benches/**" # benchmark harnesses
+  - ".docker/**" # local docker-compose dev configs (e.g. mqtt vm.args cookie)
+  - ".vscode/**" # local editor launch configs
@@ -0,0 +1,57 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#      http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Architecture Migration Rules
+
+on:
+  pull_request:
+    types: [ opened, synchronize, reopened, closed ]
+    branches: [ main ]
+    paths:
+      - "ARCHITECTURE.md"
+      - "docs/architecture/**"
+      - "scripts/check_architecture_migration_rules.sh"
+      - ".github/workflows/architecture-migration-rules.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  cancel-closed-pr-runs:
+    name: Cancel Closed PR Runs
+    if: github.event_name == 'pull_request' && github.event.action == 'closed'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Explain cancellation run
+        run: echo "PR closed; this run only cancels older runs in the same concurrency group."
+
+  architecture-migration-rules:
+    name: Architecture Migration Rules
+    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Install ripgrep
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y ripgrep
+
+      - name: Check architecture migration rules
+        run: ./scripts/check_architecture_migration_rules.sh
@@ -20,62 +20,101 @@ on:
    paths:
      - '**/Cargo.toml'
      - '**/Cargo.lock'
-      - '.github/workflows/audit.yml'
+      - 'deny.toml'
+      - '.github/actions/**'
+      - '.github/workflows/**'
+      - 'scripts/security/check_workflow_pins.sh'
  pull_request:
+    types: [ opened, synchronize, reopened, closed ]
    branches: [ main ]
    paths:
      - '**/Cargo.toml'
      - '**/Cargo.lock'
-      - '.github/workflows/audit.yml'
+      - 'deny.toml'
+      - '.github/actions/**'
+      - '.github/workflows/**'
+      - 'scripts/security/check_workflow_pins.sh'
  schedule:
-    - cron: '0 0 * * 0' # Weekly on Sunday at midnight UTC
+    - cron: '0 3 * * 0' # Weekly on Sunday 03:00 UTC (staggered after the midnight ci/build crons)
  workflow_dispatch:

 permissions:
  contents: read

+# Event-scoped groups: merges to main must not cancel the weekly scheduled
+# run (same rationale as ci.yml).
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: ${{ github.event_name != 'schedule' }}
+
 env:
  CARGO_TERM_COLOR: always

 jobs:
-  security-audit:
-    name: Security Audit
-    runs-on: ubicloud-standard-2
-    timeout-minutes: 15
+  cancel-closed-pr-runs:
+    name: Cancel Closed PR Runs
+    if: github.event_name == 'pull_request' && github.event.action == 'closed'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Explain cancellation run
+        run: echo "PR closed; this run only cancels older runs in the same concurrency group."
+
+  # RustSec advisory scanning is covered by the `advisories` check of
+  # cargo-deny below; a separate cargo-audit job would duplicate the same
+  # database lookup with a second ignore list to maintain.
+  cargo-deny:
+    name: Cargo Deny
+    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    runs-on: ubuntu-latest
+    timeout-minutes: 20
    steps:
      - name: Checkout repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7

-      - name: Install cargo-audit
-        uses: taiki-e/install-action@v2
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
        with:
-          tool: cargo-audit
+          cache-shared-key: rustfs-cargo-deny

-      - name: Run security audit
-        run: |
-          cargo audit -D warnings --json | tee audit-results.json
-
-      - name: Upload audit results
-        if: always()
-        uses: actions/upload-artifact@v6
+      - name: Install cargo-deny
+        uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
        with:
-          name: security-audit-results-${{ github.run_number }}
-          path: audit-results.json
-          retention-days: 30
+          tool: cargo-deny
+
+      - name: Run cargo-deny
+        run: cargo deny check --hide-inclusion-graph advisories sources bans licenses
+
+  workflow-pin-report:
+    name: Workflow Pin Report
+    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    runs-on: ubuntu-latest
+    timeout-minutes: 5
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Report unpinned GitHub Actions
+        run: ./scripts/security/check_workflow_pins.sh --enforce

  dependency-review:
    name: Dependency Review
-    runs-on: ubicloud-standard-2
-    if: github.event_name == 'pull_request'
+    runs-on: ubuntu-latest
+    if: github.event_name == 'pull_request' && github.event.action != 'closed'
    permissions:
      contents: read
      pull-requests: write
    steps:
      - name: Checkout repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7

      - name: Dependency Review
-        uses: actions/dependency-review-action@v4
+        uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v5
        with:
          fail-on-severity: moderate
-          comment-summary-in-pr: true
+          allow-ghsas: GHSA-2f9f-gq7v-9h6m
+          # rustfs-uring is a first-party Apache-2.0 crate, now published on
+          # crates.io (backlog#1104) rather than pulled as a git dependency.
+          # Scope the allow to the exact pinned version so a version bump forces a
+          # conscious re-review of the license/provenance claim (backlog#1181).
+          allow-dependencies-licenses: pkg:cargo/rustfs-uring@0.1.0
+          comment-summary-in-pr: always
@@ -21,6 +21,11 @@
 # 2. Upload binaries to OSS storage
 # 3. Trigger docker.yml to build and push images using the uploaded binaries
 #
+# Platform scope:
+# - Pushes to main (development builds) build the Linux targets only — they
+#   feed the dev download channel and Docker dev images. Tags, the weekly
+#   schedule and manual dispatch build the full platform matrix.
+#
 # Manual Parameters:
 # - build_docker: Build and push Docker images (default: true)
 # - platforms: Comma-separated platform IDs or 'all' (default: all)
@@ -46,7 +51,7 @@ on:
      - ".gitignore"
      - ".dockerignore"
  schedule:
-    - cron: "0 0 * * 0" # Weekly on Sunday at midnight UTC
+    - cron: "0 1 * * 0" # Weekly on Sunday 01:00 UTC (staggered after the ci.yml midnight cron)
  workflow_dispatch:
    inputs:
      build_docker:
@@ -63,6 +68,10 @@ on:
 permissions:
  contents: read

+concurrency:
+  group: ${{ github.workflow }}-${{ github.event_name == 'push' && github.ref || github.run_id }}
+  cancel-in-progress: ${{ github.event_name == 'push' }}
+
 env:
  CARGO_TERM_COLOR: always
  RUST_BACKTRACE: 1
@@ -73,7 +82,7 @@ jobs:
  # Build strategy check - determine build type based on trigger
  build-check:
    name: Build Strategy Check
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
    outputs:
      should_build: ${{ steps.check.outputs.should_build }}
      build_type: ${{ steps.check.outputs.build_type }}
@@ -82,9 +91,7 @@ jobs:
      is_prerelease: ${{ steps.check.outputs.is_prerelease }}
    steps:
      - name: Checkout repository
-        uses: actions/checkout@v6
-        with:
-          fetch-depth: 0
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7

      - name: Determine build strategy
        id: check
@@ -121,8 +128,7 @@ jobs:
            version="dev-${short_sha}"
            echo "🛠️  Development build detected"
          elif [[ "${{ github.event_name }}" == "schedule" ]] || \
-               [[ "${{ github.event_name }}" == "workflow_dispatch" ]] || \
-               [[ "${{ contains(github.event.head_commit.message, '--build') }}" == "true" ]]; then
+               [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
            # Scheduled or manual build
            should_build=true
            build_type="development"
@@ -130,11 +136,13 @@ jobs:
            echo "⚡ Manual/scheduled build detected"
          fi

-          echo "should_build=$should_build" >> $GITHUB_OUTPUT
-          echo "build_type=$build_type" >> $GITHUB_OUTPUT
-          echo "version=$version" >> $GITHUB_OUTPUT
-          echo "short_sha=$short_sha" >> $GITHUB_OUTPUT
-          echo "is_prerelease=$is_prerelease" >> $GITHUB_OUTPUT
+          {
+            echo "should_build=$should_build"
+            echo "build_type=$build_type"
+            echo "version=$version"
+            echo "short_sha=$short_sha"
+            echo "is_prerelease=$is_prerelease"
+          } >> "$GITHUB_OUTPUT"

          echo "📊 Build Summary:"
          echo "  - Should build: $should_build"
@@ -146,7 +154,8 @@ jobs:
  # Build RustFS binaries
  prepare-platform-matrix:
    name: Prepare Platform Matrix
-    runs-on: ubicloud-standard-2
+    needs: build-check
+    runs-on: ubuntu-latest
    outputs:
      matrix: ${{ steps.select.outputs.matrix }}
      selected: ${{ steps.select.outputs.selected }}
@@ -163,13 +172,24 @@ jobs:
            selected="all"
          fi

+          # Per-merge development builds only feed the dev download channel
+          # and the Docker dev images, which consume the Linux binaries; at
+          # the usual merge cadence most per-merge builds are cancelled by the
+          # next merge anyway. macOS and Windows stay covered by tag builds,
+          # the weekly scheduled build and manual dispatch.
+          if [[ "${selected}" == "all" \
+            && "${{ github.event_name }}" == "push" \
+            && "${{ needs.build-check.outputs.build_type }}" == "development" ]]; then
+            selected="linux-x86_64-musl,linux-aarch64-musl,linux-x86_64-gnu,linux-aarch64-gnu"
+            echo "Development (main push) build: restricting to Linux targets"
+          fi
+
          all='{"include":[
-            {"target_id":"linux-x86_64-musl","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":""},
-            {"target_id":"linux-aarch64-musl","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-musl","cross":true,"platform":"linux","rustflags":""},
-            {"target_id":"linux-x86_64-gnu","os":"ubicloud-standard-2","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":""},
-            {"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-2","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":""},
+            {"target_id":"linux-x86_64-musl","os":"sm-standard-2","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":""},
+            {"target_id":"linux-aarch64-musl","os":"sm-standard-2","target":"aarch64-unknown-linux-musl","cross":true,"platform":"linux","rustflags":""},
+            {"target_id":"linux-x86_64-gnu","os":"sm-standard-2","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":""},
+            {"target_id":"linux-aarch64-gnu","os":"sm-standard-2","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":""},
            {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
-            {"target_id":"macos-x86_64","os":"macos-latest","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
            {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":""}
          ]}'

@@ -202,20 +222,20 @@ jobs:
    needs: [ build-check, prepare-platform-matrix ]
    if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
    runs-on: ${{ matrix.os }}
-    timeout-minutes: 60
+    timeout-minutes: 150
    env:
      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-      # Always enable Tokio unstable features (required by dial9-tokio-telemetry).
-      # The RUSTFLAGS env var takes precedence over .cargo/config.toml [build] rustflags,
-      # so we must include --cfg tokio_unstable here explicitly; otherwise an empty
-      # RUSTFLAGS value would shadow the config-file flag and silently break tracing.
-      RUSTFLAGS: "--cfg tokio_unstable ${{ matrix.rustflags }}"
+      # Release binaries ship without dial9 telemetry and therefore do not need
+      # --cfg tokio_unstable. Telemetry builds opt in explicitly (see
+      # `make build-profiling`); crates/obs/build.rs fails the compile if the
+      # `dial9` feature is enabled without the flag.
+      RUSTFLAGS: "${{ matrix.rustflags }}"
    strategy:
      fail-fast: false
      matrix: ${{ fromJson(needs.prepare-platform-matrix.outputs.matrix) }}
    steps:
      - name: Checkout repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
        with:
          fetch-depth: 0

@@ -224,37 +244,131 @@ jobs:
        with:
          rust-version: stable
          target: ${{ matrix.target }}
-          cache-shared-key: build-${{ matrix.target }}-${{ hashFiles('**/Cargo.lock') }}
+          cache-shared-key: build-${{ matrix.target }}
          github-token: ${{ secrets.GITHUB_TOKEN }}
          cache-save-if: ${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/') }}
          install-cross-tools: ${{ matrix.cross }}

      - name: Download static console assets
        shell: bash
+        env:
+          GITHUB_TOKEN: ${{ github.token }}
        run: |
          mkdir -p ./rustfs/static
-          if [[ "${{ matrix.platform }}" == "windows" ]]; then
-            curl.exe -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" -o console.zip --retry 3 --retry-delay 5 --max-time 300
-            if [[ $? -eq 0 ]]; then
-              unzip -o console.zip -d ./rustfs/static
-              rm console.zip
+
+          verify_sha256() {
+            local expected="$1"
+            local file="$2"
+            local actual=""
+            local actual_line=""
+
+            if command -v sha256sum >/dev/null 2>&1; then
+              actual_line=$(sha256sum -- "$file") || return 1
+            elif command -v shasum >/dev/null 2>&1; then
+              actual_line=$(shasum -a 256 -- "$file") || return 1
            else
-              echo "Warning: Failed to download console assets, continuing without them"
-              echo "// Static assets not available" > ./rustfs/static/empty.txt
+              echo "No SHA-256 verification tool found" >&2
+              return 1
            fi
+            actual="${actual_line%%[[:space:]]*}"
+
+            if [ "$actual" = "$expected" ]; then
+              echo "SHA256 verified OK: $actual"
+              return 0
+            else
+              echo "SHA256 mismatch: expected=$expected actual=$actual" >&2
+              return 1
+            fi
+          }
+
+          download_console_assets() {
+            local curl_bin="curl"
+            local python_bin="python3"
+            local console_api="https://api.github.com/repos/rustfs/console/releases/latest"
+            local console_json="console-release.json"
+            local console_url
+            local console_sha256
+            local curl_auth_args=()
+            console_tag=""
+
+            if [[ "${{ matrix.platform }}" == "windows" ]]; then
+              curl_bin="curl.exe"
+            else
+              chmod +w ./rustfs/static/LICENSE || true
+            fi
+
+            if ! command -v "$python_bin" >/dev/null 2>&1; then
+              python_bin="python"
+            fi
+            if ! command -v "$python_bin" >/dev/null 2>&1; then
+              echo "No Python interpreter found for release metadata parsing" >&2
+              return 1
+            fi
+
+            if [[ -n "${GITHUB_TOKEN:-}" ]]; then
+              curl_auth_args=(-H "Authorization: Bearer ${GITHUB_TOKEN}" -H "X-GitHub-Api-Version: 2022-11-28")
+            fi
+
+            "$curl_bin" "${curl_auth_args[@]}" --fail -L "$console_api" \
+              -o "$console_json" --retry 3 --retry-delay 5 --max-time 300 || return 1
+
+            read -r console_tag console_url console_sha256 < <("$python_bin" - "$console_json" <<'PY'
+          import json
+          import re
+          import sys
+
+          with open(sys.argv[1], encoding="utf-8") as handle:
+              release = json.load(handle)
+
+          tag = release.get("tag_name", "") or "unknown"
+
+          for asset in release.get("assets", []):
+              name = asset.get("name", "")
+              digest = asset.get("digest", "")
+              url = asset.get("browser_download_url", "")
+              if name.startswith("rustfs-console-") and name.endswith(".zip") and digest.startswith("sha256:") and url:
+                  sha256 = digest.split(":", 1)[1]
+                  if not re.fullmatch(r"[0-9a-fA-F]{64}", sha256):
+                      raise SystemExit(f"console zip asset has invalid sha256 digest: {sha256}")
+                  sys.stdout.buffer.write(f"{tag} {url} {sha256}\n".encode("utf-8"))
+                  break
+          else:
+              raise SystemExit("no console zip asset with sha256 digest found")
+          PY
+            ) || return 1
+
+            if [[ -z "$console_url" || -z "$console_sha256" ]]; then
+              echo "Console release metadata is missing URL or SHA-256 digest" >&2
+              return 1
+            fi
+
+            echo "Console release: ${console_tag}"
+            echo "Downloading console asset: ${console_url}"
+            "$curl_bin" --fail -L "$console_url" -o console.zip --retry 3 --retry-delay 5 --max-time 300 || return 1
+            verify_sha256 "$console_sha256" console.zip || return 2
+            unzip -o console.zip -d ./rustfs/static || return 2
+          }
+
+          if download_console_assets; then
+            rm -f console.zip console-release.json
          else
-            chmod +w ./rustfs/static/LICENSE || true
-            curl -L "https://dl.rustfs.com/artifacts/console/rustfs-console-latest.zip" \
-              -o console.zip --retry 3 --retry-delay 5 --max-time 300
-            if [[ $? -eq 0 ]]; then
-              unzip -o console.zip -d ./rustfs/static
-              rm console.zip
-            else
-              echo "Warning: Failed to download console assets, continuing without them"
-              echo "// Static assets not available" > ./rustfs/static/empty.txt
+            status=$?
+            rm -f console.zip console-release.json
+            if [[ "$status" -eq 2 ]]; then
+              echo "Console asset integrity verification failed" >&2
            fi
+            echo "Failed to download verified console assets" >&2
+            exit 1
          fi

+          if [[ ! -s ./rustfs/static/index.html ]]; then
+            echo "Console asset archive is missing static/index.html" >&2
+            exit 1
+          fi
+
+          asset_count=$(find ./rustfs/static -type f | wc -l | tr -d '[:space:]')
+          echo "Console assets ready: version=${console_tag:-unknown}, ${asset_count} files extracted to ./rustfs/static"
+
      - name: Build RustFS
        shell: bash
        run: |
@@ -262,14 +376,8 @@ jobs:
          touch rustfs/build.rs

          if [[ "${{ matrix.cross }}" == "true" ]]; then
-            if [[ "${{ matrix.platform }}" == "windows" ]]; then
-              # Use cross for Windows ARM64
-              cargo install cross --git https://github.com/cross-rs/cross
-              cross build --release --target ${{ matrix.target }} -p rustfs --bins
-            else
-              # Use zigbuild for other cross-compilation
-              cargo zigbuild --release --target ${{ matrix.target }} -p rustfs --bins
-            fi
+            # All cross targets in the matrix are Linux; zigbuild handles them.
+            cargo zigbuild --release --target ${{ matrix.target }} -p rustfs --bins
          else
            cargo build --release --target ${{ matrix.target }} -p rustfs --bins
          fi
@@ -342,11 +450,11 @@ jobs:
            # Release/Prerelease build: rustfs-${platform}-${arch}-${variant}-v${version}.zip
            PACKAGE_NAME="${PACKAGE_BASENAME}-v${PACKAGE_VERSION}"
          fi
-
+          
          # Create zip packages for all platforms
          # Ensure zip is available
          if ! command -v zip &> /dev/null; then
-            if [[ "${{ matrix.os }}" == "ubuntu-latest" ]]; then
+            if [[ "${{ matrix.os }}" == "ubuntu-latest" || "${{ matrix.platform }}" == "linux" ]]; then
              sudo apt-get update && sudo apt-get install -y zip
            fi
          fi
@@ -398,7 +506,7 @@ jobs:
            if [[ "${{ matrix.platform }}" == "windows" ]]; then
              dir
            else
-              ls -lh ${PACKAGE_NAME}.zip
+              ls -lh "${PACKAGE_NAME}.zip"
            fi
          else
            echo "❌ Failed to create package: ${PACKAGE_NAME}.zip"
@@ -447,11 +555,13 @@ jobs:
            fi
          fi

-          echo "package_name=${PACKAGE_NAME}" >> $GITHUB_OUTPUT
-          echo "package_file=${PACKAGE_NAME}.zip" >> $GITHUB_OUTPUT
-          echo "latest_files=${LATEST_FILES}" >> $GITHUB_OUTPUT
-          echo "build_type=${BUILD_TYPE}" >> $GITHUB_OUTPUT
-          echo "version=${VERSION}" >> $GITHUB_OUTPUT
+          {
+            echo "package_name=${PACKAGE_NAME}"
+            echo "package_file=${PACKAGE_NAME}.zip"
+            echo "latest_files=${LATEST_FILES}"
+            echo "build_type=${BUILD_TYPE}"
+            echo "version=${VERSION}"
+          } >> "$GITHUB_OUTPUT"

          echo "📦 Package created: ${PACKAGE_NAME}.zip"
          if [[ -n "$LATEST_FILES" ]]; then
@@ -460,86 +570,118 @@ jobs:
          echo "🔧 Build type: ${BUILD_TYPE}"
          echo "📊 Version: ${VERSION}"

+      - name: Verify packaged console
+        if: matrix.target == 'x86_64-unknown-linux-gnu'
+        shell: bash
+        run: |
+          set -euo pipefail
+
+          find_free_port() {
+            python3 - <<'PY'
+          import socket
+
+          with socket.socket() as sock:
+              sock.bind(("127.0.0.1", 0))
+              print(sock.getsockname()[1])
+          PY
+          }
+
+          package_dir="$(mktemp -d)"
+          data_dir="$(mktemp -d)"
+          log_file="${RUNNER_TEMP}/rustfs-console-smoke.log"
+          server_pid=""
+          trap '
+            if [[ -n "${server_pid:-}" ]]; then
+              kill "$server_pid" 2>/dev/null || true
+              wait "$server_pid" 2>/dev/null || true
+            fi
+            rm -rf "$package_dir" "$data_dir"
+          ' EXIT
+
+          unzip -q "${{ steps.package.outputs.package_file }}" -d "$package_dir"
+          api_port="$(find_free_port)"
+          console_port="$(find_free_port)"
+          console_url="http://127.0.0.1:${console_port}/rustfs/console/"
+
+          "$package_dir/rustfs" server \
+            --address "127.0.0.1:${api_port}" \
+            --console-enable \
+            --console-address "127.0.0.1:${console_port}" \
+            --access-key console-smoke \
+            --secret-key console-smoke-secret \
+            "$data_dir" >"$log_file" 2>&1 &
+          server_pid=$!
+
+          for _ in {1..40}; do
+            response="$(curl --silent --output /dev/null --write-out '%{http_code} %{content_type}' "$console_url" || true)"
+            if [[ "$response" == 200\ text/html* ]]; then
+              exit 0
+            fi
+            sleep 0.25
+          done
+
+          echo "Console endpoint did not return 200 text/html: $response" >&2
+          cat "$log_file"
+          exit 1
+
      - name: Upload to GitHub artifacts
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
        with:
          name: ${{ steps.package.outputs.package_name }}
          path: "rustfs-*.zip"
          retention-days: ${{ startsWith(github.ref, 'refs/tags/') && 30 || 7 }}

-      - name: Upload to Aliyun OSS
-        if: env.OSS_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')
+      - name: Upload to Cloudflare R2
+        if: env.R2_ACCESS_KEY_ID != '' && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease' || needs.build-check.outputs.build_type == 'development')
        env:
-          OSS_ACCESS_KEY_ID: ${{ secrets.ALICLOUDOSS_KEY_ID }}
-          OSS_ACCESS_KEY_SECRET: ${{ secrets.ALICLOUDOSS_KEY_SECRET }}
-          OSS_REGION: cn-beijing
-          OSS_ENDPOINT: https://oss-accelerate.aliyuncs.com
+          R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
+          R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
+          R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
+          R2_BUCKET: ${{ secrets.R2_BUCKET }}
+          AWS_EC2_METADATA_DISABLED: true
        shell: bash
        run: |
          BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"

-          # Install ossutil (platform-specific)
-          OSSUTIL_VERSION="2.1.1"
-          case "${{ matrix.platform }}" in
-            linux)
-              if [[ "$(uname -m)" == "arm64" ]]; then
-                ARCH="arm64"
-              else
-                ARCH="amd64"
-              fi
-              OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-linux-${ARCH}.zip"
-              OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-linux-${ARCH}"
+          if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
+            echo "⚠️ R2 credentials or endpoint missing, skipping artifact upload"
+            exit 0
+          fi

-              curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
-              unzip "$OSSUTIL_ZIP"
-              mv "${OSSUTIL_DIR}/ossutil" /usr/local/bin/
-              rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
-              chmod +x /usr/local/bin/ossutil
-              OSSUTIL_BIN=ossutil
-              ;;
-            macos)
-              if [[ "$(uname -m)" == "arm64" ]]; then
-                ARCH="arm64"
-              else
-                ARCH="amd64"
-              fi
-              OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-mac-${ARCH}.zip"
-              OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-mac-${ARCH}"
+          # The self-hosted Linux runners do not ship the aws CLI (GitHub-hosted
+          # images did). Install it on demand so R2 uploads survive a fresh runner
+          # instead of hard-failing here.
+          if ! command -v aws >/dev/null 2>&1; then
+            echo "aws CLI not found on runner; installing..."
+            if command -v apt-get >/dev/null 2>&1; then
+              sudo apt-get update && sudo apt-get install -y awscli
+            elif command -v brew >/dev/null 2>&1; then
+              brew install awscli
+            else
+              echo "❌ aws CLI missing and no apt-get/brew to install it; cannot upload to R2"
+              exit 1
+            fi
+          fi

-              curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
-              unzip "$OSSUTIL_ZIP"
-              mv "${OSSUTIL_DIR}/ossutil" /usr/local/bin/
-              rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
-              chmod +x /usr/local/bin/ossutil
-              OSSUTIL_BIN=ossutil
-              ;;
-            windows)
-              OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-windows-amd64.zip"
-              OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-windows-amd64"
-
-              curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
-              unzip "$OSSUTIL_ZIP"
-              mv "${OSSUTIL_DIR}/ossutil.exe" ./ossutil.exe
-              rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
-              OSSUTIL_BIN=./ossutil.exe
-              ;;
-          esac
+          export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
+          export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
+          export AWS_DEFAULT_REGION="auto"

          # Determine upload path based on build type
          if [[ "$BUILD_TYPE" == "development" ]]; then
-            OSS_PATH="oss://rustfs-artifacts/artifacts/rustfs/dev/"
-            echo "📤 Uploading development build to OSS dev directory"
+            R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/dev/"
+            echo "📤 Uploading development build to R2 dev directory"
          else
-            OSS_PATH="oss://rustfs-artifacts/artifacts/rustfs/release/"
-            echo "📤 Uploading release build to OSS release directory"
+            R2_PATH="s3://${R2_BUCKET}/artifacts/rustfs/release/"
+            echo "📤 Uploading release build to R2 release directory"
          fi

-          # Upload all rustfs zip files to OSS using glob pattern
-          echo "📤 Uploading all rustfs-*.zip files to $OSS_PATH..."
+          # Upload all rustfs zip files to R2 using S3-compatible API
+          echo "📤 Uploading all rustfs-*.zip files to $R2_PATH..."
          for zip_file in rustfs-*.zip; do
            if [[ -f "$zip_file" ]]; then
-              echo "Uploading: $zip_file to $OSS_PATH..."
-              $OSSUTIL_BIN cp "$zip_file" "$OSS_PATH" --force
+              echo "Uploading: $zip_file to $R2_PATH..."
+              aws s3 cp "$zip_file" "$R2_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
              echo "✅ Uploaded: $zip_file"
            fi
          done
@@ -551,7 +693,7 @@ jobs:
    name: Build Summary
    needs: [ build-check, build-rustfs ]
    if: always() && needs.build-check.outputs.should_build == 'true'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
    steps:
      - name: Build completion summary
        shell: bash
@@ -603,7 +745,7 @@ jobs:
    name: Create GitHub Release
    needs: [ build-check, build-rustfs ]
    if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
    permissions:
      contents: write
    outputs:
@@ -611,7 +753,7 @@ jobs:
      release_url: ${{ steps.create.outputs.release_url }}
    steps:
      - name: Checkout repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
        with:
          fetch-depth: 0

@@ -680,8 +822,8 @@ jobs:
            RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
          fi

-          echo "release_id=$RELEASE_ID" >> $GITHUB_OUTPUT
-          echo "release_url=$RELEASE_URL" >> $GITHUB_OUTPUT
+          echo "release_id=$RELEASE_ID" >> "$GITHUB_OUTPUT"
+          echo "release_url=$RELEASE_URL" >> "$GITHUB_OUTPUT"
          echo "Created release: $RELEASE_URL"

  # Prepare and upload release assets
@@ -689,16 +831,16 @@ jobs:
    name: Upload Release Assets
    needs: [ build-check, build-rustfs, create-release ]
    if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
    permissions:
      contents: write
      actions: read
    steps:
      - name: Checkout repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7

      - name: Download all build artifacts
-        uses: actions/download-artifact@v7
+        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
        with:
          path: ./artifacts
          pattern: rustfs-*
@@ -730,16 +872,23 @@ jobs:
          cd ./release-assets

          # Generate checksums for all files (including latest versions)
-          if ls *.zip >/dev/null 2>&1; then
-            sha256sum *.zip > SHA256SUMS
-            sha512sum *.zip > SHA512SUMS
+          if compgen -G "*.zip" >/dev/null; then
+            sha256sum -- *.zip > SHA256SUMS
+            sha512sum -- *.zip > SHA512SUMS
          fi

-          # Create signature placeholder files
-          for file in *.zip; do
-            echo "# Signature for $file" > "${file}.asc"
-            echo "# GPG signature will be added in future versions" >> "${file}.asc"
-          done
+          cd ..
+          python3 scripts/security/generate_release_supply_chain_assets.py \
+            --asset-dir ./release-assets \
+            --version "$VERSION" \
+            --tag "$TAG" \
+            --build-type "${{ needs.build-check.outputs.build_type }}" \
+            --repository "${{ github.repository }}" \
+            --ref "${{ github.ref }}" \
+            --sha "${{ github.sha }}" \
+            --run-id "${{ github.run_id }}" \
+            --run-attempt "${{ github.run_attempt }}"
+          cd ./release-assets

          echo "📦 Prepared assets:"
          ls -la
@@ -765,12 +914,15 @@ jobs:

          echo "✅ All assets uploaded successfully"

-  # Update latest.json for stable releases only
+  # Update latest.json for every release tag (stable and prerelease): the
+  # project currently ships prerelease tags only, so gating this to stable
+  # left the version pointer permanently stale. release_type records whether
+  # the pointed-to version is a prerelease.
  update-latest-version:
    name: Update Latest Version
    needs: [ build-check, upload-release-assets ]
    if: startsWith(github.ref, 'refs/tags/')
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
    steps:
      - name: Update latest.json
        env:
@@ -788,16 +940,25 @@ jobs:
          VERSION="${{ needs.build-check.outputs.version }}"
          TAG="${{ needs.build-check.outputs.version }}"

+          if [[ "${{ needs.build-check.outputs.build_type }}" == "prerelease" ]]; then
+            RELEASE_TYPE="prerelease"
+          else
+            RELEASE_TYPE="stable"
+          fi
+
          # Install ossutil
          OSSUTIL_VERSION="2.1.1"
          OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-linux-amd64.zip"
          OSSUTIL_DIR="ossutil-${OSSUTIL_VERSION}-linux-amd64"
+          OSSUTIL_SHA256="60f3a808cbbdfd4b5269b8f967c6a66f564c9dacff52d93a5d21a588976ab5a2"

-          curl -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
+          curl --fail -L -o "$OSSUTIL_ZIP" "https://gosspublic.alicdn.com/ossutil/v2/${OSSUTIL_VERSION}/${OSSUTIL_ZIP}"
+          printf '%s  %s\n' "$OSSUTIL_SHA256" "$OSSUTIL_ZIP" | sha256sum -c -
          unzip "$OSSUTIL_ZIP"
-          mv "${OSSUTIL_DIR}/ossutil" /usr/local/bin/
+          OSSUTIL_BIN="${RUNNER_TEMP}/ossutil"
+          mv "${OSSUTIL_DIR}/ossutil" "$OSSUTIL_BIN"
          rm -rf "$OSSUTIL_DIR" "$OSSUTIL_ZIP"
-          chmod +x /usr/local/bin/ossutil
+          chmod +x "$OSSUTIL_BIN"

          # Create latest.json
          cat > latest.json << EOF
@@ -805,27 +966,27 @@ jobs:
            "version": "${VERSION}",
            "tag": "${TAG}",
            "release_date": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
-            "release_type": "stable",
+            "release_type": "${RELEASE_TYPE}",
            "download_url": "https://github.com/${{ github.repository }}/releases/tag/${TAG}"
          }
          EOF

          # Upload to OSS
-          ossutil cp latest.json oss://rustfs-version/latest.json --force
+          "$OSSUTIL_BIN" cp latest.json oss://rustfs-version/latest.json --force

-          echo "✅ Updated latest.json for stable release $VERSION"
+          echo "✅ Updated latest.json for ${RELEASE_TYPE} release $VERSION"

  # Publish release (remove draft status)
  publish-release:
    name: Publish Release
    needs: [ build-check, create-release, upload-release-assets ]
    if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
    permissions:
      contents: write
    steps:
      - name: Checkout repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7

      - name: Update release notes and publish
        env:
@@ -0,0 +1,70 @@
+# Copyright 2026 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Companion to ci.yml for the required "Test and Lint" status check.
+#
+# ci.yml skips docs-only pull requests via paths-ignore, but the branch
+# ruleset requires a check named "Test and Lint" — without this workflow a
+# docs-only PR would wait on that check forever. This workflow triggers on
+# exactly the paths ci.yml ignores and reports an instant success under the
+# same job name. Mixed PRs trigger both workflows and the real check still
+# gates: a required check with any failing run blocks the merge.
+# https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/troubleshooting-required-status-checks#handling-skipped-but-required-checks
+#
+# Keep the paths list below in sync with the pull_request paths-ignore list
+# in ci.yml.
+
+name: Continuous Integration (docs only)
+
+on:
+  pull_request:
+    types: [ opened, synchronize, reopened ]
+    branches: [ main ]
+    paths:
+      - "**.md"
+      - "docs/**"
+      - "deploy/**"
+      - "scripts/dev_*.sh"
+      - "scripts/probe.sh"
+      - "LICENSE*"
+      - ".gitignore"
+      - ".dockerignore"
+      - "README*"
+      - "**/*.png"
+      - "**/*.jpg"
+      - "**/*.svg"
+      - ".github/workflows/build.yml"
+      - ".github/workflows/docker.yml"
+      - ".github/workflows/audit.yml"
+
+permissions:
+  contents: read
+
+jobs:
+  test-and-lint:
+    name: Test and Lint
+    runs-on: ubuntu-latest
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      # Docs-only PRs skip the full code CI, but they are exactly where a
+      # planning-type document could be slipped in (git add -f bypasses
+      # .gitignore). Run the guard here so the required "Test and Lint" check
+      # stays meaningful for docs-only changes.
+      - name: Check no planning docs committed
+        run: ./scripts/check_no_planning_docs.sh
+
+      - name: Satisfy required check for docs-only changes
+        run: echo "Docs-only change — code CI is skipped by paths-ignore; planning-docs guard passed, reporting success for the required 'Test and Lint' check."
@@ -33,9 +33,11 @@ on:
      - ".github/workflows/build.yml"
      - ".github/workflows/docker.yml"
      - ".github/workflows/audit.yml"
-      - ".github/workflows/performance.yml"
  pull_request:
+    types: [ opened, synchronize, reopened, closed ]
    branches: [ main ]
+    # Keep this list in sync with the `paths` list in ci-docs-only.yml, which
+    # reports the required "Test and Lint" check for PRs skipped here.
    paths-ignore:
      - "**.md"
      - "docs/**"
@@ -52,9 +54,8 @@ on:
      - ".github/workflows/build.yml"
      - ".github/workflows/docker.yml"
      - ".github/workflows/audit.yml"
-      - ".github/workflows/performance.yml"
  merge_group:
-    types: [checks_requested]
+    types: [ checks_requested ]
  schedule:
    - cron: "0 0 * * 0" # Weekly on Sunday at midnight UTC
  workflow_dispatch:
@@ -62,130 +63,386 @@ on:
 permissions:
  contents: read

+# Concurrency groups are scoped per event so different triggers never cancel
+# each other: PR pushes cancel the previous run of that PR, main pushes keep
+# latest-wins semantics among themselves, and scheduled runs always complete
+# (a shared group used to let every merge kill the weekly scheduled run).
 concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: true
+  group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: ${{ github.event_name != 'schedule' }}

 env:
  CARGO_TERM_COLOR: always
  RUST_BACKTRACE: 1
-  CARGO_BUILD_JOBS: 2

 jobs:

-  skip-check:
-    name: Skip Duplicate Actions
-    permissions:
-      actions: write
-      contents: read
-    runs-on: ubicloud-standard-2
-    outputs:
-      should_skip: ${{ steps.skip_check.outputs.should_skip }}
+  cancel-closed-pr-runs:
+    name: Cancel Closed PR Runs
+    if: github.event_name == 'pull_request' && github.event.action == 'closed'
+    runs-on: ubuntu-latest
    steps:
-      - name: Skip duplicate actions
-        id: skip_check
-        uses: fkirc/skip-duplicate-actions@v5
-        with:
-          concurrent_skipping: "same_content_newer"
-          cancel_others: true
-          paths_ignore: '["*.md", "docs/**", "deploy/**"]'
-          do_not_skip: '["workflow_dispatch", "schedule", "merge_group", "release", "push"]'
+      - name: Explain cancellation run
+        run: echo "PR closed; this run only cancels older runs in the same concurrency group."

  typos:
    name: Typos
-    needs: skip-check
-    if: needs.skip-check.outputs.should_skip != 'true'
-    runs-on: ubicloud-standard-2
+    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    runs-on: ubuntu-latest
    steps:
-      - uses: actions/checkout@v6
-      - uses: dtolnay/rust-toolchain@stable
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
      - name: Typos check with custom config file
-        uses: crate-ci/typos@master
+        uses: crate-ci/typos@37bb98842b0d8c4ffebdb75301a13db0267cef89 # master
+
+  # Fast, compile-free checks that fail early so contributors get feedback in
+  # ~1 minute instead of waiting for the full test job.
+  quick-checks:
+    name: Quick Checks
+    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Install ripgrep
+        run: sudo apt-get update && sudo apt-get install -y ripgrep
+
+      - name: Install Rust toolchain
+        uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
+        with:
+          components: rustfmt
+
+      - name: Check code formatting
+        run: cargo fmt --all --check
+
+      - name: Check unsafe code allowances
+        run: ./scripts/check_unsafe_code_allowances.sh
+
+      - name: Check layered dependencies
+        run: ./scripts/check_layer_dependencies.sh
+
+      - name: Check architecture migration rules
+        run: ./scripts/check_architecture_migration_rules.sh
+
+      - name: Check tokio io-uring feature guard
+        run: ./scripts/check_no_tokio_io_uring.sh
+
+      - name: Check extension schema boundaries
+        run: ./scripts/check_extension_schema_boundaries.sh
+
+      - name: Check body-cache whitelist guard
+        run: ./scripts/check_body_cache_whitelist.sh
+
+      - name: Check no planning docs committed
+        run: ./scripts/check_no_planning_docs.sh

  test-and-lint:
    name: Test and Lint
-    needs: skip-check
-    if: needs.skip-check.outputs.should_skip != 'true'
-    runs-on: ubicloud-standard-4
+    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    runs-on: sm-standard-4
    timeout-minutes: 60
    env:
      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
    steps:
      - name: Checkout repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7

      - name: Setup Rust environment
        uses: ./.github/actions/setup
        with:
          rust-version: stable
-          cache-shared-key: ci-test-${{ hashFiles('**/Cargo.lock') }}
+          cache-shared-key: ci-test
          github-token: ${{ secrets.GITHUB_TOKEN }}
          cache-save-if: ${{ github.ref == 'refs/heads/main' }}

+      # Clippy runs before the test pass: lint failures are the most common
+      # CI-only breakage and should surface in minutes, not after 20+ minutes
+      # of tests.
+      - name: Run clippy lints
+        run: cargo clippy --all-targets -- -D warnings
+
      - name: Run tests
        run: |
-          cargo nextest run --all --exclude e2e_test
+          cargo nextest run --profile ci --all --exclude e2e_test
          cargo test --all --doc

-      - name: Check code formatting
-        run: cargo fmt --all --check
+      # rustfs/backlog#1289: fail if a seed rule's log anchor no longer exists
+      # verbatim in the source tree (log message drifted without updating the
+      # rule). Placed here where the workspace — including the la-dump-anchors
+      # bin — is already built by the clippy/test steps above.
+      - name: Check log-analyzer rule anchors
+        run: ./scripts/check_log_analyzer_rules.sh

-      - name: Run clippy lints
-        run: cargo clippy --all-targets --all-features -- -D warnings
+      - name: Upload test junit report
+        if: always()
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: junit-test-and-lint-${{ github.run_number }}
+          path: target/nextest/ci/junit.xml
+          retention-days: 3
+          if-no-files-found: ignore

-      - name: Check layered dependencies
-        run: ./scripts/check_layer_dependencies.sh
+      # Explicit gate for migration-critical suites. These tests already ran in
+      # the full nextest pass above; a single filtered nextest invocation keeps
+      # the named gate without rebuilding or re-running them one package at a time.
+      #
+      # The gate selects tests by name substring (data_movement / rebalance /
+      # decommission / source_cleanup / delete_marker), so renames can silently
+      # thin it. The script owns the filter expression and first verifies the
+      # selected-test count against the committed floor in
+      # .config/migration-gate-floor.txt before running the gate; renames or
+      # removals must update that file consciously (see the script header).
+      # Kept on the default profile (no --profile ci): a second --profile ci run
+      # would clobber target/nextest/ci/junit.xml, and none of these tests are
+      # quarantined so they gain nothing from the ci profile's retry overrides.
+      - name: Run rebalance/decommission migration proofs
+        run: ./scripts/check_migration_gate_count.sh
+
+  # Dedicated serial lane for the ILM / lifecycle integration tests. These tests
+  # drive the object layer through process-global singletons (the GLOBAL_ENV
+  # ECStore, the global tier-config manager, background-expiry workers) and bind
+  # fixed ports (9002 for the scanner suite, 9003 for the app suite), so they are
+  # marked #[ignore] to keep them out of the default parallel `cargo nextest run
+  # --all` pass. Run them here explicitly with `--run-ignored ignored-only` and
+  # `-j1` so no two of them share global state or race for a port at once.
+  # serial_test's #[serial] does NOT serialize across nextest's process-per-test
+  # boundary (see .config/nextest.toml); `-j1` is what actually serializes them.
+  # See rustfs/backlog#1148 (ilm-1) and #1155.
+  test-ilm-integration-serial:
+    name: ILM Integration (serial)
+    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    runs-on: sm-standard-4
+    timeout-minutes: 45
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-ilm-serial
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      # test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
+      # its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
+      # disk_index 0), not an EC metadata-distribution issue.
+      # restore_object_usecase_reports_ongoing_conflict_and_completion was
+      # re-enabled by backlog#1304 (restore accepts serialize on a short CAS
+      # guard; the copy-back no longer holds the #4877 whole-copy-back lock,
+      # so the mid-restore ongoing read and fast 409 rejection it asserts are
+      # the implemented contract). The remaining exclusions each hit a
+      # DIFFERENT, independent issue (all tracked under rustfs/backlog#1148;
+      # they keep #[ignore] with a backlog reference):
+      #   - test_noncurrent_{expiry,transition}_still_works_after_immediate_compensation_transition:
+      #     noncurrent transition/expiry after an immediate compensation transition.
+      - name: Run ignored ILM integration tests serially
+        run: |
+          cargo nextest run -j1 --run-ignored ignored-only \
+            -p rustfs-scanner -p rustfs \
+            -E '(binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))) and not (test(test_noncurrent_expiry_still_works_after_immediate_compensation_transition) or test(test_noncurrent_transition_still_works_after_immediate_compensation_transition))'
+
+  test-and-lint-rio-v2:
+    name: Test and Lint (rio-v2)
+    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    runs-on: sm-standard-4
+    timeout-minutes: 60
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-test-rio-v2
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      - name: Run rio-v2 clippy lints
+        run: cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
+
+      - name: Run rio-v2 feature tests
+        run: |
+          cargo nextest run -p rustfs -p rustfs-ecstore --features rio-v2
+          cargo test -p rustfs --doc --features rio-v2
+
+  test-and-lint-protocols:
+    name: "Test and Lint (${{ matrix.features.name }})"
+    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    runs-on: sm-standard-4
+    timeout-minutes: 60
+    strategy:
+      fail-fast: false
+      matrix:
+        features:
+          - name: swift
+            flags: "--features swift"
+          - name: sftp
+            flags: "--features sftp"
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-test-${{ matrix.features.name }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      - name: Run clippy with ${{ matrix.features.name }}
+        run: |
+          cargo clippy -p rustfs -p rustfs-protocols --all-targets ${{ matrix.features.flags }} -- -D warnings
+
+      - name: Run tests with ${{ matrix.features.name }}
+        run: |
+          cargo nextest run -p rustfs -p rustfs-protocols ${{ matrix.features.flags }}

  build-rustfs-debug-binary:
    name: Build RustFS Debug Binary
-    needs: skip-check
-    if: needs.skip-check.outputs.should_skip != 'true'
-    runs-on: ubicloud-standard-4
+    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    runs-on: sm-standard-4
    timeout-minutes: 30
    env:
      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
    steps:
      - name: Checkout repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7

      - name: Setup Rust environment
        uses: ./.github/actions/setup
        with:
          rust-version: stable
-          cache-shared-key: ci-rustfs-debug-binary-${{ hashFiles('**/Cargo.lock') }}
+          cache-shared-key: ci-rustfs-debug-binary
          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
          github-token: ${{ secrets.GITHUB_TOKEN }}

      - name: Build debug binary
-        run: |
-          touch rustfs/build.rs
-          cargo build -p rustfs --bins --jobs 2
+        run: cargo build -p rustfs --bins

      - name: Upload debug binary
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
        with:
          name: rustfs-debug-binary
          path: target/debug/rustfs
          if-no-files-found: error
          retention-days: 1

-  e2e-tests:
-    name: End-to-End Tests
-    needs: [ skip-check, build-rustfs-debug-binary ]
-    if: needs.skip-check.outputs.should_skip != 'true'
-    runs-on: ubicloud-standard-2
+  build-rustfs-debug-binary-rio-v2:
+    name: Build RustFS Debug Binary (rio-v2)
+    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    runs-on: sm-standard-4
+    timeout-minutes: 30
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-rustfs-debug-binary-rio-v2
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Build debug binary with rio-v2
+        run: cargo build -p rustfs --bins --features rio-v2
+
+      - name: Upload debug binary
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: rustfs-debug-binary-rio-v2
+          path: target/debug/rustfs
+          if-no-files-found: error
+          retention-days: 1
+
+  uring-integration:
+    name: io_uring Integration (real)
+    # GitHub-hosted ubuntu-latest runs a recent kernel with io_uring and, unlike
+    # a container, applies no seccomp filter that would block io_uring_setup — so
+    # the probe succeeds and the tests exercise the real UringBackend/FdCache/
+    # latch paths instead of the StdBackend fallback (rustfs/backlog#1179). The
+    # self-hosted sm-standard runners cannot guarantee this.
+    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - name: Checkout repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7

-      - name: Clean up previous test run
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-uring
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      - name: Install build dependencies
+        run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
+
+      # ext4 supports O_DIRECT; the runner's default TMPDIR may sit on tmpfs or
+      # overlayfs, where open(O_DIRECT) returns EINVAL/EOPNOTSUPP and the native
+      # read_at_direct path silently latches off to the aligned StdBackend
+      # fallback. A dedicated ext4 loopback guarantees the native O_DIRECT read
+      # path is actually exercised, not just the fallback (rustfs/backlog#1220).
+      - name: Prepare ext4 loopback for O_DIRECT coverage
        run: |
-          rm -rf /tmp/rustfs
-          rm -f /tmp/rustfs.log
+          set -euo pipefail
+          dd if=/dev/zero of=/tmp/rustfs-odirect.img bs=1M count=1024
+          mkfs.ext4 -q -F /tmp/rustfs-odirect.img
+          sudo mkdir -p /mnt/rustfs-odirect
+          sudo mount -o loop /tmp/rustfs-odirect.img /mnt/rustfs-odirect
+          sudo chmod 1777 /mnt/rustfs-odirect
+          mount | grep rustfs-odirect

+      # RUSTFS_URING_TESTS_MUST_RUN=1 makes the io_uring tests fail instead of
+      # silently skipping if io_uring is unavailable, so this leg can never pass
+      # vacuously. RUSTFS_IO_URING_READ_ENABLE=true selects the UringBackend.
+      # TMPDIR points every tempfile-based test fixture at the ext4 loopback so
+      # eligible reads keep O_DIRECT and the native path is covered end to end.
+      - name: Run ecstore io_uring integration tests on real io_uring
+        env:
+          RUSTFS_IO_URING_READ_ENABLE: "true"
+          RUSTFS_URING_TESTS_MUST_RUN: "1"
+          TMPDIR: /mnt/rustfs-odirect
+        run: cargo test -p rustfs-ecstore uring_ -- --test-threads=1 --nocapture
+
+  e2e-tests:
+    name: End-to-End Tests
+    needs: [ build-rustfs-debug-binary ]
+    runs-on: sm-standard-2
+    timeout-minutes: 30
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      # Full setup with dependency caching: the smoke-suite step below
+      # compiles the e2e_test crate, which pulls in most of the workspace.
+      # Without a restored cache this was a cold multi-GB build on every run.
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-e2e
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      # Download after the cache restore so the freshly built binary from the
+      # build job always wins over anything restored into target/debug.
      - name: Download debug binary
-        uses: actions/download-artifact@v7
+        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
        with:
          name: rustfs-debug-binary
          path: target/debug
@@ -193,15 +450,142 @@ jobs:
      - name: Make binary executable
        run: chmod +x ./target/debug/rustfs

-      - name: Setup Rust toolchain for s3s-e2e installation
-        uses: dtolnay/rust-toolchain@stable
+      # Guard the security negative-auth smoke subset (backlog#1151 sec-5)
+      # against a rename or deletion silently dropping it out of the e2e-smoke
+      # filter. The script lists what the profile selects and fails if the count
+      # of security auth-rejection tests falls below the committed floor in
+      # .config/security-smoke-floor.txt (infra-12 count-floor mechanism). Run
+      # before the smoke suite so a thinned gate fails fast; the `nextest list`
+      # here compiles the e2e_test binaries the run below reuses.
+      - name: Check security smoke subset count floor
+        run: ./scripts/check_security_smoke_count.sh check
+
+      # PR smoke subset of the in-repo e2e suite (backlog#1149 ci-4). The
+      # profile.e2e-smoke default-filter in .config/nextest.toml is the single
+      # wiring mechanism for e2e tests in CI — extend that filter instead of
+      # adding new e2e jobs here. Each test spawns its own rustfs server on a
+      # random port and reuses the downloaded debug binary above.
+      - name: Run e2e smoke suite
+        run: cargo nextest run --profile e2e-smoke -p e2e_test

      - name: Install s3s-e2e test tool
-        uses: taiki-e/cache-cargo-install-action@v2
+        uses: taiki-e/cache-cargo-install-action@7447f04c51f2ba27ca35e7f1e28fab848c5b3ba7 # v2
        with:
          tool: s3s-e2e
          git: https://github.com/s3s-project/s3s.git
-          rev: 4a04a670cf41274d9be9ab65dc36f4aa3f92fbad
+          rev: 62cb4a71dd759a6ec56b64c4c42fcc183a2c6a52
+
+      - name: Run end-to-end tests
+        run: |
+          s3s-e2e --version
+          RUN_ROOT="${RUNNER_TEMP}/rustfs-e2e-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_JOB}"
+          mkdir -p "${RUN_ROOT}"
+          RUSTFS_TEST_PORT="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()')"
+          RUSTFS_TEST_PORT="${RUSTFS_TEST_PORT}" \
+          RUSTFS_TEST_LOG="${RUN_ROOT}/rustfs.log" \
+          ./scripts/e2e-run.sh ./target/debug/rustfs "${RUN_ROOT}/data"
+
+      - name: Upload test logs
+        if: failure()
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: e2e-test-logs-${{ github.run_number }}
+          path: ${{ runner.temp }}/rustfs-e2e-*/rustfs.log
+          retention-days: 3
+
+  e2e-full:
+    name: End-to-End Tests (full merge gate)
+    # Merge gate only (backlog#1149 ci-5): the never-automated user-visible
+    # suites — KMS, object_lock, multipart_auth, quota, checksum, encryption,
+    # security-boundary, ... — via the e2e-full nextest profile. Too heavy for
+    # every PR, so it is gated to main pushes, the merge queue, and manual
+    # dispatch. protocols / the 6 cluster suites / replication / #[ignore] are
+    # owned by other lanes (see .config/nextest.toml profile.e2e-full).
+    if: >-
+      github.event_name == 'workflow_dispatch' ||
+      github.event_name == 'merge_group' ||
+      (github.event_name == 'push' && github.ref == 'refs/heads/main')
+    needs: [ build-rustfs-debug-binary ]
+    runs-on: sm-standard-2
+    timeout-minutes: 55
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-e2e
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      # Download after the cache restore so the freshly built binary from the
+      # build job always wins over anything restored into target/debug.
+      - name: Download debug binary
+        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
+        with:
+          name: rustfs-debug-binary
+          path: target/debug
+
+      - name: Make binary executable
+        run: chmod +x ./target/debug/rustfs
+
+      # Full single-node e2e lane (backlog#1149 ci-5). The e2e-full
+      # default-filter in .config/nextest.toml is the single wiring mechanism —
+      # extend that filter, never add ad-hoc e2e jobs here. Reuses the downloaded
+      # debug binary; each test spawns its own rustfs server on a random port.
+      - name: Run e2e full suite
+        run: cargo nextest run --profile e2e-full -p e2e_test
+
+      - name: Upload junit
+        if: always()
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: e2e-full-junit-${{ github.run_number }}
+          path: target/nextest/e2e-full/junit.xml
+          retention-days: 7
+
+  e2e-tests-rio-v2:
+    name: End-to-End Tests (rio-v2)
+    needs: [ build-rustfs-debug-binary-rio-v2 ]
+    runs-on: sm-standard-2
+    timeout-minutes: 30
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Clean up previous test run
+        run: |
+          rm -rf /tmp/rustfs
+          rm -f /tmp/rustfs.log
+
+      - name: Download debug binary
+        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
+        with:
+          name: rustfs-debug-binary-rio-v2
+          path: target/debug
+
+      - name: Make binary executable
+        run: chmod +x ./target/debug/rustfs
+
+      - name: Setup Rust toolchain for s3s-e2e installation
+        uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
+
+      # The sm-standard-* custom runner images (introduced in #4884) ship no C
+      # toolchain, unlike GitHub-hosted ubuntu-latest. Installing s3s-e2e below
+      # compiles it from source on a cache miss, and build scripts need cc.
+      - name: Install build tools for s3s-e2e compilation
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y build-essential cmake pkg-config libssl-dev
+
+      - name: Install s3s-e2e test tool
+        uses: taiki-e/cache-cargo-install-action@7447f04c51f2ba27ca35e7f1e28fab848c5b3ba7 # v2
+        with:
+          tool: s3s-e2e
+          git: https://github.com/s3s-project/s3s.git
+          rev: 62cb4a71dd759a6ec56b64c4c42fcc183a2c6a52

      - name: Run end-to-end tests
        run: |
@@ -210,24 +594,23 @@ jobs:

      - name: Upload test logs
        if: failure()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
        with:
-          name: e2e-test-logs-${{ github.run_number }}
+          name: e2e-test-logs-rio-v2-${{ github.run_number }}
          path: /tmp/rustfs.log
          retention-days: 3

  s3-implemented-tests:
    name: S3 Implemented Tests
-    needs: [ skip-check, build-rustfs-debug-binary ]
-    if: needs.skip-check.outputs.should_skip != 'true'
-    runs-on: ubicloud-standard-4
+    needs: [ build-rustfs-debug-binary, e2e-tests ]
+    runs-on: sm-standard-4
    timeout-minutes: 60
    steps:
      - name: Checkout repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7

      - name: Download debug binary
-        uses: actions/download-artifact@v7
+        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
        with:
          name: rustfs-debug-binary
          path: target/debug
@@ -237,17 +620,98 @@ jobs:

      - name: Run implemented s3-tests
        run: |
+          RUN_ROOT="${RUNNER_TEMP}/rustfs-s3tests-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_JOB}"
+          mkdir -p "${RUN_ROOT}"
+          S3_PORT="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()')"
          DEPLOY_MODE=binary \
          RUSTFS_BINARY=./target/debug/rustfs \
          TEST_MODE=single \
-          MAXFAIL=1 \
+          MAXFAIL=0 \
+          XDIST=4 \
+          S3_PORT="${S3_PORT}" \
+          DATA_ROOT="${RUN_ROOT}" \
+          S3TESTS_CONF=artifacts/s3tests-single/s3tests.conf \
          ./scripts/s3-tests/run.sh

      - name: Upload s3 test artifacts
        if: always()
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
        with:
          name: s3tests-implemented-${{ github.run_number }}
          path: artifacts/s3tests-single/**
          if-no-files-found: ignore
          retention-days: 3
+
+  # Dedicated lifecycle behavior lane (backlog#1148 ilm-10, master plan #1155).
+  #
+  # These ceph/s3-tests cases assert that objects/versions/uploads are actually
+  # removed by the background scanner and the stale-multipart cleanup loop, so
+  # they need a debug-accelerated day plus an enabled scanner. That configuration
+  # cannot be applied to the s3-implemented-tests lane above because a global
+  # RUSTFS_ILM_DEBUG_DAY_SECS also shrinks the x-amz-expiration header those tests
+  # assert on (test_lifecycle_expiration_header_*). Hence a separate server here.
+  #
+  # RUSTFS_ILM_DEBUG_DAY_SECS MUST stay equal to the s3-tests lc_debug_interval
+  # default (10s). test_lifecycle_expiration polls the Days=1 rule with a
+  # 4*lc_interval window while the Days=5 rule fires at 5*debug_day; observing
+  # the intermediate 4-object plateau requires 4*lc_interval < 5*debug_day, i.e.
+  # debug_day > 8. A smaller value lets the Days=5 rule fire inside the Days=1
+  # poll window so the count collapses straight to 2 and the test fails (#4764).
+  #
+  # RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES=1 is also required. A compacted directory
+  # is only re-descended (and its objects re-evaluated for ILM) once every
+  # DATA_USAGE_UPDATE_DIR_CYCLES scanner cycles (default 16). At the accelerated
+  # RUSTFS_SCANNER_CYCLE=2 that is ~32s between evaluations, so a Days=1 object
+  # that becomes due at debug_day (10s) is not actually expired until the next
+  # ~32s boundary (~42s), landing just past the 4*lc_interval (40s) poll window
+  # and making the count stall at 6 (#4767). Forcing every-cycle re-descent
+  # evaluates ILM within ~2s of the due time, well inside the poll window.
+  s3-lifecycle-behavior-tests:
+    name: S3 Lifecycle Behavior Tests
+    needs: [ build-rustfs-debug-binary ]
+    runs-on: sm-standard-4
+    timeout-minutes: 30
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Download debug binary
+        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
+        with:
+          name: rustfs-debug-binary
+          path: target/debug
+
+      - name: Make binary executable
+        run: chmod +x ./target/debug/rustfs
+
+      - name: Run lifecycle behavior s3-tests
+        run: |
+          RUN_ROOT="${RUNNER_TEMP}/rustfs-s3tests-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_JOB}"
+          mkdir -p "${RUN_ROOT}"
+          S3_PORT="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()')"
+          DEPLOY_MODE=binary \
+          RUSTFS_BINARY=./target/debug/rustfs \
+          TEST_MODE=single \
+          MAXFAIL=0 \
+          XDIST=0 \
+          IMPLEMENTED_TESTS_FILE=scripts/s3-tests/lifecycle_behavior_tests.txt \
+          RUSTFS_ILM_DEBUG_DAY_SECS=10 \
+          RUSTFS_ILM_PROCESS_TIME=1 \
+          RUSTFS_SCANNER_ENABLED=true \
+          RUSTFS_SCANNER_CYCLE=2 \
+          RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES=1 \
+          RUSTFS_SCANNER_START_DELAY_SECS=0 \
+          RUSTFS_API_STALE_UPLOADS_CLEANUP_INTERVAL=2s \
+          S3_PORT="${S3_PORT}" \
+          DATA_ROOT="${RUN_ROOT}" \
+          S3TESTS_CONF=artifacts/s3tests-single/s3tests.conf \
+          ./scripts/s3-tests/run.sh
+
+      - name: Upload s3 test artifacts
+        if: always()
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: s3tests-lifecycle-behavior-${{ github.run_number }}
+          path: artifacts/s3tests-single/**
+          if-no-files-found: ignore
+          retention-days: 3
@@ -16,7 +16,7 @@ name: CLA Check

 on:
  pull_request_target:
-    types: [opened, synchronize, reopened]
+    types: [opened, synchronize, reopened, closed]
  merge_group:
    types: [checks_requested]
  issue_comment:
@@ -28,14 +28,26 @@ permissions:
  issues: write
  checks: write

+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.event.issue.number || github.ref }}
+  cancel-in-progress: true
+
 jobs:
+  cancel-closed-pr-runs:
+    name: Cancel Closed PR Runs
+    if: github.event_name == 'pull_request_target' && github.event.action == 'closed'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Explain cancellation run
+        run: echo "PR closed; this run only cancels older runs in the same concurrency group."
+
  cla:
-    if: ${{ github.event_name != 'issue_comment' || github.event.issue.pull_request }}
+    if: ${{ (github.event_name != 'issue_comment' || github.event.issue.pull_request) && (github.event_name != 'pull_request_target' || github.event.action != 'closed') }}
    runs-on: ubuntu-latest
    steps:
      - name: Report CLA result for merge queue
        if: github.event_name == 'merge_group'
-        uses: actions/github-script@v8
+        uses: actions/github-script@ed597411d8f924073f98dfc5c65a23a2325f34cd # v8
        with:
          script: |
            await github.rest.checks.create({
@@ -54,7 +66,7 @@ jobs:
      - name: Create token for rustfs/cla
        if: github.event_name != 'merge_group'
        id: registry-token
-        uses: actions/create-github-app-token@v3
+        uses: actions/create-github-app-token@bcd2ba49218906704ab6c1aa796996da409d3eb1 # v3
        with:
          app-id: ${{ vars.CLA_BOT_APP_ID }}
          private-key: ${{ secrets.CLA_BOT_APP_PRIVATE_KEY }}
@@ -64,7 +76,7 @@ jobs:

      - name: Run CLA Bot
        if: github.event_name != 'merge_group'
-        uses: overtrue/cla-bot@v0.0.9
+        uses: overtrue/cla-bot@5e5929846a60cfb42b16d0d38e135ef67d94fc88 # v0.0.9
        with:
          github-token: ${{ github.token }}
          registry-token: ${{ steps.registry-token.outputs.token }}
@@ -0,0 +1,122 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Weekly workspace line-coverage baseline (backlog#1153 infra-5).
+#
+# NON-BLOCKING by design: this workflow only runs on schedule and manual
+# dispatch, so it never attaches a status to a PR and must never be made a
+# required check. It exists to give coverage a visible baseline and trend
+# (per-crate table in the job summary, lcov artifact kept 90 days) — the
+# per-crate ratchet for the security-critical crates builds on it later
+# (backlog#1153 infra-6, report-only first per the ci-11 ladder).
+#
+# Measurement scope matches the PR test gate (ci.yml "Run tests"):
+# `--workspace --exclude e2e_test` with the `ci` nextest profile. Doctests are
+# NOT measured (ci.yml runs them uninstrumented; `cargo llvm-cov` needs a
+# nightly toolchain to cover doctests). Trend-comparison workflow:
+# docs/testing/README.md "Coverage" section. `make coverage` is the local
+# equivalent. Scheduled failures alert via the ci-8 composite action.
+
+name: coverage
+
+on:
+  workflow_dispatch:
+  schedule:
+    # 07:00 UTC Sunday — staggered clear of the other Sunday crons: ci (00:00),
+    # build (01:00), e2e-s3tests (02:00), audit (03:00), nix-flake-update
+    # (05:00), mint (06:00), and the daily fuzz (02:00), minio-interop (03:17),
+    # e2e-replication-nightly (04:00) and performance-ab (06:00) lanes.
+    - cron: "0 7 * * 0"
+
+# Only alert-on-failure needs more than read access; it declares its own
+# job-level `issues: write`.
+permissions:
+  contents: read
+
+jobs:
+  coverage:
+    name: Workspace coverage (weekly)
+    runs-on: sm-standard-4
+    # The instrumented build cannot reuse the regular CI cache (different
+    # RUSTFLAGS), so a cold week rebuilds the workspace before running the
+    # full suite; give it double the test job's 60-minute budget.
+    timeout-minutes: 120
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+      # Match the PR gate's nextest semantics (ci.yml runs `--profile ci`):
+      # retries=0 plus the quarantine list and the ecstore-serial-flaky
+      # serialization. Set via env because `cargo llvm-cov`'s own --profile
+      # flag selects the *cargo build* profile, not the nextest profile.
+      NEXTEST_PROFILE: ci
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-coverage
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      - name: Install cargo-llvm-cov
+        uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
+        with:
+          tool: cargo-llvm-cov
+
+      - name: Install llvm-tools component
+        run: rustup component add llvm-tools-preview
+
+      - name: Run instrumented test suite
+        run: cargo llvm-cov nextest --workspace --exclude e2e_test --no-report
+
+      - name: Generate lcov and JSON reports
+        run: |
+          mkdir -p target/llvm-cov
+          cargo llvm-cov report --lcov --output-path target/llvm-cov/lcov.info
+          cargo llvm-cov report --json --output-path target/llvm-cov/coverage.json
+
+      - name: Write per-crate summary
+        run: python3 scripts/coverage_per_crate.py target/llvm-cov/coverage.json >> "$GITHUB_STEP_SUMMARY"
+
+      - name: Upload coverage artifact
+        if: always()
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: coverage-lcov-${{ github.run_number }}
+          path: |
+            target/llvm-cov/lcov.info
+            target/llvm-cov/coverage.json
+          retention-days: 90
+          if-no-files-found: ignore
+
+  alert-on-failure:
+    name: Alert on scheduled failure
+    needs: [coverage]
+    # Only scheduled runs open/append the tracking issue (backlog#1149 ci-8);
+    # manual workflow_dispatch runs stay quiet so a debugging run never files a
+    # spurious alert.
+    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+      issues: write
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+      - name: Open or update failure-tracking issue
+        uses: ./.github/actions/schedule-failure-issue
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
@@ -70,10 +70,20 @@ env:
  DOCKER_PLATFORMS: linux/amd64,linux/arm64

 jobs:
-  # Check if we should build Docker images
+  # Check if we should build Docker images.
+  # Short-circuit at job level so per-merge development builds (workflow_run
+  # from a push to main) skip the whole run without spawning a checkout job:
+  # images are only published for tag builds — build.yml push triggers cover
+  # only main and tags, so a non-main head_branch is a tag name — and for
+  # manual dispatch.
  build-check:
    name: Docker Build Check
-    runs-on: ubicloud-standard-2
+    if: >-
+      github.event_name == 'workflow_dispatch' ||
+      (github.event.workflow_run.conclusion == 'success' &&
+       github.event.workflow_run.event == 'push' &&
+       github.event.workflow_run.head_branch != 'main')
+    runs-on: ubuntu-latest
    outputs:
      should_build: ${{ steps.check.outputs.should_build }}
      should_push: ${{ steps.check.outputs.should_push }}
@@ -84,9 +94,8 @@ jobs:
      create_latest: ${{ steps.check.outputs.create_latest }}
    steps:
      - name: Checkout repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
        with:
-          fetch-depth: 0
          # For workflow_run events, checkout the specific commit that triggered the workflow
          ref: ${{ github.event.workflow_run.head_sha || github.sha }}

@@ -163,11 +172,10 @@ jobs:
                if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]] || [[ "$version" == *"rc"* ]]; then
                  build_type="prerelease"
                  is_prerelease=true
-                  # TODO: Temporary change - currently allows alpha versions to also create latest tags
-                  # After the version is stable, you need to remove the following line and restore the original logic (latest is created only for stable versions)
-                  if [[ "$version" == *"alpha"* ]]; then
+                  # Current policy: create latest tags for stable releases and selected prereleases (alpha/beta).
+                  if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]]; then
                    create_latest=true
-                     echo "🧪 Building Docker image for prerelease: $version (temporarily allowing creation of latest tag)"
+                    echo "🧪 Building Docker image for prerelease: $version (creating latest tag)"
                  else
                    echo "🧪 Building Docker image for prerelease: $version"
                  fi
@@ -216,11 +224,10 @@ jobs:
              v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
                build_type="prerelease"
                is_prerelease=true
-                # TODO: Temporary change - currently allows alpha versions to also create latest tags
-                # After the version is stable, you need to remove the if block below and restore the original logic.
-                if [[ "$input_version" == *"alpha"* ]]; then
+                # Current policy: create latest tags for stable releases and selected prereleases (alpha/beta).
+                if [[ "$version" == *"alpha"* ]] || [[ "$version" == *"beta"* ]]; then
                  create_latest=true
-                   echo "🧪 Building with prerelease version: $input_version (temporarily allowing creation of latest tag)"
+                  echo "🧪 Building with prerelease version: $input_version (creating latest tag)"
                else
                  echo "🧪 Building with prerelease version: $input_version"
                fi
@@ -240,13 +247,15 @@ jobs:
            esac
          fi

-          echo "should_build=$should_build" >> $GITHUB_OUTPUT
-          echo "should_push=$should_push" >> $GITHUB_OUTPUT
-          echo "build_type=$build_type" >> $GITHUB_OUTPUT
-          echo "version=$version" >> $GITHUB_OUTPUT
-          echo "short_sha=$short_sha" >> $GITHUB_OUTPUT
-          echo "is_prerelease=$is_prerelease" >> $GITHUB_OUTPUT
-          echo "create_latest=$create_latest" >> $GITHUB_OUTPUT
+          {
+            echo "should_build=$should_build"
+            echo "should_push=$should_push"
+            echo "build_type=$build_type"
+            echo "version=$version"
+            echo "short_sha=$short_sha"
+            echo "is_prerelease=$is_prerelease"
+            echo "create_latest=$create_latest"
+          } >> "$GITHUB_OUTPUT"

          echo "🐳 Docker Build Summary:"
          echo "  - Should build: $should_build"
@@ -265,55 +274,54 @@ jobs:
    name: Build Docker Images
    needs: build-check
    if: needs.build-check.outputs.should_build == 'true'
-    runs-on: ubicloud-standard-2
+    runs-on: dind-sm-standard-2
    timeout-minutes: 60
    strategy:
-          fail-fast: false
-          matrix:
-            include:
-              - variant: musl
-                file: Dockerfile
-                suffix: ""
-              - variant: glibc
-                file: Dockerfile.glibc
-                suffix: "-glibc"
+      fail-fast: false
+      matrix:
+        include:
+          - variant: musl
+            file: Dockerfile
+            suffix: ""
+          - variant: glibc
+            file: Dockerfile.glibc
+            suffix: "-glibc"

    steps:
      - name: Checkout repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7

      - name: Login to Docker Hub
-        uses: docker/login-action@v3
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
        with:
          username: ${{ env.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Login to GitHub Container Registry
-        uses: docker/login-action@v3
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
        with:
          registry: ghcr.io
          username: ${{ secrets.GHCR_USERNAME }}
          password: ${{ secrets.GHCR_PASSWORD }}

      - name: Login to Quay.io
-        uses: docker/login-action@v3
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
        with:
          registry: quay.io
          username: ${{ secrets.QUAY_USERNAME }}
          password: ${{ secrets.QUAY_PASSWORD }}

      - name: Set up QEMU
-        uses: docker/setup-qemu-action@v3
+        uses: docker/setup-qemu-action@c7c53464625b32c7a7e944ae62b3e17d2b600130 # v3

      - name: Set up Docker Buildx
-        uses: docker/setup-buildx-action@v3
+        uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3

      - name: Extract metadata and generate tags
        id: meta
        run: |
          BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
          VERSION="${{ needs.build-check.outputs.version }}"
-          SHORT_SHA="${{ needs.build-check.outputs.short_sha }}"
          CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
          VARIANT_SUFFIX="${{ matrix.suffix }}"

@@ -336,8 +344,8 @@ jobs:
              ;;
          esac

-          echo "docker_release=$DOCKER_RELEASE" >> $GITHUB_OUTPUT
-          echo "docker_channel=$DOCKER_CHANNEL" >> $GITHUB_OUTPUT
+          echo "docker_release=$DOCKER_RELEASE" >> "$GITHUB_OUTPUT"
+          echo "docker_channel=$DOCKER_CHANNEL" >> "$GITHUB_OUTPUT"

          echo "🐳 Docker build parameters:"
          echo "  - Original version: $VERSION"
@@ -351,9 +359,7 @@ jobs:

          # Add channel tags for prereleases and latest for stable
          if [[ "$CREATE_LATEST" == "true" ]]; then
-            # TODO: Temporary change - the current alpha version will also create the latest tag
-            # After the version is stabilized, the logic here remains unchanged, but the upstream CREATE_LATEST setting needs to be restored.
-            # Stable release (and temporary alpha versions)
+            # Create latest tags for stable releases and selected prereleases when CREATE_LATEST=true.
            TAGS="$TAGS,${{ env.REGISTRY_DOCKERHUB }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_GHCR }}:latest${VARIANT_SUFFIX},${{ env.REGISTRY_QUAY }}:latest${VARIANT_SUFFIX}"
          elif [[ "$BUILD_TYPE" == "prerelease" ]]; then
            # Prerelease channel tags (alpha, beta, rc)
@@ -371,7 +377,7 @@ jobs:
          fi

          # Output tags
-          echo "tags=$TAGS" >> $GITHUB_OUTPUT
+          echo "tags=$TAGS" >> "$GITHUB_OUTPUT"

          # Generate labels
          LABELS="org.opencontainers.image.title=RustFS"
@@ -382,7 +388,7 @@ jobs:
          LABELS="$LABELS,org.opencontainers.image.created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
          LABELS="$LABELS,org.opencontainers.image.build-type=$BUILD_TYPE"

-          echo "labels=$LABELS" >> $GITHUB_OUTPUT
+          echo "labels=$LABELS" >> "$GITHUB_OUTPUT"

          echo "🐳 Generated Docker tags:"
          echo "$TAGS" | tr ',' '\n' | sed 's/^/  - /'
@@ -390,7 +396,7 @@ jobs:
          echo "🔖 Version: $VERSION"

      - name: Build and push Docker image
-        uses: docker/build-push-action@v6
+        uses: docker/build-push-action@10e90e3645eae34f1e60eeb005ba3a3d33f178e8 # v6
        with:
          context: .
          file: ${{ matrix.file }}
@@ -410,9 +416,8 @@ jobs:
            RELEASE=${{ steps.meta.outputs.docker_release }}
            CHANNEL=${{ steps.meta.outputs.docker_channel }}
            BUILDKIT_INLINE_CACHE=1
-          # Enable advanced BuildKit features for better performance
-          provenance: false
-          sbom: false
+          provenance: true
+          sbom: true
          # Add retry mechanism by splitting the build process
          no-cache: false
          pull: true
@@ -420,12 +425,66 @@ jobs:
  # Note: Manifest creation is no longer needed as we only build one variant
  # Multi-arch manifests are automatically created by docker/build-push-action

+  # Report-only container image vulnerability scan
+  scan-docker-image:
+    name: Scan Docker Images
+    needs: [ build-check, build-docker ]
+    if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
+    runs-on: ubuntu-latest
+    permissions:
+      contents: read
+      security-events: write
+    strategy:
+      fail-fast: false
+      matrix:
+        include:
+          - variant: musl
+            suffix: ""
+          - variant: glibc
+            suffix: "-glibc"
+    steps:
+      - name: Login to GitHub Container Registry
+        # docker/login-action v3
+        uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9
+        with:
+          registry: ghcr.io
+          username: ${{ secrets.GHCR_USERNAME }}
+          password: ${{ secrets.GHCR_PASSWORD }}
+
+      - name: Scan image with Trivy
+        # aquasecurity/trivy-action v0.36.0
+        uses: aquasecurity/trivy-action@ed142fd0673e97e23eac54620cfb913e5ce36c25
+        with:
+          image-ref: ${{ env.REGISTRY_GHCR }}:${{ needs.build-check.outputs.version }}${{ matrix.suffix }}
+          format: sarif
+          output: trivy-${{ matrix.variant }}.sarif
+          ignore-unfixed: true
+          vuln-type: os,library
+          severity: CRITICAL,HIGH
+          exit-code: "0"
+
+      # Surface findings in the Security tab; an artifact alone is write-only
+      # reporting nobody reviews.
+      - name: Upload scan results to code scanning
+        uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4.37.0
+        with:
+          sarif_file: trivy-${{ matrix.variant }}.sarif
+          category: container-image-${{ matrix.variant }}
+
+      - name: Upload container scan report
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: container-image-scan-${{ matrix.variant }}
+          path: trivy-${{ matrix.variant }}.sarif
+          if-no-files-found: error
+          retention-days: 30
+
  # Docker build summary
  docker-summary:
    name: Docker Build Summary
    needs: [ build-check, build-docker ]
    if: always() && needs.build-check.outputs.should_build == 'true'
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
    steps:
      - name: Docker build completion summary
        run: |
@@ -450,10 +509,9 @@ jobs:
            "prerelease")
              echo "🧪 Prerelease Docker image has been built with ${VERSION} tags"
              echo "⚠️  This is a prerelease image - use with caution"
-              # TODO: Temporary change - alpha versions currently create the latest tag
-              # After the version is stable, you need to restore the following prompt information
-              if [[ "$VERSION" == *"alpha"* ]] && [[ "$CREATE_LATEST" == "true" ]]; then
-                echo "🏷️  Latest tag has been created for alpha version (temporary measures)"
+              # Create latest tags for stable releases and selected prereleases when CREATE_LATEST=true.
+              if [[ "$CREATE_LATEST" == "true" ]]; then
+                echo "🏷️  Latest tag has been created for prerelease: $VERSION"
              else
                echo "🚫 Latest tag NOT created for prerelease"
              fi
@@ -0,0 +1,130 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# Nightly full replication e2e lane (backlog#1147 repl-1, deps: ci-4).
+#
+# The per-PR gate (ci.yml `e2e-tests` job, `--profile e2e-smoke`) runs the 20
+# FAST replication tests. This scheduled lane runs the remaining 27
+# heavier replication e2e tests that are unfit for a per-PR gate:
+#
+#   * 2 remote-target TLS validation tests.
+#   * 12 bucket-replication data-plane/helper tests (PUT/delete + poll for
+#     convergence; two replicate over HTTPS, two pin active SSE failure
+#     contracts, and one guards event/history observers). The SSE-S3 contract
+#     remains ignored under backlog#1291.
+#   * 11 `_real_dual_node` site-replication tests (each spawns TWO rustfs
+#     servers and drives the cross-process site-replication control plane).
+#   * 1 `_real_three_node` site-replication test.
+#   * 1 `_real_single_node` service-account round-trip test.
+#
+# The selection is the [profile.e2e-repl-nightly] default-filter in
+# .config/nextest.toml — the single wiring mechanism (repl-1 / ci-4). Do NOT
+# add ad-hoc cargo-test steps here; change the filterset instead.
+#
+# Explicit division of labor: these 27 tests run ONLY here, never double-run
+# in ci-5's future e2e-full merge gate. TODO(ci-7): once the ci domain's
+# consolidated scheduled e2e workflow exists, fold this interim repl-owned lane
+# into it rather than growing a second scheduled entrypoint.
+
+name: e2e-replication-nightly
+
+on:
+  workflow_dispatch:
+  schedule:
+    # 04:00 UTC nightly — staggered clear of fuzz/e2e-s3tests (02:00),
+    # stale (01:30) and performance-ab (06:00).
+    - cron: "0 4 * * *"
+
+# Only alert-on-failure needs more than read access; it declares its own
+# job-level `issues: write`.
+permissions:
+  contents: read
+
+jobs:
+  repl-nightly:
+    name: Replication e2e (nightly)
+    # dual-node tests spawn two full rustfs servers each; the extra headroom
+    # over the per-PR sm-standard-2 e2e job keeps the parallel process fan-out
+    # from starving the runner. Mirrors the ILM serial lane's runner class.
+    runs-on: sm-standard-4
+    timeout-minutes: 45
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-e2e-repl
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      # awscurl lets the STS dual-node test actually exercise its path. Without
+      # it the test skips gracefully with a visible log line
+      # (`awscurl_available()` in crates/e2e_test/src/common.rs), so the lane
+      # still passes — installing it just upgrades that one test from skip to
+      # real coverage.
+      - name: Set up Python
+        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+        with:
+          python-version: "3.12"
+
+      - name: Install awscurl
+        run: |
+          python3 -m pip install --user --upgrade pip awscurl
+          echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
+
+      - name: Verify awscurl
+        run: test -x "$AWSCURL_PATH"
+
+      # Build the rustfs binary once up front. The e2e tests spawn it as a
+      # child process (crates/e2e_test/src/common.rs) and will build it on
+      # demand otherwise, but a single explicit build avoids several parallel
+      # nextest test processes racing to build it at once.
+      - name: Build rustfs binary
+        run: cargo build -p rustfs --bins
+
+      - name: Run replication e2e nightly suite
+        run: cargo nextest run --profile e2e-repl-nightly -p e2e_test
+
+      - name: Upload nextest junit report
+        if: always()
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: e2e-replication-nightly-junit-${{ github.run_number }}
+          path: target/nextest/e2e-repl-nightly/junit.xml
+          retention-days: 7
+          if-no-files-found: ignore
+
+  alert-on-failure:
+    name: Alert on scheduled failure
+    needs: [repl-nightly]
+    # Only scheduled runs open/append the tracking issue (backlog#1149 ci-8);
+    # manual workflow_dispatch runs stay quiet so a debugging run never files a
+    # spurious alert.
+    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+      issues: write
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+      - name: Open or update failure-tracking issue
+        uses: ./.github/actions/schedule-failure-issue
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
@@ -12,64 +12,140 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.

+# Full ceph/s3-tests sweep against RustFS.
+#
+# The per-PR compatibility gate lives in ci.yml (job s3-implemented-tests) and
+# runs only the implemented_tests.txt whitelist. This workflow complements it:
+#
+#   - Scheduled weekly full sweep (TEST_SCOPE=all): runs the ENTIRE upstream
+#     suite and reports promotion candidates (tests that newly pass) and
+#     unclassified tests. The job fails only on regressions in the implemented
+#     whitelist or on infrastructure errors — expected failures from
+#     not-yet-implemented features do not turn the run red.
+#   - Manual runs (workflow_dispatch): same, with configurable mode/scope.
+#
+# All test execution is delegated to scripts/s3-tests/run.sh (single source of
+# truth, also used locally and by the PR gate). This workflow only provisions
+# the server topology: a single node, or a real 4-node distributed cluster
+# (erasure-coded across nodes) behind an HAProxy round-robin load balancer.
+#
+# Runner infrastructure (ci-1, backlog#1149): this sweep needs BOTH a working
+# Docker daemon (to build the source image and run the single/multi topologies)
+# AND a Python toolchain with pip (for awscurl/tox and the pytest harness).
+# It previously ran on the self-hosted `sm-standard-4` label, which is served
+# by heterogeneous runners: some scheduled runs landed on minimal pods that
+# had neither `pip` (bare python3, `No module named pip`) nor `docker.sock`,
+# so "Install Python tools" died before any test executed (all 15 recorded
+# runs failed). Two changes make the infra assumptions explicit instead of
+# trusting drifting runner state:
+#   1. Pin to GitHub-hosted `ubuntu-latest`, which reliably ships Docker,
+#      docker compose, python3 and pip (no self-hosted fleet drift).
+#   2. Set up Python explicitly (actions/setup-python) before any pip usage,
+#      so the workflow stays correct even if the runner image changes.
+# The PR gate (ci.yml s3-implemented-tests) is unaffected: it avoids Docker
+# via DEPLOY_MODE=binary and defers all pip setup to run.sh's self-bootstrap.
+
 name: e2e-s3tests

 on:
  workflow_dispatch:
    inputs:
      test-mode:
-        description: "Test mode to run"
+        description: "Server topology"
        required: true
        type: choice
        default: "single"
        options:
          - single
          - multi
+      scope:
+        description: "Test scope: full upstream suite or implemented whitelist"
+        required: true
+        type: choice
+        default: "all"
+        options:
+          - all
+          - implemented
      xdist:
-        description: "Enable pytest-xdist (parallel). '0' to disable."
+        description: "pytest-xdist workers. '0' to disable."
+        required: false
+        default: "4"
+      maxfail:
+        description: "Stop after N failures. '0' to run everything."
        required: false
        default: "0"
-      maxfail:
-        description: "Stop after N failures (debug friendly)"
-        required: false
-        default: "1"
      markexpr:
-        description: "pytest -m expression (feature filters)"
+        description: "Optional pytest -m expression"
        required: false
-        default: "not lifecycle and not versioning and not s3website and not bucket_logging and not encryption"
+        default: ""
+  schedule:
+    # Weekly full sweep (Sunday 02:00 UTC): full suite, run against BOTH the
+    # single-node and the 4-node distributed topologies (matrix below).
+    - cron: "0 2 * * 0"

 env:
  # main user
-  S3_ACCESS_KEY: rustfsadmin
-  S3_SECRET_KEY: rustfsadmin
+  S3_ACCESS_KEY: rustfsadmin-ci
+  S3_SECRET_KEY: rustfssecret-ci
  # alt user (must be different from main for many s3-tests)
  S3_ALT_ACCESS_KEY: rustfsalt
  S3_ALT_SECRET_KEY: rustfsalt

  S3_REGION: us-east-1
+  S3_HOST: 127.0.0.1
+  S3_PORT: "9000"

  RUST_LOG: info
  PLATFORM: linux/amd64
  BUILDX_CACHE_SCOPE: rustfs-e2e-s3tests-source

+  # Scheduled runs have no inputs; default to a full sweep.
+  TEST_SCOPE: ${{ github.event.inputs.scope || 'all' }}
+  XDIST: ${{ github.event.inputs.xdist || '4' }}
+  MAXFAIL: ${{ github.event.inputs.maxfail || '0' }}
+  MARKEXPR: ${{ github.event.inputs.markexpr || '' }}
+
 concurrency:
  group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event.inputs['test-mode'] || 'single' }}
  cancel-in-progress: true

+# Only alert-on-failure needs more than read access; it declares its own
+# job-level `issues: write`.
+permissions:
+  contents: read
+
 defaults:
  run:
    shell: bash

 jobs:
-  s3tests-single:
-    if: github.event.inputs['test-mode'] == 'single'
-    runs-on: ubicloud-standard-2
-    timeout-minutes: 120
+  s3tests:
+    # GitHub-hosted: reliably provides Docker + docker compose + python3/pip.
+    # See the header note (ci-1) for why the self-hosted sm-standard-4 label
+    # was abandoned. TODO(ci-8): scheduled-failure alerting (auto-open issue)
+    # is added by the ci-8 composite action; do not implement it here.
+    runs-on: ubuntu-latest
+    timeout-minutes: 180
+    strategy:
+      fail-fast: false
+      matrix:
+        # Scheduled sweeps cover both topologies; manual runs use the input.
+        test-mode: ${{ github.event_name == 'schedule' && fromJSON('["single", "multi"]') || fromJSON(format('["{0}"]', github.event.inputs.test-mode || 'single')) }}
+    env:
+      TEST_MODE: ${{ matrix.test-mode }}
    steps:
-      - uses: actions/checkout@v6
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      # Provision Python explicitly rather than trusting the runner image to
+      # ship a working pip (ci-1: a bare python3 without pip is what broke the
+      # scheduled sweep). Keeps the workflow correct across runner drift.
+      - name: Set up Python
+        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
+        with:
+          python-version: "3.12"

      - name: Cache pip downloads
-        uses: actions/cache@v4
+        uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
        with:
          path: ~/.cache/pip
          key: ${{ runner.os }}-pip-e2e-s3tests-${{ hashFiles('.github/workflows/e2e-s3tests.yml') }}
@@ -82,233 +158,81 @@ jobs:
          echo "$HOME/.local/bin" >> "$GITHUB_PATH"

      - name: Enable buildx
-        uses: docker/setup-buildx-action@v3
+        uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3

      - name: Build RustFS image (source, cached)
        run: |
          DOCKER_BUILDKIT=1 docker buildx build --load \
-            --platform ${PLATFORM} \
-            --cache-from type=gha,scope=${BUILDX_CACHE_SCOPE} \
-            --cache-to type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE} \
+            --platform "${PLATFORM}" \
+            --cache-from "type=gha,scope=${BUILDX_CACHE_SCOPE}" \
+            --cache-to "type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE}" \
            -t rustfs-ci \
            -f Dockerfile.source .

-      - name: Create network
-        run: docker network inspect rustfs-net >/dev/null 2>&1 || docker network create rustfs-net
-
-      - name: Remove existing rustfs-single (if any)
-        run: docker rm -f rustfs-single >/dev/null 2>&1 || true
-
      - name: Start single RustFS
+        if: env.TEST_MODE == 'single'
        run: |
+          docker network inspect rustfs-net >/dev/null 2>&1 || docker network create rustfs-net
+          docker rm -f rustfs-single >/dev/null 2>&1 || true
+          # The four disks share one physical device on the runner (a single
+          # loopback filesystem), so the local physical-disk-independence guard
+          # would refuse to start. Bypass it — this is the CI use case the guard
+          # explicitly sanctions via RUSTFS_UNSAFE_BYPASS_DISK_CHECK.
          docker run -d --name rustfs-single \
            --network rustfs-net \
-            -p 9000:9000 \
+            -p "${S3_PORT}:9000" \
            -e RUSTFS_ADDRESS=0.0.0.0:9000 \
-            -e RUSTFS_ACCESS_KEY=$S3_ACCESS_KEY \
-            -e RUSTFS_SECRET_KEY=$S3_SECRET_KEY \
-            -e RUSTFS_VOLUMES="/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3" \
+            -e RUSTFS_ACCESS_KEY="${S3_ACCESS_KEY}" \
+            -e RUSTFS_SECRET_KEY="${S3_SECRET_KEY}" \
+            -e RUSTFS_VOLUMES="/data/rustfs{0...3}" \
+            -e RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true \
            -v /tmp/rustfs-single:/data \
            rustfs-ci

-      - name: Wait for RustFS ready
+      - name: Start 4-node distributed cluster
+        if: env.TEST_MODE == 'multi'
        run: |
-          for i in {1..60}; do
-            if curl -sf http://127.0.0.1:9000/health >/dev/null 2>&1; then
-              echo "RustFS is ready"
-              exit 0
-            fi
+          # A real distributed deployment: every node lists all endpoints in
+          # RUSTFS_VOLUMES so data is erasure-coded ACROSS nodes. Do not use
+          # node-local volume paths here — that would create four independent
+          # single-node stores behind the load balancer.
+          cat > compose.yml <<EOF
+          x-rustfs-node: &rustfs-node
+            image: rustfs-ci
+            networks: [rustfs-net]
+            environment:
+              RUSTFS_ADDRESS: "0.0.0.0:9000"
+              RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
+              RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
+              RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
+              # Each node's four disks share one physical device inside its
+              # container, so bypass the local physical-disk-independence guard
+              # (the CI use case it explicitly sanctions).
+              RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"

-            if [ "$(docker inspect -f '{{.State.Running}}' rustfs-single 2>/dev/null)" != "true" ]; then
-              echo "RustFS container not running" >&2
-              docker logs rustfs-single || true
-              exit 1
-            fi
-
-            sleep 2
-          done
-
-          echo "Health check timed out" >&2
-          docker logs rustfs-single || true
-          exit 1
-
-      - name: Generate s3tests config
-        run: |
-          export S3_HOST=127.0.0.1
-          envsubst < .github/s3tests/s3tests.conf > s3tests.conf
-
-      - name: Provision s3-tests alt user (required by suite)
-        run: |
-          # Admin API requires AWS SigV4 signing. awscurl is used by RustFS codebase as well.
-          awscurl \
-            --service s3 \
-            --region "${S3_REGION}" \
-            --access_key "${S3_ACCESS_KEY}" \
-            --secret_key "${S3_SECRET_KEY}" \
-            -X PUT \
-            -H 'Content-Type: application/json' \
-            -d '{"secretKey":"'"${S3_ALT_SECRET_KEY}"'","status":"enabled","policy":"readwrite"}' \
-            "http://127.0.0.1:9000/rustfs/admin/v3/add-user?accessKey=${S3_ALT_ACCESS_KEY}"
-
-          # Explicitly attach built-in policy via policy mapping.
-          # s3-tests relies on alt client being able to ListBuckets during setup cleanup.
-          awscurl \
-            --service s3 \
-            --region "${S3_REGION}" \
-            --access_key "${S3_ACCESS_KEY}" \
-            --secret_key "${S3_SECRET_KEY}" \
-            -X PUT \
-            "http://127.0.0.1:9000/rustfs/admin/v3/set-user-or-group-policy?policyName=readwrite&userOrGroup=${S3_ALT_ACCESS_KEY}&isGroup=false"
-
-          # Sanity check: alt user can list buckets (should not be AccessDenied).
-          awscurl \
-            --service s3 \
-            --region "${S3_REGION}" \
-            --access_key "${S3_ALT_ACCESS_KEY}" \
-            --secret_key "${S3_ALT_SECRET_KEY}" \
-            -X GET \
-            "http://127.0.0.1:9000/" >/dev/null
-
-      - name: Prepare s3-tests
-        run: |
-          git clone --depth 1 https://github.com/ceph/s3-tests.git s3-tests
-
-      - name: Run ceph s3-tests (debug friendly)
-        run: |
-          export PATH="$HOME/.local/bin:$PATH"
-          mkdir -p artifacts/s3tests-single
-
-          cd s3-tests
-
-          set -o pipefail
-
-          MAXFAIL="${{ github.event.inputs.maxfail }}"
-          if [ -z "$MAXFAIL" ]; then MAXFAIL="1"; fi
-
-          MARKEXPR="${{ github.event.inputs.markexpr }}"
-          if [ -z "$MARKEXPR" ]; then MARKEXPR="not lifecycle and not versioning and not s3website and not bucket_logging and not encryption"; fi
-
-          XDIST="${{ github.event.inputs.xdist }}"
-          if [ -z "$XDIST" ]; then XDIST="0"; fi
-          XDIST_ARGS=""
-          if [ "$XDIST" != "0" ]; then
-            # Add pytest-xdist to requirements.txt so tox installs it inside
-            # its virtualenv. Installing outside tox does NOT work.
-            echo "pytest-xdist" >> requirements.txt
-            XDIST_ARGS="-n $XDIST --dist=loadgroup"
-          fi
-
-          # Run tests from s3tests/functional (boto2+boto3 combined directory).
-          S3TEST_CONF=${GITHUB_WORKSPACE}/s3tests.conf \
-            tox -- \
-              -vv -ra --showlocals --tb=long \
-              --maxfail="$MAXFAIL" \
-              --junitxml=${GITHUB_WORKSPACE}/artifacts/s3tests-single/junit.xml \
-              $XDIST_ARGS \
-              s3tests/functional/test_s3.py \
-              -m "$MARKEXPR" \
-              2>&1 | tee ${GITHUB_WORKSPACE}/artifacts/s3tests-single/pytest.log
-
-      - name: Collect RustFS logs
-        if: always()
-        run: |
-          mkdir -p artifacts/rustfs-single
-          docker logs rustfs-single > artifacts/rustfs-single/rustfs.log 2>&1 || true
-          docker inspect rustfs-single > artifacts/rustfs-single/inspect.json || true
-
-      - name: Upload artifacts
-        if: always() && env.ACT != 'true'
-        uses: actions/upload-artifact@v6
-        with:
-          name: s3tests-single
-          path: artifacts/**
-
-  s3tests-multi:
-    if: github.event_name == 'workflow_dispatch' && github.event.inputs['test-mode'] == 'multi'
-    runs-on: ubicloud-standard-2
-    timeout-minutes: 150
-    steps:
-      - uses: actions/checkout@v6
-
-      - name: Cache pip downloads
-        uses: actions/cache@v4
-        with:
-          path: ~/.cache/pip
-          key: ${{ runner.os }}-pip-e2e-s3tests-${{ hashFiles('.github/workflows/e2e-s3tests.yml') }}
-          restore-keys: |
-            ${{ runner.os }}-pip-e2e-s3tests-
-
-      - name: Install Python tools
-        run: |
-          python3 -m pip install --user --upgrade pip awscurl tox
-          echo "$HOME/.local/bin" >> "$GITHUB_PATH"
-
-      - name: Enable buildx
-        uses: docker/setup-buildx-action@v3
-
-      - name: Build RustFS image (source, cached)
-        run: |
-          DOCKER_BUILDKIT=1 docker buildx build --load \
-            --platform ${PLATFORM} \
-            --cache-from type=gha,scope=${BUILDX_CACHE_SCOPE} \
-            --cache-to type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE} \
-            -t rustfs-ci \
-            -f Dockerfile.source .
-
-      - name: Prepare cluster compose
-        run: |
-          cat > compose.yml <<'EOF'
          services:
            rustfs1:
-              image: rustfs-ci
+              <<: *rustfs-node
              hostname: rustfs1
-              networks: [rustfs-net]
-              environment:
-                RUSTFS_ADDRESS: "0.0.0.0:9000"
-                RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
-                RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
-                RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
-              volumes:
-                - rustfs1-data:/data
+              volumes: [rustfs1-data:/data]
            rustfs2:
-              image: rustfs-ci
+              <<: *rustfs-node
              hostname: rustfs2
-              networks: [rustfs-net]
-              environment:
-                RUSTFS_ADDRESS: "0.0.0.0:9000"
-                RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
-                RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
-                RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
-              volumes:
-                - rustfs2-data:/data
+              volumes: [rustfs2-data:/data]
            rustfs3:
-              image: rustfs-ci
+              <<: *rustfs-node
              hostname: rustfs3
-              networks: [rustfs-net]
-              environment:
-                RUSTFS_ADDRESS: "0.0.0.0:9000"
-                RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
-                RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
-                RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
-              volumes:
-                - rustfs3-data:/data
+              volumes: [rustfs3-data:/data]
            rustfs4:
-              image: rustfs-ci
+              <<: *rustfs-node
              hostname: rustfs4
-              networks: [rustfs-net]
-              environment:
-                RUSTFS_ADDRESS: "0.0.0.0:9000"
-                RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
-                RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
-                RUSTFS_VOLUMES: "/data/rustfs0 /data/rustfs1 /data/rustfs2 /data/rustfs3"
-              volumes:
-                - rustfs4-data:/data
+              volumes: [rustfs4-data:/data]
            lb:
              image: haproxy:2.9
              hostname: lb
              networks: [rustfs-net]
              ports:
-                - "9000:9000"
+                - "${S3_PORT}:9000"
              volumes:
                - ./haproxy.cfg:/usr/local/etc/haproxy/haproxy.cfg:ro
          networks:
@@ -340,104 +264,97 @@ jobs:
            server s4 rustfs4:9000 check
          EOF

-      - name: Launch cluster
-        run: docker compose -f compose.yml up -d
+          docker compose -f compose.yml up -d

-      - name: Wait for LB ready
+      - name: Wait for RustFS ready
        run: |
-          for i in {1..90}; do
-            if curl -sf http://127.0.0.1:9000/health >/dev/null 2>&1; then
-              echo "Load balancer is ready"
+          for _ in {1..120}; do
+            if curl -sf "http://${S3_HOST}:${S3_PORT}/health" >/dev/null 2>&1; then
+              echo "RustFS is ready"
              exit 0
            fi
            sleep 2
          done
-          echo "LB or backend not ready" >&2
-          docker compose -f compose.yml logs --tail=200 || true
+          echo "Health check timed out" >&2
+          if [ "${TEST_MODE}" = "single" ]; then
+            docker logs rustfs-single || true
+          else
+            docker compose -f compose.yml logs --tail=200 || true
+          fi
          exit 1

-      - name: Generate s3tests config
+      - name: Run ceph s3-tests
        run: |
-          export S3_HOST=127.0.0.1
-          envsubst < .github/s3tests/s3tests.conf > s3tests.conf
+          set +e
+          DEPLOY_MODE=existing \
+          TEST_MODE="${TEST_MODE}" \
+          TEST_SCOPE="${TEST_SCOPE}" \
+          XDIST="${XDIST}" \
+          MAXFAIL="${MAXFAIL}" \
+          MARKEXPR="${MARKEXPR}" \
+          ./scripts/s3-tests/run.sh
+          RC=$?
+          set -e

-      - name: Provision s3-tests alt user (required by suite)
-        run: |
-          awscurl \
-            --service s3 \
-            --region "${S3_REGION}" \
-            --access_key "${S3_ACCESS_KEY}" \
-            --secret_key "${S3_SECRET_KEY}" \
-            -X PUT \
-            -H 'Content-Type: application/json' \
-            -d '{"secretKey":"'"${S3_ALT_SECRET_KEY}"'","status":"enabled","policy":"readwrite"}' \
-            "http://127.0.0.1:9000/rustfs/admin/v3/add-user?accessKey=${S3_ALT_ACCESS_KEY}"
-
-          awscurl \
-            --service s3 \
-            --region "${S3_REGION}" \
-            --access_key "${S3_ACCESS_KEY}" \
-            --secret_key "${S3_SECRET_KEY}" \
-            -X PUT \
-            "http://127.0.0.1:9000/rustfs/admin/v3/set-user-or-group-policy?policyName=readwrite&userOrGroup=${S3_ALT_ACCESS_KEY}&isGroup=false"
-
-          awscurl \
-            --service s3 \
-            --region "${S3_REGION}" \
-            --access_key "${S3_ALT_ACCESS_KEY}" \
-            --secret_key "${S3_ALT_SECRET_KEY}" \
-            -X GET \
-            "http://127.0.0.1:9000/" >/dev/null
-
-      - name: Prepare s3-tests
-        run: |
-          git clone --depth 1 https://github.com/ceph/s3-tests.git s3-tests
-
-      - name: Run ceph s3-tests (multi, debug friendly)
-        run: |
-          export PATH="$HOME/.local/bin:$PATH"
-          mkdir -p artifacts/s3tests-multi
-
-          cd s3-tests
-
-          set -o pipefail
-
-          MAXFAIL="${{ github.event.inputs.maxfail }}"
-          if [ -z "$MAXFAIL" ]; then MAXFAIL="1"; fi
-
-          MARKEXPR="${{ github.event.inputs.markexpr }}"
-          if [ -z "$MARKEXPR" ]; then MARKEXPR="not lifecycle and not versioning and not s3website and not bucket_logging and not encryption"; fi
-
-          XDIST="${{ github.event.inputs.xdist }}"
-          if [ -z "$XDIST" ]; then XDIST="0"; fi
-          XDIST_ARGS=""
-          if [ "$XDIST" != "0" ]; then
-            # Add pytest-xdist to requirements.txt so tox installs it inside
-            # its virtualenv. Installing outside tox does NOT work.
-            echo "pytest-xdist" >> requirements.txt
-            XDIST_ARGS="-n $XDIST --dist=loadgroup"
+          if [ "${TEST_SCOPE}" = "implemented" ]; then
+            # Whitelist run: every failure is a regression.
+            exit "${RC}"
          fi

-          # Run tests from s3tests/functional (boto2+boto3 combined directory).
-          S3TEST_CONF=${GITHUB_WORKSPACE}/s3tests.conf \
-            tox -- \
-              -vv -ra --showlocals --tb=long \
-              --maxfail="$MAXFAIL" \
-              --junitxml=${GITHUB_WORKSPACE}/artifacts/s3tests-multi/junit.xml \
-              $XDIST_ARGS \
-              s3tests/functional/test_s3.py \
-              -m "$MARKEXPR" \
-              2>&1 | tee ${GITHUB_WORKSPACE}/artifacts/s3tests-multi/pytest.log
+          # Full sweep: failures outside the implemented whitelist are
+          # inventory (promotion candidates / unimplemented features), not a
+          # gate. Fail only on whitelist regressions or infrastructure errors.
+          JUNIT="artifacts/s3tests-${TEST_MODE}/junit.xml"
+          if [ ! -f "${JUNIT}" ]; then
+            echo "No junit.xml produced — infrastructure failure (exit ${RC})" >&2
+            exit "${RC}"
+          fi
+          python3 scripts/s3-tests/report_compat.py \
+            --junit "${JUNIT}" \
+            --lists-dir scripts/s3-tests \
+            --fail-on-regression

-      - name: Collect logs
+      - name: Publish compatibility report
        if: always()
        run: |
-          mkdir -p artifacts/cluster
-          docker compose -f compose.yml logs --no-color > artifacts/cluster/cluster.log 2>&1 || true
+          REPORT="artifacts/s3tests-${TEST_MODE}/compat-report.md"
+          if [ -f "${REPORT}" ]; then
+            cat "${REPORT}" >> "$GITHUB_STEP_SUMMARY"
+          fi
+
+      - name: Collect RustFS logs
+        if: always()
+        run: |
+          mkdir -p "artifacts/rustfs-${TEST_MODE}"
+          if [ "${TEST_MODE}" = "single" ]; then
+            docker logs rustfs-single > "artifacts/rustfs-${TEST_MODE}/rustfs.log" 2>&1 || true
+            docker inspect rustfs-single > "artifacts/rustfs-${TEST_MODE}/inspect.json" || true
+          else
+            docker compose -f compose.yml logs --no-color > "artifacts/rustfs-${TEST_MODE}/cluster.log" 2>&1 || true
+          fi

      - name: Upload artifacts
        if: always() && env.ACT != 'true'
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
        with:
-          name: s3tests-multi
+          name: s3tests-${{ env.TEST_MODE }}
          path: artifacts/**
+
+  alert-on-failure:
+    name: Alert on scheduled failure
+    needs: [s3tests]
+    # `always()` is required: without it this job is skipped when a needed
+    # job fails. Alerts only for scheduled runs (backlog#1149 ci-8) — manual
+    # dispatch failures are already watched by a human.
+    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+      issues: write
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+      - name: Open or update failure-tracking issue
+        uses: ./.github/actions/schedule-failure-issue
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
@@ -0,0 +1,253 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Fuzz
+
+on:
+  pull_request:
+    types: [ opened, synchronize, reopened, closed ]
+    # PR trigger is intentionally narrow: only changes to the fuzz harness
+    # itself gate a PR. Broad crate paths (ecstore/filemeta/utils/policy/…)
+    # are covered by the nightly `schedule` run below, which fuzzes against
+    # whatever landed on main. Widening these paths previously queued a
+    # ~45min fuzz-build on nearly every PR and is why this workflow was
+    # disabled; do not re-add crate paths here.
+    paths:
+      - "fuzz/**"
+      - "scripts/fuzz/**"
+      - ".github/workflows/fuzz.yml"
+  schedule:
+    - cron: "0 2 * * *"
+  workflow_dispatch:
+    inputs:
+      profile:
+        description: "Which fuzz profile to run"
+        required: false
+        default: "both"
+        type: choice
+        options:
+          - smoke
+          - nightly
+          - both
+
+permissions:
+  contents: read
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
+
+env:
+  CARGO_TERM_COLOR: always
+  CARGO_BUILD_TARGET: x86_64-unknown-linux-gnu
+  # Fuzz targets build without the dial9 feature, so tokio_unstable is not needed.
+  RUSTFLAGS: "-C target-feature=-crt-static"
+
+jobs:
+  cancel-closed-pr-runs:
+    name: Cancel Closed PR Runs
+    if: github.event_name == 'pull_request' && github.event.action == 'closed'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Explain cancellation run
+        run: echo "PR closed; this run only cancels older runs in the same concurrency group."
+
+  # ──────────────────────────────────────────────────────────────
+  # Phase 1: Build all fuzz harness binaries once.
+  # ──────────────────────────────────────────────────────────────
+  fuzz-build:
+    name: Build Fuzz Harness
+    if: >
+      (github.event_name == 'pull_request' && github.event.action != 'closed') ||
+      github.event_name == 'schedule' ||
+      github.event_name == 'workflow_dispatch'
+    runs-on: sm-standard-4
+    timeout-minutes: 45
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: nightly
+          cache-shared-key: fuzz-${{ hashFiles('fuzz/Cargo.lock') }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' || github.event_name == 'schedule' }}
+
+      - name: Install cargo-fuzz
+        uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
+        with:
+          tool: cargo-fuzz
+
+      - name: Build all fuzz targets
+        run: BUILD_ONLY=1 ./scripts/fuzz/run.sh
+
+      - name: Stage prebuilt fuzz binaries
+        run: |
+          binary_root="fuzz/prebuilt/${CARGO_BUILD_TARGET}/release"
+          mkdir -p "${binary_root}"
+          # Keep this list in sync with the smoke/nightly matrices and
+          # scripts/fuzz/run.sh default target set (all 5 buildable bins in
+          # fuzz/Cargo.toml). The *_storage_api.rs files are `mod` submodules
+          # of their parent targets, not standalone bins.
+          for target in archive_extract bucket_validation local_metadata path_containment policy_ingress; do
+            install -Dm755 "fuzz/target/${CARGO_BUILD_TARGET}/release/${target}" "${binary_root}/${target}"
+          done
+
+      - name: Upload prebuilt fuzz binaries
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: fuzz-prebuilt-binaries-${{ github.run_number }}
+          path: |
+            fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/archive_extract
+            fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/bucket_validation
+            fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/local_metadata
+            fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/path_containment
+            fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/policy_ingress
+          if-no-files-found: error
+          retention-days: 1
+          compression-level: 0
+
+  # ──────────────────────────────────────────────────────────────
+  # Phase 2 (PR): Run smoke targets in parallel via matrix.
+  # ──────────────────────────────────────────────────────────────
+  pr-fuzz-smoke:
+    name: "Smoke / ${{ matrix.target }}"
+    needs: fuzz-build
+    if: >
+      (github.event_name == 'pull_request' && github.event.action != 'closed') ||
+      (github.event_name == 'workflow_dispatch' &&
+        (github.event.inputs.profile == 'smoke' || github.event.inputs.profile == 'both'))
+    runs-on: sm-standard-4
+    timeout-minutes: 30
+    strategy:
+      fail-fast: false
+      # One job per target runs in parallel, so wall-clock time is per-target
+      # (~60s fuzz + download/chmod overhead), not the sum across the matrix.
+      matrix:
+        target: [archive_extract, bucket_validation, local_metadata, path_containment, policy_ingress]
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Download prebuilt fuzz binaries
+        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
+        with:
+          name: fuzz-prebuilt-binaries-${{ github.run_number }}
+          path: fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release
+
+      - name: Restore executable bit
+        run: chmod +x "fuzz/prebuilt/${CARGO_BUILD_TARGET}/release/${{ matrix.target }}"
+
+      - name: Run ${{ matrix.target }}
+        env:
+          FUZZ_TARGET: ${{ matrix.target }}
+          MAX_TOTAL_TIME: 60
+          SKIP_BUILD: 1
+          USE_PREBUILT_BINARY: 1
+          PREBUILT_BINARY_DIR: ${{ github.workspace }}/fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release
+        run: ./scripts/fuzz/run.sh
+
+      - name: Upload fuzz smoke artifacts
+        if: always()
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: fuzz-smoke-${{ matrix.target }}-${{ github.run_number }}
+          path: |
+            fuzz/artifacts/**
+            fuzz/corpus/${{ matrix.target }}/**
+          if-no-files-found: ignore
+          retention-days: 7
+
+  # ──────────────────────────────────────────────────────────────
+  # Phase 2 (Nightly): Run all targets in parallel via matrix.
+  # ──────────────────────────────────────────────────────────────
+  nightly-fuzz-corpus:
+    name: "Nightly / ${{ matrix.target }}"
+    needs: fuzz-build
+    # TODO(ci-8): when the schedule-failure-issue composite action lands,
+    # add a step here (or a dependent job) that opens/updates a GitHub issue
+    # on nightly failure. ci-8 is the single alerting mechanism for all
+    # scheduled workflows; do not self-roll alerting in this workflow.
+    if: >
+      github.event_name == 'schedule' ||
+      (github.event_name == 'workflow_dispatch' &&
+        (github.event.inputs.profile == 'nightly' || github.event.inputs.profile == 'both'))
+    runs-on: sm-standard-4
+    timeout-minutes: 60
+    strategy:
+      fail-fast: false
+      matrix:
+        target: [archive_extract, bucket_validation, local_metadata, path_containment, policy_ingress]
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Download prebuilt fuzz binaries
+        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
+        with:
+          name: fuzz-prebuilt-binaries-${{ github.run_number }}
+          path: fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release
+
+      - name: Restore executable bit
+        run: chmod +x "fuzz/prebuilt/${CARGO_BUILD_TARGET}/release/${{ matrix.target }}"
+
+      - name: Run ${{ matrix.target }}
+        env:
+          FUZZ_TARGET: ${{ matrix.target }}
+          MAX_TOTAL_TIME: 300
+          SKIP_BUILD: 1
+          USE_PREBUILT_BINARY: 1
+          PREBUILT_BINARY_DIR: ${{ github.workspace }}/fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release
+        run: ./scripts/fuzz/run.sh
+
+      - name: Upload nightly fuzz artifacts
+        if: always()
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: fuzz-nightly-${{ matrix.target }}-${{ github.run_number }}
+          path: |
+            fuzz/artifacts/**
+            fuzz/corpus/${{ matrix.target }}/**
+          if-no-files-found: ignore
+          retention-days: 30
+
+  # ──────────────────────────────────────────────────────────────
+  # Nightly alerting: open/update a tracking issue on failure.
+  # ──────────────────────────────────────────────────────────────
+  alert-on-failure:
+    name: Alert on scheduled failure
+    needs: [fuzz-build, nightly-fuzz-corpus]
+    # `always()` is required: without it this job is skipped when a needed
+    # job fails. Alerts only for scheduled (nightly) runs (backlog#1149
+    # ci-8); PR and manual dispatch failures are already watched by a human.
+    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+      issues: write
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+      - name: Open or update failure-tracking issue
+        uses: ./.github/actions/schedule-failure-issue
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
@@ -18,77 +18,115 @@ on:
  workflow_run:
    workflows: [ "Build and Release" ]
    types: [ completed ]
+  workflow_dispatch:
+    inputs:
+      version:
+        description: "Release version to publish, e.g. 1.0.0-beta.1 or v1.0.0-beta.1"
+        required: true
+        default: "1.0.0-beta.1"
+        type: string

 permissions:
  contents: read

-env:
-  new_version: ${{ github.event.workflow_run.head_branch }}
-
 jobs:
  build-helm-package:
-    runs-on: ubicloud-standard-2
-    # Only run on successful builds triggered by tag pushes (version format: x.y.z or x.y.z-suffix)
+    runs-on: ubuntu-latest
    if: |
-      github.event.workflow_run.conclusion == 'success' && 
-      github.event.workflow_run.event == 'push' &&
-      contains(github.event.workflow_run.head_branch, '.')
+      github.event_name == 'workflow_dispatch' ||
+      (
+        github.event.workflow_run.conclusion == 'success' &&
+        github.event.workflow_run.event == 'push' &&
+        contains(github.event.workflow_run.head_branch, '.')
+      )
+
+    outputs:
+      raw_tag: ${{ steps.version.outputs.raw_tag }}
+      app_version: ${{ steps.version.outputs.app_version }}
+      chart_version: ${{ steps.version.outputs.chart_version }}

    steps:
      - name: Checkout helm chart repo
-        uses: actions/checkout@v6
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7

-      - name: Replace chart app version
+      - name: Normalize release version
+        id: version
        run: |
-          set -e
-          set -x
-          old_version=$(grep "^appVersion:" helm/rustfs/Chart.yaml | awk '{print $2}')
-          sed -i "s/$old_version/$new_version/g" helm/rustfs/Chart.yaml
+          set -eux
+
+          if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
+            RAW="${{ github.event.inputs.version }}"
+          else
+            RAW="${{ github.event.workflow_run.head_branch }}"
+          fi
+
+          case "$RAW" in
+            refs/tags/*)
+              RAW_TAG="${RAW#refs/tags/}"
+              ;;
+            *)
+              RAW_TAG="$RAW"
+              ;;
+          esac
+
+          ./scripts/helm_chart_version.sh "$RAW_TAG"
+
+      - name: Replace chart version and app version
+        run: |
+          set -eux
+          sed -i -E 's/^version:.*/version: "${{ steps.version.outputs.chart_version }}"/' helm/rustfs/Chart.yaml
+          sed -i -E 's/^appVersion:.*/appVersion: "${{ steps.version.outputs.app_version }}"/' helm/rustfs/Chart.yaml

      - name: Set up Helm
-        uses: azure/setup-helm@v4.3.0
+        uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
+
+      - name: Test Helm Chart Templates
+        run: ./scripts/test_helm_templates.sh

      - name: Package Helm Chart
        run: |
+          set -eux
          cp helm/README.md helm/rustfs/
-          package_version=$(echo $new_version | awk -F '-' '{print $2}' | awk -F '.' '{print $NF}') 
-          helm package ./helm/rustfs --destination helm/rustfs/ --version "0.0.$package_version"
+          helm package ./helm/rustfs \
+            --destination helm/rustfs/ \
+            --version "${{ steps.version.outputs.chart_version }}"

      - name: Upload helm package as artifact
-        uses: actions/upload-artifact@v6
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
        with:
          name: helm-package
          path: helm/rustfs/*.tgz
          retention-days: 1

  publish-helm-package:
-    runs-on: ubicloud-standard-2
+    runs-on: ubuntu-latest
    needs: [ build-helm-package ]
+    if: needs.build-helm-package.result == 'success'

    steps:
      - name: Checkout helm package repo
-        uses: actions/checkout@v6
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
        with:
          repository: rustfs/helm
          token: ${{ secrets.RUSTFS_HELM_PACKAGE }}

      - name: Download helm package
-        uses: actions/download-artifact@v7
+        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
        with:
          name: helm-package
          path: ./

      - name: Set up helm
-        uses: azure/setup-helm@v4.3.0
+        uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0

      - name: Generate index
        run: helm repo index . --url https://charts.rustfs.com

      - name: Push helm package and index file
        run: |
+          set -eux
          git config --global user.name "${{ secrets.USERNAME }}"
          git config --global user.email "${{ secrets.EMAIL_ADDRESS }}"
-          git status .
          git add .
-          git commit -m "Update rustfs helm package with $new_version."
+          git commit -m "Update rustfs helm package with ${{ needs.build-helm-package.outputs.app_version }}." || echo "No changes to commit"
          git push origin main
@@ -25,12 +25,12 @@ permissions:

 jobs:
  build:
-    runs-on: ubicloud-standard-4
+    runs-on: ubuntu-latest
    steps:
-      - uses: usthe/issues-translate-action@v2.7
+      - uses: usthe/issues-translate-action@b41f55ddc81d7d54bd542a4f289fe28ec081898e # v2.7
        with:
          IS_MODIFY_TITLE: false
-          # not require, default false, . Decide whether to modify the issue title
+          # not require, default false. Decide whether to modify the issue title
          # if true, the robot account @Issues-translate-bot must have modification permissions, invite @Issues-translate-bot to your project or use your custom bot.
          CUSTOM_BOT_NOTE: Bot detected the issue body's language is not English, translate it automatically.
          # not require. Customize the translation robot prefix message.
@@ -0,0 +1,92 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# MinIO on-disk interop: prove RustFS reads MinIO-written erasure-coded SSE
+# objects with byte-identical data and correct logical size.
+#
+# The fixtures are real MinIO backend trees generated on the fly (they are
+# gitignored, never committed), so the job regenerates them each run with
+# Docker and then runs the `#[ignore]` reader tests in
+# crates/ecstore/tests/minio_generated_read_test.rs. SSE/RIO changes run this
+# as a path-filtered PR gate; nightly and manual runs retain broader coverage.
+#
+# Runner: GitHub-hosted `ubuntu-latest`. It reliably ships Docker + Python,
+# unlike the self-hosted fleet, whose pods drift in Docker/pip availability
+# (see the infra note in e2e-s3tests.yml).
+name: minio-interop
+
+on:
+  workflow_dispatch:
+  pull_request:
+    paths:
+      - ".github/workflows/minio-interop.yml"
+      - "Cargo.lock"
+      - "Cargo.toml"
+      - "crates/ecstore/Cargo.toml"
+      - "crates/ecstore/src/io_support/rio.rs"
+      - "crates/ecstore/src/client/api_put_object_streaming.rs"
+      - "crates/ecstore/src/object_api/readers.rs"
+      - "crates/ecstore/src/set_disk/ops/object.rs"
+      - "crates/ecstore/src/sse/**"
+      - "crates/ecstore/src/store/object.rs"
+      - "crates/ecstore/tests/minio_generated_read_test.rs"
+      - "crates/kms/**"
+      - "crates/rio-v2/**"
+      - "rustfs/src/app/object_usecase.rs"
+      - "rustfs/Cargo.toml"
+      - "rustfs/src/storage/sse.rs"
+  schedule:
+    # Nightly at 03:17 UTC (offset from other nightly jobs).
+    - cron: "17 3 * * *"
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+permissions:
+  contents: read
+
+jobs:
+  minio-interop:
+    name: MinIO interop (EC + SSE read parity)
+    # Skip on forks because this job depends on the repository's CI setup.
+    if: github.repository == 'rustfs/rustfs'
+    runs-on: ubuntu-latest
+    timeout-minutes: 40
+    env:
+      # Fixed 32-byte test KMS key baked into the fixture lab; not a secret.
+      RUSTFS_MINIO_STATIC_KMS_KEY: minio-default-key:IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g=
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: ci-minio-interop
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      - name: Generate real MinIO fixtures via Docker
+        run: bash crates/rio-v2/tests/minio_fixture_lab/capture_via_docker.sh all
+
+      - name: Generate real RustFS beta.5 KMS fixture
+        run: uv run python crates/rio-v2/tests/minio_fixture_lab/capture_rustfs_beta5.py
+
+      - name: Run MinIO interop reader tests
+        run: |
+          cargo nextest run --run-ignored ignored-only \
+            -p rustfs-ecstore --features rio-v2 \
+            -E 'binary(minio_generated_read_test)'
@@ -0,0 +1,269 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# MinIO Mint against RustFS: multi-SDK client compatibility.
+#
+# ceph/s3-tests (see e2e-s3tests.yml and the ci.yml PR gate) exercises the S3
+# API semantics through boto3 only. Mint complements it by running the
+# functional test suites of many REAL client SDKs and tools (awscli, mc,
+# aws-sdk-java/go/php/ruby, minio-go/java/js/py, s3cmd, ...), which catches
+# client-specific signing, encoding, and streaming edge cases a single client
+# cannot.
+#
+# This job is REPORT-ONLY for test failures: RustFS does not pass every mint
+# suite yet, so failures are inventoried in the job summary instead of turning
+# the run red. The job fails only when mint produces no results at all
+# (infrastructure error). Once a suite passes reliably, tightening this into a
+# baseline gate (like implemented_tests.txt for s3-tests) is the natural next
+# step. Tightening this into a per-suite baseline gate is tracked as ci-3
+# (backlog#1149) and is intentionally NOT done here.
+#
+# Runner infrastructure (ci-2, backlog#1149): this job needs a working Docker
+# daemon to (a) build the RustFS source image with buildx and (b) run both the
+# RustFS server and the mint image. It previously ran on the self-hosted
+# `sm-standard-4` label, which is served by heterogeneous runners; its single
+# recorded run (28730530597, 2026-07-05) landed on a minimal pod with no
+# `/var/run/docker.sock`, so "Enable buildx" died at `docker buildx create`
+# ("failed to connect to the docker API at unix:///var/run/docker.sock") before
+# any test executed -- the same root cause ci-1 fixed for the weekly s3-tests
+# sweep. The fix is to pin to GitHub-hosted `ubuntu-latest`, which reliably
+# ships a running Docker daemon + buildx + python3, instead of trusting the
+# drifting self-hosted fleet. Tradeoff: the source Rust build is heavier on a
+# GitHub-hosted 4-vCPU runner than on a warm self-hosted host, but it stays
+# well inside the 120-minute budget and buys deterministic infra. The
+# docker-capable self-hosted `dind-sm-standard-2` label was the alternative but
+# has fewer cores and reintroduces fleet-state risk for no reliability gain.
+
+name: mint
+
+on:
+  workflow_dispatch:
+    inputs:
+      suites:
+        description: "Space-separated mint suites to run (empty = all, e.g. 'awscli mc minio-go')"
+        required: false
+        default: ""
+      mode:
+        description: "Mint mode: core (quick) or full"
+        required: true
+        type: choice
+        default: "core"
+        options:
+          - core
+          - full
+      mint-image:
+        description: "Mint image reference"
+        required: false
+        default: "minio/mint:edge"
+  schedule:
+    # Weekly, after the Sunday s3-tests full sweep (starts 02:00 UTC, up to
+    # 3h) has finished, so the two never contend for the same runner pool.
+    - cron: "0 6 * * 0"
+
+env:
+  S3_ACCESS_KEY: rustfsadmin-ci
+  S3_SECRET_KEY: rustfssecret-ci
+  S3_REGION: us-east-1
+
+  RUST_LOG: info
+  PLATFORM: linux/amd64
+  BUILDX_CACHE_SCOPE: rustfs-e2e-s3tests-source
+
+  MINT_SUITES: ${{ github.event.inputs.suites || '' }}
+  MINT_MODE: ${{ github.event.inputs.mode || 'core' }}
+  # minio/mint:edge is a rolling tag, so an unpinned run is not reproducible and
+  # can go red purely from an upstream mint change (ci-2, backlog#1149). Pin the
+  # default to the linux/amd64 digest resolved on 2026-07-10. workflow_dispatch
+  # can still override via the `mint-image` input (e.g. to test a newer edge).
+  # Refresh: docker manifest inspect minio/mint:edge --verbose | grep digest
+  #   (or, without a docker daemon:)
+  #   TOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:minio/mint:pull" | jq -r .token)
+  #   curl -sI -H "Authorization: Bearer $TOKEN" \
+  #     -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
+  #     https://registry-1.docker.io/v2/minio/mint/manifests/edge | grep -i docker-content-digest
+  MINT_IMAGE: ${{ github.event.inputs.mint-image || 'minio/mint:edge@sha256:08a05e68893c68be2a83b6f79556853ed6aa3c6c9e64c823a00853e4e55d2200' }}
+
+concurrency:
+  group: ${{ github.workflow }}-${{ github.ref }}
+  cancel-in-progress: true
+
+# Only alert-on-failure needs more than read access; it declares its own
+# job-level `issues: write`.
+permissions:
+  contents: read
+
+defaults:
+  run:
+    shell: bash
+
+jobs:
+  mint:
+    # GitHub-hosted: reliably provides a running Docker daemon + buildx +
+    # python3. See the header note (ci-2) for why the self-hosted sm-standard-4
+    # label was abandoned (no docker.sock on the pod its only run landed on).
+    # TODO(ci-8): scheduled-failure alerting (auto-open issue on a red weekly
+    # run) is added by the ci-8 composite action; do not implement it here.
+    runs-on: ubuntu-latest
+    timeout-minutes: 120
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Enable buildx
+        uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
+
+      - name: Build RustFS image (source, cached)
+        run: |
+          DOCKER_BUILDKIT=1 docker buildx build --load \
+            --platform "${PLATFORM}" \
+            --cache-from "type=gha,scope=${BUILDX_CACHE_SCOPE}" \
+            --cache-to "type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE}" \
+            -t rustfs-ci \
+            -f Dockerfile.source .
+
+      - name: Start RustFS
+        run: |
+          docker network inspect rustfs-net >/dev/null 2>&1 || docker network create rustfs-net
+          docker rm -f rustfs-mint >/dev/null 2>&1 || true
+          # The four disks share one physical device on the runner (a single
+          # loopback filesystem), so the local physical-disk-independence guard
+          # would refuse to start. Bypass it — this is the CI use case the guard
+          # explicitly sanctions via RUSTFS_UNSAFE_BYPASS_DISK_CHECK.
+          docker run -d --name rustfs-mint \
+            --network rustfs-net \
+            -p 9000:9000 \
+            -e RUSTFS_ADDRESS=0.0.0.0:9000 \
+            -e RUSTFS_ACCESS_KEY="${S3_ACCESS_KEY}" \
+            -e RUSTFS_SECRET_KEY="${S3_SECRET_KEY}" \
+            -e RUSTFS_VOLUMES="/data/rustfs{0...3}" \
+            -e RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true \
+            -v /tmp/rustfs-mint:/data \
+            rustfs-ci
+
+      - name: Wait for RustFS ready
+        run: |
+          for _ in {1..60}; do
+            if curl -sf http://127.0.0.1:9000/health >/dev/null 2>&1; then
+              echo "RustFS is ready"
+              exit 0
+            fi
+            if [ "$(docker inspect -f '{{.State.Running}}' rustfs-mint 2>/dev/null)" != "true" ]; then
+              echo "RustFS container not running" >&2
+              docker logs rustfs-mint || true
+              exit 1
+            fi
+            sleep 2
+          done
+          echo "Health check timed out" >&2
+          docker logs rustfs-mint || true
+          exit 1
+
+      - name: Run mint
+        run: |
+          mkdir -p artifacts/mint-log
+          # Mint exits non-zero when any test fails; that is inventory here,
+          # not a gate (see header comment). Capture the exit code and let the
+          # summary step decide.
+          set +e
+          # shellcheck disable=SC2086  # MINT_SUITES is intentionally word-split
+          docker run --rm --network rustfs-net \
+            -e SERVER_ENDPOINT=rustfs-mint:9000 \
+            -e ACCESS_KEY=${S3_ACCESS_KEY} \
+            -e SECRET_KEY=${S3_SECRET_KEY} \
+            -e ENABLE_HTTPS=0 \
+            -e SERVER_REGION=${S3_REGION} \
+            -e MINT_MODE=${MINT_MODE} \
+            -v "${PWD}/artifacts/mint-log:/mint/log" \
+            "${MINT_IMAGE}" ${MINT_SUITES}
+          echo "MINT_RC=$?" >> "$GITHUB_ENV"
+
+      - name: Summarize results
+        run: |
+          python3 - <<'EOF'
+          import json, os, pathlib, sys
+          from collections import Counter
+
+          log = pathlib.Path("artifacts/mint-log/log.json")
+          if not log.is_file():
+              print("mint produced no log.json - infrastructure failure", file=sys.stderr)
+              sys.exit(int(os.environ.get("MINT_RC", "1")) or 1)
+
+          per_suite: dict[str, Counter] = {}
+          failures: list[tuple[str, str, str]] = []
+          for line in log.read_text(encoding="utf-8").splitlines():
+              line = line.strip()
+              if not line:
+                  continue
+              try:
+                  entry = json.loads(line)
+              except json.JSONDecodeError:
+                  continue
+              suite = entry.get("name", "unknown")
+              status = entry.get("status", "unknown").upper()
+              per_suite.setdefault(suite, Counter())[status] += 1
+              if status == "FAIL":
+                  failures.append((suite, entry.get("function", ""), entry.get("error") or entry.get("message") or ""))
+
+          lines = ["# Mint results", "", "| Suite | Pass | Fail | N/A |", "|-------|------|------|-----|"]
+          for suite in sorted(per_suite):
+              c = per_suite[suite]
+              lines.append(f"| {suite} | {c.get('PASS', 0)} | {c.get('FAIL', 0)} | {c.get('NA', 0)} |")
+          lines.append("")
+          if failures:
+              lines.append(f"## Failures ({len(failures)})")
+              lines.append("")
+              for suite, func, err in failures:
+                  first = err.strip().splitlines()[0] if err.strip() else ""
+                  lines.append(f"- `{suite}` / `{func}` {first}")
+              lines.append("")
+              lines.append("_Report-only: failures are inventory, not a gate (see workflow header)._")
+
+          report = "\n".join(lines)
+          print(report)
+          summary = os.environ.get("GITHUB_STEP_SUMMARY")
+          if summary:
+              with open(summary, "a", encoding="utf-8") as fh:
+                  fh.write(report + "\n")
+          EOF
+
+      - name: Collect RustFS logs
+        if: always()
+        run: |
+          mkdir -p artifacts/rustfs-mint
+          docker logs rustfs-mint > artifacts/rustfs-mint/rustfs.log 2>&1 || true
+
+      - name: Upload artifacts
+        if: always() && env.ACT != 'true'
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: mint
+          path: artifacts/**
+
+  alert-on-failure:
+    name: Alert on scheduled failure
+    needs: [mint]
+    # `always()` is required: without it this job is skipped when a needed
+    # job fails. Alerts only for scheduled runs (backlog#1149 ci-8) — manual
+    # dispatch failures are already watched by a human.
+    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    permissions:
+      contents: read
+      issues: write
+    steps:
+      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+      - name: Open or update failure-tracking issue
+        uses: ./.github/actions/schedule-failure-issue
+        with:
+          github-token: ${{ secrets.GITHUB_TOKEN }}
@@ -17,7 +17,7 @@ name: Update Nix Flake
 on:
  workflow_dispatch:
  schedule:
-    - cron: '0 0 * * 0' # Weekly on Sundays
+    - cron: '0 5 * * 0' # Weekly on Sunday 05:00 UTC (staggered after the midnight ci/build crons)

 permissions:
  contents: write
@@ -31,20 +31,25 @@ jobs:
  update-flake:
    name: Update flake.lock
    runs-on: ubuntu-latest
-    timeout-minutes: 45
+    timeout-minutes: 90
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
    steps:
      - name: Checkout repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7

      - name: Install Nix
-        uses: DeterminateSystems/determinate-nix-action@v3
+        uses: DeterminateSystems/determinate-nix-action@629b284231c2a82554b724e357e47fc6020833c8 # v3
+
+      - name: Cache Nix
+        uses: DeterminateSystems/flakehub-cache-action@1f9a51a2959d3e26c7838c6f3bf9f48acae525ea # v3.20.0

      - name: Check Nix flake inputs
-        uses: DeterminateSystems/flake-checker-action@v12
+        uses: DeterminateSystems/flake-checker-action@3164002371bc90729c68af0e24d5aacf20d7c9f6 # v12

      - name: Update flake.lock
        id: update
-        uses: DeterminateSystems/update-flake-lock@main
+        uses: DeterminateSystems/update-flake-lock@fd9359ac79d0e912f1b4b947a48470b3e2799b56 # main
        with:
          git-author-name: houseme
          git-author-email: housemecn@gmail.com
@@ -56,7 +61,7 @@ jobs:
            nix
            automated
          commit-msg: "chore(deps): update flake.lock"
-          pr-reviewers: houseme, overtrue, majinghe
+          pr-reviewers: overtrue, majinghe
          token: ${{ secrets.FLAKE_UPDATE_TOKEN }}

      - name: Log PR details
@@ -24,6 +24,7 @@ on:
      - 'Cargo.lock'
      - '.github/workflows/nix.yml'
  pull_request:
+    types: [ opened, synchronize, reopened, closed ]
    branches: [ "main" ]
    paths:
      - 'flake.nix'
@@ -34,43 +35,81 @@ on:
  workflow_dispatch:

 concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
  cancel-in-progress: true
 permissions:
  contents: read
+  id-token: write

 jobs:
+  cancel-closed-pr-runs:
+    name: Cancel Closed PR Runs
+    if: github.event_name == 'pull_request' && github.event.action == 'closed'
+    runs-on: ubuntu-latest
+    steps:
+      - name: Explain cancellation run
+        run: echo "PR closed; this run only cancels older runs in the same concurrency group."
+
  nix-validation:
    name: Nix Build & Check
+    if: github.event_name != 'pull_request' || github.event.action != 'closed'
    runs-on: ubuntu-latest
-    timeout-minutes: 45
+    timeout-minutes: 60
+    permissions:
+      contents: read
+      id-token: write
+    env:
+      NIX_CURL_FLAGS: -A cargo/stable
    steps:
      - name: Checkout repository
-        uses: actions/checkout@v6
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7

      - name: Install Nix
-        uses: cachix/install-nix-action@v31
+        uses: DeterminateSystems/determinate-nix-action@4eea0b33e3d1f02ecfe37cf16e7204c424009606 # v3.21.0
        with:
-          github_access_token: ${{ secrets.GITHUB_TOKEN }}
-          extra_nix_config: |
-            experimental-features = nix-command flakes
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          extra-conf: |
+            experimental-features = nix-command flakes configurable-impure-env
+            impure-env = NIX_CURL_FLAGS
+            max-jobs = 1

-      - name: Setup Magic Nix Cache
-        uses: DeterminateSystems/magic-nix-cache-action@v13
+      - name: Cache Nix
+        uses: DeterminateSystems/flakehub-cache-action@c01e819d047464c3edf6ba778f075952af5a3aa7 # v3.21.0

-      - name: Setup Flake Checker
-        uses: DeterminateSystems/flake-checker-action@v12
+      - name: Check Nix Flake Inputs
+        uses: DeterminateSystems/flake-checker-action@3164002371bc90729c68af0e24d5aacf20d7c9f6 # v12
+        with:
+          fail-mode: true
+          ignore-missing-flake-lock: false

      - name: Verify Flake
        run: |
          echo "Checking flake structure and evaluation..."
          nix flake show
-          nix flake check --print-build-logs
+          for attempt in 1 2 3; do
+            if nix flake check --print-build-logs --fallback --option fallback true; then
+              break
+            fi
+            if [ "$attempt" -eq 3 ]; then
+              echo "nix flake check failed after 3 attempts."
+              exit 1
+            fi
+            sleep $((attempt * 15))
+          done

      - name: Build RustFS
        run: |
          echo "Building the default package..."
-          nix build .#default --print-build-logs
+          for attempt in 1 2 3; do
+            if nix build .#default --print-build-logs --fallback --option fallback true; then
+              break
+            fi
+            if [ "$attempt" -eq 3 ]; then
+              echo "nix build failed after 3 attempts."
+              exit 1
+            fi
+            sleep $((attempt * 15))
+          done

      - name: Test Binary
        run: |
--- a/Show More
+++ b/Show More