feat(obs): enhance observability with tracing spans and metrics integration

.agents/skills/adversarial-validation/SKILL.md

@@ -1,45 +1,266 @@
 ---
 name: adversarial-validation
-description: Review a final RustFS diff adversarially when the user requests adversarial review, the root AGENTS.md classifies the change as high risk, or a substantial PR is being reviewed. Do not use for ordinary questions, diagnosis, planning, status, documentation-only work, or routine low-risk implementation.
+description: Execute the Adversarial Validation policy from the root AGENTS.md — run the six reviewer roles (correctness, security, concurrency/durability, compatibility, performance, test coverage) with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, or design proposal before declaring it done.
 ---
 
-# RustFS Adversarial Validation
+# Adversarial Validation Playbooks
 
-Use the risk tier and review shape defined in the root `AGENTS.md`. This skill
-routes a review to RustFS-specific probes without loading unrelated domains.
+The policy — risk tiers, role list, protocol, exit criteria — lives in the
+root `AGENTS.md` under "Adversarial Validation (Default On)". Read it first;
+this skill does not restate it. This file adds the RustFS-specific probe
+playbook for each role: concrete attacks, where they apply, and the real
+shipped bug or rule that earns each probe its place.
 
-## Select Lenses
+## How to run a role
 
-Read only the references required by the diff:
+1. Pick the tier and the applicable roles per the root `AGENTS.md`.
+2. Run each role as an independent pass over the final diff — a parallel
+   reviewer agent where the tooling supports it, otherwise a fresh
+   sequential pass that starts from the diff and the nearest scoped
+   `AGENTS.md`, discarding the writing session's assumptions.
+3. Within a role, execute the probes whose domain the diff touches, plus any
+   attack the diff obviously invites that no probe lists — the playbook is a
+   floor, not a ceiling.
+4. Report findings (concrete failure scenario or named missing test, with
+   file:line) or the role's null report: "attacked X, Y, Z — no break
+   found". A bare pass is not a result.
 
-| Lens | When to read |
-|---|---|
-| [Correctness](references/correctness.md) | Every non-exempt adversarial review |
-| [Simplicity](references/simplicity.md) | Mechanical/standard changes and production growth |
-| [Test coverage](references/test-coverage.md) | Behavior or test changes |
-| [Security](references/security.md) | Authn/authz, IAM, RPC trust, paths, secrets, parsing, browser, encryption |
-| [Concurrency/durability](references/concurrency-durability.md) | Async shared state, locks, storage commit, cancellation, persisted queues |
-| [Compatibility](references/compatibility.md) | S3 surface, MinIO interop, metadata, wire/disk formats, mixed versions |
-| [Performance](references/performance.md) | Request/object hot paths, allocation, blocking work, fsync, fan-out |
+## Role playbooks
 
-Do not read all references as a precaution. A path name alone is insufficient;
-the changed behavior must touch the lens's domain.
+### Correctness adversary
 
-For a dedicated security audit or advisory analysis, use
-`security-advisory-lessons` instead of loading it automatically during every
-adversarial review.
+- For any change touching error aggregation or quorum decisions, build the exact disk-error slice at the quorum boundary: N disks where successes == quorum, then flip one success to an error (quorum-1) and separately inject None/nil placeholder entries into the slice. Trace whether reduce_errs (or the new equivalent) picks the placeholder as the dominant error or lets quorum-1 pass as success. Also check heal/write paths: does a per-target failure at quorum-1 return an explicit error, or silently degrade to success?
+  - Where: crates/ecstore/src/disk/error_reduce.rs; crates/ecstore/src/set_disk/{core,ops}; crates/heal
+  - Evidence: Commit 20d61c73b 'stop reduce_errs leaking nil placeholder as dominant error' (#4551) and 47c1e730c 'make erasure heal write quorum best-effort per target' (#4545); crates/ecstore/AGENTS.md: 'Do not weaken quorum checks... Prefer explicit failure over silent data corruption or implicit success.'
+- For any change in EC read/reconstruct/streaming code, trace the mid-stream error path: the first K shards read fine, then a shard turns out bitrot-corrupt or inconsistent after N bytes have already been sent to the client. Verify the error propagates as a stream error (client sees failure), not a clean end-of-body — a silently truncated GET body is data corruption. Also re-check byte accounting: sum of per-part bytes vs object size, and partNumber-to-offset routing for the first and last part.
+  - Where: crates/ecstore/src/set_disk/read.rs (reconstruct-read, inconsistent_source_indexes handling ~line 3827); codec streaming paths in crates/ecstore/src/erasure_coding
+  - Evidence: Known live bug: EC reconstruct-read failing on inconsistent shards mid-stream silently truncates GET body → client 'unexpected EOF'; commit 15808254d 'correct codec-streaming byte accounting and partNumber routing' (#4535).
+- For any listing/pagination change, construct the exact-boundary inputs: (a) exactly max_keys/max_uploads matching entries — assert the response contains max and is_truncated=false, then max+1 entries — assert exactly max returned with is_truncated=true and a correct continuation marker; (b) a delimiter listing where folding into CommonPrefixes re-fills a full page; (c) an object 'a' coexisting with prefix dir 'a/'. Off-by-one and dropped-truncation bugs live exactly at these boundaries.
+  - Where: crates/ecstore listing/merge paths (metacache, list_objects, list_multipart_uploads); rustfs/src/storage
+  - Evidence: Three recent real bugs: fefa70b31 'stop ListMultipartUploads from returning one upload past max-uploads' (#4447), d91f4d455 'report truncation when delimiter list re-folds a full page' (#4538), 7e1f7f242 'preserve CommonPrefixes when an object and same-named prefix dir coexist' (#4563).
+- Run the diff's logic with a directory object key (trailing slash, e.g. 'pre/dir/') as input. Check which layer encodes/decodes __XLDIR__ — set_disk never sees the trailing slash, so any trailing-slash branch added below the store layer is dead code and a wrong-layer bug. For delete paths, check whether options force a nil versionId onto directory keys: the resulting miss surfaces as version-not-found, not object-not-found, so callers matching only ObjectNotFound leak ghost directory entries.
+  - Where: crates/ecstore/src/store*.rs (store layer) vs crates/ecstore/src/set_disk/*; delete option construction (del_opts) and its callers
+  - Evidence: trailing-slash branches must live at store layer; del_opts injects nil version for dir keys, PR#4220 ghost-directory cleanup never fired on the real path (rustfs#4307, backlog#798 still OPEN).
+- Feed every new binary-UUID metadata read the three degenerate values: key absent, zero-length bytes, and 16 zero bytes (nil UUID). All three must mean 'no value' — any path that produces Uuid::nil() and then acts on it (e.g. sends it as a versionId) is a finding. For tier code specifically: with remote-tier version None or "", assert the tier GET/DELETE request carries no versionId parameter at all — sending versionId="" or nil yields NoSuchVersion against unversioned tier buckets.
+  - Where: crates/filemeta/src/filemeta/version.rs; crates/ecstore/src/bucket/lifecycle/; crates/ecstore/src/services/tier/; any new consumer of crates/utils/src/http/metadata_compat.rs
+  - Evidence: AGENTS.md Cross-Cutting Domain Invariants (defensive Uuid read pattern + unversioned-tier rule); docs/operations/tier-ilm-debugging.md ('nil transition_ver_id = corrupt legacy write-back, readers must filter'); commit 726f3dc18 'accept empty remote version_id in tier recovery paths' (#4552).
+- For each match/if-let on an error or algorithm enum touched by the diff, enumerate what the wildcard/else arm swallows. Inject the variants the author didn't think of — DiskNotFound during listing, an unsupported checksum algorithm, an Err from a cleanup rename/delete — and trace whether they degrade into 'not found', a wrong-but-plausible value, or silent success. Any error path that converges with the success path without logging and propagating is a finding.
+  - Where: crates/ecstore (listing, delete/rename cleanup); crates/checksums; error-mapping layers in rustfs/src/storage
+  - Evidence: Three recent real bugs of this exact shape: e0619e355 'stop treating DiskNotFound as object not-found in listing' (#4536), afaf8c681 'reject md5 instead of silently returning crc32' (#4513), f7d2b2563 'propagate disk delete/rename failures instead of swallowing them' (#4546).
+- For any change to version ordering, index lookup, or shard/part indexing: (a) call the accessor with index == len() and len()-1 — a get_idx-style bound must reject, not panic or wrap; (b) construct two versions with identical mod-times and check the sort tie-break is total and deterministic (equal keys must not compare as both before each other); (c) for EC shard math, compute shard size for object sizes 0, 1, blockSize-1, blockSize, blockSize+1 and cross-check total reconstructed length against the object size.
+  - Where: crates/filemeta/src/filemeta/*.rs (version sort, get_idx); crates/ecstore/src/erasure_coding shard-size math
+  - Evidence: Commit 8bfb00bc0 'guard get_idx bound and fix sorts_before tie-break' (#4509) — both bug classes shipped before; ecstore AGENTS.md high-risk designation for read/write/repair correctness.
+- Exercise the zero/empty end of every new size or count parameter: zero-length object PUT then GET (body must be empty, not error), part count 0, empty Vec of disks/entries into aggregation functions, and env/config values of 0 (must clamp or reject, never divide-by-zero or 'scan nothing and report zero usage'). Anywhere the diff computes a ratio, capacity, or progress percentage, plug in 0 and the max value.
+  - Where: crates/ecstore aggregation and scanner paths; crates/object-capacity; config/env parsing in touched crates
+  - Evidence: Commits 787cc77a7 'clamp zero capacity env values to safe defaults' (#4559) and 32b1094ec 'resolve a symlinked scan root instead of silently counting zero' (#4564) — zero-as-silent-wrong-answer is a recurring repo bug class.
+- Smaller-diff attack: rewrite the diff's change mentally (or actually, in scratch) as the minimal in-place edit and compare. Flag as findings: a helper function with exactly one caller introduced by this diff; a file rewrite where a 3-line edit inside the existing control flow suffices; reshaped control flow in init/locking/metadata/quorum paths beyond what the fix requires; new string literals duplicating existing constants (grep the token first); #[path] module inclusion. If the smaller diff achieves identical behavior, report it with the concrete replacement.
+  - Where: Any diff; extra scrutiny for crates/ecstore, crates/lock, rustfs/src/storage where 'preserve the existing control-flow shape' is an explicit rule
+  - Evidence: AGENTS.md 'Change Style for Existing Logic' (one-off helper ban, preserve control-flow shape in distributed/locking/metadata paths, no #[path]) and 'Constant and String Usage'; Adversarial Validation section names the smaller-diff clause as a correctness-adversary finding.
+- For any diff touching multipart or object commit paths, order the operations on paper and attack the failure point between them: kill the process (or return Err) after the commit rename but before cleanup, and after cleanup but before commit. Verify the earlier-failure case leaves the object readable and the later-failure case leaves no half-visible object; part meta files must never be deleted before the commit is durable.
+  - Where: crates/ecstore multipart commit/cleanup (set_disk/ops); rustfs/src/storage multipart handlers
+  - Evidence: Commit c77c5f047 'defer multipart part.N.meta cleanup until after commit' (#4548) — cleanup-before-commit ordering already caused a real data-loss window; the #4221 durability work shows fsync/ordering bugs are endemic here.
 
-## Review Protocol
+Null report example: "Attacked quorum-1 error reduction, exact max-keys listing boundary, trailing-slash dir keys, nil-UUID tier versionId, mid-stream reconstruct error propagation, and a minimal-diff rewrite — no break found; diff is already the minimal in-place edit."
 
-1. Freeze the exact final diff/head and list the selected lenses.
-2. Run the review shape required by root `AGENTS.md`.
-3. For each selected lens, either report a concrete finding or a null verdict
-   naming the attacks performed.
-4. A finding needs `file:line`, a triggering input/state/interleaving, the wrong
-   outcome, and a focused fix or missing regression check.
-5. Fix or rebut every finding with code-path, test, or invariant evidence.
-6. After a non-trivial edit, rerun only lenses affected by that edit against the
-   new exact diff.
+### Security reviewer
 
-Do not turn a null verdict into a long checklist. Record concise evidence that
-the relevant failure classes were attacked.
+- For every admin handler in the diff, grep the exact AdminAction constant it passes to validate_admin_request and confirm it names the operation the handler actually performs. Construct the escalation: a low-privileged user whose policy grants the wrong-but-adjacent action (e.g. Export while the handler Imports, or Update while it Lists) — if the mismatched constant lets them through, that is the bug. Also confirm read-only endpoints (metrics, list, server-info, diagnostics) still call an operation-specific admin authz path and not a mere 'credentials exist' check.
+  - Where: rustfs/src/admin/handlers/**, rustfs/src/admin/router registration; check_permissions / validate_admin_request / AdminAction::* call sites
+  - Evidence: GHSA-vcwh-pff9-64cc (ImportIam checked ExportIAMAction), GHSA-mm2q-qcmx-gw4w (ListServiceAccount used UpdateServiceAccountAdminAction), GHSA-f5cv-v44x-2xgf (/admin/v3/metrics accepted any authenticated IAM user). advisory-patterns.md 'Admin authorization and route exposure'; rustfs/src/admin/AGENTS.md 'route registration, whitelist, handler authz must agree'.
+- If the diff touches service-account or IAM import/update, treat parent, claims, accessKey, secretKey, status, policy names, and groups as attacker-controlled. Construct an ImportIam/create payload where parent points at root (or another user) and prove the code writes credentials without proving caller ownership or root authority. Separately, set deny_only=true (or 'no explicit deny') on a restricted account and check it does not skip the required allow check, letting it mint an unrestricted child.
+  - Where: crates/iam/, rustfs/src/admin/handlers (service account / import IAM), rustfs/src/auth.rs
+  - Evidence: GHSA-566f-q62r-wcr8 (attacker-controlled parent/claims/accessKey/secretKey → persistent backdoor under root), GHSA-xgr5-qc6w-vcg9 (deny_only=true skipped allow checks → privilege creation). crates/iam/AGENTS.md security boundaries; advisory-patterns.md 'IAM import, service accounts'.
+- For a changed protocol-frontend handler (FTP/FTPS/SFTP/WebDAV/gateway), enumerate ALL sibling command handlers in the same driver — not just the changed one. For each, confirm it calls the per-operation IAM authorize hook mapped to the correct S3 action (RETR→GetObject, SIZE/MDTM→HeadObject, MKD→CreateBucket, bucket probe→ListBucket/HeadBucket) BEFORE touching storage. Construct a denied-authz case and prove the storage backend is never reached.
+  - Where: crates/protocols/ (FtpsDriver, SftpDriver, WebDAV), authorize_operation call sites
+  - Evidence: GHSA-3g29-xff2-92vp (FTP RETR/SIZE/MDTM authenticated but skipped IAM), GHSA-g3vq-vv42-f647 (FTPS MKD called create_bucket without s3:CreateBucket). advisory-patterns.md: 'RustFS advisories show mixed guarded and unguarded siblings in the same driver.'
+- For any secret/token/signature/password comparison in the diff, check it uses a constant-time compare (e.g. subtle/constant_time_eq), not == or early-return byte loops. Then check the failure-response paths: construct an invalid-user request and an invalid-secret request and confirm they are indistinguishable (same error, no early length short-circuit) so an attacker cannot enumerate valid users or time-side-channel the secret.
+  - Where: crates/protocols/ (FTPS/WebDAV/FormPost auth), crates/credentials/, rustfs/src/auth.rs, RPC signature verification
+  - Evidence: GHSA-3p3x-734c-h5vx (FTPS/WebDAV early-return string equality + distinguishable invalid-user vs invalid-password). Fix commits 3c3113619 (constant-time FTPS/WebDAV) and c41062f27 (constant-time FormPost signature). 3p3x was fixed by PR #4403.
+- If the diff touches internode/RPC auth secret handling, trace whether the RPC HMAC secret can fall back to a public default (e.g. 'rustfsadmin', 'rustfs rpc') or be derived deterministically from the S3 root credentials. Construct the case where RUSTFS_RPC_SECRET is unset and confirm the code fails closed rather than silently using a default or a root-derived key. Verify RPC signing keys are independent random secrets, not reused across S3-root/RPC-HMAC/STS-JWT roles.
+  - Where: crates/credentials/, crates/ecstore/src/rpc/, internode auth setup
+  - Evidence: GHSA-r5qv-rc46-hv8q (fell back to 'rustfsadmin'), GHSA-75fx/68cw (RPC secret derivable from root creds → forgeable signatures), GHSA-h956 (hard-coded 'rustfs rpc'), GHSA-m77q (STS JWT reused root secret). Fix commit 7b2055405 (fail closed when deriving RPC secret from default credentials, PR#4402).
+- If the diff touches RPC/NodeService authentication, verify the HMAC payload binds the EXACT concrete gRPC method path (not a service prefix), the HTTP method surrogate, and a fresh timestamp. Construct captured valid metadata for method A and replay it to method B within the timestamp window — if it authorizes, the signature is under-bound. Also test stale timestamp, wrong path, wrong method, wrong secret.
+  - Where: crates/ecstore/src/rpc/, verify_rpc_signature / NodeServiceServer, x-rustfs-signature handling
+  - Evidence: GHSA-c667-rgrv-99vj (signed service prefix instead of concrete method path → cross-method replay in timestamp window). advisory-patterns.md 'RPC input validation and panic safety'; Minimum Regression Test Expectations lists replay across two methods.
+- For any RPC/gRPC handler or deserialization touched, feed empty bytes, truncated MessagePack/protobuf, invalid enum discriminants, and stale timestamps. Grep the deserialization path for unwrap()/expect()/panic-prone decode and prove malformed attacker payloads return a typed error, not a panic (remote DoS). Weak internode auth makes reachability worse, so combine with the RPC-secret probe.
+  - Where: crates/ecstore/src/rpc/, any #[derive(Deserialize)] decoded from wire bytes, RPC handler bodies
+  - Evidence: GHSA-gw2x-q739-qhcr (malformed GetMetrics reached unwrap() → remote DoS). advisory-patterns.md 'Treat all RPC payload bytes as attacker-controlled.' rust-code-quality skill: unwrap abuse.
+- Take any object key, RPC disk path, or archive/tar/zip entry name introduced or handled in the diff and construct traversal payloads: '../', URL-encoded '%2e%2e%2f', absolute paths, platform separators, empty components. Trace the value through parse → authz check → final storage path and prove (a) authz and storage normalize the SAME way, and (b) the canonicalized path cannot escape the bucket/prefix root. Attack the case where authz sees the raw attacker bucket but storage cleaning crosses into a victim bucket.
+  - Where: crates/ecstore (path join/canonicalize), rustfs/src/storage/, Snowball auto-extract / normalize_extract_entry_key, rpc read_file_stream
+  - Evidence: GHSA-pq29-69jg-9mxc (read_file_stream joined untrusted paths, no canonical boundary check), GHSA-8r6f-hmq2-28rg (traversal object keys bypassed authz), GHSA-f4vq-9ffr-m8m3 (Snowball '../victim-bucket/object' authorized raw path then storage crossed boundary). Note __XLDIR__ trailing-slash encoding is store-layer only.
+- If the diff touches multipart/copy or presigned POST, verify UploadPartCopy enforces source GetObject AND destination PutObject semantics equivalent to CopyObject, including copy-source policy conditions (not just independent source-read + dest-write). Construct a cross-bucket UploadPartCopy from a bucket the caller cannot read. For presigned POST, submit an upload that violates content-length-range, key prefix, or exact content-type and prove the server rejects it.
+  - Where: rustfs/src/storage / S3 API handlers: upload_part_copy, CompleteMultipartUpload, PostObject policy enforcement
+  - Evidence: GHSA-mx42-j6wv-px98 (UploadPartCopy missed source authz → cross-bucket exfil), GHSA-wfxj-ph3v-7mjf (missed destination copy-source policy constraint), GHSA-w5fh-f8xh-5x3p (presigned POST didn't enforce signed policy conditions). advisory-patterns.md 'S3 copy, multipart, and upload policy validation'.
+- Grep the diff for debug!/trace!/info!/error! and any ?value / {:?} on structs or response bodies that can carry secret_key, session_token, JWT claims, HMAC secrets, expected signatures, access keys beyond safe identifiers, or raw credential-bearing responses. Check custom Debug impls and merged-config dumps too. Construct the error path (invalid signature, failed auth) and confirm it does not log the secret or the derived authenticator. Verify audit/notify entries redact credential request headers.
+  - Where: rustfs/src/**, crates/iam/, crates/audit/, crates/notify/, crates/targets/, RPC signature error paths
+  - Evidence: GHSA-r54g (STS creds logged at info), GHSA-8cm2 (debug logs leaked tokens/secrets/JWT claims/raw STS bodies), GHSA-333v (invalid RPC signature log included HMAC secret + expected signature). Fix commit ee6f79110 (redact credential request headers from audit/notify, backlog#963). rustfs-logging-governance skill.
+- For any struct in the diff deserialized from untrusted input (S3 XML/JSON, lifecycle rules, bucket policy, replication config, RPC payload), check for #[serde(deny_unknown_fields)]. Construct a payload with a typo'd field (e.g. 'NoncurentDays') or an extra field and prove it is rejected, not silently ignored. Flag #[serde(default)] on security-critical fields (retention days, limits, permissions) lacking explicit post-deserialize validation, and any user-controlled integer cast with `as` (i32 as u32) — feed a negative value and check it doesn't wrap to a huge positive.
+  - Where: crates/policy/ (bucket policy), crates/ecstore lifecycle/ILM config, replication config structs, crates/protocols XML parsing
+  - Evidence: AGENTS.md 'Serde Safety'; advisory-patterns.md 'Serde deserialization' (no deny_unknown_fields found repo-wide; 'NoncurentDays' typo silently accepted; i32 as u32 wrap). Fix commit 1acd47f15 (SSE crash-loop DoS + credential reserved-char bypass, backlog#806).
+- If the diff touches SSE / encryption reader-writer composition, do not trust API metadata claiming encryption. Trace the reader wrapper order (HashReader / EncryptReader / compression / warp) and confirm EncryptReader is actually in the chain that writes to disk — construct the case where a helper unwraps a nested reader and bypasses encryption, storing plaintext. Require a regression test that inspects the ACTUAL stored bytes on disk, not just read-back. Also check encrypted-object checksums are not exposed.
+  - Where: rustfs/src/storage/ecfs.rs, crates/rio/ (reader wrappers), crates/kms/, SSE-C replication
+  - Evidence: GHSA-xrrf-67jm-3c2r (SSE metadata reported encryption while composition bypassed EncryptReader, stored plaintext). Fix commits a7b9659e7 (hide encrypted object checksums, #4529), 80cc3b1fc (preserve SSE-C checksum state, #4410). advisory-patterns.md 'SSE and on-disk storage invariants'.
+- If the diff touches CORS or the console/browser/object-preview surface: confirm default CORS does not reflect an arbitrary Origin while also sending Access-Control-Allow-Credentials: true — construct a request with a spoofed Origin and check the response. For preview, confirm attacker-controlled object content is origin-isolated (not rendered in a same-origin iframe with console creds), served with nosniff/CSP, and that preview trust derives from validated content-type + sandboxing, NOT from object name/extension (.pdf, .html). Separately, if aws:SourceIp is evaluated, spoof X-Forwarded-For / X-Real-IP as a direct (non-trusted-proxy) client and confirm the socket peer IP is used instead.
+  - Where: rustfs/src/server/layer.rs (CORS), console preview/auth code, aws:SourceIp / policy condition evaluation, X-Forwarded-For handling
+  - Evidence: GHSA-x5xv-223c-8vm7 (default CORS reflected arbitrary origins with credentials), GHSA-v9fg-3cr2-277j (preview rendered attacker HTML same-origin, exposed localStorage creds), GHSA-7gcx-wg4x-q9x6 (extension-based PDF detection bypassed sandbox), GHSA-fc6g-2gcp-2qrq (aws:SourceIp trusted client XFF). advisory-patterns.md 'Browser, CORS' + 'Trusted proxy'.
+
+Null report example: "Attacked admin action-constant matching in the two changed handlers (both call validate_admin_request with the exact AdminAction), the FTPS RETR/SIZE authz parity, and the new lifecycle struct's serde surface (has deny_unknown_fields) — no break found."
+
+### Concurrency/durability reviewer
+
+- For every new or moved lock acquisition, enumerate all other code paths that take any overlapping subset of those locks and construct the concrete ABBA interleaving (thread 1 holds A wants B, thread 2 holds B wants A). If the diff acquires 2+ locks without a comment documenting acquisition order, that alone is a finding.
+  - Where: crates/ecstore/** (namespace locks, set_disk, disk registry), crates/audit/**, crates/lock/**, any Mutex/RwLock pair in a diff
+  - Evidence: crates/ecstore/AGENTS.md 'Lock Ordering' (document order; same set in different orders = deadlock); real ABBA deadlock fixed in c0d5f938f (#4421, audit registry vs stream_cancellers)
+- If the diff touches the object write/commit path or a lock guard's lifetime, construct the timeline where the distributed lock is lost (heartbeat refresh fails / expiry) after shard writes but before the xl.meta rename commit — verify the commit is fenced on guard.is_lock_lost() (set_disk/ops/object.rs:874-880) and the diff does not move the commit outside the fenced region or drop the guard early.
+  - Where: crates/ecstore/src/set_disk/ops/object.rs, ops/multipart.rs, crates/lock/**
+  - Evidence: 1e6207c08 (#4406) fence write commit on lock loss; ddf197ba5 (#4388) heartbeat lock refresh; backlog#899 fencing comment in object.rs
+- For any change to file creation or write-then-rename: write out the exact syscall order (write tmp -> fdatasync tmp -> rename -> fsync parent dir -> fsync ancestor dirs on first object under a prefix) and simulate a power cut after each step. Flag any dropped/reordered sync, and check the change honors the durability gate (RUSTFS_DRIVE_SYNC_ENABLE, strict/relaxed/none modes, per-bucket overrides) instead of hardcoding one mode. The 'skip tmp parent fsync' optimization is only sound when the file is renamed out of tmp — verify that precondition still holds.
+  - Where: crates/ecstore/src/disk/local.rs, disk/os.rs, disk/fs.rs, crates/ecstore/src/bucket/durability.rs, set_disk/core/io_primitives.rs
+  - Evidence: PR #4221 (the repo previously had no fsync anywhere); 2df315baf/c081586e7 (#4493) fsync ancestor dirs; 062a68d15 (#4387) tmp-parent-fsync skip is rename-conditional; eaff17cad (#4397) durability modes; 54872d52d (#4478) rename_data crash harness exists — extend it for the diff
+- If the diff touches quorum counting or per-disk error aggregation, construct adversarial error vectors for reduce_errs: nil/placeholder entries, DiskNotFound mixed with FileNotFound, exactly quorum-1 agreeing errors — and show which dominant error wins. Specifically attack the case where offline-disk errors get counted as 'object does not exist', flipping a read/heal decision into data loss. Also check quorum monotonicity: a retry or heal pass must never conclude with a LOWER quorum than the original write.
+  - Where: crates/ecstore/src/disk/error_reduce.rs, crates/ecstore/src/api/mod.rs, set_disk read/heal paths
+  - Evidence: 20d61c73b (#4551) reduce_errs leaked nil placeholder as dominant error; e0619e355 (#4536) DiskNotFound treated as object-not-found in listing; quorum monotonicity flagged as open follow-up to #4221 (#4221 follow-up list); crates/ecstore/AGENTS.md 'Do not weaken quorum checks'
+- For any multi-disk fan-out (delete, rename, heal write, cleanup), trace each per-disk Result: find any `let _ =`, `.ok()`, or best-effort collapse that keeps a failed disk out of the quorum math. Construct the run where exactly write_quorum-1 disks succeed and prove the op still returns success — that is the bug. Conversely, for heal writes, check one bad target cannot fail the whole heal (best-effort per target).
+  - Where: crates/ecstore/src/set_disk/ops/*.rs, disk/disk_store.rs, crates/heal/**
+  - Evidence: f7d2b2563 (#4546) disk delete/rename failures were swallowed; 47c1e730c (#4545) heal write quorum made best-effort per target; 2b063b0c4 (#4400) disk-replacement heal missed versions
+- For every new .await placed between a state mutation and its cleanup/commit (or inside select!/timeout/spawned task that can be aborted), construct the cancellation point: client disconnects and the future is dropped exactly there. Enumerate what is left behind — tmp files, incremented counters never decremented, half-written xl.meta, a held permit/waiter — and verify cleanup runs in Drop or the state is re-entrant. Background loops the diff adds must have a hard outer timeout so a wedged awaitee cannot pin them forever.
+  - Where: crates/ecstore/** write paths, crates/object-capacity/** scanners, crates/audit/**, anything using tokio::select! or spawn+abort
+  - Evidence: d608e320f io_uring cancel-safety spike (cancellation known-hard here); 5a372557e (#4533) wedged scans needed hard outer timeout; 7b87d4d13 (#4520) waiter-count leak; e44bece00 (#4497) audit start race + paused drops
+- If the diff touches metacache/list producers or cursor resume, construct the interleaving where the producer task completes (or errors) while a reader with a saved cursor comes back for the next page — verify a completed producer is tolerated (no error, no hang) and that a re-folded/full page reports truncation instead of silently ending the listing early. Also feed a corrupt/oversized length prefix into any metacache decode the diff touches.
+  - Where: crates/ecstore/src/cache_value/metacache_set.rs, crates/ecstore/src/store/list_objects.rs, crates/filemeta/**
+  - Evidence: 91a23361e (#4531) tolerate completed metacache producers; d91f4d455 (#4538) delimiter re-fold dropped truncation flag; d2c100fd3 (#4226) corrupt length-prefix guard
+- For multipart changes, construct concurrent operations on the SAME uploadId: put_object_part racing put_object_part (same part number), abort racing complete between the parts listing and the commit rename, and list-parts racing cleanup. Verify every metadata read/list/abort holds the per-uploadId lock, and that part.N.meta cleanup is deferred until AFTER the commit rename — cleanup before commit loses parts on a crash between the two.
+  - Where: crates/ecstore/src/set_disk/ops/multipart.rs, rustfs/src/storage/
+  - Evidence: 3bc8d79fe (#4329) serialize put_object_part per uploadId; 7fb95d4fc (#4428) unlocked upload metadata reads/aborts; 93ffbdb9b (#4437) unlocked part listings; c77c5f047 (#4548) part.N.meta cleanup moved after commit
+- Any cleanup/rollback logic added near a commit: verify it runs strictly AFTER the commit is durable, is best-effort (its failure must not fail an already-committed write), and is safe under retry — i.e., re-running it after a partial first attempt must never delete the newly-committed data dir or the last surviving copy.
+  - Where: crates/ecstore/src/set_disk/ops/object.rs (rename_data tail), ops/multipart.rs, disk cleanup helpers
+  - Evidence: afc7f1d6f/d908243e6 (#4386, backlog#898) post-commit old-data-dir cleanup had to be made best-effort; e7cc719c1 (#4389) speculative tmp cleanup moved off hot path
+- If the diff does read-modify-write on any persisted shared state (bucket metadata, notify/target config, queue store), construct two concurrent writers: show whether the second write silently discards the first (lost update) — RMW must be serialized or CAS-guarded. For persisted queues/replay, construct crash-mid-replay and prove entries are neither lost nor delivered twice without an idempotency key.
+  - Where: crates/notify/**, crates/targets/** (queue store, SQL backends), crates/ecstore/src/bucket/metadata_sys.rs
+  - Evidence: 2490d4ee2 (#4425) persisted config RMW lost updates; 08e44b95f (#4505) queue store crash-safety + replay lifecycle; e008cc5da (#4500) SQL backend idempotency
+- If the diff touches erasure decode/reconstruct or streaming GET, construct the failure mid-stream: shards become inconsistent (or a disk read fails) after N bytes of the body have already been sent — verify the stream surfaces an error to the client instead of ending cleanly at a truncated length. Silent truncation on a 200 response is the known failure mode.
+  - Where: crates/ecstore/src/set_disk/read.rs (reconstruct-read validation, historically ~line 3117), crates/ecstore/src/erasure/coding/decode.rs
+  - Evidence: Known open bug: EC reconstruct-read 'inconsistent shards' mid-GET silently truncates body -> client unexpected EOF; crates/ecstore/AGENTS.md 'explicit failure over silent corruption'
+
+Null report example: "Attacked lock ordering on the new disk-registry mutex pair, lock-loss fencing across the moved commit, power-cut points around the added rename, and cancellation at the two new awaits — no break found; quorum math and metacache paths untouched by this diff."
+
+### Compatibility reviewer
+
+- Grep the diff for raw 'x-rustfs-internal-' or 'x-minio-internal-' string literals used with map.insert/remove/get instead of the metadata_compat helpers. If found, construct the MinIO-written object case: a metadata map containing ONLY 'X-Minio-Internal-<suffix>' (mixed case, no RustFS key) and trace the diff's read path — does it miss the value? Then construct the removal case: does remove leave the twin key behind so a stale MinIO-key value resurrects on next read? Also check the value-type trap: get_bytes has NO case-insensitive fallback (unlike get_str), so a diff that moves a suffix from FileInfo.metadata (String) to meta_sys (Vec<u8>) silently loses mixed-case MinIO keys.
+  - Where: Any code touching FileInfo.metadata / user_defined / meta_sys: crates/ecstore/, crates/filemeta/, rustfs/src/storage/, crates/utils/src/http/metadata_compat.rs
+  - Evidence: Repo-wide invariant in AGENTS.md 'Cross-Cutting Domain Invariants' and CLAUDE.md; helpers and the asymmetry are pinned by tests test_str_lookup_accepts_minio_metadata_case and test_get_bytes_no_case_insensitive_fallback in crates/utils/src/http/metadata_compat.rs
+- For any diff reading a binary UUID from internal metadata (transitioned-versionID, tier-free-versionID, data_dir), trace the three degenerate inputs — key absent, value empty (b""), value nil UUID — through to the outgoing tier/S3 request. The bug shape to hunt: unwrap_or_default() or Uuid::from_slice(..).unwrap_or(Uuid::nil()) turning 'no value' into Uuid::nil(), which then gets serialized as ?versionId=00000000-... and the remote tier returns NoSuchVersion. The required pattern is .and_then(|v| Uuid::from_slice(&v).ok()).filter(|u| !u.is_nil()).
+  - Where: crates/ecstore/src/bucket/lifecycle/ (bucket_lifecycle_ops.rs, tier_sweeper.rs), crates/ecstore/src/services/tier/warm_backend_*.rs, crates/filemeta/src/filemeta/version.rs
+  - Evidence: Historical production bug documented in docs/operations/tier-ilm-debugging.md ('Nil-UUID versionId sent to tier'); regression tests live in crates/filemeta/src/filemeta/version.rs; follow-up fix 726f3dc18 'accept empty remote version_id in tier recovery paths' (#4552)
+- For any diff touching tier or replication GET/DELETE against a remote S3 target, enumerate BOTH directions of the versionId contract and trace each: (a) remote version None/"" means the tier bucket is unversioned — the request must carry NO versionId parameter at all (not an empty one, not nil); (b) remote version Some(v) on a versioned target — the versionId MUST be sent, especially on version-purge deletes, or the delete lands on the wrong version / creates a delete marker instead of purging. Check whether the diff collapses these cases through a single Option/String conversion that loses the distinction.
+  - Where: crates/ecstore/src/services/tier/warm_backend*.rs, crates/ecstore/src/bucket/lifecycle/tier_sweeper.rs, replication code under crates/ecstore/src/bucket/
+  - Evidence: Invariant in AGENTS.md and docs/operations/tier-ilm-debugging.md; real bug fixed by 0fad35645 'send versionId on version-purge deletes to generic S3 targets' (#4401) — the versioned direction, and #4552 — the unversioned direction
+- If the diff changes xl.meta encoding (adds/reorders msgpack header fields, touches FileMeta::marshal_msg or codec.rs encode paths), attack downgrade and cross-vendor parse: encode an object with the new code and decode it with (a) the meta_ver<=3 read path and (b) the real-MinIO fixture tests from #4377. Then check the header signature: is it recomputed over the new bytes, or copied/hardcoded? MinIO validates it; a stale or zero signature makes MinIO reject the file. Finally verify XL_META_VERSION was not silently bumped — old RustFS/MinIO nodes reject meta_ver > 3 during a rolling upgrade.
+  - Where: crates/filemeta/src/filemeta.rs (XL_HEADER_VERSION/XL_META_VERSION, lines ~46-54), crates/filemeta/src/filemeta/codec.rs (check_xl2_v1, decode_xl_headers)
+  - Evidence: Real bug 073bc9675 'compute header signature instead of hardcoding zero' (#4343); format contract pinned in docs/architecture/minio-file-format-compat.md (write meta_ver 3, read <=3, XL2 magic); parity fixtures from a91d9cefc (#4377)
+- If the diff touches xl.meta / FileInfo decode (into_fileinfo, version parsing, part arrays), construct hostile foreign input: a MinIO- or corruption-shaped msgpack with a parts-count that disagrees with the etags/sizes array lengths, missing optional fields, and a meta_ver 2 object with legacy checksum. Trace whether the new code indexes past an array, panics, or fabricates default values instead of returning a decode error. Run the pinned legacy fixtures (test_issue_2265_legacy_meta_v2_object_compatibility, test_issue_2288) plus the #4377 real-MinIO xl.meta parse tests against the diff.
+  - Where: crates/filemeta/src/ (fileinfo.rs, filemeta.rs, filemeta/codec.rs, filemeta/version.rs)
+  - Evidence: Real bug 7efacbdf9 'validate part array lengths in into_fileinfo' (#4382); legacy meta_ver 2 regression fixtures at crates/filemeta/src/filemeta.rs (~:1130-:1174) cited by docs/architecture/minio-file-format-compat.md
+- If the diff 'corrects' a formula, constant, or layout that is a byte-for-byte MinIO port (shard-size math, bitrot hash interleaving, erasure distribution, inline-data prefix), treat the correction itself as the bug: verify against legacy on-disk data before accepting. Concretely: run crates/ecstore/tests/legacy_bitrot_read_test.rs and the ECA-18 pinning tests; check whether existing objects written by old RustFS or MinIO still verify byte-for-byte. Known trap examples: bitrot_shard_file_size's bare return for non-streaming algorithms is CORRECT MinIO whole-file behavior, and the 32-byte prefix on inline data is the HighwayHash256 bitrot hash, not corruption.
+  - Where: crates/ecstore/src/erasure/coding/bitrot.rs, crates/ecstore/src/io_support/bitrot.rs, crates/filemeta/src/fileinfo.rs
+  - Evidence: f96314a1d 'pin streaming-only bitrot layout invariant (ECA-18)' (#4553) — audit explicitly decided NOT to change the formula because it breaks legacy interop; #4377 proved the inline-data prefix is the bitrot hash
+- For any diff that copies object metadata into an S3-client-visible surface (GET/HEAD response headers, notification event userMetadata, ListObjects/replication payloads, copy-object metadata directives), construct an object carrying internal keys under BOTH prefixes and in non-canonical casing ('X-Minio-Internal-Compression') and confirm every one is stripped via is_internal_key (which is case-insensitive) — not by an exact-match filter on one prefix. Leaked internal keys are an API-semantics break and an information leak.
+  - Where: rustfs/src/storage/, crates/notify/, replication and copy_object paths in crates/ecstore/
+  - Evidence: Real bug cf8929189 'strip rustfs/minio internal metadata from event userMetadata' (#4419); is_internal_key contract in crates/utils/src/http/metadata_compat.rs
+- If the diff touches crates/protos (node.proto, models.fbs) or internode RPC request/response structs, attack the rolling-upgrade interleaving: an old node sends a message without the new field to a new node, and a new node sends the extended message to an old node. Verify proto field numbers are only appended (never reused/renumbered), FlatBuffers tables are only extended at the end, and that an absent new field decodes to a safe default on the receiving side — 'safe' meaning it must not be interpreted as success/authorization (RPC errors fail closed) and must not flip a quorum decision.
+  - Where: crates/protos/ (node.proto, models.fbs, generated/), gRPC transport and dispatch in the internode layer
+  - Evidence: 6f613317f 'optimize gRPC transport' (#4337) shows the wire layer churns; security advisory 68cw fixed by PR#4402 established RPC fail-closed as a repo rule (see .agents/skills/security-advisory-lessons)
+- For S3 handler diffs, replay the request shapes real clients actually send, not just the canonical one: mc and aws-sdk differ on path normalization (root '//' ListBuckets), virtual-host vs path style, and header casing. Then attack every pagination boundary the diff touches: request exactly max-keys/max-uploads/max-parts items and verify the response returns exactly N (not N+1), sets IsTruncated correctly, and yields a NextMarker/KeyMarker that resumes without skipping or duplicating — construct the N+1st-item case explicitly.
+  - Where: rustfs/src/storage/ S3 handlers, listing paths in crates/ecstore/src/store/ and set_disk/
+  - Evidence: Real bugs 511ad31ba 'normalize root double-slash ListBuckets requests' (#4336) and fefa70b31 'stop ListMultipartUploads from returning one upload past max-uploads' (#4447); docs/architecture/s3-compatibility-matrix.md is the compat source of truth
+- If the diff changes bucket-metadata (.metadata.bin) or IAM/config parsing structs, run it against the real MinIO RELEASE.2025-07-23 fixtures: the msgpack blob uses PascalCase field names, so any serde rename, field-type change, or derive tweak silently drops MinIO-written fields instead of erroring. Verify parse_all_configs still loads all ten config types from the fixture without loss, and that drop-in migration still decrypts MinIO-encrypted IAM/server config rather than treating ciphertext as corrupt.
+  - Where: crates/ecstore/src/bucket/metadata*, crates/ecstore/src/bucket/migration.rs, IAM/config load paths in crates/iam/ and crates/config/
+  - Evidence: Fixture parity test parses_real_minio_bucket_metadata_blob_without_loss from a91d9cefc (#4377); real bug 717cdd2ab 'decrypt MinIO IAM & server config on drop-in migration' (#4358); format matrix in docs/architecture/minio-file-format-compat.md
+- If the diff adds a compatibility shim, legacy fallback, wrapper, or old-endpoint alias (grep the diff for 'legacy', 'fallback', 'compat', 'deprecated'), verify two things: (1) it carries a RUSTFS_COMPAT_TODO(<task-id>) marker with an exact removal condition and a matching entry in the register — an unmarked shim becomes permanent dead weight; (2) the fallback's default direction is safe for old data: e.g. a new decode path must fall back to the legacy decode for old objects by default, not gate legacy reads behind an opt-in flag that makes existing data unreadable after upgrade.
+  - Where: Anywhere in the diff; register at docs/architecture/compat-cleanup-register.md; recent example: allow_inplace_legacy_fallback flag in the ecstore erasure codec streaming path
+  - Evidence: docs/architecture/compat-cleanup-register.md review checklist; d232a46b4 wired legacy decode prefetch behind a default-OFF gate while keeping legacy reads working (#4542), with arity fallout fixed in 05890d6e2 (#4573)
+
+Null report example: "Attacked dual-key metadata writes/removals against MinIO-only-key objects, nil/empty transitioned-versionID paths to the tier, xl.meta encode against meta_ver<=3 decoders and the #4377 real-MinIO fixtures, and proto field-number evolution for old-node/new-node RPC — no compatibility break found."
+
+### Performance reviewer
+
+- For every `.clone()` the diff adds or moves onto a per-request/per-object path, open the cloned type and count heap fields (String, Vec, HashMap, Bytes). If >5 heap fields or it contains an EC block buffer, construct the cost: N concurrent PUTs x M objects -> N*M deep copies per second. Demand Arc-wrapping of heavy fields or pass-by-reference; also flag new `String` allocations in header/path/signature parsing where `&str`/`Cow<str>` suffices.
+  - Where: crates/ecstore/src/set_disk/**, crates/ecstore/src/store*.rs, rustfs/src/storage/, crates/filemeta/, request handlers in rustfs/src/
+  - Evidence: crates/ecstore/AGENTS.md 'Allocation Discipline in Hot Paths' (no Clone on >5-heap-field structs, Arc for large buffers, &str/Cow for temporary computations); .agents/skills/rust-code-quality/SKILL.md ranks 'unnecessary clone in hot path' as P1 must-fix
+- For every new sync_all/sync_data/fdatasync/flush/File::sync call in the diff, trace the call chain to DurabilityMode / RUSTFS_DRIVE_SYNC_ENABLE resolution (crates/ecstore/src/disk/local.rs:291 DurabilityMode, :347 resolve_durability_mode) and to per-bucket durability overrides. Construct the run where the operator sets mode=none (or legacy RUSTFS_DRIVE_SYNC_ENABLE=false) and the new fsync still fires — that is an ungated durability cost and a regression on 4KiB writes.
+  - Where: crates/ecstore/src/disk/local.rs, crates/ecstore/src/bucket/durability.rs, crates/ecstore/src/set_disk/** (rename_data/commit paths), any crate doing tokio::fs or std::fs writes
+  - Evidence: #4221 fsync work caused a measured -10% 4KiB write regression (#814 investigation), later gated; durability modes added in eaff17cad (#4397), per-bucket tier overrides in 13e48d93a (#4407); 2df315baf (#4493) shows even ancestor-dir fsyncs are routed through the gate
+- Attack blocking-work placement from both directions: (a) find new synchronous fs calls, hashing, or EC encode/decode executed directly on an async runtime thread without spawn_blocking/block_in_place — construct the stall (a slow disk blocks a worker thread and every task queued on it); (b) find new code that splits one logical disk operation into multiple spawn_blocking hops per object — each hop is a threadpool round-trip, so K hops x N objects multiplies latency. Demand the author justify the placement with the size of the work, not habit.
+  - Where: crates/ecstore/src/disk/local.rs, crates/ecstore/src/erasure_coding/, crates/ecstore/src/bitrot/, crates/rio/
+  - Evidence: 608ab14d7 (#4554, HP-12) folded metadata open+fstat+read into a single spawn_blocking because per-op hops were measurably slow; 8fc637fb1 (#4484) moved the short EC encode inline because block_in_place cost exceeded the work — direction depends on measured work size
+- For each lock acquisition the diff adds or relocates, mark the guard's live range and list every .await and disk/RPC call inside it. Construct the contention interleaving: N concurrent requests serialize on the guard while the holder waits on IO; for namespace/multipart commit locks, compute worst-case hold time (fsync + rename per disk) against the lock's timeout. Also diff the acquisition order against other paths taking the same locks (ABBA).
+  - Where: crates/ecstore/src/set_disk/** (commit/rename paths), crates/lock/, crates/audit/ registry, any RwLock/Mutex in per-request paths
+  - Evidence: crates/ecstore/AGENTS.md 'Lock Ordering'; c0d5f938f (#4421) fixed a real ABBA deadlock between registry and stream_cancellers; #4370 history: fsync-heavy serial cross-disk commits held a lock long enough to blow test timeouts (#4370)
+- Trace exactly what executes inside the PUT commit critical section (under the object write lock, between tmp write and rename_data completion) before vs after the diff. Any newly added work there — cleanup, extra stat, additional rename, O_DIRECT write, logging — is an attack target: construct the per-PUT latency delta and demand it be moved off the critical section or parallelized across disks.
+  - Where: crates/ecstore/src/set_disk/ops/*, crates/ecstore/src/disk/local.rs rename_data path
+  - Evidence: Three real optimizations removed exactly this class of regression: e7cc719c1 (#4389) moved speculative PUT-tail tmp cleanup off the hot path, 92c8c6db7 (#4411) moved O_DIRECT shard-writes off the commit critical section, 651ccac13 (#4487) parallelized tmp xl.meta write and shard fdatasync on commit
+- Find any new loop in a batch API that performs a per-item stat/read/RPC sequentially. Construct the concrete blowup: a 1000-key DeleteObjects or a full listing page -> 1000 serial round-trips added by the diff. Demand either a gate (skip when not needed) or bounded parallelism; for startup/load paths, check for accidental O(n^2) (re-scanning the full list per item).
+  - Where: crates/ecstore/src/store_delete*.rs / batch object APIs, listing/metacache paths, crates/iam/ store loading, crates/heal/
+  - Evidence: a413729b1 (#4398) had to gate and parallelize the DeleteObjects per-object stat fanout after it shipped serial; 16a91c35e (#4537) fixed O(n^2) IAM startup load by chunking — both were diff-introduced fanouts of this exact shape
+- For every buffer the diff allocates on the encode/decode/shard path, check: is it sized with with_capacity to the EC-expanded block (not the logical size, not default-grown)? Does it copy into a fresh Vec where Bytes::slice/clone (refcount) or the io-core buffer pool would avoid the copy? Does the diff read hash and data in separate passes where one pass suffices? Construct the per-block byte-copy count before vs after. If the diff touches the io-core pool, verify gauge accounting still balances.
+  - Where: crates/ecstore/src/erasure_coding/, crates/ecstore/src/bitrot/, crates/io-core/src/pool.rs, crates/rio/
+  - Evidence: 92bf55ce6 (#4396) fixed a real regression by right-sizing BytesMut encode ingest capacity to the EC-expanded block; 47bee8b31 (#4475) merged bitrot hash+data into one read pass; 7fa3d0d4b (#4534) shows pool gauge accounting is easy to drift when touching buffer reuse
+- For every logging or instrumentation statement the diff adds, classify the call site frequency: per-request, per-object, per-shard, or per-block. Anything info!/warn!/error! at per-object frequency or higher is a finding — construct the flood (one listing under client cancellation, one 10k-object heal) and count emitted lines. New metrics/timers on the data path must be feature-gated, not always-on. Run scripts/check_logging_guardrails.sh on the diff.
+  - Where: any per-request/per-object code, especially crates/ecstore listing and heal loops, rustfs/src/storage/ handlers; scripts/check_logging_guardrails.sh
+  - Evidence: .agents/skills/rustfs-logging-governance/SKILL.md (trace level for hot-path/repetitive success events); d25ddb0e1 (#4372) fixed real listing-cancellation error-log noise; hotpath instrumentation is deliberately feature-gated (3f13d098b #4394, f262fcfce #4541 HP-14)
+- Count how many times the diff's request path parses or fetches the same metadata: xl.meta/FileMeta decoded more than once per object, bucket metadata (metadata_sys) re-fetched inside a per-object loop, or the dual x-rustfs-internal/x-minio-internal key lookup re-run repeatedly on the same map. Construct the per-request parse count before vs after; a second full FileMeta decode per GET is a finding.
+  - Where: crates/filemeta/, crates/ecstore/src/set_disk/** read paths, crates/ecstore/src/bucket/metadata_sys.rs, crates/utils/src/http/metadata_compat.rs
+  - Evidence: 608ab14d7 (#4554) exists because redundant metadata-read syscall sequences per object were measurable; CLAUDE.md dual-key metadata convention makes repeated get_bytes lookups an easy hidden double-parse
+- If the diff touches PUT/GET/commit/erasure paths and claims 'no perf impact', demand numbers, not assertion: run the criterion benches (cargo bench -p ecstore — comparison_benchmark, erasure_benchmark, rename_data_meta_benchmark, single_block_non_inline_benchmark per crates/ecstore/benches/) against origin/main, and for end-to-end paths the warp A/B relative-budget gate (scripts/run_hotpath_warp_ab.sh --baseline-ref origin/main, as .github/workflows/performance-ab.yml runs it). Probe specifically at 4KiB object size — that is where the last real regression hid.
+  - Where: crates/ecstore/benches/, .github/workflows/performance-ab.yml, scripts/run_hotpath_warp_ab.sh
+  - Evidence: crates/ecstore/AGENTS.md: 'Benchmark-sensitive changes should include measurable rationale'; performance-ab.yml (215747022 #4480) is the repo's own relative-budget gate; the #4221 regression was only visible at 4KiB writes (#814 bisect)
+
+Null report example: "Attacked the new rename_data commit-section work, durability-gate routing of the added fdatasync, guard live-range across the shard-write awaits, and per-object clone count in the PUT path; ran comparison_benchmark + rename_data_meta_benchmark vs origin/main (4KiB delta within noise) — no break found."
+
+### Test-coverage skeptic
+
+- For every behavior claim in the PR description, revert that hunk (git stash / manual undo of the changed lines) and name the exact test (`cargo test -p <crate> <test_name>`) that fails. If no test fails on revert, the behavior is untested — file a finding, not a note. Especially verify the test exercises the REAL production call path, not a lookalike helper.
+  - Where: All crates; highest value in crates/ecstore, rustfs/src/storage, crates/heal
+  - Evidence: AGENTS.md exit criterion 'Every behavior change has a test that fails without it'. Real bug: PR #4220 (ghost-directory cleanup) merged with green tests but its fix never executed on the real delete path — required follow-up rustfs#4307, backlog#798 stayed OPEN. The tests exercised a path the production flow never took.
+- Read each added/modified test and confirm it asserts the real outcome (returned value, stored bytes, error variant), not merely 'call succeeded' or 'no panic'. Flag any test whose only observable is that the function returned, and any `assert!(result.is_err())` that never checks WHICH error. Then check: does the test prove the exploit/failure form is denied, or only that the intended form still works?
+  - Where: crates/e2e_test (security_boundary_test.rs pattern), and every #[cfg(test)] module in the diff
+  - Evidence: Commit dee8e4e63 (#4466) had to rewrite 277 lines of crates/e2e_test/src/security_boundary_test.rs because 'security boundary tests' passed without asserting real outcomes. .agents/skills/rust-code-quality/SKILL.md checklist: 'Every test function has at least one assert!'; .agents/skills/security-advisory-lessons/SKILL.md: 'Does the test prove the exploit form is denied, or only that the intended form still works?'
+- When the diff adds a boolean/mode parameter or config flag, find the test that fails if the flag's effect is INVERTED inside the changed function. Tests that were mechanically updated to pass `false`/default at every call site assert nothing about the new behavior. Execute the check: flip the flag's branch in the source and confirm at least one test goes red for each branch.
+  - Where: crates/ecstore/src/set_disk/ (e.g. build_codec_streaming_part_reader), any function gaining a parameter
+  - Evidence: Commit 05890d6e2 (#4573): PR #4560 added a 15th param allow_inplace_legacy_fallback; the arity tests were fixed by passing `false` everywhere — they assert Err outcomes independent of the flag, so the fallback behavior itself has no revert-detecting test at those sites.
+- Mutation spot-check on error propagation: for each newly added `?`, `return Err`, or error-mapping line, mentally replace it with `Ok(default)`/ignore and ask which test fails. The swallowed-error bug class recurs in this repo and always ships with green tests — a fix that propagates errors needs a test that injects the failure (faulty disk, failed rename, dispatch error) and asserts the caller sees Err.
+  - Where: crates/ecstore (disk delete/rename, reduce_errs), crates/audit, crates/notify, crates/targets
+  - Evidence: Three recent fixes for the same class: f7d2b2563 (#4546, disk delete/rename failures swallowed), dbc628f16 (#4424, audit dispatch failures swallowed), 20d61c73b (#4551, reduce_errs leaking nil placeholder as dominant error). All existed while tests were green.
+- Any test touching GET/read/reconstruct/stream paths must assert the FULL body content and exact length against a known value, not status-ok or first-bytes. Construct the degraded-read case (missing/inconsistent shards forcing EC reconstruction) and assert byte-for-byte equality; a mid-stream failure that truncates the body passes every test that only checks headers or the first chunk.
+  - Where: crates/ecstore/src/set_disk/read.rs and ops/, crates/rio, crates/e2e_test GET scenarios
+  - Evidence: Known live bug: EC reconstruct-read verification failure mid-GET at set_disk read path silently truncates the body → client 'unexpected EOF'; version-independent, undetected by existing suites because none assert full-body integrity under shard inconsistency.
+- For on-disk / on-wire format changes (xl.meta, .metadata.bin, bitrot framing), reject round-trip-only tests: a struct serialized and deserialized by the same code under test cannot catch format drift. Demand the test parse a REAL captured fixture from crates/filemeta/tests/fixtures, crates/ecstore/tests/fixtures, or crates/rio-v2/tests/minio_fixture_lab — or capture a new one from a single-disk MinIO instance (RELEASE.2025-07-23 procedure from #4377).
+  - Where: crates/filemeta, crates/ecstore (headers, msgpack bucket metadata), crates/rio-v2, migration code
+  - Evidence: Commit 073bc9675 (#4343): filemeta header signature was hardcoded to zero — round-trip tests passed for months. Commit a91d9cefc (#4377) established the real-MinIO fixture convention (inline/versioned/multipart xl.meta, HighwayHash256-prefixed inline bodies) precisely because synthetic fixtures proved nothing about interop.
+- Attack new concurrency tests for flakiness-by-construction: grep the added tests for `sleep(`, fixed timeouts under ~30s on lock acquisition, and use of shared global state (disk registry, lock client, GLOBAL_*). Serialized cross-disk commits exceed small lock timeouts under full-suite CI disk load. If the test shares global state or saturates IO, it must join the `ecstore-serial-flaky` nextest test-group in .config/nextest.toml (note: serial_test's #[serial] does NOT work — nextest runs each test in its own process). Require readiness polling, never fixed sleeps.
+  - Where: crates/ecstore tests, crates/e2e_test, .config/nextest.toml
+  - Evidence: Commit 2dfa3d3c3 (#4370): concurrent_resend test flaked with Lock(Timeout 5s) on CI — six legitimate serialized cross-disk commits under IO pressure needed 30s. Commit 7c701d9f2 (#4558) created the nextest test-group after bucket_delete_* raced make_bucket into InsufficientWriteQuorum. 65849740f (#4213) deflaked global-state contamination. crates/e2e_test/AGENTS.md: 'readiness checks and explicit polling over fixed sleep-based timing'.
+- If the diff writes internal object metadata, run the dual-key mutation: delete the `x-minio-internal-<suffix>` write (keeping only `x-rustfs-internal-`) and check whether any test fails. Because `get_bytes` prefers the RustFS key, every read-back test stays green while MinIO interop is silently broken — coverage must include an assertion that BOTH keys are present in the stored metadata map.
+  - Where: crates/utils/src/http/metadata_compat.rs and all its callers in crates/ecstore and rustfs/src/storage
+  - Evidence: CLAUDE.md domain convention: metadata must be written under both x-rustfs-internal- and x-minio-internal- keys for MinIO interop; get_bytes prefers the RustFS key, making the MinIO-key half of the invariant invisible to read-back tests.
+- For changed quorum/version/UUID logic, name the tests covering the specific poison values: quorum−1 disks, nil UUID, absent vs empty vs nil-serialized UUID bytes, and remote-tier version_id of None/"" (unversioned tier bucket → no versionId sent). Mutation check: remove a `.filter(|u| !u.is_nil())` guard from the diff and confirm a test fails; if none does, the nil-UUID class is uncovered.
+  - Where: crates/ecstore (tier recovery, heal, quorum paths), crates/filemeta, code reading UUIDs from xl.meta metadata
+  - Evidence: Commit 726f3dc18 (#4552) fixed rejection of empty remote version_id in tier recovery. CLAUDE.md invariant: absent/empty/nil UUID all mean 'no value', not Uuid::nil(). docs/operations/tier-ilm-debugging.md: None/"" tier version means unversioned bucket. df9cbc4ed (#4427): unvalidated distribution values caused shuffle index panic — edge values reached production untested.
+- For any pagination/limit/truncation change, construct the exact-boundary test: result count == max (page exactly full), max+1, and a delimiter re-fold that lands precisely on the page boundary — assert both the item count AND the is_truncated/continuation marker. Off-by-one at the page boundary is a recurring shipped bug here.
+  - Where: crates/ecstore listing paths (list_objects, ListMultipartUploads, metacache), S3 handlers in rustfs/src/storage
+  - Evidence: Two shipped boundary bugs: fefa70b31 (#4447) ListMultipartUploads returned one upload past max-uploads; d91f4d455 (#4538) delimiter re-fold of a full page lost the truncation flag. Both survived existing tests because no test pinned n == max exactly.
+- Green `cargo test -p <crate>` on the touched crate is not a coverage verdict for the diff's test code itself: run `cargo clippy --all-targets -p <crate>` and a workspace-wide test BUILD (`cargo check --workspace --all-targets` at minimum) before accepting the tests as evidence. Test-only code that doesn't compile workspace-wide or fails clippy has repeatedly broken main and masked whether tests ran at all.
+  - Where: All crates; especially concurrent-branch merges into crates/ecstore
+  - Evidence: #4322 broke main because only cargo test ran (field_reassign_with_default is clippy-only). b06f3df6b (#4441) and 05890d6e2 (#4573): test code broke the workspace test build (E0061) on main after textually-clean merges, failing CI for every open PR.
+
+Null report example: "Attacked revert-detection for all 3 claimed behaviors (each has a named test that fails on revert), flag-inversion on the new fallback parameter (both branches covered in codec_streaming tests), full-body assertions on the changed GET path, and n==max pagination boundary — no coverage gap found."
+
+## Sources and maintenance
+
+Probes are distilled from shipped bugs in git history (commit/PR references
+above), GitHub security advisories (see the security-advisory-lessons
+skill), scoped `AGENTS.md` rules, and invariants under `docs/architecture/`
+and `docs/operations/`. Line numbers drift; when a cited location no longer
+matches, trust the invariant and re-locate the code. When a new bug class
+ships, add a probe with its evidence here rather than growing the policy
+section in `AGENTS.md`.

.agents/skills/adversarial-validation/references/compatibility.md

@@ -1,24 +0,0 @@
-# Compatibility Lens
-
-- Internal metadata uses `metadata_compat` helpers for dual RustFS/MinIO keys,
-  including mixed casing and removal of both twins.
-- Binary UUID metadata treats absent, empty, and nil as no value. Unversioned
-  remote tiers receive no `versionId`; versioned purge requests retain the real
-  version ID.
-- `xl.meta` changes preserve supported header/meta versions, recompute
-  signatures, decode legacy fixtures, and remain readable by old RustFS/MinIO.
-- Foreign/corrupt metadata validates parallel array lengths and missing fields;
-  it returns a decode error rather than indexing, panicking, or fabricating data.
-- Do not “correct” byte-for-byte MinIO ports without legacy fixture evidence.
-  Bitrot framing, shard math, distribution, and inline prefixes are contracts.
-- Client-visible metadata/events strip both internal prefixes
-  case-insensitively.
-- Proto fields are appended, never reused/renumbered; FlatBuffers tables extend
-  compatibly and absent new fields fail closed where authorization/quorum is
-  involved.
-- Replay real client request shapes and exact pagination boundaries for S3
-  handler changes.
-- Bucket metadata/IAM/config parsing remains compatible with pinned real MinIO
-  fixtures and encrypted migration data.
-- Compatibility shims use `RUSTFS_COMPAT_TODO(<task-id>)`, have a removal
-  condition, and default toward reading old data safely.

.agents/skills/adversarial-validation/references/concurrency-durability.md

@@ -1,23 +0,0 @@
-# Concurrency and Durability Lens
-
-- For every changed lock, enumerate overlapping lock sets and construct the
-  ABBA interleaving. Multiple-lock order must be documented and consistent.
-- Mark guard lifetimes and every `.await`, disk, and RPC call inside them.
-  Estimate contention and timeout behavior under concurrent requests.
-- Object commits remain fenced if the distributed lock is lost after shard
-  writes and before metadata rename.
-- For write/rename changes, trace `write tmp -> sync tmp -> rename -> sync parent
-  -> sync required ancestors`; simulate a crash after each step and honor the
-  configured durability gate.
-- Multi-disk fan-out counts every result. Quorum-minus-one cannot become success;
-  heal remains best-effort per target where that is the established contract.
-- At every new cancellable await between mutation and cleanup/commit, drop the
-  future and inspect leftover files, counters, permits, and replay state.
-- Multipart operations on the same upload ID are serialized where required;
-  abort/complete/list races cannot delete parts before durable commit.
-- Post-commit cleanup is best-effort, retry-safe, and cannot fail an already
-  committed write or delete the last surviving copy.
-- Persisted read-modify-write uses serialization/CAS. Queue replay is crash-safe
-  and duplicate delivery has an idempotency contract.
-- Streaming reconstruction failures after partial output surface as errors, not
-  successful EOF.

.agents/skills/adversarial-validation/references/correctness.md

@@ -1,29 +0,0 @@
-# Correctness Lens
-
-Attack the changed behavior, not every subsystem in the repository.
-
-- Trace new error paths to the caller. Inject the ignored/wildcard variants and
-  verify they cannot become success, not-found, or a plausible default.
-- Exercise zero/empty/missing, maximum, and exact-boundary inputs for every
-  changed count, size, index, page limit, or optional value.
-- For aggregation/quorum changes, test exactly quorum and quorum-minus-one with
-  mixed disk errors and nil/placeholder entries.
-- For listing/pagination, test `n == max`, `n == max + 1`, delimiter folding,
-  continuation markers, and object/prefix name collisions.
-- For EC/read/streaming changes, inject failure after partial output and verify
-  the client receives an error rather than a clean truncated body. Assert exact
-  bytes and length.
-- For multipart/object commits, fail before/after rename and cleanup; committed
-  data must remain readable and pre-commit cleanup must not destroy parts.
-- For version/index ordering, test `len - 1`, `len`, equal timestamps, missing
-  versions, and deterministic tie-breaking.
-- For directory-object behavior, trace `__XLDIR__` at the store layer; branches
-  below the layer that sees trailing slashes are dead.
-- For binary UUID metadata, absent, empty, and nil all mean no value. Never send
-  nil/empty `versionId` to an unversioned tier.
-- For agent rules/skill routers, test a trigger matrix covering ordinary
-  inquiry, low-risk implementation, explicit review, high-risk code, PR
-  creation, release, and post-PR monitoring. Each case must select only the
-  intended workflow and retain required safety/authorization boundaries.
-
-Null verdicts name only the probes relevant to the diff.

.agents/skills/adversarial-validation/references/performance.md

@@ -1,20 +0,0 @@
-# Performance Lens
-
-- For added clones/allocations on request/object/block paths, quantify copied
-  data and frequency. Recommend borrowing, move, `Bytes`/`Arc`, `Cow`, or
-  capacity reservation only for a concrete repeated cost.
-- Route every new sync/flush through the durability-mode and bucket override
-  gates; mode `none` must not pay the new fsync.
-- Keep blocking filesystem/CPU work off async runtime threads, but do not split
-  one small operation into many `spawn_blocking` round trips.
-- Measure lock hold time across I/O and compare acquisition order for ABBA.
-- Keep cleanup, extra stat/rename, and diagnostics out of the PUT commit critical
-  section when they need not be there.
-- Detect per-item serial I/O/RPC in batch APIs and accidental quadratic scans;
-  use a gate or bounded concurrency when the concrete fan-out warrants it.
-- Count buffer growth and byte copies in EC/bitrot paths; preserve pool gauge
-  balance and avoid repeated metadata decode/fetch per object.
-- Repetitive success logs stay at `trace`; metrics/instrumentation on hot paths
-  require an existing gate.
-- Claims of no impact on PUT/GET/commit/erasure paths need relevant benchmark or
-  A/B evidence, especially for 4 KiB objects.

.agents/skills/adversarial-validation/references/security.md

@@ -1,31 +0,0 @@
-# Security Lens
-
-Use `security-advisory-lessons` only for a dedicated advisory/security audit.
-For an ordinary matched diff, attack these boundaries:
-
-- Admin routes: route registration, whitelist, handler authn, and the exact
-  `AdminAction` must agree. Read-only diagnostics still require admin authz.
-- IAM/service accounts: treat parent, claims, keys, groups, status, and policy
-  names as attacker-controlled; prove ownership/root authority before writes.
-- Protocol frontends: every changed/sibling command authorizes the matching S3
-  action before reaching storage.
-- Secrets/signatures: use constant-time comparison, normalize public failures,
-  keep RPC/root/STS keys independent, and fail closed when secrets are absent.
-- RPC: bind signatures to the exact method/path and timestamp; reject replay,
-  stale, malformed, truncated, and invalid-enum payloads without panic.
-- Paths/object/archive entries: reject traversal, absolute/platform escapes,
-  and normalization differences between authz and storage.
-- Copy/multipart/presigned POST: enforce source, destination, version-aware
-  actions, copy-source conditions, and every signed policy condition.
-- Logging/errors: never expose credentials, tokens, expected signatures, raw
-  secret-bearing input, or merged configs—including via `Debug` and parse errors.
-- Untrusted serde: reject unknown fields where compatible and validate
-  security-critical defaults/ranges before numeric conversion.
-- SSE/browser/CORS/trusted proxy: inspect stored ciphertext and wrapper order;
-  isolate user content; never reflect credentialed arbitrary origins or trust
-  forwarded identity from direct clients.
-- Object Lock: unreadable/fabricated/unparsable metadata fails closed across
-  foreground, lifecycle, scanner, and force-delete paths.
-
-Security findings distinguish unauthenticated compromise from a
-low-privileged authenticated bypass.

.agents/skills/adversarial-validation/references/simplicity.md

@@ -1,22 +0,0 @@
-# Simplicity Lens
-
-- Compare the production diff with the smallest equivalent local edit. Fewer
-  lines alone are not evidence; the replacement must preserve correctness,
-  compatibility, readability, and real boundaries.
-- Search the touched crate, domain owner, `crates/utils`, `crates/common`, and
-  relevant dependencies for each new helper, constant, wrapper, or fixture.
-- Reject forced reuse when normalization, error, deadline, or durability
-  semantics differ.
-- Require a concrete trigger for every new defensive branch. Keep boundary
-  checks for disk/RPC/version data and checks immediately before destructive
-  actions.
-- Flag one-caller helpers only when they merely forward or split a short linear
-  flow without adding domain naming, invariant isolation, or useful context.
-- Ensure a replacement removes the superseded in-scope path or keeps one
-  canonical core behind a documented compatibility adapter.
-- Remove narration/change-history comments; preserve concise safety, lock,
-  durability, and compatibility invariants.
-- Treat tests, fixtures, generated code, and documentation separately from
-  production growth. Do not optimize away meaningful regression coverage.
-
-A finding must include a concrete smaller design, not a style preference.

.agents/skills/adversarial-validation/references/test-coverage.md

@@ -1,24 +0,0 @@
-# Test-Coverage Lens
-
-- For every behavior claim, name the focused test/check that fails if the
-  changed hunk is reverted. If none is practical, require the reason and
-  residual risk.
-- Confirm tests exercise the real production path and assert returned values,
-  exact bytes, stored state, or the specific error variant—not only success,
-  `is_err()`, or no panic.
-- For new flags/modes, verify each branch and ask which test fails if the branch
-  is inverted.
-- For new error propagation, inject the failure and assert the caller observes
-  it; mentally replacing `?`/`return Err` with success must break a test.
-- Streaming GET tests assert the complete body and length under degraded reads.
-- Disk/wire-format tests use pinned foreign/legacy fixtures; same-code
-  round-trips are insufficient for compatibility.
-- Concurrency tests use readiness polling, isolate global state, and avoid fixed
-  sleeps or unrealistically short timeouts. Use nextest groups when process-level
-  serialization is required.
-- Internal metadata tests assert both RustFS and MinIO keys, not only read-back
-  through a helper that prefers one key.
-- Boundary companions are distinct coverage: `n == max` vs `max + 1`, and
-  absent vs empty vs nil UUID.
-- A focused test proves only the targets/features it builds. Add compilation or
-  Clippy only for uncovered changed targets.

.agents/skills/arch-checks/SKILL.md

@@ -10,16 +10,10 @@ never weaken a check to get green.
 
 ## `check_layer_dependencies.sh` — layer DAG in `rustfs/src`
 
-Enforces `composition (server, startup/init) → interface (admin,
-storage/ecfs, storage/s3_api) → app → infra`; no upward imports. Server source
-files are composition roots, while imports of their exported HTTP contracts
-are classified as interface dependencies. Known legacy violations live in
+Enforces `interface (admin, storage/ecfs, storage/s3_api) → app → infra`; no
+upward imports. Known legacy violations live in
 `scripts/layer-dependency-baseline.txt`.
 
-Dedicated `*_test.rs` and `tests/` modules are outside this production guard.
-Inline `#[cfg(test)]` imports remain checked under their source file's layer;
-move architecture-crossing test scaffolding into a dedicated test module.
-
 - **New violation**: restructure your change so the dependency points
   downward (move the shared type/function to the lower layer).
 - **You legitimately removed a baseline entry**: run

.agents/skills/code-change-verification/SKILL.md

@@ -1,12 +1,11 @@
 ---
 name: code-change-verification
-description: Review a commit, PR, or merged patch when the user requests ordinary code-change verification. Do not combine with adversarial-validation; use that skill instead for explicitly adversarial, substantial, or high-risk RustFS reviews.
+description: Verify code changes by identifying correctness, regression, security, and performance risks from diffs or patches, then produce prioritized findings with file/line evidence and concrete fixes. Use when reviewing commits, PRs, and merged patches before/after release.
 ---
 
 # Code Change Verification
 
-Use this skill for an ordinary requested review. If the root policy or user calls
-for adversarial validation, use `adversarial-validation` instead of running both.
+Use this skill to review code changes consistently before merge, before release, and during incident follow-up.
 
 ## Quick Start
 
@@ -44,7 +43,16 @@ for adversarial validation, use `adversarial-validation` instead of running both
 
 #### Rust-specific checks (apply to all Rust changes)
 
-Run the full checklist in [rust-code-quality](../rust-code-quality/SKILL.md) — the canonical Rust review checklist for the unwrap/casting/cloning/locking/recursion/error-type/serde/test rules and the reuse-and-necessity checks (duplicated helpers, defensive branches without a nameable trigger, redundant error wrapping). Do not restate those rules here; carry its P0–P3 ratings over unchanged and use this skill's output format.
+- **unwrap/expect in production**: Search changed files for `.unwrap()` and `.expect(` outside test modules. Every `unwrap()` in production code must have a justification comment or be replaced with `?`.
+- **Silent type truncation**: Search for `as u8/u16/u32/u64/usize/i8/i16/i32/i64/isize` casts. Every `as` cast must be justified; negative-to-unsigned and large-to-small are bugs by default. Use `try_into()` or explicit clamping.
+- **Unnecessary cloning**: Check `.clone()` calls in loops, per-request paths, and on structs with >5 heap-allocated fields. Consider `Arc`, references, or `Cow<str>`.
+- **Lock ordering**: If the change acquires multiple locks, verify the order matches all other call sites. Document the order in a comment.
+- **Locks across .await**: Flag any `tokio::sync::RwLock`/`Mutex` guard held across an `.await` point without bounded hold time.
+- **Recursion depth**: If the change adds or modifies a recursive function, verify it has a depth limit or uses iterative traversal with an explicit stack.
+- **Error types**: Flag `Result<_, String>`, `Box<dyn Error>`, and missing `Error::source()` implementations in public APIs.
+- **Test assertions**: Every test function must have at least one `assert!`. Flag tests that only call code without verifying results.
+- **println/eprintln**: Search changed files for `println!`/`eprintln!` outside test modules. Production code must use `tracing` macros.
+- **Serde safety**: Structs deserialized from untrusted input (S3 API, user config) should have `#[serde(deny_unknown_fields)]`.
 
 ### 4) Findings-first output
 - Order findings by severity:
@@ -80,3 +88,4 @@ Run the full checklist in [rust-code-quality](../rust-code-quality/SKILL.md) —
   - Impact: ...
   - Fix suggestion: ...
   - Validation: ...
+

.agents/skills/code-change-verification/agents/openai.yaml

@@ -1,4 +1,4 @@
 interface:
   display_name: "Code Change Verification"
   short_description: "Prioritize risks and verify code changes before merge."
-  default_prompt: "Use $code-change-verification for an ordinary requested diff review with prioritized findings."
+  default_prompt: "Inspect a patch or diff, identify correctness/security/regression risks, and return prioritized findings with file/line evidence and fixes."

.agents/skills/pr-creation-checker/SKILL.md

@@ -1,46 +1,94 @@
 ---
 name: pr-creation-checker
-description: Perform the final RustFS PR preflight and draft compliant English title/body metadata immediately before creating or updating a PR. Do not use during implementation or as a second general code review.
+description: Prepare PR-ready diffs by validating scope, checking required verification steps, drafting a compliant English PR title/body, and surfacing blockers before opening or updating a pull request in RustFS.
 ---
 
 # PR Creation Checker
 
-Use this skill only at the PR boundary. Reuse completed diff review and
-verification evidence; do not reread the repository or rerun equivalent checks.
+Use this skill before `gh pr create`, before `gh pr edit`, or when reviewing whether a branch is ready for PR.
 
-## Preflight
+## Read sources of truth first
 
-1. Confirm the branch is based on current `origin/main` and contains only the
-   intended task diff.
-2. Inspect `git diff --stat`, `git diff --check`, and changed file names for
-   secrets, logs, generated artifacts, or unrelated edits.
-3. Confirm the checks selected by root `AGENTS.md` passed on the final diff.
-   Do not replace focused behavioral tests with a generic gate or rerun checks
-   already covered by an unchanged umbrella run.
-4. Read `.github/pull_request_template.md`. Consult `Makefile`, `.config/make/`,
-   or CI only when the required command/current gate is uncertain.
-5. Return `BLOCKED` for an unclean scope, missing required evidence, failed
-   required checks, or non-compliant metadata.
+- Read `AGENTS.md`.
+- Read `.github/pull_request_template.md`.
+- Use `Makefile` and `.config/make/` for local quality commands.
+- Use `.github/workflows/ci.yml` for CI expectations.
+- Do not restate long command matrices or template sections from memory when the files exist.
 
-## Metadata
+## Workflow
 
-- Title: English Conventional Commit, at most 72 characters, with no tool
-  prefix.
-- Body: English, exact template headings, `N/A` where needed, concise rationale,
-  actual verification commands, and material risks/rollback notes.
-- Use repository-relative paths; never include local absolute paths.
-- Keep prose paragraphs on one logical line and never include the literal
-  sequence `\n`.
-- Use a temporary body file with `gh pr create --body-file` or
-  `gh pr edit --body-file`; never pass multiline Markdown inline.
+1. Collect PR context
+- Confirm base branch, current branch, change goal, and scope.
+- Confirm whether the task is: draft a new PR, update an existing PR, or preflight-check readiness.
+- Confirm whether the branch includes only intended changes.
 
-## Output
+2. Inspect change scope
+- Review the diff and summarize what changed.
+- Call out unrelated edits, generated artifacts, logs, or secrets as blockers.
+- Mark risky areas explicitly: auth, storage, config, network, migrations, breaking changes.
+- Scan the diff for newly added string literals and confirm whether they duplicate values already defined as constants/enums/typed wrappers in the same module or shared modules.
+- Treat introducing a new hardcoded literal where a project constant already exists as a likely regression risk; require either a refactor to reuse the constant or an explicit exception explanation in the PR body.
 
-- Status: `READY` or `BLOCKED`.
-- Title.
-- Complete PR body.
-- Verification commands and results.
-- Risks or `N/A`.
+3. Verify readiness requirements
+- Require `make pre-commit` before marking PRs ready when the diff changes Rust code, product behavior, CI behavior, runtime configuration, security-sensitive logic, migrations, storage, auth, networking, or other high-risk paths.
+- For documentation-only, agent-instruction-only, or local developer-tooling-only changes, allow focused verification instead of `make pre-commit` when it directly validates the changed surface.
+- For focused verification, explain why the full gate was not run and list the scope-specific commands in the PR body.
+- If `make` is unavailable, use the equivalent commands from `.config/make/`.
+- Add scope-specific verification commands when the changed area needs more than the baseline.
+- If required checks fail, stop and return `BLOCKED`.
 
-Immediately before the GitHub write, repeat only the five preflight checks above
-against the final head.
+4. Draft PR metadata
+- Write the PR title in English using Conventional Commits and keep it within 72 characters.
+- If a generic PR workflow suggests a different title format, ignore it and follow the repository rule instead.
+- In RustFS, do not use tool-specific prefixes such as `[codex]` when the repository requires Conventional Commits.
+- Keep the PR body in English.
+- Use the exact section headings from `.github/pull_request_template.md`.
+- Fill non-applicable sections with `N/A`.
+- Include verification commands in the PR description.
+- Do not include local filesystem paths in the PR body unless the user explicitly asks for them.
+- Prefer repo-relative paths, command names, and concise summaries over machine-specific paths such as `/Users/...`.
+
+5. Prepare reviewer context
+- Summarize why the change exists.
+- Summarize what was verified.
+- Call out risks, rollout notes, config impact, and rollback notes when applicable.
+- Mention assumptions or missing context instead of guessing.
+
+6. Prepare CLI-safe output
+- When proposing `gh pr create` or `gh pr edit`, use `--body-file`, never inline `--body` for multiline markdown.
+- Return a ready-to-save PR body plus a short title.
+- If not ready, return blockers first and list the minimum steps needed to unblock.
+
+## Output format
+
+### Status
+- `READY` or `BLOCKED`
+
+### Title
+- `<type>(<scope>): <summary>`
+
+### PR Body
+- Reproduce the repository template headings exactly.
+- Fill every section.
+- Omit local absolute paths unless explicitly required.
+
+### Verification
+- List each command run.
+- State pass/fail.
+
+### Risks
+- List breaking changes, config changes, migration impact, or `N/A`.
+
+## Blocker rules
+
+- Return `BLOCKED` if a code, behavior, CI, runtime configuration, security-sensitive, migration, storage, auth, networking, or other high-risk change has not passed `make pre-commit`.
+- Return `BLOCKED` if a documentation-only, agent-instruction-only, or local developer-tooling-only change lacks focused verification for the changed surface.
+- Return `BLOCKED` if the diff contains unrelated changes that are not acknowledged.
+- Return `BLOCKED` if required template sections are missing.
+- Return `BLOCKED` if the title/body is not in English.
+- Return `BLOCKED` if the title does not follow the repository's Conventional Commit rule.
+- Return `BLOCKED` if the diff introduces string literals that should use existing constants but did not.
+
+## Reference
+
+- Use [pr-readiness-checklist.md](references/pr-readiness-checklist.md) for a short final pass before opening or editing the PR.

.agents/skills/pr-creation-checker/agents/openai.yaml

@@ -1,4 +1,4 @@
 interface:
   display_name: "PR Creation Checker"
   short_description: "Draft RustFS-ready PRs with checks, template, and blockers."
-  default_prompt: "Use $pr-creation-checker for final PR preflight and compliant English title/body metadata."
+  default_prompt: "Inspect a branch or diff, verify required PR checks, and produce a compliant English PR title/body plus blockers or readiness status."

.agents/skills/pr-creation-checker/references/pr-readiness-checklist.md

@@ -0,0 +1,16 @@
+# PR Readiness Checklist
+
+- Confirm the branch is based on current `main`.
+- Confirm the diff matches the stated scope.
+- Confirm no secrets, logs, temp files, or unrelated refactors are included.
+- Confirm `make pre-commit` passed for code, behavior, CI, runtime configuration, security-sensitive, migration, storage, auth, networking, or other high-risk changes.
+- For documentation-only, agent-instruction-only, or local developer-tooling-only changes, confirm focused verification covered the changed surface and the PR body explains why the full gate was not run.
+- Confirm extra verification commands are listed for risky changes.
+- Confirm the PR title uses Conventional Commits and stays within 72 characters.
+- Confirm the PR title does not use tool-specific prefixes such as `[codex]`.
+- Confirm the PR body is in English.
+- Confirm the PR body keeps the exact headings from `.github/pull_request_template.md`.
+- Confirm non-applicable sections are filled with `N/A`.
+- Confirm the PR body does not include local absolute paths unless explicitly required.
+- Confirm multiline GitHub CLI commands use `--body-file`.
+- Confirm new hardcoded string literals were not introduced for values already represented by existing constants/enums (including protocol labels, error identifiers, headers, and metric names), or record a justified exception.

.agents/skills/rust-code-quality/SKILL.md

@@ -1,83 +1,79 @@
 ---
 name: rust-code-quality
-description: Run a focused Rust quality review when the user requests one, when reviewing a Rust PR/commit, or when another selected review workflow delegates Rust-specific checks. Do not auto-load for every implementation edit.
+description: Enforce Rust-specific code quality rules on every code change. Use before merge to catch unwrap abuse, silent truncation, unnecessary cloning, lock ordering violations, recursion risks, and error type anti-patterns.
 ---
 
 # Rust Code Quality Gate
 
-Use this skill for a dedicated Rust review to cover rules that `cargo clippy`
-does not catch.
+Use this skill on every Rust code change to enforce quality rules that `cargo clippy` does not catch.
 
 ## Quick Start
 
 1. Identify changed `.rs` files.
 2. Run automated checks on changed files.
 3. Run manual review checklist on the diff.
-4. Resolve or rebut every finding with evidence; P0/P1 findings cannot be deferred.
+4. Report findings; block merge if P0/P1 issues exist.
 
 ## Automated Checks
 
-Use these searches to find candidates in changed `.rs` files. Inspect syntax,
-`#[cfg(test)]` scope, and the changed hunk before reporting a finding; text
-filters do not reliably distinguish production code from tests.
+Run these on every changed `.rs` file (excluding test modules):
 
 ```bash
-# 1. unwrap/expect candidates
-rg -n '\.unwrap\(\)|\.expect\(' <changed-files>
+# 1. unwrap/expect in production code
+rg -n '\.unwrap\(\)|\.expect\(' <changed-files> | grep -v '#\[cfg(test)\]' | grep -v 'test' | grep -v 'bench'
 
 # 2. Silent type truncation via `as` cast
 rg -n ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' <changed-files>
 
 # 3. String as error type
-rg -n 'Result<.*String>' <changed-files>
+rg -n 'Result<.*String>' <changed-files> | grep -v test
 
 # 4. Box<dyn Error> in public APIs
-rg -n 'Box<dyn.*Error' <changed-files>
+rg -n 'Box<dyn.*Error' <changed-files> | grep -v test
 
 # 5. println/eprintln in production
-rg -n 'println!\|eprintln!' <changed-files>
+rg -n 'println!\|eprintln!' <changed-files> | grep -v test
 
 # 6. Ordering::Relaxed usage (verify each is intentional)
 rg -n 'Ordering::Relaxed' <changed-files>
-
-# 7. Default substituted for a possibly-required value (judge each: is the value optional by domain?)
-rg -n 'unwrap_or_default\(\)|unwrap_or\(' <changed-files>
 ```
 
 ## Manual Review Checklist
 
-For the Rust diff under review, verify:
+For every Rust code change, verify:
 
 ### Error Handling
-- [ ] Every production `unwrap()` or `expect()` is infallible by type or a checked invariant; explain only non-obvious invariants, using an existing type, a useful `expect` message, or a concise comment
+- [ ] No `unwrap()` or `expect()` in production code without justification comment
 - [ ] No `Result<_, String>` in public API signatures
-- [ ] Public library APIs use domain errors unless deliberate error erasure at a boundary is part of the contract
+- [ ] No `Box<dyn Error>` in public trait/struct methods
 - [ ] `Error::source()` is overridden when inner error is stored
-- [ ] Error messages are actionable without exposing secret input
+- [ ] Error messages are actionable (what failed, with what input)
 
 ### Type Safety
 - [ ] No silent `as` truncation (negative→unsigned, large→small)
-- [ ] Fallible numeric conversions use `TryFrom`/`try_into()` and return a typed error; clamp or saturate only when the domain explicitly requires it
-- [ ] Floating-point to integer conversion validates finiteness, sign, and range before conversion
+- [ ] `try_into()` or explicit clamping used for numeric conversions
+- [ ] No `f64 as usize` without prior clamping
 
 ### Concurrency
-- [ ] Lock acquisition order is documented when multiple locks are used, and matches every other call site taking any overlapping subset (ABBA check)
-- [ ] No `tokio::sync` lock guard (read or write) held across `.await` without bounded hold time — long-lived read guards wedge writers (#4195)
-- [ ] Atomic read-modify-write uses the direct `fetch_*` operation when possible; use `compare_exchange` only for conditional updates
+- [ ] Lock acquisition order is documented when multiple locks are used
+- [ ] No `tokio::sync` write guards held across `.await` without bounded hold time
+- [ ] Concurrent counters use `compare_exchange` loops, not load-then-store
 - [ ] `std::sync::Mutex` in async context is held only briefly, never across `.await`
 
 ### Memory and Performance
-- [ ] On an identified hot path, report cloning or allocation only with a concrete per-request/per-object cost or benchmark signal
-- [ ] Prefer borrowing, moving, `Bytes`/`Arc`, or capacity reservation only when it reduces that cost without obscuring ownership or APIs
+- [ ] No `.clone()` on structs with >5 heap-allocated fields in hot paths
+- [ ] `HashMap::with_capacity()` / `Vec::with_capacity()` used when size is known
+- [ ] Large buffers wrapped in `Arc` rather than cloned
+- [ ] Temporary string computations use `&str` or `Cow<str>` instead of `String`
 
 ### Recursion Safety
-- [ ] Recursion over untrusted, persisted, or otherwise unbounded input has a depth limit or uses iterative traversal
+- [ ] Recursive functions have a depth limit or use iterative traversal
 - [ ] Tree/cache traversals handle corrupted/cyclic input safely
 
 ### Testing
-- [ ] Tests have an observable failure criterion; delegated assertions, `#[should_panic]`, snapshot/property checks, and meaningful `Result` failures do not need a redundant `assert!`
-- [ ] Use `expect` only when its message improves failure diagnosis; do not add boilerplate to self-evident test setup
-- [ ] Test volume and line count are never treated as production-code growth
+- [ ] Every test function has at least one `assert!`
+- [ ] Tests use `.expect("context")` not bare `.unwrap()`
+- [ ] No `println!`/`eprintln!` in production code (use `tracing`)
 
 ### Serde
 - [ ] Structs from untrusted input have `#[serde(deny_unknown_fields)]`
@@ -88,19 +84,12 @@ For the Rust diff under review, verify:
 - [ ] No camelCase statics or Hungarian notation
 - [ ] New string literals don't duplicate existing constants
 
-### Reuse and Necessity
-- [ ] No new helper duplicates `crates/utils`, `crates/common`, the touched crate, the likely domain-owning crate, a relevant direct dependency, or plain std/tokio behavior; reused helpers match the call site's semantics
-- [ ] No branch without a nameable concrete trigger; no re-validation of what a validated upstream layer on the same path already guarantees (Cross-Cutting Domain Invariant patterns and pre-destructive-action re-checks are load-bearing — keep them)
-- [ ] Error context attached once where actionable, not re-wrapped at every hop; no typed→generic error conversion below aggregation/quorum layers
-- [ ] Comments avoid narration and change history while completely stating non-obvious lock, `SAFETY`, durability, compatibility, and unwrap invariants
-- [ ] No near-duplicate test pinning the same code path and poison-value class as an existing test (boundary companions — n==max vs max+1, absent/empty/nil UUID — are never near-duplicates)
-
 ## Severity Classification
 
-- **P0 (Block merge)**: demonstrated data loss, security breach, remote crash, or deadlock
-- **P1 (Must fix)**: concrete correctness, compatibility, or material hot-path regression
-- **P2 (Should fix)**: avoidable duplication or maintainability issue with a concrete simpler replacement
-- **P3 (Nice to fix)**: local style or clarity issue with no behavioral risk
+- **P0 (Block merge)**: `unwrap()` in request hot path, silent truncation on user input, lock ordering violation, recursion without depth limit
+- **P1 (Must fix)**: `Result<_, String>` in public API, unnecessary clone in hot path, `Box<dyn Error>` in trait method
+- **P2 (Should fix)**: Missing `assert!` in test, `println!` in production, missing `with_capacity`
+- **P3 (Nice to fix)**: Naming convention violation, missing doc comment, `as_ptr()` vs `Arc::ptr_eq`
 
 ## Output Template
 
@@ -108,10 +97,10 @@ For the Rust diff under review, verify:
 ## Rust Code Quality Report
 
 ### Automated Scan
-- unwrap/expect candidates inspected: N
-- numeric-cast candidates inspected: N
-- error-type candidates inspected: N
-- output-macro candidates inspected: N
+- unwrap/expect in production: N found
+- as casts: N found
+- String errors: N found
+- println/eprintln: N found
 
 ### Findings
 - [P1] `path:line` — description

.agents/skills/rust-code-quality/references/checklist.md

@@ -0,0 +1,52 @@
+# Rust Code Quality Checklist
+
+Use this as a quick pre-merge checklist for every Rust code change.
+
+## Critical (P0 — block merge)
+
+| Check | Command |
+|-------|---------|
+| No `unwrap()` in request/storage hot path | `rg '\.unwrap\(\)' <files> \| grep -v test` |
+| No `as` truncation on user input | `rg ' as (u32\|usize\|i32)' <files>` |
+| Lock order consistent across call sites | Manual: trace all lock acquisitions |
+| Recursive functions have depth limit | Manual: check for `max_depth` or iterative pattern |
+| No `panic!`/`unwrap_or_else(panic!)` in production | `rg 'panic!\|unwrap_or_else.*panic' <files> \| grep -v test` |
+
+## High (P1 — must fix)
+
+| Check | Command |
+|-------|---------|
+| No `Result<_, String>` in public API | `rg 'Result<.*String>' <files> \| grep -v test` |
+| No `Box<dyn Error>` in public trait | `rg 'Box<dyn.*Error' <files> \| grep -v test` |
+| No unnecessary `.clone()` in hot path | Manual: check loops and per-request paths |
+| `Error::source()` implemented when inner error stored | Manual: check `impl Error` |
+| No `eprintln!`/`println!` in production | `rg 'println!\|eprintln!' <files> \| grep -v test` |
+
+## Medium (P2 — should fix)
+
+| Check | Command |
+|-------|---------|
+| Tests have assertions | Manual: check for `assert` in test functions |
+| `HashMap`/`Vec` use `with_capacity` when size known | Manual: check `::new()` in loops |
+| No `#![allow(dead_code)]` at crate root | `rg 'allow.dead_code' <files> \| grep 'lib.rs'` |
+| Serde structs from untrusted input have `deny_unknown_fields` | Manual: check `#[derive(Deserialize)]` |
+
+## Low (P3 — nice to fix)
+
+| Check | Command |
+|-------|---------|
+| No camelCase statics | `rg 'static ref [a-z]' <files>` |
+| `Arc::ptr_eq` instead of `as_ptr + ptr::eq` | `rg 'as_ptr\|ptr::eq' <files>` |
+| Public functions have doc comments | `rg 'pub fn' <files> \| grep -v '///'` |
+
+## Quick One-Liner
+
+```bash
+# Run all automated checks on changed files
+CHANGED=$(git diff --name-only HEAD~1 -- '*.rs' | grep -v test | grep -v bench)
+echo "=== unwrap/expect ===" && rg -c '\.unwrap\(\)|\.expect\(' $CHANGED 2>/dev/null
+echo "=== as casts ===" && rg -c ' as (u8|u16|u32|u64|usize|i8|i16|i32|i64|isize)\b' $CHANGED 2>/dev/null
+echo "=== String errors ===" && rg -c 'Result<.*String>' $CHANGED 2>/dev/null
+echo "=== println ===" && rg -c 'println!|eprintln!' $CHANGED 2>/dev/null
+echo "=== Ordering::Relaxed ===" && rg -c 'Ordering::Relaxed' $CHANGED 2>/dev/null
+```

.agents/skills/rustfs-logging-governance/SKILL.md

@@ -1,34 +1,107 @@
 ---
 name: rustfs-logging-governance
-description: Add or review RustFS `tracing` events with the repository field shape, level policy, privacy boundaries, and guardrails. Use when a change adds or edits a tracing macro/instrumentation site or the logging guardrail script.
+description: Standardize and review RustFS logging with structured `tracing` events, lower noise on hot paths, preserve security-sensitive diagnostics, and extend guardrails to prevent legacy logging patterns from returning. Use when editing or reviewing RustFS logs, startup/config diagnostics, cloud metadata logs, request validation logs, or `scripts/check_logging_guardrails.sh`.
 ---
 
 # RustFS Logging Governance
 
-Apply this skill only to changed logging sites; do not turn a local log edit into
-a broad logging cleanup.
+Use this skill when RustFS logging needs to be added, cleaned up, reviewed, or protected against regressions.
 
-## Workflow
+## Quick Start
 
-1. Read the changed function/module context and classify the site as lifecycle,
-   request/hot path, fallback, external fetch, or summary.
-2. Match neighboring structured events and reuse existing `EVENT_*`,
-   `LOG_COMPONENT_*`, and `LOG_SUBSYSTEM_*` constants.
-3. Put stable fields first (`event`, `component`, `subsystem`, `state`/`result`,
-   then context) and a short label last.
-4. Select the level by operational meaning:
-   - `error`: behavior/security-affecting failure;
-   - `warn`: degraded/fallback/operator-actionable state;
-   - `info`: low-frequency lifecycle/mode change;
-   - `debug`: targeted diagnostics;
-   - `trace`: repetitive request/object/shard success paths.
-5. Never log secrets, tokens, auth headers, credential payloads, raw
-   attacker-controlled bodies, or merged config dumps. Error strings and
-   `Debug` output are log surfaces too.
-6. Prefer one aggregate summary over inventories or startup banners.
-7. Run `./scripts/check_logging_guardrails.sh` and the checks selected by root
-   `AGENTS.md`.
+1. Identify the files whose logs are changing.
+2. Scan current `tracing` or `log` macros before editing.
+3. Convert sentence-style logs to short event-style logs.
+4. Demote hot-path success logs unless operators truly need them at `info`.
+5. Preserve failure, fallback, and security-relevant diagnostics.
+6. Update `scripts/check_logging_guardrails.sh` when a broad cleanup removes a legacy pattern class.
+7. Validate with formatting, targeted checks/tests, and the logging guardrail script.
 
-Read [logging-governance.md](references/logging-governance.md) only for a broad
-logging audit, event-model migration, or guardrail expansion. Ordinary single-
-site edits do not require the full workspace scope map.
+## Core Workflow
+
+### 1. Scope the logging surface
+
+- Read the changed module in full before touching log lines.
+- Classify the log site:
+  - lifecycle/startup
+  - request or validation path
+  - background loop or hot path
+  - fallback/degraded behavior
+  - cloud metadata or external fetch path
+  - metrics/config summary
+- Do not rewrite business logic to make logging easier.
+
+### 2. Use the RustFS event shape
+
+- Prefer fields first, message second.
+- Use short labels, not prose paragraphs.
+- Default field shape:
+  - `event`
+  - `component`
+  - `subsystem`
+  - `state` or `result`
+  - key context fields
+- Reuse stable field names and avoid inventing near-duplicates.
+
+See `references/logging-governance.md` for the event model, level policy, and anti-pattern list.
+
+### 3. Choose the right level
+
+- `error`: operation failure that affects behavior or security guarantees.
+- `warn`: degraded path, fallback, suspicious input, or operator-actionable misconfiguration.
+- `info`: low-frequency lifecycle or mode selection.
+- `debug`: targeted diagnostics and low-volume detail.
+- `trace`: hot-path and repetitive success-path events.
+
+When in doubt, lower the verbosity of normal success paths and keep structured detail in fields.
+
+### 4. Preserve security and privacy boundaries
+
+- Do not log secrets, tokens, auth headers, raw credential payloads, or merged config dumps.
+- Avoid logging raw forwarded headers or full trusted network inventories above `debug`.
+- Keep warning/error logs useful without echoing attacker-controlled payloads unnecessarily.
+
+### 5. Keep summaries aggregated
+
+- Replace multi-line startup banners or checklist logs with one structured event.
+- If metrics already express a concept, avoid duplicating it with many `info!` lines.
+- Prefer counts, modes, and sources over inventories unless debug detail is truly needed.
+
+### 6. Update guardrails when needed
+
+- Broad logging cleanup should usually extend `scripts/check_logging_guardrails.sh`.
+- Add forbidden patterns only for styles the repo has intentionally retired:
+  - sentence-style lifecycle logs
+  - noisy hot-path `info!`
+  - checklist-style summary logs
+  - legacy fallback wording that has been replaced by structured fields
+- Keep guardrails concrete and grep-friendly.
+
+### 7. Validate manually
+
+Use the smallest relevant set:
+
+```bash
+cargo fmt --all --check
+./scripts/check_logging_guardrails.sh
+cargo check -p <affected-crate>
+cargo test -p <affected-crate>
+```
+
+For broader Rust changes, add:
+
+```bash
+./scripts/check_unsafe_code_allowances.sh
+./scripts/check_architecture_migration_rules.sh
+cargo clippy -p <affected-crates> --all-targets -- -D warnings
+```
+
+## RustFS-Specific Notes
+
+- The durable RustFS logging direction is `event + component + subsystem + state/result + key context fields`.
+- `crates/concurrency` and `crates/trusted-proxies` are examples of this style for lifecycle, fallback, and cloud metadata logs.
+- `scripts/check_logging_guardrails.sh` is the enforcement point for preventing removed log styles from returning.
+
+## References
+
+- Read `references/logging-governance.md` when you need the detailed field set, anti-pattern examples, or guardrail update checklist.

.agents/skills/rustfs-logging-governance/references/logging-governance.md

@@ -1,62 +1,285 @@
-# Logging Audit and Migration Reference
+# RustFS Logging Governance Reference
 
-Read this reference only for a broad logging audit, an event-model migration,
-or a change to `scripts/check_logging_guardrails.sh`. Use `Cargo.toml` for the
-current workspace/crate list instead of maintaining one here.
+## Workspace Scope Map
 
-## Audit by Operational Role
+Use `Cargo.toml` `[workspace].members` as the source of truth for crate membership. When doing a broad logging sweep, classify crates by operational role so logs stay consistent within each role.
 
-- Server/protocol/admin: lifecycle, authorization failures, request boundaries,
-  and degraded subsystems; avoid normal request success at `info`.
-- Storage/heal/scanner/capacity: integrity failures and aggregate lifecycle;
-  avoid per-object, per-shard, and folder iteration noise.
-- IAM/policy/credentials/KMS/crypto: safe identifiers and enforcement results;
-  never emit secrets, claims, payloads, or expected authenticators.
-- Notify/audit/targets: target lifecycle and batch/backpressure summaries; avoid
-  per-event success logs.
-- Locking/concurrency/I/O foundations: contention anomalies and state changes;
-  prefer metrics for high-frequency worker/permit signals.
-- Shared type/schema crates: log at the operational caller boundary unless the
-  crate itself owns the failure context.
+### Core Server And Request Handling
 
-## Event Shape
+- `rustfs`
+  - Role: top-level server, startup, auth, admin wiring, S3 request handling.
+  - Logging focus: startup lifecycle, config summaries, authn/authz failures, protocol entrypoints, degraded subsystems.
+- `crates/protocols`
+  - Role: protocol integrations such as FTP, SFTP, WebDAV, and related server-side protocol layers.
+  - Logging focus: listener lifecycle, per-protocol enablement/disablement, request bridge failures.
+- `crates/madmin`
+  - Role: admin API contracts and management interfaces.
+  - Logging focus: admin action boundaries, validation failures, compatibility warnings.
+- `crates/trusted-proxies`
+  - Role: forwarded IP trust, proxy chain validation, cloud metadata sources.
+  - Logging focus: direct/trusted/fallback decisions, degraded metadata fetches, aggregated config summaries.
+- `crates/keystone`
+  - Role: Keystone auth integration.
+  - Logging focus: integration enablement, upstream auth failures, config safety without credential leakage.
 
-Prefer stable fields in this order when available:
+### Storage, Healing, And Data Plane
 
-1. `event`
-2. `component`
-3. `subsystem`
-4. `state` or `result`
-5. stable context such as mode, duration, reason, counts, safe identifiers, or
-   capacity/permit values
-6. short message label
+- `crates/ecstore`
+  - Role: erasure-coded storage implementation and peer/store initialization.
+  - Logging focus: disk/peer lifecycle, storage fallback, object I/O failures, avoid per-object noise.
+- `crates/heal`
+  - Role: healing orchestration and repair workflows.
+  - Logging focus: scheduler lifecycle, repair decisions, backlog or skipped work summaries, avoid repetitive task spam at `info`.
+- `crates/scanner`
+  - Role: data integrity scanning and health monitoring.
+  - Logging focus: scan lifecycle, compaction/deep-heal transitions, lag/backlog, noisy folder iteration should stay at `debug/trace`.
+- `crates/object-capacity`
+  - Role: capacity scan and refresh core.
+  - Logging focus: refresh lifecycle, degraded capacity sources, aggregate stats rather than per-object chatter.
+- `crates/filemeta`
+  - Role: file metadata parsing and helpers.
+  - Logging focus: parse failures, schema/format mismatch, avoid dumping raw metadata payloads.
+- `crates/storage-api`
+  - Role: storage contracts and shared data plane interfaces.
+  - Logging focus: contract mismatch and boundary diagnostics, usually low-volume.
+- `crates/checksums`
+  - Role: checksum helpers and validation.
+  - Logging focus: integrity failures and compatibility mismatches, not per-chunk success logs.
+- `crates/zip`
+  - Role: ZIP handling and compression helpers.
+  - Logging focus: parse/extract failures, archive path safety issues, avoid verbose file-by-file success logs.
 
-Reuse the module's constants and neighboring field names. Do not create aliases
-for the same concept.
+### Security, Identity, And Policy
 
-## Patterns to Retire
+- `crates/iam`
+  - Role: identity and access management.
+  - Logging focus: authz decision boundaries, imported payload safety, do not leak principals, secrets, or claims.
+- `crates/policy`
+  - Role: policy modeling and evaluation.
+  - Logging focus: deny/allow decision context, parser/validation failures, no raw secret-bearing request dumps.
+- `crates/credentials`
+  - Role: credential handling.
+  - Logging focus: never log secrets or tokens; only safe identifiers and redacted states.
+- `crates/kms`
+  - Role: key management service integration.
+  - Logging focus: init/health/fallback, key-source availability, never log key material.
+- `crates/crypto`
+  - Role: cryptographic helpers and security primitives.
+  - Logging focus: only algorithm or mode state, not plaintext, ciphertext, or secret-derived material.
+- `crates/security-governance`
+  - Role: security governance contracts.
+  - Logging focus: policy/state transitions and enforcement diagnostics.
+- `crates/signer`
+  - Role: request signing helpers.
+  - Logging focus: signature validation failures without expected-signature leakage.
 
-- sentence-style lifecycle announcements;
-- startup banners and checklist lines;
-- repetitive success logs at `info`/`debug`;
-- raw inventories when an aggregate count is sufficient;
-- fallback prose with values embedded in the message;
-- `?value`/`Debug` output for credential-bearing or attacker-controlled data;
-- logging a parse input when the malformed input may itself be a secret.
+### Notifications, Audit, And Targets
 
-## Guardrail Changes
+- `crates/notify`
+  - Role: notification dispatch, runtime facade, notifier implementations.
+  - Logging focus: target lifecycle, dispatch summaries, stream lag/backpressure, avoid per-event success spam.
+- `crates/audit`
+  - Role: audit target fan-out and audit pipeline management.
+  - Logging focus: pipeline lifecycle, target availability, batch dispatch summaries, avoid noisy "started successfully" prose.
+- `crates/targets`
+  - Role: target-specific configuration and utilities used by fan-out style systems.
+  - Logging focus: target selection, config validation, per-target degraded state.
+- `crates/s3-types`
+  - Role: S3 event and type definitions.
+  - Logging focus: usually minimal; keep logging at integration boundaries rather than low-level type crates.
+- `crates/s3-ops`
+  - Role: S3 operation definitions and mapping.
+  - Logging focus: mapping/contract failures, unsupported combinations, not normal-path request spam.
 
-When expanding `scripts/check_logging_guardrails.sh`:
+### Concurrency, Locking, And Runtime Foundations
 
-1. Add only files/patterns intentionally migrated in the same change.
-2. Keep patterns concrete and grep-friendly.
-3. Do not encode a style that remains valid elsewhere as a global ban.
-4. Run the guardrail script and the root validation tier.
-5. Treat the script as a floor; manually verify level, field shape, and privacy.
+- `crates/concurrency`
+  - Role: timeout, locking, backpressure, and I/O scheduling facade.
+  - Logging focus: lifecycle transitions and degraded states, not high-frequency worker/permit churn at `info`.
+- `crates/lock`
+  - Role: distributed locking implementation.
+  - Logging focus: lock lifecycle, contention anomalies, lock ordering or timeout diagnostics.
+- `crates/tls-runtime`
+  - Role: shared TLS runtime foundation.
+  - Logging focus: certificate lifecycle, reload/fallback, validation failures without sensitive dumps.
+- `crates/obs`
+  - Role: observability helpers.
+  - Logging focus: this crate shapes other crates' telemetry conventions; avoid recursive or redundant summaries.
+- `crates/io-core`
+  - Role: zero-copy I/O core primitives.
+  - Logging focus: keep very sparse; prefer metrics unless failures are actionable.
+- `crates/io-metrics`
+  - Role: I/O metrics collection.
+  - Logging focus: typically minimal; metrics should carry the hot-path signal.
+- `crates/rio`
+  - Role: Rust I/O utility layer.
+  - Logging focus: compatibility or runtime boundary failures, not fast-path internals.
+- `crates/rio-v2`
+  - Role: next-generation I/O compatibility layer.
+  - Logging focus: migration/feature-mode differences and degraded fallback between I/O paths.
+- `crates/utils`
+  - Role: shared helpers.
+  - Logging focus: usually avoid direct logging in generic helpers unless the helper is itself an operational boundary.
+- `crates/common`
+  - Role: shared data structures and helpers.
+  - Logging focus: same principle as `utils`; prefer callers to log context-rich events.
+- `crates/config`
+  - Role: configuration management.
+  - Logging focus: config source, fallback, validation, and summary aggregation; avoid dumping merged configs.
+- `crates/data-usage`
+  - Role: shared data usage models and algorithms.
+  - Logging focus: refresh lifecycle, summary stats, and degraded reads.
 
-Useful search seeds for the changed surface:
+### Schema, Contracts, And API Support
+
+- `crates/protos`
+  - Role: protobuf definitions.
+  - Logging focus: usually none inside the crate; emit logs at decode/use boundaries.
+- `crates/extension-schema`
+  - Role: extension schema contracts.
+  - Logging focus: schema validation and compatibility mismatches.
+- `crates/s3select-api`
+  - Role: S3 Select API interfaces.
+  - Logging focus: request validation and unsupported feature boundaries.
+- `crates/s3select-query`
+  - Role: S3 Select query engine.
+  - Logging focus: query parse/planning/execution failures, avoid row-level spam.
+- `crates/protocols`
+  - Role: non-S3 protocol support.
+  - Logging focus: see core server section; keep per-request verbosity below `info`.
+
+### Testing And Non-Production Crates
+
+- `crates/e2e_test`
+  - Role: end-to-end tests.
+  - Logging focus: test clarity matters more than production governance, but avoid copying test-only logging style into production crates.
+
+## Current Guardrail Coverage Map
+
+`scripts/check_logging_guardrails.sh` currently enforces retired patterns in these high-signal areas:
+
+- `rustfs/src/main.rs`
+- `rustfs/src/startup_iam.rs`
+- `rustfs/src/auth.rs`
+- `rustfs/src/protocols/client.rs`
+- `crates/audit/src/pipeline.rs`
+- `crates/audit/src/system.rs`
+- `crates/audit/src/global.rs`
+- `crates/notify/src/config_manager.rs`
+- `crates/notify/src/runtime_facade.rs`
+- `crates/notify/src/notifier.rs`
+- `crates/ecstore/src/store/peer.rs`
+- `crates/ecstore/src/store/init.rs`
+- `crates/ecstore/src/tier/tier.rs`
+- `crates/concurrency/src/workers.rs`
+- `crates/concurrency/src/manager.rs`
+- `crates/concurrency/src/lock.rs`
+- `crates/concurrency/src/deadlock.rs`
+- `crates/trusted-proxies/src/global.rs`
+- `crates/trusted-proxies/src/config/loader.rs`
+- `crates/trusted-proxies/src/proxy/metrics.rs`
+- `crates/trusted-proxies/src/proxy/validator.rs`
+- `crates/trusted-proxies/src/proxy/chain.rs`
+- `crates/trusted-proxies/src/middleware/service.rs`
+- `crates/trusted-proxies/src/cloud/detector.rs`
+- `crates/trusted-proxies/src/cloud/ranges.rs`
+- `crates/trusted-proxies/src/cloud/metadata/aws.rs`
+- `crates/trusted-proxies/src/cloud/metadata/azure.rs`
+- `crates/trusted-proxies/src/cloud/metadata/gcp.rs`
+
+When expanding coverage, prefer crates with:
+
+- repeated sentence-style lifecycle logs
+- high-frequency success-path `info!`
+- startup/config checklist banners
+- security-sensitive fallback wording
+- external fetch/retry/fallback flows
+
+That typically means the next broad candidates are `rustfs`, `crates/notify`, `crates/audit`, `crates/targets`, `crates/heal`, and `crates/scanner`.
+
+## Event Model
+
+Prefer this structure when the fields are available:
+
+- `event`
+- `component`
+- `subsystem`
+- `state` or `result`
+- stable context fields such as:
+  - `enabled`
+  - `implementation`
+  - `validation_mode`
+  - `peer_ip`
+  - `client_ip`
+  - `proxy_hops`
+  - `duration_ms`
+  - `fallback`
+  - `reason`
+  - `range_count`
+  - `hold_time_ms`
+  - `available_slots`
+  - `total_slots`
+  - `permits_in_use`
+
+## Level Policy
+
+- `error`: the operation fails and callers or security guarantees are affected.
+- `warn`: a degraded path, fallback, suspicious request, or operator-actionable config issue occurs.
+- `info`: a low-frequency lifecycle or mode transition occurs.
+- `debug`: useful diagnostics exist but normal operators do not need them all the time.
+- `trace`: hot-path and repetitive success-path details occur.
+
+## Preferred Patterns
+
+- Use a short message label:
+  - `"trusted proxy validation failed"`
+  - `"concurrency manager state changed"`
+  - `"trusted proxy cloud metadata loaded"`
+- Put key meaning into fields, not only the message text.
+- Aggregate config or metrics summaries into one log event.
+
+## Retired Patterns
+
+These should usually be removed or replaced:
+
+- Sentence-style lifecycle logs:
+  - `info!("Concurrency manager stopped")`
+  - `info!("Trusted Proxies module initialized")`
+- Checklist or banner logs:
+  - `info!("=== Application Configuration ===")`
+  - `info!("Available metrics:")`
+- Hot-path noise:
+  - `info!("worker take, {}", *available)`
+  - `debug!("Proxy validation successful in {:?}", duration)`
+- Legacy fallback prose:
+  - `"Request from private network but not trusted: ..."`
+  - `"Cloud metadata fetching is disabled"`
+
+## Guardrail Update Checklist
+
+When extending `scripts/check_logging_guardrails.sh`:
+
+1. Add the touched files to `checked_files`.
+2. Add only legacy patterns that have been intentionally retired.
+3. Keep patterns literal and grep-friendly.
+4. Run the guardrail script after changes.
+5. Avoid adding patterns for logs that are still valid elsewhere in the repo.
+
+## Validation Checklist
+
+For logging-only changes:
 
 ```bash
-rg -n 'error!|warn!|info!|debug!|trace!|#\[instrument' <changed-paths>
-rg -n '\?[^,)]|secret|token|credential|authorization|merged_config' <changed-paths>
+cargo fmt --all --check
+./scripts/check_logging_guardrails.sh
+cargo check -p <affected-crate>
+cargo test -p <affected-crate>
+```
+
+For broader Rust changes:
+
+```bash
+./scripts/check_unsafe_code_allowances.sh
+./scripts/check_architecture_migration_rules.sh
+cargo clippy -p <affected-crates> --all-targets -- -D warnings
 ```

.agents/skills/rustfs-release-publish/SKILL.md

@@ -1,233 +0,0 @@
----
-name: rustfs-release-publish
-description: "Run the end-to-end RustFS console gate, version bump, preview validation, and final-tag publication pipeline. Use only when the user explicitly asks to release or publish a RustFS version (发版/发布)."
----
-# RustFS Release Publish (preview-validated pipeline)
-
-This skill orchestrates a full release. It wraps `rustfs-release-version-bump` (which only edits version files and opens the PR) with a mandatory preview-tag validation loop before the final tag is published.
-
-Core design: **version files never carry a `-preview.N` suffix**. The preview suffix exists only in tag names. A preview tag creates a visible GitHub Release marked Prerelease and uploads versioned assets, but it never becomes GitHub Latest and never updates `*-latest`, `latest.json`, R2, Docker, or Helm channels. This works because the binary self-reports the git tag it was built from (`build::TAG` via shadow_rs, see `rustfs/src/config/cli.rs` `SHORT_VERSION`), and `build.yml` derives artifact names and preview classification from the tag name — Cargo.toml's version is only a no-tag fallback. Therefore the preview tag and the final tag can (and MUST) point at the exact same commit: what you validated is byte-for-byte the source that ships.
-
-Pipeline shape:
-
-```
-check console main against its latest Release
-  -> if ahead: publish console -> wait for Release asset + latest API
-  -> bump RustFS version files to <target> (final version, ONE commit) -> merge
-  -> tag <preview-tag> at that commit -> CI green
-  -> verify preview Release assets -> run binary locally + console checks
-  -> validate with latest rc client
-  -> tag <target> at the SAME commit (zero delta) -> re-verify CI/release
-```
-
-On validation failure: fix lands on main via normal PR (version files are already at `<target>`, no new bump PR), then tag `<preview-tag N+1>` at the new main commit and restart from Phase 2.
-
-## Required inputs
-
-- Final target version, for example `1.0.0-beta.10`.
-- Preview iteration `N` (default: next unused preview tag for that target; check with `git tag -l '<target>-preview.*'` after `git fetch --tags`).
-
-If the target version is missing or ambiguous, stop and ask before doing anything (see the semver gate below).
-
-## Semver gate — confirm the target version before touching anything
-
-Versions follow [SemVer 2.0.0](https://semver.org/). Precedence reminder:
-
-```
-1.0.0-alpha < 1.0.0-alpha.1 < 1.0.0-beta.2 < 1.0.0-beta.11 < 1.0.0-rc.1 < 1.0.0 < 1.0.1 < 1.1.0 < 2.0.0
-```
-
-Numeric prerelease identifiers compare numerically (`beta.9 < beta.10`), not lexically — see [semver.org spec item 11](https://semver.org/#spec-item-11). Preview tags are internal validation tags layered on top of the target's prerelease channel — they are never themselves a deliverable version and never appear in version files.
-
-Rules:
-
-- A request like "发个版" / "release the next version" without an exact version string is ALWAYS ambiguous. Derive the current latest tag (`git tag --sort=-v:refname | head`), then ask the user to choose via AskUserQuestion with concrete candidates, e.g. from `1.0.0-beta.10`: next prerelease `1.0.0-beta.11`, promote to `1.0.0-rc.1`, promote to stable `1.0.0`. Never guess between these — they have very different meanings (channel promotion vs. iteration) and different CI classification consequences.
-- After a stable `X.Y.Z` exists, the next version must state which component bumps: patch `X.Y.(Z+1)` for fixes only, minor `X.(Y+1).0` for backward-compatible features, major `(X+1).0.0` for breaking changes. If the user names a bump type but not a number, compute it from the latest stable tag and echo the exact resulting version back for confirmation.
-- Echo the final confirmed version string verbatim in your first status report; every later phase must use exactly that string. If at any point the user's wording and the confirmed version diverge, stop and re-confirm.
-
-## Preview tag naming
-
-- Use `<target>-preview.N` for every target, e.g. `1.0.0-beta.10-preview.3` or `1.1.0-preview.1`.
-- The canonical suffix is exactly `-preview.<digits>`. `build.yml` recognizes it before alpha/beta/rc classification and routes it to the preview-only path; any other tag containing `-preview` fails closed instead of being treated as a release.
-- A preview Release MUST be published with `isPrerelease=true` and `isLatest=false`. Any `*-latest` preview asset or preview-triggered `latest.json`, R2, Docker, or Helm publication is a pipeline failure.
-
-## Hard rules
-
-- Version files (Cargo.toml, Cargo.lock, README, flake.nix, Chart.yaml, rustfs.spec) are bumped ONCE, directly to `<target>`. Never write a `-preview.N` suffix into any version file. If `rustfs-release-version-bump` is ever asked for a `-preview` version, that is a pipeline bug — stop.
-- Preview Release assets are versioned and intentionally visible on the Releases page. Do not label them Latest or use them to update any latest distribution channel.
-- Tags have no `v` prefix. Always annotated: `git tag -a <tag> -m "Release <tag>"`.
-- The final tag MUST point at exactly `PREVIEW_HASH` — the commit the validated preview tag points at. Never tag current `main` HEAD (commits merged after validation are unvalidated), and never create an extra version-bump commit between preview and final.
-- When a previous deliverable exists, GitHub Release notes for the preview and final tags MUST use it as their shared comparison baseline: the most recently published non-preview Release before the target. Internal `-preview.N` Releases are explicitly excluded from that selection, even when they point at the same commit as the final tag. If no previous deliverable exists, omit `previous_tag_name` and record that GitHub's default baseline fallback was used.
-- Generated Release notes carry a workflow-management marker so retries can repair them. Before manually curating a generated body, remove that marker; unmarked non-placeholder notes are preserved by later workflow runs.
-- Phases run in order; a failure in any phase blocks everything after it. After the fix lands on main, restart from Phase 2 with the next preview iteration against the new `origin/main` hash — do not resume mid-pipeline against a stale hash.
-- If the release is abandoned after Phase 1 merged, main's version files claim a version that was never tagged. Either revert the bump PR or leave it to be overwritten by the next release — but tell the user explicitly and record the decision.
-- User-facing status updates in Chinese; commits, PR titles/bodies, and tag messages in English. No hard-wrapping in commit messages, PR bodies, or documentation prose — one logical line per sentence/paragraph, let soft wrap handle display.
-
-## Phase 0 — Preflight
-
-- `git status --short` clean; `git fetch origin main --tags`.
-- `gh auth status` works; confirm you can view `gh release list -L 3`.
-- Confirm the exact final target version with the user if not explicit.
-
-### Console release gate
-
-Complete this gate before changing any RustFS version file or creating any RustFS tag. RustFS `build.yml` downloads the asset returned by `repos/rustfs/console/releases/latest`, so a successful Console build alone is insufficient.
-
-1. Read the latest published Console tag and compare it with Console `main`:
-
-```bash
-CONSOLE_REPO="rustfs/console"
-CONSOLE_LATEST=$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)
-gh api "repos/${CONSOLE_REPO}/compare/${CONSOLE_LATEST}...main" \
-  --jq '{status, ahead_by, behind_by, commits: [.commits[] | {sha, message: .commit.message}]}'
-```
-
-- `ahead_by == 0`: no merged Console change is waiting for release. Still verify the current latest asset using step 4, then continue to Phase 1.
-- `ahead_by > 0` and `behind_by == 0`: publish Console before continuing. Report the merged commits and select the next unused `vX.Y.Z` tag. Default to the next patch version when the changes are fixes or backward-compatible UI work; stop for confirmation if a minor/major bump is plausible.
-- Any diverged history or `behind_by > 0`: stop and resolve the Console release baseline explicitly. Do not guess a range or publish RustFS.
-
-2. Clone/fetch `rustfs/console` into a scratch directory and record its exact `main` commit. Before creating a tag, check for a `v*` tag or Release workflow already associated with that hash. If one is in progress, wait for it instead of creating another version:
-
-```bash
-CONSOLE_SCRATCH=$(mktemp -d)
-gh repo clone "$CONSOLE_REPO" "$CONSOLE_SCRATCH/console"
-git -C "$CONSOLE_SCRATCH/console" fetch origin main --tags
-CONSOLE_HASH=$(git -C "$CONSOLE_SCRATCH/console" rev-parse origin/main)
-git -C "$CONSOLE_SCRATCH/console" tag --points-at "$CONSOLE_HASH" 'v*'
-gh run list -R "$CONSOLE_REPO" --workflow release.yml --commit "$CONSOLE_HASH" --limit 5
-```
-
-If no release exists or is running for `CONSOLE_HASH`, create the selected annotated tag at that exact hash and push it:
-
-```bash
-git -C "$CONSOLE_SCRATCH/console" tag -a "<console-tag>" -m "Release <console-tag>" "$CONSOLE_HASH"
-git -C "$CONSOLE_SCRATCH/console" push origin "<console-tag>"
-```
-
-Console tags include the `v` prefix. Pushing the tag triggers `.github/workflows/release.yml` (`🚀 Release`). Remove `CONSOLE_SCRATCH` after the gate completes.
-
-3. Find the exact tag run and wait for completion:
-
-```bash
-gh run list -R "$CONSOLE_REPO" --workflow release.yml --branch "<console-tag>" --limit 1
-gh run watch -R "$CONSOLE_REPO" "<console-run-id>" --exit-status
-```
-
-4. Block until the published Release is non-draft, the latest endpoint returns the expected tag, and `rustfs-console-<console-tag>.zip` is uploaded, non-empty, and carries a `sha256:` digest:
-
-```bash
-gh release view -R "$CONSOLE_REPO" "<console-tag>" --json isDraft,isPrerelease,assets,url
-test "$(gh api "repos/${CONSOLE_REPO}/releases/latest" --jq .tag_name)" = "<console-tag>"
-test "$(gh api "repos/${CONSOLE_REPO}/releases/tags/<console-tag>" \
-  --jq '[.assets[] | select(.name == "rustfs-console-<console-tag>.zip" and .state == "uploaded" and .size > 0 and (.digest | startswith("sha256:")))] | length')" -eq 1
-```
-
-Treat a missing/mismatched asset, digest, latest tag, or failed/cancelled workflow as BLOCKED. Do not start Phase 1 until the Console gate passes. Record `CONSOLE_TAG`, `CONSOLE_HASH`, Console run URL, and Release URL for the final report.
-
-## Phase 1 — Version bump to the final target (once)
-
-- If main's version files already read `<target>` (e.g. this is a restart after a failed preview), verify with `rg -n "<target>" Cargo.toml rustfs.spec helm/rustfs/Chart.yaml` and skip to Phase 2.
-- Otherwise invoke the `rustfs-release-version-bump` skill with the final `<target>` (NOT a preview version), full GitHub flow (commit/push/PR).
-- Get the PR merged into main. Record the resulting main commit:
-
-```bash
-git fetch origin main
-PREVIEW_HASH=$(git rev-parse origin/main)   # must contain the bump PR
-```
-
-`PREVIEW_HASH` is the single source of truth for the rest of the pipeline — report it to the user and reuse it verbatim in Phases 2 and 6. Both the preview tag and the final tag will point at it.
-
-## Phase 2 — Publish the preview tag
-
-```bash
-git tag -a "<preview-tag>" -m "Release <preview-tag>" "$PREVIEW_HASH"
-git push origin "<preview-tag>"
-```
-
-Pushing the tag triggers `.github/workflows/build.yml` ("Build and Release"); `docker.yml` chains off it via `workflow_run`.
-
-The preview run builds versioned artifacts and publishes them in a GitHub prerelease. Its latest-channel, R2, Docker, and Helm jobs must be skipped. Those publication paths run only after the final tag is pushed.
-
-On a restart (N+1), refresh `PREVIEW_HASH=$(git rev-parse origin/main)` first — it must contain the fix — and re-report it.
-
-## Phase 3 — CI and preview Release verification
-
-- Find and watch the tag build: `gh run list --workflow build.yml --branch "<preview-tag>" --limit 1` then `gh run watch <run-id>`. Every build matrix target must succeed (linux x86_64/aarch64 × musl/gnu, macos-aarch64, windows-x86_64).
-- Confirm the Release publication jobs (`create-release`, `upload-release-assets`, and `publish-release`) succeed while `update-latest-version` is skipped.
-- Verify `gh release view "<preview-tag>" --json isPrerelease,assets,url`: `isPrerelease` must be `true`, and the Release must contain all 6 versioned platform zips, checksums, SBOM, and provenance with no `-latest` assets. Confirm `gh api repos/{owner}/{repo}/releases/latest --jq .tag_name` does not return `<preview-tag>`.
-- Record `PREVIOUS_DELIVERABLE`, selected from published Releases by `publishedAt` after excluding the current tag and every `-preview.N` tag. Verify `gh release view "<preview-tag>" --json body --jq .body` contains `## What's Changed` and, when `PREVIOUS_DELIVERABLE` exists, `**Full Changelog**: https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<preview-tag>`. For a repository with no previous deliverable, verify a Full Changelog link exists and record the GitHub baseline fallback.
-- Confirm preview-triggered Docker and Helm jobs are skipped. Preview validation covers the built RustFS binaries, embedded console, and rc compatibility; Docker image construction and Helm publication are deferred to the final tag because the Dockerfiles consume GitHub Release assets.
-
-## Phase 4 — Run the artifact locally, verify the console
-
-Work inside the session scratchpad directory; never leave stray data dirs.
-
-```bash
-gh release download "<preview-tag>" -p "rustfs-macos-aarch64-v<preview-tag>.zip" -D "$SCRATCH"
-cd "$SCRATCH" && unzip -o rustfs-*.zip
-./rustfs --version        # must report the PREVIEW TAG (build::TAG), not the Cargo.toml version, plus expected short SHA
-mkdir -p data
-RUSTFS_ACCESS_KEY=rustfsadmin RUSTFS_SECRET_KEY=rustfsadmin ./rustfs ./data
-```
-
-Defaults: S3 endpoint `:9000`, embedded console `:9001`.
-
-Checks (all must pass):
-
-- `./rustfs --version` reports the preview tag name and the short SHA of `PREVIEW_HASH`. Reporting `<target>` without the `-preview.N` suffix means the build did not embed the tag — treat as FAIL and investigate before proceeding.
-- `curl -fsS http://localhost:9000/health/ready` returns ready.
-- Startup log shows the embedded console being served (this was the regression that `fix(release): require embedded console assets` guards).
-- Open `http://localhost:9001` in the browser: login with `rustfsadmin`/`rustfsadmin`; dashboard renders without JS console errors; create a bucket, upload a file, download it back (byte-identical), delete the object and bucket. Keep the server running for Phase 5.
-
-## Phase 5 — Validate with the latest rc client
-
-`rc` is the RustFS CLI client from <https://github.com/rustfs/cli>.
-
-- Ensure the latest release is installed: compare `rc --version` against `gh api repos/rustfs/cli/releases/latest --jq .tag_name`; update via `brew upgrade rustfs/tap/rc` (or download the release binary).
-- Point it at the preview server and run the command matrix, recording PASS/FAIL per command:
-
-```bash
-rc alias set preview http://localhost:9000 rustfsadmin rustfsadmin
-rc ls preview/
-rc mb preview/rel-check
-rc cp <local-file> preview/rel-check/
-rc stat preview/rel-check/<file>
-rc cat preview/rel-check/<file>          # matches source
-rc cp preview/rel-check/<file> ./out && cmp <local-file> ./out
-rc cp -r <local-dir>/ preview/rel-check/dir/
-rc find preview/rel-check --name "*"
-rc share download preview/rel-check/<file> --expire 1h   # presigned URL fetchable via curl
-rc rm preview/rel-check/<file> && rc rm -r --force preview/rel-check/dir
-rc rb preview/rel-check
-rc admin user list preview/
-rc admin user add preview/ relcheckuser relchecksecret12
-rc admin user remove preview/ relcheckuser
-rc alias remove preview
-```
-
-- Any FAIL blocks the release. Afterwards stop the server and delete the scratch data directory.
-
-## Phase 6 — Publish the final tag on the validated commit
-
-No second version bump, no release branch. The final tag goes on the exact commit the preview validated:
-
-```bash
-git fetch origin --tags
-git rev-parse "<preview-tag>^{commit}"          # must equal PREVIEW_HASH — abort if not
-git tag -a "<target>" -m "Release <target>" "$PREVIEW_HASH"
-git push origin "<target>"
-```
-
-- CI rebuilds from the same source; the only changed input is the tag name, so the binary now self-reports `<target>`.
-- Verify the final tag's complete publication path: all matrix and release jobs green; `gh release view "<target>"` shows the full versioned and `-latest` asset set plus checksums, SBOM, and provenance; Docker and Helm workflows succeed; `latest.json` points to `<target>`. A stable target must have `isPrerelease=false` and `isLatest=true`. An alpha/beta/rc target must have `isPrerelease=true`; GitHub does not permit prereleases to be Latest, but the project `latest.json` still advances to the final non-preview target.
-- Verify the final Release body contains `## What's Changed` and a Full Changelog link. When `PREVIOUS_DELIVERABLE` exists, the link MUST be `https://github.com/rustfs/rustfs/compare/<PREVIOUS_DELIVERABLE>...<target>` and the baseline MUST equal the preview Release baseline; for example, both `1.0.0-beta.12-preview.1` and `1.0.0-beta.12` compare from `1.0.0-beta.11`.
-- Optionally spot-check `./rustfs --version` from a final-tag artifact — it must report `<target>`.
-
-## Output contract
-
-Always report:
-
-- Console gate result: previous/latest Console tags, whether merged changes required a release, `CONSOLE_HASH`, and Console run/Release URLs when a release was published.
-- Target version, preview tag(s) used, `PREVIEW_HASH` (which both tags point at).
-- Per-phase result (PASS/FAIL/BLOCKED) with key evidence: preview and final Release URLs, preview `isPrerelease`/`isLatest` state, final latest-channel state, console check results, and the rc command matrix.
-- Any deviation from this pipeline and why the user approved it.

.agents/skills/rustfs-release-version-bump/SKILL.md

@@ -1,6 +1,6 @@
 ---
 name: rustfs-release-version-bump
-description: "Prepare the version-file and release-asset bump for an exact RustFS alpha/beta/stable target, with verification and optional commit/push/PR delivery. Use for an explicit version bump or when invoked by the release-publish workflow."
+description: "Publish a RustFS alpha/beta/stable release with an auditable flow: confirm target version and scope, update workspace and release assets (including strict rustfs.spec changelog identity/date/version format), run required verification, and finish with commit, push, and GitHub PR creation."
 ---
 # RustFS Release Version Bump
 
@@ -18,8 +18,6 @@ Validated baseline: release pattern used in PR `#2957`.
 
 If target version is missing or ambiguous, stop and ask before editing.
 
-Reject any target version containing `-preview`: preview identifiers are tag-only (see `rustfs-release-publish`) and must never be written into version files. If asked for one, stop and point to the release pipeline instead of editing.
-
 ## Read before editing
 
 - `AGENTS.md` (root and nearest path-specific files).
@@ -81,7 +79,10 @@ Only drop a file when the current repository release process clearly no longer r
 
 4. Verify before shipping
 - Run:
+- `cargo fmt --all`
+- `cargo fmt --all --check`
 - `make pre-commit`
+- If verification passes, run `cargo clean`.
 - If `make pre-commit` fails, return `BLOCKED` with root cause and do not silently widen scope to fix unrelated issues unless user asks.
 
 5. Commit strategy
@@ -106,7 +107,10 @@ Only drop a file when the current repository release process clearly no longer r
 - `git diff --name-only origin/main...HEAD`
 - `git diff --stat origin/main...HEAD`
 - `rg -n "<old_version>|<new_version>" Cargo.toml Cargo.lock README.md README_ZH.md flake.nix helm/rustfs/Chart.yaml rustfs.spec`
+- `cargo fmt --all`
+- `cargo fmt --all --check`
 - `make pre-commit`
+- `cargo clean`
 
 ## Output contract
 

.agents/skills/rustfs-release-version-bump/agents/openai.yaml

@@ -1,4 +1,4 @@
 interface:
   display_name: "RustFS Release Bump"
   short_description: "Prepare RustFS release branches like PR #2957."
-  default_prompt: "Use $rustfs-release-version-bump to prepare and verify an exact RustFS release-version bump."
+  default_prompt: "Use $rustfs-release-version-bump to prepare a RustFS release version, ask about any unclear version policy, and finish the commit/push/PR flow."

.agents/skills/security-advisory-lessons/SKILL.md

@@ -1,40 +1,145 @@
 ---
 name: security-advisory-lessons
-description: Perform a dedicated RustFS security/advisory review for authn/authz, IAM, RPC trust, paths, secrets, browser isolation, encryption, Object Lock, or other security boundaries. Use only when the user requests a security/advisory review or an adversarial review explicitly escalates to the full advisory map; do not auto-load solely because code touches a sensitive path.
+description: Apply RustFS security lessons distilled from repository GitHub Security Advisories. Use when making or reviewing RustFS code changes, doing security checks, handling PR review for auth/authz, IAM, storage, RPC, logging, CORS, console/browser, encryption, policy, or endpoint changes, and when deciding which security regression tests are required.
 ---
 
 # RustFS Security Advisory Lessons
 
-Use this skill as the deep security lens. For a normal adversarial review with a
-matched security surface, the concise security reference under
-`adversarial-validation` is sufficient.
+Use this skill as a RustFS-specific security lens before changing or approving code. For the distilled advisory lessons and review patterns, read [advisory-patterns.md](references/advisory-patterns.md).
 
-## Workflow
-
-1. Freeze the exact diff/head and identify the changed trust boundaries.
-2. Read [advisory-patterns.md](references/advisory-patterns.md), then apply only
-   the matching sections. Useful headings are
-   auth/admin, IAM/STS/OIDC, policy/plugins, S3/copy/multipart, protocols, paths,
-   secrets/logging/RPC, browser/CORS/proxy, SSE, Object Lock, and serde.
-3. Trace unauthenticated, low-privilege, wrong-action/owner/bucket, malformed,
-   and default-config cases. Security decisions must fail closed.
-4. Require a focused negative regression test for the bypass/exploit form, not
-   only the intended success path. State residual risk when a test is impractical.
-5. Report proven vulnerabilities separately from defense-in-depth hardening.
-
-When advisory currentness matters, fetch the live inventory instead of treating
-the reference as a status mirror:
+When currentness matters, fetch the live advisory inventory instead of relying on this skill as a status mirror:
 
 ```bash
 gh api repos/rustfs/rustfs/security-advisories --paginate \
   --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
 ```
 
-Fetch an individual advisory only when the live summary indicates a new or
-changed lesson.
+Fetch full advisory details only when the live summary suggests a new or changed lesson:
 
-## Finding Standard
+```bash
+gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
+```
 
-Each finding includes severity, `file:line`, attacker prerequisites, concrete
-input/path, impact, smallest safe fix, and a regression check. Do not exaggerate
-unauthenticated impact when the actual issue requires authenticated low privilege.
+For the full pattern map, read [advisory-patterns.md](references/advisory-patterns.md).
+
+## Workflow
+
+### 1. Scope the change
+- Identify touched routes, protocol frontends, handlers, storage paths, credentials, logs, browser surfaces, CI/release code, and policy checks.
+- Treat these paths as security-sensitive by default: `rustfs/src/admin/`, `rustfs/src/storage/`, `rustfs/src/auth.rs`, `rustfs/src/server/layer.rs`, `crates/iam/`, `crates/policy/`, `crates/credentials/`, `crates/ecstore/src/rpc/`, `crates/protocols/`, `crates/rio/`, OIDC/STS federation code, and console preview/auth code.
+
+### 2. Map to advisory classes
+- Read [advisory-patterns.md](references/advisory-patterns.md) for matching GHSA lessons.
+- Do not rely on advisory titles alone. Confirm whether the issue is authentication, authorization, input validation, storage invariant, browser isolation, logging, or operational hardening.
+
+### 3. Verify fail-closed behavior
+- Check that unauthenticated, wrong-permission, cross-user, cross-bucket, malformed-input, and default-config cases fail explicitly.
+- Prefer exact action/permission checks over broad helper calls or inferred ownership.
+- Confirm lower storage/RPC layers do not bypass checks done in upper layers.
+
+### 4. Require regression evidence
+- For behavior changes, add focused negative tests that reproduce the advisory class.
+- For sensitive fixes, include tests for the bypass form, not only the happy path.
+- If a test is impractical, explain the residual risk and provide a manual verification command.
+
+### 5. Report clearly
+- Lead with concrete findings and file/line evidence.
+- Separate proven vulnerabilities from hardening risks.
+- Avoid exaggerating unauthenticated impact when the code actually rejects unauthenticated requests but allows a low-privileged authenticated bypass.
+
+## Advisory-Derived Guardrails
+
+### Auth and admin authorization
+- Every admin or diagnostic route needs an explicit authn and authz story. Route registration, router whitelist, and handler-level authorization must agree.
+- Match the admin action to the operation exactly. Copy-paste action constants are a known RustFS vulnerability class.
+- Avoid authentication-only helpers for state-changing admin APIs; use `validate_admin_request` or the established equivalent with the right `AdminAction`.
+- Read-only admin APIs such as metrics, server info, and diagnostics still require admin authorization; checking only that credentials exist is not enough.
+- Replication admin reads can expose remote target credentials; list/get target endpoints require replication/admin authorization and must not return secrets to low-privilege callers.
+- Do not assume admin-action `Resource` scoping constrains blast radius unless the policy engine actually enforces resources for that action.
+
+### IAM and service accounts
+- Treat imported IAM payload fields as attacker-controlled: `parent`, `claims`, `accessKey`, `secretKey`, status, policy names, and groups.
+- For service account create/update/import, prove parent ownership or root/admin authority before writing credentials or claims; an action permission alone must not allow choosing root or another user as `target_user`.
+- Do not let `deny_only` or "no explicit deny" become an allow decision that skips required allow checks.
+- Test cross-user list/update/import flows with wrong, correct, self, parent, and root identities.
+
+### STS, OIDC, and federation flows
+- Every STS endpoint must have an explicit authentication story: SigV4 where required, OIDC token verification for web identity, and role/session policy validation before issuing credentials.
+- JWT session tokens must be signed and verified by a trusted issuer/key path, not by service-account-controlled material or a reused root secret.
+- Public OIDC bootstrap and callback routes must treat `Host`, `X-Forwarded-Proto`, redirect targets, `state`, and callback parameters as untrusted; credential-bearing redirects require a configured, allowlisted origin.
+- OIDC discovery and validation URLs are SSRF sinks. Resolve and classify hostnames at connection time, reject rebinding to loopback/private/link-local ranges, and do not rely on literal string checks.
+
+### S3 copy, multipart, and presigned POST
+- Multipart copy must enforce source `GetObject` and destination `PutObject` semantics equivalent to `CopyObject`, including copy-source and policy conditions.
+- Do not let `CreateMultipartUpload`, `UploadPartCopy`, `CompleteMultipartUpload`, or `AbortMultipartUpload` return success without authorization.
+- Presigned POST policies are server-side contracts. Enforce `content-length-range`, key prefix, exact metadata/content-type, and all signed policy conditions.
+
+### Protocol frontends and IAM parity
+- FTP/FTPS, SFTP, gateway, and other protocol drivers must enforce IAM per operation before calling storage backends; authentication to a protocol listener is not authorization.
+- Match protocol commands to the same S3 actions as HTTP, such as `RETR` to `GetObject`, `SIZE`/`MDTM` to `HeadObject`, `MKD` to `CreateBucket`, and bucket probes to `ListBucket` or `HeadBucket`.
+- Review every handler in a protocol driver, not only the changed handler, because RustFS advisories show mixed guarded and unguarded siblings in the same driver.
+- Regression tests for protocol frontends should deny the shared authorization hook and prove the backend is not reached for the denied command.
+- Compare protocol secrets in constant time, normalize invalid-user and invalid-secret failures where practical, and add rate limiting before exposing password-style protocol endpoints.
+
+### Paths, object keys, and filesystem access
+- Never join untrusted bucket/object/RPC path strings onto filesystem roots without normalization and boundary checks.
+- Reject or safely handle `..`, absolute paths, URL-encoded traversal, platform separators, empty components, and paths that canonicalize outside the intended root.
+- Validate both S3 object-key paths and internode/RPC disk paths; storage helpers can bypass S3 authorization if they trust already-parsed paths.
+- Archive auto-extract paths are object keys too. Validate tar/zip entry names before IAM checks and before storage writes, and prove cleaned paths cannot cross bucket or prefix boundaries.
+
+### Secrets, default credentials, and crypto
+- Do not ship hard-coded shared tokens, HMAC secrets, private keys, or production test keys.
+- Defaults for root credentials and internode/RPC auth must fail closed for network-reachable deployments or generate per-install random secrets; warnings alone are not a security boundary.
+- Keep cryptographic roles separated: root S3 credentials, RPC HMAC keys, and STS/JWT signing keys must not be reused or deterministically derived from each other.
+- License or token validation must use signatures with embedded public/verifying keys only; do not use private-key decryption as authenticity.
+- Plan key rotation and key IDs when removing exposed keys.
+
+### Logging and debug output
+- Logs must never include access keys beyond safe identifiers, secret keys, session tokens, JWT claims, HMAC secrets, expected signatures, license secrets, or raw response bodies containing credentials.
+- Treat `Debug` implementations, `?value` tracing, merged config dumps, and dependency-level HTTP body logging as leak surfaces.
+- Add log-capture tests or targeted unit tests for redaction wrappers when changing credential structs or response bodies.
+
+### RPC, parsing, and panic safety
+- Treat all RPC payload bytes as attacker-controlled. Replace `unwrap`, `expect`, and panic-prone deserialization with typed errors.
+- Malformed request tests should cover empty bytes, truncated MessagePack/protobuf, invalid enum values, stale timestamps, and invalid signatures.
+- RPC authentication must be independently strong; do not depend on S3 admin credentials unless the fallback is explicit and safe.
+- RPC signatures must bind the exact generated gRPC method path, timestamp, and request method. Service-prefix signatures must not authorize a different concrete NodeService call.
+
+### Browser, CORS, and console surfaces
+- Do not reflect arbitrary `Origin` while also allowing credentials. Default CORS should be no CORS unless explicitly configured.
+- Do not render user-controlled object content in a same-origin iframe with console credentials available to JavaScript.
+- Prefer origin separation for object preview/download, `nosniff`, CSP, strict content-type handling, and avoiding durable credentials in `localStorage`.
+- Preview safety must be based on trusted content type and sandboxing, not object names or extensions such as `.pdf`.
+- Console license/version-like metadata endpoints should expose only coarse public data unless authenticated, especially subject names and expiration timestamps.
+
+### Profiling, debug, and health endpoints
+- Profiling and debug endpoints are not health checks. They require admin auth, opt-in enablement, rate limiting, and safe responses.
+- Do not return absolute filesystem paths or other deployment layout in unauthenticated or low-privilege responses.
+- Ensure health endpoint allowlists cannot accidentally include expensive diagnostics.
+
+### Trusted proxy and network identity
+- Only honor `X-Forwarded-For` or `X-Real-IP` when the request came from a configured trusted proxy.
+- Apply the same trusted-proxy rule to scheme and host derivation; direct clients must not control security-sensitive redirects through `Host`, `X-Forwarded-Host`, or `X-Forwarded-Proto`.
+- Direct clients must use the socket peer address for `aws:SourceIp` and policy condition evaluation.
+- Add tests for direct spoofed headers and trusted-proxy headers.
+
+### SSE and storage invariants
+- Encryption metadata is not proof that bytes were encrypted on disk.
+- When touching reader/writer wrappers such as hashing, encryption, compression, or warp readers, verify wrapper order and inspect stored bytes in regression tests.
+- Avoid helper shortcuts that unwrap nested readers and accidentally bypass encryption or integrity layers.
+
+## Review Prompts
+
+Use these prompts while reviewing a diff:
+
+- Could a low-privileged authenticated user reach this path with the wrong action, parent, bucket, or source object?
+- Does a non-HTTP protocol path call the same authorization boundary as the S3 API before touching storage?
+- Does a public/default/empty config change security behavior from fail-closed to fail-open?
+- Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
+- Does this response contain stored replication, remote target, or service credentials that need redaction or stricter authorization?
+- Can this STS/OIDC path issue credentials without SigV4, trusted issuer validation, allowlisted redirects, or trusted-proxy host/scheme handling?
+- Does this outbound validation path resolve attacker-supplied hostnames and reject private, loopback, link-local, and rebound addresses at the actual connection boundary?
+- Is an archive entry, object key, or policy resource normalized differently between authorization and storage?
+- Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
+- Does a preview or browser-surface fix preserve the original security invariant when adding alternate viewers or file-type detection?
+- Does the test prove the exploit form is denied, or only that the intended form still works?

.agents/skills/security-advisory-lessons/agents/openai.yaml

@@ -1,4 +1,4 @@
 interface:
   display_name: "Security Advisory Lessons"
   short_description: "Apply advisory lessons in reviews."
-  default_prompt: "Use $security-advisory-lessons for a dedicated RustFS security review grounded in past advisories."
+  default_prompt: "Review code changes against past RustFS security advisory lessons and report concrete risks, missing tests, and recommended fixes."

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -27,29 +27,19 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 ### IAM import, service accounts, and privilege boundaries
 
 - `GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
-- `GHSA-3495-h8r9-gfqg`: `ExportIAM` wrote regular-user and service-account secret keys into exported ZIP data. Lesson: IAM export is a credential-disclosure boundary; redact, seal, or strictly justify every exported secret before treating export permission as safe.
 - `GHSA-5354-r3w2-34m8`: `AddServiceAccount` checked `CreateServiceAccountAdminAction` but trusted caller-supplied `target_user`, allowing service accounts under the root parent. Lesson: service-account create paths must validate parent ownership or root/admin authority, not only the create action.
 - `GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
 - `GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
 
 ### STS, OIDC, and federation flows
 
-- `GHSA-5qfg-mf7r-jp3w` and `GHSA-3473-5353-xhwh`: `AssumeRoleWithWebIdentity` was reachable through unauthenticated `POST /` routing and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption, and unauthenticated exemptions must be narrowed to the exact action with uniform failure responses.
-- `GHSA-jxrr-r6pv-h958`: unsigned JWT issuer data was decoded before verification to select an OIDC provider, and distinguishable failures could expose provider configuration. Lesson: web-identity routing may be unauthenticated, but pre-verification claims are untrusted routing hints; bound and rate-limit the request, normalize public errors, and verify signature, issuer, audience, and expiration before issuing credentials.
-- `GHSA-ccrv-v8v9-ch9q`, `GHSA-48rf-7j3q-3hfv`, and `GHSA-xvfh-7c9g-hpw2`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
+- `GHSA-5qfg-mf7r-jp3w`: `AssumeRoleWithWebIdentity` was reachable without the required request authentication and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption.
+- `GHSA-ccrv-v8v9-ch9q`: service-account-controlled material could self-sign JWT session tokens with forged policy claims. Lesson: session tokens must be signed by a trusted issuer/key path and validation must reject self-signed or principal-controlled tokens.
 - `GHSA-9pjf-w3c2-m32r`, `GHSA-4x2q-cpx9-9h26`, and `GHSA-xvpm-p3f7-34c3`: public OIDC authorize/callback flows trusted request `Host` or forwarded scheme when building credential-bearing redirects. Lesson: OIDC redirects must use configured allowlisted origins and trusted-proxy handling; never derive the post-login credential destination from direct client headers.
 - `GHSA-m479-9x88-94w6`, `GHSA-frwq-mfqx-83p8`, `GHSA-q9q8-rf9r-fg9f`, and `GHSA-j5c2-hhf7-6gf5`: OIDC validation accepted attacker-controlled discovery URLs because hostname checks rejected only literal forbidden IPs, allowing DNS rebinding SSRF. Lesson: outbound federation URL validation must resolve and classify hostnames at the connection boundary and reject loopback, private, link-local, and rebound addresses.
 
-### IAM policy conditions and external policy plugins
+### S3 copy, multipart, and upload policy validation
 
-- `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
-- `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
-- `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
-
-### S3 object actions, copy, multipart, and upload policy validation
-
-- `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
-- `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
 - `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
 - `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
 - `GHSA-w5fh-f8xh-5x3p`: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
@@ -68,7 +58,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 ### Secrets, defaults, and cryptographic misuse
 
-- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, `GHSA-63xc-c3w3-m2cf`, and `GHSA-ch63-6q4v-hwp5`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
+- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, and `GHSA-9gf3-jx4p-4xxf`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
 - `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
 - `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
 - `GHSA-75fx-qg6f-8rm7` and `GHSA-68cw-96m3-h2cf`: internode RPC secrets were derivable from known root credentials, making raw storage RPC signatures forgeable when explicit RPC secrets were unset. Lesson: RPC auth keys must be independent random secrets, never derived from S3 root credentials, and raw storage RPC should not share the public S3 listener without an internode-only boundary.
@@ -101,10 +91,6 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 - `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
 
-### Object Lock and retention invariants
-
-- `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
-
 ### Serde deserialization and input validation
 
 - No `#[serde(deny_unknown_fields)]` found across the entire codebase. Lesson: all structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs) should have `#[serde(deny_unknown_fields)]` to reject malformed or adversarial payloads.
@@ -120,13 +106,11 @@ Use these targeted searches when a diff touches security-sensitive code:
 rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
 rg -n "authorize_operation|FtpsDriver|SftpDriver|RETR|MKD|SIZE|MDTM|CreateBucket|GetObject|HeadObject" crates/protocols rustfs
 rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|content-length-range|starts-with" rustfs crates
-rg -n "ListBucketVersions|GetObjectVersion|versionId|VersionId|ExistingObjectTag|ForAllValues|ForAnyValue|POLICY_PLUGIN|opa" rustfs crates
 rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
 rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
 rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
 rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
 rg -n "HashReader|EncryptReader|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
-rg -n "ObjectLock|object_lock|retention|COMPLIANCE|GOVERNANCE|delete_prefix|lifecycle|scanner" rustfs crates
 rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 ```
 
@@ -136,12 +120,8 @@ rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
 - Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases, and assert denied paths do not reach the storage backend.
 - IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
 - Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
-- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, and multipart-copy source cases.
-- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, plugin mode, and built-in policy mode.
 - Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks.
 - Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
-- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
 - RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
 - Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
 - SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together.
-- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths.

.config/e2e-full-selection.txt

@@ -1,2 +0,0 @@
-sha256-darwin=b4ae71aa894e5c7795ae3eb8116f1777a7601d0f5db3898be2e48faf3329bd9b
-sha256-linux=433debd9d9defa832986269abdf0f1d131597b2d7a417ce930e17c1fd47d85ba

.config/e2e-nightly-selection.txt

@@ -1 +0,0 @@
-sha256=9b9bc336b43b70d0e06e0adb5455bf035bb18945d85d60936eb6fe4d48e0e680

.config/e2e-protocols-selection.txt

@@ -1,2 +0,0 @@
-sha256-darwin=55534a97fbd376f64c8f6c341d319017d11ff77cad6da8629a1a7f6a874e0315
-sha256-linux=c06fb8c19aed6f388b9dc61cb8251b7a44f8561a9bf764ad2b9e635598f8dc17

.config/e2e-repl-nightly-selection.txt

@@ -1 +0,0 @@
-sha256=655a3f3c1d042e694339d15caba7580518320322d1bac0f09450b37e6c09e2e7

.config/e2e-smoke-selection.txt

@@ -1 +0,0 @@
-sha256=ec27cde6ce6400723c4b372bfbd2ac61709c744294e4810af765e8a808d8e31d

.config/make/coverage.mak

@@ -1,26 +0,0 @@
-## —— Coverage --------------------------------------------------------------------------------------
-
-# Local equivalent of the weekly coverage workflow (.github/workflows/coverage.yml,
-# backlog#1153 infra-5): same measurement scope (--workspace --exclude e2e_test,
-# nextest `ci` profile) and the same per-crate table. Slow — the instrumented
-# build cannot reuse your normal target cache and then runs the whole suite.
-# Doctests are not measured (needs nightly). Outputs land in target/llvm-cov/.
-.PHONY: coverage
-coverage: core-deps ## Workspace line coverage (cargo-llvm-cov + nextest; slow, writes target/llvm-cov/)
-	@if ! command -v cargo-llvm-cov >/dev/null 2>&1; then \
-		echo >&2 "❌ cargo-llvm-cov is required for 'make coverage' but was not found."; \
-		echo >&2 "   Install it with:"; \
-		echo >&2 "     cargo install cargo-llvm-cov --locked"; \
-		echo >&2 "     rustup component add llvm-tools-preview"; \
-		exit 1; \
-	fi
-	@if ! command -v cargo-nextest >/dev/null 2>&1; then \
-		echo >&2 "❌ cargo-nextest is required for 'make coverage' (see 'make test')."; \
-		echo >&2 "   Install it with: cargo install cargo-nextest --locked"; \
-		exit 1; \
-	fi
-	NEXTEST_PROFILE=ci cargo llvm-cov nextest --workspace --exclude e2e_test --no-report
-	@mkdir -p target/llvm-cov
-	cargo llvm-cov report --lcov --output-path target/llvm-cov/lcov.info
-	cargo llvm-cov report --json --output-path target/llvm-cov/coverage.json
-	python3 scripts/coverage_per_crate.py target/llvm-cov/coverage.json

.config/make/lint-fmt.mak

@@ -60,31 +60,6 @@ body-cache-whitelist-check: ## Check the body-cache eligibility gate stays a fai
 	@echo "🧱 Checking body-cache whitelist guard..."
 	./scripts/check_body_cache_whitelist.sh
 
-.PHONY: s3s-footprint-check
-s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
-	@echo "📦 Checking s3s footprint ratchet..."
-	./scripts/check_s3s_footprint.sh
-
-.PHONY: fips-wording-check
-fips-wording-check: ## Check docs and crates/kms do not over-claim crypto capabilities
-	@echo "📣 Checking cryptographic capability wording guard..."
-	./scripts/check_fips_wording.sh
-
-.PHONY: embedded-secrets-check
-embedded-secrets-check: ## Check no private key material or credential literal is committed
-	@echo "🔑 Checking embedded secret material guard..."
-	./scripts/check_embedded_secrets.sh
-
-.PHONY: test-wiring-check
-test-wiring-check: ## Check tests stay registered and selected by their intended runners
-	@echo "🧪 Checking test wiring..."
-	python3 ./scripts/check_test_wiring.py
-
-.PHONY: log-analyzer-rules-check
-log-analyzer-rules-check: core-deps ## Check log-analyzer rule anchors still exist verbatim in source
-	@echo "🩺 Checking log-analyzer rule anchors..."
-	./scripts/check_log_analyzer_rules.sh
-
 .PHONY: compilation-check
 compilation-check: core-deps ## Run compilation check
 	@echo "🔨 Running compilation check..."

.config/make/pre-commit.mak

@@ -19,13 +19,13 @@ planning-docs-check: ## Check that no planning-type documents are committed
 	./scripts/check_no_planning_docs.sh
 
 .PHONY: pre-commit
-pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
+pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
 	@echo "✅ All pre-commit checks passed!"
 
 .PHONY: pre-pr
-pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check clippy-check test ## Run full pre-PR checks with clippy and tests
+pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check clippy-check test ## Run full pre-PR checks with clippy and tests
 	@echo "✅ All pre-PR checks passed!"
 
 .PHONY: dev-check
-dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
+dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
 	@echo "✅ Fast development checks passed!"

.config/make/tests.mak

@@ -25,21 +25,7 @@ TEST_THREADS ?= 1
 script-tests: ## Run shell script tests
 	@echo "Running script tests..."
 	./scripts/test_build_rustfs_options.sh
-	./scripts/test_docker_runtime_timezone.sh
 	./scripts/test_entrypoint_credentials.sh
-	./scripts/test_internode_grpc_ab_bench.sh
-	./scripts/test_object_batch_bench_enhanced.sh
-	./scripts/test_hotpath_warp_ab_gate.sh
-	./scripts/test_hotpath_warp_abba.sh
-	./scripts/test_exact_1mib_handoff_abba.sh
-	./scripts/test_pinned_paired_abba_bench.sh
-	./scripts/test_manual_transition_runbooks.sh
-	./scripts/check_embedded_secrets.sh --self-test
-	python3 ./scripts/check_test_wiring.py --self-test
-	python3 ./scripts/s3-tests/test_report_compat.py
-	bash -n ./scripts/validate_object_data_cache_cold_stampede.sh
-	python3 ./scripts/check_object_data_cache_follower_samples.py --self-test
-	./scripts/validate_object_data_cache_cold_stampede.sh --self-test
 
 .PHONY: test
 test: core-deps script-tests ## Run all tests (needs cargo-nextest; RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 to override)

.config/nextest.toml

@@ -1,16 +1,17 @@
 # nextest configuration for RustFS.
 #
-# Serialize the ecstore tests that share the process-wide disk registry or
-# exercise a multi-disk commit handoff across nextest process boundaries.
+# Serialize two known load-sensitive / global-state-sharing ecstore test groups
+# so the full parallel nextest suite stops producing spurious failures
+# (backlog #937). These tests pass in isolation but flake under the loaded
+# parallel run for two distinct reasons:
 #
 #   * store::bucket::tests::bucket_delete_* share process/global state (disk
 #     registry, lock client) and race make_bucket into InsufficientWriteQuorum
 #     when run concurrently with other ecstore tests.
 #   * bucket_lifecycle_ops::tests::concurrent_resend_same_part_commits_one_generation
-#     uses the shared multipart fixture and a deterministic uploadId-lock
-#     handoff, so it must not overlap another process mutating that fixture.
-#   * bucket::metadata_sys::tests::concurrent_config_writes_from_separate_nodes_do_not_lose_writes
-#     uses the shared transaction lock and must not overlap other ecstore tests.
+#     asserts a lock-acquire correctness property whose serialized cross-disk
+#     commits exceed the (already max'd, 60s) acquire deadline only when the
+#     suite saturates disk I/O.
 #
 # serial_test's #[serial] attribute does NOT serialize these across runs:
 # nextest executes each test in its own process, where the in-process
@@ -29,89 +30,28 @@
 
 [test-groups]
 ecstore-serial-flaky = { max-threads = 1 }
-embedded-test-ports = { max-threads = 1 }
-e2e-vault = { max-threads = 1 }
 
 # Reliability / fault-injection e2e tests each spawn a single-node 4-disk RustFS
 # server and manipulate its disk directories at runtime (crates/e2e_test:
-# reliability_disk_fault_test, degraded_read_eof_regression_test / dist-13, and
-# replacement_privileged_e2e_test when explicitly run as root on Linux). They
+# reliability_disk_fault_test, degraded_read_eof_regression_test / dist-13). They
 # are correct in isolation but resource-heavy; serialize them under nextest's
 # process boundary (serial_test's #[serial] does not cross it) so several 4-disk
-# servers never run at once. The e2e-full merge/main lane picks these up;
+# servers never run at once. ci-7's nightly picks these up via the e2e suite;
 # they are deliberately NOT in the fast PR `e2e-smoke` filter.
 e2e-reliability = { max-threads = 1 }
-e2e-inline-boundaries = { max-threads = 1 }
-e2e-cluster-nightly = { max-threads = 1 }
 
 # --- default profile (local): serialize the flaky groups, never retry --------
 [[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes) | test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/))'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the multipart crash-consistency scenarios (dist-2, backlog#1150):
-# each spawns a 4-disk hermetic erasure set and drives full staged-upload +
-# commit + GET cycles — the same cross-disk-commit IO shape that made
-# concurrent_resend load-sensitive. Preventive serialization only, no retries.
-# The matching ci-profile override is after [profile.ci].
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
-test-group = 'ecstore-serial-flaky'
-
-# The production-handler relocation regression builds an isolated 8-disk,
-# 2-pool store and commits a 72 MiB multipart object. Keep that cross-disk IO
-# from overlapping the ecstore commit fixtures above.
-[[profile.default.overrides]]
-filter = 'package(rustfs) & test(execute_get_object_resumes_from_relocated_pool_without_splicing_body)'
-test-group = 'ecstore-serial-flaky'
-
-# Embedded integration-test binaries discover an ephemeral port and release
-# the probe listener before RustFS binds it. Serialize that cross-process
-# TOCTOU window; retries would only hide real startup failures.
-[[profile.default.overrides]]
-filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
-test-group = 'embedded-test-ports'
-
-# Serialize the durable manual-transition checkpoint test across nextest's
-# process boundary; it mutates bucket lifecycle metadata and is not quarantined.
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the bucket-incarnation / lifecycle-fence tests. They drive
-# init_bucket_metadata_sys and bucket_metadata_sys_of, i.e. process-global
-# OnceLock state that serial_test's #[serial] cannot protect across nextest's
-# process boundary, and they delete+recreate buckets — the same shape that
-# raced into InsufficientWriteQuorum in backlog#937. Preventive only, no
-# retries. The matching ci-profile override is after [profile.ci].
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the default-off dst-dir fsync group-commit tests. They use
-# process-global test hooks/registry to deterministically freeze fsync batches;
-# no retries, just one at a time under nextest too.
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & test(dst_dir_fsync_group_commit)'
+filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(/^store::bucket::tests::bucket_delete_(mark_delete_marks|purge_removes|default_s3_delete)/))'
 test-group = 'ecstore-serial-flaky'
 
 # Serialize the 4-disk reliability / degraded-read e2e tests (see the
 # e2e-reliability test-group note above). The matching ci-profile override is at
 # the end of the file, after [profile.ci] is declared.
 [[profile.default.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
+filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
 test-group = 'e2e-reliability'
 
-[[profile.default.overrides]]
-filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
-test-group = 'e2e-inline-boundaries'
-
-# Vault KMS tests share the fixed dev-server port 8200. serial_test's #[serial]
-# does not cross nextest process boundaries, so keep these tests in one group.
-[[profile.default.overrides]]
-filter = 'package(e2e_test) & test(/^kms::kms_vault_test::/)'
-test-group = 'e2e-vault'
-
 # ---------------------------------------------------------------------------
 # ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
 # ---------------------------------------------------------------------------
@@ -140,17 +80,19 @@ path = "junit.xml"
 # profile's own overrides list, not the default profile's).
 # ===========================================================================
 
-# QUARANTINE: OPEN backlog#937 — store::bucket::tests::bucket_delete_* race
-# make_bucket into InsufficientWriteQuorum via shared global state under load.
+# QUARANTINE: OPEN backlog#937 — concurrent_resend lock-acquire deadline flakes
+# under saturated disk I/O in the full parallel suite.
 [[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(mark_delete|purge_removes|default_s3_delete)/)'
+filter = 'package(rustfs-ecstore) & test(concurrent_resend_same_part_commits_one_generation)'
 test-group = 'ecstore-serial-flaky'
 retries = 2
 
-# Keep deterministic ECStore write handoffs isolated across nextest processes.
+# QUARANTINE: OPEN backlog#937 — store::bucket::tests::bucket_delete_* race
+# make_bucket into InsufficientWriteQuorum via shared global state under load.
 [[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(concurrent_config_writes_from_separate_nodes_do_not_lose_writes))'
+filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(mark_delete_marks|purge_removes|default_s3_delete)/)'
 test-group = 'ecstore-serial-flaky'
+retries = 2
 
 # QUARANTINE: OPEN rustfs#4690 — walk_dir stall-budget accounting test depends
 # on producer/consumer timing windows that stretch past the budget on loaded
@@ -162,44 +104,11 @@ retries = 2
 # Serialize the 4-disk reliability / degraded-read e2e tests under the ci
 # profile too (see the e2e-reliability test-group note near the top). Not a
 # quarantine: no retries, just single-threaded so several 4-disk servers never
-# run concurrently when e2e-full runs the suite.
+# run concurrently when ci-7's nightly runs the full e2e suite.
 [[profile.ci.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
+filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
 test-group = 'e2e-reliability'
 
-# Serialize the multipart crash-consistency scenarios under the ci profile too
-# (see the matching default-profile override near the top). Not a quarantine:
-# no retries, just serialized 4-disk cross-disk-commit IO.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^set_disk::ops::multipart::tests::crash_consistency::/)'
-test-group = 'ecstore-serial-flaky'
-
-[[profile.ci.overrides]]
-filter = 'package(rustfs) & test(execute_get_object_resumes_from_relocated_pool_without_splicing_body)'
-test-group = 'ecstore-serial-flaky'
-
-# Match the default-profile embedded test isolation without quarantining or
-# retrying failures in CI.
-[[profile.ci.overrides]]
-filter = 'package(rustfs) & (binary(/^embedded.*_test$/) | binary(admin_diagnostic_capability_e2e))'
-test-group = 'embedded-test-ports'
-
-# Serialize the durable manual-transition checkpoint test under the ci profile
-# too. No retries: failures stay visible.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(manual_transition_page_checkpoint_persists_durable_job_progress)'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the bucket-incarnation / lifecycle-fence tests under the ci profile
-# too (see the matching default-profile override near the top). No retries.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & (test(bucket_delete_waits_for_config_mutation_fence) | test(stale_config_request_cannot_mutate_a_recreated_bucket) | test(disk_incarnation_read_detects_stale_cache_until_peer_reload) | test(lifecycle_expiry_fails_closed_on_corrupt_object_lock_metadata) | test(expiry_configs_are_resolved_from_the_owning_store))'
-test-group = 'ecstore-serial-flaky'
-
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(dst_dir_fsync_group_commit)'
-test-group = 'ecstore-serial-flaky'
-
 # ---------------------------------------------------------------------------
 # e2e-smoke profile — PR smoke subset of the e2e_test crate (backlog#1149 ci-4)
 # ---------------------------------------------------------------------------
@@ -213,10 +122,6 @@ test-group = 'ecstore-serial-flaky'
 # Each e2e test spawns its own rustfs server on a random port with an isolated
 # temp dir (crates/e2e_test/src/common.rs), so the subset is parallel-safe.
 #
-# Replication failure harness (backlog#1147 repl-8): the first clause admits
-# its four in-process fake-target self-tests. They bind random loopback ports,
-# use no external service, and finish in under a second.
-#
 # Replication PR subset (backlog#1147 repl-1): the second clause admits the 20
 # FAST bucket-replication tests from replication_extension_test — the
 # target-registration / replication-check / list / remove / delete admin paths
@@ -231,83 +136,34 @@ test-group = 'ecstore-serial-flaky'
 # the nightly profile derives its set as "the replication module MINUS this
 # allowlist", so any new replication test lands in nightly by default (never
 # silently unrun) until it is explicitly blessed as fast here. Keep the two
-# regexes byte-identical. The committed profile selection digests make changes
-# visible in CI; current counts live in docs/testing/e2e-suite-inventory.md.
+# regexes byte-identical. Count invariant: 20 here + 18 nightly = 38 total
+# (authority: `cargo nextest list`; docs/testing/e2e-suite-inventory.md).
 # HISTORY (2026-07-11): the 20 fast tests were briefly pulled out of this lane
 # (#4724) because they set a loopback (127.0.0.1) replication target that the
 # SSRF egress guard rejected on every PR after repl-1 (#4712). That is fixed —
 # the guard now honours an off-by-default opt-in and this suite's source servers
 # set it (RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) — so the allowlist below is
 # restored.
-#
-# Security negative-auth subset (backlog#1151 sec-5): the three attacker-facing
-# S3 auth-rejection suites join the first clause above by module name —
-# presigned_negative (sec-2), negative_sigv4 (sec-1, header SigV4), and
-# admin_auth (sec-4, admin gate + root-credential lifecycle). All three use
-# RustFSTestEnvironment on a random port and are parallel-safe, so they meet the
-# smoke admission criteria unchanged. This is the wiring step that makes those
-# merged suites actually execute on every PR (they were dead until listed here).
-# A rename that drops any of them out of this filter would silently thin the
-# security gate with no CI signal, so scripts/check_security_smoke_count.sh owns
-# a count-floor guard over exactly this subset (infra-12 mechanism, floor in
-# .config/security-smoke-floor.txt), invoked from the e2e-tests job in ci.yml.
-# NOT here by topology: the GHSA-3p3x FTPS/WebDAV constant-time e2e
-# (protocols::test_protocol_core_suite) binds fixed ports and needs the
-# ftps,webdav features, so it cannot join this random-port, default-feature
-# profile; its GHSA-r5qv sibling is a unit test that already runs in the
-# test-and-lint `--all --exclude e2e_test` pass. See
-# docs/testing/security-regressions.md for the full CI-execution map.
-#
-# ILM tiering main path (backlog#1148 ilm-7): the `reliant::tiering::` clause
-# admits the hermetic transition e2e. Like the fast replication pair checks it
-# spawns a second independent single-node server (the cold RustFS tier), not a
-# cluster, so it keeps the lane's parallel-safe / no-external-dependency
-# properties. The RustFS warm backend has no loopback guard (that guard is
-# replication-only), so it needs no opt-in env for its 127.0.0.1 tier target.
-#
-# Disk compression (backlog#1848): the `compression` module joins the smoke
-# lane so the multipart disk-compression roundtrips (restored after
-# rustfs/rustfs#5169 disabled them) have PR-lane signal, not just merge-gate.
-# Single-node servers on random ports with isolated temp dirs — meets the
-# admission criteria unchanged.
 [profile.e2e-smoke]
 default-filter = """
     package(e2e_test) & (
-        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|list_buckets_auth|list_buckets_iam_filter|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_object_tagging|copy_source_invalid_date|content_encoding|compression|multipart_storage_class|storage_class_capability|ssec_copy|anonymous_access|bucket_policy_check|presigned_negative|negative_sigv4|admin_auth|notification_webhook|tls_hot_reload|console_smoke|admin_iam_crud|admin_pools|sts_query_compat)_test::|^fake_s3_target::/)
+        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_source_invalid_date|content_encoding|anonymous_access|bucket_policy_check|presigned_negative)_test::/)
         | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
         | test(/^reliant::lifecycle::/)
-        | test(/^reliant::tiering::/)
     )
 """
 fail-fast = false
 
-[profile.e2e-smoke.junit]
-path = "junit.xml"
-
-# The pagination boundary cases can stall when a server/listing regression
-# prevents the continuation request from completing. Keep the timeout scoped
-# to those known failure modes so legitimate lifecycle/tiering waits retain
-# their test-level timing budget.
-[[profile.e2e-smoke.overrides]]
-filter = 'package(e2e_test) & test(/^list_objects_v2_pagination_test::tests::(test_list_objects_v2_delimiter_small_page_traverses_all|test_list_objects_v2_max_keys_above_limit_returns_token|test_list_objects_v2_maxkeys_above_limit_with_delimiter)$/)'
-slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
-
 # ---------------------------------------------------------------------------
 # e2e-repl-nightly profile — scheduled full replication e2e lane (repl-1)
 # ---------------------------------------------------------------------------
 # backlog#1147 repl-1 (deps: ci-4). Runs the SLOW / cross-process replication
 # tests that are unfit for the per-PR e2e-smoke gate:
 #
-#   * 2 remote-target TLS validation tests.
-#   * 15 bucket-replication data-plane/helper tests — they PUT/delete objects
-#     and poll until source and target converge; two replicate over HTTPS,
-#     six pin SSE replication contracts (managed SSE-S3/SSE-KMS re-encrypt on
-#     the target incl. multipart and the resync path, SSE-C and
-#     target-without-KMS stay fail-closed), and one guards event/history
-#     observers.
-#   * 12 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
+#   * 8 bucket-replication data-plane tests — they PUT/delete objects and poll
+#     until source and target converge; two replicate over HTTPS.
+#   * 9 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
 #     servers and drives the cross-process site-replication control plane.
-#   * 1 `_real_three_node` site-replication test.
 #   * 1 `_real_single_node` service-account round-trip test.
 #
 # The set is defined as "everything in replication_extension_test that is NOT
@@ -328,8 +184,9 @@ slow-timeout = { period = "60s", terminate-after = 2, grace-period = "10s" }
 # the STS dual-node test actually exercises its path (it skips gracefully with
 # a visible log line when awscurl is absent), and routes scheduled failures
 # through .github/actions/schedule-failure-issue (ci-8). Explicit division of
-# labor with e2e-full: these tests run only in the consolidated nightly
-# workflow, not in the merge/main lane.
+# labor with ci-5's future e2e-full merge gate: these tests run ONLY here, not
+# double-run there. TODO(ci-7): fold this interim repl-owned lane into the ci
+# domain's consolidated scheduled e2e workflow once it exists.
 [profile.e2e-repl-nightly]
 default-filter = """
     package(e2e_test)
@@ -342,106 +199,3 @@ fail-fast = false
 # Emitted to target/nextest/e2e-repl-nightly/junit.xml; uploaded by the nightly
 # workflow as the failure-triage artifact.
 path = "junit.xml"
-
-# ---------------------------------------------------------------------------
-# e2e-nightly profile — destructive multi-process cluster fault domains
-# ---------------------------------------------------------------------------
-# These seven modules are deliberately outside e2e-full's merge budget. Each
-# starts a real multi-process or multi-disk topology and exercises node/disk
-# loss, quorum, cleanup, notification fan-in, or admin-timeout behavior. The
-# consolidated nightly workflow runs them serially to avoid resource
-# starvation; failures are never retried.
-[profile.e2e-nightly]
-default-filter = """
-    package(e2e_test)
-    & test(/^(admin_timeout_regression_test|cluster_concurrency_test|cluster_multidrive_pool_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
-"""
-fail-fast = false
-
-[profile.e2e-nightly.junit]
-path = "junit.xml"
-
-[[profile.e2e-nightly.overrides]]
-filter = 'package(e2e_test)'
-test-group = 'e2e-cluster-nightly'
-
-# ---------------------------------------------------------------------------
-# e2e-protocols profile — serial protocol lane
-# ---------------------------------------------------------------------------
-# The suite owns fixed ports, so the nightly workflow runs this exact profile
-# with one nextest worker.
-[profile.e2e-protocols]
-default-filter = 'package(e2e_test) & test(/^protocols::/)'
-fail-fast = false
-
-[profile.e2e-protocols.junit]
-path = "junit.xml"
-
-# ---------------------------------------------------------------------------
-# e2e-full profile — merge-gate full single-node e2e lane (backlog#1149 ci-5)
-# ---------------------------------------------------------------------------
-# The merge gate (ci.yml `e2e-full` job: push main + merge_group +
-# workflow_dispatch). Runs the user-visible KMS, object-lock, multipart-auth,
-# quota, checksum, encryption,
-# security-boundary, ... — that the fast PR `e2e-smoke` subset deliberately
-# skips. Budget <= 45 min; authority for the suite count is `cargo nextest list
-# --profile e2e-full` (see docs/testing/e2e-suite-inventory.md).
-#
-# The filter is "the whole e2e_test crate MINUS the sets owned by other lanes":
-#   * protocols:: — FTPS/SFTP/WebDAV, run from the dedicated protocol profile
-#     with one worker because the suite owns fixed ports.
-#   * the 7 cluster suites that spin up a RustFSTestClusterEnvironment
-#     (cluster_concurrency, cluster_multidrive_pool, stale_multipart_cleanup_cluster,
-#     namespace_lock_quorum, heal_erasure_disk_rebuild, admin_timeout_regression,
-#     object_lambda) — too heavy for the merge budget; they run in the
-#     e2e-nightly serial cluster-fault lane.
-#   * replication_extension_test — repl-1 already splits it into the PR
-#     `e2e-smoke` (20 fast) and `e2e-repl-nightly` (55 slow) lanes and reserves
-#     it for those, so e2e-full does not double-run it.
-#   * #[ignore]d tests — nextest skips them by default (no --run-ignored); the
-#     manual-localhost:9000 reliant/policy tests are ci-13's migration.
-#
-# Each e2e test spawns its own single-node rustfs server on a random port with
-# an isolated temp dir (crates/e2e_test/src/common.rs), so the set is
-# parallel-safe — the same property e2e-smoke relies on. The exceptions are the
-# 4-disk reliability / degraded-read fault-injection tests and the fixed-port
-# Vault tests, both serialized below.
-# KNOWN-FAILURE EXCLUSIONS (characterization run 29381309848, 2026-07-15:
-# 341 ran / 32 failed on the suites' first automated run ever). Deterministic
-# product failures cannot be quarantined away with retries, so each family is
-# excluded here with its tracking issue, under the same discipline as the
-# ci-profile quarantine (docs/testing/README.md): every entry MUST cite one
-# OPEN issue, and the fixing PR MUST delete the exclusion. The passing
-# negative-path siblings of each family stay in as regression guards.
-#   * rustfs#4843 — over-limit archive entry paths hard-reject the whole
-#     archive even under ignore-errors semantics.
-[profile.e2e-full]
-default-filter = """
-    package(e2e_test)
-    & !test(/^protocols::/)
-    & !test(/^(admin_timeout_regression_test|cluster_concurrency_test|cluster_multidrive_pool_test|heal_erasure_disk_rebuild_test|namespace_lock_quorum_test|object_lambda_test|stale_multipart_cleanup_cluster_test)::/)
-    & !test(/^replication_extension_test::/)
-    & !test(/^multipart_auth_test::test_signed_put_object_extract_skips_invalid_entry_when_ignore_errors_enabled$/)
-    & !test(/^snowball_auto_extract_test::tests::snowball_auto_extract_(ignores_invalid_entries_when_requested|supports_standard_headers_with_combined_extract_options)$/)
-"""
-fail-fast = false
-
-[profile.e2e-full.junit]
-# Emitted to target/nextest/e2e-full/junit.xml; uploaded by the e2e-full job.
-path = "junit.xml"
-
-# Serialize the 4-disk reliability / degraded-read e2e tests under e2e-full too
-# (see the e2e-reliability test-group note near the top of this file). Not a
-# quarantine: no retries, just single-threaded so several 4-disk servers never
-# run concurrently.
-[[profile.e2e-full.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression|replacement_privileged_e2e)_test::/)'
-test-group = 'e2e-reliability'
-
-[[profile.e2e-full.overrides]]
-filter = 'package(e2e_test) & test(/^inline_fast_path_cluster_test::/)'
-test-group = 'e2e-inline-boundaries'
-
-[[profile.e2e-full.overrides]]
-filter = 'package(e2e_test) & test(/^kms::kms_vault_test::/)'
-test-group = 'e2e-vault'

.config/security-smoke-floor.txt

@@ -1,12 +0,0 @@
-# Committed floor for the number of security negative-auth tests selected by the
-# e2e-smoke PR profile (see scripts/check_security_smoke_count.sh, backlog#1151
-# sec-5).
-#
-# The floor equals the exact count of e2e_test cases whose name starts with a
-# security module prefix (negative_sigv4_test, presigned_negative_test,
-# admin_auth_test) that the [profile.e2e-smoke] default-filter in
-# .config/nextest.toml selects, at the time this file was last updated. CI fails
-# if the selected count drops below this number, so a rename or removal that
-# thins the security smoke gate must update this file in the same PR.
-# Adding tests does not require a bump, but bumping keeps the guard tight.
-16

.docker/observability/README.md

@@ -60,16 +60,6 @@ The file `prometheus-rules/rustfs-get-optimization-alerts.yaml` contains pre-con
 | `CodecStreamingFallbackSpike` | Warning | Codec streaming fallback > 10x baseline for 10m |
 | `IoQueueSaturation` | Warning | IO queue utilization > 90% for 5m |
 
-The file `prometheus-rules/rustfs-kms-alerts.yml` contains alerting rules for the KMS backend operation metrics. Thresholds are conservative defaults pending staging baseline calibration; response procedures live in `docs/operations/kms-observability-runbook.md`, and the matching dashboard is `deploy/observability/grafana/rustfs-kms-observability.json`.
-
-| Alert | Severity | Condition |
-|-------|----------|-----------|
-| `KmsBackendFatalErrors` | Critical | Fatal (non-retryable) attempt failures > 0 for 5m |
-| `KmsBackendHighErrorRate` | Critical | Non-success operation ratio > 5% for 10m (with traffic guard) |
-| `KmsBackendP99LatencyHigh` | Warning | Operation p99 duration (incl. retries) > 2s for 10m |
-| `KmsBackendAttemptFailureSpike` | Warning | Attempt failure rate > 0.5/s for 10m |
-| `KmsBackendRetryBudgetExhausted` | Warning | budget_exhausted / deadline_exceeded outcomes > 0.05/s for 10m |
-
 ### Enabling Alert Rules
 
 Add the alert rules file to your Prometheus configuration:
@@ -170,10 +160,6 @@ Important behavior notes:
 
 - Logs and metrics usually appear during startup, so seeing those two signals
   first is expected.
-- The OpenTelemetry bridge sends `tracing` fields as log attributes. Loki stores
-  those attributes as structured metadata, and the Collector also mirrors the
-  common troubleshooting fields into the log line so simple line filters can
-  find them.
 - Visible trace data usually requires real HTTP/S3/gRPC request traffic after
   startup, because request-path spans are created on demand.
 - `RUSTFS_OBS_LOGGER_LEVEL=info` keeps the top-level request span but filters
@@ -199,17 +185,6 @@ curl -I http://127.0.0.1:9000/health/ready
 # Jaeger:  http://localhost:16686
 ```
 
-For a structured RustFS log such as an inter-node RPC authentication failure,
-the Loki line now includes fields such as `event`, `component`, `subsystem`,
-`failure_reason`, `rpc_service`, `rpc_method`, and `expected_audience`. Useful
-LogQL checks:
-
-```logql
-{service_name="RustFS"} |= "RPC signature verification failed"
-{service_name="RustFS"} |= "failure_reason="
-{service_name="RustFS"} | failure_reason != ""
-```
-
 If logs and metrics are present but traces are sparse, the most common cause is
 "no real request traffic yet" or "`info` level filtered nested spans", not an
 OTLP routing failure.

.docker/observability/README_ZH.md

@@ -60,16 +60,6 @@
 | `CodecStreamingFallbackSpike` | 警告 | Codec streaming 回退 > 10x 基线，持续 10 分钟 |
 | `IoQueueSaturation` | 警告 | IO 队列利用率 > 90%，持续 5 分钟 |
 
-文件 `prometheus-rules/rustfs-kms-alerts.yml` 包含 KMS 后端操作指标的告警规则。阈值为保守默认值，待 staging 基线校准；响应流程见 `docs/operations/kms-observability-runbook.md`，配套仪表盘为 `deploy/observability/grafana/rustfs-kms-observability.json`。
-
-| 告警 | 级别 | 条件 |
-|------|------|------|
-| `KmsBackendFatalErrors` | 严重 | fatal（不可重试）尝试失败 > 0，持续 5 分钟 |
-| `KmsBackendHighErrorRate` | 严重 | 非 success 操作占比 > 5%，持续 10 分钟（含流量下限保护） |
-| `KmsBackendP99LatencyHigh` | 警告 | 操作 p99 耗时（含重试）> 2s，持续 10 分钟 |
-| `KmsBackendAttemptFailureSpike` | 警告 | 尝试失败率 > 0.5/s，持续 10 分钟 |
-| `KmsBackendRetryBudgetExhausted` | 警告 | budget_exhausted / deadline_exceeded 结果 > 0.05/s，持续 10 分钟 |
-
 ### 启用告警规则
 
 在 Prometheus 配置中添加告警规则文件：
@@ -169,7 +159,6 @@ RustFS 会自动在该基础 URL 后补全：
 需要注意：
 
 - 启动阶段通常会先看到日志和指标，因此“先有日志/指标、后有 trace”是正常现象。
-- OpenTelemetry bridge 会把 `tracing` 字段作为日志 attributes 发送。Loki 会将这些 attributes 存为 structured metadata，同时 Collector 会把常用排障字段镜像进日志行，方便用简单的行内容过滤直接查到。
 - 可见的 trace 数据通常依赖启动后的真实 HTTP/S3/gRPC 请求流量，因为请求路径上的 span 是按需创建的。
 - `RUSTFS_OBS_LOGGER_LEVEL=info` 会保留顶层请求 span，但会过滤掉很多 `debug` 级别的嵌套 span。
   如果 Tempo 或 Jaeger 中的 trace 看起来很稀疏，建议先改成 `RUSTFS_OBS_LOGGER_LEVEL=debug`，再判断是否是 collector 或 Tempo 问题。
@@ -193,14 +182,6 @@ curl -I http://127.0.0.1:9000/health/ready
 # Jaeger:  http://localhost:16686
 ```
 
-对于 RustFS 结构化日志，例如节点间 RPC 鉴权失败，Loki 日志行现在会包含 `event`、`component`、`subsystem`、`failure_reason`、`rpc_service`、`rpc_method`、`expected_audience` 等字段。常用 LogQL 检查：
-
-```logql
-{service_name="RustFS"} |= "RPC signature verification failed"
-{service_name="RustFS"} |= "failure_reason="
-{service_name="RustFS"} | failure_reason != ""
-```
-
 如果日志和指标已经正常，但 trace 仍然稀疏，最常见的原因通常是
 “还没有真实请求流量”或“`info` 级别过滤了嵌套 span”，而不是 OTLP 路由失败。
 

.docker/observability/otel-collector-config.yaml

@@ -29,27 +29,11 @@ processors:
     limit_mib: 1024
     spike_limit_mib: 256
   transform/logs:
-    error_mode: ignore
     log_statements:
       - context: log
         statements:
-          - set(attributes["message"], body.string) where IsString(body)
-          - set(attributes["log.body"], body.string) where IsString(body)
-          - set(body, Concat([body, " event=", attributes["event"]], "")) where IsString(body) and attributes["event"] != nil
-          - set(body, Concat([body, " component=", attributes["component"]], "")) where IsString(body) and attributes["component"] != nil
-          - set(body, Concat([body, " subsystem=", attributes["subsystem"]], "")) where IsString(body) and attributes["subsystem"] != nil
-          - set(body, Concat([body, " state=", attributes["state"]], "")) where IsString(body) and attributes["state"] != nil
-          - set(body, Concat([body, " result=", attributes["result"]], "")) where IsString(body) and attributes["result"] != nil
-          - set(body, Concat([body, " reason=", attributes["reason"]], "")) where IsString(body) and attributes["reason"] != nil
-          - set(body, Concat([body, " failure_reason=", attributes["failure_reason"]], "")) where IsString(body) and attributes["failure_reason"] != nil
-          - set(body, Concat([body, " rpc_path=", attributes["rpc_path"]], "")) where IsString(body) and attributes["rpc_path"] != nil
-          - set(body, Concat([body, " rpc_service=", attributes["rpc_service"]], "")) where IsString(body) and attributes["rpc_service"] != nil
-          - set(body, Concat([body, " rpc_method=", attributes["rpc_method"]], "")) where IsString(body) and attributes["rpc_method"] != nil
-          - set(body, Concat([body, " expected_audience=", attributes["expected_audience"]], "")) where IsString(body) and attributes["expected_audience"] != nil
-          - set(body, Concat([body, " peer_addr=", attributes["peer_addr"]], "")) where IsString(body) and attributes["peer_addr"] != nil
-          - set(body, Concat([body, " replay_scope_bootstrap_allowed=", attributes["replay_scope_bootstrap_allowed"]], "")) where IsString(body) and attributes["replay_scope_bootstrap_allowed"] != nil
-          - set(body, Concat([body, " error=", attributes["error"]], "")) where IsString(body) and attributes["error"] != nil
-          - set(body, Concat([body, " exception_message=", attributes["exception.message"]], "")) where IsString(body) and attributes["exception.message"] != nil
+          - set(attributes["message"], body.string)
+          - set(attributes["log.body"], body.string)
 
 exporters:
   otlp/tempo:

.docker/observability/prometheus-rules/rustfs-kms-alerts.yml

@@ -1,251 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# =============================================================================
-# RustFS KMS backend — Prometheus alerting rules
-# =============================================================================
-#
-# Metric source: the KMS operation-policy choke point in
-# crates/kms/src/policy.rs, except KmsKeyRotationOverdue, which reads the
-# label-less key-lifecycle gauge published by the deletion worker's sweep
-# (crates/kms/src/deletion_worker.rs). All label values are bounded static
-# strings (operation, op_class, outcome, error_class, backend, scope); key
-# identifiers, key material, and tokens never appear in labels.
-#
-# Response procedures: docs/operations/kms-observability-runbook.md
-#
-# IMPORTANT — threshold status: every numeric threshold below is a
-# conservative default chosen without a production baseline. Calibrate against
-# a staging baseline before relying on these alerts for paging, and prefer
-# loosening over tightening until the baseline exists. Formal SLO targets are
-# deliberately not encoded here (see rustfs/backlog#1584).
-#
-# NOTE: prometheus.yml loads /etc/prometheus/rules/*.yml — keep the .yml
-# extension or the file is silently ignored by the docker-compose stack.
-#
-# Validate: promtool check rules rustfs-kms-alerts.yml
-# =============================================================================
-
-groups:
-  # ==========================================================================
-  # Critical alerts — immediate action required
-  # ==========================================================================
-  - name: rustfs-kms-critical
-    interval: 30s
-    rules:
-      # ------------------------------------------------------------------
-      # 1. KmsBackendFatalErrors
-      #    Any attempt failure classified as fatal (non-retryable): auth
-      #    or permission errors, malformed requests, missing keys. The
-      #    policy never retries these, so even a low rate means real
-      #    operations are failing right now.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendFatalErrors
-        expr: |
-          sum by (operation) (rate(rustfs_kms_backend_attempt_failures_total{error_class="fatal"}[5m])) > 0
-        for: 5m
-        labels:
-          severity: critical
-          component: kms
-        annotations:
-          summary: "KMS backend fatal errors on operation {{ $labels.operation }}"
-          description: >-
-            Attempt failures classified as fatal are occurring at
-            {{ $value | printf "%.3f" }}/s on operation
-            {{ $labels.operation }}. Fatal failures are not retried:
-            each one is a KMS backend call that failed permanently
-            (authentication, permissions, malformed request, or a
-            missing key/version).
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendfatalerrors"
-
-      # ------------------------------------------------------------------
-      # 2. KmsBackendHighErrorRate
-      #    Sustained share of operations terminating without success
-      #    (fatal, budget/deadline exhaustion, admission backpressure,
-      #    or an open circuit). The cancelled outcome is excluded because
-      #    shutdowns legitimately produce it.
-      #    The traffic guard keeps a single failure on a near-idle
-      #    cluster from firing the alert.
-      #    Threshold: 5% for 10m — conservative default, calibrate
-      #    against a staging baseline.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendHighErrorRate
-        expr: |
-          (
-            sum(rate(rustfs_kms_backend_operations_total{outcome!~"success|cancelled"}[5m]))
-            /
-            clamp_min(sum(rate(rustfs_kms_backend_operations_total[5m])), 1e-9)
-          ) > 0.05
-          and
-          sum(rate(rustfs_kms_backend_operations_total[5m])) > 0.02
-        for: 10m
-        labels:
-          severity: critical
-          component: kms
-        annotations:
-          summary: "KMS backend non-success ratio above 5% for 10m"
-          description: >-
-            {{ $value | humanizePercentage }} of KMS backend operations
-            are terminating in fatal, budget_exhausted,
-            deadline_exceeded, backpressure_timeout,
-            backpressure_rejected, or circuit_open. Object encryption
-            and decryption paths depending on the KMS are degraded or
-            failing.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendhigherrorrate"
-
-  # ==========================================================================
-  # Warning alerts — investigation needed
-  # ==========================================================================
-  - name: rustfs-kms-warning
-    interval: 30s
-    rules:
-      # ------------------------------------------------------------------
-      # 3. KmsBackendP99LatencyHigh
-      #    p99 wall-clock duration of whole operations (attempts plus
-      #    backoff) is sustained above 2 seconds. Because the histogram
-      #    includes retries, a high p99 usually means the retry policy
-      #    is absorbing backend failures, not that every call is slow.
-      #    Threshold: 2s for 10m — conservative default, calibrate
-      #    against a staging baseline.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendP99LatencyHigh
-        expr: |
-          histogram_quantile(0.99,
-            sum by (le) (rate(rustfs_kms_backend_operation_duration_seconds_bucket[5m]))
-          ) > 2
-        for: 10m
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "KMS backend operation p99 latency above 2s for 10m"
-          description: >-
-            The 99th-percentile KMS backend operation duration is
-            {{ $value | humanizeDuration }}, including retries and
-            backoff. Encryption and decryption latency is leaking into
-            S3 request latency.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendp99latencyhigh"
-
-      # ------------------------------------------------------------------
-      # 4. KmsBackendAttemptFailureSpike
-      #    Aggregate attempt-failure rate (all error classes) sustained
-      #    above an absolute floor. An absolute threshold is used instead
-      #    of an offset-1d baseline ratio because fresh deployments have
-      #    no baseline and an empty offset vector would keep a ratio
-      #    alert from ever firing; switch to a baseline-relative form
-      #    (see rustfs-get-optimization-alerts.yaml for the pattern)
-      #    once a stable staging baseline exists.
-      #    Threshold: 0.5/s for 10m — conservative default, calibrate
-      #    against a staging baseline.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendAttemptFailureSpike
-        expr: |
-          sum(rate(rustfs_kms_backend_attempt_failures_total[5m])) > 0.5
-        for: 10m
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "KMS backend attempt failures above 0.5/s for 10m"
-          description: >-
-            KMS backend attempts are failing at
-            {{ $value | printf "%.2f" }}/s across all error classes.
-            The retry policy may still be masking these from callers —
-            check the error-class breakdown before it stops absorbing
-            them.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendattemptfailurespike"
-
-      # ------------------------------------------------------------------
-      # 5. KmsBackendRetryBudgetExhausted
-      #    Operations are running out of retry budget (budget_exhausted)
-      #    or operation deadline (deadline_exceeded). These surface to
-      #    callers as failed KMS operations even though every individual
-      #    failure was retryable — the backend is unhealthy for longer
-      #    than the policy can bridge.
-      #    Threshold: 0.05/s for 10m — conservative default, calibrate
-      #    against a staging baseline.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendRetryBudgetExhausted
-        expr: |
-          sum by (outcome) (rate(rustfs_kms_backend_operations_total{outcome=~"budget_exhausted|deadline_exceeded"}[5m])) > 0.05
-        for: 10m
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "KMS backend operations exhausting retry budget ({{ $labels.outcome }})"
-          description: >-
-            KMS backend operations are terminating as
-            {{ $labels.outcome }} at {{ $value | printf "%.3f" }}/s.
-            Retryable failures are outlasting the retry budget, so
-            callers are seeing hard failures.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendretrybudgetexhausted"
-
-      # ------------------------------------------------------------------
-      # 6. KmsBackendCircuitOpen
-      #    Direct circuit-state signal, independent of operation traffic.
-      #    A transient open can recover on its first half-open probe; alert
-      #    only when the circuit remains open or half-open for one minute.
-      # ------------------------------------------------------------------
-      - alert: KmsBackendCircuitOpen
-        expr: |
-          rustfs_kms_backend_circuit_open > 0
-        for: 1m
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "KMS backend circuit open ({{ $labels.backend }}/{{ $labels.scope }})"
-          description: >-
-            The KMS backend circuit for {{ $labels.backend }} scope
-            {{ $labels.scope }} has remained open or half-open for one
-            minute. Operations in this scope can terminate as
-            circuit_open until the half-open probe succeeds or returns
-            a non-retryable failure.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmsbackendcircuitopen"
-
-      # ------------------------------------------------------------------
-      # 7. KmsKeyRotationOverdue
-      #    The least recently rotated usable key has gone more than 400
-      #    days without a rotation (measured from creation for keys with
-      #    no recorded rotation). Direct gauge state published by the
-      #    deletion worker's sweep, so no traffic guard applies; the
-      #    one-hour hold only bridges scrape gaps. The worker runs only
-      #    on backends with the schedule_deletion capability, so on the
-      #    Static backend the series never exists and this alert cannot
-      #    fire — that backend cannot rotate either; see the rotation
-      #    driver matrix in docs/operations/kms-backend-security.md.
-      #    Threshold: 400 days — conservative default sitting above a
-      #    one-year rotation policy. Align it with the rotation period
-      #    your compliance policy requires, and with
-      #    RUSTFS_KMS_ROTATION_MAX_AGE_SECS so the per-key rotation_due
-      #    verdict and this aggregate alert agree.
-      # ------------------------------------------------------------------
-      - alert: KmsKeyRotationOverdue
-        expr: |
-          rustfs_kms_oldest_key_rotation_age_seconds > (400 * 86400)
-        for: 1h
-        labels:
-          severity: warning
-          component: kms
-        annotations:
-          summary: "Oldest KMS key unrotated for more than 400 days"
-          description: >-
-            The least recently rotated usable KMS key was last rotated
-            {{ $value | humanizeDuration }} ago (measured from creation
-            for keys with no recorded rotation). List keys through the
-            admin API and read rotation_due / rotation_due_reason for
-            the per-key verdict; an "unsupported" reason means the
-            backend cannot rotate at all.
-          runbook_url: "https://github.com/rustfs/rustfs/blob/main/docs/operations/kms-observability-runbook.md#kmskeyrotationoverdue"

.docker/test/site-replication/run-object-flow-check.sh

@@ -18,7 +18,6 @@ set -eu
 ACCESS_KEY="${RUSTFS_SITE_REPL_ACCESS_KEY:-rustfsadmin}"
 SECRET_KEY="${RUSTFS_SITE_REPL_SECRET_KEY:-rustfsadmin}"
 BUCKET="${RUSTFS_SITE_REPL_FLOW_BUCKET:-site-repl-flow-check}"
-DELETE_BUCKET="${RUSTFS_SITE_REPL_DELETE_BUCKET:-site-repl-delete-$(date +%Y%m%d-%H%M%S)-$$}"
 PREFIX="${RUSTFS_SITE_REPL_FLOW_PREFIX:-flow-$(date +%Y%m%d-%H%M%S)}"
 WAIT_ATTEMPTS="${RUSTFS_SITE_REPL_WAIT_ATTEMPTS:-90}"
 WAIT_SLEEP_SECONDS="${RUSTFS_SITE_REPL_WAIT_SLEEP_SECONDS:-2}"
@@ -86,39 +85,17 @@ wait_for_object() {
 
 wait_for_bucket() {
   site="$1"
-  bucket="${2:-$BUCKET}"
   attempt=1
 
   while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
-    if mc stat "$site/$bucket" >/dev/null 2>&1; then
+    if mc stat "$site/$BUCKET" >/dev/null 2>&1; then
       return 0
     fi
     sleep "$WAIT_SLEEP_SECONDS"
     attempt=$((attempt + 1))
   done
 
-  echo "bucket was not replicated in time: $site/$bucket" >&2
-  return 1
-}
-
-wait_for_bucket_delete() {
-  site="$1"
-  bucket="$2"
-  attempt=1
-
-  while [ "$attempt" -le "$WAIT_ATTEMPTS" ]; do
-    if result="$(mc stat --json "$site/$bucket" 2>&1)"; then
-      :
-    else
-      case "$result" in
-        *NoSuchBucket*) return 0 ;;
-      esac
-    fi
-    sleep "$WAIT_SLEEP_SECONDS"
-    attempt=$((attempt + 1))
-  done
-
-  echo "bucket deletion was not replicated in time: $site/$bucket" >&2
+  echo "bucket was not replicated in time: $site/$BUCKET" >&2
   return 1
 }
 
@@ -209,20 +186,6 @@ EOF
   echo "verified replicated downloads for $object_name"
 done
 
-echo "creating empty bucket for replicated delete check: $DELETE_BUCKET"
-mc mb "site1/$DELETE_BUCKET" >/dev/null
-
-for site in site1 site2 site3; do
-  wait_for_bucket "$site" "$DELETE_BUCKET"
-done
-
-echo "deleting empty bucket on site1: $DELETE_BUCKET"
-mc rb "site1/$DELETE_BUCKET" >/dev/null
-
-for site in site1 site2 site3; do
-  wait_for_bucket_delete "$site" "$DELETE_BUCKET"
-done
-
 echo "site replication object flow check passed"
 echo "bucket: $BUCKET"
 echo "prefix: $PREFIX"

.github/actions/setup/action.yml

@@ -25,13 +25,9 @@ inputs:
     required: false
     default: "rustfs-deps"
   cache-save-if:
-    description: >-
-      Whether to save the cache. The fail-safe default is 'false': a caller that
-      wants to populate a cache must opt in explicitly, so a forgotten input
-      costs a cold cache (minutes) rather than silently consuming the
-      repository-wide 10GB Actions cache quota and evicting other lanes.
+    description: "Condition for saving cache"
     required: false
-    default: "false"
+    default: "true"
   install-cross-tools:
     description: "Install cross-compilation tools"
     required: false
@@ -40,52 +36,36 @@ inputs:
     description: "Target architecture to add"
     required: false
     default: ""
-  install-build-packaging-tools:
-    description: >-
-      Install musl-tools/zip/unzip, needed for musl linking and release
-      packaging. Off for CI test lanes, which use none of them.
+  github-token:
+    description: "GitHub token for API access"
     required: false
-    default: "true"
-  install-test-tools:
-    description: >-
-      Install cargo-nextest and the rustfmt/clippy components. Off for release
-      and audit lanes, which run no tests and no lints.
-    required: false
-    default: "true"
+    default: ""
 
 runs:
   using: "composite"
   steps:
-    # protobuf-compiler is deliberately absent: the setup-protoc step below
-    # installs 35.1 into the tool cache and prepends it to PATH, so the apt
-    # build (older, and never version-matched) was shadowed on every run and
-    # simply never used.
     - name: Install system dependencies (Ubuntu)
       if: runner.os == 'Linux'
       shell: bash
       run: |
         sudo apt-get update
         sudo apt-get install -y \
+          musl-tools \
           build-essential \
           pkg-config \
           libssl-dev \
-          ripgrep
-
-    # musl-gcc is needed by the native musl release leg, and zip/unzip by the
-    # release packaging steps. No CI test lane touches any of them.
-    - name: Install packaging and cross-linking dependencies (Ubuntu)
-      if: runner.os == 'Linux' && inputs.install-build-packaging-tools == 'true'
-      shell: bash
-      run: sudo apt-get install -y musl-tools zip unzip
+          ripgrep \
+          unzip \
+          protobuf-compiler
 
     - name: Install protoc
       uses: rustfs/setup-protoc@a3705324d8f9bf5b6c3573fb6cf8ae421db55dd6 # v3.0.1
       with:
-        version: "35.1"
+        version: "34.1"
         repo-token: ${{ github.token }}
 
     - name: Install flatc
-      uses: Nugine/setup-flatc@698800de72a96bfb22cf60431dc21a2ff9a7e07b # v1
+      uses: Nugine/setup-flatc@e7855e994773ce90094a3f1626d4afc9080c23ae # v1
       with:
         version: "25.12.19"
 
@@ -94,7 +74,7 @@ runs:
       with:
         toolchain: ${{ inputs.rust-version }}
         targets: ${{ inputs.target }}
-        components: ${{ inputs.install-test-tools == 'true' && 'rustfmt, clippy' || '' }}
+        components: rustfmt, clippy
 
     - name: Install Zig
       if: inputs.install-cross-tools == 'true'
@@ -105,24 +85,12 @@ runs:
       uses: taiki-e/install-action@a21ae4029b089b9ddc45704028756f51ab8abe48 # cargo-zigbuild
 
     - name: Install cargo-nextest
-      if: inputs.install-test-tools == 'true'
       uses: taiki-e/install-action@96c7780c1d8a2b8723e12031def873a434d39d8d # nextest
 
     - name: Setup Rust cache
       uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
       with:
-        # false is rust-cache's own default. With true, cleanup.ts returns
-        # *before* pruning ~/.cargo/registry/src, and config.ts archives the
-        # whole registry — so every cache carried the unpacked source tree of
-        # every dependency, not just "a few extra crates".
-        #
-        # No coverage is lost: getPackages runs `cargo metadata --all-features`,
-        # a strict superset of any single lane's feature closure, and -sys crates
-        # are explicitly exempted from pruning (their src timestamps would
-        # otherwise trigger rebuilds). Anything pruned is re-unpacked from the
-        # .crate files still in registry/cache, whose mtimes crates.io
-        # normalises, so cargo fingerprints stay valid.
-        cache-all-crates: false
+        cache-all-crates: true
         cache-on-failure: true
         shared-key: ${{ inputs.cache-shared-key }}
         save-if: ${{ inputs.cache-save-if }}

.github/cla.yml

@@ -15,8 +15,8 @@
 enabled: true
 
 document:
-  version: v2
-  url: https://github.com/rustfs/cla/blob/main/cla/v2.md
+  version: v1
+  url: https://github.com/rustfs/cla/blob/main/cla/v1.md
 
 signing:
   mode: comment

.github/pull_request_template.md

@@ -33,4 +33,4 @@ documentation impact. Use N/A when there is no expected impact.
 
 ---
 
-Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)). If this is your first contribution, review the [CLA document](https://github.com/rustfs/cla/blob/main/cla/v2.md) and sign it by commenting `I have read and agree to the CLA.` on the PR.
+Thank you for your contribution! Please ensure your PR follows the community standards ([CODE_OF_CONDUCT.md](CODE_OF_CONDUCT.md)). If this is your first contribution, review the [CLA document](https://github.com/rustfs/cla/blob/main/cla/v1.md) and sign it by commenting `I have read and agree to the CLA.` on the PR.

.github/s3tests/README.md

@@ -46,9 +46,10 @@ lists when upstream changes.
   the PR.
 - **Weekly + manual**: `.github/workflows/e2e-s3tests.yml` runs the full
   upstream suite (`TEST_SCOPE=all`) against a Docker deployment (single node
-  or a 4-node distributed cluster behind HAProxy). The canonical gate policy
-  and compatibility-report behavior are documented in
-  [`scripts/s3-tests/README.md`](../../scripts/s3-tests/README.md).
+  or a 4-node distributed cluster behind HAProxy). It fails only on
+  regressions in the implemented whitelist and publishes a classification
+  report (`compat-report.md`, also shown in the job summary) listing promotion
+  candidates and unclassified tests.
 
 ## Running Tests Locally
 

.github/workflows/architecture-migration-rules.yml

@@ -37,7 +37,6 @@ jobs:
     name: Cancel Closed PR Runs
     if: github.event_name == 'pull_request' && github.event.action == 'closed'
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - name: Explain cancellation run
         run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -46,16 +45,13 @@ jobs:
     name: Architecture Migration Rules
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Install ripgrep
-        uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
-        with:
-          tool: ripgrep@15.2.0
+        run: |
+          sudo apt-get update
+          sudo apt-get install -y ripgrep
 
       - name: Check architecture migration rules
         run: ./scripts/check_architecture_migration_rules.sh

.github/workflows/audit.yml

@@ -23,9 +23,6 @@ on:
       - 'deny.toml'
       - '.github/actions/**'
       - '.github/workflows/**'
-      - 'scripts/release/create_or_update_release.sh'
-      - 'scripts/security/check_performance_ab_workflow.sh'
-      - 'scripts/security/check_preview_release_workflow.sh'
       - 'scripts/security/check_workflow_pins.sh'
   pull_request:
     types: [ opened, synchronize, reopened, closed ]
@@ -36,17 +33,9 @@ on:
       - 'deny.toml'
       - '.github/actions/**'
       - '.github/workflows/**'
-      - 'scripts/release/create_or_update_release.sh'
-      - 'scripts/security/check_performance_ab_workflow.sh'
-      - 'scripts/security/check_preview_release_workflow.sh'
       - 'scripts/security/check_workflow_pins.sh'
   schedule:
-    # Daily, not weekly. This schedule exists to catch RustSec advisories
-    # published against an unchanged dependency tree; at weekly cadence a new
-    # advisory could sit unnoticed for seven days. The check list is unchanged —
-    # splitting it into a light daily advisories-only run and a weekly full run
-    # would create runs where sources/bans/licenses go unverified.
-    - cron: '0 3 * * *' # Daily 03:00 UTC (staggered after the midnight ci/build crons)
+    - cron: '0 3 * * 0' # Weekly on Sunday 03:00 UTC (staggered after the midnight ci/build crons)
   workflow_dispatch:
 
 permissions:
@@ -66,7 +55,6 @@ jobs:
     name: Cancel Closed PR Runs
     if: github.event_name == 'pull_request' && github.event.action == 'closed'
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - name: Explain cancellation run
         run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -82,32 +70,11 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
-      # cargo-deny compiles nothing, so the full setup composite (apt packages,
-      # protoc, flatc, nextest, rustfmt/clippy) was pure overhead here. It does
-      # still need a real cargo: `cargo deny check` runs `cargo metadata`, and
-      # Cargo.toml pins datafusion and s3s as git dependencies, which must be
-      # materialised into ~/.cargo/git — a cold clone is hundreds of MB, so the
-      # cache stays.
-      - name: Install Rust toolchain
-        uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
-
-      # Was relying on the composite's default, which used to be "true": every
-      # PR touching Cargo.toml/Cargo.lock saved a second, PR-scoped copy of this
-      # cache and pushed the main-scoped lanes out of the 10GB quota. The
-      # default is now "false", but state it explicitly — see
-      # scripts/security/check_cache_save_if.sh.
-      - name: Setup Rust cache
-        uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32 # v2
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
         with:
-          # Same reasoning as the setup composite: true archives every
-          # dependency's unpacked source tree.
-          cache-all-crates: false
-          cache-on-failure: true
-          shared-key: rustfs-cargo-deny
-          save-if: ${{ github.ref == 'refs/heads/main' }}
+          cache-shared-key: rustfs-cargo-deny
 
       - name: Install cargo-deny
         uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
@@ -125,31 +92,13 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Report unpinned GitHub Actions
         run: ./scripts/security/check_workflow_pins.sh --enforce
 
-      - name: Check setup cache-save-if is explicit
-        run: ./scripts/security/check_cache_save_if.sh
-
-      - name: Check every job declares a timeout
-        run: ./scripts/security/check_job_timeouts.sh
-
-      - name: Check checkouts clear their credentials
-        run: ./scripts/security/check_persist_credentials.sh
-
-      - name: Check preview release workflow policy
-        run: ./scripts/security/check_preview_release_workflow.sh
-
-      - name: Check performance A/B workflow trust boundary
-        run: ./scripts/security/check_performance_ab_workflow.sh
-
   dependency-review:
     name: Dependency Review
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     if: github.event_name == 'pull_request' && github.event.action != 'closed'
     permissions:
       contents: read
@@ -157,8 +106,6 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Dependency Review
         uses: actions/dependency-review-action@a1d282b36b6f3519aa1f3fc636f609c47dddb294 # v5
@@ -171,28 +118,3 @@ jobs:
           # conscious re-review of the license/provenance claim (backlog#1181).
           allow-dependencies-licenses: pkg:cargo/rustfs-uring@0.1.0
           comment-summary-in-pr: always
-
-  alert-on-failure:
-    name: Alert on scheduled failure
-    # dependency-review is deliberately excluded: it only runs on pull_request,
-    # so it can never contribute a failure to a scheduled run.
-    needs: [cargo-deny, workflow-pin-report]
-    # A scheduled cargo-deny failure usually means the dependency tree just
-    # matched a newly published advisory — the single most important signal this
-    # workflow produces, and until now it was only visible to whoever happened to
-    # open the Actions tab. Same ci-8 mechanism coverage.yml and
-    # e2e-replication-nightly.yml already use.
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    permissions:
-      contents: read
-      issues: write
-    steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-      - name: Open or update failure-tracking issue
-        uses: ./.github/actions/schedule-failure-issue
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/build.yml

@@ -50,18 +50,12 @@ on:
       - "**/*.svg"
       - ".gitignore"
       - ".dockerignore"
-      - "flake.lock"
   schedule:
     - cron: "0 1 * * 0" # Weekly on Sunday 01:00 UTC (staggered after the ci.yml midnight cron)
   workflow_dispatch:
     inputs:
       build_docker:
-        # Advisory only. docker.yml triggers on workflow_run and its job-level
-        # condition requires the triggering event to be a tag push, so a manual
-        # dispatch of this workflow never produces images regardless of this
-        # value. Kept because the summary step reports it; wiring it up would
-        # mean teaching docker.yml's version parser a second event shape.
-        description: "Build and push Docker images after binary build (ignored: dispatch runs never reach docker.yml)"
+        description: "Build and push Docker images after binary build"
         required: false
         default: true
         type: boolean
@@ -89,7 +83,6 @@ jobs:
   build-check:
     name: Build Strategy Check
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     outputs:
       should_build: ${{ steps.check.outputs.should_build }}
       build_type: ${{ steps.check.outputs.build_type }}
@@ -99,8 +92,6 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Determine build strategy
         id: check
@@ -116,21 +107,13 @@ jobs:
 
           # Determine build type based on trigger
           if [[ "${{ startsWith(github.ref, 'refs/tags/') }}" == "true" ]]; then
-            # Tag push - preview, release, or prerelease
+            # Tag push - release or prerelease
             should_build=true
             tag_name="${GITHUB_REF#refs/tags/}"
             version="${tag_name}"
 
-            # Preview tags publish a GitHub prerelease for validation, but
-            # must not update any latest channel.
-            if [[ "$tag_name" =~ -preview\.[0-9]+$ ]]; then
-              build_type="preview"
-              is_prerelease=true
-              echo "🔍 Preview build detected: $tag_name"
-            elif [[ "$tag_name" == *"-preview"* ]]; then
-              echo "❌ Invalid preview tag: $tag_name (expected suffix: -preview.<number>)" >&2
-              exit 1
-            elif [[ "$tag_name" == *"alpha"* ]] || [[ "$tag_name" == *"beta"* ]] || [[ "$tag_name" == *"rc"* ]]; then
+            # Check if this is a prerelease
+            if [[ "$tag_name" == *"alpha"* ]] || [[ "$tag_name" == *"beta"* ]] || [[ "$tag_name" == *"rc"* ]]; then
               build_type="prerelease"
               is_prerelease=true
               echo "🚀 Prerelease build detected: $tag_name"
@@ -153,13 +136,11 @@ jobs:
             echo "⚡ Manual/scheduled build detected"
           fi
 
-          {
-            echo "should_build=$should_build"
-            echo "build_type=$build_type"
-            echo "version=$version"
-            echo "short_sha=$short_sha"
-            echo "is_prerelease=$is_prerelease"
-          } >> "$GITHUB_OUTPUT"
+          echo "should_build=$should_build" >> $GITHUB_OUTPUT
+          echo "build_type=$build_type" >> $GITHUB_OUTPUT
+          echo "version=$version" >> $GITHUB_OUTPUT
+          echo "short_sha=$short_sha" >> $GITHUB_OUTPUT
+          echo "is_prerelease=$is_prerelease" >> $GITHUB_OUTPUT
 
           echo "📊 Build Summary:"
           echo "  - Should build: $should_build"
@@ -173,7 +154,6 @@ jobs:
     name: Prepare Platform Matrix
     needs: build-check
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     outputs:
       matrix: ${{ steps.select.outputs.matrix }}
       selected: ${{ steps.select.outputs.selected }}
@@ -181,14 +161,10 @@ jobs:
       - name: Select target platforms
         id: select
         shell: bash
-        env:
-          # via env, not interpolation: a dispatch input is free-form text and
-          # would otherwise be pasted into the script for bash to evaluate.
-          RAW_PLATFORMS: ${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}
         run: |
           set -euo pipefail
 
-          selected="$RAW_PLATFORMS"
+          selected="${{ github.event_name == 'workflow_dispatch' && github.event.inputs.platforms || 'all' }}"
           selected="$(echo "${selected}" | tr -d '[:space:]')"
           if [[ -z "${selected}" ]]; then
             selected="all"
@@ -212,6 +188,7 @@ jobs:
             {"target_id":"linux-x86_64-gnu","os":"sm-standard-2","target":"x86_64-unknown-linux-gnu","cross":false,"platform":"linux","rustflags":""},
             {"target_id":"linux-aarch64-gnu","os":"sm-standard-2","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":""},
             {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
+            {"target_id":"macos-x86_64","os":"macos-26-intel","target":"x86_64-apple-darwin","cross":false,"platform":"macos","rustflags":""},
             {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":""}
           ]}'
 
@@ -243,8 +220,8 @@ jobs:
     name: Build RustFS
     needs: [ build-check, prepare-platform-matrix ]
     if: needs.build-check.outputs.should_build == 'true' && needs.prepare-platform-matrix.result == 'success'
-    runs-on: ${{ matrix.os }}
-    timeout-minutes: 150
+    runs-on: ${{ matrix.platform == 'linux' && fromJSON('["self-hosted","linux","sm-standard-2"]') || matrix.os }}
+    timeout-minutes: 90
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
       # Release binaries ship without dial9 telemetry and therefore do not need
@@ -259,7 +236,6 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
         with:
-          persist-credentials: false
           fetch-depth: 0
 
       - name: Setup Rust environment
@@ -268,17 +244,9 @@ jobs:
           rust-version: stable
           target: ${{ matrix.target }}
           cache-shared-key: build-${{ matrix.target }}
-          # main only. A cache saved on refs/tags/X is scoped to that tag: no
-          # other tag, no main run and no PR can restore it, so every release
-          # cycle wrote up to 12 entries of 1-2GB (preview tag plus final tag,
-          # six legs each) that nobody could read, evicting the hot lanes from
-          # the repo-wide 10GB quota. Tag builds still restore the main-scoped
-          # cache, since default-branch caches are readable from every ref.
-          # The one real cost: re-running a failed leg of the same tag no longer
-          # finds that tag's own warm cache and falls back to main's.
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' || startsWith(github.ref, 'refs/tags/') }}
           install-cross-tools: ${{ matrix.cross }}
-          install-test-tools: 'false'
 
       - name: Download static console assets
         shell: bash
@@ -320,7 +288,6 @@ jobs:
             local console_url
             local console_sha256
             local curl_auth_args=()
-            console_tag=""
 
             if [[ "${{ matrix.platform }}" == "windows" ]]; then
               curl_bin="curl.exe"
@@ -343,7 +310,7 @@ jobs:
             "$curl_bin" "${curl_auth_args[@]}" --fail -L "$console_api" \
               -o "$console_json" --retry 3 --retry-delay 5 --max-time 300 || return 1
 
-            read -r console_tag console_url console_sha256 < <("$python_bin" - "$console_json" <<'PY'
+            read -r console_url console_sha256 < <("$python_bin" - "$console_json" <<'PY'
           import json
           import re
           import sys
@@ -351,8 +318,6 @@ jobs:
           with open(sys.argv[1], encoding="utf-8") as handle:
               release = json.load(handle)
 
-          tag = release.get("tag_name", "") or "unknown"
-
           for asset in release.get("assets", []):
               name = asset.get("name", "")
               digest = asset.get("digest", "")
@@ -361,7 +326,7 @@ jobs:
                   sha256 = digest.split(":", 1)[1]
                   if not re.fullmatch(r"[0-9a-fA-F]{64}", sha256):
                       raise SystemExit(f"console zip asset has invalid sha256 digest: {sha256}")
-                  sys.stdout.buffer.write(f"{tag} {url} {sha256}\n".encode("utf-8"))
+                  sys.stdout.buffer.write(f"{url} {sha256}\n".encode("utf-8"))
                   break
           else:
               raise SystemExit("no console zip asset with sha256 digest found")
@@ -373,8 +338,6 @@ jobs:
               return 1
             fi
 
-            echo "Console release: ${console_tag}"
-            echo "Downloading console asset: ${console_url}"
             "$curl_bin" --fail -L "$console_url" -o console.zip --retry 3 --retry-delay 5 --max-time 300 || return 1
             verify_sha256 "$console_sha256" console.zip || return 2
             unzip -o console.zip -d ./rustfs/static || return 2
@@ -387,19 +350,12 @@ jobs:
             rm -f console.zip console-release.json
             if [[ "$status" -eq 2 ]]; then
               echo "Console asset integrity verification failed" >&2
+              exit 1
             fi
-            echo "Failed to download verified console assets" >&2
-            exit 1
+            echo "Warning: Failed to download verified console assets, continuing without them"
+            echo "// Static assets not available" > ./rustfs/static/empty.txt
           fi
 
-          if [[ ! -s ./rustfs/static/index.html ]]; then
-            echo "Console asset archive is missing static/index.html" >&2
-            exit 1
-          fi
-
-          asset_count=$(find ./rustfs/static -type f | wc -l | tr -d '[:space:]')
-          echo "Console assets ready: version=${console_tag:-unknown}, ${asset_count} files extracted to ./rustfs/static"
-
       - name: Build RustFS
         shell: bash
         run: |
@@ -481,11 +437,11 @@ jobs:
             # Release/Prerelease build: rustfs-${platform}-${arch}-${variant}-v${version}.zip
             PACKAGE_NAME="${PACKAGE_BASENAME}-v${PACKAGE_VERSION}"
           fi
-          
+
           # Create zip packages for all platforms
           # Ensure zip is available
           if ! command -v zip &> /dev/null; then
-            if [[ "${{ matrix.os }}" == "ubuntu-latest" || "${{ matrix.platform }}" == "linux" ]]; then
+            if [[ "${{ matrix.os }}" == "ubuntu-latest" ]]; then
               sudo apt-get update && sudo apt-get install -y zip
             fi
           fi
@@ -537,7 +493,7 @@ jobs:
             if [[ "${{ matrix.platform }}" == "windows" ]]; then
               dir
             else
-              ls -lh "${PACKAGE_NAME}.zip"
+              ls -lh ${PACKAGE_NAME}.zip
             fi
           else
             echo "❌ Failed to create package: ${PACKAGE_NAME}.zip"
@@ -586,13 +542,11 @@ jobs:
             fi
           fi
 
-          {
-            echo "package_name=${PACKAGE_NAME}"
-            echo "package_file=${PACKAGE_NAME}.zip"
-            echo "latest_files=${LATEST_FILES}"
-            echo "build_type=${BUILD_TYPE}"
-            echo "version=${VERSION}"
-          } >> "$GITHUB_OUTPUT"
+          echo "package_name=${PACKAGE_NAME}" >> $GITHUB_OUTPUT
+          echo "package_file=${PACKAGE_NAME}.zip" >> $GITHUB_OUTPUT
+          echo "latest_files=${LATEST_FILES}" >> $GITHUB_OUTPUT
+          echo "build_type=${BUILD_TYPE}" >> $GITHUB_OUTPUT
+          echo "version=${VERSION}" >> $GITHUB_OUTPUT
 
           echo "📦 Package created: ${PACKAGE_NAME}.zip"
           if [[ -n "$LATEST_FILES" ]]; then
@@ -601,60 +555,6 @@ jobs:
           echo "🔧 Build type: ${BUILD_TYPE}"
           echo "📊 Version: ${VERSION}"
 
-      - name: Verify packaged console
-        if: matrix.target == 'x86_64-unknown-linux-gnu'
-        shell: bash
-        run: |
-          set -euo pipefail
-
-          find_free_port() {
-            python3 - <<'PY'
-          import socket
-
-          with socket.socket() as sock:
-              sock.bind(("127.0.0.1", 0))
-              print(sock.getsockname()[1])
-          PY
-          }
-
-          package_dir="$(mktemp -d)"
-          data_dir="$(mktemp -d)"
-          log_file="${RUNNER_TEMP}/rustfs-console-smoke.log"
-          server_pid=""
-          trap '
-            if [[ -n "${server_pid:-}" ]]; then
-              kill "$server_pid" 2>/dev/null || true
-              wait "$server_pid" 2>/dev/null || true
-            fi
-            rm -rf "$package_dir" "$data_dir"
-          ' EXIT
-
-          unzip -q "${{ steps.package.outputs.package_file }}" -d "$package_dir"
-          api_port="$(find_free_port)"
-          console_port="$(find_free_port)"
-          console_url="http://127.0.0.1:${console_port}/rustfs/console/"
-
-          "$package_dir/rustfs" server \
-            --address "127.0.0.1:${api_port}" \
-            --console-enable \
-            --console-address "127.0.0.1:${console_port}" \
-            --access-key console-smoke \
-            --secret-key console-smoke-secret \
-            "$data_dir" >"$log_file" 2>&1 &
-          server_pid=$!
-
-          for _ in {1..40}; do
-            response="$(curl --silent --output /dev/null --write-out '%{http_code} %{content_type}' "$console_url" || true)"
-            if [[ "$response" == 200\ text/html* ]]; then
-              exit 0
-            fi
-            sleep 0.25
-          done
-
-          echo "Console endpoint did not return 200 text/html: $response" >&2
-          cat "$log_file"
-          exit 1
-
       - name: Upload to GitHub artifacts
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
         with:
@@ -679,19 +579,9 @@ jobs:
             exit 0
           fi
 
-          # The self-hosted Linux runners do not ship the aws CLI (GitHub-hosted
-          # images did). Install it on demand so R2 uploads survive a fresh runner
-          # instead of hard-failing here.
           if ! command -v aws >/dev/null 2>&1; then
-            echo "aws CLI not found on runner; installing..."
-            if command -v apt-get >/dev/null 2>&1; then
-              sudo apt-get update && sudo apt-get install -y awscli
-            elif command -v brew >/dev/null 2>&1; then
-              brew install awscli
-            else
-              echo "❌ aws CLI missing and no apt-get/brew to install it; cannot upload to R2"
-              exit 1
-            fi
+            echo "❌ aws CLI not found on runner; cannot upload to R2"
+            exit 1
           fi
 
           export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
@@ -725,14 +615,9 @@ jobs:
     needs: [ build-check, build-rustfs ]
     if: always() && needs.build-check.outputs.should_build == 'true'
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - name: Build completion summary
         shell: bash
-        env:
-          # dispatch input via env: free-form text must not be pasted into the
-          # script for bash to evaluate.
-          INPUT_BUILD_DOCKER: ${{ github.event.inputs.build_docker }}
         run: |
           BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
           VERSION="${{ needs.build-check.outputs.version }}"
@@ -750,10 +635,6 @@ jobs:
           echo ""
 
           case "$BUILD_TYPE" in
-            "preview")
-              echo "🔍 Preview artifacts are published in a GitHub prerelease"
-              echo "⏭️  Preview releases do not update latest channels"
-              ;;
             "development")
               echo "🛠️  Development build artifacts have been uploaded to OSS dev directory"
               echo "⚠️  This is a development build - not suitable for production use"
@@ -772,9 +653,7 @@ jobs:
 
           echo ""
           echo "🐳 Docker Images:"
-          if [[ "$BUILD_TYPE" == "preview" ]]; then
-            echo "⏭️  Preview tags do not publish Docker images"
-          elif [[ "$INPUT_BUILD_DOCKER" == "false" ]]; then
+          if [[ "${{ github.event.inputs.build_docker }}" == "false" ]]; then
             echo "⏭️  Docker image build was skipped (binary only build)"
           elif [[ "$BUILD_STATUS" == "success" ]]; then
             echo "🔄 Docker images will be built and pushed automatically via workflow_run event"
@@ -782,13 +661,12 @@ jobs:
             echo "❌ Docker image build will be skipped due to build failure"
           fi
 
-  # Create GitHub Release for every valid release tag, including previews
+  # Create GitHub Release (only for tag pushes)
   create-release:
     name: Create GitHub Release
     needs: [ build-check, build-rustfs ]
-    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
+    if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     permissions:
       contents: write
     outputs:
@@ -798,7 +676,6 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
         with:
-          persist-credentials: false
           fetch-depth: 0
 
       - name: Create GitHub Release
@@ -811,12 +688,9 @@ jobs:
           VERSION="${{ needs.build-check.outputs.version }}"
           IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
           BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
-          TARGET_COMMITISH=$(git rev-parse --verify "refs/tags/${TAG}^{commit}")
 
           # Determine release type for title
-          if [[ "$BUILD_TYPE" == "preview" ]]; then
-            RELEASE_TYPE="preview"
-          elif [[ "$BUILD_TYPE" == "prerelease" ]]; then
+          if [[ "$BUILD_TYPE" == "prerelease" ]]; then
             if [[ "$TAG" == *"alpha"* ]]; then
               RELEASE_TYPE="alpha"
             elif [[ "$TAG" == *"beta"* ]]; then
@@ -830,34 +704,61 @@ jobs:
             RELEASE_TYPE="release"
           fi
 
-          # Create release title
-          if [[ "$IS_PRERELEASE" == "true" ]]; then
-            TITLE="RustFS $VERSION (${RELEASE_TYPE})"
+          # Check if release already exists
+          if gh release view "$TAG" >/dev/null 2>&1; then
+            echo "Release $TAG already exists"
+            RELEASE_ID=$(gh release view "$TAG" --json databaseId --jq '.databaseId')
+            RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
           else
-            TITLE="RustFS $VERSION"
+            # Get release notes from tag message
+            RELEASE_NOTES=$(git tag -l --format='%(contents)' "${TAG}")
+            if [[ -z "$RELEASE_NOTES" || "$RELEASE_NOTES" =~ ^[[:space:]]*$ ]]; then
+              if [[ "$IS_PRERELEASE" == "true" ]]; then
+                RELEASE_NOTES="Pre-release ${VERSION} (${RELEASE_TYPE})"
+              else
+                RELEASE_NOTES="Release ${VERSION}"
+              fi
+            fi
+
+            # Create release title
+            if [[ "$IS_PRERELEASE" == "true" ]]; then
+              TITLE="RustFS $VERSION (${RELEASE_TYPE})"
+            else
+              TITLE="RustFS $VERSION"
+            fi
+
+            # Create the release
+            PRERELEASE_FLAG=""
+            if [[ "$IS_PRERELEASE" == "true" ]]; then
+              PRERELEASE_FLAG="--prerelease"
+            fi
+
+            gh release create "$TAG" \
+              --title "$TITLE" \
+              --notes "$RELEASE_NOTES" \
+              $PRERELEASE_FLAG \
+              --draft
+
+            RELEASE_ID=$(gh release view "$TAG" --json databaseId --jq '.databaseId')
+            RELEASE_URL=$(gh release view "$TAG" --json url --jq '.url')
           fi
 
-          ./scripts/release/create_or_update_release.sh \
-            "$TAG" \
-            "$TARGET_COMMITISH" \
-            "$TITLE" \
-            "$IS_PRERELEASE"
+          echo "release_id=$RELEASE_ID" >> $GITHUB_OUTPUT
+          echo "release_url=$RELEASE_URL" >> $GITHUB_OUTPUT
+          echo "Created release: $RELEASE_URL"
 
   # Prepare and upload release assets
   upload-release-assets:
     name: Upload Release Assets
     needs: [ build-check, build-rustfs, create-release ]
-    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
+    if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     permissions:
       contents: write
       actions: read
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Download all build artifacts
         uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
@@ -892,9 +793,9 @@ jobs:
           cd ./release-assets
 
           # Generate checksums for all files (including latest versions)
-          if compgen -G "*.zip" >/dev/null; then
-            sha256sum -- *.zip > SHA256SUMS
-            sha512sum -- *.zip > SHA512SUMS
+          if ls *.zip >/dev/null 2>&1; then
+            sha256sum *.zip > SHA256SUMS
+            sha512sum *.zip > SHA512SUMS
           fi
 
           cd ..
@@ -934,16 +835,13 @@ jobs:
 
           echo "✅ All assets uploaded successfully"
 
-  # Update latest.json for every release tag (stable and prerelease): the
-  # project currently ships prerelease tags only, so gating this to stable
-  # left the version pointer permanently stale. release_type records whether
-  # the pointed-to version is a prerelease.
+  # Update latest.json for stable releases only: prerelease tags (alpha/beta/
+  # rc) must never overwrite the stable version pointer.
   update-latest-version:
     name: Update Latest Version
-    needs: [ build-check, publish-release ]
-    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
+    needs: [ build-check, upload-release-assets ]
+    if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type == 'release'
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     steps:
       - name: Update latest.json
         env:
@@ -961,12 +859,6 @@ jobs:
           VERSION="${{ needs.build-check.outputs.version }}"
           TAG="${{ needs.build-check.outputs.version }}"
 
-          if [[ "${{ needs.build-check.outputs.build_type }}" == "prerelease" ]]; then
-            RELEASE_TYPE="prerelease"
-          else
-            RELEASE_TYPE="stable"
-          fi
-
           # Install ossutil
           OSSUTIL_VERSION="2.1.1"
           OSSUTIL_ZIP="ossutil-${OSSUTIL_VERSION}-linux-amd64.zip"
@@ -987,7 +879,7 @@ jobs:
             "version": "${VERSION}",
             "tag": "${TAG}",
             "release_date": "$(date -u +%Y-%m-%dT%H:%M:%SZ)",
-            "release_type": "${RELEASE_TYPE}",
+            "release_type": "stable",
             "download_url": "https://github.com/${{ github.repository }}/releases/tag/${TAG}"
           }
           EOF
@@ -995,40 +887,57 @@ jobs:
           # Upload to OSS
           "$OSSUTIL_BIN" cp latest.json oss://rustfs-version/latest.json --force
 
-          echo "✅ Updated latest.json for ${RELEASE_TYPE} release $VERSION"
+          echo "✅ Updated latest.json for stable release $VERSION"
 
   # Publish release (remove draft status)
   publish-release:
     name: Publish Release
     needs: [ build-check, create-release, upload-release-assets ]
-    if: startsWith(github.ref, 'refs/tags/') && (needs.build-check.outputs.build_type == 'preview' || needs.build-check.outputs.build_type == 'release' || needs.build-check.outputs.build_type == 'prerelease')
+    if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type != 'development'
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     permissions:
       contents: write
     steps:
-      - name: Publish release
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Update release notes and publish
         env:
           GH_TOKEN: ${{ github.token }}
         shell: bash
         run: |
           TAG="${{ needs.build-check.outputs.version }}"
+          VERSION="${{ needs.build-check.outputs.version }}"
+          IS_PRERELEASE="${{ needs.build-check.outputs.is_prerelease }}"
           BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
-          RELEASE_ID="${{ needs.create-release.outputs.release_id }}"
 
-          # Publish the release and correct its channel state on retries.
-          # Only a stable final release may become GitHub Latest.
-          if [[ "$BUILD_TYPE" == "release" ]]; then
-            gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-              -F draft=false \
-              -F prerelease=false \
-              -f make_latest=true >/dev/null
+          # Determine release type
+          if [[ "$BUILD_TYPE" == "prerelease" ]]; then
+            if [[ "$TAG" == *"alpha"* ]]; then
+              RELEASE_TYPE="alpha"
+            elif [[ "$TAG" == *"beta"* ]]; then
+              RELEASE_TYPE="beta"
+            elif [[ "$TAG" == *"rc"* ]]; then
+              RELEASE_TYPE="rc"
+            else
+              RELEASE_TYPE="prerelease"
+            fi
           else
-            gh api --method PATCH "repos/${GITHUB_REPOSITORY}/releases/${RELEASE_ID}" \
-              -F draft=false \
-              -F prerelease=true \
-              -f make_latest=false >/dev/null
+            RELEASE_TYPE="release"
           fi
 
+          # Get original release notes from tag
+          ORIGINAL_NOTES=$(git tag -l --format='%(contents)' "${TAG}")
+          if [[ -z "$ORIGINAL_NOTES" || "$ORIGINAL_NOTES" =~ ^[[:space:]]*$ ]]; then
+            if [[ "$IS_PRERELEASE" == "true" ]]; then
+              ORIGINAL_NOTES="Pre-release ${VERSION} (${RELEASE_TYPE})"
+            else
+              ORIGINAL_NOTES="Release ${VERSION}"
+            fi
+          fi
+
+          # Publish the release (remove draft status)
+          gh release edit "$TAG" --draft=false
+
           echo "🎉 Released $TAG successfully!"
           echo "📄 Release URL: ${{ needs.create-release.outputs.release_url }}"

.github/workflows/cache-warm.yml

@@ -1,270 +0,0 @@
-# Copyright 2026 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Sole writer of the Rust dependency caches that ci.yml restores.
-#
-# Why this is a separate workflow rather than steps inside ci.yml: ci.yml's
-# concurrency group cancels in-progress runs on main pushes, and merges land far
-# faster than its 70-minute pipeline. Measured over 15 consecutive main pushes:
-# 12 cancelled, 2 failed, 0 succeeded. A cancelled run never reaches
-# Swatinem/rust-cache's post step (cache-on-failure does not cover cancellation),
-# so the writer lanes were saving nothing and every PR paid a cold restore —
-# 11.8-20.9 minutes of "Setup Rust environment" against 0.7-3.4 warm.
-#
-# Splitting cache writing out of the test pipeline lets ci.yml keep cancelling
-# superseded runs (which is correct — nobody needs test results for a commit
-# that is already three merges behind) while the caches still get written.
-#
-# The group below deliberately does NOT cancel in progress; see the comment on
-# it for how that bounds concurrency and why it is scoped by event.
-#
-# Each job below owns exactly one shared-key and is the only place that sets
-# cache-save-if to anything but 'false' for it; every lane in ci.yml reads.
-# scripts/security/check_cache_save_if.sh keeps the declarations explicit.
-#
-# The builds are supersets of what the reading lanes compile, because a reader
-# restores only what the writer saved. Feature resolution matters here: a lane
-# built with e2e-test-hooks resolves dependency features differently, which
-# changes -Cmetadata, so the plain build does not cover it. See
-# rustfs/backlog#1600.
-
-name: Cache Warm
-
-on:
-  push:
-    branches: [ main ]
-    # Mirrors ci.yml's push paths-ignore: if a commit cannot change what ci.yml
-    # compiles, it cannot change what ci.yml needs restored either.
-    paths-ignore:
-      - "**.md"
-      - "docs/**"
-      - "deploy/**"
-      - "scripts/dev_*.sh"
-      - "scripts/probe.sh"
-      - "LICENSE*"
-      - ".gitignore"
-      - ".dockerignore"
-      - "README*"
-      - "**/*.png"
-      - "**/*.jpg"
-      - "**/*.svg"
-      - ".github/workflows/build.yml"
-      - ".github/workflows/docker.yml"
-      - ".github/workflows/audit.yml"
-  workflow_dispatch:
-    inputs:
-      emit_timings:
-        description: >-
-          Also emit cargo --timings for the ci-dev build and upload it. Used to
-          decide whether sccache is worth adopting (rustfs/backlog#1601 gate).
-        required: false
-        default: false
-        type: boolean
-
-permissions:
-  contents: read
-
-# Scoped by event. A push run and a dispatch run do not compete: GitHub keeps
-# one running plus one pending per group, so with a single shared group a
-# manually dispatched run was displaced as pending by the next merge and
-# cancelled — observed three times in a row, which made the --timings gate in
-# rustfs/backlog#1601 effectively impossible to trigger while main was busy.
-#
-# Still no cancel-in-progress: a burst of merges collapses into "current run
-# finishes, newest queued run follows" rather than a pile-up, which is what
-# bounds this workflow to one self-hosted runner per event type.
-#
-# The two paths can now overlap and race to save the same key. That is benign:
-# the loser finds the key already present and skips, and both builds produce the
-# same artifacts from the same commit.
-concurrency:
-  group: cache-warm-${{ github.event_name }}
-  cancel-in-progress: false
-
-env:
-  CARGO_TERM_COLOR: always
-
-jobs:
-  # Readers: test-and-lint, test-ilm-integration-serial, build-rustfs-debug-binary,
-  # e2e-tests, e2e-full.
-  warm-ci-dev:
-    name: Warm ci-dev
-    runs-on: sm-standard-4
-    timeout-minutes: 90
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-dev
-          cache-save-if: 'true'
-          install-build-packaging-tools: 'false'
-
-      # rustfs/backlog#1601 gate. sccache can only cache compilation units whose
-      # --emit includes link, so it covers workspace rlibs and nothing else:
-      # clippy is metadata-only, and the ~100 test binaries, the rustfs bin and
-      # every build script invoke the system linker. Before spending a bucket,
-      # credentials and a supply-chain boundary on it, measure how much of the
-      # build is actually rlib codegen.
-      #
-      # Read from the report: workspace lib codegen as a share of the build, and
-      # s3select-query's own rlib as a share. The plan adopts sccache only above
-      # 50% and 25% respectively; if linking dominates instead, the answer is
-      # mold/lld plus split-debuginfo, which is exactly the part sccache cannot
-      # touch. Off by default — this doubles the ci-dev build.
-      - name: Build ci-dev superset (with --timings)
-        if: inputs.emit_timings
-        env:
-          CARGO_BUILD_JOBS: "2"
-        run: cargo build --workspace --all-targets --timings
-
-      - name: Upload cargo timings report
-        if: inputs.emit_timings
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: cargo-timings-ci-dev
-          path: target/cargo-timings/
-          retention-days: 30
-          if-no-files-found: error
-
-      # --all-targets covers the test binaries nextest builds, including
-      # e2e_test, which test-and-lint's own run excludes. The second build adds
-      # the e2e-test-hooks feature resolution that build-rustfs-debug-binary uses
-      # and that no lint lane enables.
-      - name: Build ci-dev superset
-        env:
-          # Same limit ci.yml puts on its nextest step: this builds the same
-          # ~100 workspace test binaries, and three concurrent links saturate the
-          # self-hosted runner's overlay I/O and can wedge Cargo (#5394).
-          CARGO_BUILD_JOBS: "2"
-        run: |
-          cargo build --workspace --all-targets
-          cargo build -p rustfs --bins --features e2e-test-hooks
-
-      # Runs before rust-cache's post step, so these are the sizes it is about
-      # to archive. Reported so the cache-all-crates decision stays evidence-led:
-      # registry/src is what that flag prunes, registry/cache is what the pruned
-      # sources are re-unpacked from. See rustfs/backlog#1600.
-      - name: Report cache input sizes
-        if: always()
-        run: |
-          # tee, not a plain redirect: sent only to $GITHUB_STEP_SUMMARY these
-          # numbers are readable in the UI but absent from the job log, and the
-          # REST API exposes the log, not the summary — which made the figures
-          # unreachable for exactly the scripted comparison they exist for.
-          sizes="$(du -sh ~/.cargo/registry/src ~/.cargo/registry/cache \
-                          ~/.cargo/registry/index ~/.cargo/git target 2>/dev/null || true)"
-          echo "cache-input-sizes-begin"
-          printf '%s\n' "$sizes"
-          echo "cache-input-sizes-end"
-          {
-            echo "### Cache input sizes (ci-dev)"
-            echo '```'
-            printf '%s\n' "$sizes"
-            echo '```'
-          } >> "$GITHUB_STEP_SUMMARY"
-
-  # Readers: test-and-lint-rio-v2 (per-PR), build-rustfs-debug-binary-rio-v2
-  # (weekly schedule / manual dispatch only — dormant rio-v2 variant, see
-  # rustfs/backlog#1835 and docs/architecture/minio-file-format-compat.md).
-  # The second build below stays despite the reduced cadence: it warms the
-  # rio-v2,e2e-test-hooks feature resolution the scheduled build restores,
-  # which keeps that lane inside its 30-minute timeout.
-  warm-ci-feat-rio:
-    name: Warm ci-feat-rio
-    runs-on: sm-standard-4
-    timeout-minutes: 90
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-feat-rio
-          cache-save-if: 'true'
-          install-build-packaging-tools: 'false'
-
-      - name: Build ci-feat-rio superset
-        run: |
-          cargo build -p rustfs -p rustfs-ecstore --all-targets --features rio-v2
-          cargo build -p rustfs --bins --features rio-v2,e2e-test-hooks
-
-  # Readers: the swift and sftp legs of test-and-lint-protocols. Built in
-  # sequence rather than as `--features swift,sftp`, which is a combination no
-  # lane actually compiles; running both leaves the union in target/.
-  warm-ci-feat-proto:
-    name: Warm ci-feat-proto
-    runs-on: sm-standard-4
-    timeout-minutes: 90
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-feat-proto
-          cache-save-if: 'true'
-          install-build-packaging-tools: 'false'
-
-      - name: Build ci-feat-proto superset
-        run: |
-          cargo build -p rustfs -p rustfs-protocols --all-targets --features swift
-          cargo build -p rustfs -p rustfs-protocols --all-targets --features sftp
-
-  # Reader: uring-integration. Runs on ubuntu-latest to match it: rust-cache's
-  # key covers runner.os and arch but not the runner label or image, so a cache
-  # written on sm-standard-4 would be restored by the hosted runner as if it
-  # belonged to it.
-  warm-ci-uring:
-    name: Warm ci-uring
-    runs-on: ubuntu-latest
-    timeout-minutes: 60
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-uring
-          cache-save-if: 'true'
-          install-build-packaging-tools: 'false'
-
-      - name: Install build dependencies
-        run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
-
-      - name: Build ci-uring superset
-        run: cargo build -p rustfs-ecstore --all-targets

.github/workflows/ci-docs-only.yml

@@ -12,24 +12,18 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# Companion to ci.yml for required status checks.
+# Companion to ci.yml for the required "Test and Lint" status check.
 #
-# ci.yml skips docs-only pull requests via paths-ignore, but the branch ruleset
-# requires a check named "Test and Lint" — without this workflow a docs-only PR
-# would wait on it forever. This workflow triggers on exactly the paths ci.yml
-# ignores and reports success under the same job name. Mixed PRs trigger both
-# workflows and the real check still gates: a required check with any failing
-# run blocks the merge.
+# ci.yml skips docs-only pull requests via paths-ignore, but the branch
+# ruleset requires a check named "Test and Lint" — without this workflow a
+# docs-only PR would wait on that check forever. This workflow triggers on
+# exactly the paths ci.yml ignores and reports an instant success under the
+# same job name. Mixed PRs trigger both workflows and the real check still
+# gates: a required check with any failing run blocks the merge.
 # https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/troubleshooting-required-status-checks#handling-skipped-but-required-checks
 #
-# "Quick Checks" is mirrored here ahead of the ruleset change that will make it
-# required too (rustfs/backlog#1599). Until that change lands this job is
-# inert; mirroring it first is what lets the ruleset change happen without
-# stranding docs-only PRs on a check nobody reports.
-#
 # Keep the paths list below in sync with the pull_request paths-ignore list
-# in ci.yml, and keep the quick-checks steps below byte-identical to the
-# quick-checks job in ci.yml.
+# in ci.yml.
 
 name: Continuous Integration (docs only)
 
@@ -53,99 +47,17 @@ on:
       - ".github/workflows/build.yml"
       - ".github/workflows/docker.yml"
       - ".github/workflows/audit.yml"
-      - "flake.lock"
 
 permissions:
   contents: read
 
 jobs:
-  # Deliberately NOT a bare `echo`. Once "Quick Checks" becomes a required
-  # check, ci.yml gates every expensive job behind it, so a mixed PR reports
-  # two check runs with this name: the real one (45-51s) and this companion.
-  # GitHub has no written contract for how it picks between same-named
-  # required check runs ("latest wins" vs "any failure blocks"), so instead of
-  # relying on ordering we make both runs execute the same commands against
-  # the same merge ref — their conclusions are then necessarily identical and
-  # the choice does not matter. Keep these steps byte-identical to the
-  # quick-checks job in ci.yml (a guard script that asserts this, and the paths
-  # sync below, is tracked in rustfs/backlog#1603).
-  #
-  # For a genuinely docs-only PR this adds no strictness (no code changed, so
-  # fmt and the guards always pass) and costs ~50s of ubuntu-latest.
-  quick-checks:
-    name: Quick Checks
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Install ripgrep
-        uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
-        with:
-          tool: ripgrep@15.2.0
-
-      - name: Install Rust toolchain
-        uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
-        with:
-          components: rustfmt
-
-      - name: Check code formatting
-        run: cargo fmt --all --check
-
-      - name: Check unsafe code allowances
-        run: ./scripts/check_unsafe_code_allowances.sh
-
-      - name: Check layered dependencies
-        run: ./scripts/check_layer_dependencies.sh
-
-      - name: Check architecture migration rules
-        run: ./scripts/check_architecture_migration_rules.sh
-
-      - name: Check logging guardrails
-        run: ./scripts/check_logging_guardrails.sh
-
-      - name: Check tokio io-uring feature guard
-        run: ./scripts/check_no_tokio_io_uring.sh
-
-      - name: Check extension schema boundaries
-        run: ./scripts/check_extension_schema_boundaries.sh
-
-      - name: Check body-cache whitelist guard
-        run: ./scripts/check_body_cache_whitelist.sh
-
-      - name: Check s3s footprint ratchet
-        run: ./scripts/check_s3s_footprint.sh
-
-      - name: Check cryptographic capability wording
-        run: ./scripts/check_fips_wording.sh
-
-      - name: Check no embedded secret material
-        run: ./scripts/check_embedded_secrets.sh
-
-      - name: Check test wiring
-        run: python3 ./scripts/check_test_wiring.py
-
-      - name: Check no planning docs committed
-        run: ./scripts/check_no_planning_docs.sh
-
-      - name: Check CI paths stay in sync
-        run: ./scripts/check_ci_paths_sync.sh
-
-      - name: Check io_uring lane --lib precondition
-        run: ./scripts/check_uring_lane_lib_only.sh
-
   test-and-lint:
     name: Test and Lint
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       # Docs-only PRs skip the full code CI, but they are exactly where a
       # planning-type document could be slipped in (git add -f bypasses

.github/workflows/ci.yml

@@ -33,7 +33,6 @@ on:
       - ".github/workflows/build.yml"
       - ".github/workflows/docker.yml"
       - ".github/workflows/audit.yml"
-      - "flake.lock"
   pull_request:
     types: [ opened, synchronize, reopened, closed ]
     branches: [ main ]
@@ -55,7 +54,6 @@ on:
       - ".github/workflows/build.yml"
       - ".github/workflows/docker.yml"
       - ".github/workflows/audit.yml"
-      - "flake.lock"
   merge_group:
     types: [ checks_requested ]
   schedule:
@@ -83,7 +81,6 @@ jobs:
     name: Cancel Closed PR Runs
     if: github.event_name == 'pull_request' && github.event.action == 'closed'
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - name: Explain cancellation run
         run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -92,20 +89,13 @@ jobs:
     name: Typos
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
       - name: Typos check with custom config file
         uses: crate-ci/typos@37bb98842b0d8c4ffebdb75301a13db0267cef89 # master
 
   # Fast, compile-free checks that fail early so contributors get feedback in
   # ~1 minute instead of waiting for the full test job.
-  #
-  # These steps are mirrored byte-for-byte in ci-docs-only.yml so that a mixed
-  # PR, which reports two check runs named "Quick Checks", cannot get one red
-  # and one green. Edit both jobs together.
   quick-checks:
     name: Quick Checks
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
@@ -114,13 +104,9 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Install ripgrep
-        uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
-        with:
-          tool: ripgrep@15.2.0
+        run: sudo apt-get update && sudo apt-get install -y ripgrep
 
       - name: Install Rust toolchain
         uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
@@ -139,9 +125,6 @@ jobs:
       - name: Check architecture migration rules
         run: ./scripts/check_architecture_migration_rules.sh
 
-      - name: Check logging guardrails
-        run: ./scripts/check_logging_guardrails.sh
-
       - name: Check tokio io-uring feature guard
         run: ./scripts/check_no_tokio_io_uring.sh
 
@@ -151,179 +134,47 @@ jobs:
       - name: Check body-cache whitelist guard
         run: ./scripts/check_body_cache_whitelist.sh
 
-      - name: Check s3s footprint ratchet
-        run: ./scripts/check_s3s_footprint.sh
-
-      - name: Check cryptographic capability wording
-        run: ./scripts/check_fips_wording.sh
-
-      - name: Check no embedded secret material
-        run: ./scripts/check_embedded_secrets.sh
-
-      - name: Check test wiring
-        run: python3 ./scripts/check_test_wiring.py
-
       - name: Check no planning docs committed
         run: ./scripts/check_no_planning_docs.sh
 
-      - name: Check CI paths stay in sync
-        run: ./scripts/check_ci_paths_sync.sh
-
-      - name: Check io_uring lane --lib precondition
-        run: ./scripts/check_uring_lane_lib_only.sh
-
   test-and-lint:
     name: Test and Lint
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    needs: [ quick-checks ]
     runs-on: sm-standard-4
-    timeout-minutes: 90
-    # Both lines are required. Job-level `permissions` replaces the workflow
-    # block rather than merging with it, so declaring only `actions: write`
-    # would drop `contents: read` and break this job's checkout and the
-    # repo-token the setup action hands to setup-protoc.
-    permissions:
-      contents: read
-      actions: write
+    timeout-minutes: 60
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          # This job's token can cancel runs and delete Actions caches. Checkout
-          # otherwise writes it into .git/config, where a PR's own build.rs or
-          # proc-macro could read it back out.
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: stable
-          # Every lane in this workflow reads its cache and none writes it.
-          # cache-warm.yml is the sole writer for all four keys: this workflow
-          # cancels superseded runs on main, and a cancelled run never reaches
-          # rust-cache's post step, so writing from here saved nothing (12 of 15
-          # consecutive main-push runs were cancelled). See rustfs/backlog#1600.
-          cache-shared-key: ci-dev
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
-
-      - name: Prepare test evidence
-        run: |
-          mkdir -p artifacts/test-and-lint
-          {
-            echo "run_id=${GITHUB_RUN_ID}"
-            echo "job=${GITHUB_JOB}"
-            echo "runner=${RUNNER_NAME}"
-            echo "started_at=$(date --utc --iso-8601=seconds)"
-          } > artifacts/test-and-lint/run-metadata.txt
+          cache-shared-key: ci-test
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
       # Clippy runs before the test pass: lint failures are the most common
       # CI-only breakage and should surface in minutes, not after 20+ minutes
       # of tests.
-      # Sampled too: clippy is the natural control arm for any CARGO_BUILD_JOBS
-      # experiment, since --all-targets is check-only for workspace members and
-      # never links the ~100 test binaries the limit exists to throttle.
       - name: Run clippy lints
-        run: |
-          ./scripts/ci/resource_sampler.sh start clippy
-          trap './scripts/ci/resource_sampler.sh stop' EXIT
-          cargo clippy --all-targets -- -D warnings
+        run: cargo clippy --all-targets -- -D warnings
 
-      - name: Run nextest tests
-        env:
-          # #5394 mitigation, now under a measured experiment (backlog#1601).
-          #
-          # 2 was chosen when three concurrent workspace test links were believed
-          # to saturate the runner's overlay I/O and wedge Cargo until the 75m
-          # timeout. cgroup v2 readings from the sampler show the pod actually
-          # has 14 CPUs and 28GB (peak use 2.1GB), so 2 throttles compilation to
-          # a seventh of what is available and memory was never the constraint —
-          # the label name "sm-standard-4" had led everyone, including the
-          # original mitigation, to assume 4 cores.
-          #
-          # Raised to 3 on main pushes and manual dispatches; PRs keep 2 so the
-          # merge path is untouched while the experiment runs.
-          #
-          # Dispatch is included because push alone cannot supply the samples:
-          # this workflow cancels superseded runs on main, and only 4 of the last
-          # 20 push-triggered Test and Lint jobs reached a terminal state — at
-          # that rate ten samples would take roughly fifty merges. The
-          # concurrency group is scoped by event_name, so a dispatched run has
-          # its own group and is not cancelled by merge traffic, which makes the
-          # sample collectable on demand rather than by waiting.
-          #
-          # Baseline over 17 samples at 2:
-          # median nextest/clippy step ratio 1.95, spread 1.85-2.06. The gate-2
-          # criterion is that ratio dropping at least 10% (below ~1.76) with no
-          # 75m timeout and no run showing three consecutive samples of
-          # rustc/collect2/rust-lld in D state. If it does not, the conclusion is
-          # "this limit is not the bottleneck" — fix it back at 2 and record the
-          # experiment, which is a result, not a failure.
-          #
-          # Must stay step-level: rust-cache hashes CARGO/CC/CFLAGS/CXX/CMAKE/RUST
-          # prefixed variables from process.env into the cache key, so promoting
-          # this to job level would rotate every key on this lane.
-          CARGO_BUILD_JOBS: ${{ (github.event_name == 'push' || github.event_name == 'workflow_dispatch') && '3' || '2' }}
+      - name: Run tests
         run: |
-          mkdir -p artifacts/test-and-lint
-          ./scripts/ci/resource_sampler.sh start nextest
-          trap './scripts/ci/resource_sampler.sh stop' EXIT
-          set +e
-          NEXTEST_HIDE_PROGRESS_BAR=1 timeout --verbose --signal=TERM --kill-after=30s 75m \
-            cargo nextest run --profile ci --all --exclude e2e_test \
-            --status-level all --final-status-level all \
-            2>&1 | tee artifacts/test-and-lint/nextest.log
-          status=${PIPESTATUS[0]}
-          {
-            echo "command=cargo nextest run --profile ci --all --exclude e2e_test"
-            echo "exit_status=${status}"
-            echo "finished_at=$(date --utc --iso-8601=seconds)"
-            echo
-            echo "Remaining test-related processes:"
-            pgrep -af 'cargo|nextest|target/.*/deps/' || true
-            echo
-            echo "Kernel OOM / kill events:"
-            dmesg -T 2>/dev/null | grep -iE 'oom|out of memory|killed process' | tail -20 || true
-          } > artifacts/test-and-lint/nextest-diagnostics.txt
-          exit "${status}"
+          cargo nextest run --profile ci --all --exclude e2e_test
+          cargo test --all --doc
 
-      - name: Run documentation tests
-        run: |
-          mkdir -p artifacts/test-and-lint
-          set +e
-          timeout --verbose --signal=TERM --kill-after=30s 15m \
-            cargo test --all --doc \
-            2>&1 | tee artifacts/test-and-lint/doctest.log
-          status=${PIPESTATUS[0]}
-          {
-            echo "command=cargo test --all --doc"
-            echo "exit_status=${status}"
-            echo "finished_at=$(date --utc --iso-8601=seconds)"
-            echo
-            echo "Remaining test-related processes:"
-            pgrep -af 'cargo|rustdoc|target/.*/deps/' || true
-          } > artifacts/test-and-lint/doctest-diagnostics.txt
-          exit "${status}"
-
-      - name: Upload test reports and diagnostics
+      - name: Upload test junit report
         if: always()
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
         with:
           name: junit-test-and-lint-${{ github.run_number }}
-          path: |
-            target/nextest/ci/junit.xml
-            artifacts/test-and-lint
+          path: target/nextest/ci/junit.xml
           retention-days: 3
-          if-no-files-found: error
-
-      # rustfs/backlog#1289: fail if a seed rule's log anchor no longer exists
-      # verbatim in the source tree (log message drifted without updating the
-      # rule). Placed here where the workspace — including the la-dump-anchors
-      # bin — is already built by the clippy/test steps above.
-      - name: Check log-analyzer rule anchors
-        run: ./scripts/check_log_analyzer_rules.sh
+          if-no-files-found: ignore
 
       # Explicit gate for migration-critical suites. These tests already ran in
       # the full nextest pass above; a single filtered nextest invocation keeps
@@ -341,50 +192,6 @@ jobs:
       - name: Run rebalance/decommission migration proofs
         run: ./scripts/check_migration_gate_count.sh
 
-      # Early stop. Once this job has failed the PR cannot merge, so the sibling
-      # lanes are burning runners on a result nobody can act on: on run
-      # 30674613104 three lanes had already failed while Test and Lint and the
-      # rio-v2 variant kept going past 70 minutes.
-      #
-      # Only this job may cancel. The lanes that are NOT required checks
-      # (protocols, ILM, e2e, s3-tests) must never hold that power: a flake in
-      # one of them would turn the required "Test and Lint" into `cancelled`,
-      # which blocks the merge. Today a maintainer can merge with sftp red, and
-      # that has to stay true.
-      #
-      # These steps run last so the `if: always()` artifact upload above still
-      # captures logs and diagnostics before the run goes away.
-      - name: Annotate early-stop reason
-        if: failure() && github.event_name == 'pull_request'
-        run: |
-          {
-            echo "## CI early-stop"
-            echo "Job \`${GITHUB_JOB}\` (Test and Lint) failed; cancelling run ${GITHUB_RUN_ID} to free runners."
-            echo "Sibling jobs showing **cancelled** were stopped by this job, not by their own failure."
-          } >> "$GITHUB_STEP_SUMMARY"
-
-      # curl rather than `gh`: every existing `gh` call in this repo runs on
-      # ubuntu-latest, and the sm-standard-* images are custom and trimmed (they
-      # ship no C toolchain, see the e2e job below), so `gh` is not known to
-      # exist here.
-      #
-      # Fork PRs are excluded explicitly instead of relying on the error path:
-      # their GITHUB_TOKEN is forced read-only and job-level permissions cannot
-      # raise it, so the call would always 403. Skipping keeps their logs clean.
-      - name: Cancel run on failure (same-repo PR only)
-        if: >-
-          failure() && github.event_name == 'pull_request'
-          && github.event.pull_request.head.repo.full_name == github.repository
-        continue-on-error: true
-        env:
-          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-        run: |
-          curl -fsS -X POST \
-            -H "Authorization: Bearer ${GH_TOKEN}" \
-            -H "Accept: application/vnd.github+json" \
-            -H "X-GitHub-Api-Version: 2022-11-28" \
-            "${GITHUB_API_URL}/repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/cancel" || true
-
   # Dedicated serial lane for the ILM / lifecycle integration tests. These tests
   # drive the object layer through process-global singletons (the GLOBAL_ENV
   # ECStore, the global tier-config manager, background-expiry workers) and bind
@@ -398,64 +205,50 @@ jobs:
   test-ilm-integration-serial:
     name: ILM Integration (serial)
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    needs: [ quick-checks ]
     runs-on: sm-standard-4
-    timeout-minutes: 90
+    timeout-minutes: 45
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: stable
-          cache-shared-key: ci-dev
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
+          cache-shared-key: ci-ilm-serial
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
-      # test_transition_and_restore_flows was re-enabled by rustfs/backlog#1303:
-      # its "missing xl.meta on disk2" was a test-util bug (open_disk hardcoded
-      # disk_index 0), not an EC metadata-distribution issue.
-      # restore_object_usecase_reports_ongoing_conflict_and_completion was
-      # re-enabled by backlog#1304 (restore accepts serialize on a short CAS
-      # guard; the copy-back no longer holds the #4877 whole-copy-back lock,
-      # so the mid-restore ongoing read and fast 409 rejection it asserts are
-      # the implemented contract). The remaining exclusions each hit a
-      # DIFFERENT, independent issue (all tracked under rustfs/backlog#1148;
-      # they keep #[ignore] with a backlog reference):
-      #   - test_noncurrent_{expiry,transition}_still_works_after_immediate_compensation_transition:
-      #     noncurrent transition/expiry after an immediate compensation transition.
+      # Three scanner tests fail on main independently of this lane (restore of
+      # a transitioned multipart object, noncurrent transition/expiry after an
+      # immediate compensation transition); they keep #[ignore] with a backlog
+      # reference and are excluded here by name until fixed (rustfs/backlog#1148).
       - name: Run ignored ILM integration tests serially
         run: |
           cargo nextest run -j1 --run-ignored ignored-only \
             -p rustfs-scanner -p rustfs \
-            -E '(binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))) and not (test(test_noncurrent_expiry_still_works_after_immediate_compensation_transition) or test(test_noncurrent_transition_still_works_after_immediate_compensation_transition))'
+            -E '(binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))) and not (test(test_transition_and_restore_flows) or test(test_noncurrent_expiry_still_works_after_immediate_compensation_transition) or test(test_noncurrent_transition_still_works_after_immediate_compensation_transition))'
 
   test-and-lint-rio-v2:
     name: Test and Lint (rio-v2)
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    needs: [ quick-checks ]
     runs-on: sm-standard-4
-    timeout-minutes: 90
+    timeout-minutes: 60
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: stable
-          cache-shared-key: ci-feat-rio
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
+          cache-shared-key: ci-test-rio-v2
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
       - name: Run rio-v2 clippy lints
         run: cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
@@ -468,17 +261,10 @@ jobs:
   test-and-lint-protocols:
     name: "Test and Lint (${{ matrix.features.name }})"
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    needs: [ quick-checks ]
     runs-on: sm-standard-4
-    timeout-minutes: 90
+    timeout-minutes: 60
     strategy:
-      # On a PR, one failing protocol leg is enough to know the PR is not ready,
-      # so stop the sibling leg instead of paying another ~40 minutes for it.
-      # Everywhere else (main pushes, the merge queue, the weekly schedule) keep
-      # the full signal: there we want to know whether swift AND sftp are broken,
-      # not just whichever failed first. This is the only part of the early-stop
-      # work that also covers fork PRs, since it needs no token.
-      fail-fast: ${{ github.event_name == 'pull_request' }}
+      fail-fast: false
       matrix:
         features:
           - name: swift
@@ -490,16 +276,14 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: stable
-          cache-shared-key: ci-feat-proto
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
+          cache-shared-key: ci-test-${{ matrix.features.name }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
       - name: Run clippy with ${{ matrix.features.name }}
         run: |
@@ -512,7 +296,6 @@ jobs:
   build-rustfs-debug-binary:
     name: Build RustFS Debug Binary
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    needs: [ quick-checks ]
     runs-on: sm-standard-4
     timeout-minutes: 30
     env:
@@ -520,19 +303,17 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: stable
-          cache-shared-key: ci-dev
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
+          cache-shared-key: ci-rustfs-debug-binary
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Build debug binary
-        run: cargo build -p rustfs --bins --features e2e-test-hooks
+        run: cargo build -p rustfs --bins
 
       - name: Upload debug binary
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
@@ -544,13 +325,7 @@ jobs:
 
   build-rustfs-debug-binary-rio-v2:
     name: Build RustFS Debug Binary (rio-v2)
-    # Dormant rio-v2 variant (rustfs/backlog#1835): the feature ships in no
-    # default build, so this full-suite lane runs only on the weekly schedule
-    # and manual dispatch. Per-PR cfg-seam coverage stays with
-    # test-and-lint-rio-v2. Lifecycle and the promote-or-delete condition:
-    # docs/architecture/minio-file-format-compat.md ("rio-v2 variant lifecycle").
-    if: github.event_name == 'schedule' || github.event_name == 'workflow_dispatch'
-    needs: [ quick-checks ]
+    if: github.event_name != 'pull_request' || github.event.action != 'closed'
     runs-on: sm-standard-4
     timeout-minutes: 30
     env:
@@ -558,19 +333,17 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: stable
-          cache-shared-key: ci-feat-rio
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
+          cache-shared-key: ci-rustfs-debug-binary-rio-v2
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Build debug binary with rio-v2
-        run: cargo build -p rustfs --bins --features rio-v2,e2e-test-hooks
+        run: cargo build -p rustfs --bins --features rio-v2
 
       - name: Upload debug binary
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
@@ -582,14 +355,6 @@ jobs:
 
   uring-integration:
     name: io_uring Integration (real)
-    # The pull_request trigger includes `closed` purely so the concurrency
-    # group cancels in-flight runs of a closed PR; every other job opts out of
-    # that run with this guard (or is skipped through its `needs` chain). This
-    # job had neither, so each closed/merged PR really ran the whole io_uring
-    # suite (measured 4m17s / 7m19s / 7m31s on runs 30678272341 / 30678117601 /
-    # 30662728539) and kept the cancellation run in progress for minutes.
-    if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    needs: [ quick-checks ]
     # GitHub-hosted ubuntu-latest runs a recent kernel with io_uring and, unlike
     # a container, applies no seccomp filter that would block io_uring_setup — so
     # the probe succeeds and the tests exercise the real UringBackend/FdCache/
@@ -600,24 +365,17 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: stable
-          # Keeps its own key rather than joining ci-dev. rust-cache's key is
-          # built from runner.os/arch plus rustc and lockfile fingerprints — it
-          # does NOT include the runner label or image. ubuntu-latest and
-          # sm-standard-4 are therefore indistinguishable to it, so sharing a key
-          # would let two different system images overwrite each other's
-          # artifacts, and would make a 2-core hosted runner unpack ci-dev's ~3GB
-          # instead of this lane's ~1.3GB. cache-warm.yml warms this key on
-          # ubuntu-latest for the same reason.
           cache-shared-key: ci-uring
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      - name: Install build dependencies
+        run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
 
       # ext4 supports O_DIRECT; the runner's default TMPDIR may sit on tmpfs or
       # overlayfs, where open(O_DIRECT) returns EINVAL/EOPNOTSUPP and the native
@@ -644,17 +402,7 @@ jobs:
           RUSTFS_IO_URING_READ_ENABLE: "true"
           RUSTFS_URING_TESTS_MUST_RUN: "1"
           TMPDIR: /mnt/rustfs-odirect
-        # --lib narrows what gets compiled, not what gets run: every selected
-        # test lives in the lib target. The 7 integration binaries under
-        # crates/ecstore/tests/ each reported "running 0 tests" here, so they
-        # were compiled and linked for nothing.
-        #
-        # The `uring_` filter must stay exactly as it is. libtest matches on
-        # substring, so it also selects names containing `during_` — 6 of the 18
-        # selected tests are such incidental matches. Narrowing the filter to
-        # `io_uring` would silently drop them, which is a coverage change.
-        # scripts/check_uring_lane_lib_only.sh guards the --lib precondition.
-        run: cargo test -p rustfs-ecstore --lib uring_ -- --test-threads=1 --nocapture
+        run: cargo test -p rustfs-ecstore uring_ -- --test-threads=1 --nocapture
 
   e2e-tests:
     name: End-to-End Tests
@@ -664,8 +412,6 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       # Full setup with dependency caching: the smoke-suite step below
       # compiles the e2e_test crate, which pulls in most of the workspace.
@@ -674,9 +420,9 @@ jobs:
         uses: ./.github/actions/setup
         with:
           rust-version: stable
-          cache-shared-key: ci-dev
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
+          cache-shared-key: ci-e2e
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
       # Download after the cache restore so the freshly built binary from the
       # build job always wins over anything restored into target/debug.
@@ -689,49 +435,13 @@ jobs:
       - name: Make binary executable
         run: chmod +x ./target/debug/rustfs
 
-      # Build the e2e test graph once. The archive is reused by the smoke
-      # selection guard, security exact-count check, and run below, avoiding a
-      # second compile of the same e2e_test target on cold runners (backlog#1645).
-      - name: Archive e2e smoke test binaries
-        env:
-          NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
-          NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-smoke-list.json
-        run: |
-          cargo nextest archive --profile e2e-smoke -p e2e_test --archive-file "${NEXTEST_ARCHIVE}"
-          cargo nextest list --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" --message-format json > "${NEXTEST_LISTING}"
-          python3 ./scripts/check_test_wiring.py --check-profile e2e-smoke "${NEXTEST_LISTING}"
-          ./scripts/check_security_smoke_count.sh check "${NEXTEST_LISTING}"
-
       # PR smoke subset of the in-repo e2e suite (backlog#1149 ci-4). The
       # profile.e2e-smoke default-filter in .config/nextest.toml is the single
       # wiring mechanism for e2e tests in CI — extend that filter instead of
       # adding new e2e jobs here. Each test spawns its own rustfs server on a
       # random port and reuses the downloaded debug binary above.
       - name: Run e2e smoke suite
-        env:
-          NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
-          RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-smoke-logs
-        run: |
-          cargo nextest run --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" \
-            --status-level all --final-status-level all --failure-output final
-
-      - name: Upload e2e smoke diagnostics
-        if: failure()
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: e2e-smoke-diagnostics-${{ github.run_number }}
-          path: |
-            ${{ runner.temp }}/rustfs-e2e-smoke-logs/
-            ${{ runner.temp }}/rustfs-e2e-smoke-list.json
-          if-no-files-found: warn
-
-      - name: Upload e2e smoke JUnit report
-        if: always()
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: e2e-smoke-junit-${{ github.run_number }}
-          path: target/nextest/e2e-smoke/junit.xml
-          if-no-files-found: warn
+        run: cargo nextest run --profile e2e-smoke -p e2e_test
 
       - name: Install s3s-e2e test tool
         uses: taiki-e/cache-cargo-install-action@7447f04c51f2ba27ca35e7f1e28fab848c5b3ba7 # v2
@@ -758,109 +468,14 @@ jobs:
           path: ${{ runner.temp }}/rustfs-e2e-*/rustfs.log
           retention-days: 3
 
-  e2e-full:
-    name: End-to-End Tests (full merge gate)
-    # Merge gate only (backlog#1149 ci-5): the never-automated user-visible
-    # suites — KMS, object_lock, multipart_auth, quota, checksum, encryption,
-    # security-boundary, ... — via the e2e-full nextest profile. Too heavy for
-    # every PR, so it is gated to main pushes, the merge queue, and manual
-    # dispatch. protocols / the 7 cluster suites / replication / #[ignore] are
-    # owned by other lanes (see .config/nextest.toml profile.e2e-full).
-    if: >-
-      github.event_name == 'workflow_dispatch' ||
-      github.event_name == 'merge_group' ||
-      (github.event_name == 'push' && github.ref == 'refs/heads/main')
-    needs: [ build-rustfs-debug-binary ]
-    runs-on: sm-standard-2
-    timeout-minutes: 55
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-dev
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
-
-      - name: Set up Python
-        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
-        with:
-          python-version: "3.12"
-
-      - name: Install awscurl
-        run: |
-          python3 -m pip install --user --upgrade pip "awscurl==0.44"
-          echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
-
-      - name: Verify awscurl
-        run: test -x "$AWSCURL_PATH"
-
-      - name: Install Vault
-        run: |
-          VAULT_VERSION="1.17.6"
-          VAULT_ARCHIVE="vault_${VAULT_VERSION}_linux_amd64.zip"
-          curl -fsSLo "$RUNNER_TEMP/$VAULT_ARCHIVE" "https://releases.hashicorp.com/vault/${VAULT_VERSION}/${VAULT_ARCHIVE}"
-          echo "0cddc1fbbb88583b5ba5b845f9f8fae47c6fb39a6d48cd543c6ba6fd3ac1a669  $RUNNER_TEMP/$VAULT_ARCHIVE" | sha256sum --check --status
-          unzip -q "$RUNNER_TEMP/$VAULT_ARCHIVE" -d "$RUNNER_TEMP/vault-bin"
-          echo "RUSTFS_TEST_VAULT_BIN=$RUNNER_TEMP/vault-bin/vault" >> "$GITHUB_ENV"
-
-      - name: Verify Vault
-        run: |
-          "$RUSTFS_TEST_VAULT_BIN" version
-
-      # Download after the cache restore so the freshly built binary from the
-      # build job always wins over anything restored into target/debug.
-      - name: Download debug binary
-        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
-        with:
-          name: rustfs-debug-binary
-          path: target/debug
-
-      - name: Make binary executable
-        run: chmod +x ./target/debug/rustfs
-
-      - name: Verify e2e full membership
-        env:
-          NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-full-list.json
-        run: |
-          cargo nextest list --profile e2e-full -p e2e_test --message-format json > "${NEXTEST_LISTING}"
-          python3 ./scripts/check_test_wiring.py --check-profile e2e-full "${NEXTEST_LISTING}"
-
-      # Full single-node e2e lane (backlog#1149 ci-5). The e2e-full
-      # default-filter in .config/nextest.toml is the single wiring mechanism —
-      # extend that filter, never add ad-hoc e2e jobs here. Reuses the downloaded
-      # debug binary; each test spawns its own rustfs server on a random port.
-      - name: Run e2e full suite
-        run: cargo nextest run --profile e2e-full -p e2e_test
-
-      - name: Upload junit
-        if: always()
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: e2e-full-junit-${{ github.run_number }}
-          path: |
-            target/nextest/e2e-full/junit.xml
-            ${{ runner.temp }}/rustfs-e2e-full-list.json
-          retention-days: 7
-
   e2e-tests-rio-v2:
     name: End-to-End Tests (rio-v2)
-    # Inherits the schedule/dispatch-only gate through needs: on every other
-    # event build-rustfs-debug-binary-rio-v2 is skipped, so this job skips
-    # with it (see the dormant-variant comment on that job).
     needs: [ build-rustfs-debug-binary-rio-v2 ]
     runs-on: sm-standard-2
     timeout-minutes: 30
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Clean up previous test run
         run: |
@@ -879,14 +494,6 @@ jobs:
       - name: Setup Rust toolchain for s3s-e2e installation
         uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
 
-      # The sm-standard-* custom runner images (introduced in #4884) ship no C
-      # toolchain, unlike GitHub-hosted ubuntu-latest. Installing s3s-e2e below
-      # compiles it from source on a cache miss, and build scripts need cc.
-      - name: Install build tools for s3s-e2e compilation
-        run: |
-          sudo apt-get update
-          sudo apt-get install -y build-essential cmake pkg-config libssl-dev
-
       - name: Install s3s-e2e test tool
         uses: taiki-e/cache-cargo-install-action@7447f04c51f2ba27ca35e7f1e28fab848c5b3ba7 # v2
         with:
@@ -915,8 +522,6 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Download debug binary
         uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
@@ -977,20 +582,12 @@ jobs:
   # evaluates ILM within ~2s of the due time, well inside the poll window.
   s3-lifecycle-behavior-tests:
     name: S3 Lifecycle Behavior Tests
-    # Also gated on e2e-tests, matching s3-implemented-tests: when the e2e smoke
-    # suite is already red this lane cannot tell us anything new, and it holds a
-    # sm-standard-4 for up to 30 minutes doing so. Both lanes only download the
-    # prebuilt debug binary (no cargo build), and s3-implemented-tests — which
-    # already waits on e2e-tests — finishes later anyway, so a green PR's total
-    # wall clock is unchanged.
-    needs: [ build-rustfs-debug-binary, e2e-tests ]
+    needs: [ build-rustfs-debug-binary ]
     runs-on: sm-standard-4
     timeout-minutes: 30
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Download debug binary
         uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7

.github/workflows/cla.yml

@@ -22,18 +22,11 @@ on:
   issue_comment:
     types: [created, edited]
 
-# Least privilege at the top, widened per job below. This workflow runs on
-# pull_request_target and issue_comment, so it holds full secrets on every fork
-# PR and on any comment anyone writes — the one place in this repository where a
-# compromised action would be handed a repo-write token. It does not check out
-# or execute PR code, so there is no pwn-request path today, but the blast
-# radius should not depend on that staying true.
-#
-# contents: write in particular was never used: the signature records are
-# written to rustfs/cla through the scoped app token created below, and nothing
-# here writes to this repository's contents.
 permissions:
-  contents: read
+  contents: write
+  pull-requests: write
+  issues: write
+  checks: write
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.event.issue.number || github.ref }}
@@ -43,26 +36,14 @@ jobs:
   cancel-closed-pr-runs:
     name: Cancel Closed PR Runs
     if: github.event_name == 'pull_request_target' && github.event.action == 'closed'
-    # Echoes one line; the run exists only so the concurrency group cancels the
-    # in-flight run of a closed PR.
-    permissions: {}
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - name: Explain cancellation run
         run: echo "PR closed; this run only cancels older runs in the same concurrency group."
 
   cla:
     if: ${{ (github.event_name != 'issue_comment' || github.event.issue.pull_request) && (github.event_name != 'pull_request_target' || github.event.action != 'closed') }}
-    # checks: write reports the merge-queue check run; pull-requests and issues
-    # let cla-bot comment and label. contents stays read — see the note above.
-    permissions:
-      contents: read
-      checks: write
-      issues: write
-      pull-requests: write
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     steps:
       - name: Report CLA result for merge queue
         if: github.event_name == 'merge_group'

.github/workflows/coverage.yml

@@ -1,126 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Weekly workspace line-coverage baseline (backlog#1153 infra-5).
-#
-# NON-BLOCKING by design: this workflow only runs on schedule and manual
-# dispatch, so it never attaches a status to a PR and must never be made a
-# required check. It exists to give coverage a visible baseline and trend
-# (per-crate table in the job summary, lcov artifact kept 90 days) — the
-# per-crate ratchet for the security-critical crates builds on it later
-# (backlog#1153 infra-6, report-only first per the ci-11 ladder).
-#
-# Measurement scope matches the PR test gate (ci.yml "Run tests"):
-# `--workspace --exclude e2e_test` with the `ci` nextest profile. Doctests are
-# NOT measured (ci.yml runs them uninstrumented; `cargo llvm-cov` needs a
-# nightly toolchain to cover doctests). Trend-comparison workflow:
-# docs/testing/README.md "Coverage" section. `make coverage` is the local
-# equivalent. Scheduled failures alert via the ci-8 composite action.
-
-name: coverage
-
-on:
-  workflow_dispatch:
-  schedule:
-    # 07:00 UTC Sunday — staggered clear of the other Sunday crons: ci (00:00),
-    # build (01:00), e2e-s3tests (02:00), audit (03:00), nix-flake-update
-    # (05:00), mint (06:00), and the daily fuzz (02:00), minio-interop (03:17),
-    # e2e-replication-nightly (04:00) and performance-ab (06:00) lanes.
-    - cron: "0 7 * * 0"
-
-# Only alert-on-failure needs more than read access; it declares its own
-# job-level `issues: write`.
-permissions:
-  contents: read
-
-jobs:
-  coverage:
-    name: Workspace coverage (weekly)
-    runs-on: sm-standard-4
-    # The instrumented build cannot reuse the regular CI cache (different
-    # RUSTFLAGS), so a cold week rebuilds the workspace before running the
-    # full suite; give it double the test job's 60-minute budget.
-    timeout-minutes: 120
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-      # Match the PR gate's nextest semantics (ci.yml runs `--profile ci`):
-      # retries=0 plus the quarantine list and the ecstore-serial-flaky
-      # serialization. Set via env because `cargo llvm-cov`'s own --profile
-      # flag selects the *cargo build* profile, not the nextest profile.
-      NEXTEST_PROFILE: ci
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-coverage
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
-          install-build-packaging-tools: 'false'
-
-      - name: Install cargo-llvm-cov
-        uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
-        with:
-          tool: cargo-llvm-cov
-
-      - name: Install llvm-tools component
-        run: rustup component add llvm-tools-preview
-
-      - name: Run instrumented test suite
-        run: cargo llvm-cov nextest --workspace --exclude e2e_test --no-report
-
-      - name: Generate lcov and JSON reports
-        run: |
-          mkdir -p target/llvm-cov
-          cargo llvm-cov report --lcov --output-path target/llvm-cov/lcov.info
-          cargo llvm-cov report --json --output-path target/llvm-cov/coverage.json
-
-      - name: Write per-crate summary
-        run: python3 scripts/coverage_per_crate.py target/llvm-cov/coverage.json >> "$GITHUB_STEP_SUMMARY"
-
-      - name: Upload coverage artifact
-        if: always()
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: coverage-lcov-${{ github.run_number }}
-          path: |
-            target/llvm-cov/lcov.info
-            target/llvm-cov/coverage.json
-          retention-days: 90
-          if-no-files-found: ignore
-
-  alert-on-failure:
-    name: Alert on scheduled failure
-    needs: [coverage]
-    # Only scheduled runs open/append the tracking issue (backlog#1149 ci-8);
-    # manual workflow_dispatch runs stay quiet so a debugging run never files a
-    # spurious alert.
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    permissions:
-      contents: read
-      issues: write
-    steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-      - name: Open or update failure-tracking issue
-        uses: ./.github/actions/schedule-failure-issue
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/docker.yml

@@ -66,7 +66,7 @@ env:
   CARGO_TERM_COLOR: always
   REGISTRY_DOCKERHUB: rustfs/rustfs
   REGISTRY_GHCR: ghcr.io/${{ github.repository }}
-  REGISTRY_QUAY: quay.io/rustfs/rustfs
+  REGISTRY_QUAY: quay.io/${{ secrets.QUAY_USERNAME }}/rustfs
   DOCKER_PLATFORMS: linux/amd64,linux/arm64
 
 jobs:
@@ -82,10 +82,8 @@ jobs:
       github.event_name == 'workflow_dispatch' ||
       (github.event.workflow_run.conclusion == 'success' &&
        github.event.workflow_run.event == 'push' &&
-       github.event.workflow_run.head_branch != 'main' &&
-       !contains(github.event.workflow_run.head_branch, '-preview'))
+       github.event.workflow_run.head_branch != 'main')
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     outputs:
       should_build: ${{ steps.check.outputs.should_build }}
       should_push: ${{ steps.check.outputs.should_push }}
@@ -94,23 +92,15 @@ jobs:
       short_sha: ${{ steps.check.outputs.short_sha }}
       is_prerelease: ${{ steps.check.outputs.is_prerelease }}
       create_latest: ${{ steps.check.outputs.create_latest }}
-      source_ref: ${{ steps.check.outputs.source_ref }}
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
         with:
-          persist-credentials: false
           # For workflow_run events, checkout the specific commit that triggered the workflow
           ref: ${{ github.event.workflow_run.head_sha || github.sha }}
 
       - name: Check build conditions
         id: check
-        env:
-          # dispatch inputs via env, not `${{ }}` interpolation: they are
-          # free-form strings and would otherwise be evaluated by bash.
-          INPUT_VERSION: ${{ github.event.inputs.version }}
-          INPUT_PUSH_IMAGES: ${{ github.event.inputs.push_images }}
-          INPUT_FORCE_REBUILD: ${{ github.event.inputs.force_rebuild }}
         run: |
           should_build=false
           should_push=false
@@ -119,7 +109,6 @@ jobs:
           short_sha=""
           is_prerelease=false
           create_latest=false
-          source_ref="$GITHUB_SHA"
 
           if [[ "${{ github.event_name }}" == "workflow_run" ]]; then
             # Triggered by build workflow completion
@@ -139,7 +128,6 @@ jobs:
             # Extract version info from commit message or use commit SHA
             # Use Git to generate consistent short SHA (ensures uniqueness like build.yml)
             short_sha=$(git rev-parse --short "$HEAD_SHA")
-            source_ref="$HEAD_SHA"
 
             # Determine build type based on triggering workflow event and ref
             triggering_event="$TRIGGERING_EVENT"
@@ -213,9 +201,9 @@ jobs:
 
           elif [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
             # Manual trigger
-            input_version="$INPUT_VERSION"
+            input_version="${{ github.event.inputs.version }}"
             version="${input_version}"
-            should_push="$INPUT_PUSH_IMAGES"
+            should_push="${{ github.event.inputs.push_images }}"
             should_build=true
 
             # Get short SHA
@@ -223,7 +211,7 @@ jobs:
 
             echo "🎯 Manual Docker build triggered:"
             echo "   📋 Requested version: $input_version"
-            echo "   🔧 Force rebuild: $INPUT_FORCE_REBUILD"
+            echo "   🔧 Force rebuild: ${{ github.event.inputs.force_rebuild }}"
             echo "   🚀 Push images: $should_push"
 
             case "$input_version" in
@@ -232,13 +220,6 @@ jobs:
                 create_latest=true
                 echo "🚀 Building with latest stable release version"
                 ;;
-              *-preview*)
-                build_type="preview"
-                is_prerelease=true
-                should_build=false
-                should_push=false
-                echo "⏭️  Preview tags do not publish Docker images"
-                ;;
               # Prerelease versions (must match first, more specific)
               v*alpha*|v*beta*|v*rc*|*alpha*|*beta*|*rc*)
                 build_type="prerelease"
@@ -264,35 +245,15 @@ jobs:
                 echo "⚠️  Only release versions (latest, v1.0.0, 1.0.0) and prereleases (v1.0.0-alpha1, 1.0.0-beta2) are supported"
                 ;;
             esac
-
-            if [[ "$should_build" == true && "$input_version" != "latest" ]]; then
-              tag_ref="refs/tags/$input_version"
-              if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
-                if [[ "$input_version" == v* ]]; then
-                  tag_ref="refs/tags/${input_version#v}"
-                else
-                  tag_ref="refs/tags/v$input_version"
-                fi
-              fi
-
-              if ! git ls-remote --exit-code origin "$tag_ref" >/dev/null 2>&1; then
-                echo "❌ Release tag not found for Docker build: $input_version"
-                exit 1
-              fi
-              source_ref="$tag_ref"
-            fi
           fi
 
-          {
-            echo "should_build=$should_build"
-            echo "should_push=$should_push"
-            echo "build_type=$build_type"
-            echo "version=$version"
-            echo "short_sha=$short_sha"
-            echo "is_prerelease=$is_prerelease"
-            echo "create_latest=$create_latest"
-            echo "source_ref=$source_ref"
-          } >> "$GITHUB_OUTPUT"
+          echo "should_build=$should_build" >> $GITHUB_OUTPUT
+          echo "should_push=$should_push" >> $GITHUB_OUTPUT
+          echo "build_type=$build_type" >> $GITHUB_OUTPUT
+          echo "version=$version" >> $GITHUB_OUTPUT
+          echo "short_sha=$short_sha" >> $GITHUB_OUTPUT
+          echo "is_prerelease=$is_prerelease" >> $GITHUB_OUTPUT
+          echo "create_latest=$create_latest" >> $GITHUB_OUTPUT
 
           echo "🐳 Docker Build Summary:"
           echo "  - Should build: $should_build"
@@ -302,7 +263,6 @@ jobs:
           echo "  - Short SHA: $short_sha"
           echo "  - Is prerelease: $is_prerelease"
           echo "  - Create latest: $create_latest"
-          echo "  - Source ref: $source_ref"
 
   # Build multi-arch Docker images
   # Strategy: Build images using pre-built binaries from dl.rustfs.com
@@ -328,9 +288,6 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-          ref: ${{ needs.build-check.outputs.source_ref }}
 
       - name: Login to Docker Hub
         uses: docker/login-action@c94ce9fb468520275223c153574b00df6fe4bcc9 # v3
@@ -363,31 +320,36 @@ jobs:
         run: |
           BUILD_TYPE="${{ needs.build-check.outputs.build_type }}"
           VERSION="${{ needs.build-check.outputs.version }}"
+          SHORT_SHA="${{ needs.build-check.outputs.short_sha }}"
           CREATE_LATEST="${{ needs.build-check.outputs.create_latest }}"
           VARIANT_SUFFIX="${{ matrix.suffix }}"
 
-          # Convert version format for Dockerfile compatibility. The former
-          # DOCKER_CHANNEL was "release" down every branch and was passed as a
-          # build-arg no Dockerfile declares, so it is gone.
+          # Convert version format for Dockerfile compatibility
           case "$VERSION" in
             "latest")
+              # For stable latest, use RELEASE=latest + release CHANNEL
               DOCKER_RELEASE="latest"
+              DOCKER_CHANNEL="release"
               ;;
             v*)
               # For versioned releases (v1.0.0), remove 'v' prefix for Dockerfile
               DOCKER_RELEASE="${VERSION#v}"
+              DOCKER_CHANNEL="release"
               ;;
             *)
               # For other versions, pass as-is
               DOCKER_RELEASE="${VERSION}"
+              DOCKER_CHANNEL="release"
               ;;
           esac
 
-          echo "docker_release=$DOCKER_RELEASE" >> "$GITHUB_OUTPUT"
+          echo "docker_release=$DOCKER_RELEASE" >> $GITHUB_OUTPUT
+          echo "docker_channel=$DOCKER_CHANNEL" >> $GITHUB_OUTPUT
 
           echo "🐳 Docker build parameters:"
           echo "  - Original version: $VERSION"
           echo "  - Docker RELEASE: $DOCKER_RELEASE"
+          echo "  - Docker CHANNEL: $DOCKER_CHANNEL"
 
           # Generate tags based on build type
           # Only support release and prerelease builds (no development builds)
@@ -414,19 +376,18 @@ jobs:
           fi
 
           # Output tags
-          echo "tags=$TAGS" >> "$GITHUB_OUTPUT"
+          echo "tags=$TAGS" >> $GITHUB_OUTPUT
 
           # Generate labels
           LABELS="org.opencontainers.image.title=RustFS"
           LABELS="$LABELS,org.opencontainers.image.description=RustFS distributed object storage system"
           LABELS="$LABELS,org.opencontainers.image.version=$VERSION"
-          SOURCE_REVISION="$(git rev-parse HEAD)"
-          LABELS="$LABELS,org.opencontainers.image.revision=$SOURCE_REVISION"
+          LABELS="$LABELS,org.opencontainers.image.revision=${{ github.sha }}"
           LABELS="$LABELS,org.opencontainers.image.source=${{ github.server_url }}/${{ github.repository }}"
           LABELS="$LABELS,org.opencontainers.image.created=$(date -u +'%Y-%m-%dT%H:%M:%SZ')"
           LABELS="$LABELS,org.opencontainers.image.build-type=$BUILD_TYPE"
 
-          echo "labels=$LABELS" >> "$GITHUB_OUTPUT"
+          echo "labels=$LABELS" >> $GITHUB_OUTPUT
 
           echo "🐳 Generated Docker tags:"
           echo "$TAGS" | tr ',' '\n' | sed 's/^/  - /'
@@ -442,24 +403,18 @@ jobs:
           push: ${{ needs.build-check.outputs.should_push == 'true' }}
           tags: ${{ steps.meta.outputs.tags }}
           labels: ${{ steps.meta.outputs.labels }}
-          # No layer cache. This build compiles nothing — it downloads a
-          # release zip and runs apk/apt — so the cache could only save the
-          # minute or two those take, while creating a correctness problem: with
-          # RELEASE=latest the binary URL is resolved by curl *inside* a RUN
-          # layer, and the layer key does not include what that resolved to. A
-          # rebuild at the same RELEASE value (dispatch with version=latest, or
-          # a re-run of the same version) would hit the old layer and ship the
-          # previous release's binary. mode=max also consumed the same 10GB
-          # Actions cache quota the Rust lanes are fighting over.
-          #
-          # Only RELEASE is passed: it is the sole build-arg the Dockerfiles
-          # declare besides TARGETARCH. BUILDTIME, VERSION, BUILD_TYPE, REVISION
-          # and CHANNEL were never read by any stage (and BUILDTIME's $(date ...)
-          # was a literal here, not a shell substitution). BUILD_DATE and VCS_REF
-          # are declared by the Dockerfiles but deliberately left unset —
-          # supplying them would change the published image labels.
+          cache-from: |
+            type=gha,scope=docker-${{ matrix.variant }}
+          cache-to: |
+            type=gha,mode=max,scope=docker-${{ matrix.variant }}
           build-args: |
+            BUILDTIME=$(date -u +'%Y-%m-%dT%H:%M:%SZ')
+            VERSION=${{ needs.build-check.outputs.version }}
+            BUILD_TYPE=${{ needs.build-check.outputs.build_type }}
+            REVISION=${{ github.sha }}
             RELEASE=${{ steps.meta.outputs.docker_release }}
+            CHANNEL=${{ steps.meta.outputs.docker_channel }}
+            BUILDKIT_INLINE_CACHE=1
           provenance: true
           sbom: true
           # Add retry mechanism by splitting the build process
@@ -475,7 +430,6 @@ jobs:
     needs: [ build-check, build-docker ]
     if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     permissions:
       contents: read
       security-events: write
@@ -530,7 +484,6 @@ jobs:
     needs: [ build-check, build-docker ]
     if: always() && needs.build-check.outputs.should_build == 'true'
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - name: Docker build completion summary
         run: |

.github/workflows/e2e-replication-nightly.yml

@@ -12,28 +12,28 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# Consolidated nightly e2e lane for replication, cluster faults, and protocols.
+# Nightly full replication e2e lane (backlog#1147 repl-1, deps: ci-4).
 #
-# The per-PR gate (ci.yml `e2e-tests` job, `--profile e2e-smoke`) runs the
-# FAST replication tests. This scheduled lane runs the remaining heavier
-# replication e2e tests that are unfit for a per-PR gate: remote-target TLS
-# validation, bucket-replication data-plane/helper tests (PUT/delete + poll
-# for convergence, HTTPS targets, active SSE failure contracts, event/history
-# observers), and the `_real_dual_node` / `_real_three_node` /
-# `_real_single_node` site-replication tests that each spawn full rustfs
-# server processes.
+# The per-PR gate (ci.yml `e2e-tests` job, `--profile e2e-smoke`) runs the 20
+# FAST bucket-replication tests. This scheduled lane runs the remaining 18
+# heavier replication e2e tests that are unfit for a per-PR gate:
+#
+#   * 8 bucket-replication data-plane tests (PUT/delete + poll for convergence;
+#     two replicate over HTTPS).
+#   * 9 `_real_dual_node` site-replication tests (each spawns TWO rustfs
+#     servers and drives the cross-process site-replication control plane).
+#   * 1 `_real_single_node` service-account round-trip test.
 #
 # The selection is the [profile.e2e-repl-nightly] default-filter in
 # .config/nextest.toml — the single wiring mechanism (repl-1 / ci-4). Do NOT
-# add ad-hoc cargo-test steps here; change the filterset instead. The
-# authoritative membership and count come from
-# `cargo nextest list -p e2e_test --profile e2e-repl-nightly`; the PR/nightly
-# selection digest is committed under .config/.
+# add ad-hoc cargo-test steps here; change the filterset instead.
 #
-# Explicit division of labor: these subsets run only here and never double-run
-# in the e2e-full merge gate.
+# Explicit division of labor: these 18 tests run ONLY here, never double-run
+# in ci-5's future e2e-full merge gate. TODO(ci-7): once the ci domain's
+# consolidated scheduled e2e workflow exists, fold this interim repl-owned lane
+# into it rather than growing a second scheduled entrypoint.
 
-name: e2e-nightly
+name: e2e-replication-nightly
 
 on:
   workflow_dispatch:
@@ -47,10 +47,6 @@ on:
 permissions:
   contents: read
 
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: false
-
 jobs:
   repl-nightly:
     name: Replication e2e (nightly)
@@ -64,16 +60,14 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: stable
           cache-shared-key: ci-e2e-repl
+          github-token: ${{ secrets.GITHUB_TOKEN }}
           cache-save-if: ${{ github.ref == 'refs/heads/main' }}
-          install-build-packaging-tools: 'false'
 
       # awscurl lets the STS dual-node test actually exercise its path. Without
       # it the test skips gracefully with a visible log line
@@ -86,32 +80,16 @@ jobs:
           python-version: "3.12"
 
       - name: Install awscurl
-        run: |
-          python3 -m pip install --user --upgrade pip awscurl
-          echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
-
-      - name: Verify awscurl
-        run: test -x "$AWSCURL_PATH"
+        run: python3 -m pip install --user --upgrade pip awscurl
 
       # Build the rustfs binary once up front. The e2e tests spawn it as a
       # child process (crates/e2e_test/src/common.rs) and will build it on
       # demand otherwise, but a single explicit build avoids several parallel
       # nextest test processes racing to build it at once.
       - name: Build rustfs binary
-        run: |
-          cargo build -p rustfs --bins
-          : > target/debug/rustfs.features
-
-      - name: Verify replication e2e membership
-        env:
-          NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-repl-nightly-list.json
-        run: |
-          cargo nextest list --profile e2e-repl-nightly -p e2e_test --message-format json > "${NEXTEST_LISTING}"
-          python3 ./scripts/check_test_wiring.py --check-profile e2e-repl-nightly "${NEXTEST_LISTING}"
+        run: cargo build -p rustfs --bins
 
       - name: Run replication e2e nightly suite
-        env:
-          RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-repl-nightly-logs
         run: cargo nextest run --profile e2e-repl-nightly -p e2e_test
 
       - name: Upload nextest junit report
@@ -119,112 +97,13 @@ jobs:
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
         with:
           name: e2e-replication-nightly-junit-${{ github.run_number }}
-          path: |
-            target/nextest/e2e-repl-nightly/junit.xml
-            ${{ runner.temp }}/rustfs-e2e-repl-nightly-list.json
-            ${{ runner.temp }}/rustfs-e2e-repl-nightly-logs/
+          path: target/nextest/e2e-repl-nightly/junit.xml
           retention-days: 7
           if-no-files-found: ignore
 
-  cluster-nightly:
-    name: Cluster fault e2e (nightly)
-    runs-on: sm-standard-4
-    timeout-minutes: 90
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-e2e-nightly
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
-
-      - name: Build rustfs binary
-        run: |
-          cargo build -p rustfs --bins --features e2e-test-hooks
-          : > target/debug/rustfs.features
-
-      - name: Verify cluster fault e2e membership
-        env:
-          NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-nightly-list.json
-        run: |
-          cargo nextest list --profile e2e-nightly -p e2e_test --message-format json > "${NEXTEST_LISTING}"
-          python3 ./scripts/check_test_wiring.py --check-profile e2e-nightly "${NEXTEST_LISTING}"
-
-      - name: Run cluster fault e2e nightly suite
-        env:
-          RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-nightly-logs
-        run: cargo nextest run --profile e2e-nightly -p e2e_test
-
-      - name: Upload cluster fault diagnostics
-        if: always()
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: e2e-cluster-nightly-${{ github.run_number }}
-          path: |
-            target/nextest/e2e-nightly/junit.xml
-            ${{ runner.temp }}/rustfs-e2e-nightly-list.json
-            ${{ runner.temp }}/rustfs-e2e-nightly-logs/
-          retention-days: 7
-          if-no-files-found: warn
-
-  protocols-nightly:
-    name: Protocol e2e (nightly)
-    runs-on: sm-standard-4
-    timeout-minutes: 90
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-      RUSTFS_BUILD_FEATURES: ftps,webdav,sftp
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-e2e-protocols
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
-
-      # The suite owns fixed protocol ports and serializes its internal cases.
-      - name: Verify protocol e2e membership
-        env:
-          NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-protocols-list.json
-        run: |
-          cargo nextest list --profile e2e-protocols -p e2e_test --message-format json > "${NEXTEST_LISTING}"
-          python3 ./scripts/check_test_wiring.py --check-profile e2e-protocols "${NEXTEST_LISTING}"
-
-      - name: Run protocol e2e nightly suite
-        env:
-          RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-protocol-e2e-logs
-        run: >-
-          cargo nextest run -j 1 --profile e2e-protocols -p e2e_test --no-capture
-
-      - name: Upload protocol diagnostics
-        if: always()
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: e2e-protocol-nightly-${{ github.run_number }}
-          path: |
-            target/nextest/e2e-protocols/junit.xml
-            ${{ runner.temp }}/rustfs-e2e-protocols-list.json
-            ${{ runner.temp }}/rustfs-protocol-e2e-logs/
-          retention-days: 7
-          if-no-files-found: warn
-
   alert-on-failure:
     name: Alert on scheduled failure
-    needs: [repl-nightly, cluster-nightly, protocols-nightly]
+    needs: [repl-nightly]
     # Only scheduled runs open/append the tracking issue (backlog#1149 ci-8);
     # manual workflow_dispatch runs stay quiet so a debugging run never files a
     # spurious alert.
@@ -236,8 +115,6 @@ jobs:
       issues: write
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
       - name: Open or update failure-tracking issue
         uses: ./.github/actions/schedule-failure-issue
         with:

.github/workflows/e2e-s3tests.yml

@@ -18,9 +18,10 @@
 # runs only the implemented_tests.txt whitelist. This workflow complements it:
 #
 #   - Scheduled weekly full sweep (TEST_SCOPE=all): runs the ENTIRE upstream
-#     suite and reports promotion candidates. Regressions, unclassified tests,
-#     incomplete execution, and infrastructure errors fail the job; classified
-#     failures for not-yet-implemented features remain informational.
+#     suite and reports promotion candidates (tests that newly pass) and
+#     unclassified tests. The job fails only on regressions in the implemented
+#     whitelist or on infrastructure errors — expected failures from
+#     not-yet-implemented features do not turn the run red.
 #   - Manual runs (workflow_dispatch): same, with configurable mode/scope.
 #
 # All test execution is delegated to scripts/s3-tests/run.sh (single source of
@@ -73,19 +74,6 @@ on:
         description: "Stop after N failures. '0' to run everything."
         required: false
         default: "0"
-      shard-count:
-        description: "Exact-node-ID shard count for a targeted manual run"
-        required: false
-        default: "1"
-        type: choice
-        options:
-          - "1"
-          - "2"
-          - "4"
-      shard-index:
-        description: "Zero-based shard index for a targeted manual run"
-        required: false
-        default: "0"
       markexpr:
         description: "Optional pytest -m expression"
         required: false
@@ -116,8 +104,6 @@ env:
   XDIST: ${{ github.event.inputs.xdist || '4' }}
   MAXFAIL: ${{ github.event.inputs.maxfail || '0' }}
   MARKEXPR: ${{ github.event.inputs.markexpr || '' }}
-  S3_SHARD_COUNT: ${{ github.event_name == 'schedule' && '4' || github.event.inputs.shard-count || '1' }}
-  TEST_TIMEOUT: "300"
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event.inputs['test-mode'] || 'single' }}
@@ -134,26 +120,21 @@ defaults:
 
 jobs:
   s3tests:
-    name: s3tests (${{ matrix.test-mode }}, shard ${{ matrix.shard-index }})
     # GitHub-hosted: reliably provides Docker + docker compose + python3/pip.
     # See the header note (ci-1) for why the self-hosted sm-standard-4 label
-    # was abandoned. Scheduled failures are handled by alert-on-failure below.
+    # was abandoned. TODO(ci-8): scheduled-failure alerting (auto-open issue)
+    # is added by the ci-8 composite action; do not implement it here.
     runs-on: ubuntu-latest
     timeout-minutes: 180
     strategy:
       fail-fast: false
-      max-parallel: 2
       matrix:
         # Scheduled sweeps cover both topologies; manual runs use the input.
         test-mode: ${{ github.event_name == 'schedule' && fromJSON('["single", "multi"]') || fromJSON(format('["{0}"]', github.event.inputs.test-mode || 'single')) }}
-        shard-index: ${{ github.event_name == 'schedule' && fromJSON('[0, 1, 2, 3]') || fromJSON(format('[{0}]', github.event.inputs.shard-index || '0')) }}
     env:
       TEST_MODE: ${{ matrix.test-mode }}
-      S3_SHARD_INDEX: ${{ matrix.shard-index }}
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       # Provision Python explicitly rather than trusting the runner image to
       # ship a working pip (ci-1: a bare python3 without pip is what broke the
@@ -191,7 +172,6 @@ jobs:
       - name: Start single RustFS
         if: env.TEST_MODE == 'single'
         run: |
-          SSE_KEY="$(head -c 32 /dev/zero | base64 -w0)"
           docker network inspect rustfs-net >/dev/null 2>&1 || docker network create rustfs-net
           docker rm -f rustfs-single >/dev/null 2>&1 || true
           # The four disks share one physical device on the runner (a single
@@ -204,7 +184,6 @@ jobs:
             -e RUSTFS_ADDRESS=0.0.0.0:9000 \
             -e RUSTFS_ACCESS_KEY="${S3_ACCESS_KEY}" \
             -e RUSTFS_SECRET_KEY="${S3_SECRET_KEY}" \
-            -e RUSTFS_SSE_S3_MASTER_KEY="${SSE_KEY}" \
             -e RUSTFS_VOLUMES="/data/rustfs{0...3}" \
             -e RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true \
             -v /tmp/rustfs-single:/data \
@@ -213,7 +192,6 @@ jobs:
       - name: Start 4-node distributed cluster
         if: env.TEST_MODE == 'multi'
         run: |
-          SSE_KEY="$(head -c 32 /dev/zero | base64 -w0)"
           # A real distributed deployment: every node lists all endpoints in
           # RUSTFS_VOLUMES so data is erasure-coded ACROSS nodes. Do not use
           # node-local volume paths here — that would create four independent
@@ -226,7 +204,6 @@ jobs:
               RUSTFS_ADDRESS: "0.0.0.0:9000"
               RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
               RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
-              RUSTFS_SSE_S3_MASTER_KEY: "${SSE_KEY}"
               RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
               # Each node's four disks share one physical device inside its
               # container, so bypass the local physical-disk-independence guard
@@ -308,6 +285,7 @@ jobs:
 
       - name: Run ceph s3-tests
         run: |
+          set +e
           DEPLOY_MODE=existing \
           TEST_MODE="${TEST_MODE}" \
           TEST_SCOPE="${TEST_SCOPE}" \
@@ -315,6 +293,26 @@ jobs:
           MAXFAIL="${MAXFAIL}" \
           MARKEXPR="${MARKEXPR}" \
           ./scripts/s3-tests/run.sh
+          RC=$?
+          set -e
+
+          if [ "${TEST_SCOPE}" = "implemented" ]; then
+            # Whitelist run: every failure is a regression.
+            exit "${RC}"
+          fi
+
+          # Full sweep: failures outside the implemented whitelist are
+          # inventory (promotion candidates / unimplemented features), not a
+          # gate. Fail only on whitelist regressions or infrastructure errors.
+          JUNIT="artifacts/s3tests-${TEST_MODE}/junit.xml"
+          if [ ! -f "${JUNIT}" ]; then
+            echo "No junit.xml produced — infrastructure failure (exit ${RC})" >&2
+            exit "${RC}"
+          fi
+          python3 scripts/s3-tests/report_compat.py \
+            --junit "${JUNIT}" \
+            --lists-dir scripts/s3-tests \
+            --fail-on-regression
 
       - name: Publish compatibility report
         if: always()
@@ -339,7 +337,7 @@ jobs:
         if: always() && env.ACT != 'true'
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
         with:
-          name: s3tests-${{ env.TEST_MODE }}-shard-${{ matrix.shard-index }}
+          name: s3tests-${{ env.TEST_MODE }}
           path: artifacts/**
 
   alert-on-failure:
@@ -356,8 +354,6 @@ jobs:
       issues: write
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
       - name: Open or update failure-tracking issue
         uses: ./.github/actions/schedule-failure-issue
         with:

.github/workflows/fuzz.yml

@@ -17,17 +17,15 @@ name: Fuzz
 on:
   pull_request:
     types: [ opened, synchronize, reopened, closed ]
-    # Run when the harness or any directly fuzzed production crate changes.
+    # PR trigger is intentionally narrow: only changes to the fuzz harness
+    # itself gate a PR. Broad crate paths (ecstore/filemeta/utils/policy/…)
+    # are covered by the nightly `schedule` run below, which fuzzes against
+    # whatever landed on main. Widening these paths previously queued a
+    # ~45min fuzz-build on nearly every PR and is why this workflow was
+    # disabled; do not re-add crate paths here.
     paths:
       - "fuzz/**"
       - "scripts/fuzz/**"
-      - "crates/ecstore/**"
-      - "crates/filemeta/**"
-      - "crates/policy/**"
-      - "crates/security-governance/**"
-      - "crates/utils/**"
-      - "Cargo.toml"
-      - "Cargo.lock"
       - ".github/workflows/fuzz.yml"
   schedule:
     - cron: "0 2 * * *"
@@ -61,7 +59,6 @@ jobs:
     name: Cancel Closed PR Runs
     if: github.event_name == 'pull_request' && github.event.action == 'closed'
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - name: Explain cancellation run
         run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -76,20 +73,19 @@ jobs:
       github.event_name == 'schedule' ||
       github.event_name == 'workflow_dispatch'
     runs-on: sm-standard-4
-    timeout-minutes: 60
+    timeout-minutes: 45
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Setup Rust environment
         uses: ./.github/actions/setup
         with:
           rust-version: nightly
           cache-shared-key: fuzz-${{ hashFiles('fuzz/Cargo.lock') }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
           cache-save-if: ${{ github.ref == 'refs/heads/main' || github.event_name == 'schedule' }}
 
       - name: Install cargo-fuzz
@@ -116,7 +112,12 @@ jobs:
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
         with:
           name: fuzz-prebuilt-binaries-${{ github.run_number }}
-          path: fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/
+          path: |
+            fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/archive_extract
+            fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/bucket_validation
+            fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/local_metadata
+            fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/path_containment
+            fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/policy_ingress
           if-no-files-found: error
           retention-days: 1
           compression-level: 0
@@ -144,8 +145,6 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Download prebuilt fuzz binaries
         uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
@@ -182,7 +181,10 @@ jobs:
   nightly-fuzz-corpus:
     name: "Nightly / ${{ matrix.target }}"
     needs: fuzz-build
-    # Scheduled failures are handled by alert-on-failure below.
+    # TODO(ci-8): when the schedule-failure-issue composite action lands,
+    # add a step here (or a dependent job) that opens/updates a GitHub issue
+    # on nightly failure. ci-8 is the single alerting mechanism for all
+    # scheduled workflows; do not self-roll alerting in this workflow.
     if: >
       github.event_name == 'schedule' ||
       (github.event_name == 'workflow_dispatch' &&
@@ -198,8 +200,6 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Download prebuilt fuzz binaries
         uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
@@ -247,8 +247,6 @@ jobs:
       issues: write
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
       - name: Open or update failure-tracking issue
         uses: ./.github/actions/schedule-failure-issue
         with:

.github/workflows/helm-package.yml

@@ -32,14 +32,12 @@ permissions:
 jobs:
   build-helm-package:
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     if: |
-      (github.event_name == 'workflow_dispatch' && !contains(github.event.inputs.version, '-preview')) ||
+      github.event_name == 'workflow_dispatch' ||
       (
         github.event.workflow_run.conclusion == 'success' &&
         github.event.workflow_run.event == 'push' &&
-        contains(github.event.workflow_run.head_branch, '.') &&
-        !contains(github.event.workflow_run.head_branch, '-preview')
+        contains(github.event.workflow_run.head_branch, '.')
       )
 
     outputs:
@@ -50,26 +48,16 @@ jobs:
     steps:
       - name: Checkout helm chart repo
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
-      # Both inputs reach the shell through env rather than `${{ }}`
-      # interpolation. A git ref name may contain `$(...)` — anything without a
-      # space is a legal tag — and interpolation pastes it into the script
-      # verbatim, where bash would run it. Reading "$RAW_INPUT" instead makes it
-      # data.
       - name: Normalize release version
         id: version
-        env:
-          RAW_INPUT: ${{ github.event.inputs.version }}
-          RAW_BRANCH: ${{ github.event.workflow_run.head_branch }}
         run: |
           set -eux
 
           if [ "${{ github.event_name }}" = "workflow_dispatch" ]; then
-            RAW="$RAW_INPUT"
+            RAW="${{ github.event.inputs.version }}"
           else
-            RAW="$RAW_BRANCH"
+            RAW="${{ github.event.workflow_run.head_branch }}"
           fi
 
           case "$RAW" in
@@ -84,13 +72,10 @@ jobs:
           ./scripts/helm_chart_version.sh "$RAW_TAG"
 
       - name: Replace chart version and app version
-        env:
-          CHART_VERSION: ${{ steps.version.outputs.chart_version }}
-          APP_VERSION: ${{ steps.version.outputs.app_version }}
         run: |
           set -eux
-          sed -i -E "s/^version:.*/version: \"${CHART_VERSION}\"/" helm/rustfs/Chart.yaml
-          sed -i -E "s/^appVersion:.*/appVersion: \"${APP_VERSION}\"/" helm/rustfs/Chart.yaml
+          sed -i -E 's/^version:.*/version: "${{ steps.version.outputs.chart_version }}"/' helm/rustfs/Chart.yaml
+          sed -i -E 's/^appVersion:.*/appVersion: "${{ steps.version.outputs.app_version }}"/' helm/rustfs/Chart.yaml
 
       - name: Set up Helm
         uses: azure/setup-helm@b9e51907a09c216f16ebe8536097933489208112 # v4.3.0
@@ -115,7 +100,6 @@ jobs:
 
   publish-helm-package:
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     needs: [ build-helm-package ]
     if: needs.build-helm-package.result == 'success'
 
@@ -123,8 +107,6 @@ jobs:
       - name: Checkout helm package repo
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
         with:
-          # persist-credentials-exempt: this checkout's token IS the push credential —
-          # the job git-pushes to rustfs/helm below. Clearing it breaks chart publishing.
           repository: rustfs/helm
           token: ${{ secrets.RUSTFS_HELM_PACKAGE }}
 
@@ -140,19 +122,11 @@ jobs:
       - name: Generate index
         run: helm repo index . --url https://charts.rustfs.com
 
-      # app_version is derived from the triggering tag name, and this job holds
-      # the cross-repository push token with rustfs/helm already checked out —
-      # the worst place in the repo to paste an attacker-influenced string into
-      # a shell line. Passed through env so bash treats it as data.
       - name: Push helm package and index file
-        env:
-          GIT_USERNAME: ${{ secrets.USERNAME }}
-          GIT_EMAIL: ${{ secrets.EMAIL_ADDRESS }}
-          APP_VERSION: ${{ needs.build-helm-package.outputs.app_version }}
         run: |
           set -eux
-          git config --global user.name "${GIT_USERNAME}"
-          git config --global user.email "${GIT_EMAIL}"
+          git config --global user.name "${{ secrets.USERNAME }}"
+          git config --global user.email "${{ secrets.EMAIL_ADDRESS }}"
           git add .
-          git commit -m "Update rustfs helm package with ${APP_VERSION}." || echo "No changes to commit"
+          git commit -m "Update rustfs helm package with ${{ needs.build-helm-package.outputs.app_version }}." || echo "No changes to commit"
           git push origin main

.github/workflows/issue-translator.yml

@@ -12,13 +12,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# DISABLED. This workflow is switched off in the repository's Actions settings
-# (state: disabled_manually) and does not run on any trigger, including its cron
-# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
-# reading this file, which has already misled at least one audit — hence this
-# banner. Re-enabling is a UI action; anyone doing so should first check that the
-# workflow still matches the current CI layout. See rustfs/backlog#1603.
-#
 name: "issue-translator"
 on:
   issue_comment:
@@ -33,7 +26,6 @@ permissions:
 jobs:
   build:
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - uses: usthe/issues-translate-action@b41f55ddc81d7d54bd542a4f289fe28ec081898e # v2.7
         with:

.github/workflows/minio-interop.yml

@@ -1,123 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# MinIO on-disk interop: prove RustFS reads MinIO-written erasure-coded SSE
-# objects with byte-identical data and correct logical size.
-#
-# This is NOT a PR gate. The fixtures are real MinIO backend trees generated on
-# the fly (they are gitignored, never committed), so the job regenerates them
-# each run with Docker and then runs the `#[ignore]` reader tests in
-# rustfs/src/storage/minio_generated_read_test.rs.
-#
-# Scope: end-to-end MinIO-to-RustFS SSE interop is NOT implemented yet. Both
-# envelope parsers reject MinIO's own wrapped-DEK shape — see
-# `is_data_key_envelope` in crates/kms/src/encryption/dek.rs and the
-# `deny_unknown_fields` `LocalSseDekEnvelope` in rustfs/src/storage/sse.rs — and
-# closing that gap is tracked in rustfs/backlog#1638. Treat this job as the
-# harness for #1638, not as standing evidence that a MinIO migration reads back.
-#
-# Runner: GitHub-hosted `ubuntu-latest`. It reliably ships Docker + Python,
-# unlike the self-hosted fleet, whose pods drift in Docker/pip availability
-# (see the infra note in e2e-s3tests.yml). Nightly + manual only.
-# DISABLED. This workflow is switched off in the repository's Actions settings
-# (state: disabled_manually) and does not run on any trigger, including its cron
-# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
-# reading this file, which has already misled at least one audit — hence this
-# banner. Re-enabling is a UI action; anyone doing so should first check that the
-# workflow still matches the current CI layout. See rustfs/backlog#1603.
-#
-name: minio-interop
-
-on:
-  workflow_dispatch:
-  schedule:
-    # Nightly at 03:17 UTC (offset from other nightly jobs).
-    - cron: "17 3 * * *"
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.ref }}
-  cancel-in-progress: true
-
-permissions:
-  contents: read
-
-jobs:
-  minio-interop:
-    name: MinIO interop (EC + SSE read parity)
-    # Skip on forks: needs the repo's runners and is not a contributor gate.
-    if: github.repository == 'rustfs/rustfs'
-    runs-on: ubuntu-latest
-    timeout-minutes: 40
-    env:
-      # Fixed 32-byte test KMS key baked into the fixture lab; not a secret.
-      RUSTFS_MINIO_STATIC_KMS_KEY_B64: IyqsU3kMFloCNup4BsZtf/rmfHVcTgznO2F25CkEH1g=
-      # Single definition of "the interop tests", shared by the guard step and
-      # the run step so the two cannot drift apart.
-      #
-      # These used to live in crates/ecstore/tests/minio_generated_read_test.rs
-      # and were selected with `-p rustfs-ecstore -E
-      # 'binary(minio_generated_read_test)'`. #5435 moved them into the `rustfs`
-      # crate as a `#[cfg(test)] mod`, which deleted that test binary; the
-      # selector was never updated and has selected zero interop tests ever
-      # since (cargo-nextest 0.9.140 now rejects it outright: "operator didn't
-      # match any binary names", exit 94).
-      INTEROP_PACKAGE: rustfs
-      INTEROP_FEATURES: rio-v2
-      INTEROP_FILTER: "test(minio_generated_read_test::)"
-      INTEROP_REQUIRED_TESTS: '["reads_minio_generated_sse_s3_multipart_fixture", "reads_minio_generated_sse_kms_multipart_fixture", "rejects_minio_generated_sse_s3_fixture_with_wrong_kms_key", "rejects_minio_generated_sse_s3_fixture_with_truncated_ciphertext"]'
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-minio-interop
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
-
-      - name: Generate real MinIO fixtures via Docker
-        run: bash crates/rio-v2/tests/minio_fixture_lab/capture_via_docker.sh
-
-      # `binary(...)` at least dies loudly when nothing matches, but `test(...)`
-      # is a perfectly valid filterset that matches zero tests, so the next
-      # rename or module move would leave this job selecting nothing and
-      # reporting success without executing a single interop assertion. Count
-      # the selection and require every core reader test, while allowing new
-      # reader cases to be added without changing this guard.
-      #
-      # Count only `filter-match.status == "matches"`: the top-level
-      # `test-count` in the JSON is the package total and ignores `-E` entirely.
-      - name: Assert the interop selector still matches tests
-        run: |
-          set -euo pipefail
-          selection="$(cargo nextest list --run-ignored ignored-only \
-            -p "$INTEROP_PACKAGE" --features "$INTEROP_FEATURES" \
-            -E "$INTEROP_FILTER" --message-format json \
-            | python3 -c 'import json,os,sys; d=json.load(sys.stdin); required=json.loads(os.environ["INTEROP_REQUIRED_TESTS"]); matched=[name for suite in d.get("rust-suites", {}).values() for name,test in suite.get("testcases", {}).items() if test.get("filter-match", {}).get("status") == "matches"]; missing=[test for test in required if not any(name.endswith("minio_generated_read_test::" + test) for name in matched)]; print(len(matched)); print(",".join(missing))')"
-          count="$(printf '%s\n' "$selection" | sed -n '1p')"
-          missing="$(printf '%s\n' "$selection" | sed -n '2p')"
-          echo "interop tests selected: ${count}"
-          if [ -n "${missing}" ]; then
-            echo "::error::Selector '${INTEROP_FILTER}' in package '${INTEROP_PACKAGE}' is missing required tests: ${missing}. The MinIO interop reader tests have moved or been renamed; fix the selector instead of running an incomplete matrix. Context: rustfs/backlog#1638."
-            exit 1
-          fi
-
-      - name: Run MinIO interop reader tests
-        run: |
-          cargo nextest run --run-ignored ignored-only --no-tests=fail \
-            -p "$INTEROP_PACKAGE" --features "$INTEROP_FEATURES" \
-            -E "$INTEROP_FILTER"

.github/workflows/mint.yml

@@ -45,13 +45,6 @@
 # docker-capable self-hosted `dind-sm-standard-2` label was the alternative but
 # has fewer cores and reintroduces fleet-state risk for no reliability gain.
 
-# DISABLED. This workflow is switched off in the repository's Actions settings
-# (state: disabled_manually) and does not run on any trigger, including its cron
-# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
-# reading this file, which has already misled at least one audit — hence this
-# banner. Re-enabling is a UI action; anyone doing so should first check that the
-# workflow still matches the current CI layout. See rustfs/backlog#1603.
-#
 name: mint
 
 on:
@@ -125,8 +118,6 @@ jobs:
     timeout-minutes: 120
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Enable buildx
         uses: docker/setup-buildx-action@8d2750c68a42422c14e847fe6c8ac0403b4cbd6f # v3
@@ -144,10 +135,6 @@ jobs:
         run: |
           docker network inspect rustfs-net >/dev/null 2>&1 || docker network create rustfs-net
           docker rm -f rustfs-mint >/dev/null 2>&1 || true
-          # The four disks share one physical device on the runner (a single
-          # loopback filesystem), so the local physical-disk-independence guard
-          # would refuse to start. Bypass it — this is the CI use case the guard
-          # explicitly sanctions via RUSTFS_UNSAFE_BYPASS_DISK_CHECK.
           docker run -d --name rustfs-mint \
             --network rustfs-net \
             -p 9000:9000 \
@@ -155,7 +142,6 @@ jobs:
             -e RUSTFS_ACCESS_KEY="${S3_ACCESS_KEY}" \
             -e RUSTFS_SECRET_KEY="${S3_SECRET_KEY}" \
             -e RUSTFS_VOLUMES="/data/rustfs{0...3}" \
-            -e RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true \
             -v /tmp/rustfs-mint:/data \
             rustfs-ci
 
@@ -272,8 +258,6 @@ jobs:
       issues: write
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
       - name: Open or update failure-tracking issue
         uses: ./.github/actions/schedule-failure-issue
         with:

.github/workflows/nightly-gnu.yml

@@ -1,196 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-name: Nightly GNU Build
-
-on:
-  schedule:
-    - cron: "0 0 * * *"
-      timezone: "Asia/Shanghai"
-  workflow_dispatch:
-
-permissions:
-  contents: read
-
-concurrency:
-  group: nightly-gnu-build-main-${{ github.event_name }}
-  cancel-in-progress: ${{ github.event_name == 'workflow_dispatch' }}
-
-env:
-  CARGO_TERM_COLOR: always
-  RUST_BACKTRACE: 1
-
-jobs:
-  build:
-    name: Build x86_64 GNU
-    runs-on: sm-standard-2
-    timeout-minutes: 150
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-    steps:
-      - name: Checkout main branch
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-          ref: main
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          cache-shared-key: build-x86_64-unknown-linux-gnu
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
-          install-test-tools: 'false'
-
-      - name: Build RustFS
-        run: cargo build --release --locked --target x86_64-unknown-linux-gnu -p rustfs --bins
-
-  # Live-Vault lane for the rustfs-kms suite (rustfs/backlog#1774).
-  #
-  # RUSTFS_KMS_VAULT_TOKEN is the single switch that adds the Vault KV2 and
-  # Vault Transit backends to every for_each_backend spec in
-  # crates/kms/tests/behavior_*.rs (see crates/kms/AGENTS.md). rotate and
-  # versioning are advertised only by the Vault backends, so without this lane
-  # no CI run ever asserts the working half of behavior_rotation.rs — a
-  # rotation that silently dropped historical key versions would stay green.
-  # The same lane runs the dev-Vault #[ignore] tests and the two self-hosting
-  # live scripts (AppRole login, three-node Raft leader failover).
-  #
-  # GitHub-hosted ubuntu-latest, deliberately not the self-hosted sm-standard
-  # fleet: the HA failover script needs a working Docker daemon, and the
-  # self-hosted fleet is heterogeneous — a docker-dependent workflow has been
-  # burned by it before (see the banner in e2e-s3tests.yml, rustfs/backlog#1149).
-  kms-vault-lane:
-    name: KMS live Vault lane
-    runs-on: ubuntu-latest
-    timeout-minutes: 90
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-      # Root token of the ephemeral loopback dev server. Not a secret: the
-      # server lives only for this job, listens on 127.0.0.1, and holds only
-      # keys the tests create. The literal value matters — the dev-Vault
-      # #[ignore] fixtures in crates/kms/src/backends/vault.rs hardcode it.
-      VAULT_LANE_TOKEN: dev-only-token
-      VAULT_LANE_ADDR: http://127.0.0.1:8200
-      # Keeps a runner-level proxy from swallowing the loopback dev-server
-      # traffic (see crates/kms/AGENTS.md). Actions env keys are
-      # case-insensitive, so only the uppercase form is set; reqwest reads
-      # either casing.
-      NO_PROXY: 127.0.0.1,localhost
-    steps:
-      - name: Checkout main branch
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-          ref: main
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          # Dedicated key: rust-cache cannot tell runner images apart, so
-          # sharing a key with an sm-standard lane would let two different
-          # system images overwrite each other's artifacts (same reasoning as
-          # ci.yml's ci-uring lane). Saved from this nightly job itself so the
-          # next night starts warm.
-          cache-shared-key: kms-vault-lane
-          cache-save-if: 'true'
-          install-build-packaging-tools: 'false'
-          install-test-tools: 'false'
-
-      - name: Install Vault CLI
-        run: |
-          set -euo pipefail
-          wget -qO- https://apt.releases.hashicorp.com/gpg | sudo gpg --dearmor -o /usr/share/keyrings/hashicorp-archive-keyring.gpg
-          echo "deb [signed-by=/usr/share/keyrings/hashicorp-archive-keyring.gpg] https://apt.releases.hashicorp.com $(lsb_release -cs) main" | sudo tee /etc/apt/sources.list.d/hashicorp.list >/dev/null
-          sudo apt-get update -qq
-          sudo apt-get install -y -qq vault
-          vault version
-
-      - name: Start Vault dev server with KV2 and Transit engines
-        run: |
-          set -euo pipefail
-          nohup vault server -dev \
-            -dev-root-token-id="${VAULT_LANE_TOKEN}" \
-            -dev-listen-address=127.0.0.1:8200 >/tmp/vault-dev.log 2>&1 &
-          for _ in $(seq 1 60); do
-            if curl -fsS "${VAULT_LANE_ADDR}/v1/sys/health" >/dev/null 2>&1; then
-              break
-            fi
-            sleep 1
-          done
-          curl -fsS "${VAULT_LANE_ADDR}/v1/sys/health"
-          export VAULT_ADDR="${VAULT_LANE_ADDR}" VAULT_TOKEN="${VAULT_LANE_TOKEN}"
-          # Dev mode mounts KV v2 at secret/ by default; Transit is explicit.
-          # Prove both engines actually work rather than assuming the defaults.
-          vault secrets enable transit
-          vault kv put secret/rustfs-ci-lane-probe value=ok >/dev/null
-          vault kv get secret/rustfs-ci-lane-probe >/dev/null
-          vault write -f transit/keys/rustfs-ci-lane-probe >/dev/null
-
-      - name: Run rustfs-kms suite with the Vault lane on
-        env:
-          RUSTFS_KMS_VAULT_TOKEN: ${{ env.VAULT_LANE_TOKEN }}
-          RUSTFS_KMS_VAULT_ADDR: ${{ env.VAULT_LANE_ADDR }}
-        run: cargo test -p rustfs-kms --locked
-
-      - name: Run dev-Vault ignored tests
-        env:
-          RUSTFS_KMS_VAULT_TOKEN: ${{ env.VAULT_LANE_TOKEN }}
-          RUSTFS_KMS_VAULT_ADDR: ${{ env.VAULT_LANE_ADDR }}
-        # Filters select the dev-Vault-only #[ignore] tests. The AWS #[ignore]
-        # tests (backends::aws, service_manager) stay excluded — they need real
-        # AWS credentials and create billable keys. The AppRole and HA #[ignore]
-        # tests are excluded here because their own scripts below provision the
-        # Vault topology they need.
-        run: |
-          set -euo pipefail
-          cargo test -p rustfs-kms --locked --lib backends::contract_tests -- --ignored
-          cargo test -p rustfs-kms --locked --lib backends::vault -- --ignored
-          cargo test -p rustfs-kms --locked --test vault_fault_injection -- --ignored
-
-      - name: Run AppRole live checks (self-hosting ephemeral Vault)
-        run: bash scripts/test/vault_approle_kms_live.sh
-
-      - name: Show Vault dev server log on failure
-        if: failure()
-        run: tail -n 200 /tmp/vault-dev.log || true
-
-  # Three-node Raft leader failover (crates/kms/tests/vault_ha_failover_live.rs,
-  # first validated by rustfs/rustfs#5653). Its own job so an election-timing
-  # flake cannot mask the main lane's verdict, and vice versa. The script
-  # provisions and tears down its own Docker cluster.
-  kms-vault-ha-failover:
-    name: KMS Vault HA failover lane
-    runs-on: ubuntu-latest
-    timeout-minutes: 60
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-      NO_PROXY: 127.0.0.1,localhost
-    steps:
-      - name: Checkout main branch
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-          ref: main
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          cache-shared-key: kms-vault-lane
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
-          install-test-tools: 'false'
-
-      - name: Run HA leader failover live checks (three-node Raft cluster in Docker)
-        run: bash scripts/test/vault_ha_kms_live.sh

.github/workflows/nix-flake-update.yml

@@ -19,12 +19,9 @@ on:
   schedule:
     - cron: '0 5 * * 0' # Weekly on Sunday 05:00 UTC (staggered after the midnight ci/build crons)
 
-# GITHUB_TOKEN only needs to read the repository here: the branch push and the
-# pull request are both created by update-flake-lock using the
-# FLAKE_UPDATE_TOKEN PAT below, not by this token. Leaving write on it hands a
-# repo-write credential to an unattended weekly job that does not use it.
 permissions:
-  contents: read
+  contents: write
+  pull-requests: write
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
@@ -40,10 +37,6 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        # persist-credentials-exempt: update-flake-lock pushes the branch and opens
-        # the PR. It passes FLAKE_UPDATE_TOKEN to create-pull-request itself rather
-        # than reusing .git/config, but that is unverified — exempt until a
-        # workflow_dispatch run confirms it (rustfs/backlog#1602).
 
       - name: Install Nix
         uses: DeterminateSystems/determinate-nix-action@629b284231c2a82554b724e357e47fc6020833c8 # v3

.github/workflows/nix.yml

@@ -12,13 +12,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# DISABLED. This workflow is switched off in the repository's Actions settings
-# (state: disabled_manually) and does not run on any trigger, including its cron
-# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
-# reading this file, which has already misled at least one audit — hence this
-# banner. Re-enabling is a UI action; anyone doing so should first check that the
-# workflow still matches the current CI layout. See rustfs/backlog#1603.
-#
 name: Nix CI
 
 on:
@@ -53,7 +46,6 @@ jobs:
     name: Cancel Closed PR Runs
     if: github.event_name == 'pull_request' && github.event.action == 'closed'
     runs-on: ubuntu-latest
-    timeout-minutes: 10
     steps:
       - name: Explain cancellation run
         run: echo "PR closed; this run only cancels older runs in the same concurrency group."
@@ -71,8 +63,6 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
 
       - name: Install Nix
         uses: DeterminateSystems/determinate-nix-action@4eea0b33e3d1f02ecfe37cf16e7204c424009606 # v3.21.0

.github/workflows/package.yml

@@ -1,562 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Package Workflow - Build DEB/RPM packages
-#
-# This workflow builds DEB and RPM packages from pre-built Linux binaries
-# and uploads them to Cloudflare R2 and the GitHub release.
-#
-# Trigger:
-# - workflow_run: automatically package after "Build and Release" completes
-#   for a release tag (the mac/windows/linux binaries are already uploaded
-#   to the GitHub release before packaging starts)
-# - workflow_dispatch: manual fallback (backfill / re-run) with optional tag/run_id
-#
-# Flow:
-# 1. Resolve the triggering Build workflow run for the release tag
-# 2. Download Linux binaries (x86_64-gnu, aarch64-gnu) from build artifacts
-# 3. Build DEB packages for amd64 and arm64
-# 4. Build RPM packages for x86_64 and aarch64
-# 5. Upload all packages to Cloudflare R2 and the GitHub release
-
-name: Package DEB/RPM
-
-permissions:
-  # contents: write is required to upload packages to the GitHub release
-  contents: write
-  actions: read
-
-on:
-  # Follows the same pattern as docker.yml: run after the release build
-  # workflow completes, so packaging is triggered only by release tags
-  # (e.g. 1.0.0-rc.2, 1.0.0-rc.3), never by development builds.
-  workflow_run:
-    workflows: [ "Build and Release" ]
-    types: [ completed ]
-  workflow_dispatch:
-    inputs:
-      tag:
-        description: "Release tag to package (e.g. 1.0.0-beta.12). Leave empty for latest main build."
-        required: false
-        type: string
-      build_run_id:
-        description: "Build workflow run ID (overrides tag lookup)"
-        required: false
-        type: string
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.event.workflow_run.head_branch || github.event.inputs.tag || github.run_id }}
-  cancel-in-progress: true
-
-env:
-  HEAD_BRANCH: ${{ github.event.workflow_run.head_branch }}
-  WORKFLOW_RUN_ID: ${{ github.event.workflow_run.id }}
-
-jobs:
-  # Resolve which build run to use and extract version info
-  resolve:
-    name: Resolve Build
-    # Auto-trigger only from successful tag builds of "Build and Release".
-    # Tag pushes arrive as event == push with head_branch != main (a
-    # non-main push head_branch is the release tag name). Manual dispatch
-    # stays available as a fallback for backfills and re-runs.
-    if: >-
-      github.event_name == 'workflow_dispatch' ||
-      (github.event.workflow_run.conclusion == 'success' &&
-       github.event.workflow_run.event == 'push' &&
-       github.event.workflow_run.head_branch != 'main')
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    outputs:
-      version: ${{ steps.resolve.outputs.version }}
-      build_type: ${{ steps.resolve.outputs.build_type }}
-      build_run_id: ${{ steps.resolve.outputs.build_run_id }}
-      tag: ${{ steps.resolve.outputs.tag }}
-    steps:
-      - name: Resolve build run
-        id: resolve
-        shell: bash
-        env:
-          GH_TOKEN: ${{ github.token }}
-          INPUT_TAG: ${{ github.event.inputs.tag }}
-          INPUT_RUN_ID: ${{ github.event.inputs.build_run_id }}
-        run: |
-          set -euo pipefail
-
-          # Determine tag
-          if [[ "${{ github.event_name }}" == "workflow_run" ]]; then
-            TAG="${HEAD_BRANCH}"
-          elif [[ -n "$INPUT_TAG" ]]; then
-            TAG="$INPUT_TAG"
-          else
-            TAG=""
-          fi
-
-          echo "Tag: ${TAG:-<none>}"
-
-          # Determine build run ID
-          BUILD_RUN_ID=""
-
-          if [[ -n "$INPUT_RUN_ID" ]]; then
-            # Explicit run ID takes priority
-            BUILD_RUN_ID="$INPUT_RUN_ID"
-            echo "Using explicit build run ID: $BUILD_RUN_ID"
-
-          elif [[ "${{ github.event_name }}" == "workflow_run" ]]; then
-            # Use the Build and Release run that triggered this workflow
-            BUILD_RUN_ID="${WORKFLOW_RUN_ID}"
-            echo "Using triggering workflow run: $BUILD_RUN_ID"
-
-          elif [[ -n "$TAG" ]]; then
-            # Find the build run that produced this tag
-            echo "Looking for build run for tag: $TAG"
-            BUILD_RUN_ID=$(gh api \
-              "repos/${{ github.repository }}/actions/workflows/build.yml/runs?branch=${TAG}&status=success&per_page=1" \
-              --jq '.workflow_runs[0].id' 2>/dev/null || echo "")
-
-            if [[ -z "$BUILD_RUN_ID" || "$BUILD_RUN_ID" == "null" ]]; then
-              # Tag might not be a branch; try event=push with head_branch matching
-              BUILD_RUN_ID=$(gh api \
-                "repos/${{ github.repository }}/actions/workflows/build.yml/runs?event=push&status=success&per_page=100" \
-                --jq ".workflow_runs[] | select(.head_branch == \"$TAG\") | .id" 2>/dev/null | head -1 || echo "")
-            fi
-
-            if [[ -z "$BUILD_RUN_ID" || "$BUILD_RUN_ID" == "null" ]]; then
-              echo "❌ No successful build run found for tag: $TAG"
-              exit 1
-            fi
-            echo "Found build run: $BUILD_RUN_ID"
-
-          else
-            # No tag — latest successful main build
-            echo "No tag specified, looking for latest main build"
-            BUILD_RUN_ID=$(gh api \
-              "repos/${{ github.repository }}/actions/workflows/build.yml/runs?branch=main&status=success&per_page=1" \
-              --jq '.workflow_runs[0].id' 2>/dev/null || echo "")
-
-            if [[ -z "$BUILD_RUN_ID" || "$BUILD_RUN_ID" == "null" ]]; then
-              echo "❌ No successful main build found"
-              exit 1
-            fi
-            echo "Latest main build: $BUILD_RUN_ID"
-          fi
-
-          # Determine version and build type
-          if [[ -n "$TAG" ]]; then
-            VERSION="$TAG"
-            if [[ "$TAG" == *"-preview"* ]]; then
-              BUILD_TYPE="preview"
-            elif [[ "$TAG" == *"alpha"* || "$TAG" == *"beta"* || "$TAG" == *"rc"* ]]; then
-              BUILD_TYPE="prerelease"
-            else
-              BUILD_TYPE="release"
-            fi
-          else
-            SHORT_SHA=$(gh api "repos/${{ github.repository }}/actions/runs/${BUILD_RUN_ID}" \
-              --jq '.head_sha' 2>/dev/null | head -c 7)
-            VERSION="dev-${SHORT_SHA}"
-            BUILD_TYPE="development"
-          fi
-
-          {
-            echo "version=$VERSION"
-            echo "build_type=$BUILD_TYPE"
-            echo "build_run_id=$BUILD_RUN_ID"
-            echo "tag=${TAG}"
-          } >> "$GITHUB_OUTPUT"
-
-          echo "📊 Resolved:"
-          echo "  Version: $VERSION"
-          echo "  Build type: $BUILD_TYPE"
-          echo "  Build run ID: $BUILD_RUN_ID"
-
-  # Build DEB and RPM packages for each architecture
-  package:
-    name: Package (${{ matrix.arch }})
-    needs: resolve
-    runs-on: ubuntu-latest
-    timeout-minutes: 30
-    strategy:
-      fail-fast: false
-      max-parallel: 1
-      matrix:
-        include:
-          - arch: x86_64
-            deb_arch: amd64
-            rpm_arch: x86_64
-            artifact_name: "rustfs-linux-x86_64-gnu"
-          - arch: aarch64
-            deb_arch: arm64
-            rpm_arch: aarch64
-            artifact_name: "rustfs-linux-aarch64-gnu"
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Download binary artifact from build run
-        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
-        with:
-          pattern: ${{ matrix.artifact_name }}*
-          path: ./binary-artifact
-          run-id: ${{ needs.resolve.outputs.build_run_id }}
-          github-token: ${{ github.token }}
-          merge-multiple: true
-
-      - name: Extract binary
-        id: binary
-        shell: bash
-        run: |
-          set -euo pipefail
-
-          ZIP_FILE=$(find ./binary-artifact -name "*.zip" -type f | head -1)
-          if [[ -z "$ZIP_FILE" ]]; then
-            echo "❌ No binary artifact found"
-            ls -la ./binary-artifact/ || true
-            exit 1
-          fi
-
-          echo "Found artifact: $ZIP_FILE"
-
-          mkdir -p ./bin
-          unzip -o "$ZIP_FILE" -d ./bin
-
-          if [[ ! -f ./bin/rustfs ]]; then
-            echo "❌ rustfs binary not found in archive"
-            exit 1
-          fi
-
-          chmod +x ./bin/rustfs
-          ls -lh ./bin/rustfs
-          echo "✅ Binary extracted"
-
-      - name: Build DEB package
-        id: deb
-        shell: bash
-        run: |
-          set -euo pipefail
-
-          VERSION="${{ needs.resolve.outputs.version }}"
-          DEB_ARCH="${{ matrix.deb_arch }}"
-          # DEB version: replace - with ~ (1.0.0-beta.12 -> 1.0.0~beta.12)
-          # Use a variable for ~ to prevent tilde expansion by bash
-          TILDE='~'
-          DEB_VERSION="${VERSION/-/$TILDE}"
-          PKG_DIR="rustfs_${DEB_VERSION}_${DEB_ARCH}"
-
-          echo "Building DEB: ${PKG_DIR}.deb"
-
-          mkdir -p "${PKG_DIR}/DEBIAN"
-          mkdir -p "${PKG_DIR}/usr/bin"
-          mkdir -p "${PKG_DIR}/etc/default"
-          mkdir -p "${PKG_DIR}/lib/systemd/system"
-          mkdir -p "${PKG_DIR}/usr/share/doc/rustfs"
-
-          cp ./bin/rustfs "${PKG_DIR}/usr/bin/"
-          chmod 755 "${PKG_DIR}/usr/bin/rustfs"
-
-          cp deploy/build/rustfs.service "${PKG_DIR}/lib/systemd/system/"
-
-          cat > "${PKG_DIR}/etc/default/rustfs" << 'ENVEOF'
-          # RustFS Environment Configuration
-          # See https://rustfs.com/docs/ for more information
-          # RUSTFS_VOLUMES=""
-          # RUSTFS_ROOT_USER=""
-          # RUSTFS_ROOT_PASSWORD=""
-          ENVEOF
-
-          cat > "${PKG_DIR}/DEBIAN/control" << EOF
-          Package: rustfs
-          Version: ${DEB_VERSION}
-          Section: utils
-          Priority: optional
-          Architecture: ${DEB_ARCH}
-          Depends: libc6 (>= 2.31)
-          Maintainer: RustFS Team <support@rustfs.com>
-          Description: High-performance distributed object storage
-           RustFS is a high-performance distributed object storage software
-           built using Rust. It is compatible with MinIO and S3 API.
-          Homepage: https://rustfs.com
-          EOF
-
-          # Declare /etc/default/rustfs as a conffile so dpkg preserves user
-          # modifications on upgrade instead of silently overwriting them.
-          cat > "${PKG_DIR}/DEBIAN/conffiles" << 'CONFFILES'
-          /etc/default/rustfs
-          CONFFILES
-
-          cat > "${PKG_DIR}/DEBIAN/postinst" << 'POSTINST'
-          #!/bin/bash
-          set -e
-          if ! getent passwd rustfs > /dev/null 2>&1; then
-              useradd -r -s /bin/false -d /opt/rustfs rustfs
-          fi
-          mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
-          chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
-          if [ -d /run/systemd/system ]; then
-              systemctl daemon-reload
-          fi
-          echo "RustFS installed. Configure /etc/default/rustfs then: systemctl start rustfs"
-          POSTINST
-          chmod 755 "${PKG_DIR}/DEBIAN/postinst"
-
-          cat > "${PKG_DIR}/DEBIAN/prerm" << 'PRERM'
-          #!/bin/bash
-          set -e
-          if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
-              systemctl stop rustfs
-          fi
-          PRERM
-          chmod 755 "${PKG_DIR}/DEBIAN/prerm"
-
-          cat > "${PKG_DIR}/DEBIAN/postrm" << 'POSTRM'
-          #!/bin/bash
-          set -e
-          if [ -d /run/systemd/system ]; then
-              systemctl daemon-reload
-          fi
-          POSTRM
-          chmod 755 "${PKG_DIR}/DEBIAN/postrm"
-
-          cp LICENSE "${PKG_DIR}/usr/share/doc/rustfs/"
-          cp README.md "${PKG_DIR}/usr/share/doc/rustfs/"
-
-          fakeroot dpkg-deb --build "${PKG_DIR}"
-
-          DEB_FILE="${PKG_DIR}.deb"
-          ls -lh "$DEB_FILE"
-          echo "deb_file=$DEB_FILE" >> "$GITHUB_OUTPUT"
-          echo "✅ DEB built: $DEB_FILE"
-
-      - name: Build RPM package
-        id: rpm
-        shell: bash
-        run: |
-          set -euo pipefail
-
-          VERSION="${{ needs.resolve.outputs.version }}"
-          RPM_ARCH="${{ matrix.rpm_arch }}"
-
-          echo "Building RPM for ${RPM_ARCH}"
-
-          sudo apt-get update && sudo apt-get install -y ruby ruby-dev build-essential
-          sudo gem install fpm
-
-          # Create config file for fpm (DEB build creates it in its package dir structure,
-          # but fpm needs the file to exist before packaging)
-          mkdir -p ./tmp-pkg/etc/default
-          cat > ./tmp-pkg/etc/default/rustfs << 'ENVEOF'
-          # RustFS Environment Configuration
-          # See https://rustfs.com/docs/ for more information
-          # RUSTFS_VOLUMES=""
-          # RUSTFS_ROOT_USER=""
-          # RUSTFS_ROOT_PASSWORD=""
-          ENVEOF
-
-          fpm -s dir -t rpm \
-            --name rustfs \
-            --version "$VERSION" \
-            --architecture "$RPM_ARCH" \
-            --depends "glibc >= 2.31" \
-            --maintainer "RustFS Team <support@rustfs.com>" \
-            --description "High-performance distributed object storage" \
-            --url "https://rustfs.com" \
-            --license "Apache-2.0" \
-            --after-install <(cat <<'POSTINST'
-          #!/bin/bash
-          set -e
-          if ! getent passwd rustfs > /dev/null 2>&1; then
-              useradd -r -s /bin/false -d /opt/rustfs rustfs
-          fi
-          mkdir -p /opt/rustfs /data/rustfs /var/log/rustfs
-          chown rustfs:rustfs /opt/rustfs /data/rustfs /var/log/rustfs
-          if [ -d /run/systemd/system ]; then
-              systemctl daemon-reload
-          fi
-          POSTINST
-            ) \
-            --before-remove <(cat <<'PRERM'
-          #!/bin/bash
-          set -e
-          if [ -d /run/systemd/system ] && systemctl is-active --quiet rustfs; then
-              systemctl stop rustfs
-          fi
-          PRERM
-            ) \
-            --after-remove <(cat <<'POSTRM'
-          #!/bin/bash
-          set -e
-          if [ -d /run/systemd/system ]; then
-              systemctl daemon-reload
-          fi
-          POSTRM
-            ) \
-            --config-files /etc/default/rustfs \
-            ./bin/rustfs=/usr/bin/rustfs \
-            ./tmp-pkg/etc/default/rustfs=/etc/default/rustfs \
-            deploy/build/rustfs.service=/lib/systemd/system/rustfs.service \
-            LICENSE=/usr/share/doc/rustfs/LICENSE \
-            README.md=/usr/share/doc/rustfs/README.md
-
-          RPM_FILE=$(ls -1 rustfs-*.rpm 2>/dev/null | head -1)
-          if [[ -z "$RPM_FILE" ]]; then
-            echo "❌ RPM build failed"
-            exit 1
-          fi
-
-          ls -lh "$RPM_FILE"
-          echo "rpm_file=$RPM_FILE" >> "$GITHUB_OUTPUT"
-          echo "✅ RPM built: $RPM_FILE"
-
-      - name: Upload packages to artifacts
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: packages-${{ matrix.arch }}
-          path: |
-            *.deb
-            *.rpm
-          retention-days: 30
-
-      - name: Upload packages to Cloudflare R2
-        if: env.R2_ACCESS_KEY_ID != ''
-        env:
-          R2_ACCESS_KEY_ID: ${{ secrets.R2_ACCESS_KEY_ID }}
-          R2_SECRET_ACCESS_KEY: ${{ secrets.R2_SECRET_ACCESS_KEY }}
-          R2_ENDPOINT: ${{ secrets.R2_ENDPOINT }}
-          R2_BUCKET: ${{ secrets.R2_BUCKET }}
-          AWS_EC2_METADATA_DISABLED: true
-        shell: bash
-        run: |
-          set -euo pipefail
-
-          if [[ -z "$R2_ACCESS_KEY_ID" || -z "$R2_SECRET_ACCESS_KEY" || -z "$R2_ENDPOINT" || -z "$R2_BUCKET" ]]; then
-            echo "⚠️ R2 credentials missing, skipping upload"
-            exit 0
-          fi
-
-          if ! command -v aws >/dev/null 2>&1; then
-            sudo apt-get update && sudo apt-get install -y awscli
-          fi
-
-          export AWS_ACCESS_KEY_ID="$R2_ACCESS_KEY_ID"
-          export AWS_SECRET_ACCESS_KEY="$R2_SECRET_ACCESS_KEY"
-          export AWS_DEFAULT_REGION="auto"
-
-          BUILD_TYPE="${{ needs.resolve.outputs.build_type }}"
-          if [[ "$BUILD_TYPE" == "development" ]]; then
-            R2_PREFIX="artifacts/rustfs/packages/dev"
-          else
-            R2_PREFIX="artifacts/rustfs/packages/release"
-          fi
-          R2_PATH="s3://${R2_BUCKET}/${R2_PREFIX}/"
-
-          echo "📤 Uploading to $R2_PATH"
-
-          DEB_FILE="${{ steps.deb.outputs.deb_file }}"
-          RPM_FILE="${{ steps.rpm.outputs.rpm_file }}"
-
-          for f in "$DEB_FILE" "$RPM_FILE"; do
-            if [[ -n "$f" && -f "$f" ]]; then
-              echo "Uploading: $f"
-              aws s3 cp "$f" "$R2_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
-            fi
-          done
-
-          echo "✅ Upload complete"
-
-          # Also upload as latest for release/prerelease
-          if [[ "$BUILD_TYPE" == "release" || "$BUILD_TYPE" == "prerelease" ]]; then
-            LATEST_PATH="s3://${R2_BUCKET}/artifacts/rustfs/packages/latest/"
-            for f in "$DEB_FILE" "$RPM_FILE"; do
-              if [[ -n "$f" && -f "$f" ]]; then
-                echo "Uploading latest: $(basename "$f")"
-                aws s3 cp "$f" "$LATEST_PATH" --endpoint-url "$R2_ENDPOINT" --only-show-errors
-              fi
-            done
-            echo "✅ Latest packages updated"
-          fi
-
-      - name: Upload packages to GitHub Release
-        if: needs.resolve.outputs.tag != ''
-        env:
-          GH_TOKEN: ${{ github.token }}
-        shell: bash
-        run: |
-          set -euo pipefail
-
-          TAG="${{ needs.resolve.outputs.tag }}"
-          DEB_FILE="${{ steps.deb.outputs.deb_file }}"
-          RPM_FILE="${{ steps.rpm.outputs.rpm_file }}"
-
-          # Upload the packages, then refresh the release checksums so the new
-          # assets are covered, matching the binary release flow.
-          for f in "$DEB_FILE" "$RPM_FILE"; do
-            if [[ -n "$f" && -f "$f" ]]; then
-              echo "📤 Uploading $(basename "$f") to GitHub release ${TAG}..."
-              gh release upload "$TAG" "$f" --clobber
-            fi
-          done
-
-          CHECKSUM_DIR="$(mktemp -d)"
-          gh release download "$TAG" -p 'SHA256SUMS' -p 'SHA512SUMS' \
-            -D "$CHECKSUM_DIR" --clobber
-
-          for spec in "SHA256SUMS:sha256sum" "SHA512SUMS:sha512sum"; do
-            asset="${spec%%:*}"
-            checksum_cmd="${spec##*:}"
-            checksum_file="${CHECKSUM_DIR}/${asset}"
-
-            for f in "$DEB_FILE" "$RPM_FILE"; do
-              if [[ -n "$f" && -f "$f" ]]; then
-                base="$(basename "$f")"
-                # GitHub stores release asset names with '~' normalized to '.'
-                # (e.g. rustfs_1.0.0~rc.2_amd64.deb is stored as
-                # rustfs_1.0.0.rc.2_amd64.deb), so checksum entries must
-                # reference the name as stored on the release.
-                github_base="${base//\~/.}"
-                # Remove any stale entry (both naming variants), then append
-                grep -Fv -- "$base" "$checksum_file" > "${checksum_file}.tmp" || true
-                grep -Fv -- "$github_base" "${checksum_file}.tmp" > "${checksum_file}.tmp2" || true
-                mv "${checksum_file}.tmp2" "$checksum_file"
-                digest=$("$checksum_cmd" -- "$f" | awk '{print $1}')
-                printf '%s  %s\n' "$digest" "$github_base" >> "$checksum_file"
-              fi
-            done
-
-            echo "📤 Updating ${asset} for release ${TAG}..."
-            gh release upload "$TAG" "$checksum_file" --clobber
-          done
-
-          echo "✅ GitHub release assets updated"
-
-  # Summary
-  summary:
-    name: Summary
-    needs: [ resolve, package ]
-    if: always()
-    runs-on: ubuntu-latest
-    timeout-minutes: 5
-    steps:
-      - name: Print summary
-        shell: bash
-        run: |
-          echo "## 📦 Package Summary" >> "$GITHUB_STEP_SUMMARY"
-          echo "" >> "$GITHUB_STEP_SUMMARY"
-          echo "| Item | Value |" >> "$GITHUB_STEP_SUMMARY"
-          echo "|------|-------|" >> "$GITHUB_STEP_SUMMARY"
-          echo "| Version | \`${{ needs.resolve.outputs.version }}\` |" >> "$GITHUB_STEP_SUMMARY"
-          echo "| Build Type | ${{ needs.resolve.outputs.build_type }} |" >> "$GITHUB_STEP_SUMMARY"
-          echo "| Build Run | #${{ needs.resolve.outputs.build_run_id }} |" >> "$GITHUB_STEP_SUMMARY"
-          echo "| Package Status | ${{ needs.package.result }} |" >> "$GITHUB_STEP_SUMMARY"

.github/workflows/performance-ab.yml

@@ -17,18 +17,11 @@
 # Two entry points, honestly scoped:
 #   * schedule (nightly, on main): post-merge detection — catches a regression
 #     within 24h of landing, not before merge.
-#   * workflow_dispatch: an explicitly selected trusted ref.
-# The dispatch input can run the gate with --allow-regression so a deliberate
-# correctness cost (e.g. the #4221 fsync durability fix) is recorded, not
-# blocked (rustfs/backlog#935 correction 1).
+#   * pull_request labeled `perf-ab`: opt-in pre-merge gate for a specific PR.
+# The `perf-deliberate-tradeoff` label runs the gate with --allow-regression so
+# a deliberate correctness cost (e.g. the #4221 fsync durability fix) is
+# recorded but does not block (rustfs/backlog#935 correction 1).
 
-# DISABLED. This workflow is switched off in the repository's Actions settings
-# (state: disabled_manually) and does not run on any trigger, including its cron
-# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
-# reading this file, which has already misled at least one audit — hence this
-# banner. Re-enabling is a UI action; anyone doing so should first check that the
-# workflow still matches the current CI layout. See rustfs/backlog#1603.
-#
 name: Performance A/B
 
 on:
@@ -46,95 +39,46 @@ on:
         required: false
         default: false
         type: boolean
-  push:
-    # Every main commit pre-builds and caches its release binary (perf-3) so the
-    # nightly A/B restores a ready baseline instead of paying the double build.
-    branches: [main]
+  pull_request:
+    types: [labeled, synchronize, reopened]
 
 permissions:
   contents: read
+  pull-requests: write
+
+# Per-PR: a new push cancels the previous (up to 90-minute) A/B run instead of
+# stacking them. Nightly schedule and manual dispatch get a unique group and
+# always run to completion.
+concurrency:
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
+  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
 
 env:
   CARGO_TERM_COLOR: always
   RUST_BACKTRACE: 1
 
 jobs:
-  # perf-3: on every push to main, build the release binary once and cache it
-  # keyed by commit SHA (rustfs-baseline-<sha>). The warp-ab measurements
-  # restore this instead of paying the ~32min-per-side source
-  # build. That double build is what pushed the expanded 24-cell nightly past its
-  # ceiling — 2026-07-11..07-14 all cancelled on the 120min timeout. Incremental
-  # builds off the shared cargo cache keep each push cheap, and building on the
-  # same sm-standard-2 runner the A/B measures on guarantees the cached binary is
-  # ABI-identical. Do NOT source this from build.yml's per-merge artifact: those
-  # are cancelled ~7/8 of the time and are not a reliable baseline.
-  build-baseline-cache:
-    name: Build + cache baseline binary
-    if: github.event_name == 'push'
-    runs-on: sm-standard-2
-    # Latest-wins: consumers only ever restore the binary for the *current*
-    # origin/main tip, so when pushes land faster than the ~65min build, a
-    # superseded build's output is dead weight — cancel it instead of stacking
-    # hour-long jobs on the shared runner pool. A skipped intermediate SHA at
-    # most costs one same-commit self-heal in the A/B job.
-    concurrency:
-      group: perf-baseline-build-main
-      cancel-in-progress: true
-    # #4806 put thin LTO + codegen-units=1 on [profile.release], pushing a
-    # single release build past 60min on this runner — every cache build on
-    # 2026-07-15 died on the old 60min ceiling ("exceeded the maximum execution
-    # time of 1h0m0s") and the cache never populated. The measured binary must
-    # keep the production profile, so the budget absorbs the build instead.
-    timeout-minutes: 100
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: warp-ab-${{ hashFiles('**/Cargo.lock') }}
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
-
-      - name: Build release rustfs
-        run: cargo build --release --bin rustfs
-
-      - name: Stage binary for cache
-        run: |
-          set -euo pipefail
-          mkdir -p baseline-bin
-          cp target/release/rustfs baseline-bin/rustfs
-
-      - name: Cache baseline binary by SHA
-        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
-        with:
-          path: baseline-bin/rustfs
-          key: rustfs-baseline-${{ github.sha }}
-
   warp-ab:
     name: Warp A/B budget gate
-    # Always run on schedule / manual dispatch. Never on push — that event only
-    # feeds build-baseline-cache above.
+    # Opt-in on PRs: only run when the `perf-ab` label is present, and for
+    # `labeled` events only when the label being added is `perf-ab` itself —
+    # adding an unrelated label to an opted-in PR must not re-run the gate.
+    # Always run on schedule / manual dispatch.
     if: >-
-      github.event_name == 'schedule' ||
-      github.event_name == 'workflow_dispatch'
+      github.event_name != 'pull_request' ||
+      (contains(github.event.pull_request.labels.*.name, 'perf-ab') &&
+       (github.event.action != 'labeled' || github.event.label.name == 'perf-ab'))
     runs-on: sm-standard-2
-    # With perf-3's cached baseline binary the common (cache-hit) nightly is
-    # measurement-only and finishes well under 50min. This ceiling stays
-    # generous only to absorb the same-commit cache-miss self-heal (~65min
-    # single build with the post-#4806 LTO profile + measurement). A timeout
-    # surfaces via the alert-on-failure job (it fires on cancelled/timed-out,
-    # not just failure). perf-6 recalibrates the budget once the noise study
-    # lands.
+    # Phase-0 stopgap: the baseline+candidate release double-build alone is
+    # ~65min on this runner, so 90min left no room for a real full-matrix
+    # measurement (the earlier nightly runs only ever failed *before*
+    # measuring). 120min gives the 24-cell short matrix headroom until perf-3
+    # caches the baseline binary and restores a tighter budget.
     timeout-minutes: 120
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
         with:
-          persist-credentials: false
           fetch-depth: 0 # baseline is built from origin/main
 
       - name: Setup Rust environment
@@ -143,6 +87,7 @@ jobs:
           rust-version: stable
           cache-shared-key: warp-ab-${{ hashFiles('**/Cargo.lock') }}
           cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
 
       - name: Install warp
         run: |
@@ -154,152 +99,39 @@ jobs:
 
       - name: Decide exemption
         id: exempt
-        env:
-          INPUT_ALLOW_REGRESSION: ${{ github.event.inputs.allow_regression }}
         run: |
           allow="false"
-          if [[ "$INPUT_ALLOW_REGRESSION" == "true" ]]; then
+          if [[ "${{ github.event_name }}" == "pull_request" ]] \
+            && ${{ contains(github.event.pull_request.labels.*.name, 'perf-deliberate-tradeoff') }}; then
+            allow="true"
+          fi
+          if [[ "${{ github.event.inputs.allow_regression }}" == "true" ]]; then
             allow="true"
           fi
           echo "allow_regression=$allow" >> "$GITHUB_OUTPUT"
 
-      # perf-3: resolve the commits so the cache can be keyed by SHA. The
-      # baseline is origin/main; the candidate is the checked-out ref. On the
-      # nightly (checkout == main) they are the same commit, so one cached binary
-      # serves both phases and the run does zero source builds.
-      - name: Resolve baseline / candidate commits
-        id: commits
-        run: |
-          set -euo pipefail
-          baseline_sha="$(git rev-parse origin/main)"
-          candidate_sha="$(git rev-parse HEAD)"
-          echo "baseline_sha=$baseline_sha" >> "$GITHUB_OUTPUT"
-          echo "candidate_sha=$candidate_sha" >> "$GITHUB_OUTPUT"
-          echo "baseline commit: $baseline_sha"
-          echo "candidate commit: $candidate_sha"
-
-      # Exact-key restore of the baseline binary built by build-baseline-cache
-      # when origin/main last landed. A miss (binary evicted or not built yet)
-      # leaves cache-hit unset and the rig falls back to a source build.
-      - name: Restore cached baseline binary
-        id: baseline_cache
-        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
-        with:
-          path: baseline-bin/rustfs
-          key: rustfs-baseline-${{ steps.commits.outputs.baseline_sha }}
-
-      # Self-heal: on a nightly/dispatch run where the candidate commit IS the
-      # baseline commit, a cache miss would make the rig build the same commit
-      # twice (~65min per side with the post-#4806 LTO profile — no job budget
-      # fits that). Build it once here, reuse it for both phases, and save it
-      # back to the cache so the next run hits.
-      - name: Build baseline on cache miss (same-commit self-heal)
-        id: selfheal
-        if: >-
-          steps.baseline_cache.outputs.cache-hit != 'true' &&
-          steps.commits.outputs.baseline_sha == steps.commits.outputs.candidate_sha
-        run: |
-          set -euo pipefail
-          cargo build --release --bin rustfs
-          mkdir -p baseline-bin
-          cp target/release/rustfs baseline-bin/rustfs
-          echo "built=true" >> "$GITHUB_OUTPUT"
-
-      - name: Build baseline on cache miss (different candidate)
-        id: baseline_build
-        if: >-
-          steps.baseline_cache.outputs.cache-hit != 'true' &&
-          steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha
-        run: |
-          set -euo pipefail
-          baseline_root="$RUNNER_TEMP/rustfs-baseline-${{ github.run_id }}"
-          baseline_target="$RUNNER_TEMP/rustfs-baseline-target-${{ github.run_id }}"
-          git worktree add --detach "$baseline_root" "${{ steps.commits.outputs.baseline_sha }}"
-          cargo build --release --manifest-path "$baseline_root/Cargo.toml" --bin rustfs --target-dir "$baseline_target"
-          mkdir -p baseline-bin
-          cp "$baseline_target/release/rustfs" baseline-bin/rustfs
-          git worktree remove --force "$baseline_root"
-          echo "built=true" >> "$GITHUB_OUTPUT"
-
-      - name: Build candidate binary
-        id: candidate_build
-        if: steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha
-        run: |
-          set -euo pipefail
-          cargo build --release --bin rustfs
-          mkdir -p candidate-bin
-          cp target/release/rustfs candidate-bin/rustfs
-          echo "built=true" >> "$GITHUB_OUTPUT"
-
-      - name: Save self-healed baseline to cache
-        if: steps.selfheal.outputs.built == 'true' || steps.baseline_build.outputs.built == 'true'
-        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
-        with:
-          path: baseline-bin/rustfs
-          key: rustfs-baseline-${{ steps.commits.outputs.baseline_sha }}
-
       - name: Run warp A/B and gate
         id: ab
-        env:
-          INPUT_DURATION: ${{ github.event.inputs.duration }}
         run: |
           set -euo pipefail
-          # The formal runner executes A1 baseline -> B1 candidate -> B2 candidate
-          # -> A2 baseline for each workload and drive-sync cell. It requires three
-          # rounds per leg to emit tail latency and error-rate evidence.
-          # --health-timeout 180 outlasts the server's own 120s startup-readiness
-          # budget, which the rig's previous 60s health poll undershot (the first
-          # two nightly failures). perf-6 recalibrates these once the noise study
-          # lands.
-          duration="${INPUT_DURATION:-12s}"
-          baseline_sha="${{ steps.commits.outputs.baseline_sha }}"
-          candidate_sha="${{ steps.commits.outputs.candidate_sha }}"
-          baseline_hit="${{ steps.baseline_cache.outputs.cache-hit }}"
-          selfheal_built="${{ steps.selfheal.outputs.built }}"
-          baseline_built="${{ steps.baseline_build.outputs.built }}"
-          candidate_built="${{ steps.candidate_build.outputs.built }}"
-
-          args=(--duration "$duration" --rounds 3 --cooldown 5 --health-timeout 180 --baseline-revision "$baseline_sha" --candidate-revision "$candidate_sha")
-
-          if [[ "$baseline_hit" == "true" || "$selfheal_built" == "true" || "$baseline_built" == "true" ]]; then
-            chmod +x baseline-bin/rustfs
-            base_bin="$PWD/baseline-bin/rustfs"
-            args+=(--baseline-bin "$base_bin")
-            if [[ "$baseline_hit" == "true" ]]; then
-              base_src="actions-cache (rustfs-baseline-$baseline_sha)"
-            elif [[ "$selfheal_built" == "true" ]]; then
-              base_src="source build (cache self-heal, saved as rustfs-baseline-$baseline_sha)"
-            else
-              base_src="isolated origin/main source build (saved as rustfs-baseline-$baseline_sha)"
-            fi
-            if [[ "$candidate_sha" == "$baseline_sha" ]]; then
-              # Nightly on main: the candidate is the same commit as the baseline,
-              # so reuse the one binary for both phases and skip all builds.
-              args+=(--candidate-bin "$base_bin")
-              cand_src="same binary as baseline (same commit)"
-            elif [[ "$candidate_built" == "true" ]]; then
-              chmod +x candidate-bin/rustfs
-              args+=(--candidate-bin "$PWD/candidate-bin/rustfs")
-              cand_src="source build of the checked-out ref"
-            else
-              echo "::error::candidate binary was not built" >&2
-              exit 2
-            fi
-          else
-            echo "::error::baseline binary was not restored or built" >&2
-            exit 2
-          fi
-
-          echo "baseline binary: $base_src"
-          echo "candidate binary: $cand_src"
-
+          # Budget note: the baseline+candidate release double-build (~65 min,
+          # cached away later by perf-3) dominates the 90-min job, so the
+          # measurement runs a short warp matrix — duration/rounds/cooldown are
+          # kept small to fit all 24 cells (6 workloads x 2 phases x 2 drive-sync)
+          # under budget rather than dropping cells. --health-timeout 180 outlasts
+          # the server's own 120s startup-readiness budget, which is what the
+          # rig's previous 60s health poll undershot (the first two nightly
+          # failures). perf-6 will recalibrate these once the pipeline is green.
+          duration="${{ github.event.inputs.duration || '12s' }}"
+          args=(--baseline-ref origin/main
+            --duration "$duration" --rounds 2 --cooldown 5 --health-timeout 180)
           if [[ "${{ steps.exempt.outputs.allow_regression }}" == "true" ]]; then
-            args+=(--allow-regression --exemption-reason "workflow dispatch override")
+            args+=(--allow-regression --exemption-reason "labeled perf-deliberate-tradeoff / dispatch override")
           fi
           # Do not let a gate FAIL abort the job here; capture status and surface
-          # it after the step summary is written.
+          # it after the PR comment is posted.
           set +e
-          bash scripts/run_hotpath_warp_abba.sh "${args[@]}"
+          bash scripts/run_hotpath_warp_ab.sh "${args[@]}"
           echo "status=$?" >> "$GITHUB_OUTPUT"
           set -e
           # Locate the newest run dir + gate.md for the summary/comment/artifact
@@ -307,10 +139,10 @@ jobs:
           # holds server-logs/ for diagnosis.
           # Run dirs are UTC-timestamp names (no special chars); ls is safe here.
           # shellcheck disable=SC2012
-          run_dir="$(ls -td target/hotpath-abba/*/ 2>/dev/null | head -n1 || true)"
+          run_dir="$(ls -td target/hotpath-ab/*/ 2>/dev/null | head -n1 || true)"
           echo "run_dir=${run_dir%/}" >> "$GITHUB_OUTPUT"
           # shellcheck disable=SC2012
-          gate_md="$(ls -t target/hotpath-abba/*/candidate_gate.md 2>/dev/null | head -n1 || true)"
+          gate_md="$(ls -t target/hotpath-ab/*/gate.md 2>/dev/null | head -n1 || true)"
           echo "gate_md=$gate_md" >> "$GITHUB_OUTPUT"
 
       - name: Upload A/B results
@@ -318,10 +150,10 @@ jobs:
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
         with:
           name: hotpath-warp-ab-${{ github.run_number }}
-          # Includes per-cell median_summary.csv / baseline_compare.csv, both gates,
+          # Includes per-cell median_summary.csv / baseline_compare.csv, gate.md,
           # and server-logs/ (rustfs.log + startup env per phase) so a failed run
           # is diagnosable. Short retention: this is churny nightly debug data.
-          path: target/hotpath-abba/
+          path: target/hotpath-ab/
           if-no-files-found: warn
           retention-days: 14
 
@@ -362,15 +194,26 @@ jobs:
             fi
           } >> "$GITHUB_STEP_SUMMARY"
 
-      # Scheduled failure alerting is handled by the alert-on-failure job below
-      # (perf-2 consuming ci-8's schedule-failure-issue composite action).
+      - name: Comment gate result on PR
+        if: always() && github.event_name == 'pull_request' && steps.ab.outputs.gate_md != ''
+        env:
+          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
+        run: |
+          gh pr comment "${{ github.event.pull_request.number }}" --body-file "${{ steps.ab.outputs.gate_md }}"
+
+      # TODO(ci-8/perf-2): scheduled/dispatch failures are currently silent. Once
+      # ci-8 lands the .github/actions/schedule-failure-issue composite action,
+      # perf-2 adds a step here guarded by
+      #   if: failure() && github.event_name != 'pull_request'
+      # that calls it (label perf-nightly-failure, append to an existing open
+      # issue) instead of hand-rolling gh CLI dedup. Do not implement it here.
 
       - name: Enforce gate
         if: always()
         run: |
           status="${{ steps.ab.outputs.status }}"
           if [[ "$status" != "0" ]]; then
-            echo "::error::warp A/B budget gate failed (exit $status). See the step summary / gate.md artifact." >&2
+            echo "::error::warp A/B budget gate failed (exit $status). See the step summary / PR comment / gate.md artifact." >&2
             exit "$status"
           fi
           echo "warp A/B budget gate passed."
@@ -380,15 +223,8 @@ jobs:
     needs: [warp-ab]
     # `always()` is required: without it this job is skipped when a needed
     # job fails. Alerts only for scheduled (nightly) runs (backlog#1149
-    # ci-8); manual dispatch failures are already watched by a human.
-    # `cancelled` is included alongside `failure` on purpose: a job that hits
-    # timeout-minutes ends as `cancelled`, and the 2026-07-11..07-14 nightly
-    # timeouts went silent precisely because the guard was failure-only. The
-    # composite action already reports cancelled/timed-out jobs in the issue
-    # body.
-    if: >-
-      always() && github.event_name == 'schedule' &&
-      (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
+    # ci-8); PR and manual dispatch failures are already watched by a human.
+    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
     runs-on: ubuntu-latest
     timeout-minutes: 10
     permissions:
@@ -396,8 +232,6 @@ jobs:
       issues: write
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
       - name: Open or update failure-tracking issue
         uses: ./.github/actions/schedule-failure-issue
         with:

.github/workflows/runner-hygiene.yml

@@ -1,81 +0,0 @@
-# Copyright 2026 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Asserts that the self-hosted runners are still ephemeral — one job per pod.
-#
-# This repository is public and its pull_request jobs run on those runners,
-# executing the PR's own build.rs, proc-macros and tests. The only thing keeping
-# that code from reaching a later job is that each ARC pod handles exactly one
-# job and is then destroyed. That guarantee lives in the ARC scale-set
-# configuration, outside this repository, where it can be changed without any PR
-# — so it is asserted here from the outside, against real run data, instead of
-# being assumed.
-#
-# Monthly rather than per-PR: the property changes only when someone
-# reconfigures the scale set, and the check costs a few dozen API calls.
-# See docs/ci/runners.md and rustfs/backlog#1602.
-
-name: Runner Hygiene
-
-on:
-  schedule:
-    - cron: "0 6 1 * *" # Monthly, 1st at 06:00 UTC (after the daily audit cron)
-  workflow_dispatch:
-
-permissions:
-  contents: read
-
-concurrency:
-  group: runner-hygiene
-  cancel-in-progress: false
-
-jobs:
-  check-ephemerality:
-    name: Check runner ephemerality
-    runs-on: ubuntu-latest
-    timeout-minutes: 15
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      # Exit 2 (inconclusive / broken) is deliberately not a pass: a window
-      # where every sm-* job was still queued would otherwise look identical to
-      # a clean bill of health.
-      - name: Assert one job per self-hosted runner
-        env:
-          GH_TOKEN: ${{ secrets.GITHUB_TOKEN }}
-        run: ./scripts/ci/check_runner_ephemerality.sh 40
-
-  alert-on-failure:
-    name: Alert on scheduled failure
-    needs: [check-ephemerality]
-    # Same ci-8 mechanism as coverage.yml, audit.yml and the nightly lanes:
-    # scheduled runs file a tracking issue, manual dispatch stays quiet so
-    # debugging never produces a spurious alert.
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    permissions:
-      contents: read
-      issues: write
-    steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-      - name: Open or update failure-tracking issue
-        uses: ./.github/actions/schedule-failure-issue
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/schedule-failure-alert-drill.yml

@@ -24,13 +24,6 @@
 # The run itself is expected to end red (the forced failure); only the
 # alert-on-failure job result matters.
 
-# DISABLED. This workflow is switched off in the repository's Actions settings
-# (state: disabled_manually) and does not run on any trigger, including its cron
-# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
-# reading this file, which has already misled at least one audit — hence this
-# banner. Re-enabling is a UI action; anyone doing so should first check that the
-# workflow still matches the current CI layout. See rustfs/backlog#1603.
-#
 name: Schedule Failure Alert Drill
 
 on:
@@ -63,8 +56,6 @@ jobs:
       issues: write
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
       - name: Open or update failure-tracking issue
         uses: ./.github/actions/schedule-failure-issue
         with:

.github/workflows/stale.yml

@@ -12,13 +12,6 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-# DISABLED. This workflow is switched off in the repository's Actions settings
-# (state: disabled_manually) and does not run on any trigger, including its cron
-# and workflow_dispatch. That state lives in GitHub's UI and is invisible when
-# reading this file, which has already misled at least one audit — hence this
-# banner. Re-enabling is a UI action; anyone doing so should first check that the
-# workflow still matches the current CI layout. See rustfs/backlog#1603.
-#
 name: "Mark stale issues"
 on:
   schedule:
@@ -27,7 +20,6 @@ on:
 jobs:
   stale:
     runs-on: ubuntu-latest
-    timeout-minutes: 30
     steps:
       - uses: actions/stale@5bef64f19d7facfb25b37b414482c7164d639639 # v9
         with:

.github/workflows/star-history.yml

@@ -1,27 +0,0 @@
-name: Star History
-
-on:
-  schedule:
-    - cron: "17 3 * * *"
-  workflow_dispatch:
-
-permissions:
-  contents: write
-
-concurrency:
-  group: star-history
-  cancel-in-progress: false
-
-jobs:
-  update:
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    steps:
-      - uses: overtrue/repo-visuals-action@72f34d24769ff5d341956da2f23952594ef2f1e2 # v1.3.0
-        with:
-          github-token: ${{ github.token }}
-          output-branch: star-history
-          output-path: .
-          chart-style: gradient
-          animate: "true"
-          contributors: "true"

.github/workflows/windows-filesystem.yml

@@ -1,90 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#      http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-name: Windows Filesystem Tests
-
-on:
-  push:
-    branches: [ main ]
-    paths:
-      - "crates/ecstore/src/disk/**"
-      - "crates/ecstore/src/store/init_format.rs"
-      - "crates/ecstore/Cargo.toml"
-      - "Cargo.toml"
-      - "Cargo.lock"
-      - ".github/actions/setup/**"
-      - ".github/workflows/windows-filesystem.yml"
-  pull_request:
-    branches: [ main ]
-    paths:
-      - "crates/ecstore/src/disk/**"
-      - "crates/ecstore/src/store/init_format.rs"
-      - "crates/ecstore/Cargo.toml"
-      - "Cargo.toml"
-      - "Cargo.lock"
-      - ".github/actions/setup/**"
-      - ".github/workflows/windows-filesystem.yml"
-  workflow_dispatch:
-
-permissions:
-  contents: read
-
-concurrency:
-  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
-  cancel-in-progress: true
-
-env:
-  CARGO_TERM_COLOR: always
-  FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-  RUST_BACKTRACE: 1
-
-jobs:
-  rename-safety:
-    name: Rename Safety
-    runs-on: windows-latest
-    timeout-minutes: 60
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: build-x86_64-pc-windows-msvc
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
-          install-test-tools: 'false'
-
-      - name: Check production Windows dependencies
-        shell: pwsh
-        run: cargo check -p rustfs-ecstore --lib
-
-      - name: Test guarded rename publication
-        shell: pwsh
-        run: cargo test -p rustfs-ecstore --lib rename_all_ -- --nocapture
-
-      - name: Test Windows handle guards
-        shell: pwsh
-        run: cargo test -p rustfs-ecstore --lib windows_ -- --nocapture
-
-      - name: Test startup temporary-directory cleanup
-        shell: pwsh
-        run: cargo test -p rustfs-ecstore --lib cleanup_tmp_on_startup_ -- --nocapture
-
-      - name: Test fresh format publication
-        shell: pwsh
-        run: cargo test -p rustfs-ecstore --lib fresh_format_load_initializes_all_disks -- --nocapture

.gitignore

@@ -83,7 +83,3 @@ worktrees/*
 
 # Local AI-agent review artifacts (omo evidence dumps)
 .omo/
-
-# insta scratch files; the accepted .snap files ARE the assertions and are committed
-*.snap.new
-*.pending-snap

.mimocode/skills/issue-triage/SKILL.md

@@ -1,116 +0,0 @@
----
-name: issue-triage
-description: Triage a GitHub issue — determine if it is already fixed, needs implementation, or should be closed. Searches related commits and PRs, verifies implementation status, and posts a triage comment or closes the issue. Use when the user provides an issue URL and asks whether it can be closed or needs work.
----
-
-# Issue Triage
-
-Use this skill when the user provides a GitHub issue URL and asks "can this be closed?", "is this already implemented?", "check completion status", or similar triage questions.
-
-## Workflow
-
-### 1. Fetch issue context
-
-```bash
-gh issue view <N> --repo <owner/repo> --json title,body,state,comments,labels,updatedAt
-```
-
-Read the issue body to understand what was requested. Extract:
-- The specific feature/fix/behavior described.
-- Any linked PRs or commits mentioned in the body or comments.
-- Any checklist items or sub-issues.
-
-### 2. Search for related work
-
-Search git history for commits referencing the issue:
-```bash
-git log --oneline --all --grep="<N>" | head -30
-```
-
-Search for related PRs:
-```bash
-gh pr list --search "fixes #<N> OR closes #<N> OR #<N>" --state all --json number,title,state,mergedAt
-```
-
-If the issue mentions specific PRs, check their status:
-```bash
-gh pr view <PR_N> --json state,mergedAt,title
-```
-
-### 3. Verify implementation
-
-For each linked or related PR that is merged, verify the fix is actually present on the current main branch:
-```bash
-git log --oneline main | grep -i "<keyword>"
-# or
-git log --oneline main --grep="<PR_N>"
-```
-
-If the issue describes a specific defect, check the relevant code to confirm the fix is in place:
-```bash
-grep -n "<pattern>" crates/<relevant>/src/<file>.rs
-```
-
-For issues with checklists, verify each item individually. If sub-items are tracked as separate issues, check those too:
-```bash
-gh issue view <SUB_N> --repo <owner/repo> --json state
-```
-
-### 4. Determine verdict
-
-- **All items fixed and merged**: Close with a summary comment listing what was fixed and which PRs.
-- **Some items fixed, some remaining**: Comment with status of each item. Do not close.
-- **Not yet implemented**: Comment with a summary of what remains. Do not close.
-- **Superseded or no longer relevant**: Close with explanation.
-
-### 5. Take action
-
-Close with comment:
-```bash
-gh issue close <N> --repo <owner/repo> --comment "<body>"
-```
-
-Comment without closing:
-```bash
-gh issue comment <N> --repo <owner/repo> --body-file /tmp/triage.md
-```
-
-Update issue labels if needed:
-```bash
-gh issue edit <N> --repo <owner/repo> --add-label "completed" --remove-label "needs-triage"
-```
-
-Always use `--body-file` for multiline content, never inline `--body`.
-
-### 6. Handle multi-issue batches
-
-When the user asks to check multiple issues (e.g., "check all issues by user X" or "scan backlog for closable issues"):
-1. List the issues: `gh issue list --repo <repo> --author <user> --state open --json number,title,updatedAt`
-2. For each issue, run steps 1-5 above.
-3. Report a summary table of all triaged issues with verdicts.
-
-## Output format
-
-### Issue Triage: #<N> — <title>
-
-**State**: OPEN / CLOSED
-**Linked PRs**: <list with merge status>
-
-#### Assessment
-<what was requested vs what is implemented>
-
-#### Verdict
-- Close — all items resolved by <PR list>
-- Keep open — <remaining items>
-- Not started — <what needs to be done>
-
-#### Action taken
-- Closed with comment / Commented / No action
-
-## Notes
-
-- The user may ask in Chinese ("是否可以关闭", "检查完成情况"); respond in the same language.
-- When closing, always include a summary of what was fixed and which PRs resolved it — this creates a useful audit trail.
-- For issues in `rustfs/backlog`, use `--repo rustfs/backlog`.
-- For issues in `rustfs/rustfs`, use `--repo rustfs/rustfs`.
-- If the issue has sub-issues (GitHub sub-issues API), check each one's state before declaring the parent complete.

.mimocode/skills/pr-review/SKILL.md

@@ -1,147 +0,0 @@
----
-name: pr-review
-description: Review a GitHub PR end-to-end from a URL or number — fetch metadata, inspect the diff, run multi-role adversarial review, check CI status, and post the review comment. Use when the user provides a PR link and asks to review it.
----
-
-# PR Review
-
-Use this skill when the user provides a GitHub PR URL or number and asks to review it. This covers the full review lifecycle: data gathering, code review, CI verification, and posting the result.
-
-## Prerequisites
-
-- Read `AGENTS.md` for the repository's adversarial validation policy and change-style rules.
-- The `adversarial-validation` skill handles the review role playbooks; this skill orchestrates the workflow around it.
-
-## Workflow
-
-### 1. Gather PR context
-
-```bash
-gh pr view <N> --json title,author,state,body,additions,deletions,changedFiles,commits,baseRefName,headRefName
-gh pr diff <N> --name-only
-```
-
-Read the PR body and linked issues to understand the change's purpose. If the PR references an issue, fetch that too:
-```bash
-gh issue view <ISSUE> --json title,body,state
-```
-
-### 2. Fetch the diff and classify the change
-
-```bash
-git fetch origin pull/<N>/head:pr-<N>
-git diff main...pr-<N> --stat
-```
-
-Classify the change by risk tier (per AGENTS.md):
-- **Exempt**: docs/comments/instruction-only, formatting, typos.
-- **Mechanical**: renames, file moves, test-only or tooling changes.
-- **Standard** (default): any behavior change.
-- **High risk**: locking, erasure coding, quorum/heal, replication, multipart, RPC, lifecycle/tiering, metadata formats, persistence/fsync, IAM/KMS/auth, on-disk/on-wire formats, S3 API-visible behavior.
-
-### 3. Cluster changed files and delegate review
-
-Group the changed files into logical clusters (by crate or functional area). For each cluster, spawn a subagent with a focused review prompt that includes:
-- The cluster's changed files and their diffs.
-- The applicable adversarial role probes (from the `adversarial-validation` skill).
-- The repository's AGENTS.md rules relevant to that domain.
-
-For standard-tier changes: correctness adversary + simplicity adversary + test-coverage skeptic, plus every role whose domain the diff touches.
-For high-risk changes: run all seven roles.
-
-Each subagent must produce findings (concrete failure scenario with file:line) or a null report ("attacked X, Y, Z — no break found").
-
-### 4. Check CI status
-
-```bash
-gh pr checks <N>
-```
-
-If any checks fail, investigate:
-```bash
-gh run view --log-failed --job=<JOB_ID>
-```
-
-Determine whether failures are pre-existing (on main), flaky, or caused by the PR.
-
-### 5. Synthesize findings
-
-Combine all subagent findings into a structured review:
-- **Summary**: one-paragraph overview of the change and overall assessment.
-- **Findings**: each finding with severity (critical/major/minor/nit), file:line, concrete failure scenario, and suggested fix.
-- **CI status**: pass/fail with notes on any failures.
-- **Verdict**: APPROVE, REQUEST_CHANGES, or COMMENT.
-
-### 6. Post the review
-
-Write the review body to a temp file and post via CLI:
-```bash
-# Request changes
-gh pr review <N> --request-changes --body-file /tmp/pr_review.md
-
-# Approve
-gh pr review <N> --approve --body-file /tmp/pr_review.md
-
-# Comment only (no verdict)
-gh pr review <N> --comment --body-file /tmp/pr_review.md
-```
-
-For inline comments on specific lines, use the GitHub API:
-```bash
-cat > /tmp/pr_review.json <<'EOF'
-{
-  "body": "review body",
-  "event": "REQUEST_CHANGES",
-  "comments": [
-    {
-      "path": "crates/foo/src/bar.rs",
-      "line": 42,
-      "body": "finding description"
-    }
-  ]
-}
-EOF
-gh api --method POST /repos/{owner}/{repo}/pulls/<N>/reviews --input /tmp/pr_review.json
-```
-
-Always use `--body-file` or `--input`, never inline multiline `--body`.
-
-### 7. Handle follow-up
-
-If the review requests changes:
-- Monitor for new commits: `gh pr view <N> --json commits`
-- Re-review changed files only: `git diff pr-<N>..origin/pull/<N>/head`
-- Update the review when findings are addressed.
-
-If CI was failing due to pre-existing main breakage:
-- Comment on the PR noting the failure is pre-existing.
-- Suggest updating the branch: `gh pr update-branch <N>`
-
-## Output format
-
-### PR Review: #<N> — <title>
-
-**Author**: <author>
-**Risk tier**: exempt | mechanical | standard | high-risk
-**Changed files**: <count> across <cluster count> clusters
-
-#### Summary
-<one-paragraph overview>
-
-#### Findings
-| Severity | Location | Finding |
-|----------|----------|---------|
-| critical | file:line | concrete failure scenario |
-
-#### CI Status
-- All checks pass / Failing: <details>
-
-#### Verdict
-APPROVE / REQUEST_CHANGES / COMMENT
-
-## Notes
-
-- The user may ask for review in Chinese; respond in the same language but keep the review body in English per AGENTS.md rules.
-- When the user asks for "多角色对抗 review", run the full adversarial validation protocol — this skill's step 3 covers that.
-- If the PR is from a fork, check `maintainerCanModify` before attempting to push fixes.
-- For very large PRs (>50 files), cluster aggressively and delegate in parallel to keep review time reasonable.

.vscode/launch.json

@@ -1,7 +1,45 @@
 {
     "version": "0.2.0",
     "configurations": [
-          {
+        {
+            "name": "Debug RustFS observability (OTLP)",
+            "type": "lldb",
+            "request": "launch",
+            "cargo": {
+                "args": [
+                    "build",
+                    "--bin=rustfs",
+                    "--package=rustfs"
+                ],
+                "filter": {
+                    "name": "rustfs",
+                    "kind": "bin"
+                }
+            },
+            "args": [],
+            "cwd": "${workspaceFolder}",
+            "env": {
+                "RUST_LOG": "rustfs=debug,ecstore=info,s3s=info,iam=info",
+                "RUST_BACKTRACE": "full",
+                "RUSTFS_ACCESS_KEY": "rustfsadmin",
+                "RUSTFS_SECRET_KEY": "rustfsadmin",
+                "RUSTFS_VOLUMES": "./target/observability/data{1...4}",
+                "RUSTFS_ADDRESS": ":9000",
+                "RUSTFS_CONSOLE_ENABLE": "true",
+                "RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
+                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
+                "RUSTFS_OBS_ENDPOINT": "http://127.0.0.1:4318",
+                "RUSTFS_OBS_TRACES_EXPORT_ENABLED": "true",
+                "RUSTFS_OBS_METRICS_EXPORT_ENABLED": "true",
+                "RUSTFS_OBS_LOGS_EXPORT_ENABLED": "true",
+                "RUSTFS_OBS_USE_STDOUT": "true",
+                "RUSTFS_OBS_LOG_DIRECTORY": "./target/observability/logs",
+                "RUSTFS_OBS_METER_INTERVAL": "5",
+                "RUSTFS_OBS_SERVICE_NAME": "rustfs-observability-local",
+                "RUSTFS_OBS_ENVIRONMENT": "development"
+            }
+        },
+        {
             "type": "lldb",
             "request": "launch",
             "name": "Debug(only) executable 'rustfs'",
@@ -172,7 +210,7 @@
             ],
         },
         {
-            "name": "Debug executable target/debug/rustfs with sse kms",
+            "name": "Debug executable target/debug/rustfs with sse",
             "type":  "lldb",
             "request": "launch",
             "program": "${workspaceFolder}/target/debug/rustfs",
@@ -200,7 +238,7 @@
                 // 2. kms local backend test key
                 // "RUSTFS_KMS_ENABLE": "true",
                 // "RUSTFS_KMS_BACKEND": "local",
-                // "RUSTFS_KMS_KEY_DIR": "/tmp/kms-key-dir",
+                // "RUSTFS_KMS_KEY_DIR": "./target/kms-key-dir",
                 // "RUSTFS_KMS_LOCAL_MASTER_KEY": "my-secret-key", // Some Password
                 // "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
 
@@ -212,40 +250,13 @@
                 // "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
 
                 // 4. kms vault transit backend test key
-                // "RUSTFS_KMS_ENABLE": "true",
-                // "RUSTFS_KMS_BACKEND": "vault-transit",
-                // "RUSTFS_KMS_VAULT_ADDRESS": "http://127.0.0.1:8200",
-                // "RUSTFS_KMS_VAULT_TOKEN": "Dev Token",
-                // "RUSTFS_KMS_VAULT_MOUNT_PATH": "transit",
-                // "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
-                
-                // 5、kms static backend test key
                 "RUSTFS_KMS_ENABLE": "true",
-                "RUSTFS_KMS_BACKEND": "static",
-                "RUSTFS_KMS_STATIC_SECRET_KEY": "rustfs-master-key:2dfNXGHlsEflGVCxb+5DIdGEl1sIvtwX+QfmYasi5QM="
-            },
-            "sourceLanguages": [
-                "rust"
-            ],
-        },
-        {
-            "name": "Debug executable target/debug/rustfs with local sse",
-            "type": "lldb",
-            "request": "launch",
-            "program": "${workspaceFolder}/target/debug/rustfs",
-            "args": [],
-            "cwd": "${workspaceFolder}",
-            "env": {
-                "RUSTFS_ACCESS_KEY": "rustfsadmin",
-                "RUSTFS_SECRET_KEY": "rustfsadmin",
-                "RUSTFS_VOLUMES": "./target/volumes/test{1...4}",
-                "RUSTFS_ADDRESS": ":9000",
-                "RUSTFS_CONSOLE_ENABLE": "true",
-                "RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
-                "RUSTFS_OBS_LOG_DIRECTORY": "./target/logs",
-                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
-                "RUSTFS_SSE_S3_MASTER_KEY": "xGb3aYSp825j2tPpg8JrUzghiXsIkfdOtmrsJ/iafiM=",
-                "RUST_LOG": "rustfs=debug,ecstore=debug,s3s=debug,iam=debug",
+                "RUSTFS_KMS_BACKEND": "vault-transit",
+                "RUSTFS_KMS_VAULT_ADDRESS": "http://127.0.0.1:8200",
+                "RUSTFS_KMS_VAULT_TOKEN": "Dev Token",
+                "RUSTFS_KMS_VAULT_MOUNT_PATH": "transit",
+                "RUSTFS_KMS_DEFAULT_KEY_ID": "rustfs-master-key",
+
             },
             "sourceLanguages": [
                 "rust"

AGENTS.md

@@ -1,255 +1,288 @@
-# RustFS Agent Instructions
+# RustFS Agent Instructions (Global)
 
-This file contains repository-wide rules. Use the nearest subdirectory
-`AGENTS.md` for path-specific invariants.
+This root file keeps repository-wide rules only.
+Use the nearest subdirectory `AGENTS.md` for path-specific guidance.
 
-## Precedence
+## Rule Precedence
 
 1. System/developer instructions.
-2. The current user request.
-3. The nearest `AGENTS.md`.
-4. This file.
+2. Current user/task instructions.
+3. The nearest `AGENTS.md` in the current path.
+4. This file (global defaults).
 
-## Operating Model
+If repo-level instructions conflict, follow the nearest file and keep behavior aligned with CI.
 
-- Inquiry, diagnosis, review, and planning tasks are read-only unless the user
-  explicitly requests changes.
-- For implementation, read the relevant code, tests, and local guidance, then
-  make the smallest change that satisfies the request.
-- State assumptions only when they affect behavior or verification. Ask only
-  when a wrong assumption would materially change the result.
-- Do not load every skill or inspect unrelated modules preemptively. Select a
-  skill only when its description directly matches the request or changed
-  surface.
-- Avoid repeated reads and equivalent verification commands once enough
-  evidence exists.
+## Execution Discipline
 
-## Worktree and Disk Hygiene
+- Read the relevant existing code, tests, and local guidance before changing behavior.
+- State assumptions when they affect the implementation or verification path.
+- If a task has multiple plausible interpretations, list the options briefly and choose the narrowest reasonable path; ask when the ambiguity would make the change risky.
+- For multi-step work, keep the plan minimal and tied to verifiable outcomes.
+- Avoid redundant file reads, repeated commands, and unnecessary exploratory work once enough context is available.
+- A good result is a minimal diff with clear assumptions, no over-engineering, and independent verification that survives Adversarial Validation (below).
 
-- Start implementation from the latest `origin/main` and confirm the requested
-  change is not already present.
-- An existing clean, isolated task worktree is sufficient. Create another
-  worktree only when the current checkout is shared, dirty with unrelated work,
-  or belongs to another task.
-- Never commit from a shared checkout. Use an `overtrue/` feature branch unless
-  the user requests another name.
-- Check free space before artifact-heavy builds, tests, coverage, or downloads.
-  Re-check before a broad gate when space is tight.
-- Remove only task-owned temporary/build artifacts. Never delete another task's
-  worktree or uncommitted data.
-- At handoff, mention disk or cleanup details only when they affected execution
-  or artifacts/worktrees remain intentionally.
+## Communication and Language
 
-## Change Style
+- Respond in the same language used by the requester.
+- Keep source code, comments, commit messages, and PR title/body in English.
+- Be concise. Avoid sycophantic openers, closing fluff, and verbose status reporting.
 
-- Preserve existing control flow unless changing it is required for correctness.
-- Prefer a direct local edit over new files, wrappers, managers, or speculative
-  abstractions.
-- Add a helper only when it removes current duplication, names a real domain
-  boundary, or isolates a non-trivial invariant.
-- Remove an in-scope path superseded by the change. If compatibility requires it,
-  adapt at the boundary to one canonical core and use the repository's
-  `RUSTFS_COMPAT_TODO` policy.
-- Comments explain non-obvious invariants or reasons. Do not narrate code or
-  record change history.
-- Mention unrelated problems when useful; do not fix them in a narrow task.
+## Change Style for Existing Logic
 
-## Reuse and Boundary Rules
+- Prefer direct, local code over extracting one-off helpers.
+- Extract a helper only when logic is reused or the extraction materially clarifies a non-trivial flow.
+- Use Rust's default module file layout (`mod foo;` with `foo.rs` or `foo/mod.rs`/`foo/*.rs`).
+  Avoid `#[path = "..."]` for module inclusion; move files into the canonical module tree instead.
+  If an unavoidable generated-code, FFI, or test-fixture exception remains, keep it local and document why the canonical layout cannot work.
+- Solve only the requested problem; do not add speculative features, configurability, or adjacent improvements.
+- Prefer editing existing code over rewriting files or reshaping unrelated logic.
+- Modify only what is required and remove only artifacts introduced by your own changes.
+- Preserve the existing control-flow and logic shape when fixing bugs or addressing review comments, especially in init, distributed coordination, locking, metadata, and concurrency paths.
+- Do not refactor existing code only to make it easier to unit test.
+- Keep fixes narrowly aligned with the requested behavior; avoid semantic-adjacent rewrites while touching sensitive paths.
+- Keep code elegant, concise, and direct. Prefer minimal, readable implementations over over-engineering and excessive abstraction. Use comments to clarify non-obvious intent and invariants, not to compensate for unclear code.
+- Mention unrelated issues when useful, but do not fix them as part of a narrow task.
 
-- Before adding helpers, constants, fixtures, or wrappers, search the touched
-  crate, the domain-owning crate, `crates/utils`, `crates/common`, and relevant
-  direct dependencies.
-- Reuse requires matching semantics: normalization, error types, deadlines,
-  durability, and compatibility must fit the call site. A narrowly named local
-  helper is better than forced reuse with different semantics.
-- Validate untrusted input at its trust boundary, then trust the validated type.
-  Values crossing disk, RPC, persistence, or version boundaries remain
-  untrusted at every consumer.
-- Re-check boundary values immediately before destructive actions such as
-  delete, overwrite, or quorum decisions.
-- Every new branch needs a concrete triggering input/state. For decoded or peer
-  data, corruption and mixed-version input are valid triggers.
-- Required values must return a typed error when absent or corrupt; do not use a
-  default that converts corruption into a plausible result.
-- Attach error context once where it is actionable. Do not erase typed errors
-  below aggregation or quorum layers.
+## Constant and String Usage
+
+- Before introducing new string literals, search for existing constants/enums that already represent the same semantic value.
+- Reuse existing constants for protocol labels, error identifiers, header keys, event names, metric names, command tags, and similar fixed tokens.
+- If a new string is truly unique, define a local constant near related logic and avoid scattering the literal across multiple sites.
+- When changing existing behavior, keep naming and format consistency by aligning with established project constants.
 
 ## Sources of Truth
 
-- Workspace membership: `Cargo.toml`.
-- Local gates: `Makefile` and `.config/make/`.
-- CI gates: `.github/workflows/ci.yml`.
-- PR format: `.github/pull_request_template.md`.
-- Architecture routing: `ARCHITECTURE.md` and `docs/architecture/README.md`.
-- Agent skills: `.agents/skills/*/SKILL.md`.
+- Workspace layout and crate membership: `Cargo.toml` (`[workspace].members`)
+- Local quality commands: `Makefile` and `.config/make/`
+- CI quality gates: `.github/workflows/ci.yml`
+- PR template: `.github/pull_request_template.md`
+- High-level architecture and crate map: `ARCHITECTURE.md`
+- Migration guardrails, readiness contracts, support matrices:
+  `docs/architecture/README.md` (routes by audience)
+- Shared agent skills (all tools): `.agents/skills/` — each `SKILL.md` carries
+  a frontmatter `description` stating when it applies. Scan the descriptions
+  before starting a task and follow any skill that matches, even if your tool
+  does not auto-load skills:
+  `grep -m1 '^description:' .agents/skills/*/SKILL.md`
+  Claude Code reads them through the `.claude/skills` symlink; add new skills
+  to `.agents/skills/` only, never as separate copies per tool
 
-Do not commit one-shot plans, trackers, migration ledgers, benchmark snapshots,
-or agent scratch notes. Durable architecture belongs under `docs/architecture/`,
-operations under `docs/operations/`, and testing references under
-`docs/testing/`. `scripts/check_no_planning_docs.sh` enforces this boundary.
+Avoid duplicating long crate lists or command matrices in instruction files.
+Reference the source files above instead.
 
-## Verification
+Do not commit planning-type documents — one-shot implementation/optimization
+plans, task trackers, migration-progress ledgers, phase/PR templates,
+issue-scoped benchmark-result snapshots or optimization conclusions, or
+agent-generated working notes (e.g. anything a `superpowers`/scratch workflow
+produces). Keep that work in the issue tracker or your local worktree, not in
+the repository. Only durable reference — the architecture set under
+`docs/architecture/`, repeatable operational runbooks under `docs/operations/`,
+and the test-suite references under `docs/testing/` — belongs in version
+control; `.gitignore` ignores everything else under `docs/` by default, so a new
+plan file will not be tracked unless someone force-adds it — don't.
+`scripts/check_no_planning_docs.sh` (wired into `make pre-commit`/`pre-pr` and
+CI) fails the build if anything is committed under `docs/superpowers/`, even via
+`git add -f`.
 
-Select checks from the final task-owned diff. Scoped `AGENTS.md` files may add a
-concrete path-specific check, but must not replace this tiering with a generic
-full-workspace gate.
+## Verification Before PR
 
-### Documentation and Instructions
+Convert changes into independently verifiable outcomes. Prefer focused tests for behavior changes and run the relevant checks before declaring completion.
+Non-exempt changes must also pass Adversarial Validation (next section) before the checks below count as completion.
 
-For prose, comments, agent instructions, and skill metadata that cannot affect
-runtime/build output:
+For code changes, run and pass the following before opening a PR:
 
-- Run `git diff --check`.
-- Run the relevant documentation guard or skill validator when applicable.
-- Skip Cargo formatting, compilation, Clippy, tests, `make pre-commit`, and
-  `make pre-pr`.
+```bash
+make pre-pr
+```
 
-### Non-Behavioral Source Changes
+Before committing code changes, prefer focused verification for the touched
+surface and use the faster local gate when a broad smoke check is needed:
 
-- Run the formatter/validator for the changed language.
-- Add compilation or doctests only when syntax or executable examples changed.
+```bash
+make pre-commit
+```
 
-### Localized Behavior Changes
+For migration batches, do not run the full `make pre-pr` gate before every
+intermediate commit. Use focused tests and `make pre-commit` during
+development, then reserve `make pre-pr` for the final PR-ready branch.
 
-- Run `cargo fmt --all --check` for Rust changes.
-- Run the narrowest test that exercises the changed behavior.
-- Add package-scoped `cargo check` or Clippy only for targets, features, public
-  APIs, error handling, or control flow not compiled by the focused test.
-- Use `make pre-commit` only when its repository-wide fast checks add confidence
-  beyond the focused checks.
+Before pushing code changes, make sure formatting is clean:
 
-### Broad or High-Risk Changes
+- Run `cargo fmt --all`.
+- Run `cargo fmt --all --check` and ensure no files are modified unexpectedly.
 
-After the required adversarial review, run `make pre-pr` when targeted coverage
-cannot bound the impact, including dependency/toolchain/build-matrix changes,
-unbounded cross-crate APIs, or locking, durability, erasure coding, replication,
-RPC, IAM/KMS/auth, cryptography, on-disk/on-wire, and S3-visible behavior.
+If `make` is unavailable, run the equivalent checks defined under `.config/make/`.
+Documentation-only or instruction-only changes are exempt from the verification commands above (including the `.config/make/` equivalents), though any locally installed git pre-commit hooks may still run on commit unless explicitly skipped.
+After build-based verification completes, clean generated build artifacts before wrapping up to avoid unnecessary disk usage.
+Do not open a PR with code changes when the required checks fail.
+Make a failing check pass by fixing the cause, never by weakening the gate:
+do not loosen or skip a guard script, add entries to a baseline or allowance
+list, suppress a lint with `#[allow]`, mark a failing test `#[ignore]`, or
+delete or relax a failing assertion to get green. If a check itself is wrong,
+change it deliberately and state the rationale in the PR.
 
-`make pre-pr` includes `make pre-commit`; never run both for the same unchanged
-diff. Do not repeat a check already covered by a successful umbrella gate.
-Rerun only checks affected by later edits.
+For flaky tests, do not paper over them with retries. Follow the flake policy
+in [docs/testing/README.md](docs/testing/README.md) (open an issue within 24h,
+quarantine with an issue link, fix or delete within 30 days); the local
+`default` nextest profile never retries.
 
-Never weaken a gate to get green: do not add baselines/allowances, suppress
-lints, ignore tests, or relax assertions unless changing that policy is itself
-the reviewed task. Follow `docs/testing/README.md` for flaky tests.
+## Adversarial Validation (Default On)
 
-## Adversarial Validation
+Every non-exempt output (see Risk tiers) — code change, bug fix, or
+design/solution proposal — passes multi-role adversarial review before it
+counts as done.
+Author confidence is not evidence: each role's job is to refute the change,
+not to bless it.
 
-Adversarial validation applies to final implementation diffs, explicitly
-requested adversarial/design reviews, and agent-instruction changes that alter
-execution. Ordinary questions, diagnoses, status reports, non-adversarial code
-reviews, and low-risk planning do not trigger it.
+### Risk tiers
 
-Risk and review shape:
+Pick the tier from the riskiest file touched; when in doubt, pick the higher.
 
-- **Exempt:** documentation, comments, formatting, or typos with no runtime,
-  build, test, or agent-execution effect.
-- **Mechanical:** renames, moves, test/tooling-only changes, and agent-rule
-  changes. Run correctness and simplicity lenses.
-- **Standard:** localized behavior changes. Run one integrated final-diff pass
-  covering correctness, simplicity, and test coverage; add only domain lenses
-  matched by the diff.
-- **High risk / substantial PR review:** high risk includes locking,
-  erasure/quorum/heal, replication, multipart, RPC, lifecycle/tiering,
-  persistence/fsync, IAM/KMS/auth, cryptography, on-disk/on-wire formats, and
-  S3-visible semantics. Cover all applicable lenses using exactly two
-  independent reviewers when delegation is explicitly authorized. Split the
-  lenses between them. Otherwise perform two fresh sequential passes.
+- **Exempt:** docs/comments/instruction-only changes, formatting, typos with
+  no runtime surface. Skip this section.
+- **Mechanical:** pure renames, file moves, test-only or tooling changes —
+  correctness adversary only.
+- **Standard (the default):** any change that affects behavior.
+- **High risk:** touches locking, erasure coding, quorum/heal, replication,
+  multipart, RPC, lifecycle/tiering, metadata formats (`xl.meta`),
+  persistence/fsync, IAM/KMS/auth, on-disk or on-wire formats, or
+  S3 API-visible behavior.
 
-Available domain lenses are security, concurrency/durability, compatibility,
-and performance. Select `.agents/skills/adversarial-validation/SKILL.md` for an
-explicit adversarial request, a high-risk change, or a substantial PR review;
-then read only its matching role references. A routine standard pass does not
-load the playbook unless the reviewer needs a RustFS-specific probe.
+### Roles
 
-A finding must name a concrete input/state/interleaving and wrong outcome, or a
-specific missing regression check, with `file:line`. Resolve it by fixing the
-diff or rebutting it with code-path/test/invariant evidence. After a non-trivial
-fix, rerun only affected lenses.
+Run each applicable role as an independent pass over the final diff (or
+proposal text) — parallel reviewer agents where the tooling supports them,
+otherwise sequential passes that each start fresh from the diff and the
+nearest scoped `AGENTS.md`, discarding the writing session's assumptions.
+Each role either produces findings or reports "attacked X, Y, Z — no break
+found"; a bare pass is not a result. Repo-specific attack probes for every
+role live in `.agents/skills/adversarial-validation/` — run them, they
+encode this repo's shipped bugs.
 
-For high-risk PRs, record one concise verdict per covered lens in the PR body.
+- **Correctness adversary** — construct a concrete input/state/interleaving
+  that yields wrong output, data loss, or a crash. Probe error paths and edge
+  values (empty, nil UUID, zero-length, quorum−1, missing version). For code
+  diffs, a materially smaller or more idiomatic diff achieving the same
+  behavior is also a finding (see Change Style for Existing Logic).
+- **Security reviewer** — authn/authz bypass, injection, secret leakage,
+  untrusted deserialization (see Serde Safety), path traversal, timing leaks.
+- **Concurrency/durability reviewer** — lock ordering, races, cancellation,
+  partial failure, retry/idempotency, crash and power-loss ordering.
+- **Compatibility reviewer** — S3 API surface, MinIO interop, on-disk and
+  on-wire formats, mixed-version upgrade/downgrade paths.
+- **Performance reviewer** — allocation and cloning on hot paths, lock hold
+  time across IO, sync or CPU-heavy work on async runtime threads, added
+  fsync/flush outside the durability gate, hot-path logging noise. A
+  measurable regression on a per-request or per-object path is a finding.
+- **Test-coverage skeptic** — for each claimed behavior, name the test that
+  fails if the change is reverted; then name a changed line that could be
+  wrong while all tests stay green — if one exists, coverage is insufficient.
+  A missing test is a finding, not a note.
 
-## Pull Request Lifecycle
+Standard tier: correctness adversary + test-coverage skeptic, plus every
+role whose domain the diff touches (async or shared-state code →
+concurrency; parsing of untrusted input → security; public crate API shape
+→ compatibility; per-request or per-object hot paths → performance).
+High risk: all six roles.
 
-- Creating or updating a PR includes one immediate snapshot of checks,
-  mergeability, reviews, and unresolved threads.
-- Unless the user explicitly requests monitoring, a release workflow requires
-  it, or an automation already owns it, hand off after the PR is open with the
-  current state and next event to watch. Do not delay ordinary handoff with
-  fixed quiet-period sleeps.
-- For requested monitoring, use event-driven or bounded waits. Report only state
-  changes, actionable failures, or a meaningful prolonged delay.
-- Investigate failures/comments before changing code. Fix task-attributable
-  issues, rerun affected verification, push, reply or resolve the thread, then
-  resume the requested monitor.
-- Never merge without required reviewer approval or explicit authority.
-- After an observed merge, verify the commit reached the base, then clean the
-  task worktree/branch when safe. Preserve unmerged work for closed PRs unless
-  deletion was explicitly authorized.
+### Protocol
+
+1. A finding states a concrete failure scenario (input/state → wrong
+   outcome) or names a missing test, with severity and file:line. "Looks
+   risky" is not a finding.
+2. Resolve every finding: fix it, or rebut it with evidence — a test, a
+   traced code path, or a cited invariant. Restated intent and "unlikely"
+   are not rebuttals.
+3. After non-trivial fixes, re-run the roles whose domain the fix touched.
+4. For proposals with no diff, roles attack assumptions, failure modes,
+   migration/rollback, and testability instead — including the simplest
+   rejected alternative and the blast radius when the design fails.
+
+### Exit criteria
+
+- Every applicable role has run; every finding is fixed or rebutted with
+  evidence.
+- Every behavior change has a test that fails without it.
+- The Verification Before PR gates pass — adversarial review supplements
+  those gates, never replaces them.
+- High risk only: record a one-line verdict per role in the PR description.
 
 ## Git and PR Baseline
 
-- Follow Conventional Commits; keep the subject at most 72 characters.
-- Source comments, commits, PR titles, and PR bodies are in English.
-- Keep every heading from `.github/pull_request_template.md`; use `N/A` where
-  needed and include commands actually run.
-- Use `--body-file` for multiline `gh pr create`/`gh pr edit` content.
-- PR/issue/discussion content must not contain the literal sequence `\n` or
-  hard-wrapped prose paragraphs.
-- Do not include local absolute paths or tool-specific labels/prefixes in GitHub
-  content.
-- Resolve review threads after the underlying issue is fixed. If declining a
-  suggestion, reply with a short evidence-based reason.
+- Use feature branches based on the latest `main`.
+- Assume other agent sessions work this repository concurrently. Never commit
+  in a shared checkout; do all work on a dedicated feature branch, preferably
+  in a dedicated worktree.
+- Immediately before branching, fetch `origin/main` and branch from it;
+  confirm the target issue is not already fixed there before writing code.
+- Follow Conventional Commits, with subject length <= 72 characters.
+- Keep PR title and description in English.
+- Use `.github/pull_request_template.md` and keep all section headings.
+- Use `N/A` for non-applicable template sections.
+- Include verification commands in the PR description.
+- When using `gh pr create`/`gh pr edit`, write the markdown body to a file
+  and pass `--body-file`; multiline inline `--body` is unsafe — backticks and
+  shell expansion can corrupt content or trigger unintended commands.
+  Pattern: `cat > /tmp/pr_body.md <<'EOF' ... EOF`, then
+  `--body-file /tmp/pr_body.md` (keep the file outside the checkout).
+- Do not include the literal sequence `\n` in any GitHub issue, pull request, or discussion comment.
+- Do not hard-wrap prose in PR/issue/discussion bodies; write each paragraph as a
+  single line and let it reflow. GitHub renders single newlines inside a paragraph
+  as line breaks, so mid-sentence wrapping shows up as ugly breaks. Only break lines
+  for list items, code blocks, and deliberate separators.
+- After fixing code review comments or CI findings, always mark corresponding review
+  comments/threads as resolved before returning to the user.
+- In handling review comments, confirm the underlying issue before changing code.
+  If a suggested change is not appropriate for behavior or risk, reply with a
+  concise rationale instead of blindly applying it.
 
 ## Security Baseline
 
 - Never commit secrets, credentials, or key material.
 - Use environment variables or vault tooling for sensitive configuration.
-- For localhost-sensitive tests, bypass proxies explicitly.
-- Untrusted S3 XML/JSON, lifecycle, policy, replication, and RPC structures use
-  strict deserialization where compatibility permits. Security-critical
-  defaults require explicit validation.
+- For localhost-sensitive tests, verify proxy settings to avoid traffic leakage.
 
-## Logging
+## Tools
 
-For every added or edited `tracing` call:
+### xl.meta decode tool Quick Use
 
-- Reuse the module's `EVENT_*`, `LOG_COMPONENT_*`, and `LOG_SUBSYSTEM_*`
-  constants and field shape.
-- Put fields first and a short label last.
-- Use `error` for behavior/security failure, `warn` for degradation/fallback,
-  `info` for low-frequency lifecycle, `debug` for diagnostics, and `trace` for
-  repetitive request/object success paths.
-- Never log secrets, credential payloads, or merged configs.
+```
+cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/file/xl.meta"
+```
 
-Use `.agents/skills/rustfs-logging-governance/SKILL.md` for logging changes.
+## Serde Safety
 
-## Cross-Cutting Storage Invariants
+- Add `#[serde(deny_unknown_fields)]` to structs deserialized from untrusted input (S3 API XML/JSON, lifecycle rules, bucket policies, replication configs).
+- When `deny_unknown_fields` is impractical (backward compatibility), at minimum log unknown fields at `warn` level.
+- Never use `#[serde(default)]` on security-critical fields without explicit validation of the resulting value.
 
-- Write internal object metadata under both `x-rustfs-internal-<suffix>` and
-  `x-minio-internal-<suffix>` using
-  `crates/utils/src/http/metadata_compat.rs` helpers.
-- Read binary UUID metadata with
-  `.and_then(|v| Uuid::from_slice(&v).ok()).filter(|u| !u.is_nil())`; absent,
-  empty, and nil all mean no value.
-- Remote-tier version `None` or `""` means an unversioned bucket; send no
-  `versionId` on tier GET/DELETE.
-- `DataUsageCacheInfo` and `DataUsageEntry` keep their hand-written map
-  serialization and new fields remain `#[serde(default)]` for older readers.
+## Cross-Cutting Domain Invariants
 
-## Naming
+- Write internal object metadata under **both** `x-rustfs-internal-<suffix>`
+  and `x-minio-internal-<suffix>` keys (MinIO interop). Use the helpers in
+  `crates/utils/src/http/metadata_compat.rs` (`get_bytes` prefers the RustFS
+  key); never write only one of the two.
+- Read binary UUID metadata defensively:
+  `.and_then(|v| Uuid::from_slice(&v).ok()).filter(|u| !u.is_nil())` —
+  absent, empty, and nil all mean "no value", never `Uuid::nil()`.
+- A remote-tier version of `None`/`""` means the tier bucket is unversioned:
+  send **no** `versionId` on tier GET/DELETE.
 
-Use Rust API naming: `SCREAMING_SNAKE_CASE` constants/statics, `snake_case`
-functions/variables, and `PascalCase` types. Do not rename unrelated existing
-violations.
+## Naming Conventions
 
-## Scoped Guidance
+- Follow Rust API Guidelines for naming: `SCREAMING_SNAKE_CASE` for statics and constants, `snake_case` for functions and variables, `PascalCase` for types.
+- Do not use camelCase or Hungarian notation (e.g., `globalDeploymentIDPtr` → `GLOBAL_DEPLOYMENT_ID`).
+- If existing code violates naming conventions, do not widen the violation in new code. Do not rename existing symbols as part of an unrelated task; mention the violation instead (see Change Style for Existing Logic).
 
-Before editing, locate the nearest instructions with:
+## Scoped Guidance in This Repository
+
+Many crates and modules carry their own `AGENTS.md` with path-specific rules
+(security boundaries, lock ordering, domain invariants). Before editing a
+path, check for the nearest one:
 
 ```bash
 git ls-files '*AGENTS.md'
 ```
 
-The nearest file wins for domain invariants. Keep generic workflow and
-validation policy in this root file.
+The nearest file wins. Do not maintain a hand-written index of these files
+here — it goes stale.

ARCHITECTURE.md

@@ -1,6 +1,6 @@
 # ARCHITECTURE.md
 
-> Last updated: 2026-08-12 · Revision: 3
+> Last updated: 2026-07-02 · Revision: 2
 >
 > This document describes the high-level architecture of RustFS.
 > If you want to familiarize yourself with the code base, you are in the right place!
@@ -31,7 +31,7 @@ HTTP request
       → storage/ecfs (erasure coding, encryption, checksums)
         → ecstore (disk pool selection, data distribution)
           → rio (reader pipeline: encrypt → compress → hash → write)
-            → io-core (buffer pool, storage profiling, admission control)
+            → io-core (zero-copy I/O, buffer pool, direct I/O)
               → local disk / remote disk via RPC
 ```
 
@@ -41,7 +41,7 @@ The repository is a Cargo workspace with a flat `crates/` layout:
 
 ```
 rustfs/                      # Workspace root (virtual manifest)
-├── rustfs/                  # Main binary + library crate
+├── rustfs/                  # Main binary + library crate (75K lines)
 │   └── src/
 │       ├── main.rs          # Entry point, startup sequence
 │       ├── lib.rs           # Module tree root
@@ -53,9 +53,9 @@ rustfs/                      # Workspace root (virtual manifest)
 │       ├── config/          # CLI args, config parsing, workload profiles
 │       └── ...
 ├── crates/                  # library crates (authoritative list: Cargo.toml [workspace].members)
-│   ├── ecstore/             # Erasure-coded storage engine
+│   ├── ecstore/             # Erasure-coded storage engine (⚠️ 87K lines)
 │   ├── rio/                 # Reader I/O pipeline (encrypt, compress, hash)
-│   ├── io-core/             # Buffer pool, storage profiling, backpressure/deadlock policy, lock optimizer, operation progress
+│   ├── io-core/             # Zero-copy I/O, scheduling, buffer pool
 │   ├── io-metrics/          # I/O metrics collection
 │   ├── common/              # Shared runtime state, globals, data usage types
 │   ├── config/              # Configuration types and parsing
@@ -83,28 +83,124 @@ A request flows **downward** through the layers. No layer should reach upward
 
 ### Crate Reference
 
-`Cargo.toml` is the authoritative workspace membership and `cargo tree` is the
-authoritative dependency graph. This overview deliberately avoids line-count
-and dependency-depth snapshots because both quickly become stale during
-refactors.
+> Depth levels, line counts, and crate counts in this section are a
+> point-in-time snapshot and drift with refactors. Treat them as orders of
+> magnitude; `Cargo.toml` and `cargo tree` are the source of truth.
+
+Crates are organized in a dependency DAG with 9 depth levels (0 = leaf, 8 = top):
+
+```
+Depth 0 — LEAF (no internal deps):
+  appauth, checksums, config, credentials, crypto, io-metrics,
+  madmin, s3-common, workers, zip
+
+Depth 1:
+  io-core (→ io-metrics)
+  policy (→ config, credentials, crypto)
+  utils                                   (historical → config edge removed; now effectively leaf)
+
+Depth 2:
+  concurrency, filemeta, keystone, kms, lock, obs,
+  signer, targets, trusted-proxies
+
+Depth 3:
+  common                                  (historical → filemeta/madmin edges removed; now effectively leaf)
+
+Depth 4:
+  object-capacity, protos, rio
+
+Depth 5 — CORE:
+  ecstore (16 internal deps, 11 dependents — the architectural heart)
+
+Depth 6:
+  audit, heal, iam, metrics, notify, s3select-api, scanner
+
+Depth 7:
+  object-io, protocols, s3select-query
+
+Depth 8 — TOP:
+  rustfs (35 internal deps — the binary, depends on almost everything)
+```
 
 #### By Domain
 
-| Domain | Current workspace crates | Responsibility |
-|--------|--------------------------|----------------|
-| Foundation | `checksums`, `common`, `config`, `data-usage`, `utils` | Shared configuration, data-usage models, utilities, and checksums. |
-| I/O and storage | `concurrency`, `ecstore`, `filemeta`, `heal`, `io-core`, `io-metrics`, `lifecycle`, `lock`, `object-capacity`, `object-data-cache`, `replication`, `rio`, `rio-v2`, `scanner`, `storage-api` | Erasure-coded object storage, metadata, recovery, lifecycle, replication, locking, cache, and I/O pipelines. |
-| Security and identity | `credentials`, `crypto`, `iam`, `keystone`, `kms`, `policy`, `security-governance`, `signer`, `tls-runtime`, `trusted-proxies` | Credentials, authentication, authorization, encryption, key management, TLS, and security contracts. |
-| Protocols and contracts | `extension-schema`, `madmin`, `protos`, `protocols`, `s3-ops`, `s3-types`, `s3select-api`, `s3select-query` | Admin, inter-node, S3, S3 Select, and optional protocol contracts. |
-| Operations and integration | `audit`, `notify`, `obs`, `targets`, `zip` | Auditing, observability, event delivery, notification targets, and archive support. |
-| Test support | `e2e_test`, `test-utils` | End-to-end validation and shared test bootstrap utilities. |
+**Core Infrastructure:**
 
-The `rustfs` binary crate composes these libraries into the running server.
-`ecstore` remains the storage engine at the architectural center; its internal
-module split is tracked under `docs/architecture/`. `rio-v2` is the
-feature-gated MinIO on-disk format compatibility I/O layer; it ships in no
-default build (lifecycle:
-[docs/architecture/minio-file-format-compat.md](docs/architecture/minio-file-format-compat.md)).
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `config` | 3.3K | Configuration types and environment parsing |
+| `utils` | 8.7K | Pure utilities (paths, compression, network, retry) |
+| `common` | 4.4K | Shared runtime state, globals, data usage types, metrics |
+| `madmin` | 5.5K | Admin API request/response types |
+
+**I/O Pipeline:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `io-core` | 6.5K | Zero-copy I/O, buffer pool, direct I/O, scheduling, backpressure |
+| `io-metrics` | 4.5K | I/O operation metrics and counters |
+| `rio` | 6.9K | Composable reader chain (encrypt → compress → hash → limit) |
+| `object-io` | 2.4K | High-level object read/write using rio + ecstore |
+| `concurrency` | 0.8K | Shared concurrency contract types: workload admission snapshots, worker-slot pool, policy types (runtime control lives in `rustfs/src/storage`) |
+
+**Storage Engine:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `ecstore` | 87K | ⚠️ Erasure-coded storage: disks, pools, buckets, replication, lifecycle |
+| `filemeta` | 10K | File/object metadata types and versioning |
+| `checksums` | 732 | Checksum computation |
+| `lock` | 7.1K | Distributed lock manager |
+| `heal` | 5.9K | Data healing / bitrot repair |
+| `scanner` | 5.4K | Background data usage scanner |
+| `object-capacity` | 2.5K | Capacity tracking and management |
+
+**Security & Auth:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `crypto` | 1.6K | Encryption primitives |
+| `credentials` | 713 | Credential types (access key / secret key) |
+| `signer` | 1.4K | S3 v4 request signing |
+| `iam` | 9.0K | Identity and access management |
+| `policy` | 8.8K | Policy engine (S3 bucket/IAM policies) |
+| `kms` | 8.1K | Key management service integration |
+| `keystone` | 1.9K | OpenStack Keystone auth |
+| `appauth` | 143 | Application-level auth tokens |
+
+**Protocol & API:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `protos` | 5.7K | Protobuf/gRPC definitions for inter-node RPC |
+| `protocols` | 18K | FTP/FTPS, WebDAV, Swift API support |
+| `s3-common` | 738 | Shared S3 types |
+| `s3select-api` | 1.9K | S3 Select interface |
+| `s3select-query` | 3.6K | S3 Select query engine |
+
+**Observability:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `metrics` | 8.4K | Prometheus metric collectors |
+| `io-metrics` | 4.5K | I/O-specific metrics |
+| `obs` | 5.6K | OpenTelemetry tracing and telemetry |
+| `audit` | 2.4K | Audit logging |
+
+**Events:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `notify` | 5.5K | Event notification system |
+| `targets` | 3.2K | Notification targets (Kafka, AMQP, webhook, etc.) |
+
+**Other:**
+
+| Crate | Lines | Purpose |
+|-------|-------|---------|
+| `trusted-proxies` | 4.0K | Trusted proxy / IP forwarding |
+| `zip` | 986 | ZIP archive support for bulk downloads |
+| `workers` | 136 | Simple worker abstraction |
 
 ## Architecture Invariants
 
@@ -116,50 +212,25 @@ default build (lifecycle:
    No upward imports.
 
 2. **Leaf crates have zero internal dependencies.** `config`, `credentials`, `crypto`,
-   `io-metrics`, and `madmin` should depend only on external crates.
+   `io-metrics`, `madmin`, `s3-common` should depend only on external crates.
    - ✅ RESOLVED: the historical `utils → config` and `common → filemeta`/`madmin`
      edges were removed; do not reintroduce them (see Known Structural Issues).
 
 3. **Each type has exactly one definition.** Types shared across crates must be defined
    in one crate and re-exported or imported by others.
-   - ⚠️ VIOLATED: `ReplicationStats` names three unrelated types
-     (`crates/data-usage/src/data_usage.rs`,
-     `crates/obs/src/metrics/collectors/replication.rs`,
-     `crates/ecstore/src/bucket/replication/replication_state.rs`) — a naming
-     collision, not copies; renaming is tracked in rustfs/backlog#1847.
-   - `LastMinuteLatency` has two deliberately different implementations: the
-     per-second bucketed accumulator in `crates/common/src/last_minute.rs` and
-     the in-memory endpoint-health sample tracker in
-     `crates/ecstore/src/bucket/bucket_target_sys.rs` (its doc comment explains
-     why it stays local).
-   - ✅ RESOLVED: `BackpressureConfig` and `DataUsageInfo` each have exactly one
-     definition (`crates/io-core/src/backpressure.rs`,
-     `crates/data-usage/src/data_usage.rs`). The zero-consumer
-     `BackpressureSettings` copy that lingered in io-metrics was removed
-     (rustfs/backlog#1833).
+   - ⚠️ VIOLATED: `ReplicationStats` (4 copies), `LastMinuteLatency` (3 copies),
+     `BackpressureConfig` (3 copies), `DataUsageInfo` (2 copies).
 
 4. **ecstore does not know about HTTP or S3 protocol details.** It operates on
    storage-level abstractions (objects, buckets, disks, pools).
-   - ⚠️ VIOLATED: 58 files under `crates/ecstore/src` reference `s3s`
-     (`rg -l 's3s' crates/ecstore/src | wc -l`), `crates/ecstore/src/client/`
-     is a ~9.4K-line embedded S3 HTTP client, and `crates/ecstore/Cargo.toml`
-     depends on `s3s`, `http`, `hyper`/`hyper-util`/`hyper-rustls`, and
-     `reqwest`. Target state: the engine's need to act as an S3 client
-     (tiering, replication targets) is served by an extracted client crate,
-     and ecstore holds no wire or DTO types.
 
 5. **The `rustfs` binary crate is the only place that wires everything together.**
    Individual crates should be testable in isolation.
 
 6. **Error types use `thiserror` with descriptive names** (e.g., `StorageError`,
    not bare `Error`).
-   - ✅ RESOLVED (strategy): `snafu` is gone from source
-     (`rg -l snafu crates/ rustfs/` is empty) and library code no longer uses
-     `anyhow` (remaining hits are test code and the `e2e_test` crate; `heal`
-     uses `thiserror`).
-   - ⚠️ VIOLATED (naming): 6 crates still export a bare `pub enum Error`:
-     `crypto`, `filemeta`, `heal`, `iam`, `policy`, and `replication`
-     (`src/resync.rs`) — all `thiserror`-derived.
+   - ⚠️ VIOLATED: 6 crates use `pub enum Error`; 2 crates use `snafu`;
+      `heal` use `anyhow` in library code.
 
 ## Known Structural Issues
 
@@ -168,25 +239,13 @@ default build (lifecycle:
 
 ### Critical
 
-- **scanner/data-usage duplicate `.usage-cache.bin` serialization types.** The
-  original finding ("common/scanner code duplication, ~3K lines") is resolved:
-  `scanner` imports the shared data-usage types from `rustfs-data-usage` (see
-  the `pub use rustfs_data_usage::…` re-exports at the top of
-  `crates/scanner/src/data_usage_define.rs`). What remains: `scanner` and
-  `data-usage` each hold their own serialization types for the scanner cache
-  file (`DataUsageCacheInfo`/`DataUsageEntryInfo` in
-  `crates/scanner/src/data_usage_define.rs` vs
-  `DataUsageCacheInfo`/`DataUsageEntry` in
-  `crates/data-usage/src/data_usage.rs`); convergence is tracked in
-  rustfs/backlog#1828.
+- **common/scanner code duplication (~3K lines).** `scanner` depends on `common`
+  but maintains its own copies of `DataUsageInfo`, `LastMinuteLatency`, and related
+  types instead of importing them.
 
-- **ecstore is a monolith (265 files, ~288K lines — roughly half is inline
-  `#[cfg(test)]` code).** Measured with
-  `find crates/ecstore/src -name '*.rs' | xargs wc -l`. It contains disk
-  management, bucket management, erasure coding, replication, lifecycle, RPC,
-  and configuration — all in one crate. It should be decomposed along its
-  existing subdirectories; the split plan lives in
-  [docs/architecture/ecstore-module-split-plan.md](docs/architecture/ecstore-module-split-plan.md).
+- **ecstore is a monolith (87K lines, 163 files).** It contains disk management,
+  bucket management, erasure coding, replication, lifecycle, RPC, and configuration
+  — all in one crate. It should be decomposed along its existing subdirectories.
 
 ### High
 
@@ -194,26 +253,19 @@ default build (lifecycle:
   `common → filemeta/madmin` edges must stay removed so leaf/helper crates do
   not regain upward dependencies.
 
-- **Three-layer backpressure/deadlock policy bridging** across io-core,
-  concurrency, and `rustfs/src/storage`. The config types are no longer
-  duplicated (`BackpressureConfig` and `DeadlockDetectorConfig` are each
-  defined once, in io-core). Storage policies expose and consume explicit
-  projections into the concurrency/io-core policy shapes, and workload
+- **Three-layer BackpressureConfig/DeadlockConfig duplication** across io-core,
+  concurrency, and `rustfs/src/storage`. Storage policies now expose and consume
+  explicit projections into the concurrency/io-core policy shapes, and workload
   admission snapshots are composed through provider registries; later work
   should use those bridges before deleting compatibility wrappers.
 
 ### Medium
 
-- **Bare `Error` naming.** Error-handling strategy has converged on `thiserror`
-  (no `snafu`, no `anyhow` in library code); the remaining inconsistency is the
-  bare `pub enum Error` naming in the 6 crates listed under Invariant 6.
+- **Inconsistent error handling.** Three strategies (thiserror/snafu/anyhow) and
+  mixed naming (bare `Error` vs descriptive names).
 
-- **`common` is mostly parked domain code, not shared utilities.** Of its
-  6,724 lines, ~83% is scanner/heal domain code stranded there to break
-  dependency cycles (`metrics.rs`, ~4,810 lines of scanner-domain metrics;
-  `heal_channel.rs`, ~776 lines of heal-domain channel types). The
-  "common vs utils" naming ambiguity is secondary to moving that code to its
-  domain owners.
+- **Ambiguous common vs utils boundary.** Both described as "utilities and data
+  structures." Need clear ownership rules.
 
 ## Cross-Cutting Concerns
 
@@ -279,7 +331,7 @@ The binary (`main.rs`) boots in this order:
 
 ```
                             ┌─────────┐
-                            │  rustfs │  (binary + lib)
+                            │  rustfs │  (binary + lib, 75K lines)
                             │  main   │
                             └────┬────┘
                                  │
@@ -302,7 +354,7 @@ The binary (`main.rs`) boots in this order:
               │                  │                  │
         ┌─────▼──────┐    ┌──────▼──────┐    ┌──────▼──────┐
         │  ecstore   │    │     rio     │    │   io-core   │
-        │   (core)   │    │  (readers)  │    │ (buffers)   │
+        │ (87K,core) │    │  (readers)  │    │ (zero-copy) │
         └─────┬──────┘    └─────────────┘    └─────────────┘
               │
      ┌─────┬──┼──┬─────┬──────┐
@@ -314,7 +366,7 @@ The binary (`main.rs`) boots in this order:
 
 - **"Where does S3 PutObject go?"**
   `server/` routes → `app/object_usecase` validates → `storage/ecfs` encodes →
-  `ecstore` distributes → `rio` encrypts/compresses → `io-core` supplies buffers
+  `ecstore` distributes → `rio` encrypts/compresses → `io-core` writes
 
 - **"Where are bucket policies enforced?"**
   `app/bucket_usecase` calls into `crates/policy/`

CHANGELOG.md

@@ -9,14 +9,8 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Fixed
 - **Helm Ingress**: `customAnnotations` are now merged with class-specific annotations (nginx/traefik) instead of being ignored when `ingress.className` is set.
-- **Per-pool erasure parity**: Erasure parity (STANDARD and reduced-redundancy) is now resolved independently for every pool instead of reusing the first pool's value. A heterogeneous topology — for example a 4-drive pool plus a 2-drive pool created during expansion — previously inherited the first pool's parity and could resolve to zero data shards in the smaller pool, panicking Reed-Solomon construction on write. Automatic parity now resolves per pool (for example `2+2` in the 4-drive pool and `1+1` in the 2-drive pool). Fixes #4801.
 
 ### Added
-- **NATS JetStream Publish Path**: Opt-in at-least-once delivery for the NATS notify and audit targets. A NATS Core publish flushes to the connection without awaiting a broker acknowledgement, so an event can be lost across a broker restart or a reconnect after the send queue has already cleared it. A queued event now clears only after the JetStream `PublishAck`, so bucket notifications survive those interruptions. Off by default and byte-identical to the NATS Core path when disabled.
-  - Three configuration keys per target: `JETSTREAM_ENABLE`, `JETSTREAM_STREAM_NAME`, and `JETSTREAM_ACK_TIMEOUT_SECS`, under the `RUSTFS_NOTIFY_NATS_` and `RUSTFS_AUDIT_NATS_` prefixes
-  - Durable store-and-forward with a stable dedup id sent as the `Nats-Msg-Id` header, so a replay after a crash is collapsed by the server duplicate window
-  - Pre-flight stream validation, and a bounded failed-events store (count and TTL). Only a non-retryable rejection is recorded in the failed-events store. A retryable condition keeps the entry on the live queue until it is delivered
-  - Operator guide at `docs/operations/nats-jetstream.md`
 - **OpenStack Keystone Authentication Integration**: Full support for OpenStack Keystone authentication via X-Auth-Token headers
   - Tower-based middleware (`KeystoneAuthLayer`) self-contained within `rustfs-keystone` crate
   - Task-local storage for async-safe credential passing between middleware and auth handlers
@@ -39,7 +33,6 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ### Changed
 - **HTTP Server Stack**: Integrated `KeystoneAuthLayer` middleware from `rustfs-keystone` crate into service stack (positioned after ReadinessGateLayer)
-- **Storage-class validation on startup (upgrade note)**: A persisted explicit storage class (`RUSTFS_STORAGE_CLASS_STANDARD` / `RUSTFS_STORAGE_CLASS_RRS`, for example `EC:2`) is now validated against the actual per-pool drive counts at startup and rejected when a pool cannot satisfy it. This is fail-closed and correct, but a cluster that persisted a storage class larger than a small or heterogeneous pool can hold (for example `EC:2` alongside a 2-drive pool), which earlier releases accepted and silently resolved to an invalid layout, will now refuse to start after upgrade. To recover, unset `RUSTFS_STORAGE_CLASS_STANDARD` so the server derives a valid per-pool default automatically, or set it to a value every pool can satisfy.
 - **IAMAuth**: Enhanced `get_secret_key()` to return empty secret for Keystone credentials (bypasses signature validation)
 - **Auth Module**: Modified `check_key_valid()` to retrieve Keystone credentials from task-local storage and determine admin status
 - **`StorageBackend` trait**: extended with multipart upload methods (`create_multipart_upload`, `upload_part`, `complete_multipart_upload`, `abort_multipart_upload`) plus `upload_part_copy`. Streaming-upload code path is now available to FTPS, WebDAV, and Swift drivers as well.

CLAUDE.md

@@ -31,8 +31,6 @@ make build-docker BUILD_OS=ubuntu22.04
 - Architecture, layering, crate map: [ARCHITECTURE.md](ARCHITECTURE.md)
 - Migration guardrails & readiness contracts: [docs/architecture/](docs/architecture/README.md)
 - CI gates: `.github/workflows/ci.yml` (source of truth; never copy its steps into docs)
-- Test-layer taxonomy, per-layer entry commands, serial/nextest rules, flake
-  policy: [docs/testing/README.md](docs/testing/README.md)
 - Tier/ILM transition debugging (xl.meta inspection, versionId tracing):
   [docs/operations/tier-ilm-debugging.md](docs/operations/tier-ilm-debugging.md)
 

CONTRIBUTING.md

@@ -68,8 +68,6 @@ make pre-pr
 
 > `make test` requires [cargo-nextest](https://nexte.st) (CI runs it and only nextest honours `.config/nextest.toml` test-groups). Install it with `cargo install cargo-nextest --locked` or a prebuilt binary (see https://nexte.st/docs/installation/). To run the plain `cargo test` fallback anyway (results not authoritative — serialization semantics differ from CI), set `RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1`.
 
-> For the full test-layer taxonomy (unit / ecstore black-box / e2e / s3s-e2e / S3 compatibility / chaos / fuzz / bench), each layer's entry command, the naming conventions the migration gate depends on, and the serial/nextest rules, see [docs/testing/README.md](docs/testing/README.md).
-
 ### 🔒 Automated Pre-commit Hooks
 #### What `make pre-commit` and `make pre-pr` actually run
 
@@ -91,9 +89,8 @@ A green `make pre-commit` is not enough to open a pull request.
 `make pre-pr` is the **full** gate: it runs all of the guard checks above,
 then `clippy-check` (`cargo clippy --all-targets --all-features -- -D warnings`)
 and `test` (shell script tests, workspace tests excluding `e2e_test`, and doc
-tests). Complete the applicable multi-role adversarial review described in
-`AGENTS.md` before running `make pre-pr`; then run the gate before opening or
-updating a pull request. This is what CI enforces.
+tests). Run `make pre-pr` before opening or updating a pull request — this is
+what CI enforces.
 
 ### 🔒 Git Pre-commit Hooks (optional)
 
@@ -151,9 +148,8 @@ Example output when formatting fails:
 2. **Format your code**: `make fmt` or `cargo fmt --all`
 3. **Run the fast gate**: `make pre-commit` (no clippy, no tests)
 4. **Commit your changes**: `git commit -m "your message"`
-5. **Complete the applicable multi-role adversarial review** for non-exempt changes (see `AGENTS.md`)
-6. **Run the full gate before opening/updating a PR**: `make pre-pr` (clippy + tests)
-7. **Push to your branch**: `git push`
+5. **Run the full gate before opening/updating a PR**: `make pre-pr` (clippy + tests)
+6. **Push to your branch**: `git push`
 
 ### 🛠️ IDE Integration
 

Cargo.toml

@@ -31,7 +31,6 @@ members = [
     "crates/lifecycle", # Lifecycle rule evaluation contracts
     "crates/kms", # Key Management Service
     "crates/lock", # Distributed locking implementation
-    "crates/log-analyzer", # Offline log fault-analysis core (rustfs diagnose)
     "crates/madmin", # Management dashboard and admin API interface
     "crates/notify", # Notification system for events
     "crates/obs", # Observability utilities
@@ -41,7 +40,7 @@ members = [
     "crates/protocols", # Protocol implementations (FTPS, SFTP, etc.)
     "crates/protos", # Protocol buffer definitions
     "crates/rio", # Rust I/O utilities and abstractions
-    "crates/rio-v2", # MinIO on-disk format compatibility I/O layer (feature-gated, ships in no default build)
+    "crates/rio-v2", # Next-generation Rust I/O compatibility layer
     "crates/replication", # Replication contracts and wire formats
     "crates/concurrency", # Concurrency management for RustFS - timeout, locking, backpressure, and I/O scheduling
     "crates/s3-types", # S3 event type definitions
@@ -53,7 +52,6 @@ members = [
     "crates/extension-schema", # Extension schema contracts
     "crates/signer", # client signer
     "crates/storage-api", # Storage API contracts
-    "crates/test-utils", # Shared test bootstrap helpers (dev-dependency only)
     "crates/targets", # Target-specific configurations and utilities
     "crates/trusted-proxies", # Trusted proxies management
     "crates/tls-runtime", # Project-wide TLS runtime foundation
@@ -68,8 +66,8 @@ resolver = "3"
 edition = "2024"
 license = "Apache-2.0"
 repository = "https://github.com/rustfs/rustfs"
-rust-version = "1.97.1"
-version = "1.0.0-rc.3"
+rust-version = "1.96.0"
+version = "1.0.0-beta.8"
 homepage = "https://rustfs.com"
 description = "RustFS is a high-performance distributed object storage software built using Rust, one of the most popular languages worldwide. "
 keywords = ["RustFS", "Minio", "object-storage", "filesystem", "s3"]
@@ -86,94 +84,92 @@ redundant_clone = "warn"
 
 [workspace.dependencies]
 # RustFS Internal Crates
-rustfs = { path = "./rustfs", version = "1.0.0-rc.3" }
-rustfs-heal = { path = "crates/heal", version = "1.0.0-rc.3" }
-rustfs-audit = { path = "crates/audit", version = "1.0.0-rc.3" }
-rustfs-checksums = { path = "crates/checksums", version = "1.0.0-rc.3" }
-rustfs-common = { path = "crates/common", version = "1.0.0-rc.3" }
-rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-rc.3" }
-rustfs-config = { path = "./crates/config", version = "1.0.0-rc.3" }
-rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-rc.3" }
-rustfs-credentials = { path = "crates/credentials", version = "1.0.0-rc.3" }
-rustfs-crypto = { path = "crates/crypto", version = "1.0.0-rc.3" }
-rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-rc.3" }
-rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-rc.3" }
-rustfs-iam = { path = "crates/iam", version = "1.0.0-rc.3" }
-rustfs-keystone = { path = "crates/keystone", version = "1.0.0-rc.3" }
-rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-rc.3" }
-rustfs-kms = { path = "crates/kms", version = "1.0.0-rc.3" }
-rustfs-lock = { path = "crates/lock", version = "1.0.0-rc.3" }
-rustfs-madmin = { path = "crates/madmin", version = "1.0.0-rc.3" }
-rustfs-notify = { path = "crates/notify", version = "1.0.0-rc.3" }
-rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-rc.3" }
-rustfs-io-core = { path = "crates/io-core", version = "1.0.0-rc.3" }
-rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-rc.3" }
-rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-rc.3", default-features = false }
-rustfs-log-analyzer = { path = "crates/log-analyzer", version = "1.0.0-rc.3" }
-rustfs-obs = { path = "crates/obs", version = "1.0.0-rc.3" }
-rustfs-policy = { path = "crates/policy", version = "1.0.0-rc.3" }
-rustfs-protos = { path = "crates/protos", version = "1.0.0-rc.3" }
-rustfs-protocols = { path = "crates/protocols", version = "1.0.0-rc.3" }
-rustfs-replication = { path = "crates/replication", version = "1.0.0-rc.3" }
-rustfs-rio = { path = "crates/rio", version = "1.0.0-rc.3" }
-rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-rc.3" }
-rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-rc.3" }
-rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-rc.3" }
-rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-rc.3" }
-rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-rc.3" }
-rustfs-scanner = { path = "crates/scanner", version = "1.0.0-rc.3" }
-rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-rc.3" }
-rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-rc.3" }
-rustfs-signer = { path = "crates/signer", version = "1.0.0-rc.3" }
-rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-rc.3" }
-rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-rc.3" }
-rustfs-targets = { path = "crates/targets", version = "1.0.0-rc.3" }
-rustfs-test-utils = { path = "crates/test-utils", version = "1.0.0-rc.3" }
-rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-rc.3" }
-rustfs-utils = { path = "crates/utils", version = "1.0.0-rc.3" }
-rustfs-zip = { path = "./crates/zip", version = "1.0.0-rc.3" }
+rustfs = { path = "./rustfs", version = "1.0.0-beta.8" }
+rustfs-heal = { path = "crates/heal", version = "1.0.0-beta.8" }
+rustfs-audit = { path = "crates/audit", version = "1.0.0-beta.8" }
+rustfs-checksums = { path = "crates/checksums", version = "1.0.0-beta.8" }
+rustfs-common = { path = "crates/common", version = "1.0.0-beta.8" }
+rustfs-data-usage = { path = "crates/data-usage", version = "1.0.0-beta.8" }
+rustfs-config = { path = "./crates/config", version = "1.0.0-beta.8" }
+rustfs-concurrency = { path = "./crates/concurrency", version = "1.0.0-beta.8" }
+rustfs-credentials = { path = "crates/credentials", version = "1.0.0-beta.8" }
+rustfs-crypto = { path = "crates/crypto", version = "1.0.0-beta.8" }
+rustfs-ecstore = { path = "crates/ecstore", version = "1.0.0-beta.8" }
+rustfs-filemeta = { path = "crates/filemeta", version = "1.0.0-beta.8" }
+rustfs-iam = { path = "crates/iam", version = "1.0.0-beta.8" }
+rustfs-keystone = { path = "crates/keystone", version = "1.0.0-beta.8" }
+rustfs-lifecycle = { path = "crates/lifecycle", version = "1.0.0-beta.8" }
+rustfs-kms = { path = "crates/kms", version = "1.0.0-beta.8" }
+rustfs-lock = { path = "crates/lock", version = "1.0.0-beta.8" }
+rustfs-madmin = { path = "crates/madmin", version = "1.0.0-beta.8" }
+rustfs-notify = { path = "crates/notify", version = "1.0.0-beta.8" }
+rustfs-io-metrics = { path = "crates/io-metrics", version = "1.0.0-beta.8" }
+rustfs-io-core = { path = "crates/io-core", version = "1.0.0-beta.8" }
+rustfs-object-capacity = { path = "crates/object-capacity", version = "1.0.0-beta.8" }
+rustfs-object-data-cache = { path = "crates/object-data-cache", version = "1.0.0-beta.8" }
+rustfs-obs = { path = "crates/obs", version = "1.0.0-beta.8" }
+rustfs-policy = { path = "crates/policy", version = "1.0.0-beta.8" }
+rustfs-protos = { path = "crates/protos", version = "1.0.0-beta.8" }
+rustfs-protocols = { path = "crates/protocols", version = "1.0.0-beta.8" }
+rustfs-replication = { path = "crates/replication", version = "1.0.0-beta.8" }
+rustfs-rio = { path = "crates/rio", version = "1.0.0-beta.8" }
+rustfs-rio-v2 = { path = "crates/rio-v2", version = "1.0.0-beta.8" }
+rustfs-s3-types = { path = "crates/s3-types", version = "1.0.0-beta.8" }
+rustfs-s3-ops = { path = "crates/s3-ops", version = "1.0.0-beta.8" }
+rustfs-s3select-api = { path = "crates/s3select-api", version = "1.0.0-beta.8" }
+rustfs-s3select-query = { path = "crates/s3select-query", version = "1.0.0-beta.8" }
+rustfs-scanner = { path = "crates/scanner", version = "1.0.0-beta.8" }
+rustfs-security-governance = { path = "crates/security-governance", version = "1.0.0-beta.8" }
+rustfs-extension-schema = { path = "crates/extension-schema", version = "1.0.0-beta.8" }
+rustfs-signer = { path = "crates/signer", version = "1.0.0-beta.8" }
+rustfs-storage-api = { path = "crates/storage-api", version = "1.0.0-beta.8" }
+rustfs-trusted-proxies = { path = "crates/trusted-proxies", version = "1.0.0-beta.8" }
+rustfs-targets = { path = "crates/targets", version = "1.0.0-beta.8" }
+rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.0-beta.8" }
+rustfs-utils = { path = "crates/utils", version = "1.0.0-beta.8" }
+rustfs-zip = { path = "./crates/zip", version = "1.0.0-beta.8" }
 
 # Async Runtime and Networking
 async-channel = "2.5.0"
-async_zip = { default-features = false, version = "0.0.18" }
-mysql_async = { default-features = false, version = "0.37" }
-async-compression = { version = "0.4.43" }
+async_zip = { version = "0.0.18", default-features = false, features = ["tokio", "deflate"] }
+mysql_async = { version = "0.37", default-features = false, features = ["default-rustls", "tracing"] }
+async-compression = { version = "0.4.42" }
 async-recursion = "1.1.1"
-async-trait = "0.1.92"
-async-nats = { version = "0.50.0", default-features = false }
+async-trait = "0.1.89"
+async-nats = "0.49.1"
 axum = "0.8.9"
-futures = "0.3.34"
-futures-core = "0.3.34"
+futures = "0.3.32"
+futures-core = "0.3.32"
 futures-lite = "2.6.1"
-futures-util = "0.3.34"
+futures-util = "0.3.32"
 pollster = "1.0.1"
-pulsar = { default-features = false, version = "6.8.0" }
-lapin = { default-features = false, version = "4.10.0" }
-hyper = { version = "1.11.0" }
-hyper-rustls = { default-features = false, version = "0.27.9" }
-hyper-util = { version = "0.1.20" }
-http = "1.5.0"
-http-body = "1.1.0"
-http-body-util = "0.1.5"
+pulsar = { version = "6.8.0", default-features = false, features = ["tokio-rustls-runtime", "telemetry"] }
+lapin = { version = "4.10.0", default-features = false, features = ["tokio", "rustls", "rustls--aws_lc_rs"] }
+hyper = { version = "1.10.1", features = ["http2", "http1", "server"] }
+hyper-rustls = { version = "0.27.9", default-features = false, features = ["native-tokio", "http1", "tls12", "logging", "http2", "aws-lc-rs", "webpki-roots"] }
+hyper-util = { version = "0.1.20", features = ["tokio", "server-auto", "server-graceful", "tracing"] }
+http = "1.4.2"
+http-body = "1.0.1"
+http-body-util = "0.1.3"
 minlz = "1.2.3"
-reqwest = "0.13.4"
+reqwest = { version = "0.13.4", default-features = false, features = ["rustls", "charset", "http2", "system-proxy", "stream", "json", "blocking", "query", "form"] }
 rustfs-kafka-async = { version = "1.2.0" }
-socket2 = { version = "0.6.5" }
-tokio = { version = "1.53.1" }
-tokio-rustls = { default-features = false, version = "0.26.4" }
-tokio-stream = { version = "0.1.19" }
+socket2 = { version = "0.6.4", features = ["all"] }
+tokio = { version = "1.52.3", features = ["fs", "rt-multi-thread"] }
+tokio-rustls = { version = "0.26.4", default-features = false, features = ["logging", "tls12", "aws-lc-rs"] }
+tokio-stream = { version = "0.1.18" }
 tokio-test = "0.4.5"
-tokio-util = { version = "0.7.19" }
-tonic = { version = "0.14.6" }
+tokio-util = { version = "0.7.18", features = ["io", "compat"] }
+tonic = { version = "0.14.6", features = ["gzip", "deflate"] }
 tonic-prost = { version = "0.14.6" }
 tonic-prost-build = { version = "0.14.6" }
-tower = { version = "0.5.3" }
-tower-http = { version = "0.7.0" }
+tower = { version = "0.5.3", features = ["timeout"] }
+tower-http = { version = "0.7.0", features = ["cors"] }
 
 # Serialization and Data Formats
-apache-avro = { version = "0.22.0", features = ["snappy", "zstandard"] }
-bytes = { version = "1.12.1" }
-bytesize = "2.7.0"
+apache-avro = "0.21.0"
+bytes = { version = "1.12.1", features = ["serde"] }
+bytesize = "2.4.2"
 byteorder = "1.5.0"
 flatbuffers = "25.12.19"
 form_urlencoded = "1.2.2"
@@ -181,9 +177,8 @@ prost = "0.14.4"
 quick-xml = "0.41.0"
 rmp = { version = "0.8.15" }
 rmp-serde = { version = "1.3.1" }
-serde = { version = "1.0.229" }
-serde_ignored = { version = "0.1" }
-serde_json = { version = "1.0.151" }
+serde = { version = "1.0.228", features = ["derive"] }
+serde_json = { version = "1.0.150", features = ["raw_value"] }
 serde_urlencoded = "0.7.1"
 
 # Cryptography and Security
@@ -191,173 +186,163 @@ serde_urlencoded = "0.7.1"
 # matching stable releases are not available yet, while previous stable lines
 # have incompatible APIs. Keep them exact-pinned and monitor upstream for stable
 # releases.
-aes-gcm = { version = "=0.11.0" }
+aes-gcm = { version = "=0.11.0", features = ["rand_core"] }
 argon2 = { version = "=0.6.0-rc.8" }
 blake2 = "=0.11.0-rc.6"
 chacha20poly1305 = { version = "=0.11.0" }
 crc-fast = "1.10.0"
 hmac = { version = "0.13.0" }
-jsonwebtoken = { version = "11.0.0" }
-openidconnect = { default-features = false, version = "4.0" }
+jsonwebtoken = { version = "10.4.0", features = ["aws_lc_rs"] }
+openidconnect = { version = "4.0", default-features = false, features = ["accept-rfc3339-timestamps"] }
 pbkdf2 = "0.13.0"
 rsa = { version = "=0.10.0-rc.18" }
-rustls = { default-features = false, version = "0.23.43" }
+rustls = { version = "0.23.41", default-features = false, features = ["aws-lc-rs", "logging", "tls12", "prefer-post-quantum", "std"] }
 rustls-native-certs = "0.8"
-rustls-pki-types = "1.15.1"
-x509-parser = "0.18.1"
+rustls-pki-types = "1.15.0"
 sha1 = "0.11.0"
 sha2 = "0.11.0"
 subtle = "2.6"
-zeroize = { version = "1.9.0" }
+zeroize = { version = "1.9.0", features = ["derive"] }
 
 # Time and Date
-chrono = { version = "0.4.45" }
+chrono = { version = "0.4.45", features = ["serde"] }
 humantime = "2.4.0"
-jiff = { version = "0.2.35" }
-time = { version = "0.3.55" }
+jiff = { version = "0.2.32", features = ["serde"] }
+time = { version = "0.3.53", features = ["std", "parsing", "formatting", "macros", "serde"] }
 
 # Database
-deadpool-postgres = { version = "0.14" }
-tokio-postgres = { default-features = false, version = "0.7.18" }
+deadpool-postgres = { version = "0.14", features = ["rt_tokio_1"] }
+tokio-postgres = { version = "0.7.18", default-features = false, features = ["runtime", "with-serde_json-1"] }
 tokio-postgres-rustls = "0.14.0"
 
 # Utilities and Tools
-anyhow = "1.0.104"
+anyhow = "1.0.103"
 arc-swap = "1.9.2"
-astral-tokio-tar = "0.6.4"
+astral-tokio-tar = "0.6.3"
 atoi = "3.1.0"
 atomic_enum = "0.3.0"
-aws-config = { version = "1.10.1" }
+aws-config = { version = "1.9.0" }
 aws-credential-types = { version = "1.3.0" }
-aws-sdk-kms = { default-features = false, version = "1.115.0" }
-aws-sdk-s3 = { default-features = false, version = "1.142.0" }
-aws-sdk-sts = { default-features = false, version = "1.111.0" }
-aws-smithy-http-client = { default-features = false, version = "1.4.0" }
-aws-smithy-runtime-api = { version = "1.15.0" }
-aws-smithy-types = { version = "1.6.2" }
-base64 = "0.23.1"
+aws-sdk-s3 = { version = "1.138.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
+aws-smithy-http-client = { version = "1.2.0", default-features = false, features = ["default-client", "rustls-aws-lc"] }
+aws-smithy-runtime-api = { version = "1.13.0", features = ["http-1x"] }
+aws-smithy-types = { version = "1.6.1" }
+base64 = "0.22.1"
 base64-simd = "0.8.0"
 brotli = "8.0.4"
-clap = { version = "4.6.6" }
-const-str = { version = "1.1.0" }
+clap = { version = "4.6.1", features = ["derive", "env"] }
+const-str = { version = "1.1.0", features = ["std", "proc"] }
 convert_case = "0.11.0"
-criterion = { version = "0.8" }
+criterion = { version = "0.8", features = ["html_reports"] }
 crossbeam-queue = "0.3.13"
 crossbeam-channel = "0.5.16"
 crossbeam-deque = "0.8.7"
 crossbeam-utils = "0.8.22"
-datafusion = { default-features = false, version = "55.0.0" }
+datafusion = { git = "https://github.com/apache/datafusion.git", rev = "dae03ee062b2abf986de8df12ea82fb1578a2d99" }
 derive_builder = "0.20.2"
-enumset = "1.1.14"
+enumset = "1.1.13"
 faster-hex = "0.10.0"
 flate2 = "1.1.9"
-glob = "0.3.4"
-google-cloud-storage = "1.17.0"
-google-cloud-auth = "1.15.0"
-hashbrown = { version = "0.17.1" }
+glob = "0.3.3"
+google-cloud-storage = "1.15.0"
+google-cloud-auth = "1.13.0"
+hashbrown = { version = "0.17.1", features = ["serde", "rayon"] }
 hex = "0.4.3"
 hex-simd = "0.8.0"
 highway = { version = "1.3.0" }
-hostname = "0.4.2"
-ipnetwork = { version = "0.21.1" }
+ipnetwork = { version = "0.21.1", features = ["serde"] }
 lazy_static = "1.5.0"
-libc = "0.2.189"
+libc = "0.2.186"
 libsystemd = "0.7.2"
 local-ip-address = "0.6.13"
-log = "0.4"
 memmap2 = "0.9.11"
 lz4 = "1.28.1"
 matchit = "0.9.2"
 md-5 = "0.11.0"
+md5 = "0.8.1"
 mime_guess = "2.0.5"
-moka = { version = "0.12.16" }
+moka = { version = "0.12.15", features = ["future"] }
 netif = "0.1.6"
 num_cpus = { version = "1.17.0" }
 nvml-wrapper = "0.12.1"
 parking_lot = "0.12.5"
 path-absolutize = "4.0.1"
+path-clean = "1.0.1"
 percent-encoding = "2.3.2"
 pin-project-lite = "0.2.17"
 pretty_assertions = "1.4.1"
-rand = { version = "0.10.2" }
-ratelimit = "2.0.0"
+rand = { version = "0.10.2", features = ["serde"] }
+ratelimit = "0.10.1"
 rayon = "1.12.0"
-reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "8.0.2" }
+reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "7.0.1", features = ["simd-accel"] }
+#reed-solomon-erasure = { version = "6.0", features = ["simd-accel"], git = "https://github.com/houseme/reed-solomon-erasure",rev = "main" }
 reed-solomon-simd = "3.1.0"
-regex = { version = "1.13.1" }
-rumqttc = { package = "rumqttc-next", version = "0.34.0" }
-redis = { version = "1.6.0" }
-rustify = { version = "0.7", default-features = false }
-rustix = { version = "1.1.4" }
+regex = { version = "1.13.0" }
+rumqttc = { package = "rumqttc-next", version = "0.33.2", features = ["websocket"] }
+redis = { version = "1.3.0", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
+rustix = { version = "1.1.4", features = ["fs"] }
 rust-embed = { version = "8.12.0" }
 rustc-hash = { version = "2.1.3" }
-s3s = { git = "https://github.com/rustfs/s3s.git", rev = "ed70cb048cc4be168419d461cb9ac3c2c7fa6d5a" }
-serial_test = "4.0.1"
-shadow-rs = { default-features = false, version = "2.0.0" }
+s3s = { version = "0.14.1", features = ["minio"] }
+serial_test = "3.5.0"
+shadow-rs = { version = "2.0.0", default-features = false }
 siphasher = "1.0.3"
-smallvec = { version = "1.15.2" }
-compact_str = "0.10.0"
-snap = "1.1.2"
-starshard = { version = "2.2.2" }
-strum = { version = "0.28.0" }
+smallvec = { version = "1.15.2", features = ["serde"] }
+smartstring = "1.0.1"
+snap = "1.1.1"
+starshard = { version = "2.2.1", features = ["rayon", "async", "serde"] }
+strum = { version = "0.28.0", features = ["derive"] }
 sysinfo = "0.39.6"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
 test-case = "3.3.1"
-thiserror = "2.0.20"
+thiserror = "2.0.18"
 tracing = { version = "0.1.44" }
 tracing-appender = "0.2.5"
-tracing-core = "0.1.36"
 tracing-error = "0.2.1"
 tracing-opentelemetry = { version = "0.33" }
-tracing-subscriber = { version = "0.3.23" }
+tracing-subscriber = { version = "0.3.23", features = ["env-filter", "time"] }
 transform-stream = "0.3.1"
 url = "2.5.8"
 urlencoding = "2.1.3"
-uuid = { version = "1.24.1" }
+uuid = { version = "1.23.5", features = ["v4", "fast-rng", "macro-diagnostics"] }
 vaultrs = { version = "0.8.0" }
-tar = "0.4.46"
 walkdir = "2.5.0"
-winapi-util = "0.1.11"
 windows = { version = "0.62.2" }
-windows-sys = "0.61.2"
-xxhash-rust = { version = "0.8.18" }
+xxhash-rust = { version = "0.8.16", features = ["xxh64", "xxh3"] }
 zip = "8.6.0"
 zstd = "0.13.3"
 
 # Observability and Metrics
 metrics = "0.24.6"
-metrics-util = "0.20"
 dial9-tokio-telemetry = "0.3"
 opentelemetry = { version = "0.32.0" }
-opentelemetry-appender-tracing = { version = "0.32.0" }
-opentelemetry-otlp = { version = "0.32.0" }
-opentelemetry-proto = { version = "0.32.0", default-features = false, features = ["metrics", "gen-tonic-messages"] }
-opentelemetry_sdk = { version = "0.32.1" }
-opentelemetry-semantic-conventions = { version = "0.32.1" }
+opentelemetry-appender-tracing = { version = "0.32.0", features = ["experimental_span_attributes", "experimental_metadata_attributes"] }
+opentelemetry-otlp = { version = "0.32.0", features = ["gzip-http", "reqwest-rustls"] }
+opentelemetry_sdk = { version = "0.32.1", features = ["rt-tokio"] }
+opentelemetry-semantic-conventions = { version = "0.32.1", features = ["semconv_experimental"] }
 opentelemetry-stdout = { version = "0.32.0" }
-pyroscope = { version = "2.1.1" }
+pyroscope = { version = "2.1.0", features = ["backend-pprof-rs"] }
 
 # FTP and SFTP
-libunftp = { version = "0.23.0" }
+libunftp = { version = "0.23.0", features = ["experimental"] }
 unftp-core = "0.1.0"
-suppaftp = { version = "10.0.2" }
-rcgen = { version = "0.14.9", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
-russh = { version = "0.62.7" }
-russh-sftp = "2.4.0"
+suppaftp = { version = "10.0.0", features = ["tokio", "tokio-rustls-aws-lc-rs"] }
+rcgen = "0.14.8"
+russh = { version = "0.62.2", features = ["serde"] }
+russh-sftp = "2.3.0"
 
 # WebDAV
 dav-server = "0.11.0"
 
 # Performance Analysis and Memory Profiling
-mimalloc = { version = "0.1.52", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11" }
-libmimalloc-sys = { version = "0.1.49", git = "https://github.com/xonatius/mimalloc_rust.git", rev = "6d4c41bb10c6d9da1d1b6f07b38c4cc051667f11", features = ["extended"] }
-hotpath = { version = "0.23.3", default-features = false }
+mimalloc = "0.1"
+hotpath = "0.21"
 # Snapshot testing for output format regression detection
-insta = { version = "1.48" }
+insta = { version = "1.48", features = ["yaml", "json"] }
 
 [workspace.metadata.cargo-shear]
-ignored = ["hotpath", "rustfs"]
+ignored = ["rustfs"]
 
 [profile.dev]
 # Full debuginfo roughly doubles compile+link time and produces multi-GB
@@ -368,16 +353,12 @@ debug = "line-tables-only"
 
 [profile.release]
 opt-level = 3
-lto = "thin"
-codegen-units = 1
-debug = 0
-split-debuginfo = "off"
-strip = "symbols"
 
 [profile.production]
 inherits = "release"
+lto = "fat"
+codegen-units = 1
 
 [profile.profiling]
 inherits = "release"
 debug = true
-strip = "none"

Dockerfile

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM alpine:3.24.1 AS build
+FROM alpine:3.23.4 AS build
 
 ARG TARGETARCH
 ARG RELEASE=latest
@@ -70,7 +70,7 @@ RUN set -eux; \
     rm -rf rustfs.zip /build/.tmp || true
 
 
-FROM alpine:3.24.1
+FROM alpine:3.23.4
 
 ARG RELEASE=latest
 ARG BUILD_DATE
@@ -88,15 +88,8 @@ LABEL name="RustFS" \
       url="https://rustfs.com" \
       license="Apache-2.0"
 
-# Upgrade base-image packages so published images pick up security fixes
-# (e.g. openssl/libssl3 CVEs) without waiting for a new Alpine point release.
-RUN apk upgrade --no-cache && \
-    apk add --no-cache \
-    ca-certificates \
-    coreutils \
-    curl \
-    tzdata \
-    && test "$(TZ=Asia/Kolkata date +%z)" = "+0530"
+RUN apk update && \
+    apk add --no-cache ca-certificates coreutils curl
 
 COPY --from=build /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
 COPY --from=build /build/rustfs /usr/bin/rustfs

Dockerfile.decommission-local

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM rust:1.97.1-trixie
+FROM rust:1.95-trixie
 
 RUN set -eux; \
     export DEBIAN_FRONTEND=noninteractive; \

Dockerfile.glibc

@@ -12,7 +12,7 @@
 # See the License for the specific language governing permissions and
 # limitations under the License.
 
-FROM ubuntu:26.04 AS build
+FROM ubuntu:24.04 AS build
 
 ARG TARGETARCH
 ARG RELEASE=latest
@@ -76,7 +76,7 @@ RUN set -eux; \
     chmod +x /build/rustfs; \
     rm -rf rustfs.zip /build/.tmp || true
 
-FROM ubuntu:26.04
+FROM ubuntu:24.04
 
 ARG RELEASE=latest
 ARG BUILD_DATE
@@ -93,14 +93,9 @@ LABEL name="RustFS" \
       url="https://rustfs.com" \
       license="Apache-2.0"
 
-# Upgrade base-image packages so published images pick up security fixes
-# (e.g. tar/gzip/perl CVEs) without waiting for a new Ubuntu point release.
-RUN apt-get update && apt-get upgrade -y \
-    && DEBIAN_FRONTEND=noninteractive apt-get install -y --no-install-recommends \
+RUN apt-get update && apt-get install -y --no-install-recommends \
     ca-certificates \
     curl \
-    tzdata \
-    && test "$(TZ=Asia/Kolkata date +%z)" = "+0530" \
     && rm -rf /var/lib/apt/lists/*
 
 COPY --from=build /build/rustfs /usr/bin/rustfs

Dockerfile.source

@@ -32,7 +32,7 @@ ARG RUSTFS_BUILD_FEATURES=""
 # -----------------------------
 # Build stage
 # -----------------------------
-FROM rust:1.97.1-trixie AS builder
+FROM rust:1.95-trixie AS builder
 
 # Re-declare args after FROM
 ARG TARGETPLATFORM
@@ -208,7 +208,7 @@ CMD ["cargo", "run", "--bin", "rustfs", "--"]
 # -----------------------------
 # Runtime stage (Ubuntu minimal)
 # -----------------------------
-FROM ubuntu:26.04
+FROM ubuntu:24.04
 
 ARG BUILD_DATE
 ARG VCS_REF
@@ -223,7 +223,6 @@ LABEL name="RustFS (dev-local)" \
 RUN set -eux; \
     export DEBIAN_FRONTEND=noninteractive; \
     apt-get update; \
-    apt-get upgrade -y; \
     apt-get install -y --no-install-recommends \
     ca-certificates \
     curl \

README.md

@@ -116,7 +116,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # Using specific version
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-rc.3
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.8
 ```
 
 If you use [podman](https://github.com/containers/podman) instead of docker, you can install the RustFS with the below command
@@ -163,7 +163,6 @@ docker run -d --name rustfs -p 9000:9000 \
   -e RUSTFS_NOTIFY_WEBHOOK_ENABLE_PRIMARY=on \
   -e RUSTFS_NOTIFY_WEBHOOK_ENDPOINT_PRIMARY=http://<host-ip>:3020/webhook \
   -e RUSTFS_NOTIFY_WEBHOOK_QUEUE_DIR_PRIMARY=/tmp/rustfs-events \
-  -e RUSTFS_OUTBOUND_ALLOW_ORIGINS=http://<host-ip>:3020 \
   rustfs/rustfs:latest
 ```
 
@@ -172,11 +171,6 @@ Notes:
 - For ARN `arn:rustfs:sqs::primary:webhook`, use instance-scoped env vars with `_PRIMARY`.
 - If queue dir is omitted, default is `/opt/rustfs/events`; ensure it is writable by the container runtime user.
 - `RUSTFS_NOTIFY_WEBHOOK_SKIP_TLS_VERIFY_PRIMARY` defaults to `false`; enabling it skips webhook TLS certificate verification, allows MITM attacks, and emits a startup warning. Prefer `RUSTFS_NOTIFY_WEBHOOK_CLIENT_CA_PRIMARY` for private CAs.
-- Since `1.0.0-beta.11`, webhook endpoints on private or container networks
-  (`Docker Compose service names`, `host.docker.internal`, RFC 1918 addresses) are
-  blocked unless their exact `scheme://host:port` origin is listed in
-  `RUSTFS_OUTBOUND_ALLOW_ORIGINS` (the origin only, without the path). See
-  [Outbound Connection Policy](docs/operations/outbound-connection-policy.md).
 
 **NOTE**: We recommend reviewing the `docker-compose.yml` file before running. It defines several services including Grafana, Prometheus, and Jaeger, which are helpful for RustFS observability. If you wish to start Redis or Nginx containers, you can specify the corresponding profiles.
 
@@ -224,10 +218,7 @@ For scanner pacing, cycle budgets, bitrot cadence, lifecycle transition status,
 and single-node single-disk idle CPU tuning, see
 [Scanner Runtime Controls](docs/operations/scanner-runtime-controls.md). For
 repeatable scanner-pressure validation, see
-[Scanner Benchmark Runbook](docs/operations/scanner-benchmark-runbook.md). For
-drive timeout knobs on slow storage — including the walk stall budget that
-governs `ListObjects` on large prefixes — see
-[Drive Timeout Tuning](docs/operations/drive-timeout-tuning.md).
+[Scanner Benchmark Runbook](docs/operations/scanner-benchmark-runbook.md).
 
 ### 5\. Nix Flake (Option 5)
 
@@ -268,7 +259,7 @@ rustfs --help
 2. **Create a Bucket**: Use the console to create a new bucket for your objects.
 3. **Upload Objects**: You can upload files directly through the console or use S3-compatible APIs/clients to interact with your RustFS instance.
 
-**NOTE**: To access the RustFS instance via `https`, please refer to the [TLS Configuration Docs](https://docs.rustfs.com/integration/tls-configured).
+**NOTE**: To access the RustFS instance via `https`, please refer to the [TLS Configuration Docs](https://docs.rustfs.com/integration/tls-configured.html).
 
 ### OIDC Roles Claim (Microsoft Entra ID)
 
@@ -344,18 +335,12 @@ If you have any questions or need assistance:
 RustFS is a community-driven project, and we appreciate all contributions. Check out the [Contributors](https://github.com/rustfs/rustfs/graphs/contributors) page to see the amazing people who have helped make RustFS better.
 
 <a href="https://github.com/rustfs/rustfs/graphs/contributors">
-<picture>
-  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rustfs/rustfs/star-history/contributors-dark.svg">
-  <img src="https://raw.githubusercontent.com/rustfs/rustfs/star-history/contributors-light.svg" alt="RustFS contributors">
-</picture>
+<img src="https://opencollective.com/rustfs/contributors.svg?width=890&limit=500&button=false" alt="Contributors" />
 </a>
 
 ## Star History
 
-<picture>
-  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rustfs/rustfs/star-history/star-history-dark.svg">
-  <img src="https://raw.githubusercontent.com/rustfs/rustfs/star-history/star-history-light.svg" alt="RustFS star history chart">
-</picture>
+[![Star History Chart](https://api.star-history.com/svg?repos=rustfs/rustfs&type=date&legend=top-left)](https://www.star-history.com/#rustfs/rustfs&type=date&legend=top-left)
 
 ## License
 

README_ZH.md

@@ -113,7 +113,7 @@ chown -R 10001:10001 data logs
 docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:latest
 
 # 使用指定版本运行
-docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-rc.3
+docker run -d -p 9000:9000 -p 9001:9001 -v $(pwd)/data:/data -v $(pwd)/logs:/logs rustfs/rustfs:1.0.0-beta.8
 ```
 
 如果您通过绑定挂载启用 TLS 证书目录，也请用同样方式准备该目录：
@@ -214,7 +214,7 @@ rustfs --help
 2. **创建存储桶**: 使用控制台为您​​的对象创建一个新的存储桶 (Bucket)。
 3. **上传对象**: 您可以直接通过控制台上传文件，或使用 S3 兼容的 API/客户端与您的 RustFS 实例进行交互。
 
-**注意**: 如果您希望通过 `https` 访问 RustFS 实例，请参考 [TLS 配置文档](https://docs.rustfs.com/integration/tls-configured)。
+**注意**: 如果您希望通过 `https` 访问 RustFS 实例，请参考 [TLS 配置文档](https://docs.rustfs.com/integration/tls-configured.html)。
 
 ## 文档
 
@@ -247,18 +247,12 @@ rustfs --help
 RustFS 是一个社区驱动的项目，我们感谢所有的贡献。请查看 [贡献者](https://github.com/rustfs/rustfs/graphs/contributors) 页面，看看那些让 RustFS 变得更好的了不起的人们。
 
 <a href="https://github.com/rustfs/rustfs/graphs/contributors">
-<picture>
-  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rustfs/rustfs/star-history/contributors-dark.svg">
-  <img src="https://raw.githubusercontent.com/rustfs/rustfs/star-history/contributors-light.svg" alt="RustFS 贡献者">
-</picture>
+<img src="https://opencollective.com/rustfs/contributors.svg?width=890&limit=500&button=false" alt="Contributors" />
 </a>
 
 ## Star 历史
 
-<picture>
-  <source media="(prefers-color-scheme: dark)" srcset="https://raw.githubusercontent.com/rustfs/rustfs/star-history/star-history-dark.svg">
-  <img src="https://raw.githubusercontent.com/rustfs/rustfs/star-history/star-history-light.svg" alt="RustFS Star 历史图表">
-</picture>
+[![Star History Chart](https://api.star-history.com/svg?repos=rustfs/rustfs&type=date&legend=top-left)](https://www.star-history.com/#rustfs/rustfs&type=date&legend=top-left)
 
 ## 许可证
 

_typos.toml

@@ -40,7 +40,6 @@ mak = "mak"
 gae = "gae"
 GAE = "GAE"
 thr = "thr"
-mis = "mis"
 # s3-tests original test names (cannot be changed)
 nonexisted = "nonexisted"
 consts = "consts"
@@ -48,9 +47,6 @@ consts = "consts"
 Hashi = "Hashi"  # HashiCorp
 # Accept alternate spelling used in parser/XML comments.
 unparseable = "unparseable"
-# Disaster-recovery objectives: recovery time and recovery point.
-RTO = "RTO"
-rto = "rto"
 
 [files]
 extend-exclude = []

build-rustfs.sh

@@ -217,7 +217,7 @@ setup_rust_environment() {
     # Set up environment variables for musl targets
     if [[ "$PLATFORM" == *"musl"* ]]; then
         print_message $YELLOW "Setting up environment for musl target..."
-        export RUSTFLAGS="${RUSTFLAGS:+$RUSTFLAGS }-C target-feature=-crt-static"
+        export RUSTFLAGS="--cfg tokio_unstable -C target-feature=-crt-static"
 
         # For cargo-zigbuild, set up additional environment variables
         if command -v cargo-zigbuild &> /dev/null; then
@@ -434,7 +434,7 @@ build_binary() {
         fi
     else
         # Native compilation
-        build_cmd="RUSTFLAGS='${RUSTFLAGS:+$RUSTFLAGS }-Clink-arg=-lm' cargo build"
+        build_cmd="RUSTFLAGS='--cfg tokio_unstable -Clink-arg=-lm' cargo build"
     fi
 
     if [ "$BUILD_TYPE" = "release" ]; then

crates/AGENTS.md

@@ -19,9 +19,7 @@ Applies to all paths under `crates/`.
 
 - Document lock acquisition order when a module uses multiple locks. Never acquire the same set of locks in different orders across code paths.
 - Never hold a `tokio::sync::RwLock`/`Mutex` write guard across `.await` points unless the critical section is unavoidably async and the hold time is bounded.
-- Prefer direct atomic `fetch_*` operations for unconditional updates and
-  `compare_exchange` loops only for conditional updates such as peaks or
-  adaptive state.
+- Prefer `compare_exchange` loops over load-then-store for concurrent counters (peak values, adaptive heuristics).
 - When resetting multi-field atomic statistics, use a version/sequence counter or accept that concurrent readers may see partial snapshots; document the tradeoff.
 - `std::sync::Mutex` is acceptable in async context only when held for a brief, non-`await`-containing critical section. If in doubt, use `tokio::sync::Mutex`.
 
@@ -42,9 +40,7 @@ Applies to all paths under `crates/`.
 - Keep unit tests close to the module they test.
 - Keep integration tests under each crate's `tests/` directory.
 - Add regression tests for bug fixes and behavior changes.
-- Every test needs an observable failure criterion. Direct assertions,
-  delegated assertions, snapshots/properties, `#[should_panic]`, and meaningful
-  `Result` failures are all valid; a call that can silently succeed is not.
+- Every test function must contain at least one `assert!`/`assert_eq!`/`assert_matches!`. A test that only calls code without asserting is not a test.
 - In tests, prefer `.expect("context: what was being tested")` over bare `.unwrap()`. A test failure should tell you which operation failed and with what input.
 
 ## Async and Performance

crates/audit/AGENTS.md

@@ -50,3 +50,4 @@ crate.
   - `cargo test -p rustfs-audit`
   - Focused: `cargo test -p rustfs-audit --test pipeline_layer_test`
   - Focused: `cargo test -p rustfs-audit pipeline`
+- Full gate before commit: `make pre-commit`

crates/audit/Cargo.toml

@@ -25,45 +25,19 @@ documentation = "https://docs.rs/rustfs-audit/latest/rustfs_audit/"
 keywords = ["audit", "target", "management", "fan-out", "RustFS"]
 categories = ["web-programming", "development-tools", "asynchronous", "api-bindings"]
 
-[features]
-default = []
-hotpath = [
-    "hotpath/hotpath",
-    "hotpath/tokio",
-    "hotpath/futures",
-    "rustfs-config/hotpath",
-    "rustfs-s3-types/hotpath",
-    "rustfs-targets/hotpath",
-]
-hotpath-alloc = [
-    "hotpath",
-    "hotpath/hotpath-alloc",
-    "rustfs-config/hotpath-alloc",
-    "rustfs-s3-types/hotpath-alloc",
-    "rustfs-targets/hotpath-alloc",
-]
-hotpath-cpu = [
-    "hotpath",
-    "hotpath/hotpath-cpu",
-    "rustfs-config/hotpath-cpu",
-    "rustfs-s3-types/hotpath-cpu",
-    "rustfs-targets/hotpath-cpu",
-]
-
 [dependencies]
-hotpath.workspace = true
 rustfs-targets = { workspace = true }
-rustfs-config = { workspace = true, features = ["audit", "server-config-model"] }
+rustfs-config = { workspace = true, features = ["audit", "constants", "server-config-model"] }
 rustfs-s3-types = { workspace = true }
-const-str = { workspace = true, features = ["std", "proc"] }
+chrono = { workspace = true }
+const-str = { workspace = true }
 futures = { workspace = true }
-hashbrown = { workspace = true, features = ["serde", "rayon"] }
-jiff = { workspace = true, features = ["serde"] }
+hashbrown = { workspace = true }
 metrics = { workspace = true }
-serde = { workspace = true, features = ["derive"] }
-serde_json = { workspace = true, features = ["raw_value"] }
+serde = { workspace = true }
+serde_json = { workspace = true }
 thiserror = { workspace = true }
-tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread", "time", "macros"] }
+tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread", "rt", "time", "macros"] }
 tracing = { workspace = true, features = ["std", "attributes"] }
 
 [dev-dependencies]

crates/audit/src/entity.rs

@@ -12,8 +12,8 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
+use chrono::{DateTime, Utc};
 use hashbrown::HashMap;
-use jiff::Timestamp;
 use rustfs_s3_types::EventName;
 use serde::{Deserialize, Serialize};
 use serde_json::Value;
@@ -151,8 +151,8 @@ pub struct AuditEntry {
     pub deployment_id: Option<String>,
     #[serde(rename = "siteName", skip_serializing_if = "Option::is_none")]
     pub site_name: Option<String>,
-    #[serde(with = "jiff::fmt::serde::timestamp::millisecond::required")]
-    pub time: Timestamp,
+    #[serde(with = "chrono::serde::ts_milliseconds")]
+    pub time: DateTime<Utc>,
     pub event: EventName,
     #[serde(rename = "type", skip_serializing_if = "Option::is_none")]
     pub entry_type: Option<String>,
@@ -198,7 +198,7 @@ impl AuditEntryBuilder {
     pub fn new(version: impl Into<String>, event: EventName, trigger: impl Into<String>, api: ApiDetails) -> Self {
         Self(AuditEntry {
             version: version.into(),
-            time: Timestamp::now(),
+            time: Utc::now(),
             event,
             trigger: trigger.into(),
             api,
@@ -232,7 +232,7 @@ impl AuditEntryBuilder {
         self
     }
 
-    pub fn time(mut self, time: Timestamp) -> Self {
+    pub fn time(mut self, time: DateTime<Utc>) -> Self {
         self.0.time = time;
         self
     }
@@ -342,23 +342,4 @@ mod tests {
         assert_eq!(value["requestID"], Value::String("req-audit-123".to_string()));
         assert!(value.get("request_id").is_none(), "historical audit contract must not expose request_id");
     }
-
-    #[test]
-    fn audit_entry_time_serializes_as_epoch_milliseconds() {
-        let entry = AuditEntryBuilder::new(
-            "1",
-            EventName::ObjectCreatedPut,
-            "s3",
-            ApiDetailsBuilder::new()
-                .name("PutObject")
-                .status("OK")
-                .status_code(200)
-                .build(),
-        )
-        .time(Timestamp::from_millisecond(1_711_423_698_870).expect("timestamp should be valid"))
-        .build();
-
-        let value = serde_json::to_value(entry).expect("audit entry should serialize");
-        assert_eq!(value["time"], Value::Number(1_711_423_698_870_i64.into()));
-    }
 }

crates/audit/src/pipeline.rs

@@ -32,7 +32,6 @@ const EVENT_AUDIT_BATCH_DISPATCH_COMPLETED: &str = "audit_batch_dispatch_complet
 const EVENT_AUDIT_TARGET_STATE_CHANGED: &str = "audit_target_state_changed";
 const EVENT_AUDIT_REPLAY_DELIVERED: &str = "audit_replay_delivered";
 const EVENT_AUDIT_REPLAY_RETRY_SCHEDULED: &str = "audit_replay_retry_scheduled";
-const EVENT_AUDIT_REPLAY_RETRY_EXHAUSTED: &str = "audit_replay_retry_exhausted";
 const EVENT_AUDIT_REPLAY_DROPPED: &str = "audit_replay_dropped";
 const EVENT_AUDIT_REPLAY_STREAM_STATUS: &str = "audit_replay_stream_status";
 
@@ -282,7 +281,6 @@ impl AuditPipeline {
                 let delivery = target.delivery_snapshot();
                 AuditTargetMetricSnapshot {
                     failed_messages: delivery.failed_messages,
-                    failed_store_length: delivery.failed_store_length,
                     queue_length: delivery.queue_length,
                     target_id: target.id().to_string(),
                     total_messages: delivery.total_messages,
@@ -292,8 +290,8 @@ impl AuditPipeline {
     }
 
     pub async fn snapshot_target_health(&self) -> Vec<rustfs_targets::RuntimeTargetHealthSnapshot> {
-        let targets = self.registry.lock().await.list_target_values();
-        rustfs_targets::health_snapshots_for_targets(targets).await
+        let registry = self.registry.lock().await;
+        registry.runtime_manager().health_snapshots().await
     }
 }
 
@@ -465,16 +463,18 @@ impl AuditRuntimeFacade {
                             target.record_final_failure();
                             observability::record_target_failure();
                         }
-                        ReplayEvent::RetryExhausted { detail, key, target } => {
+                        ReplayEvent::RetryExhausted { key, target } => {
                             warn!(
-                                event = EVENT_AUDIT_REPLAY_RETRY_EXHAUSTED,
+                                event = EVENT_AUDIT_REPLAY_DROPPED,
                                 component = LOG_COMPONENT_AUDIT,
                                 subsystem = LOG_SUBSYSTEM_PIPELINE,
                                 target_id = %target.id(),
                                 replay_key = %key,
-                                error = %detail,
-                                "audit replay retry budget exhausted, entry stays queued and retries"
+                                reason = "retry_exhausted",
+                                "audit replay delivery"
                             );
+                            target.record_final_failure();
+                            observability::record_target_failure();
                         }
                         ReplayEvent::UnreadableEntry { key, error, target } => {
                             warn!(
@@ -570,7 +570,7 @@ mod tests {
     use rustfs_targets::target::{EntityTarget, QueuedPayload, QueuedPayloadMeta};
     use rustfs_targets::{StoreError, Target, TargetError};
     use std::sync::Arc;
-    use tokio::sync::{Mutex, Notify};
+    use tokio::sync::Mutex;
 
     /// Mock target whose `save()` outcome is fixed at construction so tests can
     /// force full-success / full-failure / partial-failure fan-outs.
@@ -578,7 +578,6 @@ mod tests {
     struct MockTarget {
         id: TargetID,
         fail: bool,
-        health_gate: Option<(Arc<Notify>, Arc<Notify>)>,
     }
 
     impl MockTarget {
@@ -586,14 +585,8 @@ mod tests {
             Self {
                 id: TargetID::new(id.to_string(), "webhook".to_string()),
                 fail,
-                health_gate: None,
             }
         }
-
-        fn with_health_gate(mut self, started: Arc<Notify>, release: Arc<Notify>) -> Self {
-            self.health_gate = Some((started, release));
-            self
-        }
     }
 
     #[async_trait]
@@ -606,10 +599,6 @@ mod tests {
         }
 
         async fn is_active(&self) -> Result<bool, TargetError> {
-            if let Some((started, release)) = &self.health_gate {
-                started.notify_one();
-                release.notified().await;
-            }
             Ok(true)
         }
 
@@ -684,24 +673,6 @@ mod tests {
         pipeline.dispatch(entry()).await.expect("no targets should return Ok");
     }
 
-    #[tokio::test]
-    async fn health_probe_does_not_hold_the_registry_lock() {
-        let started = Arc::new(Notify::new());
-        let release = Arc::new(Notify::new());
-        let pipeline = pipeline_with(vec![MockTarget::new("blocked", false).with_health_gate(started.clone(), release.clone())]);
-        let registry = Arc::clone(&pipeline.registry);
-        let snapshot_task = tokio::spawn(async move { pipeline.snapshot_target_health().await });
-        started.notified().await;
-
-        let guard = tokio::time::timeout(std::time::Duration::from_secs(1), registry.lock())
-            .await
-            .expect("network health probe must not retain the audit registry lock");
-        drop(guard);
-        release.notify_one();
-
-        assert_eq!(snapshot_task.await.expect("snapshot task should finish").len(), 1);
-    }
-
     // backlog#962: dispatch_batch must mirror dispatch and propagate a
     // whole-batch loss instead of returning Ok.
     #[tokio::test]

crates/audit/src/system.rs

@@ -30,7 +30,6 @@ const EVENT_AUDIT_CONFIG_RELOADED: &str = "audit_config_reloaded";
 #[derive(Debug, Clone, Default, PartialEq, Eq)]
 pub struct AuditTargetMetricSnapshot {
     pub failed_messages: u64,
-    pub failed_store_length: u64,
     pub queue_length: u64,
     pub target_id: String,
     pub total_messages: u64,

crates/audit/tests/performance_test.rs

@@ -97,7 +97,7 @@ async fn test_audit_log_dispatch_performance() {
         return; // Alternatively: assert!(false, "AuditSystem failed to start");
     }
 
-    use jiff::Timestamp;
+    use chrono::Utc;
     use rustfs_targets::EventName;
     use serde_json::json;
     use std::collections::HashMap;
@@ -136,7 +136,7 @@ async fn test_audit_log_dispatch_performance() {
         version: "1".to_string(),
         deployment_id: Some(format!("test-deployment-{id}")),
         site_name: Some("test-site".to_string()),
-        time: Timestamp::now(),
+        time: Utc::now(),
         event: EventName::ObjectCreatedPut,
         entry_type: Some("object".to_string()),
         trigger: "api".to_string(),
@@ -298,7 +298,7 @@ fn test_performance_requirements() {
     for i in 0..3000 {
         // Simulate event name parsing and processing
         let _event_id = format!("s3:ObjectCreated:Put_{i}");
-        let _timestamp = jiff::Timestamp::now().to_string();
+        let _timestamp = chrono::Utc::now().to_rfc3339();
 
         // Simulate basic audit entry creation overhead
         let _entry_size = 512; // bytes

crates/audit/tests/pipeline_layer_test.rs

@@ -236,19 +236,12 @@ async fn audit_pipeline_reports_empty_runtime_snapshots() {
 }
 
 #[tokio::test]
-async fn stopping_audit_replay_workers_is_a_no_op_when_there_are_none() {
+async fn audit_runtime_facade_stops_empty_replay_workers() {
     let registry = Arc::new(Mutex::new(AuditRegistry::new()));
     let replay_workers = Arc::new(RwLock::new(rustfs_targets::ReplayWorkerManager::new()));
-    let facade = AuditRuntimeFacade::new(registry, Arc::clone(&replay_workers));
+    let facade = AuditRuntimeFacade::new(registry, replay_workers);
 
     facade.stop_replay_workers().await;
-
-    // The stop path takes the manager's workers and hands them to the adapter,
-    // so an empty facade must leave it empty rather than wedge it, and a second
-    // call — which shutdown paths make — must stay harmless (rustfs/backlog#1836).
-    assert!(replay_workers.read().await.is_empty());
-    facade.stop_replay_workers().await;
-    assert!(replay_workers.read().await.is_empty());
 }
 
 #[tokio::test]

crates/audit/tests/system_integration_test.rs

@@ -264,7 +264,7 @@ fn create_sample_audit_entry() -> AuditEntry {
 }
 
 fn create_sample_audit_entry_with_id(id: u32) -> AuditEntry {
-    use jiff::Timestamp;
+    use chrono::Utc;
     use rustfs_targets::EventName;
     use serde_json::json;
 
@@ -301,7 +301,7 @@ fn create_sample_audit_entry_with_id(id: u32) -> AuditEntry {
         version: "1".to_string(),
         deployment_id: Some(format!("test-deployment-{id}")),
         site_name: Some("test-site".to_string()),
-        time: Timestamp::now(),
+        time: Utc::now(),
         event: EventName::ObjectCreatedPut,
         entry_type: Some("object".to_string()),
         trigger: "api".to_string(),