refactor(ecstore): migrate the bucket bandwidth monitor into InstanceContext (Phase 5 Slice 10) (#4489)

Phase 5 Slice 9 (backlog#939): move the lifecycle expiry state and transition
state — two stateful service handles defined in bucket_lifecycle_ops — into the
per-instance InstanceContext, following the lazy-materialization pattern from
Slices 7-8.

- InstanceContext gains `expiry_state: OnceLock<Arc<RwLock<ExpiryState>>>` and
  `transition_state: OnceLock<Arc<TransitionState>>`, each materialized lazily
  on first access (both constructors are cheap — empty worker vecs/channels,
  workers started later — so lazy init reproduces the eager static's
  create-once-then-share).
- The lifecycle owner helpers `get_global_expiry_state` / `get_global_transition_state`
  and the runtime-source handles `expiry_state_handle` / `transition_state_handle`
  keep their signatures and route through the current instance's context; the
  two statics (and the now-unused lazy_static import) are removed. The scanner
  boundary and lifecycle-state arch guards still pass.

Tests: expiry/transition state are stable within an instance (same Arc) and
distinct across instances.

Verification: cargo test -p rustfs-ecstore (20 instance-context tests green),
cargo clippy -p rustfs-ecstore --all-targets (clean), make pre-commit (pass).

Refs: backlog#939 (Phase 5, Slice 9).

.agents/skills/adversarial-validation/SKILL.md

@@ -1,266 +0,0 @@
----
-name: adversarial-validation
-description: Execute the Adversarial Validation policy from the root AGENTS.md — run the six reviewer roles (correctness, security, concurrency/durability, compatibility, performance, test coverage) with RustFS-specific attack probes. Use on every behavior-affecting code change, bug fix, or design proposal before declaring it done.
----
-
-# Adversarial Validation Playbooks
-
-The policy — risk tiers, role list, protocol, exit criteria — lives in the
-root `AGENTS.md` under "Adversarial Validation (Default On)". Read it first;
-this skill does not restate it. This file adds the RustFS-specific probe
-playbook for each role: concrete attacks, where they apply, and the real
-shipped bug or rule that earns each probe its place.
-
-## How to run a role
-
-1. Pick the tier and the applicable roles per the root `AGENTS.md`.
-2. Run each role as an independent pass over the final diff — a parallel
-   reviewer agent where the tooling supports it, otherwise a fresh
-   sequential pass that starts from the diff and the nearest scoped
-   `AGENTS.md`, discarding the writing session's assumptions.
-3. Within a role, execute the probes whose domain the diff touches, plus any
-   attack the diff obviously invites that no probe lists — the playbook is a
-   floor, not a ceiling.
-4. Report findings (concrete failure scenario or named missing test, with
-   file:line) or the role's null report: "attacked X, Y, Z — no break
-   found". A bare pass is not a result.
-
-## Role playbooks
-
-### Correctness adversary
-
-- For any change touching error aggregation or quorum decisions, build the exact disk-error slice at the quorum boundary: N disks where successes == quorum, then flip one success to an error (quorum-1) and separately inject None/nil placeholder entries into the slice. Trace whether reduce_errs (or the new equivalent) picks the placeholder as the dominant error or lets quorum-1 pass as success. Also check heal/write paths: does a per-target failure at quorum-1 return an explicit error, or silently degrade to success?
-  - Where: crates/ecstore/src/disk/error_reduce.rs; crates/ecstore/src/set_disk/{core,ops}; crates/heal
-  - Evidence: Commit 20d61c73b 'stop reduce_errs leaking nil placeholder as dominant error' (#4551) and 47c1e730c 'make erasure heal write quorum best-effort per target' (#4545); crates/ecstore/AGENTS.md: 'Do not weaken quorum checks... Prefer explicit failure over silent data corruption or implicit success.'
-- For any change in EC read/reconstruct/streaming code, trace the mid-stream error path: the first K shards read fine, then a shard turns out bitrot-corrupt or inconsistent after N bytes have already been sent to the client. Verify the error propagates as a stream error (client sees failure), not a clean end-of-body — a silently truncated GET body is data corruption. Also re-check byte accounting: sum of per-part bytes vs object size, and partNumber-to-offset routing for the first and last part.
-  - Where: crates/ecstore/src/set_disk/read.rs (reconstruct-read, inconsistent_source_indexes handling ~line 3827); codec streaming paths in crates/ecstore/src/erasure_coding
-  - Evidence: Known live bug: EC reconstruct-read failing on inconsistent shards mid-stream silently truncates GET body → client 'unexpected EOF'; commit 15808254d 'correct codec-streaming byte accounting and partNumber routing' (#4535).
-- For any listing/pagination change, construct the exact-boundary inputs: (a) exactly max_keys/max_uploads matching entries — assert the response contains max and is_truncated=false, then max+1 entries — assert exactly max returned with is_truncated=true and a correct continuation marker; (b) a delimiter listing where folding into CommonPrefixes re-fills a full page; (c) an object 'a' coexisting with prefix dir 'a/'. Off-by-one and dropped-truncation bugs live exactly at these boundaries.
-  - Where: crates/ecstore listing/merge paths (metacache, list_objects, list_multipart_uploads); rustfs/src/storage
-  - Evidence: Three recent real bugs: fefa70b31 'stop ListMultipartUploads from returning one upload past max-uploads' (#4447), d91f4d455 'report truncation when delimiter list re-folds a full page' (#4538), 7e1f7f242 'preserve CommonPrefixes when an object and same-named prefix dir coexist' (#4563).
-- Run the diff's logic with a directory object key (trailing slash, e.g. 'pre/dir/') as input. Check which layer encodes/decodes __XLDIR__ — set_disk never sees the trailing slash, so any trailing-slash branch added below the store layer is dead code and a wrong-layer bug. For delete paths, check whether options force a nil versionId onto directory keys: the resulting miss surfaces as version-not-found, not object-not-found, so callers matching only ObjectNotFound leak ghost directory entries.
-  - Where: crates/ecstore/src/store*.rs (store layer) vs crates/ecstore/src/set_disk/*; delete option construction (del_opts) and its callers
-  - Evidence: trailing-slash branches must live at store layer; del_opts injects nil version for dir keys, PR#4220 ghost-directory cleanup never fired on the real path (rustfs#4307, backlog#798 still OPEN).
-- Feed every new binary-UUID metadata read the three degenerate values: key absent, zero-length bytes, and 16 zero bytes (nil UUID). All three must mean 'no value' — any path that produces Uuid::nil() and then acts on it (e.g. sends it as a versionId) is a finding. For tier code specifically: with remote-tier version None or "", assert the tier GET/DELETE request carries no versionId parameter at all — sending versionId="" or nil yields NoSuchVersion against unversioned tier buckets.
-  - Where: crates/filemeta/src/filemeta/version.rs; crates/ecstore/src/bucket/lifecycle/; crates/ecstore/src/services/tier/; any new consumer of crates/utils/src/http/metadata_compat.rs
-  - Evidence: AGENTS.md Cross-Cutting Domain Invariants (defensive Uuid read pattern + unversioned-tier rule); docs/operations/tier-ilm-debugging.md ('nil transition_ver_id = corrupt legacy write-back, readers must filter'); commit 726f3dc18 'accept empty remote version_id in tier recovery paths' (#4552).
-- For each match/if-let on an error or algorithm enum touched by the diff, enumerate what the wildcard/else arm swallows. Inject the variants the author didn't think of — DiskNotFound during listing, an unsupported checksum algorithm, an Err from a cleanup rename/delete — and trace whether they degrade into 'not found', a wrong-but-plausible value, or silent success. Any error path that converges with the success path without logging and propagating is a finding.
-  - Where: crates/ecstore (listing, delete/rename cleanup); crates/checksums; error-mapping layers in rustfs/src/storage
-  - Evidence: Three recent real bugs of this exact shape: e0619e355 'stop treating DiskNotFound as object not-found in listing' (#4536), afaf8c681 'reject md5 instead of silently returning crc32' (#4513), f7d2b2563 'propagate disk delete/rename failures instead of swallowing them' (#4546).
-- For any change to version ordering, index lookup, or shard/part indexing: (a) call the accessor with index == len() and len()-1 — a get_idx-style bound must reject, not panic or wrap; (b) construct two versions with identical mod-times and check the sort tie-break is total and deterministic (equal keys must not compare as both before each other); (c) for EC shard math, compute shard size for object sizes 0, 1, blockSize-1, blockSize, blockSize+1 and cross-check total reconstructed length against the object size.
-  - Where: crates/filemeta/src/filemeta/*.rs (version sort, get_idx); crates/ecstore/src/erasure_coding shard-size math
-  - Evidence: Commit 8bfb00bc0 'guard get_idx bound and fix sorts_before tie-break' (#4509) — both bug classes shipped before; ecstore AGENTS.md high-risk designation for read/write/repair correctness.
-- Exercise the zero/empty end of every new size or count parameter: zero-length object PUT then GET (body must be empty, not error), part count 0, empty Vec of disks/entries into aggregation functions, and env/config values of 0 (must clamp or reject, never divide-by-zero or 'scan nothing and report zero usage'). Anywhere the diff computes a ratio, capacity, or progress percentage, plug in 0 and the max value.
-  - Where: crates/ecstore aggregation and scanner paths; crates/object-capacity; config/env parsing in touched crates
-  - Evidence: Commits 787cc77a7 'clamp zero capacity env values to safe defaults' (#4559) and 32b1094ec 'resolve a symlinked scan root instead of silently counting zero' (#4564) — zero-as-silent-wrong-answer is a recurring repo bug class.
-- Smaller-diff attack: rewrite the diff's change mentally (or actually, in scratch) as the minimal in-place edit and compare. Flag as findings: a helper function with exactly one caller introduced by this diff; a file rewrite where a 3-line edit inside the existing control flow suffices; reshaped control flow in init/locking/metadata/quorum paths beyond what the fix requires; new string literals duplicating existing constants (grep the token first); #[path] module inclusion. If the smaller diff achieves identical behavior, report it with the concrete replacement.
-  - Where: Any diff; extra scrutiny for crates/ecstore, crates/lock, rustfs/src/storage where 'preserve the existing control-flow shape' is an explicit rule
-  - Evidence: AGENTS.md 'Change Style for Existing Logic' (one-off helper ban, preserve control-flow shape in distributed/locking/metadata paths, no #[path]) and 'Constant and String Usage'; Adversarial Validation section names the smaller-diff clause as a correctness-adversary finding.
-- For any diff touching multipart or object commit paths, order the operations on paper and attack the failure point between them: kill the process (or return Err) after the commit rename but before cleanup, and after cleanup but before commit. Verify the earlier-failure case leaves the object readable and the later-failure case leaves no half-visible object; part meta files must never be deleted before the commit is durable.
-  - Where: crates/ecstore multipart commit/cleanup (set_disk/ops); rustfs/src/storage multipart handlers
-  - Evidence: Commit c77c5f047 'defer multipart part.N.meta cleanup until after commit' (#4548) — cleanup-before-commit ordering already caused a real data-loss window; the #4221 durability work shows fsync/ordering bugs are endemic here.
-
-Null report example: "Attacked quorum-1 error reduction, exact max-keys listing boundary, trailing-slash dir keys, nil-UUID tier versionId, mid-stream reconstruct error propagation, and a minimal-diff rewrite — no break found; diff is already the minimal in-place edit."
-
-### Security reviewer
-
-- For every admin handler in the diff, grep the exact AdminAction constant it passes to validate_admin_request and confirm it names the operation the handler actually performs. Construct the escalation: a low-privileged user whose policy grants the wrong-but-adjacent action (e.g. Export while the handler Imports, or Update while it Lists) — if the mismatched constant lets them through, that is the bug. Also confirm read-only endpoints (metrics, list, server-info, diagnostics) still call an operation-specific admin authz path and not a mere 'credentials exist' check.
-  - Where: rustfs/src/admin/handlers/**, rustfs/src/admin/router registration; check_permissions / validate_admin_request / AdminAction::* call sites
-  - Evidence: GHSA-vcwh-pff9-64cc (ImportIam checked ExportIAMAction), GHSA-mm2q-qcmx-gw4w (ListServiceAccount used UpdateServiceAccountAdminAction), GHSA-f5cv-v44x-2xgf (/admin/v3/metrics accepted any authenticated IAM user). advisory-patterns.md 'Admin authorization and route exposure'; rustfs/src/admin/AGENTS.md 'route registration, whitelist, handler authz must agree'.
-- If the diff touches service-account or IAM import/update, treat parent, claims, accessKey, secretKey, status, policy names, and groups as attacker-controlled. Construct an ImportIam/create payload where parent points at root (or another user) and prove the code writes credentials without proving caller ownership or root authority. Separately, set deny_only=true (or 'no explicit deny') on a restricted account and check it does not skip the required allow check, letting it mint an unrestricted child.
-  - Where: crates/iam/, rustfs/src/admin/handlers (service account / import IAM), rustfs/src/auth.rs
-  - Evidence: GHSA-566f-q62r-wcr8 (attacker-controlled parent/claims/accessKey/secretKey → persistent backdoor under root), GHSA-xgr5-qc6w-vcg9 (deny_only=true skipped allow checks → privilege creation). crates/iam/AGENTS.md security boundaries; advisory-patterns.md 'IAM import, service accounts'.
-- For a changed protocol-frontend handler (FTP/FTPS/SFTP/WebDAV/gateway), enumerate ALL sibling command handlers in the same driver — not just the changed one. For each, confirm it calls the per-operation IAM authorize hook mapped to the correct S3 action (RETR→GetObject, SIZE/MDTM→HeadObject, MKD→CreateBucket, bucket probe→ListBucket/HeadBucket) BEFORE touching storage. Construct a denied-authz case and prove the storage backend is never reached.
-  - Where: crates/protocols/ (FtpsDriver, SftpDriver, WebDAV), authorize_operation call sites
-  - Evidence: GHSA-3g29-xff2-92vp (FTP RETR/SIZE/MDTM authenticated but skipped IAM), GHSA-g3vq-vv42-f647 (FTPS MKD called create_bucket without s3:CreateBucket). advisory-patterns.md: 'RustFS advisories show mixed guarded and unguarded siblings in the same driver.'
-- For any secret/token/signature/password comparison in the diff, check it uses a constant-time compare (e.g. subtle/constant_time_eq), not == or early-return byte loops. Then check the failure-response paths: construct an invalid-user request and an invalid-secret request and confirm they are indistinguishable (same error, no early length short-circuit) so an attacker cannot enumerate valid users or time-side-channel the secret.
-  - Where: crates/protocols/ (FTPS/WebDAV/FormPost auth), crates/credentials/, rustfs/src/auth.rs, RPC signature verification
-  - Evidence: GHSA-3p3x-734c-h5vx (FTPS/WebDAV early-return string equality + distinguishable invalid-user vs invalid-password). Fix commits 3c3113619 (constant-time FTPS/WebDAV) and c41062f27 (constant-time FormPost signature). 3p3x was fixed by PR #4403.
-- If the diff touches internode/RPC auth secret handling, trace whether the RPC HMAC secret can fall back to a public default (e.g. 'rustfsadmin', 'rustfs rpc') or be derived deterministically from the S3 root credentials. Construct the case where RUSTFS_RPC_SECRET is unset and confirm the code fails closed rather than silently using a default or a root-derived key. Verify RPC signing keys are independent random secrets, not reused across S3-root/RPC-HMAC/STS-JWT roles.
-  - Where: crates/credentials/, crates/ecstore/src/rpc/, internode auth setup
-  - Evidence: GHSA-r5qv-rc46-hv8q (fell back to 'rustfsadmin'), GHSA-75fx/68cw (RPC secret derivable from root creds → forgeable signatures), GHSA-h956 (hard-coded 'rustfs rpc'), GHSA-m77q (STS JWT reused root secret). Fix commit 7b2055405 (fail closed when deriving RPC secret from default credentials, PR#4402).
-- If the diff touches RPC/NodeService authentication, verify the HMAC payload binds the EXACT concrete gRPC method path (not a service prefix), the HTTP method surrogate, and a fresh timestamp. Construct captured valid metadata for method A and replay it to method B within the timestamp window — if it authorizes, the signature is under-bound. Also test stale timestamp, wrong path, wrong method, wrong secret.
-  - Where: crates/ecstore/src/rpc/, verify_rpc_signature / NodeServiceServer, x-rustfs-signature handling
-  - Evidence: GHSA-c667-rgrv-99vj (signed service prefix instead of concrete method path → cross-method replay in timestamp window). advisory-patterns.md 'RPC input validation and panic safety'; Minimum Regression Test Expectations lists replay across two methods.
-- For any RPC/gRPC handler or deserialization touched, feed empty bytes, truncated MessagePack/protobuf, invalid enum discriminants, and stale timestamps. Grep the deserialization path for unwrap()/expect()/panic-prone decode and prove malformed attacker payloads return a typed error, not a panic (remote DoS). Weak internode auth makes reachability worse, so combine with the RPC-secret probe.
-  - Where: crates/ecstore/src/rpc/, any #[derive(Deserialize)] decoded from wire bytes, RPC handler bodies
-  - Evidence: GHSA-gw2x-q739-qhcr (malformed GetMetrics reached unwrap() → remote DoS). advisory-patterns.md 'Treat all RPC payload bytes as attacker-controlled.' rust-code-quality skill: unwrap abuse.
-- Take any object key, RPC disk path, or archive/tar/zip entry name introduced or handled in the diff and construct traversal payloads: '../', URL-encoded '%2e%2e%2f', absolute paths, platform separators, empty components. Trace the value through parse → authz check → final storage path and prove (a) authz and storage normalize the SAME way, and (b) the canonicalized path cannot escape the bucket/prefix root. Attack the case where authz sees the raw attacker bucket but storage cleaning crosses into a victim bucket.
-  - Where: crates/ecstore (path join/canonicalize), rustfs/src/storage/, Snowball auto-extract / normalize_extract_entry_key, rpc read_file_stream
-  - Evidence: GHSA-pq29-69jg-9mxc (read_file_stream joined untrusted paths, no canonical boundary check), GHSA-8r6f-hmq2-28rg (traversal object keys bypassed authz), GHSA-f4vq-9ffr-m8m3 (Snowball '../victim-bucket/object' authorized raw path then storage crossed boundary). Note __XLDIR__ trailing-slash encoding is store-layer only.
-- If the diff touches multipart/copy or presigned POST, verify UploadPartCopy enforces source GetObject AND destination PutObject semantics equivalent to CopyObject, including copy-source policy conditions (not just independent source-read + dest-write). Construct a cross-bucket UploadPartCopy from a bucket the caller cannot read. For presigned POST, submit an upload that violates content-length-range, key prefix, or exact content-type and prove the server rejects it.
-  - Where: rustfs/src/storage / S3 API handlers: upload_part_copy, CompleteMultipartUpload, PostObject policy enforcement
-  - Evidence: GHSA-mx42-j6wv-px98 (UploadPartCopy missed source authz → cross-bucket exfil), GHSA-wfxj-ph3v-7mjf (missed destination copy-source policy constraint), GHSA-w5fh-f8xh-5x3p (presigned POST didn't enforce signed policy conditions). advisory-patterns.md 'S3 copy, multipart, and upload policy validation'.
-- Grep the diff for debug!/trace!/info!/error! and any ?value / {:?} on structs or response bodies that can carry secret_key, session_token, JWT claims, HMAC secrets, expected signatures, access keys beyond safe identifiers, or raw credential-bearing responses. Check custom Debug impls and merged-config dumps too. Construct the error path (invalid signature, failed auth) and confirm it does not log the secret or the derived authenticator. Verify audit/notify entries redact credential request headers.
-  - Where: rustfs/src/**, crates/iam/, crates/audit/, crates/notify/, crates/targets/, RPC signature error paths
-  - Evidence: GHSA-r54g (STS creds logged at info), GHSA-8cm2 (debug logs leaked tokens/secrets/JWT claims/raw STS bodies), GHSA-333v (invalid RPC signature log included HMAC secret + expected signature). Fix commit ee6f79110 (redact credential request headers from audit/notify, backlog#963). rustfs-logging-governance skill.
-- For any struct in the diff deserialized from untrusted input (S3 XML/JSON, lifecycle rules, bucket policy, replication config, RPC payload), check for #[serde(deny_unknown_fields)]. Construct a payload with a typo'd field (e.g. 'NoncurentDays') or an extra field and prove it is rejected, not silently ignored. Flag #[serde(default)] on security-critical fields (retention days, limits, permissions) lacking explicit post-deserialize validation, and any user-controlled integer cast with `as` (i32 as u32) — feed a negative value and check it doesn't wrap to a huge positive.
-  - Where: crates/policy/ (bucket policy), crates/ecstore lifecycle/ILM config, replication config structs, crates/protocols XML parsing
-  - Evidence: AGENTS.md 'Serde Safety'; advisory-patterns.md 'Serde deserialization' (no deny_unknown_fields found repo-wide; 'NoncurentDays' typo silently accepted; i32 as u32 wrap). Fix commit 1acd47f15 (SSE crash-loop DoS + credential reserved-char bypass, backlog#806).
-- If the diff touches SSE / encryption reader-writer composition, do not trust API metadata claiming encryption. Trace the reader wrapper order (HashReader / EncryptReader / compression / warp) and confirm EncryptReader is actually in the chain that writes to disk — construct the case where a helper unwraps a nested reader and bypasses encryption, storing plaintext. Require a regression test that inspects the ACTUAL stored bytes on disk, not just read-back. Also check encrypted-object checksums are not exposed.
-  - Where: rustfs/src/storage/ecfs.rs, crates/rio/ (reader wrappers), crates/kms/, SSE-C replication
-  - Evidence: GHSA-xrrf-67jm-3c2r (SSE metadata reported encryption while composition bypassed EncryptReader, stored plaintext). Fix commits a7b9659e7 (hide encrypted object checksums, #4529), 80cc3b1fc (preserve SSE-C checksum state, #4410). advisory-patterns.md 'SSE and on-disk storage invariants'.
-- If the diff touches CORS or the console/browser/object-preview surface: confirm default CORS does not reflect an arbitrary Origin while also sending Access-Control-Allow-Credentials: true — construct a request with a spoofed Origin and check the response. For preview, confirm attacker-controlled object content is origin-isolated (not rendered in a same-origin iframe with console creds), served with nosniff/CSP, and that preview trust derives from validated content-type + sandboxing, NOT from object name/extension (.pdf, .html). Separately, if aws:SourceIp is evaluated, spoof X-Forwarded-For / X-Real-IP as a direct (non-trusted-proxy) client and confirm the socket peer IP is used instead.
-  - Where: rustfs/src/server/layer.rs (CORS), console preview/auth code, aws:SourceIp / policy condition evaluation, X-Forwarded-For handling
-  - Evidence: GHSA-x5xv-223c-8vm7 (default CORS reflected arbitrary origins with credentials), GHSA-v9fg-3cr2-277j (preview rendered attacker HTML same-origin, exposed localStorage creds), GHSA-7gcx-wg4x-q9x6 (extension-based PDF detection bypassed sandbox), GHSA-fc6g-2gcp-2qrq (aws:SourceIp trusted client XFF). advisory-patterns.md 'Browser, CORS' + 'Trusted proxy'.
-
-Null report example: "Attacked admin action-constant matching in the two changed handlers (both call validate_admin_request with the exact AdminAction), the FTPS RETR/SIZE authz parity, and the new lifecycle struct's serde surface (has deny_unknown_fields) — no break found."
-
-### Concurrency/durability reviewer
-
-- For every new or moved lock acquisition, enumerate all other code paths that take any overlapping subset of those locks and construct the concrete ABBA interleaving (thread 1 holds A wants B, thread 2 holds B wants A). If the diff acquires 2+ locks without a comment documenting acquisition order, that alone is a finding.
-  - Where: crates/ecstore/** (namespace locks, set_disk, disk registry), crates/audit/**, crates/lock/**, any Mutex/RwLock pair in a diff
-  - Evidence: crates/ecstore/AGENTS.md 'Lock Ordering' (document order; same set in different orders = deadlock); real ABBA deadlock fixed in c0d5f938f (#4421, audit registry vs stream_cancellers)
-- If the diff touches the object write/commit path or a lock guard's lifetime, construct the timeline where the distributed lock is lost (heartbeat refresh fails / expiry) after shard writes but before the xl.meta rename commit — verify the commit is fenced on guard.is_lock_lost() (set_disk/ops/object.rs:874-880) and the diff does not move the commit outside the fenced region or drop the guard early.
-  - Where: crates/ecstore/src/set_disk/ops/object.rs, ops/multipart.rs, crates/lock/**
-  - Evidence: 1e6207c08 (#4406) fence write commit on lock loss; ddf197ba5 (#4388) heartbeat lock refresh; backlog#899 fencing comment in object.rs
-- For any change to file creation or write-then-rename: write out the exact syscall order (write tmp -> fdatasync tmp -> rename -> fsync parent dir -> fsync ancestor dirs on first object under a prefix) and simulate a power cut after each step. Flag any dropped/reordered sync, and check the change honors the durability gate (RUSTFS_DRIVE_SYNC_ENABLE, strict/relaxed/none modes, per-bucket overrides) instead of hardcoding one mode. The 'skip tmp parent fsync' optimization is only sound when the file is renamed out of tmp — verify that precondition still holds.
-  - Where: crates/ecstore/src/disk/local.rs, disk/os.rs, disk/fs.rs, crates/ecstore/src/bucket/durability.rs, set_disk/core/io_primitives.rs
-  - Evidence: PR #4221 (the repo previously had no fsync anywhere); 2df315baf/c081586e7 (#4493) fsync ancestor dirs; 062a68d15 (#4387) tmp-parent-fsync skip is rename-conditional; eaff17cad (#4397) durability modes; 54872d52d (#4478) rename_data crash harness exists — extend it for the diff
-- If the diff touches quorum counting or per-disk error aggregation, construct adversarial error vectors for reduce_errs: nil/placeholder entries, DiskNotFound mixed with FileNotFound, exactly quorum-1 agreeing errors — and show which dominant error wins. Specifically attack the case where offline-disk errors get counted as 'object does not exist', flipping a read/heal decision into data loss. Also check quorum monotonicity: a retry or heal pass must never conclude with a LOWER quorum than the original write.
-  - Where: crates/ecstore/src/disk/error_reduce.rs, crates/ecstore/src/api/mod.rs, set_disk read/heal paths
-  - Evidence: 20d61c73b (#4551) reduce_errs leaked nil placeholder as dominant error; e0619e355 (#4536) DiskNotFound treated as object-not-found in listing; quorum monotonicity flagged as open follow-up to #4221 (#4221 follow-up list); crates/ecstore/AGENTS.md 'Do not weaken quorum checks'
-- For any multi-disk fan-out (delete, rename, heal write, cleanup), trace each per-disk Result: find any `let _ =`, `.ok()`, or best-effort collapse that keeps a failed disk out of the quorum math. Construct the run where exactly write_quorum-1 disks succeed and prove the op still returns success — that is the bug. Conversely, for heal writes, check one bad target cannot fail the whole heal (best-effort per target).
-  - Where: crates/ecstore/src/set_disk/ops/*.rs, disk/disk_store.rs, crates/heal/**
-  - Evidence: f7d2b2563 (#4546) disk delete/rename failures were swallowed; 47c1e730c (#4545) heal write quorum made best-effort per target; 2b063b0c4 (#4400) disk-replacement heal missed versions
-- For every new .await placed between a state mutation and its cleanup/commit (or inside select!/timeout/spawned task that can be aborted), construct the cancellation point: client disconnects and the future is dropped exactly there. Enumerate what is left behind — tmp files, incremented counters never decremented, half-written xl.meta, a held permit/waiter — and verify cleanup runs in Drop or the state is re-entrant. Background loops the diff adds must have a hard outer timeout so a wedged awaitee cannot pin them forever.
-  - Where: crates/ecstore/** write paths, crates/object-capacity/** scanners, crates/audit/**, anything using tokio::select! or spawn+abort
-  - Evidence: d608e320f io_uring cancel-safety spike (cancellation known-hard here); 5a372557e (#4533) wedged scans needed hard outer timeout; 7b87d4d13 (#4520) waiter-count leak; e44bece00 (#4497) audit start race + paused drops
-- If the diff touches metacache/list producers or cursor resume, construct the interleaving where the producer task completes (or errors) while a reader with a saved cursor comes back for the next page — verify a completed producer is tolerated (no error, no hang) and that a re-folded/full page reports truncation instead of silently ending the listing early. Also feed a corrupt/oversized length prefix into any metacache decode the diff touches.
-  - Where: crates/ecstore/src/cache_value/metacache_set.rs, crates/ecstore/src/store/list_objects.rs, crates/filemeta/**
-  - Evidence: 91a23361e (#4531) tolerate completed metacache producers; d91f4d455 (#4538) delimiter re-fold dropped truncation flag; d2c100fd3 (#4226) corrupt length-prefix guard
-- For multipart changes, construct concurrent operations on the SAME uploadId: put_object_part racing put_object_part (same part number), abort racing complete between the parts listing and the commit rename, and list-parts racing cleanup. Verify every metadata read/list/abort holds the per-uploadId lock, and that part.N.meta cleanup is deferred until AFTER the commit rename — cleanup before commit loses parts on a crash between the two.
-  - Where: crates/ecstore/src/set_disk/ops/multipart.rs, rustfs/src/storage/
-  - Evidence: 3bc8d79fe (#4329) serialize put_object_part per uploadId; 7fb95d4fc (#4428) unlocked upload metadata reads/aborts; 93ffbdb9b (#4437) unlocked part listings; c77c5f047 (#4548) part.N.meta cleanup moved after commit
-- Any cleanup/rollback logic added near a commit: verify it runs strictly AFTER the commit is durable, is best-effort (its failure must not fail an already-committed write), and is safe under retry — i.e., re-running it after a partial first attempt must never delete the newly-committed data dir or the last surviving copy.
-  - Where: crates/ecstore/src/set_disk/ops/object.rs (rename_data tail), ops/multipart.rs, disk cleanup helpers
-  - Evidence: afc7f1d6f/d908243e6 (#4386, backlog#898) post-commit old-data-dir cleanup had to be made best-effort; e7cc719c1 (#4389) speculative tmp cleanup moved off hot path
-- If the diff does read-modify-write on any persisted shared state (bucket metadata, notify/target config, queue store), construct two concurrent writers: show whether the second write silently discards the first (lost update) — RMW must be serialized or CAS-guarded. For persisted queues/replay, construct crash-mid-replay and prove entries are neither lost nor delivered twice without an idempotency key.
-  - Where: crates/notify/**, crates/targets/** (queue store, SQL backends), crates/ecstore/src/bucket/metadata_sys.rs
-  - Evidence: 2490d4ee2 (#4425) persisted config RMW lost updates; 08e44b95f (#4505) queue store crash-safety + replay lifecycle; e008cc5da (#4500) SQL backend idempotency
-- If the diff touches erasure decode/reconstruct or streaming GET, construct the failure mid-stream: shards become inconsistent (or a disk read fails) after N bytes of the body have already been sent — verify the stream surfaces an error to the client instead of ending cleanly at a truncated length. Silent truncation on a 200 response is the known failure mode.
-  - Where: crates/ecstore/src/set_disk/read.rs (reconstruct-read validation, historically ~line 3117), crates/ecstore/src/erasure/coding/decode.rs
-  - Evidence: Known open bug: EC reconstruct-read 'inconsistent shards' mid-GET silently truncates body -> client unexpected EOF; crates/ecstore/AGENTS.md 'explicit failure over silent corruption'
-
-Null report example: "Attacked lock ordering on the new disk-registry mutex pair, lock-loss fencing across the moved commit, power-cut points around the added rename, and cancellation at the two new awaits — no break found; quorum math and metacache paths untouched by this diff."
-
-### Compatibility reviewer
-
-- Grep the diff for raw 'x-rustfs-internal-' or 'x-minio-internal-' string literals used with map.insert/remove/get instead of the metadata_compat helpers. If found, construct the MinIO-written object case: a metadata map containing ONLY 'X-Minio-Internal-<suffix>' (mixed case, no RustFS key) and trace the diff's read path — does it miss the value? Then construct the removal case: does remove leave the twin key behind so a stale MinIO-key value resurrects on next read? Also check the value-type trap: get_bytes has NO case-insensitive fallback (unlike get_str), so a diff that moves a suffix from FileInfo.metadata (String) to meta_sys (Vec<u8>) silently loses mixed-case MinIO keys.
-  - Where: Any code touching FileInfo.metadata / user_defined / meta_sys: crates/ecstore/, crates/filemeta/, rustfs/src/storage/, crates/utils/src/http/metadata_compat.rs
-  - Evidence: Repo-wide invariant in AGENTS.md 'Cross-Cutting Domain Invariants' and CLAUDE.md; helpers and the asymmetry are pinned by tests test_str_lookup_accepts_minio_metadata_case and test_get_bytes_no_case_insensitive_fallback in crates/utils/src/http/metadata_compat.rs
-- For any diff reading a binary UUID from internal metadata (transitioned-versionID, tier-free-versionID, data_dir), trace the three degenerate inputs — key absent, value empty (b""), value nil UUID — through to the outgoing tier/S3 request. The bug shape to hunt: unwrap_or_default() or Uuid::from_slice(..).unwrap_or(Uuid::nil()) turning 'no value' into Uuid::nil(), which then gets serialized as ?versionId=00000000-... and the remote tier returns NoSuchVersion. The required pattern is .and_then(|v| Uuid::from_slice(&v).ok()).filter(|u| !u.is_nil()).
-  - Where: crates/ecstore/src/bucket/lifecycle/ (bucket_lifecycle_ops.rs, tier_sweeper.rs), crates/ecstore/src/services/tier/warm_backend_*.rs, crates/filemeta/src/filemeta/version.rs
-  - Evidence: Historical production bug documented in docs/operations/tier-ilm-debugging.md ('Nil-UUID versionId sent to tier'); regression tests live in crates/filemeta/src/filemeta/version.rs; follow-up fix 726f3dc18 'accept empty remote version_id in tier recovery paths' (#4552)
-- For any diff touching tier or replication GET/DELETE against a remote S3 target, enumerate BOTH directions of the versionId contract and trace each: (a) remote version None/"" means the tier bucket is unversioned — the request must carry NO versionId parameter at all (not an empty one, not nil); (b) remote version Some(v) on a versioned target — the versionId MUST be sent, especially on version-purge deletes, or the delete lands on the wrong version / creates a delete marker instead of purging. Check whether the diff collapses these cases through a single Option/String conversion that loses the distinction.
-  - Where: crates/ecstore/src/services/tier/warm_backend*.rs, crates/ecstore/src/bucket/lifecycle/tier_sweeper.rs, replication code under crates/ecstore/src/bucket/
-  - Evidence: Invariant in AGENTS.md and docs/operations/tier-ilm-debugging.md; real bug fixed by 0fad35645 'send versionId on version-purge deletes to generic S3 targets' (#4401) — the versioned direction, and #4552 — the unversioned direction
-- If the diff changes xl.meta encoding (adds/reorders msgpack header fields, touches FileMeta::marshal_msg or codec.rs encode paths), attack downgrade and cross-vendor parse: encode an object with the new code and decode it with (a) the meta_ver<=3 read path and (b) the real-MinIO fixture tests from #4377. Then check the header signature: is it recomputed over the new bytes, or copied/hardcoded? MinIO validates it; a stale or zero signature makes MinIO reject the file. Finally verify XL_META_VERSION was not silently bumped — old RustFS/MinIO nodes reject meta_ver > 3 during a rolling upgrade.
-  - Where: crates/filemeta/src/filemeta.rs (XL_HEADER_VERSION/XL_META_VERSION, lines ~46-54), crates/filemeta/src/filemeta/codec.rs (check_xl2_v1, decode_xl_headers)
-  - Evidence: Real bug 073bc9675 'compute header signature instead of hardcoding zero' (#4343); format contract pinned in docs/architecture/minio-file-format-compat.md (write meta_ver 3, read <=3, XL2 magic); parity fixtures from a91d9cefc (#4377)
-- If the diff touches xl.meta / FileInfo decode (into_fileinfo, version parsing, part arrays), construct hostile foreign input: a MinIO- or corruption-shaped msgpack with a parts-count that disagrees with the etags/sizes array lengths, missing optional fields, and a meta_ver 2 object with legacy checksum. Trace whether the new code indexes past an array, panics, or fabricates default values instead of returning a decode error. Run the pinned legacy fixtures (test_issue_2265_legacy_meta_v2_object_compatibility, test_issue_2288) plus the #4377 real-MinIO xl.meta parse tests against the diff.
-  - Where: crates/filemeta/src/ (fileinfo.rs, filemeta.rs, filemeta/codec.rs, filemeta/version.rs)
-  - Evidence: Real bug 7efacbdf9 'validate part array lengths in into_fileinfo' (#4382); legacy meta_ver 2 regression fixtures at crates/filemeta/src/filemeta.rs (~:1130-:1174) cited by docs/architecture/minio-file-format-compat.md
-- If the diff 'corrects' a formula, constant, or layout that is a byte-for-byte MinIO port (shard-size math, bitrot hash interleaving, erasure distribution, inline-data prefix), treat the correction itself as the bug: verify against legacy on-disk data before accepting. Concretely: run crates/ecstore/tests/legacy_bitrot_read_test.rs and the ECA-18 pinning tests; check whether existing objects written by old RustFS or MinIO still verify byte-for-byte. Known trap examples: bitrot_shard_file_size's bare return for non-streaming algorithms is CORRECT MinIO whole-file behavior, and the 32-byte prefix on inline data is the HighwayHash256 bitrot hash, not corruption.
-  - Where: crates/ecstore/src/erasure/coding/bitrot.rs, crates/ecstore/src/io_support/bitrot.rs, crates/filemeta/src/fileinfo.rs
-  - Evidence: f96314a1d 'pin streaming-only bitrot layout invariant (ECA-18)' (#4553) — audit explicitly decided NOT to change the formula because it breaks legacy interop; #4377 proved the inline-data prefix is the bitrot hash
-- For any diff that copies object metadata into an S3-client-visible surface (GET/HEAD response headers, notification event userMetadata, ListObjects/replication payloads, copy-object metadata directives), construct an object carrying internal keys under BOTH prefixes and in non-canonical casing ('X-Minio-Internal-Compression') and confirm every one is stripped via is_internal_key (which is case-insensitive) — not by an exact-match filter on one prefix. Leaked internal keys are an API-semantics break and an information leak.
-  - Where: rustfs/src/storage/, crates/notify/, replication and copy_object paths in crates/ecstore/
-  - Evidence: Real bug cf8929189 'strip rustfs/minio internal metadata from event userMetadata' (#4419); is_internal_key contract in crates/utils/src/http/metadata_compat.rs
-- If the diff touches crates/protos (node.proto, models.fbs) or internode RPC request/response structs, attack the rolling-upgrade interleaving: an old node sends a message without the new field to a new node, and a new node sends the extended message to an old node. Verify proto field numbers are only appended (never reused/renumbered), FlatBuffers tables are only extended at the end, and that an absent new field decodes to a safe default on the receiving side — 'safe' meaning it must not be interpreted as success/authorization (RPC errors fail closed) and must not flip a quorum decision.
-  - Where: crates/protos/ (node.proto, models.fbs, generated/), gRPC transport and dispatch in the internode layer
-  - Evidence: 6f613317f 'optimize gRPC transport' (#4337) shows the wire layer churns; security advisory 68cw fixed by PR#4402 established RPC fail-closed as a repo rule (see .agents/skills/security-advisory-lessons)
-- For S3 handler diffs, replay the request shapes real clients actually send, not just the canonical one: mc and aws-sdk differ on path normalization (root '//' ListBuckets), virtual-host vs path style, and header casing. Then attack every pagination boundary the diff touches: request exactly max-keys/max-uploads/max-parts items and verify the response returns exactly N (not N+1), sets IsTruncated correctly, and yields a NextMarker/KeyMarker that resumes without skipping or duplicating — construct the N+1st-item case explicitly.
-  - Where: rustfs/src/storage/ S3 handlers, listing paths in crates/ecstore/src/store/ and set_disk/
-  - Evidence: Real bugs 511ad31ba 'normalize root double-slash ListBuckets requests' (#4336) and fefa70b31 'stop ListMultipartUploads from returning one upload past max-uploads' (#4447); docs/architecture/s3-compatibility-matrix.md is the compat source of truth
-- If the diff changes bucket-metadata (.metadata.bin) or IAM/config parsing structs, run it against the real MinIO RELEASE.2025-07-23 fixtures: the msgpack blob uses PascalCase field names, so any serde rename, field-type change, or derive tweak silently drops MinIO-written fields instead of erroring. Verify parse_all_configs still loads all ten config types from the fixture without loss, and that drop-in migration still decrypts MinIO-encrypted IAM/server config rather than treating ciphertext as corrupt.
-  - Where: crates/ecstore/src/bucket/metadata*, crates/ecstore/src/bucket/migration.rs, IAM/config load paths in crates/iam/ and crates/config/
-  - Evidence: Fixture parity test parses_real_minio_bucket_metadata_blob_without_loss from a91d9cefc (#4377); real bug 717cdd2ab 'decrypt MinIO IAM & server config on drop-in migration' (#4358); format matrix in docs/architecture/minio-file-format-compat.md
-- If the diff adds a compatibility shim, legacy fallback, wrapper, or old-endpoint alias (grep the diff for 'legacy', 'fallback', 'compat', 'deprecated'), verify two things: (1) it carries a RUSTFS_COMPAT_TODO(<task-id>) marker with an exact removal condition and a matching entry in the register — an unmarked shim becomes permanent dead weight; (2) the fallback's default direction is safe for old data: e.g. a new decode path must fall back to the legacy decode for old objects by default, not gate legacy reads behind an opt-in flag that makes existing data unreadable after upgrade.
-  - Where: Anywhere in the diff; register at docs/architecture/compat-cleanup-register.md; recent example: allow_inplace_legacy_fallback flag in the ecstore erasure codec streaming path
-  - Evidence: docs/architecture/compat-cleanup-register.md review checklist; d232a46b4 wired legacy decode prefetch behind a default-OFF gate while keeping legacy reads working (#4542), with arity fallout fixed in 05890d6e2 (#4573)
-
-Null report example: "Attacked dual-key metadata writes/removals against MinIO-only-key objects, nil/empty transitioned-versionID paths to the tier, xl.meta encode against meta_ver<=3 decoders and the #4377 real-MinIO fixtures, and proto field-number evolution for old-node/new-node RPC — no compatibility break found."
-
-### Performance reviewer
-
-- For every `.clone()` the diff adds or moves onto a per-request/per-object path, open the cloned type and count heap fields (String, Vec, HashMap, Bytes). If >5 heap fields or it contains an EC block buffer, construct the cost: N concurrent PUTs x M objects -> N*M deep copies per second. Demand Arc-wrapping of heavy fields or pass-by-reference; also flag new `String` allocations in header/path/signature parsing where `&str`/`Cow<str>` suffices.
-  - Where: crates/ecstore/src/set_disk/**, crates/ecstore/src/store*.rs, rustfs/src/storage/, crates/filemeta/, request handlers in rustfs/src/
-  - Evidence: crates/ecstore/AGENTS.md 'Allocation Discipline in Hot Paths' (no Clone on >5-heap-field structs, Arc for large buffers, &str/Cow for temporary computations); .agents/skills/rust-code-quality/SKILL.md ranks 'unnecessary clone in hot path' as P1 must-fix
-- For every new sync_all/sync_data/fdatasync/flush/File::sync call in the diff, trace the call chain to DurabilityMode / RUSTFS_DRIVE_SYNC_ENABLE resolution (crates/ecstore/src/disk/local.rs:291 DurabilityMode, :347 resolve_durability_mode) and to per-bucket durability overrides. Construct the run where the operator sets mode=none (or legacy RUSTFS_DRIVE_SYNC_ENABLE=false) and the new fsync still fires — that is an ungated durability cost and a regression on 4KiB writes.
-  - Where: crates/ecstore/src/disk/local.rs, crates/ecstore/src/bucket/durability.rs, crates/ecstore/src/set_disk/** (rename_data/commit paths), any crate doing tokio::fs or std::fs writes
-  - Evidence: #4221 fsync work caused a measured -10% 4KiB write regression (#814 investigation), later gated; durability modes added in eaff17cad (#4397), per-bucket tier overrides in 13e48d93a (#4407); 2df315baf (#4493) shows even ancestor-dir fsyncs are routed through the gate
-- Attack blocking-work placement from both directions: (a) find new synchronous fs calls, hashing, or EC encode/decode executed directly on an async runtime thread without spawn_blocking/block_in_place — construct the stall (a slow disk blocks a worker thread and every task queued on it); (b) find new code that splits one logical disk operation into multiple spawn_blocking hops per object — each hop is a threadpool round-trip, so K hops x N objects multiplies latency. Demand the author justify the placement with the size of the work, not habit.
-  - Where: crates/ecstore/src/disk/local.rs, crates/ecstore/src/erasure_coding/, crates/ecstore/src/bitrot/, crates/rio/
-  - Evidence: 608ab14d7 (#4554, HP-12) folded metadata open+fstat+read into a single spawn_blocking because per-op hops were measurably slow; 8fc637fb1 (#4484) moved the short EC encode inline because block_in_place cost exceeded the work — direction depends on measured work size
-- For each lock acquisition the diff adds or relocates, mark the guard's live range and list every .await and disk/RPC call inside it. Construct the contention interleaving: N concurrent requests serialize on the guard while the holder waits on IO; for namespace/multipart commit locks, compute worst-case hold time (fsync + rename per disk) against the lock's timeout. Also diff the acquisition order against other paths taking the same locks (ABBA).
-  - Where: crates/ecstore/src/set_disk/** (commit/rename paths), crates/lock/, crates/audit/ registry, any RwLock/Mutex in per-request paths
-  - Evidence: crates/ecstore/AGENTS.md 'Lock Ordering'; c0d5f938f (#4421) fixed a real ABBA deadlock between registry and stream_cancellers; #4370 history: fsync-heavy serial cross-disk commits held a lock long enough to blow test timeouts (#4370)
-- Trace exactly what executes inside the PUT commit critical section (under the object write lock, between tmp write and rename_data completion) before vs after the diff. Any newly added work there — cleanup, extra stat, additional rename, O_DIRECT write, logging — is an attack target: construct the per-PUT latency delta and demand it be moved off the critical section or parallelized across disks.
-  - Where: crates/ecstore/src/set_disk/ops/*, crates/ecstore/src/disk/local.rs rename_data path
-  - Evidence: Three real optimizations removed exactly this class of regression: e7cc719c1 (#4389) moved speculative PUT-tail tmp cleanup off the hot path, 92c8c6db7 (#4411) moved O_DIRECT shard-writes off the commit critical section, 651ccac13 (#4487) parallelized tmp xl.meta write and shard fdatasync on commit
-- Find any new loop in a batch API that performs a per-item stat/read/RPC sequentially. Construct the concrete blowup: a 1000-key DeleteObjects or a full listing page -> 1000 serial round-trips added by the diff. Demand either a gate (skip when not needed) or bounded parallelism; for startup/load paths, check for accidental O(n^2) (re-scanning the full list per item).
-  - Where: crates/ecstore/src/store_delete*.rs / batch object APIs, listing/metacache paths, crates/iam/ store loading, crates/heal/
-  - Evidence: a413729b1 (#4398) had to gate and parallelize the DeleteObjects per-object stat fanout after it shipped serial; 16a91c35e (#4537) fixed O(n^2) IAM startup load by chunking — both were diff-introduced fanouts of this exact shape
-- For every buffer the diff allocates on the encode/decode/shard path, check: is it sized with with_capacity to the EC-expanded block (not the logical size, not default-grown)? Does it copy into a fresh Vec where Bytes::slice/clone (refcount) or the io-core buffer pool would avoid the copy? Does the diff read hash and data in separate passes where one pass suffices? Construct the per-block byte-copy count before vs after. If the diff touches the io-core pool, verify gauge accounting still balances.
-  - Where: crates/ecstore/src/erasure_coding/, crates/ecstore/src/bitrot/, crates/io-core/src/pool.rs, crates/rio/
-  - Evidence: 92bf55ce6 (#4396) fixed a real regression by right-sizing BytesMut encode ingest capacity to the EC-expanded block; 47bee8b31 (#4475) merged bitrot hash+data into one read pass; 7fa3d0d4b (#4534) shows pool gauge accounting is easy to drift when touching buffer reuse
-- For every logging or instrumentation statement the diff adds, classify the call site frequency: per-request, per-object, per-shard, or per-block. Anything info!/warn!/error! at per-object frequency or higher is a finding — construct the flood (one listing under client cancellation, one 10k-object heal) and count emitted lines. New metrics/timers on the data path must be feature-gated, not always-on. Run scripts/check_logging_guardrails.sh on the diff.
-  - Where: any per-request/per-object code, especially crates/ecstore listing and heal loops, rustfs/src/storage/ handlers; scripts/check_logging_guardrails.sh
-  - Evidence: .agents/skills/rustfs-logging-governance/SKILL.md (trace level for hot-path/repetitive success events); d25ddb0e1 (#4372) fixed real listing-cancellation error-log noise; hotpath instrumentation is deliberately feature-gated (3f13d098b #4394, f262fcfce #4541 HP-14)
-- Count how many times the diff's request path parses or fetches the same metadata: xl.meta/FileMeta decoded more than once per object, bucket metadata (metadata_sys) re-fetched inside a per-object loop, or the dual x-rustfs-internal/x-minio-internal key lookup re-run repeatedly on the same map. Construct the per-request parse count before vs after; a second full FileMeta decode per GET is a finding.
-  - Where: crates/filemeta/, crates/ecstore/src/set_disk/** read paths, crates/ecstore/src/bucket/metadata_sys.rs, crates/utils/src/http/metadata_compat.rs
-  - Evidence: 608ab14d7 (#4554) exists because redundant metadata-read syscall sequences per object were measurable; CLAUDE.md dual-key metadata convention makes repeated get_bytes lookups an easy hidden double-parse
-- If the diff touches PUT/GET/commit/erasure paths and claims 'no perf impact', demand numbers, not assertion: run the criterion benches (cargo bench -p ecstore — comparison_benchmark, erasure_benchmark, rename_data_meta_benchmark, single_block_non_inline_benchmark per crates/ecstore/benches/) against origin/main, and for end-to-end paths the warp A/B relative-budget gate (scripts/run_hotpath_warp_ab.sh --baseline-ref origin/main, as .github/workflows/performance-ab.yml runs it). Probe specifically at 4KiB object size — that is where the last real regression hid.
-  - Where: crates/ecstore/benches/, .github/workflows/performance-ab.yml, scripts/run_hotpath_warp_ab.sh
-  - Evidence: crates/ecstore/AGENTS.md: 'Benchmark-sensitive changes should include measurable rationale'; performance-ab.yml (215747022 #4480) is the repo's own relative-budget gate; the #4221 regression was only visible at 4KiB writes (#814 bisect)
-
-Null report example: "Attacked the new rename_data commit-section work, durability-gate routing of the added fdatasync, guard live-range across the shard-write awaits, and per-object clone count in the PUT path; ran comparison_benchmark + rename_data_meta_benchmark vs origin/main (4KiB delta within noise) — no break found."
-
-### Test-coverage skeptic
-
-- For every behavior claim in the PR description, revert that hunk (git stash / manual undo of the changed lines) and name the exact test (`cargo test -p <crate> <test_name>`) that fails. If no test fails on revert, the behavior is untested — file a finding, not a note. Especially verify the test exercises the REAL production call path, not a lookalike helper.
-  - Where: All crates; highest value in crates/ecstore, rustfs/src/storage, crates/heal
-  - Evidence: AGENTS.md exit criterion 'Every behavior change has a test that fails without it'. Real bug: PR #4220 (ghost-directory cleanup) merged with green tests but its fix never executed on the real delete path — required follow-up rustfs#4307, backlog#798 stayed OPEN. The tests exercised a path the production flow never took.
-- Read each added/modified test and confirm it asserts the real outcome (returned value, stored bytes, error variant), not merely 'call succeeded' or 'no panic'. Flag any test whose only observable is that the function returned, and any `assert!(result.is_err())` that never checks WHICH error. Then check: does the test prove the exploit/failure form is denied, or only that the intended form still works?
-  - Where: crates/e2e_test (security_boundary_test.rs pattern), and every #[cfg(test)] module in the diff
-  - Evidence: Commit dee8e4e63 (#4466) had to rewrite 277 lines of crates/e2e_test/src/security_boundary_test.rs because 'security boundary tests' passed without asserting real outcomes. .agents/skills/rust-code-quality/SKILL.md checklist: 'Every test function has at least one assert!'; .agents/skills/security-advisory-lessons/SKILL.md: 'Does the test prove the exploit form is denied, or only that the intended form still works?'
-- When the diff adds a boolean/mode parameter or config flag, find the test that fails if the flag's effect is INVERTED inside the changed function. Tests that were mechanically updated to pass `false`/default at every call site assert nothing about the new behavior. Execute the check: flip the flag's branch in the source and confirm at least one test goes red for each branch.
-  - Where: crates/ecstore/src/set_disk/ (e.g. build_codec_streaming_part_reader), any function gaining a parameter
-  - Evidence: Commit 05890d6e2 (#4573): PR #4560 added a 15th param allow_inplace_legacy_fallback; the arity tests were fixed by passing `false` everywhere — they assert Err outcomes independent of the flag, so the fallback behavior itself has no revert-detecting test at those sites.
-- Mutation spot-check on error propagation: for each newly added `?`, `return Err`, or error-mapping line, mentally replace it with `Ok(default)`/ignore and ask which test fails. The swallowed-error bug class recurs in this repo and always ships with green tests — a fix that propagates errors needs a test that injects the failure (faulty disk, failed rename, dispatch error) and asserts the caller sees Err.
-  - Where: crates/ecstore (disk delete/rename, reduce_errs), crates/audit, crates/notify, crates/targets
-  - Evidence: Three recent fixes for the same class: f7d2b2563 (#4546, disk delete/rename failures swallowed), dbc628f16 (#4424, audit dispatch failures swallowed), 20d61c73b (#4551, reduce_errs leaking nil placeholder as dominant error). All existed while tests were green.
-- Any test touching GET/read/reconstruct/stream paths must assert the FULL body content and exact length against a known value, not status-ok or first-bytes. Construct the degraded-read case (missing/inconsistent shards forcing EC reconstruction) and assert byte-for-byte equality; a mid-stream failure that truncates the body passes every test that only checks headers or the first chunk.
-  - Where: crates/ecstore/src/set_disk/read.rs and ops/, crates/rio, crates/e2e_test GET scenarios
-  - Evidence: Known live bug: EC reconstruct-read verification failure mid-GET at set_disk read path silently truncates the body → client 'unexpected EOF'; version-independent, undetected by existing suites because none assert full-body integrity under shard inconsistency.
-- For on-disk / on-wire format changes (xl.meta, .metadata.bin, bitrot framing), reject round-trip-only tests: a struct serialized and deserialized by the same code under test cannot catch format drift. Demand the test parse a REAL captured fixture from crates/filemeta/tests/fixtures, crates/ecstore/tests/fixtures, or crates/rio-v2/tests/minio_fixture_lab — or capture a new one from a single-disk MinIO instance (RELEASE.2025-07-23 procedure from #4377).
-  - Where: crates/filemeta, crates/ecstore (headers, msgpack bucket metadata), crates/rio-v2, migration code
-  - Evidence: Commit 073bc9675 (#4343): filemeta header signature was hardcoded to zero — round-trip tests passed for months. Commit a91d9cefc (#4377) established the real-MinIO fixture convention (inline/versioned/multipart xl.meta, HighwayHash256-prefixed inline bodies) precisely because synthetic fixtures proved nothing about interop.
-- Attack new concurrency tests for flakiness-by-construction: grep the added tests for `sleep(`, fixed timeouts under ~30s on lock acquisition, and use of shared global state (disk registry, lock client, GLOBAL_*). Serialized cross-disk commits exceed small lock timeouts under full-suite CI disk load. If the test shares global state or saturates IO, it must join the `ecstore-serial-flaky` nextest test-group in .config/nextest.toml (note: serial_test's #[serial] does NOT work — nextest runs each test in its own process). Require readiness polling, never fixed sleeps.
-  - Where: crates/ecstore tests, crates/e2e_test, .config/nextest.toml
-  - Evidence: Commit 2dfa3d3c3 (#4370): concurrent_resend test flaked with Lock(Timeout 5s) on CI — six legitimate serialized cross-disk commits under IO pressure needed 30s. Commit 7c701d9f2 (#4558) created the nextest test-group after bucket_delete_* raced make_bucket into InsufficientWriteQuorum. 65849740f (#4213) deflaked global-state contamination. crates/e2e_test/AGENTS.md: 'readiness checks and explicit polling over fixed sleep-based timing'.
-- If the diff writes internal object metadata, run the dual-key mutation: delete the `x-minio-internal-<suffix>` write (keeping only `x-rustfs-internal-`) and check whether any test fails. Because `get_bytes` prefers the RustFS key, every read-back test stays green while MinIO interop is silently broken — coverage must include an assertion that BOTH keys are present in the stored metadata map.
-  - Where: crates/utils/src/http/metadata_compat.rs and all its callers in crates/ecstore and rustfs/src/storage
-  - Evidence: CLAUDE.md domain convention: metadata must be written under both x-rustfs-internal- and x-minio-internal- keys for MinIO interop; get_bytes prefers the RustFS key, making the MinIO-key half of the invariant invisible to read-back tests.
-- For changed quorum/version/UUID logic, name the tests covering the specific poison values: quorum−1 disks, nil UUID, absent vs empty vs nil-serialized UUID bytes, and remote-tier version_id of None/"" (unversioned tier bucket → no versionId sent). Mutation check: remove a `.filter(|u| !u.is_nil())` guard from the diff and confirm a test fails; if none does, the nil-UUID class is uncovered.
-  - Where: crates/ecstore (tier recovery, heal, quorum paths), crates/filemeta, code reading UUIDs from xl.meta metadata
-  - Evidence: Commit 726f3dc18 (#4552) fixed rejection of empty remote version_id in tier recovery. CLAUDE.md invariant: absent/empty/nil UUID all mean 'no value', not Uuid::nil(). docs/operations/tier-ilm-debugging.md: None/"" tier version means unversioned bucket. df9cbc4ed (#4427): unvalidated distribution values caused shuffle index panic — edge values reached production untested.
-- For any pagination/limit/truncation change, construct the exact-boundary test: result count == max (page exactly full), max+1, and a delimiter re-fold that lands precisely on the page boundary — assert both the item count AND the is_truncated/continuation marker. Off-by-one at the page boundary is a recurring shipped bug here.
-  - Where: crates/ecstore listing paths (list_objects, ListMultipartUploads, metacache), S3 handlers in rustfs/src/storage
-  - Evidence: Two shipped boundary bugs: fefa70b31 (#4447) ListMultipartUploads returned one upload past max-uploads; d91f4d455 (#4538) delimiter re-fold of a full page lost the truncation flag. Both survived existing tests because no test pinned n == max exactly.
-- Green `cargo test -p <crate>` on the touched crate is not a coverage verdict for the diff's test code itself: run `cargo clippy --all-targets -p <crate>` and a workspace-wide test BUILD (`cargo check --workspace --all-targets` at minimum) before accepting the tests as evidence. Test-only code that doesn't compile workspace-wide or fails clippy has repeatedly broken main and masked whether tests ran at all.
-  - Where: All crates; especially concurrent-branch merges into crates/ecstore
-  - Evidence: #4322 broke main because only cargo test ran (field_reassign_with_default is clippy-only). b06f3df6b (#4441) and 05890d6e2 (#4573): test code broke the workspace test build (E0061) on main after textually-clean merges, failing CI for every open PR.
-
-Null report example: "Attacked revert-detection for all 3 claimed behaviors (each has a named test that fails on revert), flag-inversion on the new fallback parameter (both branches covered in codec_streaming tests), full-body assertions on the changed GET path, and n==max pagination boundary — no coverage gap found."
-
-## Sources and maintenance
-
-Probes are distilled from shipped bugs in git history (commit/PR references
-above), GitHub security advisories (see the security-advisory-lessons
-skill), scoped `AGENTS.md` rules, and invariants under `docs/architecture/`
-and `docs/operations/`. Line numbers drift; when a cited location no longer
-matches, trust the invariant and re-locate the code. When a new bug class
-ships, add a probe with its evidence here rather than growing the policy
-section in `AGENTS.md`.

.agents/skills/arch-checks/SKILL.md

@@ -47,12 +47,5 @@ consider adding it to the script's `checked_files` list.
 Instruction/architecture docs (`AGENTS.md`, `CLAUDE.md`, `ARCHITECTURE.md`,
 `docs/architecture/*.md`) must not reference repo file paths that no longer
 exist. If your refactor moved code, update the docs that point at it — the
-error message lists `doc -> stale-path` pairs.
-
-## `check_no_planning_docs.sh`
-
-Planning-type documents must not be committed (see AGENTS.md "Sources of
-Truth"). The guard fails if anything is tracked under `docs/superpowers/` —
-`.gitignore` already ignores it, but `git add -f` bypasses that, so this closes
-the hole. Fix by removing the listed file(s) with `git rm`; keep the plan or
-spec in the issue tracker or a local worktree instead.
+error message lists `doc -> stale-path` pairs. Historical plans under
+`docs/superpowers/plans/` are exempt.

.agents/skills/security-advisory-lessons/SKILL.md

@@ -26,7 +26,7 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 
 ### 1. Scope the change
 - Identify touched routes, protocol frontends, handlers, storage paths, credentials, logs, browser surfaces, CI/release code, and policy checks.
-- Treat these paths as security-sensitive by default: `rustfs/src/admin/`, `rustfs/src/storage/`, `rustfs/src/auth.rs`, `rustfs/src/server/layer.rs`, `crates/iam/`, `crates/policy/`, `crates/credentials/`, `crates/ecstore/src/rpc/`, `crates/protocols/`, `crates/rio/`, OIDC/STS federation code, and console preview/auth code.
+- Treat these paths as security-sensitive by default: `rustfs/src/admin/`, `rustfs/src/storage/`, `rustfs/src/auth.rs`, `rustfs/src/server/layer.rs`, `crates/iam/`, `crates/policy/`, `crates/credentials/`, `crates/ecstore/src/rpc/`, `crates/protocols/`, `crates/rio/`, and console preview/auth code.
 
 ### 2. Map to advisory classes
 - Read [advisory-patterns.md](references/advisory-patterns.md) for matching GHSA lessons.
@@ -59,16 +59,10 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 
 ### IAM and service accounts
 - Treat imported IAM payload fields as attacker-controlled: `parent`, `claims`, `accessKey`, `secretKey`, status, policy names, and groups.
-- For service account create/update/import, prove parent ownership or root/admin authority before writing credentials or claims; an action permission alone must not allow choosing root or another user as `target_user`.
+- For service account create/update/import, prove parent ownership or root/admin authority before writing credentials or claims.
 - Do not let `deny_only` or "no explicit deny" become an allow decision that skips required allow checks.
 - Test cross-user list/update/import flows with wrong, correct, self, parent, and root identities.
 
-### STS, OIDC, and federation flows
-- Every STS endpoint must have an explicit authentication story: SigV4 where required, OIDC token verification for web identity, and role/session policy validation before issuing credentials.
-- JWT session tokens must be signed and verified by a trusted issuer/key path, not by service-account-controlled material or a reused root secret.
-- Public OIDC bootstrap and callback routes must treat `Host`, `X-Forwarded-Proto`, redirect targets, `state`, and callback parameters as untrusted; credential-bearing redirects require a configured, allowlisted origin.
-- OIDC discovery and validation URLs are SSRF sinks. Resolve and classify hostnames at connection time, reject rebinding to loopback/private/link-local ranges, and do not rely on literal string checks.
-
 ### S3 copy, multipart, and presigned POST
 - Multipart copy must enforce source `GetObject` and destination `PutObject` semantics equivalent to `CopyObject`, including copy-source and policy conditions.
 - Do not let `CreateMultipartUpload`, `UploadPartCopy`, `CompleteMultipartUpload`, or `AbortMultipartUpload` return success without authorization.
@@ -119,7 +113,6 @@ For the full pattern map, read [advisory-patterns.md](references/advisory-patter
 
 ### Trusted proxy and network identity
 - Only honor `X-Forwarded-For` or `X-Real-IP` when the request came from a configured trusted proxy.
-- Apply the same trusted-proxy rule to scheme and host derivation; direct clients must not control security-sensitive redirects through `Host`, `X-Forwarded-Host`, or `X-Forwarded-Proto`.
 - Direct clients must use the socket peer address for `aws:SourceIp` and policy condition evaluation.
 - Add tests for direct spoofed headers and trusted-proxy headers.
 
@@ -137,8 +130,6 @@ Use these prompts while reviewing a diff:
 - Does a public/default/empty config change security behavior from fail-closed to fail-open?
 - Is any attacker-controlled value later used as a path, policy condition, credential identity, log field, URL, Origin, or response body?
 - Does this response contain stored replication, remote target, or service credentials that need redaction or stricter authorization?
-- Can this STS/OIDC path issue credentials without SigV4, trusted issuer validation, allowlisted redirects, or trusted-proxy host/scheme handling?
-- Does this outbound validation path resolve attacker-supplied hostnames and reject private, loopback, link-local, and rebound addresses at the actual connection boundary?
 - Is an archive entry, object key, or policy resource normalized differently between authorization and storage?
 - Is the same operation implemented in multiple paths, such as `CopyObject` vs `UploadPartCopy`, and do all paths enforce the same security contract?
 - Does a preview or browser-surface fix preserve the original security invariant when adding alternate viewers or file-type detection?

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -27,17 +27,9 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 ### IAM import, service accounts, and privilege boundaries
 
 - `GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
-- `GHSA-5354-r3w2-34m8`: `AddServiceAccount` checked `CreateServiceAccountAdminAction` but trusted caller-supplied `target_user`, allowing service accounts under the root parent. Lesson: service-account create paths must validate parent ownership or root/admin authority, not only the create action.
 - `GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
 - `GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
 
-### STS, OIDC, and federation flows
-
-- `GHSA-5qfg-mf7r-jp3w`: `AssumeRoleWithWebIdentity` was reachable without the required request authentication and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption.
-- `GHSA-ccrv-v8v9-ch9q`: service-account-controlled material could self-sign JWT session tokens with forged policy claims. Lesson: session tokens must be signed by a trusted issuer/key path and validation must reject self-signed or principal-controlled tokens.
-- `GHSA-9pjf-w3c2-m32r`, `GHSA-4x2q-cpx9-9h26`, and `GHSA-xvpm-p3f7-34c3`: public OIDC authorize/callback flows trusted request `Host` or forwarded scheme when building credential-bearing redirects. Lesson: OIDC redirects must use configured allowlisted origins and trusted-proxy handling; never derive the post-login credential destination from direct client headers.
-- `GHSA-m479-9x88-94w6`, `GHSA-frwq-mfqx-83p8`, `GHSA-q9q8-rf9r-fg9f`, and `GHSA-j5c2-hhf7-6gf5`: OIDC validation accepted attacker-controlled discovery URLs because hostname checks rejected only literal forbidden IPs, allowing DNS rebinding SSRF. Lesson: outbound federation URL validation must resolve and classify hostnames at the connection boundary and reject loopback, private, link-local, and rebound addresses.
-
 ### S3 copy, multipart, and upload policy validation
 
 - `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
@@ -58,7 +50,7 @@ Update this file only when an advisory adds or changes a reusable lesson, affect
 
 ### Secrets, defaults, and cryptographic misuse
 
-- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, and `GHSA-9gf3-jx4p-4xxf`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
+- `GHSA-j59h-h7q5-q348`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, and console surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
 - `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
 - `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
 - `GHSA-75fx-qg6f-8rm7` and `GHSA-68cw-96m3-h2cf`: internode RPC secrets were derivable from known root credentials, making raw storage RPC signatures forgeable when explicit RPC secrets were unset. Lesson: RPC auth keys must be independent random secrets, never derived from S3 root credentials, and raw storage RPC should not share the public S3 listener without an internode-only boundary.

.cargo/config.toml

@@ -14,20 +14,13 @@
 
 # RustFS Cargo configuration
 
-# NOTE: `--cfg tokio_unstable` is deliberately NOT set here.
-#
-# It used to be a global `[build] rustflags` entry so that the (default-off)
-# `dial9` telemetry feature could compile. That made every build depend on
-# Tokio's unstable, non-semver API, and it broke silently whenever a caller
-# exported their own RUSTFLAGS — an environment RUSTFLAGS replaces the value
-# from this file rather than appending to it.
-#
-# The flag is now scoped to telemetry builds, which opt in explicitly:
-#
-#     make build-profiling
-#     RUSTFLAGS="--cfg tokio_unstable" cargo build --features dial9
-#
-# `crates/obs/build.rs` fails the compile if the `dial9` feature is on without
-# the flag, so the two can no longer drift apart unnoticed.
-#
-# For CPU profiling, add `-C force-frame-pointers=yes` to that RUSTFLAGS value.
+# Enable tokio_unstable cfg for dial9-tokio-telemetry support
+# This allows dial9 to hook into Tokio's internal runtime events
+[build]
+# Enable Tokio unstable features required by dial9-tokio-telemetry for runtime tracing.
+# See: https://docs.rs/tokio/latest/tokio/#unstable-features
+rustflags = ["--cfg", "tokio_unstable"]
+
+# Enable frame pointers for CPU profiling (Linux only, optional but recommended)
+# Uncomment the following line for better CPU profiling data
+# rustflags = ["--cfg", "tokio_unstable", "-C", "force-frame-pointers=yes"]

.config/make/build.mak

@@ -35,25 +35,6 @@ build-gnu-arm64: ## Build aarch64 GNU version
 	./build-rustfs.sh --platform aarch64-unknown-linux-gnu
 
 
-## —— Profiling build (dial9 Tokio runtime telemetry) ------------------------------------------
-
-# dial9 hooks Tokio's unstable runtime instrumentation, so it needs
-# `--cfg tokio_unstable`. That flag is deliberately absent from
-# .cargo/config.toml: it is not free, and release binaries do not carry it.
-# Setting RUSTFLAGS here replaces (never appends to) the config-file value, and
-# crates/obs/build.rs fails the build if the feature and the flag disagree.
-#
-# There are no task-dump or S3-upload features — see the notes in
-# crates/obs/Cargo.toml for why.
-DIAL9_FEATURES ?= dial9
-DIAL9_RUSTFLAGS ?= --cfg tokio_unstable
-
-.PHONY: build-profiling
-build-profiling: ## Build RustFS with dial9 Tokio runtime telemetry (diagnostic builds only)
-	@echo "🔬 Building RustFS with dial9 telemetry (features: $(DIAL9_FEATURES))..."
-	@echo "⚠️  Diagnostic build: telemetry writes trace segments to disk continuously."
-	RUSTFLAGS="$(DIAL9_RUSTFLAGS)" cargo build --release --bin rustfs --features $(DIAL9_FEATURES)
-
 .PHONY: build-cross-all
 build-cross-all: core-deps ## Build binaries for all architectures
 	@echo "🔧 Building all target architectures..."

.config/make/check.mak

@@ -11,16 +11,14 @@ check-%:
 
 # Warning-only check
 # Checks for optional dependencies and issues a warning if not found
+# (e.g., cargo-nextest for enhanced testing)
 warn-%:
 	@command -v $* >/dev/null 2>&1 || { \
 		echo >&2 "⚠️ '$*' is not installed."; \
 	}
 
 # For checking dependencies use check-<dep-name> or warn-<dep-name>
-#
-# NOTE: cargo-nextest is a HARD dependency of `make test`, gated inside the
-# test recipe itself (with a RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 escape hatch)
-# rather than a warn-only prerequisite here — see .config/make/tests.mak.
-.PHONY: core-deps fmt-deps
+.PHONY: core-deps fmt-deps test-deps
 core-deps: check-cargo ## Check core dependencies
 fmt-deps: check-rustfmt ## Check lint and formatting dependencies
+test-deps: warn-cargo-nextest ## Check tests dependencies

.config/make/lint-fmt.mak

@@ -15,7 +15,7 @@ fmt-check: core-deps fmt-deps ## Check code formatting
 .PHONY: clippy-check
 clippy-check: core-deps ## Run clippy checks
 	@echo "🔍 Running clippy checks..."
-	cargo clippy --all-targets -- -D warnings
+	cargo clippy --all-targets --all-features -- -D warnings
 
 .PHONY: clippy-fix
 clippy-fix: core-deps ## Apply clippy fixes
@@ -50,16 +50,6 @@ tokio-io-uring-check: ## Check tokio io-uring runtime feature stays removed
 	@echo "🚫 Checking tokio io-uring feature guard..."
 	./scripts/check_no_tokio_io_uring.sh
 
-.PHONY: extension-schema-check
-extension-schema-check: ## Check extension-schema stays a lightweight contract crate
-	@echo "🧩 Checking extension schema boundaries..."
-	./scripts/check_extension_schema_boundaries.sh
-
-.PHONY: body-cache-whitelist-check
-body-cache-whitelist-check: ## Check the body-cache eligibility gate stays a fail-closed allow-list
-	@echo "🧱 Checking body-cache whitelist guard..."
-	./scripts/check_body_cache_whitelist.sh
-
 .PHONY: compilation-check
 compilation-check: core-deps ## Run compilation check
 	@echo "🔨 Running compilation check..."

.config/make/pre-commit.mak

@@ -13,19 +13,14 @@ doc-paths-check: ## Check that instruction/architecture docs reference existing
 	@echo "📄 Checking doc path references..."
 	./scripts/check_doc_paths.sh
 
-.PHONY: planning-docs-check
-planning-docs-check: ## Check that no planning-type documents are committed
-	@echo "📄 Checking for committed planning docs..."
-	./scripts/check_no_planning_docs.sh
-
 .PHONY: pre-commit
-pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
+pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check doc-paths-check quick-check ## Run fast pre-commit checks without clippy/full tests
 	@echo "✅ All pre-commit checks passed!"
 
 .PHONY: pre-pr
-pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check clippy-check test ## Run full pre-PR checks with clippy and tests
+pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check doc-paths-check clippy-check test ## Run full pre-PR checks with clippy and tests
 	@echo "✅ All pre-PR checks passed!"
 
 .PHONY: dev-check
-dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check doc-paths-check planning-docs-check quick-check ## Run fast local development checks
+dev-check: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check tokio-io-uring-check doc-paths-check quick-check ## Run fast local development checks
 	@echo "✅ Fast development checks passed!"

.config/make/tests.mak

@@ -2,25 +2,6 @@
 
 TEST_THREADS ?= 1
 
-# cargo-nextest is a HARD dependency of `make test`.
-#
-# nextest changes test semantics vs plain `cargo test`: it runs every test in
-# its own process (so serial_test's #[serial] mutex does not serialize across
-# tests) and it is the only runner that honours .config/nextest.toml
-# [test-groups] (e.g. the ecstore-serial-flaky serialization guard). CI runs
-# nextest (.github/actions/setup/action.yml installs it), so a silent fallback
-# to `cargo test` would run with different serialization behaviour than CI and
-# mask (or invent) flakes.
-#
-# Installing cargo-nextest:
-#   cargo install cargo-nextest --locked          # from source
-#   # or a prebuilt binary (faster) via the taiki-e installer / get.nexte.st:
-#   #   https://nexte.st/docs/installation/
-#
-# Escape hatch: set RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 to run the plain
-# `cargo test` fallback anyway. Results are NOT authoritative — semantics
-# differ from CI and .config/nextest.toml test-groups will NOT apply.
-
 .PHONY: script-tests
 script-tests: ## Run shell script tests
 	@echo "Running script tests..."
@@ -28,38 +9,13 @@ script-tests: ## Run shell script tests
 	./scripts/test_entrypoint_credentials.sh
 
 .PHONY: test
-test: core-deps script-tests ## Run all tests (needs cargo-nextest; RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 to override)
+test: core-deps test-deps script-tests ## Run all tests
 	@echo "🧪 Running tests..."
 	@if command -v cargo-nextest >/dev/null 2>&1; then \
 		cargo nextest run --all --exclude e2e_test; \
-	elif [ "$${RUSTFS_ALLOW_CARGO_TEST_FALLBACK:-0}" = "1" ]; then \
-		echo >&2 "⚠️  ============================================================================"; \
-		echo >&2 "⚠️  cargo-nextest NOT found — running the 'cargo test' fallback (opt-in)."; \
-		echo >&2 "⚠️  TEST SEMANTICS DIFFER FROM CI; results are NOT authoritative:"; \
-		echo >&2 "⚠️    * nextest runs each test in its own process; 'cargo test' does not,"; \
-		echo >&2 "⚠️      so serial_test #[serial] serialization behaves differently."; \
-		echo >&2 "⚠️    * .config/nextest.toml [test-groups] will NOT apply (e.g. the"; \
-		echo >&2 "⚠️      ecstore-serial-flaky group), so load-sensitive tests may flake here"; \
-		echo >&2 "⚠️      but pass on CI (or vice versa)."; \
-		echo >&2 "⚠️  Install cargo-nextest and re-run before trusting these results."; \
-		echo >&2 "⚠️  ============================================================================"; \
-		cargo test --workspace --exclude e2e_test -- --nocapture --test-threads="$(TEST_THREADS)"; \
 	else \
-		echo >&2 "❌ cargo-nextest is required for 'make test' but was not found."; \
-		echo >&2 ""; \
-		echo >&2 "   RustFS tests run under cargo-nextest (process-per-test isolation)."; \
-		echo >&2 "   CI runs nextest and .config/nextest.toml [test-groups] only take effect"; \
-		echo >&2 "   under nextest. Plain 'cargo test' has different serialization semantics"; \
-		echo >&2 "   and is NOT a faithful substitute."; \
-		echo >&2 ""; \
-		echo >&2 "   Install it with either:"; \
-		echo >&2 "     cargo install cargo-nextest --locked"; \
-		echo >&2 "   or a prebuilt binary (faster) — see https://nexte.st/docs/installation/"; \
-		echo >&2 ""; \
-		echo >&2 "   To run the plain 'cargo test' fallback anyway (results NOT authoritative;"; \
-		echo >&2 "   serialization semantics differ from CI), re-run with:"; \
-		echo >&2 "     RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1 make test"; \
-		exit 1; \
+		echo "ℹ️ cargo-nextest not found; falling back to 'cargo test'"; \
+		cargo test --workspace --exclude e2e_test -- --nocapture --test-threads="$(TEST_THREADS)"; \
 	fi
 	cargo test --all --doc
 

.config/migration-gate-floor.txt

@@ -1,10 +0,0 @@
-# Committed floor for the number of tests selected by the migration-critical
-# CI gate (see scripts/check_migration_gate_count.sh, backlog#1153 infra-12).
-#
-# The floor equals the exact count of rustfs-ecstore --lib tests matching the
-# gate filter (name substrings: data_movement, rebalance, decommission,
-# source_cleanup, delete_marker) at the time this file was last updated.
-# CI fails if the selected count drops below this number, so renames or
-# removals that thin the gate must update this file in the same PR.
-# Adding tests does not require a bump, but bumping keeps the guard tight.
-571

.config/nextest.toml

@@ -1,201 +0,0 @@
-# nextest configuration for RustFS.
-#
-# Serialize two known load-sensitive / global-state-sharing ecstore test groups
-# so the full parallel nextest suite stops producing spurious failures
-# (backlog #937). These tests pass in isolation but flake under the loaded
-# parallel run for two distinct reasons:
-#
-#   * store::bucket::tests::bucket_delete_* share process/global state (disk
-#     registry, lock client) and race make_bucket into InsufficientWriteQuorum
-#     when run concurrently with other ecstore tests.
-#   * bucket_lifecycle_ops::tests::concurrent_resend_same_part_commits_one_generation
-#     asserts a lock-acquire correctness property whose serialized cross-disk
-#     commits exceed the (already max'd, 60s) acquire deadline only when the
-#     suite saturates disk I/O.
-#
-# serial_test's #[serial] attribute does NOT serialize these across runs:
-# nextest executes each test in its own process, where the in-process
-# serial_test mutex has no effect. A nextest test-group with max-threads = 1 is
-# the mechanism that actually serializes across nextest's process boundary.
-#
-# ---------------------------------------------------------------------------
-# Profiles
-# ---------------------------------------------------------------------------
-# The `default` profile is what local `cargo nextest run` uses. It NEVER
-# retries: a red test locally means a real failure to investigate, not noise to
-# paper over. The `ci` profile (below) is the strict CI gate: global
-# retries = 0 so a new race's first occurrence is never masked, plus a
-# narrowly-scoped quarantine list (retries = 2) for tests with a tracked OPEN
-# flake issue. Flake policy lives in docs/testing/README.md.
-
-[test-groups]
-ecstore-serial-flaky = { max-threads = 1 }
-
-# Reliability / fault-injection e2e tests each spawn a single-node 4-disk RustFS
-# server and manipulate its disk directories at runtime (crates/e2e_test:
-# reliability_disk_fault_test, degraded_read_eof_regression_test / dist-13). They
-# are correct in isolation but resource-heavy; serialize them under nextest's
-# process boundary (serial_test's #[serial] does not cross it) so several 4-disk
-# servers never run at once. ci-7's nightly picks these up via the e2e suite;
-# they are deliberately NOT in the fast PR `e2e-smoke` filter.
-e2e-reliability = { max-threads = 1 }
-
-# --- default profile (local): serialize the flaky groups, never retry --------
-[[profile.default.overrides]]
-filter = 'package(rustfs-ecstore) & (test(concurrent_resend_same_part_commits_one_generation) | test(/^store::bucket::tests::bucket_delete_(mark_delete_marks|purge_removes|default_s3_delete)/))'
-test-group = 'ecstore-serial-flaky'
-
-# Serialize the 4-disk reliability / degraded-read e2e tests (see the
-# e2e-reliability test-group note above). The matching ci-profile override is at
-# the end of the file, after [profile.ci] is declared.
-[[profile.default.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
-test-group = 'e2e-reliability'
-
-# ---------------------------------------------------------------------------
-# ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
-# ---------------------------------------------------------------------------
-[profile.ci]
-# Strict: a new race must fail on its first occurrence, never be retried away.
-retries = 0
-# Report every failure in one run instead of bailing on the first.
-fail-fast = false
-
-[profile.ci.junit]
-# Emitted to target/nextest/ci/junit.xml; uploaded as a CI artifact.
-# Tests that pass only after a quarantine retry are marked `flaky` here — that
-# marker is the observable signal the flake policy is built around.
-path = "junit.xml"
-
-# ===========================================================================
-# QUARANTINE — flaky tests granted retries = 2 under the ci profile ONLY.
-#
-# RULES (enforced by review, see docs/testing/README.md):
-#   * Every entry MUST link exactly one OPEN issue tracking the flake.
-#   * An entry stays until the issue is fixed (test made robust) or the test is
-#     deleted — 30-day policy. No entry may exist without a live issue link.
-#
-# Each entry also re-declares the `ecstore-serial-flaky` test-group so the
-# serialization holds under the ci profile (nextest evaluates a named
-# profile's own overrides list, not the default profile's).
-# ===========================================================================
-
-# QUARANTINE: OPEN backlog#937 — concurrent_resend lock-acquire deadline flakes
-# under saturated disk I/O in the full parallel suite.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(concurrent_resend_same_part_commits_one_generation)'
-test-group = 'ecstore-serial-flaky'
-retries = 2
-
-# QUARANTINE: OPEN backlog#937 — store::bucket::tests::bucket_delete_* race
-# make_bucket into InsufficientWriteQuorum via shared global state under load.
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(/^store::bucket::tests::bucket_delete_(mark_delete_marks|purge_removes|default_s3_delete)/)'
-test-group = 'ecstore-serial-flaky'
-retries = 2
-
-# QUARANTINE: OPEN rustfs#4690 — walk_dir stall-budget accounting test depends
-# on producer/consumer timing windows that stretch past the budget on loaded
-# CI runners (regression test for rustfs#4644; failed on a zero-Rust-diff PR).
-[[profile.ci.overrides]]
-filter = 'package(rustfs-ecstore) & test(walk_dir_does_not_charge_consumer_backpressure_to_the_stall_budget)'
-retries = 2
-
-# Serialize the 4-disk reliability / degraded-read e2e tests under the ci
-# profile too (see the e2e-reliability test-group note near the top). Not a
-# quarantine: no retries, just single-threaded so several 4-disk servers never
-# run concurrently when ci-7's nightly runs the full e2e suite.
-[[profile.ci.overrides]]
-filter = 'package(e2e_test) & test(/^(reliability_disk_fault|degraded_read_eof_regression)_test::/)'
-test-group = 'e2e-reliability'
-
-# ---------------------------------------------------------------------------
-# e2e-smoke profile — PR smoke subset of the e2e_test crate (backlog#1149 ci-4)
-# ---------------------------------------------------------------------------
-# PR smoke subset of the e2e_test crate (backlog#1149 ci-4). This profile is
-# the single wiring mechanism for e2e tests in CI: other suites join by
-# extending this filter (or a sibling profile), never by adding ad-hoc e2e
-# jobs to ci.yml. Admission criteria (see crates/e2e_test/README.md): fast,
-# single-node topology, no external dependencies (no awscurl / Vault / fixed
-# ports / pre-started server), no #[ignore].
-#
-# Each e2e test spawns its own rustfs server on a random port with an isolated
-# temp dir (crates/e2e_test/src/common.rs), so the subset is parallel-safe.
-#
-# Replication PR subset (backlog#1147 repl-1): the second clause admits the 20
-# FAST bucket-replication tests from replication_extension_test — the
-# target-registration / replication-check / list / remove / delete admin paths
-# that validate config synchronously and never wait for asynchronous
-# replication convergence. Each spawns its own single-node rustfs server(s) on
-# random ports (source, plus an independent single-node target for the pair
-# checks — NOT a cluster), so the subset stays parallel-safe and single-digit
-# seconds. The data-plane tests that poll for convergence and all
-# `_real_dual_node` / `_real_single_node`
-# site-replication tests run in the [profile.e2e-repl-nightly] lane below, NOT
-# here. This allowlist is the single source of truth for the PR/nightly split:
-# the nightly profile derives its set as "the replication module MINUS this
-# allowlist", so any new replication test lands in nightly by default (never
-# silently unrun) until it is explicitly blessed as fast here. Keep the two
-# regexes byte-identical. Count invariant: 20 here + 18 nightly = 38 total
-# (authority: `cargo nextest list`; docs/testing/e2e-suite-inventory.md).
-# HISTORY (2026-07-11): the 20 fast tests were briefly pulled out of this lane
-# (#4724) because they set a loopback (127.0.0.1) replication target that the
-# SSRF egress guard rejected on every PR after repl-1 (#4712). That is fixed —
-# the guard now honours an off-by-default opt-in and this suite's source servers
-# set it (RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET) — so the allowlist below is
-# restored.
-[profile.e2e-smoke]
-default-filter = """
-    package(e2e_test) & (
-        test(/^(delete_marker_migration_semantics|version_id_regression|list_objects_v2_pagination|list_object_versions_regression|list_objects_duplicates|list_buckets_double_slash|leading_slash_key|special_chars|create_bucket_region|delete_objects_versioning|head_object_consistency|head_object_range|copy_object_metadata|copy_source_invalid_date|content_encoding|anonymous_access|bucket_policy_check|presigned_negative)_test::/)
-        | test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
-        | test(/^reliant::lifecycle::/)
-    )
-"""
-fail-fast = false
-
-# ---------------------------------------------------------------------------
-# e2e-repl-nightly profile — scheduled full replication e2e lane (repl-1)
-# ---------------------------------------------------------------------------
-# backlog#1147 repl-1 (deps: ci-4). Runs the SLOW / cross-process replication
-# tests that are unfit for the per-PR e2e-smoke gate:
-#
-#   * 8 bucket-replication data-plane tests — they PUT/delete objects and poll
-#     until source and target converge; two replicate over HTTPS.
-#   * 9 `_real_dual_node` site-replication tests — each spawns TWO full rustfs
-#     servers and drives the cross-process site-replication control plane.
-#   * 1 `_real_single_node` service-account round-trip test.
-#
-# The set is defined as "everything in replication_extension_test that is NOT
-# in the e2e-smoke PR allowlist above" (the negated clause is byte-identical to
-# the allowlist), so a newly added replication test automatically runs here
-# until it is explicitly promoted to the fast PR subset — no replication test
-# is ever silently left out of CI.
-#
-# #[serial] does NOT serialize under nextest (process-per-test; see the file
-# header). These tests need no cross-test serialization: each spawns its own
-# server(s) on random ports with isolated temp dirs, so they are parallel-safe
-# by construction — the same property the e2e-smoke subset relies on. If load
-# on the runner surfaces a real flake, quarantine the specific test with an
-# OPEN issue link (ci-10 / backlog#937 policy), never blanket-retry or exclude.
-#
-# Wired by .github/workflows/e2e-replication-nightly.yml (schedule +
-# workflow_dispatch), which builds the rustfs binary once, installs awscurl so
-# the STS dual-node test actually exercises its path (it skips gracefully with
-# a visible log line when awscurl is absent), and routes scheduled failures
-# through .github/actions/schedule-failure-issue (ci-8). Explicit division of
-# labor with ci-5's future e2e-full merge gate: these tests run ONLY here, not
-# double-run there. TODO(ci-7): fold this interim repl-owned lane into the ci
-# domain's consolidated scheduled e2e workflow once it exists.
-[profile.e2e-repl-nightly]
-default-filter = """
-    package(e2e_test)
-    & test(/^replication_extension_test::/)
-    & !test(/^replication_extension_test::(test_replication_check_succeeds_with_remote_target|test_replication_check_rejects_target_without_object_lock|test_set_remote_target_rejects_unversioned_source_bucket|test_replication_check_rejects_unversioned_source_bucket|test_replication_check_rejects_missing_replication_config|test_replication_check_rejects_invalid_bucket|test_set_remote_target_rejects_same_bucket_on_same_deployment|test_set_remote_target_rejects_unversioned_target_bucket|test_set_remote_target_update_requires_arn|test_set_remote_target_update_rejects_missing_target|test_set_remote_target_rejects_invalid_target_url|test_set_remote_target_rejects_self_signed_https_target_without_skip_tls_verify|test_set_remote_target_rejects_private_ca_https_target_without_ca_cert_pem|test_list_remote_targets_rejects_empty_bucket|test_list_remote_targets_rejects_invalid_bucket|test_remove_remote_target_rejects_missing_target|test_remove_remote_target_rejects_missing_arn|test_remove_remote_target_rejects_invalid_bucket|test_remove_remote_target_rejects_target_used_by_replication|test_delete_bucket_replication_removes_remote_target)$/)
-"""
-fail-fast = false
-
-[profile.e2e-repl-nightly.junit]
-# Emitted to target/nextest/e2e-repl-nightly/junit.xml; uploaded by the nightly
-# workflow as the failure-triage artifact.
-path = "junit.xml"

.docker/compose/docker-compose.cluster.local-build.yml

@@ -23,8 +23,8 @@ services:
       - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
       - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
@@ -46,7 +46,7 @@ services:
       - RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
       - RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
       - RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
-      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
+      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-0}
     extra_hosts:
       - "host.docker.internal:host-gateway"
     volumes:
@@ -69,8 +69,8 @@ services:
       - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
       - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
@@ -92,7 +92,7 @@ services:
       - RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
       - RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
       - RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
-      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
+      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-0}
     extra_hosts:
       - "host.docker.internal:host-gateway"
     volumes:
@@ -115,8 +115,8 @@ services:
       - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
       - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
@@ -138,7 +138,7 @@ services:
       - RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
       - RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
       - RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
-      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
+      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-0}
     extra_hosts:
       - "host.docker.internal:host-gateway"
     volumes:
@@ -161,8 +161,8 @@ services:
       - RUSTFS_VOLUMES=${RUSTFS_VOLUMES:-http://node{1...4}:9000/data/rustfs{0...3}}
       - RUSTFS_ADDRESS=:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
+      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin}
+      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfsadmin}
       - RUSTFS_OBS_ENDPOINT=${RUSTFS_OBS_ENDPOINT:-http://host.docker.internal:4318}
       - RUSTFS_OBS_LOGGER_LEVEL=${RUSTFS_OBS_LOGGER_LEVEL:-info}
       - RUSTFS_OBS_USE_STDOUT=${RUSTFS_OBS_USE_STDOUT:-false}
@@ -184,7 +184,7 @@ services:
       - RUSTFS_INTERNODE_OFFLINE_BYPASS=${RUSTFS_INTERNODE_OFFLINE_BYPASS:-false}
       - RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS=${RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS:-5}
       - RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD=${RUSTFS_INTERNODE_OFFLINE_FAILURE_THRESHOLD:-3}
-      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-1}
+      - RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES=${RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES:-0}
     extra_hosts:
       - "host.docker.internal:host-gateway"
     volumes:

.docker/compose/docker-compose.cluster.yaml

@@ -21,8 +21,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
     platform: linux/amd64
     ports:
       - "9000:9000" # Map port 9001 of the host to port 9000 of the container
@@ -38,8 +38,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
     platform: linux/amd64
     ports:
       - "9001:9000" # Map port 9002 of the host to port 9000 of the container
@@ -55,8 +55,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
     platform: linux/amd64
     ports:
       - "9002:9000" # Map port 9003 of the host to port 9000 of the container
@@ -72,8 +72,8 @@ services:
       - RUSTFS_VOLUMES=http://node{0...3}:9000/data/rustfs{0...3}
       - RUSTFS_ADDRESS=0.0.0.0:9000
       - RUSTFS_CONSOLE_ENABLE=true
-      - RUSTFS_ACCESS_KEY=${RUSTFS_ACCESS_KEY:-rustfsadmin-local}
-      - RUSTFS_SECRET_KEY=${RUSTFS_SECRET_KEY:-rustfssecret-local}
+      - RUSTFS_ACCESS_KEY=rustfsadmin
+      - RUSTFS_SECRET_KEY=rustfsadmin
     platform: linux/amd64
     ports:
       - "9003:9000" # Map port 9004 of the host to port 9000 of the container

.docker/observability/README.md

@@ -45,7 +45,6 @@ Three pre-built Grafana dashboards are included for monitoring RustFS GET perfor
 | **GET Rollout Health** | `grafana-get-rollout-health.json` | Monitors optimization rollout: latency by reader path, early-stop hit rate, codec streaming usage, pipeline failures |
 | **GET Data Integrity** | `grafana-get-data-integrity.json` | Monitors data safety: bitrot verify failures, decode errors, short reads, shard read outcomes |
 | **GET Resource Impact** | `grafana-get-resource-impact.json` | Monitors resource usage: concurrent requests, IO queue utilization, disk permit wait, RSS trend |
-| **Object Data Cache** | `grafana-object-data-cache.json` | Monitors the GET body cache (`rustfs_object_data_cache_*`): hit ratio, lookup/plan/fill outcomes, fill duration quantiles, hit vs fill throughput, entries/weighted bytes, inflight fills, memory-pressure skips, invalidations, and size-class breakdowns |
 
 ### Prometheus Alert Rules
 

.docker/observability/README_ZH.md

@@ -45,7 +45,6 @@
 | **GET 发布健康度** | `grafana-get-rollout-health.json` | 监控优化发布：按 reader path 的延迟、early-stop 命中率、codec streaming 使用率、pipeline 失败率 |
 | **GET 数据完整性** | `grafana-get-data-integrity.json` | 监控数据安全：bitrot 校验失败、decode 错误、short read、shard 读取结果 |
 | **GET 资源影响** | `grafana-get-resource-impact.json` | 监控资源使用：并发请求数、IO 队列利用率、disk permit 等待、RSS 趋势 |
-| **对象数据缓存** | `grafana-object-data-cache.json` | 监控 GET body 缓存（`rustfs_object_data_cache_*`）：命中率、查找/规划/填充结果、填充耗时分位、命中 vs 填充吞吐、条目数/加权字节、在途填充、内存压力拒绝、失效、按尺寸档拆分 |
 
 ### Prometheus 告警规则
 

.github/AGENTS.md

@@ -1,28 +1,21 @@
 # GitHub Workflow Instructions
 
-Applies to `.github/`.
+Applies to `.github/` and repository pull-request operations.
 
 ## Pull Requests
 
-PR conventions (English title/body, template usage, `--body-file` with the
-heredoc pattern, review-thread etiquette) live in the root `AGENTS.md` under
-"Git and PR Baseline" — that section is the single normative home; do not
-duplicate its rules here.
-
-## Workflow Changes
-
-- Any change touching `.github/workflows/` must pass `actionlint` (run from
-  the repo root) with zero findings before commit/PR. Install via
-  `brew install actionlint`, or the download script in the actionlint repo
-  (rhysd/actionlint); `make pre-pr` does not cover workflow files, so this is
-  the required check for them.
-- Custom self-hosted runner labels (`sm-standard-*`, `dind-*`) are declared
-  in `.github/actionlint.yaml`. When introducing a new `runs-on:` label,
-  declare it there in the same change. Never use the config or `-ignore` to
-  silence real findings — fix the workflow (root `AGENTS.md`: make checks
-  pass by fixing the cause, not by weakening the gate).
-- If `shellcheck` is installed, actionlint also lints `run:` scripts; treat
-  those findings as part of the gate.
+- PR titles and descriptions must be in English.
+- Use `.github/pull_request_template.md` for every PR body.
+- Keep all template section headings.
+- Use `N/A` for non-applicable sections.
+- Include verification commands in the PR details.
+- For `gh pr create` and `gh pr edit`, always write markdown body to a file and pass `--body-file`.
+- Do not use multiline inline `--body`; backticks and shell expansion can corrupt content or trigger unintended commands.
+- Recommended pattern:
+  - `cat > /tmp/pr_body.md <<'EOF'`
+  - `...markdown...`
+  - `EOF`
+  - `gh pr create ... --body-file /tmp/pr_body.md`
 
 ## CI Alignment
 

.github/actionlint.yaml

@@ -1,7 +0,0 @@
-self-hosted-runner:
-  # Custom labels of this repository's self-hosted runners. Keep in sync with
-  # the labels actually used in `runs-on:` across .github/workflows/.
-  labels:
-    - sm-standard-2
-    - sm-standard-4
-    - dind-sm-standard-2

.github/actions/schedule-failure-issue/README.md

@@ -1,74 +0,0 @@
-# schedule-failure-issue
-
-Opens (or updates) a GitHub issue when a **scheduled** workflow run fails.
-This is the single alerting mechanism for all timed pipelines
-(rustfs/backlog#1149 ci-8, arbitration G3): scheduled workflows must consume
-this action instead of inventing their own notification paths.
-
-## Behavior
-
-- Issue title: `[scheduled-failure] <workflow name>` — the workflow name is
-  the dedupe key.
-- If an **open** issue with that exact title exists, the failure is appended
-  as a comment; otherwise a new issue is created (labeled `infrastructure` by
-  default). Closing the issue resets the cycle: the next failure opens a
-  fresh one.
-- The issue body / comment includes the run URL, run attempt, event, ref and
-  the names of the failed (or timed-out/cancelled) jobs of the current run
-  attempt.
-- Requires `gh` and `jq` on the runner (both preinstalled on GitHub-hosted
-  runners such as `ubuntu-latest`).
-
-## Usage
-
-Add a final job to the workflow. `always()` is required — without it the job
-is skipped when a needed job fails; `contains(needs.*.result, 'failure')`
-makes it act only when something actually failed. Scope `issues: write` to
-this job only; no other job may gain permissions.
-
-```yaml
-  alert-on-failure:
-    name: Alert on scheduled failure
-    needs: [<the jobs to watch>]
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    permissions:
-      contents: read
-      issues: write
-    steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-      - name: Open or update failure-tracking issue
-        uses: ./.github/actions/schedule-failure-issue
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-```
-
-Notes:
-
-- Upstream jobs that are *cancelled* (e.g. a manually cancelled run) do not
-  trigger the alert — a human was already looking. Job `timeout-minutes`
-  expiry surfaces as `failure` and does alert.
-- Manual `workflow_dispatch` runs never alert; a human is watching those.
-
-## Current consumers
-
-- `.github/workflows/e2e-s3tests.yml` (weekly full s3-tests sweep)
-- `.github/workflows/mint.yml` (weekly multi-SDK mint run)
-- `.github/workflows/fuzz.yml` (nightly fuzz corpus jobs)
-- `.github/workflows/performance-ab.yml` (nightly warp A/B gate)
-- **Pending:** the ci-7 e2e nightly-full workflow does not exist yet
-  (backlog#1149 ci-7). When it lands, wire it up with the snippet above.
-
-## Drill
-
-`.github/workflows/schedule-failure-alert-drill.yml` is a manual-only
-workflow that forces a job failure and runs this action through the exact
-consumer wiring. Use it to verify the alert path end to end:
-
-```bash
-gh workflow run schedule-failure-alert-drill.yml -R rustfs/rustfs --ref <branch>
-```
-
-Run it twice to verify both paths (issue creation, then comment dedupe), and
-close the `[scheduled-failure] Schedule Failure Alert Drill` issue afterwards.

.github/actions/schedule-failure-issue/action.yml

@@ -1,104 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-name: "Schedule Failure Issue"
-description: >-
-  Open (or update) a tracking issue when a scheduled workflow run fails.
-  Dedupes by workflow name: if an open issue titled
-  "[scheduled-failure] <workflow name>" already exists, the failure is
-  appended as a comment; otherwise a new issue is created. This is the
-  single alerting mechanism for all scheduled pipelines (backlog#1149 ci-8).
-
-inputs:
-  github-token:
-    description: >-
-      Token with issues:write on the repository. Pass secrets.GITHUB_TOKEN
-      from a job that declares `permissions: issues: write` (scope the
-      permission to the alert job only, never workflow-wide).
-    required: true
-  workflow-name:
-    description: "Workflow name used for the issue title (the dedupe key)."
-    required: false
-    default: ${{ github.workflow }}
-  label:
-    description: >-
-      Label applied when a new issue is created. Must already exist in the
-      repository; if applying it fails, the issue is created without a label.
-      Set to an empty string to skip labeling.
-    required: false
-    default: "infrastructure"
-
-runs:
-  using: "composite"
-  steps:
-    - name: Open or update failure-tracking issue
-      shell: bash
-      env:
-        GH_TOKEN: ${{ inputs.github-token }}
-        WORKFLOW_NAME: ${{ inputs.workflow-name }}
-        ISSUE_LABEL: ${{ inputs.label }}
-      run: |
-        set -euo pipefail
-
-        title="[scheduled-failure] ${WORKFLOW_NAME}"
-        run_url="${GITHUB_SERVER_URL}/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}"
-
-        # Failed job names for this run attempt. The alert job runs while the
-        # run as a whole is still in progress, so inspect the jobs that have
-        # already completed with a non-success conclusion.
-        failed_jobs="$(gh api \
-          "repos/${GITHUB_REPOSITORY}/actions/runs/${GITHUB_RUN_ID}/attempts/${GITHUB_RUN_ATTEMPT}/jobs" \
-          --paginate \
-          --jq '.jobs[]
-                | select(.conclusion == "failure" or .conclusion == "timed_out" or .conclusion == "cancelled")
-                | "- `\(.name)` (\(.conclusion))"')"
-        if [ -z "${failed_jobs}" ]; then
-          failed_jobs="- (failed job not recorded yet — see the run page)"
-        fi
-
-        body="$(cat <<EOF
-        Scheduled run of **${WORKFLOW_NAME}** failed.
-
-        - Run: ${run_url} (attempt ${GITHUB_RUN_ATTEMPT})
-        - Event: \`${GITHUB_EVENT_NAME}\`
-        - Ref: \`${GITHUB_REF_NAME}\` @ \`${GITHUB_SHA}\`
-
-        Failed jobs:
-        ${failed_jobs}
-        EOF
-        )"
-
-        # Dedupe by exact title among OPEN issues (a closed issue means the
-        # earlier breakage was resolved; a new failure gets a fresh issue).
-        # GitHub search strips the [] characters, so search loosely and match
-        # the exact title with jq.
-        existing="$(gh issue list --repo "${GITHUB_REPOSITORY}" --state open --limit 100 \
-          --search "\"scheduled-failure\" in:title" --json number,title \
-          | jq -r --arg title "${title}" 'map(select(.title == $title)) | (.[0].number // empty)')"
-
-        if [ -n "${existing}" ]; then
-          echo "Appending comment to existing open issue #${existing}"
-          gh issue comment "${existing}" --repo "${GITHUB_REPOSITORY}" --body "${body}"
-          exit 0
-        fi
-
-        echo "Creating new issue: ${title}"
-        if [ -n "${ISSUE_LABEL}" ]; then
-          if gh issue create --repo "${GITHUB_REPOSITORY}" \
-               --title "${title}" --body "${body}" --label "${ISSUE_LABEL}"; then
-            exit 0
-          fi
-          echo "::warning::issue creation with label '${ISSUE_LABEL}' failed (missing label?); retrying without a label"
-        fi
-        gh issue create --repo "${GITHUB_REPOSITORY}" --title "${title}" --body "${body}"

.github/secret_scanning.yml

@@ -1,14 +0,0 @@
-# GitHub secret scanning path exclusions.
-# Secrets detected under these paths are auto-closed as "ignored by
-# configuration" and are not blocked by push protection.
-#
-# Deliberately NOT excluded: production source files (e.g. rustfs/src/**,
-# crates/*/src/**) even when a hit sits inside a #[cfg(test)] module or a
-# doc-comment example — excluding the file would also stop scanning the
-# production code in it. Close those alerts manually as "used in tests".
-paths-ignore:
-  - "crates/e2e_test/**" # dedicated e2e test crate, test credentials throughout
-  - "**/tests/**" # Rust integration-test dirs (crates/*/tests, rustfs/tests)
-  - "**/benches/**" # benchmark harnesses
-  - ".docker/**" # local docker-compose dev configs (e.g. mqtt vm.args cookie)
-  - ".vscode/**" # local editor launch configs

.github/workflows/audit.yml

@@ -35,17 +35,15 @@ on:
       - '.github/workflows/**'
       - 'scripts/security/check_workflow_pins.sh'
   schedule:
-    - cron: '0 3 * * 0' # Weekly on Sunday 03:00 UTC (staggered after the midnight ci/build crons)
+    - cron: '0 0 * * 0' # Weekly on Sunday at midnight UTC
   workflow_dispatch:
 
 permissions:
   contents: read
 
-# Event-scoped groups: merges to main must not cancel the weekly scheduled
-# run (same rationale as ci.yml).
 concurrency:
-  group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }}
-  cancel-in-progress: ${{ github.event_name != 'schedule' }}
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
 
 env:
   CARGO_TERM_COLOR: always
@@ -59,9 +57,32 @@ jobs:
       - name: Explain cancellation run
         run: echo "PR closed; this run only cancels older runs in the same concurrency group."
 
-  # RustSec advisory scanning is covered by the `advisories` check of
-  # cargo-deny below; a separate cargo-audit job would duplicate the same
-  # database lookup with a second ignore list to maintain.
+  security-audit:
+    name: Security Audit
+    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    runs-on: ubuntu-latest
+    timeout-minutes: 15
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Install cargo-audit
+        uses: taiki-e/install-action@bffeee26d4db9be238a4ea78d8826604ebcb594d # v2
+        with:
+          tool: cargo-audit
+
+      - name: Run security audit
+        run: |
+          cargo audit -D warnings --json | tee audit-results.json
+
+      - name: Upload audit results
+        if: always()
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: security-audit-results-${{ github.run_number }}
+          path: audit-results.json
+          retention-days: 30
+
   cargo-deny:
     name: Cargo Deny
     if: github.event_name != 'pull_request' || github.event.action != 'closed'
@@ -112,9 +133,4 @@ jobs:
         with:
           fail-on-severity: moderate
           allow-ghsas: GHSA-2f9f-gq7v-9h6m
-          # rustfs-uring is a first-party Apache-2.0 crate, now published on
-          # crates.io (backlog#1104) rather than pulled as a git dependency.
-          # Scope the allow to the exact pinned version so a version bump forces a
-          # conscious re-review of the license/provenance claim (backlog#1181).
-          allow-dependencies-licenses: pkg:cargo/rustfs-uring@0.1.0
           comment-summary-in-pr: always

.github/workflows/build.yml

@@ -21,11 +21,6 @@
 # 2. Upload binaries to OSS storage
 # 3. Trigger docker.yml to build and push images using the uploaded binaries
 #
-# Platform scope:
-# - Pushes to main (development builds) build the Linux targets only — they
-#   feed the dev download channel and Docker dev images. Tags, the weekly
-#   schedule and manual dispatch build the full platform matrix.
-#
 # Manual Parameters:
 # - build_docker: Build and push Docker images (default: true)
 # - platforms: Comma-separated platform IDs or 'all' (default: all)
@@ -51,7 +46,7 @@ on:
       - ".gitignore"
       - ".dockerignore"
   schedule:
-    - cron: "0 1 * * 0" # Weekly on Sunday 01:00 UTC (staggered after the ci.yml midnight cron)
+    - cron: "0 0 * * 0" # Weekly on Sunday at midnight UTC
   workflow_dispatch:
     inputs:
       build_docker:
@@ -92,6 +87,8 @@ jobs:
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+        with:
+          fetch-depth: 0
 
       - name: Determine build strategy
         id: check
@@ -128,7 +125,8 @@ jobs:
             version="dev-${short_sha}"
             echo "🛠️  Development build detected"
           elif [[ "${{ github.event_name }}" == "schedule" ]] || \
-               [[ "${{ github.event_name }}" == "workflow_dispatch" ]]; then
+               [[ "${{ github.event_name }}" == "workflow_dispatch" ]] || \
+               [[ "${{ contains(github.event.head_commit.message, '--build') }}" == "true" ]]; then
             # Scheduled or manual build
             should_build=true
             build_type="development"
@@ -152,7 +150,6 @@ jobs:
   # Build RustFS binaries
   prepare-platform-matrix:
     name: Prepare Platform Matrix
-    needs: build-check
     runs-on: ubuntu-latest
     outputs:
       matrix: ${{ steps.select.outputs.matrix }}
@@ -170,18 +167,6 @@ jobs:
             selected="all"
           fi
 
-          # Per-merge development builds only feed the dev download channel
-          # and the Docker dev images, which consume the Linux binaries; at
-          # the usual merge cadence most per-merge builds are cancelled by the
-          # next merge anyway. macOS and Windows stay covered by tag builds,
-          # the weekly scheduled build and manual dispatch.
-          if [[ "${selected}" == "all" \
-            && "${{ github.event_name }}" == "push" \
-            && "${{ needs.build-check.outputs.build_type }}" == "development" ]]; then
-            selected="linux-x86_64-musl,linux-aarch64-musl,linux-x86_64-gnu,linux-aarch64-gnu"
-            echo "Development (main push) build: restricting to Linux targets"
-          fi
-
           all='{"include":[
             {"target_id":"linux-x86_64-musl","os":"sm-standard-2","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":""},
             {"target_id":"linux-aarch64-musl","os":"sm-standard-2","target":"aarch64-unknown-linux-musl","cross":true,"platform":"linux","rustflags":""},
@@ -224,11 +209,11 @@ jobs:
     timeout-minutes: 90
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-      # Release binaries ship without dial9 telemetry and therefore do not need
-      # --cfg tokio_unstable. Telemetry builds opt in explicitly (see
-      # `make build-profiling`); crates/obs/build.rs fails the compile if the
-      # `dial9` feature is enabled without the flag.
-      RUSTFLAGS: "${{ matrix.rustflags }}"
+      # Always enable Tokio unstable features (required by dial9-tokio-telemetry).
+      # The RUSTFLAGS env var takes precedence over .cargo/config.toml [build] rustflags,
+      # so we must include --cfg tokio_unstable here explicitly; otherwise an empty
+      # RUSTFLAGS value would shadow the config-file flag and silently break tracing.
+      RUSTFLAGS: "--cfg tokio_unstable ${{ matrix.rustflags }}"
     strategy:
       fail-fast: false
       matrix: ${{ fromJson(needs.prepare-platform-matrix.outputs.matrix) }}
@@ -798,6 +783,12 @@ jobs:
             sha512sum *.zip > SHA512SUMS
           fi
 
+          # Create signature placeholder files
+          for file in *.zip; do
+            echo "# Signature for $file" > "${file}.asc"
+            echo "# GPG signature will be added in future versions" >> "${file}.asc"
+          done
+
           cd ..
           python3 scripts/security/generate_release_supply_chain_assets.py \
             --asset-dir ./release-assets \
@@ -835,12 +826,11 @@ jobs:
 
           echo "✅ All assets uploaded successfully"
 
-  # Update latest.json for stable releases only: prerelease tags (alpha/beta/
-  # rc) must never overwrite the stable version pointer.
+  # Update latest.json for stable releases only
   update-latest-version:
     name: Update Latest Version
     needs: [ build-check, upload-release-assets ]
-    if: startsWith(github.ref, 'refs/tags/') && needs.build-check.outputs.build_type == 'release'
+    if: startsWith(github.ref, 'refs/tags/')
     runs-on: ubuntu-latest
     steps:
       - name: Update latest.json

.github/workflows/ci-docs-only.yml

@@ -1,70 +0,0 @@
-# Copyright 2026 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Companion to ci.yml for the required "Test and Lint" status check.
-#
-# ci.yml skips docs-only pull requests via paths-ignore, but the branch
-# ruleset requires a check named "Test and Lint" — without this workflow a
-# docs-only PR would wait on that check forever. This workflow triggers on
-# exactly the paths ci.yml ignores and reports an instant success under the
-# same job name. Mixed PRs trigger both workflows and the real check still
-# gates: a required check with any failing run blocks the merge.
-# https://docs.github.com/en/repositories/configuring-branches-and-merges-in-your-repository/defining-the-mergeability-of-pull-requests/troubleshooting-required-status-checks#handling-skipped-but-required-checks
-#
-# Keep the paths list below in sync with the pull_request paths-ignore list
-# in ci.yml.
-
-name: Continuous Integration (docs only)
-
-on:
-  pull_request:
-    types: [ opened, synchronize, reopened ]
-    branches: [ main ]
-    paths:
-      - "**.md"
-      - "docs/**"
-      - "deploy/**"
-      - "scripts/dev_*.sh"
-      - "scripts/probe.sh"
-      - "LICENSE*"
-      - ".gitignore"
-      - ".dockerignore"
-      - "README*"
-      - "**/*.png"
-      - "**/*.jpg"
-      - "**/*.svg"
-      - ".github/workflows/build.yml"
-      - ".github/workflows/docker.yml"
-      - ".github/workflows/audit.yml"
-
-permissions:
-  contents: read
-
-jobs:
-  test-and-lint:
-    name: Test and Lint
-    runs-on: ubuntu-latest
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-
-      # Docs-only PRs skip the full code CI, but they are exactly where a
-      # planning-type document could be slipped in (git add -f bypasses
-      # .gitignore). Run the guard here so the required "Test and Lint" check
-      # stays meaningful for docs-only changes.
-      - name: Check no planning docs committed
-        run: ./scripts/check_no_planning_docs.sh
-
-      - name: Satisfy required check for docs-only changes
-        run: echo "Docs-only change — code CI is skipped by paths-ignore; planning-docs guard passed, reporting success for the required 'Test and Lint' check."

.github/workflows/ci.yml

@@ -33,11 +33,10 @@ on:
       - ".github/workflows/build.yml"
       - ".github/workflows/docker.yml"
       - ".github/workflows/audit.yml"
+      - ".github/workflows/performance.yml"
   pull_request:
     types: [ opened, synchronize, reopened, closed ]
     branches: [ main ]
-    # Keep this list in sync with the `paths` list in ci-docs-only.yml, which
-    # reports the required "Test and Lint" check for PRs skipped here.
     paths-ignore:
       - "**.md"
       - "docs/**"
@@ -54,6 +53,7 @@ on:
       - ".github/workflows/build.yml"
       - ".github/workflows/docker.yml"
       - ".github/workflows/audit.yml"
+      - ".github/workflows/performance.yml"
   merge_group:
     types: [ checks_requested ]
   schedule:
@@ -63,13 +63,9 @@ on:
 permissions:
   contents: read
 
-# Concurrency groups are scoped per event so different triggers never cancel
-# each other: PR pushes cancel the previous run of that PR, main pushes keep
-# latest-wins semantics among themselves, and scheduled runs always complete
-# (a shared group used to let every merge kill the weekly scheduled run).
 concurrency:
-  group: ${{ github.workflow }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.ref }}
-  cancel-in-progress: ${{ github.event_name != 'schedule' }}
+  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.ref }}
+  cancel-in-progress: true
 
 env:
   CARGO_TERM_COLOR: always
@@ -85,12 +81,33 @@ jobs:
       - name: Explain cancellation run
         run: echo "PR closed; this run only cancels older runs in the same concurrency group."
 
+  skip-check:
+    name: Skip Duplicate Actions
+    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    permissions:
+      actions: write
+      contents: read
+    runs-on: ubuntu-latest
+    outputs:
+      should_skip: ${{ steps.skip_check.outputs.should_skip }}
+    steps:
+      - name: Skip duplicate actions
+        id: skip_check
+        uses: fkirc/skip-duplicate-actions@f75f66ce1886f00957d99748a42c724f4330bdcf # v5
+        with:
+          concurrent_skipping: "same_content_newer"
+          cancel_others: true
+          paths_ignore: '["*.md", "docs/**", "deploy/**"]'
+          do_not_skip: '["workflow_dispatch", "schedule", "merge_group", "release", "push"]'
+
   typos:
     name: Typos
-    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    needs: skip-check
+    if: needs.skip-check.outputs.should_skip != 'true'
     runs-on: ubuntu-latest
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+      - uses: dtolnay/rust-toolchain@29eef336d9b2848a0b548edc03f92a220660cdb8 # stable
       - name: Typos check with custom config file
         uses: crate-ci/typos@37bb98842b0d8c4ffebdb75301a13db0267cef89 # master
 
@@ -98,7 +115,8 @@ jobs:
   # ~1 minute instead of waiting for the full test job.
   quick-checks:
     name: Quick Checks
-    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    needs: skip-check
+    if: needs.skip-check.outputs.should_skip != 'true'
     runs-on: ubuntu-latest
     timeout-minutes: 10
     steps:
@@ -128,18 +146,10 @@ jobs:
       - name: Check tokio io-uring feature guard
         run: ./scripts/check_no_tokio_io_uring.sh
 
-      - name: Check extension schema boundaries
-        run: ./scripts/check_extension_schema_boundaries.sh
-
-      - name: Check body-cache whitelist guard
-        run: ./scripts/check_body_cache_whitelist.sh
-
-      - name: Check no planning docs committed
-        run: ./scripts/check_no_planning_docs.sh
-
   test-and-lint:
     name: Test and Lint
-    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    needs: skip-check
+    if: needs.skip-check.outputs.should_skip != 'true'
     runs-on: sm-standard-4
     timeout-minutes: 60
     env:
@@ -156,84 +166,26 @@ jobs:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
-      # Clippy runs before the test pass: lint failures are the most common
-      # CI-only breakage and should surface in minutes, not after 20+ minutes
-      # of tests.
-      - name: Run clippy lints
-        run: cargo clippy --all-targets -- -D warnings
-
       - name: Run tests
         run: |
-          cargo nextest run --profile ci --all --exclude e2e_test
+          cargo nextest run --all --exclude e2e_test
           cargo test --all --doc
 
-      - name: Upload test junit report
-        if: always()
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: junit-test-and-lint-${{ github.run_number }}
-          path: target/nextest/ci/junit.xml
-          retention-days: 3
-          if-no-files-found: ignore
-
       # Explicit gate for migration-critical suites. These tests already ran in
       # the full nextest pass above; a single filtered nextest invocation keeps
       # the named gate without rebuilding or re-running them one package at a time.
-      #
-      # The gate selects tests by name substring (data_movement / rebalance /
-      # decommission / source_cleanup / delete_marker), so renames can silently
-      # thin it. The script owns the filter expression and first verifies the
-      # selected-test count against the committed floor in
-      # .config/migration-gate-floor.txt before running the gate; renames or
-      # removals must update that file consciously (see the script header).
-      # Kept on the default profile (no --profile ci): a second --profile ci run
-      # would clobber target/nextest/ci/junit.xml, and none of these tests are
-      # quarantined so they gain nothing from the ci profile's retry overrides.
       - name: Run rebalance/decommission migration proofs
-        run: ./scripts/check_migration_gate_count.sh
-
-  # Dedicated serial lane for the ILM / lifecycle integration tests. These tests
-  # drive the object layer through process-global singletons (the GLOBAL_ENV
-  # ECStore, the global tier-config manager, background-expiry workers) and bind
-  # fixed ports (9002 for the scanner suite, 9003 for the app suite), so they are
-  # marked #[ignore] to keep them out of the default parallel `cargo nextest run
-  # --all` pass. Run them here explicitly with `--run-ignored ignored-only` and
-  # `-j1` so no two of them share global state or race for a port at once.
-  # serial_test's #[serial] does NOT serialize across nextest's process-per-test
-  # boundary (see .config/nextest.toml); `-j1` is what actually serializes them.
-  # See rustfs/backlog#1148 (ilm-1) and #1155.
-  test-ilm-integration-serial:
-    name: ILM Integration (serial)
-    if: github.event_name != 'pull_request' || github.event.action != 'closed'
-    runs-on: sm-standard-4
-    timeout-minutes: 45
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-ilm-serial
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
-
-      # Three scanner tests fail on main independently of this lane (restore of
-      # a transitioned multipart object, noncurrent transition/expiry after an
-      # immediate compensation transition); they keep #[ignore] with a backlog
-      # reference and are excluded here by name until fixed (rustfs/backlog#1148).
-      - name: Run ignored ILM integration tests serially
         run: |
-          cargo nextest run -j1 --run-ignored ignored-only \
-            -p rustfs-scanner -p rustfs \
-            -E '(binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))) and not (test(test_transition_and_restore_flows) or test(test_noncurrent_expiry_still_works_after_immediate_compensation_transition) or test(test_noncurrent_transition_still_works_after_immediate_compensation_transition))'
+          cargo nextest run -p rustfs-ecstore --lib \
+            -E 'test(data_movement) or test(rebalance) or test(decommission) or test(source_cleanup) or test(delete_marker)'
+
+      - name: Run clippy lints
+        run: cargo clippy --all-targets -- -D warnings
 
   test-and-lint-rio-v2:
     name: Test and Lint (rio-v2)
-    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    needs: skip-check
+    if: needs.skip-check.outputs.should_skip != 'true'
     runs-on: sm-standard-4
     timeout-minutes: 60
     env:
@@ -250,17 +202,18 @@ jobs:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
-      - name: Run rio-v2 clippy lints
-        run: cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
-
       - name: Run rio-v2 feature tests
         run: |
           cargo nextest run -p rustfs -p rustfs-ecstore --features rio-v2
           cargo test -p rustfs --doc --features rio-v2
 
+      - name: Run rio-v2 clippy lints
+        run: cargo clippy -p rustfs -p rustfs-ecstore --all-targets --features rio-v2 -- -D warnings
+
   test-and-lint-protocols:
     name: "Test and Lint (${{ matrix.features.name }})"
-    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    needs: skip-check
+    if: needs.skip-check.outputs.should_skip != 'true'
     runs-on: sm-standard-4
     timeout-minutes: 60
     strategy:
@@ -285,17 +238,18 @@ jobs:
           github-token: ${{ secrets.GITHUB_TOKEN }}
           cache-save-if: ${{ github.ref == 'refs/heads/main' }}
 
-      - name: Run clippy with ${{ matrix.features.name }}
-        run: |
-          cargo clippy -p rustfs -p rustfs-protocols --all-targets ${{ matrix.features.flags }} -- -D warnings
-
       - name: Run tests with ${{ matrix.features.name }}
         run: |
           cargo nextest run -p rustfs -p rustfs-protocols ${{ matrix.features.flags }}
 
+      - name: Run clippy with ${{ matrix.features.name }}
+        run: |
+          cargo clippy -p rustfs -p rustfs-protocols --all-targets ${{ matrix.features.flags }} -- -D warnings
+
   build-rustfs-debug-binary:
     name: Build RustFS Debug Binary
-    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    needs: skip-check
+    if: needs.skip-check.outputs.should_skip != 'true'
     runs-on: sm-standard-4
     timeout-minutes: 30
     env:
@@ -325,7 +279,8 @@ jobs:
 
   build-rustfs-debug-binary-rio-v2:
     name: Build RustFS Debug Binary (rio-v2)
-    if: github.event_name != 'pull_request' || github.event.action != 'closed'
+    needs: skip-check
+    if: needs.skip-check.outputs.should_skip != 'true'
     runs-on: sm-standard-4
     timeout-minutes: 30
     env:
@@ -353,67 +308,17 @@ jobs:
           if-no-files-found: error
           retention-days: 1
 
-  uring-integration:
-    name: io_uring Integration (real)
-    # GitHub-hosted ubuntu-latest runs a recent kernel with io_uring and, unlike
-    # a container, applies no seccomp filter that would block io_uring_setup — so
-    # the probe succeeds and the tests exercise the real UringBackend/FdCache/
-    # latch paths instead of the StdBackend fallback (rustfs/backlog#1179). The
-    # self-hosted sm-standard runners cannot guarantee this.
-    runs-on: ubuntu-latest
-    timeout-minutes: 30
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-uring
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
-
-      - name: Install build dependencies
-        run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
-
-      # ext4 supports O_DIRECT; the runner's default TMPDIR may sit on tmpfs or
-      # overlayfs, where open(O_DIRECT) returns EINVAL/EOPNOTSUPP and the native
-      # read_at_direct path silently latches off to the aligned StdBackend
-      # fallback. A dedicated ext4 loopback guarantees the native O_DIRECT read
-      # path is actually exercised, not just the fallback (rustfs/backlog#1220).
-      - name: Prepare ext4 loopback for O_DIRECT coverage
-        run: |
-          set -euo pipefail
-          dd if=/dev/zero of=/tmp/rustfs-odirect.img bs=1M count=1024
-          mkfs.ext4 -q -F /tmp/rustfs-odirect.img
-          sudo mkdir -p /mnt/rustfs-odirect
-          sudo mount -o loop /tmp/rustfs-odirect.img /mnt/rustfs-odirect
-          sudo chmod 1777 /mnt/rustfs-odirect
-          mount | grep rustfs-odirect
-
-      # RUSTFS_URING_TESTS_MUST_RUN=1 makes the io_uring tests fail instead of
-      # silently skipping if io_uring is unavailable, so this leg can never pass
-      # vacuously. RUSTFS_IO_URING_READ_ENABLE=true selects the UringBackend.
-      # TMPDIR points every tempfile-based test fixture at the ext4 loopback so
-      # eligible reads keep O_DIRECT and the native path is covered end to end.
-      - name: Run ecstore io_uring integration tests on real io_uring
-        env:
-          RUSTFS_IO_URING_READ_ENABLE: "true"
-          RUSTFS_URING_TESTS_MUST_RUN: "1"
-          TMPDIR: /mnt/rustfs-odirect
-        run: cargo test -p rustfs-ecstore uring_ -- --test-threads=1 --nocapture
-
   e2e-tests:
     name: End-to-End Tests
-    needs: [ build-rustfs-debug-binary ]
+    needs: [ skip-check, build-rustfs-debug-binary ]
+    if: needs.skip-check.outputs.should_skip != 'true'
     runs-on: sm-standard-2
     timeout-minutes: 30
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
 
-      # Full setup with dependency caching: the smoke-suite step below
+      # Full setup with dependency caching: the migration-proof step below
       # compiles the e2e_test crate, which pulls in most of the workspace.
       # Without a restored cache this was a cold multi-GB build on every run.
       - name: Setup Rust environment
@@ -435,13 +340,8 @@ jobs:
       - name: Make binary executable
         run: chmod +x ./target/debug/rustfs
 
-      # PR smoke subset of the in-repo e2e suite (backlog#1149 ci-4). The
-      # profile.e2e-smoke default-filter in .config/nextest.toml is the single
-      # wiring mechanism for e2e tests in CI — extend that filter instead of
-      # adding new e2e jobs here. Each test spawns its own rustfs server on a
-      # random port and reuses the downloaded debug binary above.
-      - name: Run e2e smoke suite
-        run: cargo nextest run --profile e2e-smoke -p e2e_test
+      - name: Run delete-marker migration proof
+        run: cargo test -p e2e_test delete_marker_migration_semantics -- --nocapture --test-threads=1
 
       - name: Install s3s-e2e test tool
         uses: taiki-e/cache-cargo-install-action@7447f04c51f2ba27ca35e7f1e28fab848c5b3ba7 # v2
@@ -470,7 +370,8 @@ jobs:
 
   e2e-tests-rio-v2:
     name: End-to-End Tests (rio-v2)
-    needs: [ build-rustfs-debug-binary-rio-v2 ]
+    needs: [ skip-check, build-rustfs-debug-binary-rio-v2 ]
+    if: needs.skip-check.outputs.should_skip != 'true'
     runs-on: sm-standard-2
     timeout-minutes: 30
     steps:
@@ -516,7 +417,8 @@ jobs:
 
   s3-implemented-tests:
     name: S3 Implemented Tests
-    needs: [ build-rustfs-debug-binary, e2e-tests ]
+    needs: [ skip-check, build-rustfs-debug-binary, e2e-tests ]
+    if: needs.skip-check.outputs.should_skip != 'true'
     runs-on: sm-standard-4
     timeout-minutes: 60
     steps:
@@ -555,77 +457,3 @@ jobs:
           path: artifacts/s3tests-single/**
           if-no-files-found: ignore
           retention-days: 3
-
-  # Dedicated lifecycle behavior lane (backlog#1148 ilm-10, master plan #1155).
-  #
-  # These ceph/s3-tests cases assert that objects/versions/uploads are actually
-  # removed by the background scanner and the stale-multipart cleanup loop, so
-  # they need a debug-accelerated day plus an enabled scanner. That configuration
-  # cannot be applied to the s3-implemented-tests lane above because a global
-  # RUSTFS_ILM_DEBUG_DAY_SECS also shrinks the x-amz-expiration header those tests
-  # assert on (test_lifecycle_expiration_header_*). Hence a separate server here.
-  #
-  # RUSTFS_ILM_DEBUG_DAY_SECS MUST stay equal to the s3-tests lc_debug_interval
-  # default (10s). test_lifecycle_expiration polls the Days=1 rule with a
-  # 4*lc_interval window while the Days=5 rule fires at 5*debug_day; observing
-  # the intermediate 4-object plateau requires 4*lc_interval < 5*debug_day, i.e.
-  # debug_day > 8. A smaller value lets the Days=5 rule fire inside the Days=1
-  # poll window so the count collapses straight to 2 and the test fails (#4764).
-  #
-  # RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES=1 is also required. A compacted directory
-  # is only re-descended (and its objects re-evaluated for ILM) once every
-  # DATA_USAGE_UPDATE_DIR_CYCLES scanner cycles (default 16). At the accelerated
-  # RUSTFS_SCANNER_CYCLE=2 that is ~32s between evaluations, so a Days=1 object
-  # that becomes due at debug_day (10s) is not actually expired until the next
-  # ~32s boundary (~42s), landing just past the 4*lc_interval (40s) poll window
-  # and making the count stall at 6 (#4767). Forcing every-cycle re-descent
-  # evaluates ILM within ~2s of the due time, well inside the poll window.
-  s3-lifecycle-behavior-tests:
-    name: S3 Lifecycle Behavior Tests
-    needs: [ build-rustfs-debug-binary ]
-    runs-on: sm-standard-4
-    timeout-minutes: 30
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-
-      - name: Download debug binary
-        uses: actions/download-artifact@37930b1c2abaa49bbe596cd826c3c89aef350131 # v7
-        with:
-          name: rustfs-debug-binary
-          path: target/debug
-
-      - name: Make binary executable
-        run: chmod +x ./target/debug/rustfs
-
-      - name: Run lifecycle behavior s3-tests
-        run: |
-          RUN_ROOT="${RUNNER_TEMP}/rustfs-s3tests-${GITHUB_RUN_ID}-${GITHUB_RUN_ATTEMPT}-${GITHUB_JOB}"
-          mkdir -p "${RUN_ROOT}"
-          S3_PORT="$(python3 -c 'import socket; s=socket.socket(); s.bind(("127.0.0.1", 0)); print(s.getsockname()[1]); s.close()')"
-          DEPLOY_MODE=binary \
-          RUSTFS_BINARY=./target/debug/rustfs \
-          TEST_MODE=single \
-          MAXFAIL=0 \
-          XDIST=0 \
-          IMPLEMENTED_TESTS_FILE=scripts/s3-tests/lifecycle_behavior_tests.txt \
-          RUSTFS_ILM_DEBUG_DAY_SECS=10 \
-          RUSTFS_ILM_PROCESS_TIME=1 \
-          RUSTFS_SCANNER_ENABLED=true \
-          RUSTFS_SCANNER_CYCLE=2 \
-          RUSTFS_DATA_USAGE_UPDATE_DIR_CYCLES=1 \
-          RUSTFS_SCANNER_START_DELAY_SECS=0 \
-          RUSTFS_API_STALE_UPLOADS_CLEANUP_INTERVAL=2s \
-          S3_PORT="${S3_PORT}" \
-          DATA_ROOT="${RUN_ROOT}" \
-          S3TESTS_CONF=artifacts/s3tests-single/s3tests.conf \
-          ./scripts/s3-tests/run.sh
-
-      - name: Upload s3 test artifacts
-        if: always()
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: s3tests-lifecycle-behavior-${{ github.run_number }}
-          path: artifacts/s3tests-single/**
-          if-no-files-found: ignore
-          retention-days: 3

.github/workflows/docker.yml

@@ -70,19 +70,9 @@ env:
   DOCKER_PLATFORMS: linux/amd64,linux/arm64
 
 jobs:
-  # Check if we should build Docker images.
-  # Short-circuit at job level so per-merge development builds (workflow_run
-  # from a push to main) skip the whole run without spawning a checkout job:
-  # images are only published for tag builds — build.yml push triggers cover
-  # only main and tags, so a non-main head_branch is a tag name — and for
-  # manual dispatch.
+  # Check if we should build Docker images
   build-check:
     name: Docker Build Check
-    if: >-
-      github.event_name == 'workflow_dispatch' ||
-      (github.event.workflow_run.conclusion == 'success' &&
-       github.event.workflow_run.event == 'push' &&
-       github.event.workflow_run.head_branch != 'main')
     runs-on: ubuntu-latest
     outputs:
       should_build: ${{ steps.check.outputs.should_build }}
@@ -96,6 +86,7 @@ jobs:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
         with:
+          fetch-depth: 0
           # For workflow_run events, checkout the specific commit that triggered the workflow
           ref: ${{ github.event.workflow_run.head_sha || github.sha }}
 
@@ -430,9 +421,6 @@ jobs:
     needs: [ build-check, build-docker ]
     if: needs.build-check.outputs.should_build == 'true' && needs.build-check.outputs.should_push == 'true'
     runs-on: ubuntu-latest
-    permissions:
-      contents: read
-      security-events: write
     strategy:
       fail-fast: false
       matrix:
@@ -462,16 +450,9 @@ jobs:
           severity: CRITICAL,HIGH
           exit-code: "0"
 
-      # Surface findings in the Security tab; an artifact alone is write-only
-      # reporting nobody reviews.
-      - name: Upload scan results to code scanning
-        uses: github/codeql-action/upload-sarif@99df26d4f13ea111d4ec1a7dddef6063f76b97e9 # v4.37.0
-        with:
-          sarif_file: trivy-${{ matrix.variant }}.sarif
-          category: container-image-${{ matrix.variant }}
-
       - name: Upload container scan report
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        # actions/upload-artifact v7.0.1
+        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a
         with:
           name: container-image-scan-${{ matrix.variant }}
           path: trivy-${{ matrix.variant }}.sarif

.github/workflows/e2e-replication-nightly.yml

@@ -1,121 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Nightly full replication e2e lane (backlog#1147 repl-1, deps: ci-4).
-#
-# The per-PR gate (ci.yml `e2e-tests` job, `--profile e2e-smoke`) runs the 20
-# FAST bucket-replication tests. This scheduled lane runs the remaining 18
-# heavier replication e2e tests that are unfit for a per-PR gate:
-#
-#   * 8 bucket-replication data-plane tests (PUT/delete + poll for convergence;
-#     two replicate over HTTPS).
-#   * 9 `_real_dual_node` site-replication tests (each spawns TWO rustfs
-#     servers and drives the cross-process site-replication control plane).
-#   * 1 `_real_single_node` service-account round-trip test.
-#
-# The selection is the [profile.e2e-repl-nightly] default-filter in
-# .config/nextest.toml — the single wiring mechanism (repl-1 / ci-4). Do NOT
-# add ad-hoc cargo-test steps here; change the filterset instead.
-#
-# Explicit division of labor: these 18 tests run ONLY here, never double-run
-# in ci-5's future e2e-full merge gate. TODO(ci-7): once the ci domain's
-# consolidated scheduled e2e workflow exists, fold this interim repl-owned lane
-# into it rather than growing a second scheduled entrypoint.
-
-name: e2e-replication-nightly
-
-on:
-  workflow_dispatch:
-  schedule:
-    # 04:00 UTC nightly — staggered clear of fuzz/e2e-s3tests (02:00),
-    # stale (01:30) and performance-ab (06:00).
-    - cron: "0 4 * * *"
-
-# Only alert-on-failure needs more than read access; it declares its own
-# job-level `issues: write`.
-permissions:
-  contents: read
-
-jobs:
-  repl-nightly:
-    name: Replication e2e (nightly)
-    # dual-node tests spawn two full rustfs servers each; the extra headroom
-    # over the per-PR sm-standard-2 e2e job keeps the parallel process fan-out
-    # from starving the runner. Mirrors the ILM serial lane's runner class.
-    runs-on: sm-standard-4
-    timeout-minutes: 45
-    env:
-      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-e2e-repl
-          github-token: ${{ secrets.GITHUB_TOKEN }}
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
-
-      # awscurl lets the STS dual-node test actually exercise its path. Without
-      # it the test skips gracefully with a visible log line
-      # (`awscurl_available()` in crates/e2e_test/src/common.rs), so the lane
-      # still passes — installing it just upgrades that one test from skip to
-      # real coverage.
-      - name: Set up Python
-        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
-        with:
-          python-version: "3.12"
-
-      - name: Install awscurl
-        run: python3 -m pip install --user --upgrade pip awscurl
-
-      # Build the rustfs binary once up front. The e2e tests spawn it as a
-      # child process (crates/e2e_test/src/common.rs) and will build it on
-      # demand otherwise, but a single explicit build avoids several parallel
-      # nextest test processes racing to build it at once.
-      - name: Build rustfs binary
-        run: cargo build -p rustfs --bins
-
-      - name: Run replication e2e nightly suite
-        run: cargo nextest run --profile e2e-repl-nightly -p e2e_test
-
-      - name: Upload nextest junit report
-        if: always()
-        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
-        with:
-          name: e2e-replication-nightly-junit-${{ github.run_number }}
-          path: target/nextest/e2e-repl-nightly/junit.xml
-          retention-days: 7
-          if-no-files-found: ignore
-
-  alert-on-failure:
-    name: Alert on scheduled failure
-    needs: [repl-nightly]
-    # Only scheduled runs open/append the tracking issue (backlog#1149 ci-8);
-    # manual workflow_dispatch runs stay quiet so a debugging run never files a
-    # spurious alert.
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    permissions:
-      contents: read
-      issues: write
-    steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-      - name: Open or update failure-tracking issue
-        uses: ./.github/actions/schedule-failure-issue
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/e2e-s3tests.yml

@@ -28,22 +28,6 @@
 # truth, also used locally and by the PR gate). This workflow only provisions
 # the server topology: a single node, or a real 4-node distributed cluster
 # (erasure-coded across nodes) behind an HAProxy round-robin load balancer.
-#
-# Runner infrastructure (ci-1, backlog#1149): this sweep needs BOTH a working
-# Docker daemon (to build the source image and run the single/multi topologies)
-# AND a Python toolchain with pip (for awscurl/tox and the pytest harness).
-# It previously ran on the self-hosted `sm-standard-4` label, which is served
-# by heterogeneous runners: some scheduled runs landed on minimal pods that
-# had neither `pip` (bare python3, `No module named pip`) nor `docker.sock`,
-# so "Install Python tools" died before any test executed (all 15 recorded
-# runs failed). Two changes make the infra assumptions explicit instead of
-# trusting drifting runner state:
-#   1. Pin to GitHub-hosted `ubuntu-latest`, which reliably ships Docker,
-#      docker compose, python3 and pip (no self-hosted fleet drift).
-#   2. Set up Python explicitly (actions/setup-python) before any pip usage,
-#      so the workflow stays correct even if the runner image changes.
-# The PR gate (ci.yml s3-implemented-tests) is unaffected: it avoids Docker
-# via DEPLOY_MODE=binary and defers all pip setup to run.sh's self-bootstrap.
 
 name: e2e-s3tests
 
@@ -85,8 +69,8 @@ on:
 
 env:
   # main user
-  S3_ACCESS_KEY: rustfsadmin-ci
-  S3_SECRET_KEY: rustfssecret-ci
+  S3_ACCESS_KEY: rustfsadmin
+  S3_SECRET_KEY: rustfsadmin
   # alt user (must be different from main for many s3-tests)
   S3_ALT_ACCESS_KEY: rustfsalt
   S3_ALT_SECRET_KEY: rustfsalt
@@ -109,22 +93,13 @@ concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}-${{ github.event.inputs['test-mode'] || 'single' }}
   cancel-in-progress: true
 
-# Only alert-on-failure needs more than read access; it declares its own
-# job-level `issues: write`.
-permissions:
-  contents: read
-
 defaults:
   run:
     shell: bash
 
 jobs:
   s3tests:
-    # GitHub-hosted: reliably provides Docker + docker compose + python3/pip.
-    # See the header note (ci-1) for why the self-hosted sm-standard-4 label
-    # was abandoned. TODO(ci-8): scheduled-failure alerting (auto-open issue)
-    # is added by the ci-8 composite action; do not implement it here.
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-4
     timeout-minutes: 180
     strategy:
       fail-fast: false
@@ -136,14 +111,6 @@ jobs:
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
 
-      # Provision Python explicitly rather than trusting the runner image to
-      # ship a working pip (ci-1: a bare python3 without pip is what broke the
-      # scheduled sweep). Keeps the workflow correct across runner drift.
-      - name: Set up Python
-        uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6.3.0
-        with:
-          python-version: "3.12"
-
       - name: Cache pip downloads
         uses: actions/cache@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
         with:
@@ -163,9 +130,9 @@ jobs:
       - name: Build RustFS image (source, cached)
         run: |
           DOCKER_BUILDKIT=1 docker buildx build --load \
-            --platform "${PLATFORM}" \
-            --cache-from "type=gha,scope=${BUILDX_CACHE_SCOPE}" \
-            --cache-to "type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE}" \
+            --platform ${PLATFORM} \
+            --cache-from type=gha,scope=${BUILDX_CACHE_SCOPE} \
+            --cache-to type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE} \
             -t rustfs-ci \
             -f Dockerfile.source .
 
@@ -174,18 +141,13 @@ jobs:
         run: |
           docker network inspect rustfs-net >/dev/null 2>&1 || docker network create rustfs-net
           docker rm -f rustfs-single >/dev/null 2>&1 || true
-          # The four disks share one physical device on the runner (a single
-          # loopback filesystem), so the local physical-disk-independence guard
-          # would refuse to start. Bypass it — this is the CI use case the guard
-          # explicitly sanctions via RUSTFS_UNSAFE_BYPASS_DISK_CHECK.
           docker run -d --name rustfs-single \
             --network rustfs-net \
-            -p "${S3_PORT}:9000" \
+            -p ${S3_PORT}:9000 \
             -e RUSTFS_ADDRESS=0.0.0.0:9000 \
-            -e RUSTFS_ACCESS_KEY="${S3_ACCESS_KEY}" \
-            -e RUSTFS_SECRET_KEY="${S3_SECRET_KEY}" \
+            -e RUSTFS_ACCESS_KEY=${S3_ACCESS_KEY} \
+            -e RUSTFS_SECRET_KEY=${S3_SECRET_KEY} \
             -e RUSTFS_VOLUMES="/data/rustfs{0...3}" \
-            -e RUSTFS_UNSAFE_BYPASS_DISK_CHECK=true \
             -v /tmp/rustfs-single:/data \
             rustfs-ci
 
@@ -205,10 +167,6 @@ jobs:
               RUSTFS_ACCESS_KEY: ${S3_ACCESS_KEY}
               RUSTFS_SECRET_KEY: ${S3_SECRET_KEY}
               RUSTFS_VOLUMES: "http://rustfs{1...4}:9000/data/rustfs{0...3}"
-              # Each node's four disks share one physical device inside its
-              # container, so bypass the local physical-disk-independence guard
-              # (the CI use case it explicitly sanctions).
-              RUSTFS_UNSAFE_BYPASS_DISK_CHECK: "true"
 
           services:
             rustfs1:
@@ -268,7 +226,7 @@ jobs:
 
       - name: Wait for RustFS ready
         run: |
-          for _ in {1..120}; do
+          for i in {1..120}; do
             if curl -sf "http://${S3_HOST}:${S3_PORT}/health" >/dev/null 2>&1; then
               echo "RustFS is ready"
               exit 0
@@ -339,22 +297,3 @@ jobs:
         with:
           name: s3tests-${{ env.TEST_MODE }}
           path: artifacts/**
-
-  alert-on-failure:
-    name: Alert on scheduled failure
-    needs: [s3tests]
-    # `always()` is required: without it this job is skipped when a needed
-    # job fails. Alerts only for scheduled runs (backlog#1149 ci-8) — manual
-    # dispatch failures are already watched by a human.
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    permissions:
-      contents: read
-      issues: write
-    steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-      - name: Open or update failure-tracking issue
-        uses: ./.github/actions/schedule-failure-issue
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/fuzz.yml

@@ -17,16 +17,13 @@ name: Fuzz
 on:
   pull_request:
     types: [ opened, synchronize, reopened, closed ]
-    # PR trigger is intentionally narrow: only changes to the fuzz harness
-    # itself gate a PR. Broad crate paths (ecstore/filemeta/utils/policy/…)
-    # are covered by the nightly `schedule` run below, which fuzzes against
-    # whatever landed on main. Widening these paths previously queued a
-    # ~45min fuzz-build on nearly every PR and is why this workflow was
-    # disabled; do not re-add crate paths here.
     paths:
       - "fuzz/**"
       - "scripts/fuzz/**"
       - ".github/workflows/fuzz.yml"
+      - "crates/ecstore/**"
+      - "crates/filemeta/**"
+      - "crates/utils/**"
   schedule:
     - cron: "0 2 * * *"
   workflow_dispatch:
@@ -51,8 +48,7 @@ concurrency:
 env:
   CARGO_TERM_COLOR: always
   CARGO_BUILD_TARGET: x86_64-unknown-linux-gnu
-  # Fuzz targets build without the dial9 feature, so tokio_unstable is not needed.
-  RUSTFLAGS: "-C target-feature=-crt-static"
+  RUSTFLAGS: "--cfg tokio_unstable -C target-feature=-crt-static"
 
 jobs:
   cancel-closed-pr-runs:
@@ -100,11 +96,7 @@ jobs:
         run: |
           binary_root="fuzz/prebuilt/${CARGO_BUILD_TARGET}/release"
           mkdir -p "${binary_root}"
-          # Keep this list in sync with the smoke/nightly matrices and
-          # scripts/fuzz/run.sh default target set (all 5 buildable bins in
-          # fuzz/Cargo.toml). The *_storage_api.rs files are `mod` submodules
-          # of their parent targets, not standalone bins.
-          for target in archive_extract bucket_validation local_metadata path_containment policy_ingress; do
+          for target in path_containment bucket_validation local_metadata; do
             install -Dm755 "fuzz/target/${CARGO_BUILD_TARGET}/release/${target}" "${binary_root}/${target}"
           done
 
@@ -113,11 +105,9 @@ jobs:
         with:
           name: fuzz-prebuilt-binaries-${{ github.run_number }}
           path: |
-            fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/archive_extract
+            fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/path_containment
             fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/bucket_validation
             fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/local_metadata
-            fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/path_containment
-            fuzz/prebuilt/${{ env.CARGO_BUILD_TARGET }}/release/policy_ingress
           if-no-files-found: error
           retention-days: 1
           compression-level: 0
@@ -136,10 +126,8 @@ jobs:
     timeout-minutes: 30
     strategy:
       fail-fast: false
-      # One job per target runs in parallel, so wall-clock time is per-target
-      # (~60s fuzz + download/chmod overhead), not the sum across the matrix.
       matrix:
-        target: [archive_extract, bucket_validation, local_metadata, path_containment, policy_ingress]
+        target: [path_containment, bucket_validation, local_metadata]
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
@@ -181,10 +169,6 @@ jobs:
   nightly-fuzz-corpus:
     name: "Nightly / ${{ matrix.target }}"
     needs: fuzz-build
-    # TODO(ci-8): when the schedule-failure-issue composite action lands,
-    # add a step here (or a dependent job) that opens/updates a GitHub issue
-    # on nightly failure. ci-8 is the single alerting mechanism for all
-    # scheduled workflows; do not self-roll alerting in this workflow.
     if: >
       github.event_name == 'schedule' ||
       (github.event_name == 'workflow_dispatch' &&
@@ -194,7 +178,7 @@ jobs:
     strategy:
       fail-fast: false
       matrix:
-        target: [archive_extract, bucket_validation, local_metadata, path_containment, policy_ingress]
+        target: [path_containment, bucket_validation, local_metadata]
     env:
       FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
     steps:
@@ -229,25 +213,3 @@ jobs:
             fuzz/corpus/${{ matrix.target }}/**
           if-no-files-found: ignore
           retention-days: 30
-
-  # ──────────────────────────────────────────────────────────────
-  # Nightly alerting: open/update a tracking issue on failure.
-  # ──────────────────────────────────────────────────────────────
-  alert-on-failure:
-    name: Alert on scheduled failure
-    needs: [fuzz-build, nightly-fuzz-corpus]
-    # `always()` is required: without it this job is skipped when a needed
-    # job fails. Alerts only for scheduled (nightly) runs (backlog#1149
-    # ci-8); PR and manual dispatch failures are already watched by a human.
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    permissions:
-      contents: read
-      issues: write
-    steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-      - name: Open or update failure-tracking issue
-        uses: ./.github/actions/schedule-failure-issue
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/mint.yml

@@ -26,24 +26,7 @@
 # the run red. The job fails only when mint produces no results at all
 # (infrastructure error). Once a suite passes reliably, tightening this into a
 # baseline gate (like implemented_tests.txt for s3-tests) is the natural next
-# step. Tightening this into a per-suite baseline gate is tracked as ci-3
-# (backlog#1149) and is intentionally NOT done here.
-#
-# Runner infrastructure (ci-2, backlog#1149): this job needs a working Docker
-# daemon to (a) build the RustFS source image with buildx and (b) run both the
-# RustFS server and the mint image. It previously ran on the self-hosted
-# `sm-standard-4` label, which is served by heterogeneous runners; its single
-# recorded run (28730530597, 2026-07-05) landed on a minimal pod with no
-# `/var/run/docker.sock`, so "Enable buildx" died at `docker buildx create`
-# ("failed to connect to the docker API at unix:///var/run/docker.sock") before
-# any test executed -- the same root cause ci-1 fixed for the weekly s3-tests
-# sweep. The fix is to pin to GitHub-hosted `ubuntu-latest`, which reliably
-# ships a running Docker daemon + buildx + python3, instead of trusting the
-# drifting self-hosted fleet. Tradeoff: the source Rust build is heavier on a
-# GitHub-hosted 4-vCPU runner than on a warm self-hosted host, but it stays
-# well inside the 120-minute budget and buys deterministic infra. The
-# docker-capable self-hosted `dind-sm-standard-2` label was the alternative but
-# has fewer cores and reintroduces fleet-state risk for no reliability gain.
+# step.
 
 name: mint
 
@@ -67,13 +50,12 @@ on:
         required: false
         default: "minio/mint:edge"
   schedule:
-    # Weekly, after the Sunday s3-tests full sweep (starts 02:00 UTC, up to
-    # 3h) has finished, so the two never contend for the same runner pool.
-    - cron: "0 6 * * 0"
+    # Weekly, after the Sunday s3-tests full sweep.
+    - cron: "0 4 * * 0"
 
 env:
-  S3_ACCESS_KEY: rustfsadmin-ci
-  S3_SECRET_KEY: rustfssecret-ci
+  S3_ACCESS_KEY: rustfsadmin
+  S3_SECRET_KEY: rustfsadmin
   S3_REGION: us-east-1
 
   RUST_LOG: info
@@ -82,39 +64,21 @@ env:
 
   MINT_SUITES: ${{ github.event.inputs.suites || '' }}
   MINT_MODE: ${{ github.event.inputs.mode || 'core' }}
-  # minio/mint:edge is a rolling tag, so an unpinned run is not reproducible and
-  # can go red purely from an upstream mint change (ci-2, backlog#1149). Pin the
-  # default to the linux/amd64 digest resolved on 2026-07-10. workflow_dispatch
-  # can still override via the `mint-image` input (e.g. to test a newer edge).
-  # Refresh: docker manifest inspect minio/mint:edge --verbose | grep digest
-  #   (or, without a docker daemon:)
-  #   TOKEN=$(curl -s "https://auth.docker.io/token?service=registry.docker.io&scope=repository:minio/mint:pull" | jq -r .token)
-  #   curl -sI -H "Authorization: Bearer $TOKEN" \
-  #     -H "Accept: application/vnd.docker.distribution.manifest.v2+json" \
-  #     https://registry-1.docker.io/v2/minio/mint/manifests/edge | grep -i docker-content-digest
-  MINT_IMAGE: ${{ github.event.inputs.mint-image || 'minio/mint:edge@sha256:08a05e68893c68be2a83b6f79556853ed6aa3c6c9e64c823a00853e4e55d2200' }}
+  # NOTE: minio/mint:edge is a rolling tag. If sweeps get noisy from upstream
+  # mint changes, pin this to a digest known to work.
+  MINT_IMAGE: ${{ github.event.inputs.mint-image || 'minio/mint:edge' }}
 
 concurrency:
   group: ${{ github.workflow }}-${{ github.ref }}
   cancel-in-progress: true
 
-# Only alert-on-failure needs more than read access; it declares its own
-# job-level `issues: write`.
-permissions:
-  contents: read
-
 defaults:
   run:
     shell: bash
 
 jobs:
   mint:
-    # GitHub-hosted: reliably provides a running Docker daemon + buildx +
-    # python3. See the header note (ci-2) for why the self-hosted sm-standard-4
-    # label was abandoned (no docker.sock on the pod its only run landed on).
-    # TODO(ci-8): scheduled-failure alerting (auto-open issue on a red weekly
-    # run) is added by the ci-8 composite action; do not implement it here.
-    runs-on: ubuntu-latest
+    runs-on: sm-standard-4
     timeout-minutes: 120
     steps:
       - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
@@ -125,9 +89,9 @@ jobs:
       - name: Build RustFS image (source, cached)
         run: |
           DOCKER_BUILDKIT=1 docker buildx build --load \
-            --platform "${PLATFORM}" \
-            --cache-from "type=gha,scope=${BUILDX_CACHE_SCOPE}" \
-            --cache-to "type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE}" \
+            --platform ${PLATFORM} \
+            --cache-from type=gha,scope=${BUILDX_CACHE_SCOPE} \
+            --cache-to type=gha,mode=max,scope=${BUILDX_CACHE_SCOPE} \
             -t rustfs-ci \
             -f Dockerfile.source .
 
@@ -139,15 +103,15 @@ jobs:
             --network rustfs-net \
             -p 9000:9000 \
             -e RUSTFS_ADDRESS=0.0.0.0:9000 \
-            -e RUSTFS_ACCESS_KEY="${S3_ACCESS_KEY}" \
-            -e RUSTFS_SECRET_KEY="${S3_SECRET_KEY}" \
+            -e RUSTFS_ACCESS_KEY=${S3_ACCESS_KEY} \
+            -e RUSTFS_SECRET_KEY=${S3_SECRET_KEY} \
             -e RUSTFS_VOLUMES="/data/rustfs{0...3}" \
             -v /tmp/rustfs-mint:/data \
             rustfs-ci
 
       - name: Wait for RustFS ready
         run: |
-          for _ in {1..60}; do
+          for i in {1..60}; do
             if curl -sf http://127.0.0.1:9000/health >/dev/null 2>&1; then
               echo "RustFS is ready"
               exit 0
@@ -243,22 +207,3 @@ jobs:
         with:
           name: mint
           path: artifacts/**
-
-  alert-on-failure:
-    name: Alert on scheduled failure
-    needs: [mint]
-    # `always()` is required: without it this job is skipped when a needed
-    # job fails. Alerts only for scheduled runs (backlog#1149 ci-8) — manual
-    # dispatch failures are already watched by a human.
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    permissions:
-      contents: read
-      issues: write
-    steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-      - name: Open or update failure-tracking issue
-        uses: ./.github/actions/schedule-failure-issue
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/nix-flake-update.yml

@@ -17,7 +17,7 @@ name: Update Nix Flake
 on:
   workflow_dispatch:
   schedule:
-    - cron: '0 5 * * 0' # Weekly on Sunday 05:00 UTC (staggered after the midnight ci/build crons)
+    - cron: '0 0 * * 0' # Weekly on Sundays
 
 permissions:
   contents: write

.github/workflows/performance-ab.yml

@@ -30,9 +30,9 @@ on:
   workflow_dispatch:
     inputs:
       duration:
-        description: "warp duration per round (short by default to fit the double-build budget)"
+        description: "warp duration per round"
         required: false
-        default: "12s"
+        default: "30s"
         type: string
       allow_regression:
         description: "Pass the gate despite a FAIL (deliberate tradeoff)"
@@ -46,13 +46,6 @@ permissions:
   contents: read
   pull-requests: write
 
-# Per-PR: a new push cancels the previous (up to 90-minute) A/B run instead of
-# stacking them. Nightly schedule and manual dispatch get a unique group and
-# always run to completion.
-concurrency:
-  group: ${{ github.workflow }}-${{ github.event.pull_request.number || github.run_id }}
-  cancel-in-progress: ${{ github.event_name == 'pull_request' }}
-
 env:
   CARGO_TERM_COLOR: always
   RUST_BACKTRACE: 1
@@ -60,21 +53,13 @@ env:
 jobs:
   warp-ab:
     name: Warp A/B budget gate
-    # Opt-in on PRs: only run when the `perf-ab` label is present, and for
-    # `labeled` events only when the label being added is `perf-ab` itself —
-    # adding an unrelated label to an opted-in PR must not re-run the gate.
-    # Always run on schedule / manual dispatch.
+    # Opt-in on PRs: only run when the `perf-ab` label is present. Always run on
+    # schedule / manual dispatch.
     if: >-
       github.event_name != 'pull_request' ||
-      (contains(github.event.pull_request.labels.*.name, 'perf-ab') &&
-       (github.event.action != 'labeled' || github.event.label.name == 'perf-ab'))
+      contains(github.event.pull_request.labels.*.name, 'perf-ab')
     runs-on: sm-standard-2
-    # Phase-0 stopgap: the baseline+candidate release double-build alone is
-    # ~65min on this runner, so 90min left no room for a real full-matrix
-    # measurement (the earlier nightly runs only ever failed *before*
-    # measuring). 120min gives the 24-cell short matrix headroom until perf-3
-    # caches the baseline binary and restores a tighter budget.
-    timeout-minutes: 120
+    timeout-minutes: 90
     steps:
       - name: Checkout repository
         uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
@@ -114,17 +99,7 @@ jobs:
         id: ab
         run: |
           set -euo pipefail
-          # Budget note: the baseline+candidate release double-build (~65 min,
-          # cached away later by perf-3) dominates the 90-min job, so the
-          # measurement runs a short warp matrix — duration/rounds/cooldown are
-          # kept small to fit all 24 cells (6 workloads x 2 phases x 2 drive-sync)
-          # under budget rather than dropping cells. --health-timeout 180 outlasts
-          # the server's own 120s startup-readiness budget, which is what the
-          # rig's previous 60s health poll undershot (the first two nightly
-          # failures). perf-6 will recalibrate these once the pipeline is green.
-          duration="${{ github.event.inputs.duration || '12s' }}"
-          args=(--baseline-ref origin/main
-            --duration "$duration" --rounds 2 --cooldown 5 --health-timeout 180)
+          args=(--baseline-ref origin/main --duration "${{ github.event.inputs.duration || '30s' }}")
           if [[ "${{ steps.exempt.outputs.allow_regression }}" == "true" ]]; then
             args+=(--allow-regression --exemption-reason "labeled perf-deliberate-tradeoff / dispatch override")
           fi
@@ -134,14 +109,7 @@ jobs:
           bash scripts/run_hotpath_warp_ab.sh "${args[@]}"
           echo "status=$?" >> "$GITHUB_OUTPUT"
           set -e
-          # Locate the newest run dir + gate.md for the summary/comment/artifact
-          # steps. On a startup failure there is no gate.md, but the run dir still
-          # holds server-logs/ for diagnosis.
-          # Run dirs are UTC-timestamp names (no special chars); ls is safe here.
-          # shellcheck disable=SC2012
-          run_dir="$(ls -td target/hotpath-ab/*/ 2>/dev/null | head -n1 || true)"
-          echo "run_dir=${run_dir%/}" >> "$GITHUB_OUTPUT"
-          # shellcheck disable=SC2012
+          # Locate the newest gate.md for the comment/artifact steps.
           gate_md="$(ls -t target/hotpath-ab/*/gate.md 2>/dev/null | head -n1 || true)"
           echo "gate_md=$gate_md" >> "$GITHUB_OUTPUT"
 
@@ -150,49 +118,8 @@ jobs:
         uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
         with:
           name: hotpath-warp-ab-${{ github.run_number }}
-          # Includes per-cell median_summary.csv / baseline_compare.csv, gate.md,
-          # and server-logs/ (rustfs.log + startup env per phase) so a failed run
-          # is diagnosable. Short retention: this is churny nightly debug data.
           path: target/hotpath-ab/
           if-no-files-found: warn
-          retention-days: 14
-
-      - name: Write gate summary
-        if: always()
-        run: |
-          set -euo pipefail
-          status="${{ steps.ab.outputs.status }}"
-          gate_md="${{ steps.ab.outputs.gate_md }}"
-          run_dir="${{ steps.ab.outputs.run_dir }}"
-          {
-            echo "## Hotpath warp A/B — run ${{ github.run_number }}"
-            echo
-            if [[ "$status" == "0" ]]; then
-              echo "Rig/gate exit: \`0\` (pass or warn)."
-            else
-              echo "Rig/gate exit: \`${status:-unknown}\` — **FAILED**."
-            fi
-            echo
-            if [[ -n "$gate_md" && -f "$gate_md" ]]; then
-              cat "$gate_md"
-            else
-              echo "No \`gate.md\` produced — the rig failed **before** the gate"
-              echo "(most likely server startup / health). Failing phase(s) below;"
-              echo "full logs in the \`hotpath-warp-ab-${{ github.run_number }}\` artifact."
-              if [[ -n "$run_dir" && -d "$run_dir/server-logs" ]]; then
-                for f in "$run_dir"/server-logs/*.log; do
-                  [[ -f "$f" ]] || continue
-                  echo
-                  echo "<details><summary>$(basename "$f")</summary>"
-                  echo
-                  echo '```'
-                  tail -n 30 "$f"
-                  echo '```'
-                  echo "</details>"
-                done
-              fi
-            fi
-          } >> "$GITHUB_STEP_SUMMARY"
 
       - name: Comment gate result on PR
         if: always() && github.event_name == 'pull_request' && steps.ab.outputs.gate_md != ''
@@ -201,38 +128,12 @@ jobs:
         run: |
           gh pr comment "${{ github.event.pull_request.number }}" --body-file "${{ steps.ab.outputs.gate_md }}"
 
-      # TODO(ci-8/perf-2): scheduled/dispatch failures are currently silent. Once
-      # ci-8 lands the .github/actions/schedule-failure-issue composite action,
-      # perf-2 adds a step here guarded by
-      #   if: failure() && github.event_name != 'pull_request'
-      # that calls it (label perf-nightly-failure, append to an existing open
-      # issue) instead of hand-rolling gh CLI dedup. Do not implement it here.
-
       - name: Enforce gate
         if: always()
         run: |
           status="${{ steps.ab.outputs.status }}"
           if [[ "$status" != "0" ]]; then
-            echo "::error::warp A/B budget gate failed (exit $status). See the step summary / PR comment / gate.md artifact." >&2
+            echo "::error::warp A/B budget gate failed (exit $status). See the PR comment / gate.md artifact." >&2
             exit "$status"
           fi
           echo "warp A/B budget gate passed."
-
-  alert-on-failure:
-    name: Alert on scheduled failure
-    needs: [warp-ab]
-    # `always()` is required: without it this job is skipped when a needed
-    # job fails. Alerts only for scheduled (nightly) runs (backlog#1149
-    # ci-8); PR and manual dispatch failures are already watched by a human.
-    if: always() && github.event_name == 'schedule' && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    permissions:
-      contents: read
-      issues: write
-    steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-      - name: Open or update failure-tracking issue
-        uses: ./.github/actions/schedule-failure-issue
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}

.github/workflows/performance.yml

@@ -0,0 +1,146 @@
+# Copyright 2024 RustFS Team
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+name: Performance Testing
+
+on:
+  push:
+    branches: [ main ]
+    paths:
+      - "**/*.rs"
+      - "**/Cargo.toml"
+      - "**/Cargo.lock"
+      - ".github/workflows/performance.yml"
+  workflow_dispatch:
+    inputs:
+      profile_duration:
+        description: "Profiling duration in seconds"
+        required: false
+        default: "120"
+        type: string
+
+permissions:
+  contents: read
+
+env:
+  CARGO_TERM_COLOR: always
+  RUST_BACKTRACE: 1
+
+jobs:
+  performance-profile:
+    name: Performance Profiling
+    runs-on: sm-standard-2
+    timeout-minutes: 30
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: nightly
+          cache-shared-key: perf-${{ hashFiles('**/Cargo.lock') }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+
+      - name: Install additional nightly components
+        run: rustup component add llvm-tools-preview
+
+      - name: Install samply profiler
+        uses: taiki-e/cache-cargo-install-action@7447f04c51f2ba27ca35e7f1e28fab848c5b3ba7 # v2
+        with:
+          tool: samply
+
+      - name: Configure kernel for profiling
+        run: echo '1' | sudo tee /proc/sys/kernel/perf_event_paranoid
+
+      - name: Prepare test environment
+        run: |
+          # Create test volumes
+          for i in {0..4}; do
+            mkdir -p ./target/volume/test$i
+          done
+
+          # Set environment variables
+          echo "RUSTFS_VOLUMES=./target/volume/test{0...4}" >> $GITHUB_ENV
+          echo "RUST_LOG=rustfs=info,ecstore=info,s3s=info,iam=info,rustfs-obs=info" >> $GITHUB_ENV
+
+      - name: Verify console static assets
+        run: |
+          # Console static assets are already embedded in the repository
+          echo "Console static assets size: $(du -sh rustfs/static/)"
+          echo "Console static assets are embedded via rust-embed, no external download needed"
+
+      - name: Build with profiling optimizations
+        run: |
+          RUSTFLAGS="-C force-frame-pointers=yes -C debug-assertions=off --cfg tokio_unstable" \
+          cargo +nightly build --profile profiling -p rustfs --bins
+
+      - name: Run performance profiling
+        id: profiling
+        run: |
+          DURATION="${{ github.event.inputs.profile_duration || '120' }}"
+          echo "Running profiling for ${DURATION} seconds..."
+
+          timeout "${DURATION}s" samply record \
+            --output samply-profile.json \
+            ./target/profiling/rustfs ${RUSTFS_VOLUMES} || true
+
+          if [ -f "samply-profile.json" ]; then
+            echo "profile_generated=true" >> $GITHUB_OUTPUT
+            echo "Profile generated successfully"
+          else
+            echo "profile_generated=false" >> $GITHUB_OUTPUT
+            echo "::warning::Profile data not generated"
+          fi
+
+      - name: Upload profile data
+        if: steps.profiling.outputs.profile_generated == 'true'
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: performance-profile-${{ github.run_number }}
+          path: samply-profile.json
+          retention-days: 30
+
+  benchmark:
+    name: Benchmark Tests
+    runs-on: sm-standard-2
+    timeout-minutes: 45
+    env:
+      FORCE_JAVASCRIPT_ACTIONS_TO_NODE24: "true"
+    steps:
+      - name: Checkout repository
+        uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+
+      - name: Setup Rust environment
+        uses: ./.github/actions/setup
+        with:
+          rust-version: stable
+          cache-shared-key: bench-${{ hashFiles('**/Cargo.lock') }}
+          github-token: ${{ secrets.GITHUB_TOKEN }}
+          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
+
+      - name: Run benchmarks
+        run: |
+          cargo bench --package ecstore --bench comparison_benchmark -- --output-format json | \
+          tee benchmark-results.json
+
+      - name: Upload benchmark results
+        uses: actions/upload-artifact@b7c566a772e6b6bfb58ed0dc250532a479d7789f # v6
+        with:
+          name: benchmark-results-${{ github.run_number }}
+          path: benchmark-results.json
+          retention-days: 7

.github/workflows/schedule-failure-alert-drill.yml

@@ -1,62 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Manual drill for the scheduled-failure alerting path (backlog#1149 ci-8).
-#
-# Forces a job failure and then runs .github/actions/schedule-failure-issue
-# through the exact `needs` + `always() && contains(needs.*.result,
-# 'failure')` wiring used by the real consumers (e2e-s3tests, mint, fuzz,
-# performance-ab). Dispatch it twice to verify both the issue-creation and
-# the dedupe-comment paths, then close the resulting
-# "[scheduled-failure] Schedule Failure Alert Drill" issue.
-#
-# The run itself is expected to end red (the forced failure); only the
-# alert-on-failure job result matters.
-
-name: Schedule Failure Alert Drill
-
-on:
-  workflow_dispatch:
-
-permissions:
-  contents: read
-
-jobs:
-  forced-failure:
-    name: Forced failure
-    runs-on: ubuntu-latest
-    timeout-minutes: 5
-    steps:
-      - name: Fail on purpose
-        run: |
-          echo "Deliberate failure so alert-on-failure exercises the real consumer wiring."
-          exit 1
-
-  alert-on-failure:
-    name: Alert on scheduled failure
-    needs: [forced-failure]
-    # Mirrors the consumer wiring, minus the schedule-event guard (this
-    # workflow is dispatch-only by design).
-    if: always() && contains(needs.*.result, 'failure')
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    permissions:
-      contents: read
-      issues: write
-    steps:
-      - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
-      - name: Open or update failure-tracking issue
-        uses: ./.github/actions/schedule-failure-issue
-        with:
-          github-token: ${{ secrets.GITHUB_TOKEN }}

.gitignore

@@ -55,8 +55,8 @@ docs/*
 !docs/architecture/**
 !docs/operations/
 !docs/operations/**
-!docs/testing/
-!docs/testing/**
+!docs/superpowers/
+!docs/superpowers/**
 docs/heal-scanner-logging-governance.md
 docs/benchmark/rustfs-target-bench/
 docs/benchmark/*.md

.vscode/launch.json

@@ -1,45 +1,7 @@
 {
     "version": "0.2.0",
     "configurations": [
-        {
-            "name": "Debug RustFS observability (OTLP)",
-            "type": "lldb",
-            "request": "launch",
-            "cargo": {
-                "args": [
-                    "build",
-                    "--bin=rustfs",
-                    "--package=rustfs"
-                ],
-                "filter": {
-                    "name": "rustfs",
-                    "kind": "bin"
-                }
-            },
-            "args": [],
-            "cwd": "${workspaceFolder}",
-            "env": {
-                "RUST_LOG": "rustfs=debug,ecstore=info,s3s=info,iam=info",
-                "RUST_BACKTRACE": "full",
-                "RUSTFS_ACCESS_KEY": "rustfsadmin",
-                "RUSTFS_SECRET_KEY": "rustfsadmin",
-                "RUSTFS_VOLUMES": "./target/observability/data{1...4}",
-                "RUSTFS_ADDRESS": ":9000",
-                "RUSTFS_CONSOLE_ENABLE": "true",
-                "RUSTFS_CONSOLE_ADDRESS": "127.0.0.1:9001",
-                "RUSTFS_UNSAFE_BYPASS_DISK_CHECK": "true",
-                "RUSTFS_OBS_ENDPOINT": "http://127.0.0.1:4318",
-                "RUSTFS_OBS_TRACES_EXPORT_ENABLED": "true",
-                "RUSTFS_OBS_METRICS_EXPORT_ENABLED": "true",
-                "RUSTFS_OBS_LOGS_EXPORT_ENABLED": "true",
-                "RUSTFS_OBS_USE_STDOUT": "true",
-                "RUSTFS_OBS_LOG_DIRECTORY": "./target/observability/logs",
-                "RUSTFS_OBS_METER_INTERVAL": "5",
-                "RUSTFS_OBS_SERVICE_NAME": "rustfs-observability-local",
-                "RUSTFS_OBS_ENVIRONMENT": "development"
-            }
-        },
-        {
+          {
             "type": "lldb",
             "request": "launch",
             "name": "Debug(only) executable 'rustfs'",

AGENTS.md

@@ -19,7 +19,7 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
 - If a task has multiple plausible interpretations, list the options briefly and choose the narrowest reasonable path; ask when the ambiguity would make the change risky.
 - For multi-step work, keep the plan minimal and tied to verifiable outcomes.
 - Avoid redundant file reads, repeated commands, and unnecessary exploratory work once enough context is available.
-- A good result is a minimal diff with clear assumptions, no over-engineering, and independent verification that survives Adversarial Validation (below).
+- A good result is a minimal diff with clear assumptions, no over-engineering, and independent verification.
 
 ## Communication and Language
 
@@ -59,35 +59,17 @@ If repo-level instructions conflict, follow the nearest file and keep behavior a
 - High-level architecture and crate map: `ARCHITECTURE.md`
 - Migration guardrails, readiness contracts, support matrices:
   `docs/architecture/README.md` (routes by audience)
-- Shared agent skills (all tools): `.agents/skills/` — each `SKILL.md` carries
-  a frontmatter `description` stating when it applies. Scan the descriptions
-  before starting a task and follow any skill that matches, even if your tool
-  does not auto-load skills:
-  `grep -m1 '^description:' .agents/skills/*/SKILL.md`
-  Claude Code reads them through the `.claude/skills` symlink; add new skills
-  to `.agents/skills/` only, never as separate copies per tool
+- Historical implementation plans and trackers: `docs/superpowers/plans/`
+- Shared agent skills (all tools): `.agents/skills/` — Claude Code reads them
+  through the `.claude/skills` symlink; add new skills to `.agents/skills/`
+  only, never as separate copies per tool
 
 Avoid duplicating long crate lists or command matrices in instruction files.
 Reference the source files above instead.
 
-Do not commit planning-type documents — one-shot implementation/optimization
-plans, task trackers, migration-progress ledgers, phase/PR templates,
-issue-scoped benchmark-result snapshots or optimization conclusions, or
-agent-generated working notes (e.g. anything a `superpowers`/scratch workflow
-produces). Keep that work in the issue tracker or your local worktree, not in
-the repository. Only durable reference — the architecture set under
-`docs/architecture/`, repeatable operational runbooks under `docs/operations/`,
-and the test-suite references under `docs/testing/` — belongs in version
-control; `.gitignore` ignores everything else under `docs/` by default, so a new
-plan file will not be tracked unless someone force-adds it — don't.
-`scripts/check_no_planning_docs.sh` (wired into `make pre-commit`/`pre-pr` and
-CI) fails the build if anything is committed under `docs/superpowers/`, even via
-`git add -f`.
-
 ## Verification Before PR
 
 Convert changes into independently verifiable outcomes. Prefer focused tests for behavior changes and run the relevant checks before declaring completion.
-Non-exempt changes must also pass Adversarial Validation (next section) before the checks below count as completion.
 
 For code changes, run and pass the following before opening a PR:
 
@@ -112,119 +94,19 @@ Before pushing code changes, make sure formatting is clean:
 - Run `cargo fmt --all --check` and ensure no files are modified unexpectedly.
 
 If `make` is unavailable, run the equivalent checks defined under `.config/make/`.
-Documentation-only or instruction-only changes are exempt from the verification commands above (including the `.config/make/` equivalents), though any locally installed git pre-commit hooks may still run on commit unless explicitly skipped.
+Documentation-only or instruction-only changes are exempt from the verification commands above (including the `.config/make/` equivalents), though any installed git pre-commit hooks (for example, from `make setup-hooks`) may still run on commit unless explicitly skipped.
 After build-based verification completes, clean generated build artifacts before wrapping up to avoid unnecessary disk usage.
 Do not open a PR with code changes when the required checks fail.
-Make a failing check pass by fixing the cause, never by weakening the gate:
-do not loosen or skip a guard script, add entries to a baseline or allowance
-list, suppress a lint with `#[allow]`, mark a failing test `#[ignore]`, or
-delete or relax a failing assertion to get green. If a check itself is wrong,
-change it deliberately and state the rationale in the PR.
-
-For flaky tests, do not paper over them with retries. Follow the flake policy
-in [docs/testing/README.md](docs/testing/README.md) (open an issue within 24h,
-quarantine with an issue link, fix or delete within 30 days); the local
-`default` nextest profile never retries.
-
-## Adversarial Validation (Default On)
-
-Every non-exempt output (see Risk tiers) — code change, bug fix, or
-design/solution proposal — passes multi-role adversarial review before it
-counts as done.
-Author confidence is not evidence: each role's job is to refute the change,
-not to bless it.
-
-### Risk tiers
-
-Pick the tier from the riskiest file touched; when in doubt, pick the higher.
-
-- **Exempt:** docs/comments/instruction-only changes, formatting, typos with
-  no runtime surface. Skip this section.
-- **Mechanical:** pure renames, file moves, test-only or tooling changes —
-  correctness adversary only.
-- **Standard (the default):** any change that affects behavior.
-- **High risk:** touches locking, erasure coding, quorum/heal, replication,
-  multipart, RPC, lifecycle/tiering, metadata formats (`xl.meta`),
-  persistence/fsync, IAM/KMS/auth, on-disk or on-wire formats, or
-  S3 API-visible behavior.
-
-### Roles
-
-Run each applicable role as an independent pass over the final diff (or
-proposal text) — parallel reviewer agents where the tooling supports them,
-otherwise sequential passes that each start fresh from the diff and the
-nearest scoped `AGENTS.md`, discarding the writing session's assumptions.
-Each role either produces findings or reports "attacked X, Y, Z — no break
-found"; a bare pass is not a result. Repo-specific attack probes for every
-role live in `.agents/skills/adversarial-validation/` — run them, they
-encode this repo's shipped bugs.
-
-- **Correctness adversary** — construct a concrete input/state/interleaving
-  that yields wrong output, data loss, or a crash. Probe error paths and edge
-  values (empty, nil UUID, zero-length, quorum−1, missing version). For code
-  diffs, a materially smaller or more idiomatic diff achieving the same
-  behavior is also a finding (see Change Style for Existing Logic).
-- **Security reviewer** — authn/authz bypass, injection, secret leakage,
-  untrusted deserialization (see Serde Safety), path traversal, timing leaks.
-- **Concurrency/durability reviewer** — lock ordering, races, cancellation,
-  partial failure, retry/idempotency, crash and power-loss ordering.
-- **Compatibility reviewer** — S3 API surface, MinIO interop, on-disk and
-  on-wire formats, mixed-version upgrade/downgrade paths.
-- **Performance reviewer** — allocation and cloning on hot paths, lock hold
-  time across IO, sync or CPU-heavy work on async runtime threads, added
-  fsync/flush outside the durability gate, hot-path logging noise. A
-  measurable regression on a per-request or per-object path is a finding.
-- **Test-coverage skeptic** — for each claimed behavior, name the test that
-  fails if the change is reverted; then name a changed line that could be
-  wrong while all tests stay green — if one exists, coverage is insufficient.
-  A missing test is a finding, not a note.
-
-Standard tier: correctness adversary + test-coverage skeptic, plus every
-role whose domain the diff touches (async or shared-state code →
-concurrency; parsing of untrusted input → security; public crate API shape
-→ compatibility; per-request or per-object hot paths → performance).
-High risk: all six roles.
-
-### Protocol
-
-1. A finding states a concrete failure scenario (input/state → wrong
-   outcome) or names a missing test, with severity and file:line. "Looks
-   risky" is not a finding.
-2. Resolve every finding: fix it, or rebut it with evidence — a test, a
-   traced code path, or a cited invariant. Restated intent and "unlikely"
-   are not rebuttals.
-3. After non-trivial fixes, re-run the roles whose domain the fix touched.
-4. For proposals with no diff, roles attack assumptions, failure modes,
-   migration/rollback, and testability instead — including the simplest
-   rejected alternative and the blast radius when the design fails.
-
-### Exit criteria
-
-- Every applicable role has run; every finding is fixed or rebutted with
-  evidence.
-- Every behavior change has a test that fails without it.
-- The Verification Before PR gates pass — adversarial review supplements
-  those gates, never replaces them.
-- High risk only: record a one-line verdict per role in the PR description.
 
 ## Git and PR Baseline
 
 - Use feature branches based on the latest `main`.
-- Assume other agent sessions work this repository concurrently. Never commit
-  in a shared checkout; do all work on a dedicated feature branch, preferably
-  in a dedicated worktree.
-- Immediately before branching, fetch `origin/main` and branch from it;
-  confirm the target issue is not already fixed there before writing code.
 - Follow Conventional Commits, with subject length <= 72 characters.
 - Keep PR title and description in English.
 - Use `.github/pull_request_template.md` and keep all section headings.
 - Use `N/A` for non-applicable template sections.
 - Include verification commands in the PR description.
-- When using `gh pr create`/`gh pr edit`, write the markdown body to a file
-  and pass `--body-file`; multiline inline `--body` is unsafe — backticks and
-  shell expansion can corrupt content or trigger unintended commands.
-  Pattern: `cat > /tmp/pr_body.md <<'EOF' ... EOF`, then
-  `--body-file /tmp/pr_body.md` (keep the file outside the checkout).
+- When using `gh pr create`/`gh pr edit`, use `--body-file` instead of inline `--body` for multiline markdown.
 - Do not include the literal sequence `\n` in any GitHub issue, pull request, or discussion comment.
 - Do not hard-wrap prose in PR/issue/discussion bodies; write each paragraph as a
   single line and let it reflow. GitHub renders single newlines inside a paragraph
@@ -256,23 +138,11 @@ cargo run -p rustfs-filemeta --example dump_fileinfo -- "/path/to/file/xl.meta"
 - When `deny_unknown_fields` is impractical (backward compatibility), at minimum log unknown fields at `warn` level.
 - Never use `#[serde(default)]` on security-critical fields without explicit validation of the resulting value.
 
-## Cross-Cutting Domain Invariants
-
-- Write internal object metadata under **both** `x-rustfs-internal-<suffix>`
-  and `x-minio-internal-<suffix>` keys (MinIO interop). Use the helpers in
-  `crates/utils/src/http/metadata_compat.rs` (`get_bytes` prefers the RustFS
-  key); never write only one of the two.
-- Read binary UUID metadata defensively:
-  `.and_then(|v| Uuid::from_slice(&v).ok()).filter(|u| !u.is_nil())` —
-  absent, empty, and nil all mean "no value", never `Uuid::nil()`.
-- A remote-tier version of `None`/`""` means the tier bucket is unversioned:
-  send **no** `versionId` on tier GET/DELETE.
-
 ## Naming Conventions
 
 - Follow Rust API Guidelines for naming: `SCREAMING_SNAKE_CASE` for statics and constants, `snake_case` for functions and variables, `PascalCase` for types.
 - Do not use camelCase or Hungarian notation (e.g., `globalDeploymentIDPtr` → `GLOBAL_DEPLOYMENT_ID`).
-- If existing code violates naming conventions, do not widen the violation in new code. Do not rename existing symbols as part of an unrelated task; mention the violation instead (see Change Style for Existing Logic).
+- If existing code violates naming conventions, do not widen the violation in new code. Fix opportunistically when touching the surrounding area.
 
 ## Scoped Guidance in This Repository
 

ARCHITECTURE.md

@@ -141,7 +141,7 @@ Depth 8 — TOP:
 | `io-metrics` | 4.5K | I/O operation metrics and counters |
 | `rio` | 6.9K | Composable reader chain (encrypt → compress → hash → limit) |
 | `object-io` | 2.4K | High-level object read/write using rio + ecstore |
-| `concurrency` | 0.8K | Shared concurrency contract types: workload admission snapshots, worker-slot pool, policy types (runtime control lives in `rustfs/src/storage`) |
+| `concurrency` | 1.8K | Concurrency control wrappers over io-core |
 
 **Storage Engine:**
 

CLAUDE.md

@@ -36,7 +36,12 @@ make build-docker BUILD_OS=ubuntu22.04
 
 ## Domain conventions worth knowing up front
 
-Repo-wide domain invariants (dual internal metadata keys, defensive UUID
-reads, unversioned tier buckets) live in [AGENTS.md](AGENTS.md) under
-"Cross-Cutting Domain Invariants" — read them before touching metadata or
-tiering code.
+- Internal object metadata is written under **both** `x-rustfs-internal-<suffix>`
+  and `x-minio-internal-<suffix>` keys for MinIO interop
+  (`crates/utils/src/http/metadata_compat.rs`; `get_bytes` prefers the RustFS
+  key). Never write only one of the two.
+- Binary metadata values (UUIDs) must be read defensively:
+  `.and_then(|v| Uuid::from_slice(&v).ok()).filter(|u| !u.is_nil())` —
+  absent/empty/nil all mean "no value", not `Uuid::nil()`.
+- A remote-tier version of `None`/`""` means the tier bucket is unversioned:
+  send **no** `versionId` on tier GET/DELETE.

CONTRIBUTING.md

@@ -8,9 +8,9 @@ This guide covers the local development environment and the checks expected befo
 
 **MANDATORY**: All code must be properly formatted before committing. This project enforces strict formatting standards to maintain code consistency and readability.
 
-#### Verification Requirements
+#### Pre-commit Requirements
 
-Before submitting your changes for review, you **MUST**:
+Before every commit, you **MUST**:
 
 1. **Format your code**:
 
@@ -47,57 +47,33 @@ make fmt
 # Check if code is properly formatted
 make fmt-check
 
-# Run clippy checks (all targets, all features, -D warnings)
-make clippy-check
+# Run clippy checks
+make clippy
 
-# Fast workspace compilation check (excludes e2e_test)
-make quick-check
+# Run compilation check
+make check
 
-# Full compilation check (cargo check --all-targets)
-make compilation-check
-
-# Run tests (shell script tests + workspace tests + doc tests)
+# Run tests
 make test
 
-# Fast pre-commit gate — see below for exactly what it runs
+# Run all pre-commit checks (format + clippy + check + test)
 make pre-commit
 
-# Full pre-PR gate (pre-commit gates + clippy + tests)
-make pre-pr
+# Setup git hooks (one-time setup)
+make setup-hooks
 ```
 
-> `make test` requires [cargo-nextest](https://nexte.st) (CI runs it and only nextest honours `.config/nextest.toml` test-groups). Install it with `cargo install cargo-nextest --locked` or a prebuilt binary (see https://nexte.st/docs/installation/). To run the plain `cargo test` fallback anyway (results not authoritative — serialization semantics differ from CI), set `RUSTFS_ALLOW_CARGO_TEST_FALLBACK=1`.
-
 ### 🔒 Automated Pre-commit Hooks
-#### What `make pre-commit` and `make pre-pr` actually run
 
-`make pre-commit` is the **fast** gate. It runs, in order
-(see `.config/make/pre-commit.mak`):
+This project includes a pre-commit hook that automatically runs before each commit to ensure:
 
-1. `fmt-check` — `cargo fmt --all --check`
-2. `unsafe-code-check` — `./scripts/check_unsafe_code_allowances.sh`
-3. `architecture-migration-check` — `./scripts/check_architecture_migration_rules.sh`
-4. `logging-guardrails-check` — `./scripts/check_logging_guardrails.sh`
-5. `tokio-io-uring-check` — `./scripts/check_no_tokio_io_uring.sh`
-6. `extension-schema-check` — `./scripts/check_extension_schema_boundaries.sh`
-7. `doc-paths-check` — `./scripts/check_doc_paths.sh`
-8. `quick-check` — `cargo check --workspace --exclude e2e_test`
+- ✅ Code is properly formatted (`cargo fmt --all --check`)
+- ✅ No clippy warnings (`cargo clippy --all-targets --all-features -- -D warnings`)
+- ✅ Code compiles successfully (`cargo check --all-targets`)
 
-**`make pre-commit` does NOT run clippy and does NOT run any tests.**
-A green `make pre-commit` is not enough to open a pull request.
+#### Setting Up Pre-commit Hooks
 
-`make pre-pr` is the **full** gate: it runs all of the guard checks above,
-then `clippy-check` (`cargo clippy --all-targets --all-features -- -D warnings`)
-and `test` (shell script tests, workspace tests excluding `e2e_test`, and doc
-tests). Run `make pre-pr` before opening or updating a pull request — this is
-what CI enforces.
-
-### 🔒 Git Pre-commit Hooks (optional)
-
-Git hooks are **not** versioned in this repository, so a fresh clone has no
-active pre-commit hook. If you add your own `.git/hooks/pre-commit` (a good
-choice is a one-liner that runs `make pre-commit`), you can mark it executable
-with:
+Run this command once after cloning the repository:
 
 ```bash
 make setup-hooks
@@ -109,9 +85,6 @@ Or manually:
 chmod +x .git/hooks/pre-commit
 ```
 
-With or without a hook, the expectation is the same: run `make pre-commit`
-before committing and `make pre-pr` before opening a pull request.
-
 ### 📝 Formatting Configuration
 
 The project uses the following rustfmt configuration (defined in `rustfmt.toml`):
@@ -124,7 +97,7 @@ single_line_let_else_max_width = 100
 
 ### 🚫 Commit Prevention
 
-If you set up a pre-commit hook and your code doesn't meet the formatting requirements, the hook will:
+If your code doesn't meet the formatting requirements, the pre-commit hook will:
 
 1. **Block the commit** and show clear error messages
 2. **Provide exact commands** to fix the issues
@@ -146,10 +119,9 @@ Example output when formatting fails:
 
 1. **Make your changes**
 2. **Format your code**: `make fmt` or `cargo fmt --all`
-3. **Run the fast gate**: `make pre-commit` (no clippy, no tests)
+3. **Run pre-commit checks**: `make pre-commit`
 4. **Commit your changes**: `git commit -m "your message"`
-5. **Run the full gate before opening/updating a PR**: `make pre-pr` (clippy + tests)
-6. **Push to your branch**: `git push`
+5. **Push to your branch**: `git push`
 
 ### 🛠️ IDE Integration
 

Cargo.toml

@@ -142,7 +142,7 @@ futures = "0.3.32"
 futures-core = "0.3.32"
 futures-lite = "2.6.1"
 futures-util = "0.3.32"
-pollster = "1.0.1"
+pollster = "0.4.0"
 pulsar = { version = "6.8.0", default-features = false, features = ["tokio-rustls-runtime", "telemetry"] }
 lapin = { version = "4.10.0", default-features = false, features = ["tokio", "rustls", "rustls--aws_lc_rs"] }
 hyper = { version = "1.10.1", features = ["http2", "http1", "server"] }
@@ -168,7 +168,7 @@ tower-http = { version = "0.7.0", features = ["cors"] }
 
 # Serialization and Data Formats
 apache-avro = "0.21.0"
-bytes = { version = "1.12.1", features = ["serde"] }
+bytes = { version = "1.12.0", features = ["serde"] }
 bytesize = "2.4.2"
 byteorder = "1.5.0"
 flatbuffers = "25.12.19"
@@ -207,7 +207,7 @@ zeroize = { version = "1.9.0", features = ["derive"] }
 # Time and Date
 chrono = { version = "0.4.45", features = ["serde"] }
 humantime = "2.4.0"
-jiff = { version = "0.2.32", features = ["serde"] }
+jiff = { version = "0.2.31", features = ["serde"] }
 time = { version = "0.3.53", features = ["std", "parsing", "formatting", "macros", "serde"] }
 
 # Database
@@ -221,12 +221,12 @@ arc-swap = "1.9.2"
 astral-tokio-tar = "0.6.3"
 atoi = "3.1.0"
 atomic_enum = "0.3.0"
-aws-config = { version = "1.9.0" }
-aws-credential-types = { version = "1.3.0" }
-aws-sdk-s3 = { version = "1.138.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
-aws-smithy-http-client = { version = "1.2.0", default-features = false, features = ["default-client", "rustls-aws-lc"] }
-aws-smithy-runtime-api = { version = "1.13.0", features = ["http-1x"] }
-aws-smithy-types = { version = "1.6.1" }
+aws-config = { version = "1.8.18" }
+aws-credential-types = { version = "1.2.14" }
+aws-sdk-s3 = { version = "1.137.0", default-features = false, features = ["sigv4a", "default-https-client", "rt-tokio"] }
+aws-smithy-http-client = { version = "1.1.13", default-features = false, features = ["default-client", "rustls-aws-lc"] }
+aws-smithy-runtime-api = { version = "1.12.3", features = ["http-1x"] }
+aws-smithy-types = { version = "1.5.0" }
 base64 = "0.22.1"
 base64-simd = "0.8.0"
 brotli = "8.0.4"
@@ -259,14 +259,14 @@ memmap2 = "0.9.11"
 lz4 = "1.28.1"
 matchit = "0.9.2"
 md-5 = "0.11.0"
-md5 = "0.8.1"
+md5 = "0.8.0"
 mime_guess = "2.0.5"
 moka = { version = "0.12.15", features = ["future"] }
 netif = "0.1.6"
 num_cpus = { version = "1.17.0" }
 nvml-wrapper = "0.12.1"
 parking_lot = "0.12.5"
-path-absolutize = "4.0.1"
+path-absolutize = "3.1.1"
 path-clean = "1.0.1"
 percent-encoding = "2.3.2"
 pin-project-lite = "0.2.17"
@@ -277,11 +277,11 @@ rayon = "1.12.0"
 reed-solomon-erasure = { package = "rustfs-erasure-codec", version = "7.0.1", features = ["simd-accel"] }
 #reed-solomon-erasure = { version = "6.0", features = ["simd-accel"], git = "https://github.com/houseme/reed-solomon-erasure",rev = "main" }
 reed-solomon-simd = "3.1.0"
-regex = { version = "1.13.0" }
+regex = { version = "1.12.4" }
 rumqttc = { package = "rumqttc-next", version = "0.33.2", features = ["websocket"] }
 redis = { version = "1.3.0", features = ["connection-manager", "tokio-rustls-comp", "tls-rustls-insecure"] }
 rustix = { version = "1.1.4", features = ["fs"] }
-rust-embed = { version = "8.12.0" }
+rust-embed = { version = "8.11.0" }
 rustc-hash = { version = "2.1.3" }
 s3s = { version = "0.14.1", features = ["minio"] }
 serial_test = "3.5.0"
@@ -292,7 +292,7 @@ smartstring = "1.0.1"
 snap = "1.1.1"
 starshard = { version = "2.2.1", features = ["rayon", "async", "serde"] }
 strum = { version = "0.28.0", features = ["derive"] }
-sysinfo = "0.39.6"
+sysinfo = "0.39.5"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
 test-case = "3.3.1"
@@ -305,9 +305,10 @@ tracing-subscriber = { version = "0.3.23", features = ["env-filter", "time"] }
 transform-stream = "0.3.1"
 url = "2.5.8"
 urlencoding = "2.1.3"
-uuid = { version = "1.23.5", features = ["v4", "fast-rng", "macro-diagnostics"] }
+uuid = { version = "1.23.4", features = ["v4", "fast-rng", "macro-diagnostics"] }
 vaultrs = { version = "0.8.0" }
 walkdir = "2.5.0"
+wildmatch = { version = "2.6.1", features = ["serde"] }
 windows = { version = "0.62.2" }
 xxhash-rust = { version = "0.8.16", features = ["xxh64", "xxh3"] }
 zip = "8.6.0"
@@ -322,7 +323,7 @@ opentelemetry-otlp = { version = "0.32.0", features = ["gzip-http", "reqwest-rus
 opentelemetry_sdk = { version = "0.32.1", features = ["rt-tokio"] }
 opentelemetry-semantic-conventions = { version = "0.32.1", features = ["semconv_experimental"] }
 opentelemetry-stdout = { version = "0.32.0" }
-pyroscope = { version = "2.1.0", features = ["backend-pprof-rs"] }
+pyroscope = { version = "2.0.6", features = ["backend-pprof-rs"] }
 
 # FTP and SFTP
 libunftp = { version = "0.23.0", features = ["experimental"] }

Makefile

@@ -64,11 +64,10 @@ How to use me:
 
 	🔧 Code Quality:
 		make fmt                                 # Format code
-		make clippy-check                        # Run clippy (all targets, all features, -D warnings)
-		make quick-check                         # Fast workspace compile check (excludes e2e_test)
-		make test                                # Script tests + workspace tests + doc tests
-		make pre-commit                          # Fast gate: fmt-check + guard scripts + quick-check (NO clippy, NO tests)
-		make pre-pr                              # Full pre-PR gate: pre-commit gates + clippy-check + test
+		make clippy                              # Run clippy checks
+		make test                                # Run tests
+		make pre-commit                          # Run fast pre-commit checks
+		make pre-pr                              # Run full pre-PR checks
 
 	🚀 Quick Start:
 		make build                               # Build RustFS binary

_typos.toml

@@ -45,8 +45,6 @@ nonexisted = "nonexisted"
 consts = "consts"
 # Swift API - company/product names
 Hashi = "Hashi"  # HashiCorp
-# Accept alternate spelling used in parser/XML comments.
-unparseable = "unparseable"
 
 [files]
 extend-exclude = []

crates/audit/src/error.rs

@@ -35,9 +35,6 @@ pub enum AuditError {
     #[error("System already initialized")]
     AlreadyInitialized,
 
-    #[error("Audit system is paused; entry was not accepted")]
-    Paused,
-
     #[error("Storage not available: {0}")]
     StorageNotAvailable(String),
 

crates/audit/src/global.rs

@@ -12,7 +12,7 @@
 //  See the License for the specific language governing permissions and
 //  limitations under the License.
 
-use crate::{AuditEntry, AuditError, AuditResult, AuditSystem, system::AuditTargetMetricSnapshot};
+use crate::{AuditEntry, AuditResult, AuditSystem, system::AuditTargetMetricSnapshot};
 use rustfs_config::server_config::Config;
 use std::sync::{Arc, OnceLock};
 use tracing::{debug, error, trace};
@@ -78,27 +78,10 @@ pub async fn resume_audit_system() -> AuditResult<()> {
 
 /// Dispatch an audit log entry to all targets
 pub async fn dispatch_audit_log(entry: Arc<AuditEntry>) -> AuditResult<()> {
-    let Some(system) = audit_system() else {
-        debug!(
-            event = EVENT_AUDIT_ENTRY_DROPPED,
-            component = LOG_COMPONENT_AUDIT,
-            subsystem = LOG_SUBSYSTEM_GLOBAL,
-            reason = "system_not_initialized",
-            "Dropped audit entry"
-        );
-        return Ok(());
-    };
-
-    // Single state read (backlog#984): the previous code checked `is_running()`
-    // and then called `dispatch()`, which re-read the state. Between the two
-    // reads the system could transition (e.g. Running -> Stopping) and
-    // `dispatch()` would return an error the caller never expected. Let
-    // `dispatch()` be the single authority on the current state and interpret
-    // its "not accepting" errors as a deliberate skip, while still surfacing
-    // real delivery failures (backlog#962).
-    match system.dispatch(entry).await {
-        Ok(()) => Ok(()),
-        Err(AuditError::NotInitialized(_)) | Err(AuditError::Paused) => {
+    if let Some(system) = audit_system() {
+        if system.is_running().await {
+            system.dispatch(entry).await
+        } else {
             trace!(
                 event = EVENT_AUDIT_ENTRY_DROPPED,
                 component = LOG_COMPONENT_AUDIT,
@@ -108,7 +91,15 @@ pub async fn dispatch_audit_log(entry: Arc<AuditEntry>) -> AuditResult<()> {
             );
             Ok(())
         }
-        Err(e) => Err(e),
+    } else {
+        debug!(
+            event = EVENT_AUDIT_ENTRY_DROPPED,
+            component = LOG_COMPONENT_AUDIT,
+            subsystem = LOG_SUBSYSTEM_GLOBAL,
+            reason = "system_not_initialized",
+            "Dropped audit entry"
+        );
+        Ok(())
     }
 }
 

crates/audit/src/pipeline.rs

@@ -45,12 +45,6 @@ impl AuditPipeline {
         Self { registry }
     }
 
-    /// Fans an audit entry out to every configured target concurrently.
-    ///
-    /// Delivery across targets is unordered: the per-target `save()` calls run
-    /// via `join_all` and may complete in any order. Ordering of entries within
-    /// a single target is preserved by that target's own store/queue, not by
-    /// this fan-out.
     pub async fn dispatch(&self, entry: Arc<AuditEntry>) -> AuditResult<()> {
         let start_time = std::time::Instant::now();
 
@@ -196,14 +190,8 @@ impl AuditPipeline {
                         data: (*entry).clone(),
                     };
                     match target.save(Arc::new(entity_target)).await {
-                        Ok(_) => {
-                            success_count += 1;
-                            observability::record_target_success();
-                        }
-                        Err(e) => {
-                            observability::record_target_failure();
-                            errors.push(e);
-                        }
+                        Ok(_) => success_count += 1,
+                        Err(e) => errors.push(e),
                     }
                 }
                 (target.id().to_string(), success_count, errors)
@@ -263,12 +251,6 @@ impl AuditPipeline {
             ));
         }
 
-        // Record the aggregate event outcome so batch dispatch reports the same
-        // observability signal as single dispatch (backlog#984): full success or
-        // partial failure both count as a delivered audit event here, since at
-        // least one target accepted every entry that reached this point.
-        observability::record_audit_success(dispatch_time);
-
         Ok(())
     }
 
@@ -559,135 +541,3 @@ impl AuditRuntimeFacade {
         self.runtime_adapter.stop_replay_workers(&mut replay_workers).await;
     }
 }
-
-#[cfg(test)]
-mod tests {
-    use super::AuditPipeline;
-    use crate::{AuditEntry, AuditError, AuditRegistry};
-    use async_trait::async_trait;
-    use rustfs_targets::arn::TargetID;
-    use rustfs_targets::store::{Key, Store};
-    use rustfs_targets::target::{EntityTarget, QueuedPayload, QueuedPayloadMeta};
-    use rustfs_targets::{StoreError, Target, TargetError};
-    use std::sync::Arc;
-    use tokio::sync::Mutex;
-
-    /// Mock target whose `save()` outcome is fixed at construction so tests can
-    /// force full-success / full-failure / partial-failure fan-outs.
-    #[derive(Clone)]
-    struct MockTarget {
-        id: TargetID,
-        fail: bool,
-    }
-
-    impl MockTarget {
-        fn new(id: &str, fail: bool) -> Self {
-            Self {
-                id: TargetID::new(id.to_string(), "webhook".to_string()),
-                fail,
-            }
-        }
-    }
-
-    #[async_trait]
-    impl<E> Target<E> for MockTarget
-    where
-        E: rustfs_targets::PluginEvent,
-    {
-        fn id(&self) -> TargetID {
-            self.id.clone()
-        }
-
-        async fn is_active(&self) -> Result<bool, TargetError> {
-            Ok(true)
-        }
-
-        async fn save(&self, _event: Arc<EntityTarget<E>>) -> Result<(), TargetError> {
-            if self.fail {
-                Err(TargetError::Configuration("forced save failure".to_string()))
-            } else {
-                Ok(())
-            }
-        }
-
-        async fn send_raw_from_store(&self, _key: Key, _body: Vec<u8>, _meta: QueuedPayloadMeta) -> Result<(), TargetError> {
-            Ok(())
-        }
-
-        async fn close(&self) -> Result<(), TargetError> {
-            Ok(())
-        }
-
-        fn store(&self) -> Option<&(dyn Store<QueuedPayload, Error = StoreError, Key = Key> + Send + Sync)> {
-            None
-        }
-
-        fn clone_dyn(&self) -> Box<dyn Target<E> + Send + Sync> {
-            Box::new(self.clone())
-        }
-
-        fn is_enabled(&self) -> bool {
-            true
-        }
-    }
-
-    fn pipeline_with(targets: Vec<MockTarget>) -> AuditPipeline {
-        let mut registry = AuditRegistry::new();
-        for target in targets {
-            registry.add_target(target.id.to_string(), Box::new(target));
-        }
-        AuditPipeline::new(Arc::new(Mutex::new(registry)))
-    }
-
-    fn entry() -> Arc<AuditEntry> {
-        Arc::new(AuditEntry::default())
-    }
-
-    // backlog#962: when every target rejects the event it is lost outright, so
-    // dispatch must return Err rather than swallowing the failures as Ok.
-    #[tokio::test]
-    async fn dispatch_returns_err_when_all_targets_fail() {
-        let pipeline = pipeline_with(vec![MockTarget::new("a:webhook", true), MockTarget::new("b:webhook", true)]);
-        let result = pipeline.dispatch(entry()).await;
-        assert!(matches!(result, Err(AuditError::Target(_))), "expected Err, got {result:?}");
-    }
-
-    // A partially-successful fan-out means the entry reached at least one sink,
-    // so dispatch reports success (degradation is logged, not propagated).
-    #[tokio::test]
-    async fn dispatch_returns_ok_on_partial_failure() {
-        let pipeline = pipeline_with(vec![MockTarget::new("ok:webhook", false), MockTarget::new("bad:webhook", true)]);
-        pipeline.dispatch(entry()).await.expect("partial success should return Ok");
-    }
-
-    #[tokio::test]
-    async fn dispatch_returns_ok_when_all_targets_succeed() {
-        let pipeline = pipeline_with(vec![MockTarget::new("a:webhook", false), MockTarget::new("b:webhook", false)]);
-        pipeline.dispatch(entry()).await.expect("all-success should return Ok");
-    }
-
-    // No configured targets is a benign no-op, not a failure.
-    #[tokio::test]
-    async fn dispatch_returns_ok_with_no_targets() {
-        let pipeline = pipeline_with(vec![]);
-        pipeline.dispatch(entry()).await.expect("no targets should return Ok");
-    }
-
-    // backlog#962: dispatch_batch must mirror dispatch and propagate a
-    // whole-batch loss instead of returning Ok.
-    #[tokio::test]
-    async fn dispatch_batch_returns_err_when_all_targets_fail() {
-        let pipeline = pipeline_with(vec![MockTarget::new("a:webhook", true)]);
-        let result = pipeline.dispatch_batch(vec![entry(), entry()]).await;
-        assert!(matches!(result, Err(AuditError::Target(_))), "expected Err, got {result:?}");
-    }
-
-    #[tokio::test]
-    async fn dispatch_batch_returns_ok_when_all_targets_succeed() {
-        let pipeline = pipeline_with(vec![MockTarget::new("a:webhook", false), MockTarget::new("b:webhook", false)]);
-        pipeline
-            .dispatch_batch(vec![entry(), entry()])
-            .await
-            .expect("all-success batch should return Ok");
-    }
-}

crates/audit/src/system.rs

@@ -89,20 +89,14 @@ impl AuditSystem {
         registry.create_audit_targets_from_config(config).await
     }
 
-    /// Stops any active replay workers and closes the currently installed
-    /// targets without touching the system state. Lock order is `registry`
-    /// then `stream_cancellers` to stay consistent with every other path that
-    /// holds both locks (see `runtime_status_snapshot`, backlog#961).
-    async fn shutdown_runtime_targets(&self) -> AuditResult<()> {
-        let mut registry = self.registry.lock().await;
-        let mut replay_workers = self.stream_cancellers.write().await;
-        self.runtime_facade()
-            .shutdown_runtime(&mut registry, &mut replay_workers)
-            .await
-    }
-
     async fn clear_runtime_targets(&self) -> AuditResult<()> {
-        self.shutdown_runtime_targets().await?;
+        {
+            let mut registry = self.registry.lock().await;
+            let mut replay_workers = self.stream_cancellers.write().await;
+            self.runtime_facade()
+                .shutdown_runtime(&mut registry, &mut replay_workers)
+                .await?;
+        }
 
         let mut state = self.state.write().await;
         *state = AuditSystemState::Stopped;
@@ -129,16 +123,6 @@ impl AuditSystem {
             "audit system state"
         );
 
-        // Stop-before-start (backlog#970): tear down the existing replay workers
-        // and close the currently installed targets *before* activating the new
-        // set. Activation spawns fresh replay workers for store-backed targets,
-        // so if the old workers were still running they would drain the same
-        // persistent queue concurrently with the new ones and re-deliver
-        // entries. Shutting the old runtime down first keeps at most one active
-        // worker per store across a reload. `replace_targets` performs a second
-        // (now no-op) shutdown before installing, which is idempotent.
-        self.shutdown_runtime_targets().await?;
-
         let activation = self.runtime_facade().activate_targets_with_replay(targets).await;
         self.runtime_facade().replace_targets(activation).await?;
 
@@ -155,30 +139,21 @@ impl AuditSystem {
     /// # Returns
     /// * `AuditResult<()>` - Result indicating success or failure
     pub async fn start(&self, config: Config) -> AuditResult<()> {
-        // Claim the `Starting` transition atomically while holding the write
-        // lock (backlog#978): the previous code released the lock after the
-        // check and re-acquired it later to set `Starting`, so two concurrent
-        // `start()` calls (or `start()` racing `reload`) could both pass the
-        // check and double-activate. Transitioning to `Starting` before
-        // dropping the guard makes a concurrent caller observe `Starting` and
-        // return early instead.
-        {
-            let mut state = self.state.write().await;
+        let state = self.state.write().await;
 
-            match *state {
-                AuditSystemState::Running => {
-                    return Err(AuditError::AlreadyInitialized);
-                }
-                AuditSystemState::Starting => {
-                    warn_audit_state("starting", Some("already_starting"));
-                    return Ok(());
-                }
-                _ => {}
+        match *state {
+            AuditSystemState::Running => {
+                return Err(AuditError::AlreadyInitialized);
             }
-
-            *state = AuditSystemState::Starting;
+            AuditSystemState::Starting => {
+                warn_audit_state("starting", Some("already_starting"));
+                return Ok(());
+            }
+            _ => {}
         }
 
+        drop(state);
+
         info!(
             event = EVENT_AUDIT_SYSTEM_STATE,
             component = LOG_COMPONENT_AUDIT,
@@ -198,7 +173,11 @@ impl AuditSystem {
 
         match self.create_targets_from_config(&config).await {
             Ok(targets) => {
-                // State is already `Starting` (claimed atomically above).
+                {
+                    let mut state = self.state.write().await;
+                    *state = AuditSystemState::Starting;
+                }
+
                 self.commit_runtime_targets(targets, AuditSystemState::Running).await?;
                 info_audit_state("running", None, None);
                 Ok(())
@@ -339,13 +318,7 @@ impl AuditSystem {
         match *state {
             AuditSystemState::Running => {}
             AuditSystemState::Paused => {
-                // Do not silently return Ok while paused (backlog#978): the
-                // entry is neither delivered nor persisted, so reporting success
-                // would corrupt the audit trail. Surface an explicit `Paused`
-                // error and let the caller apply its policy (the global helper
-                // treats this as a deliberate skip; direct API callers can
-                // decide otherwise).
-                return Err(AuditError::Paused);
+                return Ok(());
             }
             _ => {
                 return Err(AuditError::NotInitialized("Audit system is not running".to_string()));
@@ -575,7 +548,6 @@ fn warn_audit_state(state: &str, reason: Option<&str>) {
 #[cfg(test)]
 mod tests {
     use super::{AuditSystem, AuditSystemState};
-    use crate::{AuditEntry, AuditError};
     use async_trait::async_trait;
     use rustfs_targets::ReplayWorkerManager;
     use rustfs_targets::arn::TargetID;
@@ -733,95 +705,4 @@ mod tests {
             .await
             .expect("audit lock paths deadlocked (backlog#961 regression)");
     }
-
-    /// backlog#978: a paused system must not report success while silently
-    /// dropping the entry. `dispatch` should surface an explicit `Paused` error.
-    #[tokio::test]
-    async fn dispatch_while_paused_returns_error_not_ok() {
-        let system = AuditSystem::new();
-        {
-            let mut state = system.state.write().await;
-            *state = AuditSystemState::Paused;
-        }
-
-        let result = system.dispatch(Arc::new(AuditEntry::default())).await;
-        assert!(
-            matches!(result, Err(AuditError::Paused)),
-            "paused dispatch must return Err(Paused), got {result:?}"
-        );
-    }
-
-    /// backlog#978: `start()` now claims the `Starting` transition atomically
-    /// under the state lock, so racing `start()` calls cannot both pass the
-    /// check and double-activate. Hammer concurrent starts and assert the
-    /// workload completes (no deadlock/panic) and converges to a consistent
-    /// final state.
-    #[tokio::test(flavor = "multi_thread", worker_threads = 4)]
-    async fn concurrent_start_does_not_hang_or_double_activate() {
-        use std::time::Duration;
-
-        let system = AuditSystem::new();
-        let mut handles = Vec::new();
-        for _ in 0..8 {
-            let s = system.clone();
-            handles.push(tokio::spawn(async move {
-                // Empty config activates no targets, so a completed start settles
-                // the system back to `Stopped`.
-                let _ = s.start(rustfs_config::server_config::Config(HashMap::new())).await;
-            }));
-        }
-
-        let workload = async {
-            for handle in handles {
-                handle.await.expect("start task panicked");
-            }
-        };
-        tokio::time::timeout(Duration::from_secs(30), workload)
-            .await
-            .expect("concurrent start deadlocked (backlog#978 regression)");
-
-        assert_eq!(system.get_state().await, AuditSystemState::Stopped);
-    }
-
-    /// backlog#970: a reload/commit must tear down the previous replay workers
-    /// and close the old targets before activating the replacement set, so the
-    /// old and new workers never drain the same store concurrently. Seed an old
-    /// target plus a replay worker, commit a new target, and assert the old one
-    /// was closed and its worker stopped while the new one is installed.
-    #[tokio::test]
-    async fn commit_closes_old_targets_before_installing_new() {
-        let system = AuditSystem::new();
-
-        let old = TestTarget::new("old", "webhook");
-        let old_close = Arc::clone(&old.close_calls);
-        {
-            let mut registry = system.registry.lock().await;
-            registry.add_target("old:webhook".to_string(), Box::new(old));
-        }
-        {
-            let mut replay_workers = system.stream_cancellers.write().await;
-            let (cancel_tx, _cancel_rx) = mpsc::channel(1);
-            replay_workers.insert("old:webhook".to_string(), cancel_tx);
-        }
-        {
-            let mut state = system.state.write().await;
-            *state = AuditSystemState::Running;
-        }
-
-        let new = TestTarget::new("new", "webhook");
-        let new_close = Arc::clone(&new.close_calls);
-        system
-            .commit_runtime_targets(vec![Box::new(new)], AuditSystemState::Running)
-            .await
-            .expect("commit should succeed");
-
-        // Old target closed exactly once during the pre-install shutdown.
-        assert_eq!(old_close.load(Ordering::SeqCst), 1);
-        // New target installed and left open.
-        assert_eq!(new_close.load(Ordering::SeqCst), 0);
-        assert_eq!(system.list_targets().await, vec!["new:webhook".to_string()]);
-        // Old replay worker stopped; the store-less new target adds none.
-        assert_eq!(system.runtime_status_snapshot().await.replay_worker_count, 0);
-        assert_eq!(system.get_state().await, AuditSystemState::Running);
-    }
 }

crates/checksums/src/error.rs

@@ -36,7 +36,7 @@ impl fmt::Display for UnknownChecksumAlgorithmError {
     fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
         write!(
             f,
-            r#"unknown checksum algorithm "{}", please pass a known algorithm name ("crc32", "crc32c", "crc64nvme", "sha1", "sha256")"#,
+            r#"unknown checksum algorithm "{}", please pass a known algorithm name ("crc32", "crc32c", "sha1", "sha256", "md5")"#,
             self.checksum_algorithm
         )
     }

crates/checksums/src/lib.rs

@@ -43,6 +43,8 @@ pub enum ChecksumAlgorithm {
     #[default]
     Crc32,
     Crc32c,
+    #[deprecated]
+    Md5,
     Sha1,
     Sha256,
     Crc64Nvme,
@@ -60,6 +62,9 @@ impl FromStr for ChecksumAlgorithm {
             Ok(Self::Sha1)
         } else if checksum_algorithm.eq_ignore_ascii_case(SHA_256_NAME) {
             Ok(Self::Sha256)
+        } else if checksum_algorithm.eq_ignore_ascii_case(MD5_NAME) {
+            // MD5 is now an alias for the default Crc32 since it is deprecated
+            Ok(Self::Crc32)
         } else if checksum_algorithm.eq_ignore_ascii_case(CRC_64_NVME_NAME) {
             Ok(Self::Crc64Nvme)
         } else {
@@ -74,6 +79,8 @@ impl ChecksumAlgorithm {
             Self::Crc32 => Box::<Crc32>::default(),
             Self::Crc32c => Box::<Crc32c>::default(),
             Self::Crc64Nvme => Box::<Crc64Nvme>::default(),
+            #[allow(deprecated)]
+            Self::Md5 => Box::<Crc32>::default(),
             Self::Sha1 => Box::<Sha1>::default(),
             Self::Sha256 => Box::<Sha256>::default(),
         }
@@ -84,6 +91,8 @@ impl ChecksumAlgorithm {
             Self::Crc32 => CRC_32_NAME,
             Self::Crc32c => CRC_32_C_NAME,
             Self::Crc64Nvme => CRC_64_NVME_NAME,
+            #[allow(deprecated)]
+            Self::Md5 => MD5_NAME,
             Self::Sha1 => SHA_1_NAME,
             Self::Sha256 => SHA_256_NAME,
         }
@@ -291,7 +300,6 @@ struct Md5 {
     hasher: md5::Md5,
 }
 
-#[allow(dead_code)]
 impl Md5 {
     fn update(&mut self, bytes: &[u8]) {
         use md5::Digest;
@@ -436,23 +444,4 @@ mod tests {
             .expect_err("it should error");
         assert_eq!("some invalid checksum algorithm", error.checksum_algorithm());
     }
-
-    #[test]
-    fn test_unknown_algorithm_error_message_lists_supported_algorithms() {
-        let error = "nope".parse::<ChecksumAlgorithm>().expect_err("it should error");
-        let message = error.to_string();
-        assert!(message.contains("crc64nvme"), "message should advertise crc64nvme: {message}");
-        assert!(!message.contains("md5"), "message should not advertise the unsupported md5: {message}");
-    }
-
-    #[test]
-    fn test_md5_is_not_a_supported_checksum_algorithm() {
-        // MD5 is not an accepted S3 checksum algorithm here: parsing it must fail
-        // loudly rather than silently substituting a CRC32 hasher.
-        let error = "md5".parse::<ChecksumAlgorithm>().expect_err("md5 should not parse");
-        assert_eq!("md5", error.checksum_algorithm());
-
-        let error = "MD5".parse::<ChecksumAlgorithm>().expect_err("md5 should not parse");
-        assert_eq!("MD5", error.checksum_algorithm());
-    }
 }

crates/common/src/last_minute.rs

@@ -144,14 +144,11 @@ impl LastMinuteLatency {
             merged.last_sec = o.last_sec;
         }
 
-        // Both operands must be read in their forwarded form so aged-out
-        // ring-buffer slots stay zeroed: `x` is the forwarded copy of `o`,
-        // and `self` is forwarded in place in the `else` branch above.
         for i in 0..merged.totals.len() {
             merged.totals[i] = AccElem {
-                total: self.totals[i].total.wrapping_add(x.totals[i].total),
-                n: self.totals[i].n.wrapping_add(x.totals[i].n),
-                size: self.totals[i].size.wrapping_add(x.totals[i].size),
+                total: self.totals[i].total + o.totals[i].total,
+                n: self.totals[i].n + o.totals[i].n,
+                size: self.totals[i].size + o.totals[i].size,
             }
         }
         merged
@@ -375,105 +372,6 @@ mod tests {
         assert_eq!(merged.totals[0].total, 30);
     }
 
-    #[test]
-    fn test_last_minute_latency_merge_ages_out_stale_slots_self_newer() {
-        // self.last_sec > o.last_sec branch: `o` is forwarded to self.last_sec,
-        // which zeroes the ring-buffer slots for seconds 1001..=1010, i.e.
-        // indices 41..=50. Data parked in one of those slots is stale and must
-        // be excluded from the merged result.
-        let mut newer = LastMinuteLatency::default();
-        let mut older = LastMinuteLatency::default();
-
-        newer.last_sec = 1010;
-        older.last_sec = 1000;
-
-        // Stale slot: second 1005 -> index 45, cleared by forward_to(1010).
-        let stale_idx = (1005 % 60) as usize;
-        older.totals[stale_idx].total = 111;
-        older.totals[stale_idx].n = 5;
-        older.totals[stale_idx].size = 999;
-
-        // In-window slot: older's own last_sec (1000 -> index 40) is kept.
-        let kept_idx = (1000 % 60) as usize;
-        older.totals[kept_idx].total = 3;
-        older.totals[kept_idx].n = 1;
-        older.totals[kept_idx].size = 30;
-
-        // newer's current data (1010 -> index 50) must survive.
-        let newer_idx = (1010 % 60) as usize;
-        newer.totals[newer_idx].total = 7;
-        newer.totals[newer_idx].n = 1;
-        newer.totals[newer_idx].size = 70;
-
-        let merged = newer.merge(&older);
-
-        assert_eq!(merged.last_sec, 1010);
-        // The stale older slot is aged out -> excluded from the sum.
-        assert_eq!(merged.totals[stale_idx].total, 0);
-        assert_eq!(merged.totals[stale_idx].n, 0);
-        assert_eq!(merged.totals[stale_idx].size, 0);
-        // In-window older data is retained.
-        assert_eq!(merged.totals[kept_idx].total, 3);
-        assert_eq!(merged.totals[kept_idx].n, 1);
-        // newer data is retained.
-        assert_eq!(merged.totals[newer_idx].total, 7);
-    }
-
-    #[test]
-    fn test_last_minute_latency_merge_ages_out_of_window_self_newer() {
-        // self.last_sec > o.last_sec with a full-window gap (>= 60s): all of
-        // `o` is aged out and only `self`'s data remains.
-        let mut newer = LastMinuteLatency::default();
-        let mut older = LastMinuteLatency::default();
-
-        newer.last_sec = 1070;
-        older.last_sec = 1000; // gap of 70 >= 60 -> all of older is aged out
-
-        // 1070 % 60 == 1000 % 60 == 10, so both write the same slot; the fix
-        // must yield exactly newer's value, not newer + stale older.
-        let idx = (1070 % 60) as usize;
-        older.totals[idx].total = 111;
-        older.totals[idx].n = 5;
-        older.totals[idx].size = 999;
-        newer.totals[idx].total = 7;
-        newer.totals[idx].n = 1;
-        newer.totals[idx].size = 70;
-
-        let merged = newer.merge(&older);
-
-        assert_eq!(merged.last_sec, 1070);
-        assert_eq!(merged.totals[idx].total, 7);
-        assert_eq!(merged.totals[idx].n, 1);
-        assert_eq!(merged.totals[idx].size, 70);
-    }
-
-    #[test]
-    fn test_last_minute_latency_merge_ages_out_of_window_o_newer() {
-        // Mirror of the above for the else branch: `self` is older and is
-        // forwarded to o.last_sec, aging out self's out-of-window data.
-        let mut older = LastMinuteLatency::default();
-        let mut newer = LastMinuteLatency::default();
-
-        older.last_sec = 1000;
-        newer.last_sec = 1070; // gap of 70 >= 60 -> all of older (self) is aged out
-
-        let idx = (1070 % 60) as usize; // 1000 % 60 == 1070 % 60 == 10
-        older.totals[idx].total = 111;
-        older.totals[idx].n = 5;
-        older.totals[idx].size = 999;
-        newer.totals[idx].total = 7;
-        newer.totals[idx].n = 1;
-        newer.totals[idx].size = 70;
-
-        let merged = older.merge(&newer);
-
-        assert_eq!(merged.last_sec, 1070);
-        // self's stale data aged out; only newer's data remains.
-        assert_eq!(merged.totals[idx].total, 7);
-        assert_eq!(merged.totals[idx].n, 1);
-        assert_eq!(merged.totals[idx].size, 70);
-    }
-
     #[test]
     fn test_last_minute_latency_window_wraparound() {
         let mut latency = LastMinuteLatency::default();

crates/common/src/metrics.rs

@@ -750,11 +750,6 @@ pub struct Metrics {
     last_scan_cycle_duration_millis: AtomicU64,
     last_scan_cycle_objects_scanned: AtomicU64,
     last_scan_cycle_directories_scanned: AtomicU64,
-    /// Lifetime count of object versions walked by the data scanner, counted
-    /// for every version regardless of whether any lifecycle rule applies.
-    /// This is the honest source for `rustfs_scanner_versions_scanned_total`;
-    /// ILM-checked versions are tracked separately on the `Lifecycle` source.
-    lifetime_versions_scanned: AtomicU64,
     last_scan_cycle_bucket_drive_scans: AtomicU64,
     last_scan_cycle_bucket_drive_failures: AtomicU64,
     last_scan_cycle_yield_events: AtomicU64,
@@ -1111,9 +1106,6 @@ pub struct ScannerMetricsReport {
     pub oldest_active_path_age_seconds: u64,
     pub life_time_ops: HashMap<String, u64>,
     pub life_time_ilm: HashMap<String, u64>,
-    /// Lifetime object versions scanned, independent of ILM configuration.
-    #[serde(default)]
-    pub versions_scanned: u64,
     pub last_minute: ScannerLastMinute,
     pub active_paths: Vec<String>,
     pub current_scan_mode: String,
@@ -1712,7 +1704,6 @@ impl Metrics {
             last_scan_cycle_duration_millis: AtomicU64::new(0),
             last_scan_cycle_objects_scanned: AtomicU64::new(0),
             last_scan_cycle_directories_scanned: AtomicU64::new(0),
-            lifetime_versions_scanned: AtomicU64::new(0),
             last_scan_cycle_bucket_drive_scans: AtomicU64::new(0),
             last_scan_cycle_bucket_drive_failures: AtomicU64::new(0),
             last_scan_cycle_yield_events: AtomicU64::new(0),
@@ -2121,17 +2112,6 @@ impl Metrics {
         self.record_scanner_source_work(source, ScannerSourceWorkUpdate::checked(count));
     }
 
-    /// Record `count` object versions walked by the data scanner.
-    ///
-    /// Every version the scanner visits is counted here, independent of
-    /// lifecycle configuration, so `rustfs_scanner_versions_scanned_total`
-    /// reflects real scan coverage even on clusters with no ILM rules. ILM
-    /// evaluation coverage is recorded separately via the `Lifecycle` source's
-    /// `checked` counter and surfaces as `rustfs_ilm_versions_scanned_total`.
-    pub fn record_scanner_versions_scanned(&self, count: u64) {
-        self.lifetime_versions_scanned.fetch_add(count, Ordering::Relaxed);
-    }
-
     pub fn record_scanner_source_queued(&self, source: ScannerWorkSource, count: u64) {
         self.record_scanner_source_work(source, ScannerSourceWorkUpdate::queued(count));
     }
@@ -2715,7 +2695,6 @@ impl Metrics {
             .map(|(disk, state)| format!("{disk}/{}", state.path))
             .collect();
         m.current_scan_mode = self.current_scan_mode().as_str().to_string();
-        m.versions_scanned = self.lifetime_versions_scanned.load(Ordering::Relaxed);
         m.leader_lock_state = self.scanner_leader_lock_state.read().await.clone();
         m.leader_lock_held_by_this_process = self.scanner_leader_lock_held.load(Ordering::Relaxed);
         m.leader_lock_last_error = self.scanner_leader_lock_last_error.read().await.clone();
@@ -3075,50 +3054,6 @@ mod tests {
         assert_eq!(report.current_disk_bucket_scans_active, 0);
     }
 
-    #[tokio::test]
-    async fn report_counts_scanned_versions_independent_of_lifecycle() {
-        let metrics = Metrics::new();
-
-        // No ILM configuration touched: scanned versions must still accrue so
-        // rustfs_scanner_versions_scanned_total reflects real coverage.
-        metrics.record_scanner_versions_scanned(3);
-        metrics.record_scanner_versions_scanned(5);
-
-        let report = metrics.report().await;
-
-        assert_eq!(report.versions_scanned, 8);
-        // ILM-checked versions live on the Lifecycle source and stay zero when
-        // no lifecycle evaluation ran.
-        let lifecycle_checked = report
-            .source_work
-            .iter()
-            .find(|s| s.source == "lifecycle")
-            .map(|s| s.checked)
-            .unwrap_or_default();
-        assert_eq!(lifecycle_checked, 0);
-    }
-
-    #[tokio::test]
-    async fn report_tracks_lifecycle_checked_versions_separately() {
-        let metrics = Metrics::new();
-
-        // Simulate the scanner walking versions and, on a lifecycle-configured
-        // bucket, handing a subset to the ILM evaluator.
-        metrics.record_scanner_versions_scanned(10);
-        metrics.record_scanner_source_checked(ScannerWorkSource::Lifecycle, 4);
-
-        let report = metrics.report().await;
-
-        assert_eq!(report.versions_scanned, 10, "total scanned versions independent of ILM");
-        let lifecycle_checked = report
-            .source_work
-            .iter()
-            .find(|s| s.source == "lifecycle")
-            .map(|s| s.checked)
-            .unwrap_or_default();
-        assert_eq!(lifecycle_checked, 4, "ILM-checked versions tracked on the Lifecycle source");
-    }
-
     #[tokio::test]
     async fn report_derives_scanner_pacing_pressure() {
         let metrics = Metrics::new();

crates/concurrency/Cargo.toml

@@ -6,17 +6,22 @@ license.workspace = true
 repository.workspace = true
 rust-version.workspace = true
 homepage.workspace = true
-description = "Shared concurrency contract types for RustFS - workload admission, queue snapshots, worker pools, and policy types"
-keywords = ["rustfs", "concurrency", "admission", "backpressure", "workers"]
+description = "Concurrency management for RustFS - timeout, locking, backpressure, and I/O scheduling"
+keywords = ["rustfs", "concurrency", "timeout", "backpressure", "scheduling"]
 categories = ["concurrency", "filesystem"]
 
 [dependencies]
 # Internal crates
 rustfs-io-core = { workspace = true }
+rustfs-io-metrics = { workspace = true }
 serde = { workspace = true }
 
 # Async runtime
-tokio = { workspace = true, features = ["sync"] }
+tokio = { workspace = true, features = ["sync", "time", "rt"] }
+tokio-util = { workspace = true }
+
+# Error handling
+thiserror = { workspace = true }
 
 # Logging
 tracing = { workspace = true }
@@ -24,4 +29,18 @@ tracing = { workspace = true }
 [dev-dependencies]
 insta = { workspace = true }
 serde_json = { workspace = true }
-tokio = { workspace = true, features = ["test-util", "macros", "rt-multi-thread"] }
+tokio = { workspace = true, features = ["test-util","macros","rt-multi-thread"] }
+
+[features]
+default = ["timeout", "lock", "deadlock", "backpressure", "scheduler"]
+
+# Feature modules
+timeout = []
+lock = []
+deadlock = []
+backpressure = []
+scheduler = []
+
+[package.metadata.docs.rs]
+all-features = true
+rustdoc-args = ["--cfg", "docsrs"]

crates/concurrency/src/backpressure.rs

@@ -12,15 +12,17 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-//! Shared backpressure policy type.
-//!
-//! The runtime backpressure implementation (byte-watermark pipes and
-//! monitors) lives in `rustfs/src/storage/backpressure.rs`; this module only
-//! carries the watermark policy type that implementation shares.
+//! Backpressure management
 
-use rustfs_io_core::BackpressureConfig as CoreBackpressureConfig;
+use rustfs_io_core::{
+    BackpressureConfig as CoreBackpressureConfig, BackpressureMonitor as CoreBackpressureMonitor, BackpressureState,
+};
+use rustfs_io_metrics::backpressure_metrics;
+use std::sync::Arc;
+use std::time::Instant;
+use tokio::io::{DuplexStream, duplex};
 
-/// Watermark policy for duplex-pipe backpressure.
+/// Facade policy for duplex-pipe watermark backpressure.
 #[derive(Debug, Clone, Copy)]
 pub struct PipeBackpressurePolicy {
     /// Buffer size in bytes
@@ -52,9 +54,9 @@ impl PipeBackpressurePolicy {
         (self.buffer_size as u64 * self.low_watermark as u64 / 100) as usize
     }
 
-    /// Convert the policy into the reusable io-core admission-pressure config.
+    /// Convert the facade policy into the reusable io-core admission-pressure config.
     ///
-    /// The caller still owns duplex buffer sizing, but the shared
+    /// The concurrency layer still owns duplex buffer sizing, but the shared
     /// overload/admission primitive lives in `io-core`.
     pub fn to_core_config(&self) -> CoreBackpressureConfig {
         CoreBackpressureConfig {
@@ -67,28 +69,157 @@ impl PipeBackpressurePolicy {
     }
 }
 
+/// Backpressure manager
+pub struct BackpressureManager {
+    config: PipeBackpressurePolicy,
+    core_config: CoreBackpressureConfig,
+    monitor: Arc<CoreBackpressureMonitor>,
+}
+
+impl BackpressureManager {
+    /// Create a new backpressure manager
+    pub fn new(buffer_size: usize, high_watermark: u32, low_watermark: u32) -> Self {
+        Self::from_policy(PipeBackpressurePolicy {
+            buffer_size,
+            high_watermark,
+            low_watermark,
+        })
+    }
+
+    /// Create a new backpressure manager from the facade policy type.
+    pub fn from_policy(config: PipeBackpressurePolicy) -> Self {
+        let core_config = config.to_core_config();
+        Self {
+            config,
+            core_config: core_config.clone(),
+            monitor: Arc::new(CoreBackpressureMonitor::new(core_config)),
+        }
+    }
+
+    /// Get the configuration
+    pub fn config(&self) -> &PipeBackpressurePolicy {
+        &self.config
+    }
+
+    /// Get the derived io-core admission-pressure configuration.
+    pub fn core_config(&self) -> &CoreBackpressureConfig {
+        &self.core_config
+    }
+
+    /// Get the monitor
+    pub fn monitor(&self) -> Arc<CoreBackpressureMonitor> {
+        self.monitor.clone()
+    }
+
+    /// Create a backpressure pipe
+    pub fn create_pipe(&self) -> BackpressurePipe {
+        BackpressurePipe::new(self.config, self.monitor.clone())
+    }
+
+    /// Get current state
+    pub fn state(&self) -> BackpressureState {
+        self.monitor.state()
+    }
+
+    /// Check if backpressure is active
+    pub fn is_active(&self) -> bool {
+        self.monitor.is_active()
+    }
+}
+
+/// Backpressure pipe wrapping tokio's duplex
+pub struct BackpressurePipe {
+    reader: DuplexStream,
+    writer: DuplexStream,
+    config: PipeBackpressurePolicy,
+    monitor: Arc<CoreBackpressureMonitor>,
+    created_at: Instant,
+}
+
+/// Shared pipe metadata snapshot for facade-level backpressure pipes.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct BackpressurePipeMeta {
+    /// Configured duplex buffer capacity in bytes.
+    pub buffer_capacity: usize,
+    /// Current backpressure state reported by the shared core monitor.
+    pub state: BackpressureState,
+    /// Age of the pipe since creation.
+    pub age: std::time::Duration,
+}
+
+impl BackpressurePipe {
+    fn new(config: PipeBackpressurePolicy, monitor: Arc<CoreBackpressureMonitor>) -> Self {
+        let (reader, writer) = duplex(config.buffer_size);
+
+        Self {
+            reader,
+            writer,
+            config,
+            monitor,
+            created_at: Instant::now(),
+        }
+    }
+
+    /// Get the reader end
+    pub fn reader(&mut self) -> &mut DuplexStream {
+        &mut self.reader
+    }
+
+    /// Get the writer end
+    pub fn writer(&mut self) -> &mut DuplexStream {
+        &mut self.writer
+    }
+
+    /// Split into reader and writer
+    pub fn into_split(self) -> (DuplexStream, DuplexStream) {
+        (self.reader, self.writer)
+    }
+
+    /// Get the configuration
+    pub fn config(&self) -> &PipeBackpressurePolicy {
+        &self.config
+    }
+
+    /// Get current state
+    pub fn state(&self) -> BackpressureState {
+        self.monitor.state()
+    }
+
+    /// Get the age of this pipe
+    pub fn age(&self) -> std::time::Duration {
+        self.created_at.elapsed()
+    }
+
+    /// Get a compact metadata snapshot for the pipe.
+    pub fn meta(&self) -> BackpressurePipeMeta {
+        BackpressurePipeMeta {
+            buffer_capacity: self.config.buffer_size,
+            state: self.state(),
+            age: self.age(),
+        }
+    }
+
+    /// Check if should apply backpressure
+    pub fn should_apply_backpressure(&self) -> bool {
+        let should = self.monitor.should_apply_backpressure();
+        if should {
+            backpressure_metrics::record_backpressure_activation();
+        }
+        should
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
 
     #[test]
-    fn test_backpressure_policy_defaults() {
+    fn test_backpressure_config() {
         let config = PipeBackpressurePolicy::default();
         assert_eq!(config.buffer_size, 4 * 1024 * 1024);
         assert!(config.high_watermark > config.low_watermark);
     }
 
-    #[test]
-    fn test_backpressure_policy_watermark_bytes() {
-        let config = PipeBackpressurePolicy {
-            buffer_size: 1000,
-            high_watermark: 80,
-            low_watermark: 50,
-        };
-        assert_eq!(config.high_watermark_bytes(), 800);
-        assert_eq!(config.low_watermark_bytes(), 500);
-    }
-
     #[test]
     fn test_backpressure_policy_to_core_config() {
         let policy = PipeBackpressurePolicy::default();
@@ -97,4 +228,33 @@ mod tests {
         assert_eq!(core.low_water_mark, policy.low_watermark as f64 / 100.0);
         assert!(core.enabled);
     }
+
+    #[test]
+    fn test_backpressure_manager() {
+        let manager = BackpressureManager::new(1024, 80, 50);
+        assert_eq!(manager.state(), BackpressureState::Normal);
+    }
+
+    #[test]
+    fn test_backpressure_pipe() {
+        let manager = BackpressureManager::new(1024, 80, 50);
+        let pipe = manager.create_pipe();
+        assert_eq!(pipe.state(), BackpressureState::Normal);
+        assert_eq!(pipe.meta().buffer_capacity, 1024);
+    }
+
+    #[test]
+    fn test_backpressure_pipe_meta_is_read_only() {
+        let manager = BackpressureManager::new(2048, 80, 50);
+        let pipe = manager.create_pipe();
+        let first = pipe.meta();
+        let second = pipe.meta();
+
+        assert_eq!(first.buffer_capacity, 2048);
+        assert_eq!(first.state, BackpressureState::Normal);
+        assert_eq!(second.buffer_capacity, first.buffer_capacity);
+        assert_eq!(second.state, first.state);
+        assert!(second.age >= first.age);
+        assert_eq!(manager.state(), BackpressureState::Normal);
+    }
 }

crates/concurrency/src/config.rs

@@ -0,0 +1,242 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Configuration for concurrency management
+
+use crate::{
+    backpressure::PipeBackpressurePolicy, deadlock::DeadlockMonitorPolicy, scheduler::SchedulerPolicy,
+    timeout::TimeoutManagerPolicy,
+};
+use std::time::Duration;
+
+/// Feature flags for concurrency modules
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct ConcurrencyFeatures {
+    /// Enable timeout control
+    pub timeout: bool,
+    /// Enable lock optimization
+    pub lock: bool,
+    /// Enable deadlock detection
+    pub deadlock: bool,
+    /// Enable backpressure management
+    pub backpressure: bool,
+    /// Enable I/O scheduling
+    pub scheduler: bool,
+}
+
+impl Default for ConcurrencyFeatures {
+    fn default() -> Self {
+        Self {
+            timeout: cfg!(feature = "timeout"),
+            lock: cfg!(feature = "lock"),
+            deadlock: cfg!(feature = "deadlock"),
+            backpressure: cfg!(feature = "backpressure"),
+            scheduler: cfg!(feature = "scheduler"),
+        }
+    }
+}
+
+impl ConcurrencyFeatures {
+    /// Create with all features enabled
+    pub fn all() -> Self {
+        Self {
+            timeout: true,
+            lock: true,
+            deadlock: true,
+            backpressure: true,
+            scheduler: true,
+        }
+    }
+
+    /// Create with no features enabled
+    pub fn none() -> Self {
+        Self {
+            timeout: false,
+            lock: false,
+            deadlock: false,
+            backpressure: false,
+            scheduler: false,
+        }
+    }
+
+    /// Check if any feature is enabled
+    pub fn any_enabled(&self) -> bool {
+        self.timeout || self.lock || self.deadlock || self.backpressure || self.scheduler
+    }
+}
+
+/// Facade policy for lock manager behavior.
+#[derive(Debug, Clone, Copy)]
+pub struct LockManagerPolicy {
+    /// Enable lock optimization.
+    pub enabled: bool,
+    /// Lock acquisition timeout.
+    pub acquire_timeout: Duration,
+}
+
+impl Default for LockManagerPolicy {
+    fn default() -> Self {
+        Self {
+            enabled: true,
+            acquire_timeout: Duration::from_secs(5),
+        }
+    }
+}
+
+/// Main configuration for concurrency management
+#[derive(Debug, Clone, Default)]
+pub struct ConcurrencyConfig {
+    /// Feature flags
+    pub features: ConcurrencyFeatures,
+    /// Timeout facade policy.
+    pub timeout_policy: TimeoutManagerPolicy,
+    /// Lock facade policy.
+    pub lock_policy: LockManagerPolicy,
+    /// Deadlock facade policy.
+    pub deadlock_policy: DeadlockMonitorPolicy,
+    /// Backpressure facade policy.
+    pub backpressure_policy: PipeBackpressurePolicy,
+    /// Scheduler facade policy.
+    pub scheduler_policy: SchedulerPolicy,
+}
+
+impl ConcurrencyConfig {
+    /// Create configuration from environment variables
+    pub fn from_env() -> Self {
+        let mut config = Self::default();
+
+        // Read from environment if available
+        if let Ok(val) = std::env::var("RUSTFS_TIMEOUT_DEFAULT")
+            && let Ok(secs) = val.parse::<u64>()
+        {
+            config.timeout_policy.default_timeout = Duration::from_secs(secs);
+        }
+
+        if let Ok(val) = std::env::var("RUSTFS_TIMEOUT_MAX")
+            && let Ok(secs) = val.parse::<u64>()
+        {
+            config.timeout_policy.max_timeout = Duration::from_secs(secs);
+        }
+
+        if let Ok(val) = std::env::var("RUSTFS_BACKPRESSURE_BUFFER_SIZE")
+            && let Ok(size) = val.parse::<usize>()
+        {
+            config.backpressure_policy.buffer_size = size;
+        }
+
+        if let Ok(val) = std::env::var("RUSTFS_IO_BUFFER_SIZE")
+            && let Ok(size) = val.parse::<usize>()
+        {
+            config.scheduler_policy.base_buffer_size = size;
+        }
+
+        config
+    }
+
+    /// Validate configuration
+    pub fn validate(&self) -> Result<(), ConfigError> {
+        if self.timeout_policy.default_timeout > self.timeout_policy.max_timeout {
+            return Err(ConfigError::InvalidTimeout("default_timeout cannot exceed max_timeout".to_string()));
+        }
+        if self.timeout_policy.min_timeout > self.timeout_policy.max_timeout {
+            return Err(ConfigError::InvalidTimeout("min_timeout cannot exceed max_timeout".to_string()));
+        }
+
+        if self.backpressure_policy.high_watermark <= self.backpressure_policy.low_watermark
+            || self.backpressure_policy.high_watermark > 100
+        {
+            return Err(ConfigError::InvalidBackpressure(
+                "high_watermark must be > low_watermark and <= 100".to_string(),
+            ));
+        }
+
+        if self.scheduler_policy.base_buffer_size > self.scheduler_policy.max_buffer_size {
+            return Err(ConfigError::InvalidScheduler(
+                "base_buffer_size cannot exceed max_buffer_size".to_string(),
+            ));
+        }
+
+        Ok(())
+    }
+}
+
+/// Configuration error
+#[allow(clippy::enum_variant_names)]
+#[derive(Debug, Clone, thiserror::Error)]
+pub enum ConfigError {
+    /// Invalid timeout configuration
+    #[error("Invalid timeout config: {0}")]
+    InvalidTimeout(String),
+
+    /// Invalid backpressure configuration
+    #[error("Invalid backpressure config: {0}")]
+    InvalidBackpressure(String),
+
+    /// Invalid scheduler configuration
+    #[error("Invalid scheduler config: {0}")]
+    InvalidScheduler(String),
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_default_config() {
+        let config = ConcurrencyConfig::default();
+        assert!(config.validate().is_ok());
+    }
+
+    #[test]
+    fn test_invalid_timeout() {
+        let config = ConcurrencyConfig {
+            timeout_policy: TimeoutManagerPolicy {
+                default_timeout: Duration::from_secs(100),
+                max_timeout: Duration::from_secs(50),
+                enable_dynamic: true,
+                ..Default::default()
+            },
+            ..Default::default()
+        };
+        assert!(
+            config.validate().is_err(),
+            "validate() should return an error when default_timeout > max_timeout"
+        );
+    }
+
+    #[test]
+    fn test_invalid_min_timeout() {
+        let config = ConcurrencyConfig {
+            timeout_policy: TimeoutManagerPolicy {
+                min_timeout: Duration::from_secs(100),
+                max_timeout: Duration::from_secs(50),
+                ..Default::default()
+            },
+            ..Default::default()
+        };
+        assert!(
+            config.validate().is_err(),
+            "validate() should return an error when min_timeout > max_timeout"
+        );
+    }
+
+    #[test]
+    fn test_features() {
+        let features = ConcurrencyFeatures::all();
+        assert!(features.any_enabled());
+
+        let features = ConcurrencyFeatures::none();
+        assert!(!features.any_enabled());
+    }
+}

crates/concurrency/src/deadlock.rs

@@ -12,16 +12,15 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-//! Shared deadlock-monitor policy type.
-//!
-//! The runtime request-hang / deadlock detection loop lives in
-//! `rustfs/src/storage/deadlock_detector.rs`; this module only carries the
-//! monitor policy type that implementation shares.
+//! Deadlock detection management
 
-use rustfs_io_core::DeadlockDetectorConfig as CoreDeadlockConfig;
-use std::time::Duration;
+use rustfs_io_core::{DeadlockDetector as CoreDeadlockDetector, DeadlockDetectorConfig as CoreDeadlockConfig, LockType};
+use rustfs_io_metrics::deadlock_metrics;
+use std::collections::HashMap;
+use std::sync::Arc;
+use std::time::{Duration, Instant};
 
-/// Policy for the request-hang deadlock monitor.
+/// Facade policy for the concurrency-layer deadlock monitor.
 #[derive(Debug, Clone, Copy)]
 pub struct DeadlockMonitorPolicy {
     /// Enable deadlock detection
@@ -43,7 +42,7 @@ impl Default for DeadlockMonitorPolicy {
 }
 
 impl DeadlockMonitorPolicy {
-    /// Convert the policy into the reusable io-core deadlock config.
+    /// Convert the facade policy into the reusable io-core deadlock config.
     pub fn to_core_config(&self) -> CoreDeadlockConfig {
         CoreDeadlockConfig {
             enabled: self.enabled,
@@ -53,14 +52,182 @@ impl DeadlockMonitorPolicy {
     }
 }
 
+/// Deadlock manager
+pub struct DeadlockManager {
+    config: DeadlockMonitorPolicy,
+    detector: Arc<CoreDeadlockDetector>,
+    running: Arc<tokio::sync::Mutex<bool>>,
+}
+
+impl DeadlockManager {
+    /// Create a new deadlock manager
+    pub fn new(enabled: bool, check_interval: Duration, hang_threshold: Duration) -> Self {
+        Self::from_policy(DeadlockMonitorPolicy {
+            enabled,
+            check_interval,
+            hang_threshold,
+        })
+    }
+
+    /// Create a new deadlock manager from the facade policy type.
+    pub fn from_policy(config: DeadlockMonitorPolicy) -> Self {
+        let core_config = config.to_core_config();
+        Self {
+            config,
+            detector: Arc::new(CoreDeadlockDetector::new(core_config)),
+            running: Arc::new(tokio::sync::Mutex::new(false)),
+        }
+    }
+
+    /// Get the configuration
+    pub fn config(&self) -> &DeadlockMonitorPolicy {
+        &self.config
+    }
+
+    /// Get the core detector
+    pub fn detector(&self) -> Arc<CoreDeadlockDetector> {
+        self.detector.clone()
+    }
+
+    /// Start the deadlock detection background task
+    pub async fn start(&self) {
+        if !self.config.enabled {
+            return;
+        }
+
+        let mut running = self.running.lock().await;
+        if *running {
+            return;
+        }
+        *running = true;
+        drop(running);
+
+        tracing::info!(
+            event = "deadlock_monitor.lifecycle",
+            component = "concurrency",
+            subsystem = "deadlock",
+            state = "started",
+            check_interval_ms = self.config.check_interval.as_millis(),
+            hang_threshold_ms = self.config.hang_threshold.as_millis(),
+            "deadlock monitor state changed"
+        );
+    }
+
+    /// Stop the deadlock detection
+    pub async fn stop(&self) {
+        let mut running = self.running.lock().await;
+        *running = false;
+
+        tracing::info!(
+            event = "deadlock_monitor.lifecycle",
+            component = "concurrency",
+            subsystem = "deadlock",
+            state = "stopped",
+            check_interval_ms = self.config.check_interval.as_millis(),
+            hang_threshold_ms = self.config.hang_threshold.as_millis(),
+            "deadlock monitor state changed"
+        );
+    }
+
+    /// Create a request tracker
+    pub fn track_request(&self, request_id: String, description: String) -> RequestTracker {
+        RequestTracker::new(request_id, description, self.detector.clone())
+    }
+
+    /// Register a lock
+    pub fn register_lock(&self, lock_type: LockType) -> u64 {
+        self.detector.register_lock(lock_type)
+    }
+
+    /// Unregister a lock
+    pub fn unregister_lock(&self, lock_id: u64) {
+        self.detector.unregister_lock(lock_id);
+    }
+
+    /// Detect deadlock
+    pub fn detect_deadlock(&self) -> Option<Vec<u64>> {
+        let result = self.detector.detect_deadlock();
+        if let Some(ref cycle) = result {
+            deadlock_metrics::record_deadlock_detected(cycle.len());
+        }
+        result
+    }
+}
+
+/// Lightweight compatibility wrapper for request-scoped deadlock bookkeeping.
+///
+/// This type intentionally stays minimal in the concurrency layer. Rich
+/// request-level lock/resource diagnostics belong to
+/// `rustfs::storage::deadlock_detector::RequestResourceTracker`.
+pub struct RequestTracker {
+    request_id: String,
+    description: String,
+    start_time: Instant,
+    resources: HashMap<String, Vec<String>>,
+    detector: Arc<CoreDeadlockDetector>,
+}
+
+impl RequestTracker {
+    fn new(request_id: String, description: String, detector: Arc<CoreDeadlockDetector>) -> Self {
+        let start_time = Instant::now();
+        detector.register_request(&request_id, 1); // Use placeholder thread ID
+
+        Self {
+            request_id,
+            description,
+            start_time,
+            resources: HashMap::new(),
+            detector,
+        }
+    }
+
+    /// Get the request ID
+    pub fn request_id(&self) -> &str {
+        &self.request_id
+    }
+
+    /// Get the description
+    pub fn description(&self) -> &str {
+        &self.description
+    }
+
+    /// Get the elapsed time
+    pub fn elapsed(&self) -> Duration {
+        self.start_time.elapsed()
+    }
+
+    /// Record a lock acquisition
+    pub fn record_lock_acquire(&mut self, lock_id: u64, resource: String) {
+        self.resources.entry("locks".to_string()).or_default().push(resource);
+        self.detector.record_acquire(lock_id, 1); // Use placeholder thread ID
+        deadlock_metrics::record_lock_acquisition("read");
+    }
+
+    /// Return a read-only view of tracked resource names.
+    pub fn resources(&self) -> &HashMap<String, Vec<String>> {
+        &self.resources
+    }
+
+    /// Record a lock release
+    pub fn record_lock_release(&mut self, lock_id: u64) {
+        self.detector.record_release(lock_id);
+    }
+}
+
+impl Drop for RequestTracker {
+    fn drop(&mut self) {
+        self.detector.unregister_request(&self.request_id);
+    }
+}
+
 #[cfg(test)]
 mod tests {
     use super::*;
 
     #[test]
-    fn test_deadlock_policy_defaults_disabled() {
-        let policy = DeadlockMonitorPolicy::default();
-        assert!(!policy.enabled);
+    fn test_deadlock_manager_creation() {
+        let manager = DeadlockManager::new(false, Duration::from_secs(10), Duration::from_secs(60));
+        assert!(!manager.config().enabled);
     }
 
     #[test]
@@ -71,4 +238,16 @@ mod tests {
         assert_eq!(core.detection_interval, policy.check_interval);
         assert_eq!(core.max_hold_time, policy.hang_threshold);
     }
+
+    #[tokio::test]
+    async fn test_request_tracker() {
+        let manager = DeadlockManager::new(true, Duration::from_secs(10), Duration::from_secs(60));
+        let mut tracker = manager.track_request("req-1".to_string(), "test request".to_string());
+        let lock_id = manager.register_lock(LockType::Mutex);
+        tracker.record_lock_acquire(lock_id, "bucket/key".to_string());
+
+        assert_eq!(tracker.request_id(), "req-1");
+        assert_eq!(tracker.description(), "test request");
+        assert_eq!(tracker.resources().get("locks").map(Vec::len), Some(1));
+    }
 }

crates/concurrency/src/lib.rs

@@ -12,40 +12,167 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-//! # RustFS Concurrency Contracts
+//! # RustFS Concurrency Management
 //!
-//! Shared concurrency contract types for RustFS:
+//! This crate provides comprehensive concurrency management for RustFS,
+//! including timeout control, lock optimization, deadlock detection,
+//! backpressure management, and I/O scheduling.
 //!
-//! - [`workload`]: workload admission snapshot/contract types consumed by
-//!   ecstore, heal, and the server for admission-aware throttling.
-//! - [`GetObjectQueueSnapshot`]: disk permit queue usage snapshot for
-//!   GetObject orchestration.
-//! - [`workers`]: a bounded semaphore-backed worker-slot pool.
-//! - [`PipeBackpressurePolicy`] / [`DeadlockMonitorPolicy`]: policy types
-//!   shared with the runtime implementations.
+//! ## Features
 //!
-//! The actual runtime concurrency control — size-aware timeouts, byte-watermark
-//! backpressure, request-hang/deadlock detection, and I/O scheduling — is
-//! implemented in `rustfs/src/storage/*` on top of the `rustfs-io-core`
-//! primitives. This crate only carries the data and contract types those
-//! implementations share; it does not run any background tasks itself.
+//! All features are controlled by feature flags and can be enabled/disabled at compile time:
+//!
+//! - **timeout**: Dynamic timeout calculation based on data size and transfer rate
+//! - **lock**: Early lock release to reduce contention
+//! - **deadlock**: Request tracking and cycle detection
+//! - **backpressure**: Buffer-based flow control
+//! - **scheduler**: Adaptive buffer sizing and priority queuing
+//!
+//! ## Architecture
+//!
+//! ```text
+//! rustfs-concurrency (Business Layer)
+//!     ├── timeout (Timeout Control)
+//!     ├── lock (Lock Optimization)
+//!     ├── deadlock (Deadlock Detection)
+//!     ├── backpressure (Backpressure Management)
+//!     └── scheduler (I/O Scheduling)
+//!          │
+//!          ├── rustfs-io-core (Core Algorithms)
+//!          └── rustfs-io-metrics (Metrics Collection)
+//! ```
+//!
+//! ## Usage
+//!
+//! ```rust,no_run
+//! use rustfs_concurrency::{ConcurrencyConfig, ConcurrencyManager};
+//!
+//! # #[tokio::main]
+//! # async fn main() {
+//! // Create manager with all features enabled
+//! let config = ConcurrencyConfig::default();
+//! let manager = ConcurrencyManager::new(config);
+//!
+//! // Start services
+//! manager.start().await;
+//!
+//! // Use timeout control (if enabled)
+//! if manager.is_timeout_enabled() {
+//!     let timeout_manager = manager.timeout();
+//!     let _ = timeout_manager;
+//! }
+//!
+//! // Use lock optimization (if enabled)
+//! if manager.is_lock_enabled() {
+//!     let lock_manager = manager.lock();
+//!     let _ = lock_manager;
+//! }
+//!
+//! // Stop services
+//! manager.stop().await;
+//! # }
+//! ```
 
 #![deny(missing_docs)]
 #![deny(unsafe_code)]
+#![cfg_attr(docsrs, feature(doc_cfg))]
 
-// Re-exported io-core type used by downstream timeout implementations.
-pub use rustfs_io_core::OperationProgress;
+// Re-export core types from io-core
+pub use rustfs_io_core::{
+    // Backpressure types
+    BackpressureConfig as CoreBackpressureConfig,
+    BackpressureMonitor as CoreBackpressureMonitor,
+    BackpressureState,
 
-mod backpressure;
+    // Deadlock types
+    DeadlockDetector as CoreDeadlockDetector,
+    IoLoadLevel,
+    IoLoadMetrics,
+    IoPriority,
+    // Scheduler types
+    IoScheduler,
+    IoSchedulingContext,
+    LockInfo,
+    LockOptimizer as CoreLockOptimizer,
+
+    // Lock types
+    LockStats as CoreLockStats,
+    LockType,
+    // Timeout types
+    OperationProgress,
+    TimeoutError,
+    TimeoutStats,
+    WaitGraphEdge,
+
+    calculate_adaptive_timeout,
+    estimate_bytes_per_second,
+};
+
+// Module declarations with feature gates
+#[cfg(feature = "timeout")]
+mod timeout;
+
+#[cfg(feature = "lock")]
+mod lock;
+
+#[cfg(feature = "deadlock")]
 mod deadlock;
-mod queue;
+
+#[cfg(feature = "backpressure")]
+mod backpressure;
+
+#[cfg(feature = "scheduler")]
+mod scheduler;
+
 pub mod workers;
 pub mod workload;
 
-pub use backpressure::PipeBackpressurePolicy;
-pub use deadlock::DeadlockMonitorPolicy;
-pub use queue::GetObjectQueueSnapshot;
+// Public module exports with feature gates
+#[cfg(feature = "timeout")]
+pub use timeout::{TimeoutGuard, TimeoutManager, TimeoutManagerPolicy};
+
+#[cfg(feature = "lock")]
+pub use lock::{LockConfig, LockManager, LockScopeGuard, OptimizedLockGuard};
+
+#[cfg(feature = "deadlock")]
+pub use deadlock::{DeadlockManager, DeadlockMonitorPolicy, RequestTracker};
+
+#[cfg(feature = "backpressure")]
+pub use backpressure::{BackpressureManager, BackpressurePipe, PipeBackpressurePolicy};
+
+#[cfg(feature = "scheduler")]
+pub use scheduler::{IoStrategy, SchedulerManager, SchedulerPolicy};
+
+// Configuration
+mod config;
+pub use config::{ConcurrencyConfig, ConcurrencyFeatures};
+
+// Manager
+mod manager;
+pub use manager::{ConcurrencyManager, GetObjectQueueSnapshot};
 pub use workload::{
     AdmissionState, WorkloadAdmissionRegistrySnapshot, WorkloadAdmissionSnapshot, WorkloadAdmissionSnapshotProvider,
     WorkloadClass,
 };
+
+// Prelude for convenient imports
+pub mod prelude {
+    //! Prelude module for convenient imports
+
+    #[cfg(feature = "timeout")]
+    pub use crate::timeout::{TimeoutGuard, TimeoutManager, TimeoutManagerPolicy};
+
+    #[cfg(feature = "lock")]
+    pub use crate::lock::{LockConfig, LockManager, LockScopeGuard, OptimizedLockGuard};
+
+    #[cfg(feature = "deadlock")]
+    pub use crate::deadlock::{DeadlockManager, DeadlockMonitorPolicy, RequestTracker};
+
+    #[cfg(feature = "backpressure")]
+    pub use crate::backpressure::{BackpressureManager, BackpressurePipe, PipeBackpressurePolicy};
+
+    #[cfg(feature = "scheduler")]
+    pub use crate::scheduler::{IoStrategy, SchedulerManager, SchedulerPolicy};
+
+    pub use crate::{AdmissionState, ConcurrencyConfig, ConcurrencyFeatures, ConcurrencyManager, WorkloadAdmissionSnapshot};
+}

crates/concurrency/src/lock.rs

@@ -0,0 +1,227 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Lock optimization management
+
+use rustfs_io_core::{LockOptimizer as CoreLockOptimizer, LockStats};
+use rustfs_io_metrics::lock_metrics;
+use std::sync::Arc;
+use std::time::{Duration, Instant};
+
+/// Lock configuration
+#[derive(Debug, Clone)]
+pub struct LockConfig {
+    /// Enable lock optimization
+    pub enabled: bool,
+    /// Lock acquisition timeout
+    pub acquire_timeout: Duration,
+}
+
+impl Default for LockConfig {
+    fn default() -> Self {
+        Self {
+            enabled: true,
+            acquire_timeout: Duration::from_secs(5),
+        }
+    }
+}
+
+/// Lock manager
+pub struct LockManager {
+    config: LockConfig,
+    optimizer: Arc<CoreLockOptimizer>,
+}
+
+impl LockManager {
+    /// Create a new lock manager
+    pub fn new(enabled: bool, acquire_timeout: Duration) -> Self {
+        let config = LockConfig {
+            enabled,
+            acquire_timeout,
+        };
+
+        let core_config = rustfs_io_core::LockOptimizeConfig {
+            enabled,
+            acquire_timeout,
+            max_hold_time_warning: Duration::from_millis(100),
+            adaptive_spin: true,
+            max_spin_iterations: 1000,
+        };
+
+        Self {
+            config,
+            optimizer: Arc::new(CoreLockOptimizer::new(core_config)),
+        }
+    }
+
+    /// Get the configuration
+    pub fn config(&self) -> &LockConfig {
+        &self.config
+    }
+
+    /// Get the core optimizer
+    pub fn optimizer(&self) -> Arc<CoreLockOptimizer> {
+        self.optimizer.clone()
+    }
+
+    /// Get lock statistics
+    pub fn stats(&self) -> &LockStats {
+        self.optimizer.stats()
+    }
+
+    /// Optimize a lock guard
+    pub fn optimize<G>(&self, guard: G, resource: impl Into<String>) -> OptimizedLockGuard<G> {
+        OptimizedLockGuard::new(guard, resource, self.optimizer.clone())
+    }
+
+    /// Check if optimization is enabled
+    pub fn is_enabled(&self) -> bool {
+        self.config.enabled
+    }
+}
+
+/// Optimized lock guard with early release support
+pub struct OptimizedLockGuard<G> {
+    guard: Option<G>,
+    acquire_time: Instant,
+    released: bool,
+    resource: String,
+    optimizer: Arc<CoreLockOptimizer>,
+}
+
+impl<G> OptimizedLockGuard<G> {
+    fn new(guard: G, resource: impl Into<String>, optimizer: Arc<CoreLockOptimizer>) -> Self {
+        optimizer.on_acquire();
+        lock_metrics::record_lock_optimization_enabled(optimizer.config().enabled);
+
+        Self {
+            guard: Some(guard),
+            acquire_time: Instant::now(),
+            released: false,
+            resource: resource.into(),
+            optimizer,
+        }
+    }
+
+    /// Get the lock hold time
+    pub fn hold_time(&self) -> Duration {
+        self.acquire_time.elapsed()
+    }
+
+    /// Check if the lock has been released
+    pub fn is_released(&self) -> bool {
+        self.released
+    }
+
+    /// Release the lock early
+    pub fn early_release(&mut self) {
+        if self.released {
+            return;
+        }
+
+        let hold_time = self.hold_time();
+        self.guard.take();
+        self.released = true;
+
+        self.optimizer.on_release(hold_time);
+        lock_metrics::record_lock_hold_time(hold_time);
+
+        tracing::debug!(
+            event = "lock_guard.release",
+            component = "concurrency",
+            subsystem = "lock",
+            release_mode = "early",
+            resource = %self.resource,
+            hold_time_ms = hold_time.as_millis(),
+            "lock guard released"
+        );
+    }
+
+    /// Get a reference to the underlying guard
+    pub fn as_ref(&self) -> Option<&G> {
+        if self.released { None } else { self.guard.as_ref() }
+    }
+}
+
+impl<G> Drop for OptimizedLockGuard<G> {
+    fn drop(&mut self) {
+        if !self.released {
+            let hold_time = self.hold_time();
+            self.guard.take();
+            self.released = true;
+
+            self.optimizer.on_release(hold_time);
+            lock_metrics::record_lock_hold_time(hold_time);
+
+            tracing::debug!(
+                event = "lock_guard.release",
+                component = "concurrency",
+                subsystem = "lock",
+                release_mode = "drop",
+                resource = %self.resource,
+                hold_time_ms = hold_time.as_millis(),
+                "lock guard released"
+            );
+        }
+    }
+}
+
+/// Lock scope guard for RAII semantics
+pub struct LockScopeGuard<G> {
+    guard: Option<G>,
+}
+
+impl<G> LockScopeGuard<G> {
+    /// Create a new scope guard
+    pub fn new(guard: G) -> Self {
+        Self { guard: Some(guard) }
+    }
+
+    /// Get a reference to the guard
+    pub fn as_ref(&self) -> Option<&G> {
+        self.guard.as_ref()
+    }
+}
+
+impl<G> Drop for LockScopeGuard<G> {
+    fn drop(&mut self) {
+        self.guard.take();
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+    use std::sync::Mutex;
+
+    #[test]
+    fn test_lock_manager_creation() {
+        let manager = LockManager::new(true, Duration::from_secs(5));
+        assert!(manager.is_enabled());
+    }
+
+    #[test]
+    fn test_optimized_lock_guard() {
+        let manager = LockManager::new(true, Duration::from_secs(5));
+        let mutex = Mutex::new(42);
+        let guard = mutex.lock().unwrap();
+
+        let optimized = manager.optimize(guard, "test_resource");
+        assert!(!optimized.is_released());
+
+        let mut optimized = optimized;
+        optimized.early_release();
+        assert!(optimized.is_released());
+    }
+}

crates/concurrency/src/manager.rs

@@ -0,0 +1,397 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Main concurrency manager
+
+use crate::config::{ConcurrencyConfig, ConfigError};
+use std::sync::Arc;
+
+/// Snapshot of disk permit queue usage for GetObject orchestration.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct GetObjectQueueSnapshot {
+    /// Total permits configured for disk reads.
+    pub total_permits: usize,
+    /// Permits currently in use.
+    pub permits_in_use: usize,
+}
+
+impl GetObjectQueueSnapshot {
+    /// Create a queue snapshot from total and available permits.
+    pub fn from_available_permits(total_permits: usize, available_permits: usize) -> Self {
+        Self {
+            total_permits,
+            permits_in_use: total_permits.saturating_sub(available_permits),
+        }
+    }
+
+    /// Return currently available permits.
+    pub fn permits_available(&self) -> usize {
+        self.total_permits.saturating_sub(self.permits_in_use)
+    }
+
+    /// Return queue utilization percentage in the 0-100 range.
+    pub fn utilization_percent(&self) -> f64 {
+        if self.total_permits == 0 {
+            0.0
+        } else {
+            (self.permits_in_use as f64 / self.total_permits as f64) * 100.0
+        }
+    }
+
+    /// Return whether the queue is considered congested.
+    pub fn is_congested(&self, threshold_percent: f64) -> bool {
+        self.utilization_percent() > threshold_percent
+    }
+}
+
+/// Main concurrency manager that provides access to all concurrency features
+pub struct ConcurrencyManager {
+    config: ConcurrencyConfig,
+
+    #[cfg(feature = "timeout")]
+    timeout: Arc<crate::timeout::TimeoutManager>,
+
+    #[cfg(feature = "lock")]
+    lock: Arc<crate::lock::LockManager>,
+
+    #[cfg(feature = "deadlock")]
+    deadlock: Arc<crate::deadlock::DeadlockManager>,
+
+    #[cfg(feature = "backpressure")]
+    backpressure: Arc<crate::backpressure::BackpressureManager>,
+
+    #[cfg(feature = "scheduler")]
+    scheduler: Arc<crate::scheduler::SchedulerManager>,
+}
+
+impl ConcurrencyManager {
+    fn build(config: ConcurrencyConfig) -> Self {
+        Self {
+            #[cfg(feature = "timeout")]
+            timeout: Arc::new(crate::timeout::TimeoutManager::from_policy(config.timeout_policy)),
+
+            #[cfg(feature = "lock")]
+            lock: Arc::new(crate::lock::LockManager::new(
+                config.lock_policy.enabled,
+                config.lock_policy.acquire_timeout,
+            )),
+
+            #[cfg(feature = "deadlock")]
+            deadlock: Arc::new(crate::deadlock::DeadlockManager::from_policy(config.deadlock_policy)),
+
+            #[cfg(feature = "backpressure")]
+            backpressure: Arc::new(crate::backpressure::BackpressureManager::from_policy(config.backpressure_policy)),
+
+            #[cfg(feature = "scheduler")]
+            scheduler: Arc::new(crate::scheduler::SchedulerManager::from_policy(config.scheduler_policy)),
+
+            config,
+        }
+    }
+
+    /// Try to create a new concurrency manager with the given configuration.
+    pub fn try_new(config: ConcurrencyConfig) -> Result<Self, ConfigError> {
+        config.validate()?;
+        Ok(Self::build(config))
+    }
+
+    /// Create a new concurrency manager with the given configuration.
+    ///
+    /// Invalid configurations are downgraded to the default configuration instead of
+    /// panicking so startup/runtime callers can remain fail-safe in production paths.
+    pub fn new(config: ConcurrencyConfig) -> Self {
+        match Self::try_new(config) {
+            Ok(manager) => manager,
+            Err(err) => {
+                tracing::warn!(
+                    event = "concurrency_manager.invalid_config_fallback",
+                    component = "concurrency",
+                    subsystem = "manager",
+                    error = %err,
+                    "Invalid concurrency configuration detected; falling back to defaults"
+                );
+                Self::build(ConcurrencyConfig::default())
+            }
+        }
+    }
+
+    /// Create with default configuration
+    pub fn with_defaults() -> Self {
+        Self::build(ConcurrencyConfig::default())
+    }
+
+    /// Try to create a manager from environment-derived configuration.
+    pub fn try_from_env() -> Result<Self, ConfigError> {
+        Self::try_new(ConcurrencyConfig::from_env())
+    }
+
+    /// Create from environment variables
+    pub fn from_env() -> Self {
+        match Self::try_from_env() {
+            Ok(manager) => manager,
+            Err(err) => {
+                tracing::warn!(
+                    event = "concurrency_manager.invalid_env_fallback",
+                    component = "concurrency",
+                    subsystem = "manager",
+                    error = %err,
+                    "Invalid environment-derived concurrency configuration detected; falling back to defaults"
+                );
+                Self::build(ConcurrencyConfig::default())
+            }
+        }
+    }
+
+    /// Get the configuration
+    pub fn config(&self) -> &ConcurrencyConfig {
+        &self.config
+    }
+
+    // ============================================
+    // Feature enable checks
+    // ============================================
+
+    /// Check if timeout feature is enabled
+    pub fn is_timeout_enabled(&self) -> bool {
+        #[cfg(feature = "timeout")]
+        {
+            self.config.features.timeout
+        }
+        #[cfg(not(feature = "timeout"))]
+        {
+            false
+        }
+    }
+
+    /// Check if lock feature is enabled
+    pub fn is_lock_enabled(&self) -> bool {
+        #[cfg(feature = "lock")]
+        {
+            self.config.features.lock
+        }
+        #[cfg(not(feature = "lock"))]
+        {
+            false
+        }
+    }
+
+    /// Check if deadlock feature is enabled
+    pub fn is_deadlock_enabled(&self) -> bool {
+        #[cfg(feature = "deadlock")]
+        {
+            self.config.features.deadlock
+        }
+        #[cfg(not(feature = "deadlock"))]
+        {
+            false
+        }
+    }
+
+    /// Check if backpressure feature is enabled
+    pub fn is_backpressure_enabled(&self) -> bool {
+        #[cfg(feature = "backpressure")]
+        {
+            self.config.features.backpressure
+        }
+        #[cfg(not(feature = "backpressure"))]
+        {
+            false
+        }
+    }
+
+    /// Check if scheduler feature is enabled
+    pub fn is_scheduler_enabled(&self) -> bool {
+        #[cfg(feature = "scheduler")]
+        {
+            self.config.features.scheduler
+        }
+        #[cfg(not(feature = "scheduler"))]
+        {
+            false
+        }
+    }
+
+    // ============================================
+    // Feature accessors
+    // ============================================
+
+    /// Get timeout manager
+    #[cfg(feature = "timeout")]
+    pub fn timeout(&self) -> Arc<crate::timeout::TimeoutManager> {
+        self.timeout.clone()
+    }
+
+    /// Get lock manager
+    #[cfg(feature = "lock")]
+    pub fn lock(&self) -> Arc<crate::lock::LockManager> {
+        self.lock.clone()
+    }
+
+    /// Get deadlock manager
+    #[cfg(feature = "deadlock")]
+    pub fn deadlock(&self) -> Arc<crate::deadlock::DeadlockManager> {
+        self.deadlock.clone()
+    }
+
+    /// Get backpressure manager
+    #[cfg(feature = "backpressure")]
+    pub fn backpressure(&self) -> Arc<crate::backpressure::BackpressureManager> {
+        self.backpressure.clone()
+    }
+
+    /// Get scheduler manager
+    #[cfg(feature = "scheduler")]
+    pub fn scheduler(&self) -> Arc<crate::scheduler::SchedulerManager> {
+        self.scheduler.clone()
+    }
+
+    // ============================================
+    // Lifecycle management
+    // ============================================
+
+    /// Start all enabled services (e.g., deadlock detection background task)
+    pub async fn start(&self) {
+        #[cfg(feature = "deadlock")]
+        {
+            if self.config.deadlock_policy.enabled {
+                self.deadlock.start().await;
+            }
+        }
+
+        tracing::info!(
+            event = "concurrency_manager.lifecycle",
+            component = "concurrency",
+            subsystem = "manager",
+            state = "started",
+            timeout_enabled = self.is_timeout_enabled(),
+            lock_enabled = self.is_lock_enabled(),
+            deadlock_enabled = self.is_deadlock_enabled(),
+            backpressure_enabled = self.is_backpressure_enabled(),
+            scheduler_enabled = self.is_scheduler_enabled(),
+            "concurrency manager state changed"
+        );
+    }
+
+    /// Stop all services
+    pub async fn stop(&self) {
+        #[cfg(feature = "deadlock")]
+        {
+            self.deadlock.stop().await;
+        }
+
+        tracing::info!(
+            event = "concurrency_manager.lifecycle",
+            component = "concurrency",
+            subsystem = "manager",
+            state = "stopped",
+            "concurrency manager state changed"
+        );
+    }
+}
+
+impl Default for ConcurrencyManager {
+    fn default() -> Self {
+        Self::with_defaults()
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_queue_snapshot() {
+        let snapshot = GetObjectQueueSnapshot::from_available_permits(64, 16);
+        assert_eq!(snapshot.permits_in_use, 48);
+        assert_eq!(snapshot.permits_available(), 16);
+        assert!(snapshot.is_congested(70.0));
+    }
+
+    #[test]
+    fn test_queue_snapshot_clamps_over_available_permits() {
+        let snapshot = GetObjectQueueSnapshot::from_available_permits(64, 96);
+        assert_eq!(snapshot.permits_in_use, 0);
+        assert_eq!(snapshot.permits_available(), 64);
+        assert_eq!(snapshot.utilization_percent(), 0.0);
+        assert!(!snapshot.is_congested(0.0));
+    }
+
+    #[test]
+    fn test_queue_snapshot_handles_zero_total_permits() {
+        let snapshot = GetObjectQueueSnapshot::from_available_permits(0, 0);
+        assert_eq!(snapshot.permits_in_use, 0);
+        assert_eq!(snapshot.permits_available(), 0);
+        assert_eq!(snapshot.utilization_percent(), 0.0);
+        assert!(!snapshot.is_congested(0.0));
+    }
+
+    #[test]
+    fn test_manager_creation() {
+        let manager = ConcurrencyManager::with_defaults();
+        assert!(manager.config().validate().is_ok());
+    }
+
+    #[test]
+    fn test_try_new_returns_error_for_invalid_config() {
+        let config = ConcurrencyConfig {
+            timeout_policy: crate::TimeoutManagerPolicy {
+                default_timeout: std::time::Duration::from_secs(10),
+                max_timeout: std::time::Duration::from_secs(1),
+                ..Default::default()
+            },
+            ..Default::default()
+        };
+
+        let result = ConcurrencyManager::try_new(config);
+        assert!(matches!(result, Err(ConfigError::InvalidTimeout(_))));
+    }
+
+    #[test]
+    fn test_new_falls_back_to_default_for_invalid_config() {
+        let config = ConcurrencyConfig {
+            timeout_policy: crate::TimeoutManagerPolicy {
+                default_timeout: std::time::Duration::from_secs(10),
+                max_timeout: std::time::Duration::from_secs(1),
+                ..Default::default()
+            },
+            ..Default::default()
+        };
+
+        let manager = ConcurrencyManager::new(config);
+        assert!(manager.config().validate().is_ok());
+        assert_eq!(
+            manager.config().timeout_policy.default_timeout,
+            ConcurrencyConfig::default().timeout_policy.default_timeout
+        );
+    }
+
+    #[tokio::test]
+    async fn test_manager_lifecycle() {
+        let manager = ConcurrencyManager::with_defaults();
+        manager.start().await;
+        manager.stop().await;
+    }
+
+    #[test]
+    fn test_feature_checks() {
+        let manager = ConcurrencyManager::with_defaults();
+
+        // These should return the feature flag status
+        let _ = manager.is_timeout_enabled();
+        let _ = manager.is_lock_enabled();
+        let _ = manager.is_deadlock_enabled();
+        let _ = manager.is_backpressure_enabled();
+        let _ = manager.is_scheduler_enabled();
+    }
+}

crates/concurrency/src/queue.rs

@@ -1,84 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Disk permit queue snapshot for GetObject orchestration.
-
-/// Snapshot of disk permit queue usage for GetObject orchestration.
-#[derive(Debug, Clone, Copy, PartialEq, Eq)]
-pub struct GetObjectQueueSnapshot {
-    /// Total permits configured for disk reads.
-    pub total_permits: usize,
-    /// Permits currently in use.
-    pub permits_in_use: usize,
-}
-
-impl GetObjectQueueSnapshot {
-    /// Create a queue snapshot from total and available permits.
-    pub fn from_available_permits(total_permits: usize, available_permits: usize) -> Self {
-        Self {
-            total_permits,
-            permits_in_use: total_permits.saturating_sub(available_permits),
-        }
-    }
-
-    /// Return currently available permits.
-    pub fn permits_available(&self) -> usize {
-        self.total_permits.saturating_sub(self.permits_in_use)
-    }
-
-    /// Return queue utilization percentage in the 0-100 range.
-    pub fn utilization_percent(&self) -> f64 {
-        if self.total_permits == 0 {
-            0.0
-        } else {
-            (self.permits_in_use as f64 / self.total_permits as f64) * 100.0
-        }
-    }
-
-    /// Return whether the queue is considered congested.
-    pub fn is_congested(&self, threshold_percent: f64) -> bool {
-        self.utilization_percent() > threshold_percent
-    }
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[test]
-    fn test_queue_snapshot() {
-        let snapshot = GetObjectQueueSnapshot::from_available_permits(64, 16);
-        assert_eq!(snapshot.permits_in_use, 48);
-        assert_eq!(snapshot.permits_available(), 16);
-        assert!(snapshot.is_congested(70.0));
-    }
-
-    #[test]
-    fn test_queue_snapshot_clamps_over_available_permits() {
-        let snapshot = GetObjectQueueSnapshot::from_available_permits(64, 96);
-        assert_eq!(snapshot.permits_in_use, 0);
-        assert_eq!(snapshot.permits_available(), 64);
-        assert_eq!(snapshot.utilization_percent(), 0.0);
-        assert!(!snapshot.is_congested(0.0));
-    }
-
-    #[test]
-    fn test_queue_snapshot_handles_zero_total_permits() {
-        let snapshot = GetObjectQueueSnapshot::from_available_permits(0, 0);
-        assert_eq!(snapshot.permits_in_use, 0);
-        assert_eq!(snapshot.permits_available(), 0);
-        assert_eq!(snapshot.utilization_percent(), 0.0);
-        assert!(!snapshot.is_congested(0.0));
-    }
-}

crates/concurrency/src/scheduler.rs

@@ -0,0 +1,278 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! I/O scheduler management
+
+use rustfs_io_core::{
+    IoLoadLevel, IoPriority, IoScheduler as CoreIoScheduler, IoSchedulingContext,
+    io_profile::{AccessPattern, StorageMedia},
+};
+use rustfs_io_metrics::io_metrics;
+use std::sync::Arc;
+use std::time::Duration;
+
+/// Facade policy for the concurrency-layer scheduler manager.
+#[derive(Debug, Clone, Copy)]
+pub struct SchedulerPolicy {
+    /// Base buffer size
+    pub base_buffer_size: usize,
+    /// Maximum buffer size
+    pub max_buffer_size: usize,
+    /// High priority threshold
+    pub high_priority_threshold: usize,
+    /// Low priority threshold
+    pub low_priority_threshold: usize,
+}
+
+impl Default for SchedulerPolicy {
+    fn default() -> Self {
+        Self {
+            base_buffer_size: 64 * 1024,              // 64KB
+            max_buffer_size: 4 * 1024 * 1024,         // 4MB
+            high_priority_threshold: 1024 * 1024,     // 1MB
+            low_priority_threshold: 10 * 1024 * 1024, // 10MB
+        }
+    }
+}
+
+impl SchedulerPolicy {
+    /// Convert facade policy to io-core scheduler config.
+    pub fn to_core_config(&self) -> rustfs_io_core::IoSchedulerConfig {
+        rustfs_io_core::IoSchedulerConfig {
+            base_buffer_size: self.base_buffer_size,
+            max_buffer_size: self.max_buffer_size,
+            high_priority_size_threshold: self.high_priority_threshold,
+            low_priority_size_threshold: self.low_priority_threshold,
+            ..rustfs_io_core::IoSchedulerConfig::default()
+        }
+    }
+}
+
+/// Scheduler manager
+pub struct SchedulerManager {
+    config: SchedulerPolicy,
+    core_config: rustfs_io_core::IoSchedulerConfig,
+    scheduler: Arc<CoreIoScheduler>,
+}
+
+impl SchedulerManager {
+    /// Create a new scheduler manager
+    pub fn new(
+        base_buffer_size: usize,
+        max_buffer_size: usize,
+        high_priority_threshold: usize,
+        low_priority_threshold: usize,
+    ) -> Self {
+        Self::from_policy(SchedulerPolicy {
+            base_buffer_size,
+            max_buffer_size,
+            high_priority_threshold,
+            low_priority_threshold,
+        })
+    }
+
+    /// Create a scheduler manager from facade policy.
+    pub fn from_policy(config: SchedulerPolicy) -> Self {
+        let core_config = config.to_core_config();
+
+        Self {
+            config,
+            core_config: core_config.clone(),
+            scheduler: Arc::new(CoreIoScheduler::new(core_config)),
+        }
+    }
+
+    /// Get the configuration
+    pub fn config(&self) -> &SchedulerPolicy {
+        &self.config
+    }
+
+    /// Get the derived io-core scheduler config.
+    pub fn core_config(&self) -> &rustfs_io_core::IoSchedulerConfig {
+        &self.core_config
+    }
+
+    /// Get the scheduler
+    pub fn scheduler(&self) -> Arc<CoreIoScheduler> {
+        self.scheduler.clone()
+    }
+
+    /// Create an I/O strategy
+    pub fn create_strategy(&self) -> IoStrategy {
+        IoStrategy::new(self.config, self.scheduler.clone())
+    }
+
+    /// Calculate buffer size
+    pub fn calculate_buffer_size(
+        &self,
+        file_size: i64,
+        media: StorageMedia,
+        pattern: AccessPattern,
+        load: IoLoadLevel,
+        concurrent: usize,
+    ) -> usize {
+        let strategy = self.create_strategy();
+        strategy.calculate_buffer_size(file_size, media, pattern, load, concurrent)
+    }
+
+    /// Get I/O priority
+    pub fn get_priority(&self, size: i64) -> IoPriority {
+        IoPriority::from_size(size, self.config.high_priority_threshold, self.config.low_priority_threshold)
+    }
+}
+
+/// I/O strategy
+pub struct IoStrategy {
+    config: SchedulerPolicy,
+    scheduler: Arc<CoreIoScheduler>,
+}
+
+impl IoStrategy {
+    fn new(config: SchedulerPolicy, scheduler: Arc<CoreIoScheduler>) -> Self {
+        Self { config, scheduler }
+    }
+
+    /// Calculate buffer size with multi-factor strategy
+    pub fn calculate_buffer_size(
+        &self,
+        file_size: i64,
+        media: StorageMedia,
+        pattern: AccessPattern,
+        load: IoLoadLevel,
+        concurrent: usize,
+    ) -> usize {
+        // Create scheduling context
+        let _ctx = IoSchedulingContext::new(file_size, self.config.base_buffer_size)
+            .with_sequential(matches!(pattern, AccessPattern::Sequential))
+            .with_media(media);
+
+        // Get base buffer size from core scheduler
+        let permit_wait = Duration::from_millis(10); // Default wait time
+        let is_sequential = matches!(pattern, AccessPattern::Sequential);
+        let core_strategy = self.scheduler.calculate_strategy(file_size, permit_wait, is_sequential);
+        let base_size = core_strategy.buffer_size;
+
+        // Apply multi-factor adjustments
+        let adjusted_size = self.apply_adjustments(base_size, media, pattern, load, concurrent);
+
+        // Record metrics
+        io_metrics::record_io_scheduler_decision(adjusted_size, load.as_str(), pattern.as_str());
+
+        adjusted_size.min(self.config.max_buffer_size)
+    }
+
+    fn apply_adjustments(
+        &self,
+        base_size: usize,
+        media: StorageMedia,
+        pattern: AccessPattern,
+        load: IoLoadLevel,
+        concurrent: usize,
+    ) -> usize {
+        let mut size = base_size;
+
+        // Media adjustment
+        size = match media {
+            StorageMedia::Nvme => (size as f64 * 1.5) as usize,
+            StorageMedia::Ssd => (size as f64 * 1.2) as usize,
+            StorageMedia::Hdd => size,
+            _ => size,
+        };
+
+        // Pattern adjustment
+        size = match pattern {
+            AccessPattern::Sequential => (size as f64 * 1.5) as usize,
+            AccessPattern::Random => (size as f64 * 0.5) as usize,
+            _ => size,
+        };
+
+        // Load adjustment
+        size = match load {
+            IoLoadLevel::Low => (size as f64 * 1.2) as usize,
+            IoLoadLevel::Medium => size,
+            IoLoadLevel::High => (size as f64 * 0.7) as usize,
+            IoLoadLevel::Critical => (size as f64 * 0.5) as usize,
+        };
+
+        // Concurrency adjustment
+        if concurrent > 10 {
+            size = (size as f64 * 0.8) as usize;
+        }
+
+        size
+    }
+
+    /// Get the configuration
+    pub fn config(&self) -> &SchedulerPolicy {
+        &self.config
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_scheduler_config() {
+        let config = SchedulerPolicy::default();
+        assert!(config.base_buffer_size < config.max_buffer_size);
+    }
+
+    #[test]
+    fn test_scheduler_policy_to_core_config() {
+        let policy = SchedulerPolicy::default();
+        let core = policy.to_core_config();
+        assert_eq!(core.base_buffer_size, policy.base_buffer_size);
+        assert_eq!(core.max_buffer_size, policy.max_buffer_size);
+        assert_eq!(core.high_priority_size_threshold, policy.high_priority_threshold);
+        assert_eq!(core.low_priority_size_threshold, policy.low_priority_threshold);
+    }
+
+    #[test]
+    fn test_scheduler_default_thresholds_remain_stable() {
+        let policy = SchedulerPolicy::default();
+        assert_eq!(policy.base_buffer_size, 64 * 1024);
+        assert_eq!(policy.max_buffer_size, 4 * 1024 * 1024);
+        assert_eq!(policy.high_priority_threshold, 1024 * 1024);
+        assert_eq!(policy.low_priority_threshold, 10 * 1024 * 1024);
+    }
+
+    #[test]
+    fn test_scheduler_priority_boundaries_remain_stable() {
+        let policy = SchedulerPolicy::default();
+        let manager = SchedulerManager::from_policy(policy);
+
+        assert_eq!(manager.get_priority(1_048_575), IoPriority::High);
+        assert_eq!(manager.get_priority(1_048_576), IoPriority::Normal);
+        assert_eq!(manager.get_priority(10_485_760), IoPriority::Normal);
+        assert_eq!(manager.get_priority(10_485_761), IoPriority::Low);
+    }
+
+    #[test]
+    fn test_scheduler_manager() {
+        let manager = SchedulerManager::new(1024, 4096, 512, 2048);
+        let priority = manager.get_priority(100);
+        assert!(priority.is_high());
+    }
+
+    #[test]
+    fn test_io_strategy() {
+        let manager = SchedulerManager::new(1024, 4096, 512, 2048);
+        let strategy = manager.create_strategy();
+
+        let size = strategy.calculate_buffer_size(1024 * 1024, StorageMedia::Ssd, AccessPattern::Sequential, IoLoadLevel::Low, 1);
+
+        assert!(size > 0);
+    }
+}

crates/concurrency/src/timeout.rs

@@ -0,0 +1,222 @@
+// Copyright 2024 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+//! Timeout management for operations
+
+use rustfs_io_core::{TimeoutConfig as CoreTimeoutConfig, TimeoutError, calculate_adaptive_timeout};
+use std::time::{Duration, Instant};
+use tokio_util::sync::CancellationToken;
+
+/// Facade policy for the concurrency-layer timeout manager.
+#[derive(Debug, Clone, Copy)]
+pub struct TimeoutManagerPolicy {
+    /// Default timeout duration
+    pub default_timeout: Duration,
+    /// Maximum timeout duration
+    pub max_timeout: Duration,
+    /// Minimum timeout floor (prevents dynamic calculation from going too low).
+    pub min_timeout: Duration,
+    /// Enable dynamic timeout calculation
+    pub enable_dynamic: bool,
+}
+
+impl Default for TimeoutManagerPolicy {
+    fn default() -> Self {
+        Self {
+            default_timeout: Duration::from_secs(30),
+            max_timeout: Duration::from_secs(300),
+            min_timeout: Duration::from_secs(5),
+            enable_dynamic: true,
+        }
+    }
+}
+
+impl TimeoutManagerPolicy {
+    /// Convert the facade policy into the reusable io-core timeout configuration.
+    ///
+    /// This keeps the concurrency layer explicitly wired to the shared core
+    /// timeout primitives without changing the facade's public behavior.
+    pub fn to_core_config(&self) -> CoreTimeoutConfig {
+        CoreTimeoutConfig {
+            base_timeout: self.default_timeout,
+            timeout_per_mb: Duration::ZERO,
+            max_timeout: self.max_timeout,
+            min_timeout: self.min_timeout,
+            get_object_timeout: self.default_timeout,
+            put_object_timeout: self.max_timeout,
+            list_objects_timeout: self.default_timeout,
+            enable_dynamic_timeout: self.enable_dynamic,
+        }
+    }
+}
+
+/// Timeout manager
+pub struct TimeoutManager {
+    config: TimeoutManagerPolicy,
+    core_config: CoreTimeoutConfig,
+}
+
+impl TimeoutManager {
+    /// Create a new timeout manager
+    pub fn new(default_timeout: Duration, max_timeout: Duration, enable_dynamic: bool) -> Self {
+        let min_timeout = default_timeout.min(max_timeout);
+        Self::from_policy(TimeoutManagerPolicy {
+            default_timeout,
+            max_timeout,
+            min_timeout,
+            enable_dynamic,
+        })
+    }
+
+    /// Create a new timeout manager from the facade policy type.
+    pub fn from_policy(config: TimeoutManagerPolicy) -> Self {
+        let config = TimeoutManagerPolicy {
+            // Guard clamp(min, max) from panic when callers provide an
+            // out-of-order policy (or very small max_timeout).
+            min_timeout: config.min_timeout.min(config.max_timeout),
+            ..config
+        };
+        let core_config = config.to_core_config();
+        Self { config, core_config }
+    }
+
+    /// Get the configuration
+    pub fn config(&self) -> &TimeoutManagerPolicy {
+        &self.config
+    }
+
+    /// Get the derived io-core timeout configuration.
+    pub fn core_config(&self) -> &CoreTimeoutConfig {
+        &self.core_config
+    }
+
+    /// Calculate timeout for a given size
+    pub fn calculate_timeout(&self, size: u64, _history: &[Duration]) -> Duration {
+        if !self.config.enable_dynamic {
+            return self.config.default_timeout;
+        }
+
+        calculate_adaptive_timeout(self.core_config.base_timeout, None, 0, size)
+            .clamp(self.core_config.min_timeout, self.core_config.max_timeout)
+    }
+
+    /// Wrap an operation with timeout control
+    pub async fn wrap_operation<F, T, E>(&self, operation: F, timeout: Option<Duration>) -> Result<T, TimeoutError>
+    where
+        F: std::future::Future<Output = Result<T, E>>,
+        E: Into<TimeoutError>,
+    {
+        let timeout = timeout.unwrap_or(self.config.default_timeout);
+
+        match tokio::time::timeout(timeout, operation).await {
+            Ok(Ok(result)) => Ok(result),
+            Ok(Err(e)) => Err(e.into()),
+            Err(_) => Err(TimeoutError::TimedOut(timeout)),
+        }
+    }
+
+    /// Create a timeout guard for manual timeout control
+    pub fn create_guard(&self, timeout: Option<Duration>) -> TimeoutGuard {
+        TimeoutGuard::new(timeout.unwrap_or(self.core_config.base_timeout))
+    }
+}
+
+/// Timeout guard for manual timeout control
+pub struct TimeoutGuard {
+    timeout: Duration,
+    start: Instant,
+    cancel_token: CancellationToken,
+}
+
+impl TimeoutGuard {
+    fn new(timeout: Duration) -> Self {
+        Self {
+            timeout,
+            start: Instant::now(),
+            cancel_token: CancellationToken::new(),
+        }
+    }
+
+    /// Check if timeout has elapsed
+    pub fn is_timed_out(&self) -> bool {
+        self.start.elapsed() > self.timeout
+    }
+
+    /// Get remaining time
+    pub fn remaining(&self) -> Duration {
+        self.timeout.saturating_sub(self.start.elapsed())
+    }
+
+    /// Get the cancellation token
+    pub fn cancel_token(&self) -> CancellationToken {
+        self.cancel_token.clone()
+    }
+
+    /// Cancel the operation
+    pub fn cancel(&self) {
+        self.cancel_token.cancel();
+    }
+}
+
+#[cfg(test)]
+mod tests {
+    use super::*;
+
+    #[test]
+    fn test_timeout_config() {
+        let config = TimeoutManagerPolicy::default();
+        assert!(config.default_timeout < config.max_timeout);
+    }
+
+    #[test]
+    fn test_timeout_policy_to_core_config() {
+        let policy = TimeoutManagerPolicy::default();
+        let core = policy.to_core_config();
+        assert_eq!(core.base_timeout, policy.default_timeout);
+        assert_eq!(core.max_timeout, policy.max_timeout);
+        assert_eq!(core.min_timeout, policy.min_timeout);
+        assert_eq!(core.get_object_timeout, policy.default_timeout);
+        assert!(core.enable_dynamic_timeout);
+    }
+
+    #[test]
+    fn test_timeout_manager_new_sanitizes_min_timeout_with_small_max_timeout() {
+        let manager = TimeoutManager::new(Duration::from_secs(1), Duration::from_secs(1), true);
+        let timeout = manager.calculate_timeout(1024, &[]);
+        assert_eq!(timeout, Duration::from_secs(1));
+    }
+
+    #[test]
+    fn test_timeout_manager_from_policy_sanitizes_min_timeout() {
+        let manager = TimeoutManager::from_policy(TimeoutManagerPolicy {
+            default_timeout: Duration::from_secs(30),
+            max_timeout: Duration::from_secs(1),
+            min_timeout: Duration::from_secs(5),
+            enable_dynamic: true,
+        });
+
+        assert_eq!(manager.config().min_timeout, Duration::from_secs(1));
+        assert_eq!(manager.core_config().min_timeout, Duration::from_secs(1));
+    }
+
+    #[tokio::test]
+    async fn test_wrap_operation_success() {
+        let manager = TimeoutManager::new(Duration::from_secs(5), Duration::from_secs(10), true);
+
+        let result = manager.wrap_operation(async { Ok::<_, TimeoutError>(42) }, None).await;
+
+        assert!(result.is_ok());
+        assert_eq!(result.unwrap(), 42);
+    }
+}

crates/concurrency/src/workers.rs

@@ -15,19 +15,14 @@
 //! Worker slot limiter used by long-running background workflows.
 
 use std::sync::Arc;
-use tokio::sync::{Semaphore, watch};
+use tokio::sync::{Mutex, Notify};
 use tracing::{debug, trace};
 
 /// Cooperative worker-slot controller for async tasks.
-///
-/// Slot acquisition is backed by a fair FIFO [`Semaphore`], while drain
-/// waiters observe the in-use count through a dedicated [`watch`] channel.
-/// The two wakeup sources are deliberately separate so a drain waiter can
-/// never consume a wakeup destined for a pending [`take`](Self::take).
 pub struct Workers {
-    semaphore: Semaphore,         // Available working slots
-    in_use: watch::Sender<usize>, // Slots currently held; drain waiters watch it
-    limit: usize,                 // Maximum number of concurrent jobs
+    available: Mutex<usize>, // Available working slots
+    notify: Notify,          // Used to notify waiting tasks
+    limit: usize,            // Maximum number of concurrent jobs
 }
 
 impl Workers {
@@ -36,70 +31,86 @@ impl Workers {
         if n == 0 {
             return Err("n must be > 0");
         }
-        if n > Semaphore::MAX_PERMITS {
-            return Err("n exceeds the maximum supported number of slots");
-        }
 
         Ok(Arc::new(Self {
-            semaphore: Semaphore::new(n),
-            in_use: watch::Sender::new(0),
+            available: Mutex::new(n),
+            notify: Notify::new(),
             limit: n,
         }))
     }
 
     /// Acquire a worker slot, waiting until one becomes available.
     pub async fn take(&self) {
-        let permit = match self.semaphore.acquire().await {
-            Ok(permit) => permit,
-            // The semaphore is owned by `self` and never closed.
-            Err(_) => unreachable!("worker semaphore is never closed"),
-        };
-        permit.forget(); // returned manually via give()
-        self.in_use.send_modify(|held| *held += 1);
-        trace!(
-            event = "worker_slot.acquire",
-            component = "concurrency",
-            subsystem = "workers",
-            state = "granted",
-            available_slots = self.semaphore.available_permits(),
-            total_slots = self.limit,
-            permits_in_use = *self.in_use.borrow(),
-            "worker slot updated"
-        );
+        loop {
+            let mut available = self.available.lock().await;
+            if *available == 0 {
+                trace!(
+                    event = "worker_slot.acquire",
+                    component = "concurrency",
+                    subsystem = "workers",
+                    state = "waiting",
+                    available_slots = *available,
+                    total_slots = self.limit,
+                    "worker slot pending"
+                );
+                drop(available);
+                self.notify.notified().await;
+            } else {
+                *available -= 1;
+                trace!(
+                    event = "worker_slot.acquire",
+                    component = "concurrency",
+                    subsystem = "workers",
+                    state = "granted",
+                    available_slots = *available,
+                    total_slots = self.limit,
+                    permits_in_use = self.limit.saturating_sub(*available),
+                    "worker slot updated"
+                );
+                break;
+            }
+        }
     }
 
     /// Release a worker slot.
-    ///
-    /// A `give()` that is not paired with a prior [`take`](Self::take) is
-    /// clamped: it never inflates capacity beyond `limit`.
     pub async fn give(&self) {
-        let mut released = false;
-        self.in_use.send_modify(|held| {
-            if *held > 0 {
-                *held -= 1;
-                released = true;
-            }
-        });
-        if released {
-            self.semaphore.add_permits(1);
-        }
+        let mut available = self.available.lock().await;
+        *available = (*available).saturating_add(1).min(self.limit); // avoid over-release beyond limit
         trace!(
             event = "worker_slot.release",
             component = "concurrency",
             subsystem = "workers",
             state = "released",
-            available_slots = self.semaphore.available_permits(),
+            available_slots = *available,
             total_slots = self.limit,
-            permits_in_use = *self.in_use.borrow(),
+            permits_in_use = self.limit.saturating_sub(*available),
             "worker slot updated"
         );
+        self.notify.notify_one(); // Notify a waiting task
     }
 
     /// Wait until all worker slots are released.
     pub async fn wait(&self) {
-        let mut rx = self.in_use.subscribe();
-        // The sender is owned by `self`, so it outlives this borrow.
-        let _ = rx.wait_for(|&held| held == 0).await;
+        loop {
+            {
+                let available = self.available.lock().await;
+                if *available == self.limit {
+                    break;
+                }
+                trace!(
+                    event = "worker_slot.wait",
+                    component = "concurrency",
+                    subsystem = "workers",
+                    state = "waiting",
+                    available_slots = *available,
+                    total_slots = self.limit,
+                    permits_in_use = self.limit.saturating_sub(*available),
+                    "worker drain pending"
+                );
+            }
+            // Wait until all slots are freed
+            self.notify.notified().await;
+        }
         debug!(
             event = "worker_slot.wait",
             component = "concurrency",
@@ -114,7 +125,7 @@ impl Workers {
 
     /// Return the current number of available worker slots.
     pub async fn available(&self) -> usize {
-        self.limit.saturating_sub(*self.in_use.borrow())
+        *self.available.lock().await
     }
 }
 
@@ -122,7 +133,7 @@ impl Workers {
 mod tests {
     use super::*;
     use std::time::Duration;
-    use tokio::time::{sleep, timeout};
+    use tokio::time::sleep;
 
     #[tokio::test]
     async fn test_workers() {
@@ -157,64 +168,4 @@ mod tests {
 
         assert_eq!(workers.available().await, 2);
     }
-
-    /// Regression for S08: with limit=1, a pending drain waiter must never
-    /// steal the wakeup destined for a pending taker.
-    #[tokio::test]
-    async fn test_drain_waiter_does_not_steal_take_wakeup() {
-        let workers = Workers::new(1).unwrap();
-        workers.take().await;
-
-        let taker = {
-            let workers = workers.clone();
-            tokio::spawn(async move { workers.take().await })
-        };
-        let drainer = {
-            let workers = workers.clone();
-            tokio::spawn(async move { workers.wait().await })
-        };
-        // Let both tasks block before releasing the slot.
-        sleep(Duration::from_millis(50)).await;
-        workers.give().await;
-
-        timeout(Duration::from_secs(1), taker)
-            .await
-            .expect("taker must be woken by give()")
-            .unwrap();
-
-        workers.give().await;
-        timeout(Duration::from_secs(1), drainer)
-            .await
-            .expect("drain waiter must complete once all slots are free")
-            .unwrap();
-        assert_eq!(workers.available().await, 1);
-    }
-
-    /// Regression for S17: two rapid give() calls must wake two pending
-    /// takers; permits must not merge into a single wakeup.
-    #[tokio::test]
-    async fn test_rapid_gives_wake_all_pending_takers() {
-        let workers = Workers::new(2).unwrap();
-        workers.take().await;
-        workers.take().await;
-
-        let takers: Vec<_> = (0..2)
-            .map(|_| {
-                let workers = workers.clone();
-                tokio::spawn(async move { workers.take().await })
-            })
-            .collect();
-        // Let both takers queue up before any slot is released.
-        sleep(Duration::from_millis(50)).await;
-        workers.give().await;
-        workers.give().await;
-
-        for taker in takers {
-            timeout(Duration::from_secs(1), taker)
-                .await
-                .expect("every pending taker must be woken")
-                .unwrap();
-        }
-        assert_eq!(workers.available().await, 0);
-    }
 }

crates/config/README.md

@@ -62,10 +62,6 @@ Current guidance:
 - `RUSTFS_CORS_ALLOWED_ORIGINS` defaults to empty, so the S3 endpoint emits no generic CORS headers unless configured. Set `*` for wildcard origins without credentials, or a comma-separated allow-list for credentialed explicit origins.
 - `RUSTFS_CONSOLE_CORS_ALLOWED_ORIGINS` defaults to `*` for the console service.
 
-## Browser redirect environment variables
-
-- `RUSTFS_BROWSER_REDIRECT_URL` sets the externally reachable browser origin used for OIDC callback, console success redirect, and logout fallback URLs. Configure it to the public scheme and authority without a path, for example `https://console.example.com`. In load-balancer deployments, keep OIDC authorize and callback requests on the same backend node because the in-flight OIDC `state` is local to the RustFS node.
-
 ## Scanner environment aliases
 
 - `RUSTFS_SCANNER_SPEED` (canonical, also accepts `MINIO_SCANNER_SPEED`)

crates/config/src/constants/app.rs

@@ -143,41 +143,6 @@ pub const ENV_MINIO_CI: &str = "MINIO_CI";
 /// Default flag value for bypassing local physical disk independence checks.
 pub const DEFAULT_UNSAFE_BYPASS_DISK_CHECK: bool = false;
 
-/// Environment variable selecting how startup waits for DNS/topology
-/// convergence in multi-node setups. One of `auto` (default), `orchestrated`,
-/// `bounded`, or `fail-fast`.
-///
-/// - `orchestrated`: wait effectively indefinitely for recoverable DNS/topology
-///   errors so the process stays Running (readiness stays false) instead of
-///   exiting and triggering a Kubernetes pod restart loop.
-/// - `bounded`: wait for a finite window (see
-///   `ENV_STARTUP_TOPOLOGY_WAIT_TIMEOUT`) then fail with an actionable error.
-/// - `fail-fast`: fail on the first non-transient resolution error (CI/local).
-/// - `auto`: distributed URL endpoints use orchestrated on Kubernetes and
-///   bounded otherwise; local path endpoints use fail-fast (no DNS to await).
-pub const ENV_STARTUP_TOPOLOGY_WAIT_MODE: &str = "RUSTFS_STARTUP_TOPOLOGY_WAIT_MODE";
-
-/// Environment variable bounding how long startup waits for topology/DNS
-/// convergence before failing in `bounded` mode. Accepts durations such as
-/// `3m`, `90s`, `500ms`, or a bare number of seconds.
-pub const ENV_STARTUP_TOPOLOGY_WAIT_TIMEOUT: &str = "RUSTFS_STARTUP_TOPOLOGY_WAIT_TIMEOUT";
-
-/// Environment variable capping the per-attempt backoff delay while retrying
-/// startup topology/DNS convergence. Accepts durations such as `8s`.
-pub const ENV_STARTUP_TOPOLOGY_RETRY_MAX_DELAY: &str = "RUSTFS_STARTUP_TOPOLOGY_RETRY_MAX_DELAY";
-
-/// Environment variable injected into every Kubernetes pod; its presence marks
-/// an orchestrated deployment for startup topology auto-detection.
-pub const ENV_KUBERNETES_SERVICE_HOST: &str = "KUBERNETES_SERVICE_HOST";
-
-/// Default bounded-mode wait window (seconds) for non-Kubernetes distributed
-/// deployments before startup topology convergence is declared failed.
-pub const DEFAULT_STARTUP_TOPOLOGY_WAIT_TIMEOUT_SECS: u64 = 180;
-
-/// Default per-attempt backoff cap (seconds) while retrying startup
-/// topology/DNS convergence.
-pub const DEFAULT_STARTUP_TOPOLOGY_RETRY_MAX_DELAY_SECS: u64 = 8;
-
 /// Environment variable for server access key.
 pub const ENV_RUSTFS_ACCESS_KEY: &str = "RUSTFS_ACCESS_KEY";
 
@@ -209,12 +174,6 @@ pub const ENV_RUSTFS_CONSOLE_ENABLE: &str = "RUSTFS_CONSOLE_ENABLE";
 /// Environment variable for console server address.
 pub const ENV_RUSTFS_CONSOLE_ADDRESS: &str = "RUSTFS_CONSOLE_ADDRESS";
 
-/// Public browser entrypoint used to build OIDC callback and console redirects.
-///
-/// This should be the externally reachable scheme and authority, without a path.
-/// Example: `RUSTFS_BROWSER_REDIRECT_URL=https://console.example.com`.
-pub const ENV_RUSTFS_BROWSER_REDIRECT_URL: &str = "RUSTFS_BROWSER_REDIRECT_URL";
-
 /// Environment variable for server tls path.
 pub const ENV_RUSTFS_TLS_PATH: &str = "RUSTFS_TLS_PATH";
 
@@ -389,7 +348,6 @@ mod tests {
             RUSTFS_TLS_CERT,
             DEFAULT_ADDRESS,
             DEFAULT_CONSOLE_ADDRESS,
-            ENV_RUSTFS_BROWSER_REDIRECT_URL,
         ];
 
         for constant in &string_constants {

crates/config/src/constants/capacity.rs

@@ -45,6 +45,9 @@ pub const ENV_CAPACITY_METRICS_INTERVAL: &str = "RUSTFS_CAPACITY_METRICS_INTERVA
 /// Environment variable for following symbolic links during capacity calculation
 pub const ENV_CAPACITY_FOLLOW_SYMLINKS: &str = "RUSTFS_CAPACITY_FOLLOW_SYMLINKS";
 
+/// Environment variable for maximum symlink follow depth
+pub const ENV_CAPACITY_MAX_SYMLINK_DEPTH: &str = "RUSTFS_CAPACITY_MAX_SYMLINK_DEPTH";
+
 /// Environment variable for enabling dynamic timeout calculation
 pub const ENV_CAPACITY_ENABLE_DYNAMIC_TIMEOUT: &str = "RUSTFS_CAPACITY_ENABLE_DYNAMIC_TIMEOUT";
 
@@ -54,6 +57,9 @@ pub const ENV_CAPACITY_MIN_TIMEOUT: &str = "RUSTFS_CAPACITY_MIN_TIMEOUT";
 /// Environment variable for maximum capacity calculation timeout
 pub const ENV_CAPACITY_MAX_TIMEOUT: &str = "RUSTFS_CAPACITY_MAX_TIMEOUT";
 
+/// Environment variable for progress stall detection timeout
+pub const ENV_CAPACITY_STALL_TIMEOUT: &str = "RUSTFS_CAPACITY_STALL_TIMEOUT";
+
 // ============================================================================
 // Default Values
 // ============================================================================
@@ -94,6 +100,10 @@ pub const DEFAULT_CAPACITY_METRICS_INTERVAL_SECS: u64 = 600;
 /// Default: false (disabled for safety)
 pub const DEFAULT_CAPACITY_FOLLOW_SYMLINKS: bool = false;
 
+/// Maximum symlink follow depth
+/// Default: 3 levels
+pub const DEFAULT_CAPACITY_MAX_SYMLINK_DEPTH: u8 = 3;
+
 /// Enable dynamic timeout calculation based on directory characteristics
 /// Default: true (enabled)
 pub const DEFAULT_CAPACITY_ENABLE_DYNAMIC_TIMEOUT: bool = true;
@@ -106,6 +116,10 @@ pub const DEFAULT_CAPACITY_MIN_TIMEOUT_SECS: u64 = 2;
 /// Default: 15 seconds
 pub const DEFAULT_CAPACITY_MAX_TIMEOUT_SECS: u64 = 15;
 
+/// Progress stall detection timeout in seconds
+/// Default: 20 seconds
+pub const DEFAULT_CAPACITY_STALL_TIMEOUT_SECS: u64 = 20;
+
 // ============================================================================
 // Tests
 // ============================================================================
@@ -125,8 +139,10 @@ mod tests {
         assert_eq!(ENV_CAPACITY_SAMPLE_RATE, "RUSTFS_CAPACITY_SAMPLE_RATE");
         assert_eq!(ENV_CAPACITY_METRICS_INTERVAL, "RUSTFS_CAPACITY_METRICS_INTERVAL");
         assert_eq!(ENV_CAPACITY_FOLLOW_SYMLINKS, "RUSTFS_CAPACITY_FOLLOW_SYMLINKS");
+        assert_eq!(ENV_CAPACITY_MAX_SYMLINK_DEPTH, "RUSTFS_CAPACITY_MAX_SYMLINK_DEPTH");
         assert_eq!(ENV_CAPACITY_ENABLE_DYNAMIC_TIMEOUT, "RUSTFS_CAPACITY_ENABLE_DYNAMIC_TIMEOUT");
         assert_eq!(ENV_CAPACITY_MIN_TIMEOUT, "RUSTFS_CAPACITY_MIN_TIMEOUT");
         assert_eq!(ENV_CAPACITY_MAX_TIMEOUT, "RUSTFS_CAPACITY_MAX_TIMEOUT");
+        assert_eq!(ENV_CAPACITY_STALL_TIMEOUT, "RUSTFS_CAPACITY_STALL_TIMEOUT");
     }
 }

crates/config/src/constants/env.rs

@@ -41,24 +41,6 @@ pub const ENV_ILM_PROCESS_TIME_DEPRECATED: &str = "_RUSTFS_ILM_PROCESS_TIME";
 /// Default ILM process boundary in seconds (24h).
 pub const DEFAULT_ILM_PROCESS_TIME_SECS: i32 = 86400;
 
-/// **TEST/DEBUG ONLY** — lifecycle "day" length override, in seconds.
-///
-/// Modeled on Ceph RGW's `rgw_lc_debug_interval`. When set to a positive
-/// integer `N`, lifecycle rule evaluation treats one Days-based "day" as `N`
-/// seconds instead of [`DEFAULT_ILM_DAY_SECS`] (86400), so `Days`-based
-/// expiration/transition/noncurrent rules become exercisable in seconds under
-/// test. When unset (or set to `0`/an invalid value), behavior is byte-for-byte
-/// identical to production.
-///
-/// This accelerates data deletion and MUST NEVER be set in a production
-/// deployment. Enabling it emits a `WARN` log. It only rescales relative
-/// `Days` deadlines; absolute `Date`-based rules are unaffected.
-pub const ENV_ILM_DEBUG_DAY_SECS: &str = "RUSTFS_ILM_DEBUG_DAY_SECS";
-
-/// Number of seconds in one lifecycle "day" (86400) used as the default when
-/// [`ENV_ILM_DEBUG_DAY_SECS`] is unset.
-pub const DEFAULT_ILM_DAY_SECS: u32 = 86400;
-
 /// Medium-drawn lines separator
 /// This is used to separate words in environment variable names.
 pub const ENV_WORD_DELIMITER_DASH: &str = "-";

crates/config/src/constants/internode.rs

@@ -148,18 +148,14 @@ pub const DEFAULT_INTERNODE_OFFLINE_REPROBE_SECS: u64 = 5;
 const _: () = assert!(!DEFAULT_INTERNODE_PREWARM);
 const _: () = assert!(!DEFAULT_INTERNODE_OFFLINE_BYPASS);
 
-/// Extra attempts for idempotent, read-only/reentrant control-plane and object-read RPCs
-/// (`DiskInfo`, `ReadAll`, `ReadMetadata`, `ReadVersion`) on transient network failures, with
-/// exponential backoff (grpc-optimization P3-3).
+/// Extra attempts for idempotent, read-only/reentrant control-plane RPCs (e.g. `DiskInfo`) on
+/// transient network failures, with exponential backoff (grpc-optimization P3-3).
 ///
-/// Defaults to `1`: a freshly-committed object lives on only `write_quorum` disks until heal
-/// reconstructs the rest, so a single reset-by-peer on a metadata read during that
-/// read-after-write window can erode the read quorum and surface as a spurious
-/// `InsufficientReadQuorum` (issue #2761). One bounded retry absorbs that transient failure.
-/// **Only** idempotent reads use this; write/lock RPCs (`WriteAll`/`RenameData`/`Delete*`/`Lock`
-/// /`UnLock`) must never auto-retry, to preserve quorum and idempotency semantics (see `CLAUDE.md`).
+/// Defaults to `0` (disabled) — retries change latency-on-failure behavior and are opt-in. **Only**
+/// idempotent reads use this; write/lock RPCs (`WriteAll`/`RenameData`/`Delete*`/`Lock`/`UnLock`)
+/// must never auto-retry, to preserve quorum and idempotency semantics (see `CLAUDE.md`).
 pub const ENV_INTERNODE_IDEMPOTENT_READ_RETRIES: &str = "RUSTFS_INTERNODE_IDEMPOTENT_READ_RETRIES";
-pub const DEFAULT_INTERNODE_IDEMPOTENT_READ_RETRIES: usize = 1;
+pub const DEFAULT_INTERNODE_IDEMPOTENT_READ_RETRIES: usize = 0;
 
 // ── Control/bulk channel isolation (P1) ──
 // Large `bytes`-carrying unary RPCs (ReadAll/WriteAll/ReadMultiple/BatchReadVersion) otherwise
@@ -198,12 +194,6 @@ pub const ENV_INTERNODE_HTTP_POOL_MAX_IDLE_PER_HOST: &str = "RUSTFS_INTERNODE_HT
 pub const ENV_INTERNODE_HTTP_POOL_IDLE_TIMEOUT_SECS: &str = "RUSTFS_INTERNODE_HTTP_POOL_IDLE_TIMEOUT_SECS";
 
 /// Internode HTTP/2 initial stream window size in bytes.
-///
-/// NOTE (backlog#805-C3): all `RUSTFS_INTERNODE_HTTP2_*` tuning (window sizes,
-/// adaptive window, keepalive timeout) only takes effect when the internode
-/// connection actually negotiates HTTP/2, which happens via TLS-ALPN. Over
-/// plaintext internode transport the connection is HTTP/1.1 and these settings
-/// are silently inert — enable internode TLS to use them.
 pub const ENV_INTERNODE_HTTP2_INITIAL_STREAM_WINDOW_SIZE: &str = "RUSTFS_INTERNODE_HTTP2_INITIAL_STREAM_WINDOW_SIZE";
 
 /// Internode HTTP/2 initial connection window size in bytes.
@@ -287,8 +277,7 @@ mod tests {
     fn internode_p3_lifecycle_defaults_are_opt_in() {
         // The opt-in (default-off) invariants are asserted at compile time next to the definitions.
         assert_eq!(DEFAULT_INTERNODE_OFFLINE_REPROBE_SECS, 5);
-        // One bounded retry for idempotent reads to absorb read-after-write transient failures (#2761).
-        assert_eq!(DEFAULT_INTERNODE_IDEMPOTENT_READ_RETRIES, 1);
+        assert_eq!(DEFAULT_INTERNODE_IDEMPOTENT_READ_RETRIES, 0);
         assert_eq!(ENV_INTERNODE_PREWARM, "RUSTFS_INTERNODE_PREWARM");
         assert_eq!(ENV_INTERNODE_OFFLINE_BYPASS, "RUSTFS_INTERNODE_OFFLINE_BYPASS");
         assert_eq!(ENV_INTERNODE_OFFLINE_REPROBE_SECS, "RUSTFS_INTERNODE_OFFLINE_REPROBE_SECS");

crates/config/src/constants/proxy.rs

@@ -53,10 +53,8 @@ pub const DEFAULT_TRUSTED_PROXY_LOG_FAILED_VALIDATIONS: bool = true;
 // ==================== Trusted Proxy Networks ====================
 /// Environment variable for the list of trusted proxy networks (comma-separated IP/CIDR).
 pub const ENV_TRUSTED_PROXY_PROXIES: &str = "RUSTFS_TRUSTED_PROXY_NETWORKS";
-/// Default trusted networks are loopback-only. Private/RFC1918 ranges are not
-/// trusted by default: an operator must explicitly add the specific proxy
-/// addresses in front of this server to have forwarding headers honored.
-pub const DEFAULT_TRUSTED_PROXY_PROXIES: &str = "127.0.0.1,::1";
+/// Default trusted networks include localhost and common private ranges.
+pub const DEFAULT_TRUSTED_PROXY_PROXIES: &str = "127.0.0.1,::1,10.0.0.0/8,172.16.0.0/12,192.168.0.0/16,fd00::/8";
 
 /// Environment variable for additional trusted proxy networks (production specific).
 pub const ENV_TRUSTED_PROXY_EXTRA_PROXIES: &str = "RUSTFS_TRUSTED_PROXY_EXTRA_NETWORKS";

crates/config/src/constants/runtime.rs

@@ -33,16 +33,9 @@ pub const ENV_RUNTIME_DIAL9_OUTPUT_DIR: &str = "RUSTFS_RUNTIME_DIAL9_OUTPUT_DIR"
 pub const ENV_RUNTIME_DIAL9_FILE_PREFIX: &str = "RUSTFS_RUNTIME_DIAL9_FILE_PREFIX";
 pub const ENV_RUNTIME_DIAL9_MAX_FILE_SIZE: &str = "RUSTFS_RUNTIME_DIAL9_MAX_FILE_SIZE";
 pub const ENV_RUNTIME_DIAL9_ROTATION_COUNT: &str = "RUSTFS_RUNTIME_DIAL9_ROTATION_COUNT";
-/// Accepted but not honoured: dial9's S3 uploader depends on a vulnerable
-/// rustls-webpki. Setting this logs a warning. See rustfs/backlog#1157.
 pub const ENV_RUNTIME_DIAL9_S3_BUCKET: &str = "RUSTFS_RUNTIME_DIAL9_S3_BUCKET";
-/// Accepted but not honoured; see [`ENV_RUNTIME_DIAL9_S3_BUCKET`].
 pub const ENV_RUNTIME_DIAL9_S3_PREFIX: &str = "RUSTFS_RUNTIME_DIAL9_S3_PREFIX";
-// Note: there are deliberately no task-dump knobs. dial9 only captures task
-// dumps for futures spawned through `dial9_tokio_telemetry::spawn`, and RustFS
-// spawns with `tokio::spawn` throughout, so the switch could never do anything.
-// Measured: 0 dumps via tokio::spawn vs 14709 via dial9::spawn on an identical
-// workload. See rustfs/backlog#1157 (D9-16).
+pub const ENV_RUNTIME_DIAL9_SAMPLING_RATE: &str = "RUSTFS_RUNTIME_DIAL9_SAMPLING_RATE";
 
 // Default values for Tokio runtime
 pub const DEFAULT_WORKER_THREADS: usize = 16;
@@ -63,6 +56,7 @@ pub const DEFAULT_RUNTIME_DIAL9_OUTPUT_DIR: &str = "/var/log/rustfs/telemetry";
 pub const DEFAULT_RUNTIME_DIAL9_FILE_PREFIX: &str = "rustfs-tokio";
 pub const DEFAULT_RUNTIME_DIAL9_MAX_FILE_SIZE: u64 = 100 * 1024 * 1024; // 100MB
 pub const DEFAULT_RUNTIME_DIAL9_ROTATION_COUNT: usize = 10;
+pub const DEFAULT_RUNTIME_DIAL9_SAMPLING_RATE: f64 = 1.0; // 100% sampling
 // Note: S3 bucket/prefix have no default; absence means upload is disabled (modeled as Option<String>)
 
 /// Maximum transition workers used as a local fallback when runtime env is unset.
@@ -118,25 +112,6 @@ pub const ENV_OBJECT_SEEK_SUPPORT_THRESHOLD: &str = "RUSTFS_OBJECT_SEEK_SUPPORT_
 pub const DEFAULT_OBJECT_SEEK_SUPPORT_THRESHOLD: usize = 10 * 1024 * 1024;
 
 // Object data cache configuration
-
-/// Enables the in-memory object body cache, which serves whole-object GET
-/// responses without an erasure read.
-///
-/// # Timing side channel
-///
-/// A cache hit skips the erasure read, bitrot verification and decode, so it is
-/// reliably faster than a miss. Any principal authorized to read an object can
-/// therefore infer, by timing a single GET, whether *someone* read that object
-/// within the entry's lifetime (see `RUSTFS_OBJECT_DATA_CACHE_TTL_SECS` and
-/// `..._TIME_TO_IDLE_SECS`).
-///
-/// This never crosses an authorization boundary — the probe requires read
-/// access to the exact bucket and object, checked before the cache is consulted
-/// — but it does disclose co-tenants' recent access patterns on objects the
-/// observer may already read. Leave the cache disabled where access-pattern
-/// confidentiality matters, such as a bucket shared read-only between competing
-/// tenants. Timing noise is not a viable mitigation: it would cost exactly the
-/// latency the cache exists to save.
 pub const ENV_OBJECT_DATA_CACHE_ENABLE: &str = "RUSTFS_OBJECT_DATA_CACHE_ENABLE";
 pub const ENV_OBJECT_DATA_CACHE_MODE: &str = "RUSTFS_OBJECT_DATA_CACHE_MODE";
 pub const ENV_OBJECT_DATA_CACHE_MAX_BYTES: &str = "RUSTFS_OBJECT_DATA_CACHE_MAX_BYTES";

crates/config/src/observability/mod.rs

@@ -37,10 +37,6 @@ pub const ENV_OBS_METER_INTERVAL: &str = "RUSTFS_OBS_METER_INTERVAL";
 pub const ENV_OBS_SERVICE_NAME: &str = "RUSTFS_OBS_SERVICE_NAME";
 pub const ENV_OBS_SERVICE_VERSION: &str = "RUSTFS_OBS_SERVICE_VERSION";
 pub const ENV_OBS_ENVIRONMENT: &str = "RUSTFS_OBS_ENVIRONMENT";
-/// Stable OpenTelemetry service instance identity for this RustFS process.
-pub const ENV_OBS_INSTANCE_ID: &str = "RUSTFS_OBS_INSTANCE_ID";
-/// Optional stable RustFS deployment identity used to correlate cluster telemetry.
-pub const ENV_OBS_CLUSTER_ID: &str = "RUSTFS_OBS_CLUSTER_ID";
 
 /// Per-signal enable/disable flags (default: true)
 pub const ENV_OBS_TRACES_EXPORT_ENABLED: &str = "RUSTFS_OBS_TRACES_EXPORT_ENABLED";
@@ -135,8 +131,6 @@ mod tests {
         assert_eq!(ENV_OBS_SERVICE_NAME, "RUSTFS_OBS_SERVICE_NAME");
         assert_eq!(ENV_OBS_SERVICE_VERSION, "RUSTFS_OBS_SERVICE_VERSION");
         assert_eq!(ENV_OBS_ENVIRONMENT, "RUSTFS_OBS_ENVIRONMENT");
-        assert_eq!(ENV_OBS_INSTANCE_ID, "RUSTFS_OBS_INSTANCE_ID");
-        assert_eq!(ENV_OBS_CLUSTER_ID, "RUSTFS_OBS_CLUSTER_ID");
         assert_eq!(ENV_OBS_LOGGER_LEVEL, "RUSTFS_OBS_LOGGER_LEVEL");
         assert_eq!(ENV_OBS_LOG_STDOUT_ENABLED, "RUSTFS_OBS_LOG_STDOUT_ENABLED");
         assert_eq!(ENV_OBS_LOG_DIRECTORY, "RUSTFS_OBS_LOG_DIRECTORY");

crates/data-usage/src/data_usage.rs

@@ -284,24 +284,6 @@ impl SizeHistogram {
     }
 
     pub fn to_map(&self) -> HashMap<String, u64> {
-        // Numeric interval bounds, kept in lockstep with `add` above. The
-        // rollup for the v1-compat `BETWEEN_1024B_AND_1_MB` bucket is derived
-        // from these bounds rather than the display names to avoid undercounting
-        // the sub-ranges in [1 KiB, 512 KiB).
-        const ONE_MIB: u64 = 1024 * 1024;
-        let intervals = [
-            (0, 1024),                          // LESS_THAN_1024_B
-            (1024, 64 * 1024 - 1),              // BETWEEN_1024_B_AND_64_KB
-            (64 * 1024, 256 * 1024 - 1),        // BETWEEN_64_KB_AND_256_KB
-            (256 * 1024, 512 * 1024 - 1),       // BETWEEN_256_KB_AND_512_KB
-            (512 * 1024, ONE_MIB - 1),          // BETWEEN_512_KB_AND_1_MB
-            (1024, ONE_MIB - 1),                // BETWEEN_1024B_AND_1_MB (v1-compat rollup)
-            (ONE_MIB, 10 * ONE_MIB - 1),        // BETWEEN_1_MB_AND_10_MB
-            (10 * ONE_MIB, 64 * ONE_MIB - 1),   // BETWEEN_10_MB_AND_64_MB
-            (64 * ONE_MIB, 128 * ONE_MIB - 1),  // BETWEEN_64_MB_AND_128_MB
-            (128 * ONE_MIB, 512 * ONE_MIB - 1), // BETWEEN_128_MB_AND_512_MB
-            (512 * ONE_MIB, u64::MAX),          // GREATER_THAN_512_MB
-        ];
         let names = [
             "LESS_THAN_1024_B",
             "BETWEEN_1024_B_AND_64_KB",
@@ -316,21 +298,14 @@ impl SizeHistogram {
             "GREATER_THAN_512_MB",
         ];
 
-        // Sum every sub-bucket whose interval lies entirely within [1024, 1 MiB),
-        // excluding the compat bucket itself, to form the v1-compat rollup.
-        let compat_rollup: u64 = self
-            .0
-            .iter()
-            .zip(intervals.iter())
-            .zip(names.iter())
-            .filter(|((_, (start, end)), name)| name != &&"BETWEEN_1024B_AND_1_MB" && *start >= 1024 && *end < ONE_MIB)
-            .map(|((count, _), _)| *count)
-            .sum();
-
         let mut res = HashMap::new();
+        let mut spl_count = 0;
         for (count, name) in self.0.iter().zip(names.iter()) {
             if name == &"BETWEEN_1024B_AND_1_MB" {
-                res.insert(name.to_string(), compat_rollup);
+                res.insert(name.to_string(), spl_count);
+            } else if name.starts_with("BETWEEN_") && name.contains("_KB_") && name.contains("_MB") {
+                spl_count += count;
+                res.insert(name.to_string(), *count);
             } else {
                 res.insert(name.to_string(), *count);
             }
@@ -434,7 +409,7 @@ impl ReplicationAllStats {
         if self.replica_size != 0 && self.replica_count != 0 {
             return false;
         }
-        for v in self.targets.values() {
+        for (_, v) in self.targets.iter() {
             if !v.empty() {
                 return false;
             }
@@ -1343,24 +1318,6 @@ mod tests {
         assert_eq!(summary1.versions, 15);
     }
 
-    #[test]
-    fn test_size_histogram_compat_rollup_sums_all_sub_buckets() {
-        let mut hist = SizeHistogram::default();
-        // One object in each of the four sub-ranges within [1024, 1 MiB).
-        hist.add(32 * 1024); // [1024, 64 KiB)
-        hist.add(128 * 1024); // [64 KiB, 256 KiB)
-        hist.add(384 * 1024); // [256 KiB, 512 KiB)
-        hist.add(768 * 1024); // [512 KiB, 1 MiB)
-
-        let map = hist.to_map();
-
-        assert_eq!(map["BETWEEN_1024B_AND_1_MB"], 4);
-        assert_eq!(map["BETWEEN_1024_B_AND_64_KB"], 1);
-        assert_eq!(map["BETWEEN_64_KB_AND_256_KB"], 1);
-        assert_eq!(map["BETWEEN_256_KB_AND_512_KB"], 1);
-        assert_eq!(map["BETWEEN_512_KB_AND_1_MB"], 1);
-    }
-
     #[test]
     fn test_data_usage_cache_merge_adds_missing_child() {
         let mut base = DataUsageCache::default();

crates/e2e_test/AGENTS.md

@@ -2,11 +2,6 @@
 
 Applies to `crates/e2e_test/`.
 
-Contributor guide (how to run suites, add a test, the harness/fixture
-inventory, `#[ignore]` classes, and the CI map) lives in
-[`README.md`](README.md). This file holds the crate rules that agents must
-follow.
-
 ## Test Reliability
 
 - Keep end-to-end tests deterministic and environment-aware.

crates/e2e_test/README.md

@@ -1,276 +0,0 @@
-# e2e_test
-
-End-to-end test suite for RustFS. Each test spawns a **real `rustfs` binary**
-(built on demand from the workspace) and drives it over the network with the
-AWS SDK (`aws-sdk-s3`), raw HTTP (`reqwest` / `awscurl`), or a protocol client
-(FTPS / WebDAV / SFTP). This is the black-box integration layer: exhaustive
-end-to-end behavior lives here, unit behavior stays in the source crates
-(see [`AGENTS.md`](AGENTS.md)).
-
-The harness lives in [`src/common.rs`](src/common.rs) (single-node +
-cluster environments, S3 client construction, `awscurl` helpers) and
-[`src/chaos.rs`](src/chaos.rs) (in-process disk fault injection). Crate-wide
-test conventions and environment-safety rules are in
-[`AGENTS.md`](AGENTS.md); this file is the contributor guide.
-
-## Module map (~50 modules)
-
-Registered in [`src/lib.rs`](src/lib.rs). Grouped by concern:
-
-| Group | Location | What it covers |
-| --- | --- | --- |
-| **functional** | top-level `*_test.rs` | S3 data plane: `list_objects_*`, `copy_object_*`, `delete_objects_versioning`, `head_object_*`, `checksum_upload`, `compression`, `content_encoding`, `special_chars`, `leading_slash_key`, `create_bucket_region`, `quota`, `data_usage`, `snowball_auto_extract`, `mc_mirror_small_bucket`, `archive_download_integrity`, `version_id_regression`, `delete_marker_migration_semantics` |
-| **object_lock** | [`src/object_lock/`](src/object_lock) | Retention / legal-hold / WORM semantics |
-| **kms** | [`src/kms/`](src/kms) | SSE-S3 / SSE-KMS / SSE-C, local + Vault backends, multipart encryption. Own guide: [`src/kms/README.md`](src/kms/README.md) |
-| **policy** | [`src/policy/`](src/policy), `existing_object_tag_policy_test`, `bucket_policy_check_test`, `anonymous_access_test`, `security_boundary_test`, `multipart_auth_test` | IAM / bucket-policy / STS session policy, policy variables, anonymous access, DoS/SSRF boundaries. Own guide: [`src/policy/README.md`](src/policy/README.md) |
-| **protocols** | [`src/protocols/`](src/protocols) | FTPS, WebDAV, SFTP compliance. Fixed ports, own guide: [`src/protocols/README.md`](src/protocols/README.md) |
-| **reliant** | [`src/reliant/`](src/reliant) | Tests that reuse an **externally started** server (SQL/select, conditional writes, lifecycle, deleted-object reads, node-interact). Run via [`scripts/run_e2e_tests.sh`](../../scripts/run_e2e_tests.sh); see [`src/reliant/README.md`](src/reliant/README.md) |
-| **cluster** | `cluster_concurrency_test`, `stale_multipart_cleanup_cluster_test`, `namespace_lock_quorum_test`, `admin_timeout_regression_test`, `object_lambda_test`, `replication_extension_test` | Multi-node scenarios via `RustFSTestClusterEnvironment` |
-| **chaos / reliability** | [`src/chaos.rs`](src/chaos.rs), `reliability_disk_fault_test`, `heal_erasure_disk_rebuild_test`, `server_startup_failfast_test` | Disk offline/replace/corrupt, EC rebuild, heal, fail-fast startup |
-
-## How to run
-
-All commands assume repo root. `cargo test` triggers an on-demand build of the
-`rustfs` binary from [`src/common.rs`](src/common.rs) (`rustfs_binary_path`) on
-first use — the first invocation is slow, later ones reuse the binary.
-
-```bash
-# Whole crate (default = ignored tests skipped)
-cargo nextest run -p e2e_test
-
-# One module
-cargo nextest run -p e2e_test -E 'test(list_objects_v2_pagination_test)'
-
-# PR smoke subset (see "CI smoke subset" below)
-cargo nextest run --profile e2e-smoke -p e2e_test
-
-# ILM serial lane — ignored lifecycle tests, single-threaded (mirrors CI)
-cargo nextest run -j1 --run-ignored ignored-only -p rustfs-scanner -p rustfs \
-  -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))'
-
-# Protocols suite — fixed ports, MUST be single-threaded, gated by build features
-RUSTFS_BUILD_FEATURES=ftps,webdav,sftp \
-  cargo test -p e2e_test test_protocol_core_suite -- --test-threads=1 --nocapture
-```
-
-The protocols suite has its own contract (fixed bind ports 9022–9301,
-`--test-threads=1`, feature-gated scheduling) documented in
-[`src/protocols/README.md`](src/protocols/README.md). `RUSTFS_BUILD_FEATURES`
-selects which features the spawned binary is built with; leave it unset to run
-every protocol entry.
-
-### `#[ignore]` semantics
-
-Ignored tests are excluded from the default `cargo nextest run` pass because
-they need something the default runner does not provide. **Do not maintain a
-static count here** — it rots (the set shrinks as ci-13 / ilm-3 activate
-suites). Read the live sources instead:
-
-```bash
-rg -n '#\[ignore' crates/e2e_test/src   # every ignore + its reason string
-```
-
-The reason string on each attribute is the classifier. Current classes:
-
-- **Needs a pre-started server** — `"requires running RustFS server at
-  localhost:9000"` / `"Connects to existing rustfs server"`. These are the
-  `reliant/*` and `policy/test_runner` tests; start a server first (e.g.
-  [`scripts/run_e2e_tests.sh`](../../scripts/run_e2e_tests.sh)) or use
-  `--run-ignored`.
-- **Heavy / external tool** — `"Starts a rustfs server; enable when running
-  full E2E"`, `"requires awscurl and spawns a real RustFS server"`. Spawn their
-  own server and/or need `awscurl` on `PATH`.
-- **Serial / global-state (ILM lane)** — lifecycle tests bind fixed ports and
-  share process-global singletons; run via the ILM serial lane above.
-
-## How to add a test
-
-### Single-node (the common case)
-
-Use `RustFSTestEnvironment` from [`src/common.rs`](src/common.rs). It picks a
-**random free port** and a **unique temp dir** per instance, so tests are
-parallel-safe by construction and clean up on `Drop`:
-
-```rust
-use crate::common::{RustFSTestEnvironment, TEST_BUCKET};
-
-#[tokio::test]
-async fn my_case() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;       // waits for readiness
-    let client = env.create_s3_client();          // aws-sdk-s3 Client
-    env.create_test_bucket(TEST_BUCKET).await?;
-    // ... drive `client` ...
-    Ok(())
-}
-```
-
-Register the module in [`src/lib.rs`](src/lib.rs) under `#[cfg(test)]`.
-
-### Cluster
-
-Use `RustFSTestClusterEnvironment::new(node_count)` then `.start()`; it spawns
-`node_count` servers over a shared erasure set and hands out per-node S3 clients
-via `create_s3_client(idx)` / `create_all_clients()`. See
-`cluster_concurrency_test.rs` and `namespace_lock_quorum_test.rs` for patterns.
-
-### Fixture / helper inventory (`src/common.rs`)
-
-| Helper | Purpose |
-| --- | --- |
-| `RustFSTestEnvironment::new` / `with_address` | Single-node env; random or fixed address |
-| `start_rustfs_server` / `_with_env` / `_without_cleanup` | Spawn the server (optional extra args / env vars / no pre-cleanup) |
-| `wait_for_server_ready` | Poll readiness before issuing requests |
-| `create_s3_client` / `create_test_bucket` / `delete_test_bucket` | aws-sdk-s3 client + bucket lifecycle |
-| `find_available_port` | Random free port (isolation primitive) |
-| `rustfs_binary_path` / `_with_features` | Locate/build the binary; honors `RUSTFS_BUILD_FEATURES` |
-| `requested_rustfs_build_features` / `rustfs_build_feature_enabled` | Feature-gate a test to what the binary was built with |
-| `awscurl_available` + `execute_awscurl` / `awscurl_post` / `_get` / `_put` / `_delete` / `awscurl_post_sts_form_urlencoded` | Admin/STS API calls via `awscurl` (skip gracefully when absent) |
-| `replication_fast_env` | Env vars that shrink replication timers (from repl-4); pass to `start_rustfs_server_with_env` |
-| `local_http_client` / `init_logging` | Loopback HTTP client; idempotent tracing init |
-| `RustFSTestClusterEnvironment` (`new`/`start`/`start_node`/`stop_node`/`create_all_clients`) | Multi-node harness |
-| Constants: `DEFAULT_ACCESS_KEY`, `DEFAULT_SECRET_KEY`, `TEST_BUCKET`, `ENV_RUSTFS_BUILD_FEATURES` | Shared credentials / bucket name / env-var name |
-
-Fault injectors live in [`src/chaos.rs`](src/chaos.rs): `DiskFaultHarness`
-(`take_disk_offline`, `bring_disk_online`, `replace_disk_with_empty`,
-`corrupt_object_shard`, `object_metadata_exists_on_disk`, `kill_server` /
-`restart_server`) plus `signed_admin_post`.
-
-### Isolation rules
-
-- **Port**: never hard-code a port for single-node tests — `new()` allocates a
-  random one. Fixed ports (protocols, ILM lane) force `--test-threads=1` / a
-  serial CI lane.
-- **Temp dir**: each env owns a temp dir cleaned on `Drop`; do not write under
-  a shared path.
-- **Orphans**: `RustFSTestEnvironment` kills its child on `Drop`, but a panicked
-  or `kill -9`'d run can leak a `rustfs` process holding a port — see
-  Troubleshooting.
-
-### `#[serial]` vs nextest reality
-
-`serial_test`'s `#[serial]` uses an **in-process** mutex. Under nextest each
-test runs in its **own process**, so `#[serial]` does **not** serialize across
-tests there — see the header of [`.config/nextest.toml`](../../.config/nextest.toml).
-Real cross-test serialization comes from a nextest test-group (`max-threads =
-1`) or a `-j1` CI lane. Single-node e2e tests should instead be parallel-safe by
-construction (random port + isolated temp dir) and need no serialization.
-
-## CI map
-
-`e2e_test` is **excluded** from the main `cargo nextest run --profile ci --all`
-pass ([`.github/workflows/ci.yml`](../../.github/workflows/ci.yml) line 158,
-`--exclude e2e_test`) — the whole crate is too slow to gate every PR. Subsets
-join CI through the nextest profile system only (never as ad-hoc jobs):
-
-| Suite | Runs where | Status |
-| --- | --- | --- |
-| Smoke subset (`e2e-smoke` profile) | `e2e-tests` job, every PR | **Active** (backlog#1149 ci-4) |
-| `s3s-e2e` black-box | `e2e-tests` + `e2e-tests-rio-v2` jobs | **Active** (external conformance tool) |
-| ILM / lifecycle (ignored) | `test-ilm-integration-serial` lane, `-j1` | **Active** (backlog#1148 ilm-1) |
-| KMS suite | — | Not in CI yet (backlog#1149 ci-5) |
-| Protocols (FTPS/WebDAV/SFTP) | — | Not in CI yet (backlog#1149 ci-7) |
-| Replication (fast subset) | `e2e-smoke` profile, `e2e-tests` job, every PR | **Active** (backlog#1147 repl-1) |
-| Replication (slow + dual-node) | `e2e-repl-nightly` profile, scheduled workflow | **Active** (backlog#1147 repl-1) |
-| `reliant/*` (pre-started server) | — | Manual only |
-
-Links: [`ci.yml`](../../.github/workflows/ci.yml) `e2e-tests` (line 347),
-`test-ilm-integration-serial` (line 196). The `e2e-smoke` `default-filter` in
-[`.config/nextest.toml`](../../.config/nextest.toml) is the **single wiring
-mechanism** — extend that filter (or add a sibling profile) to admit more
-tests; do not add e2e jobs to `ci.yml`. repl-1 / ilm-3 are landing in parallel
-and may add lanes; keep the table above easy to extend.
-
-## Troubleshooting
-
-**Reproduce a CI failure locally** — run the exact profile/lane:
-
-```bash
-# Smoke (e2e-tests job) — includes the 20 fast replication tests
-cargo nextest run --profile e2e-smoke -p e2e_test
-# Replication nightly lane (16 slow + dual-node tests; install awscurl for the
-# STS dual-node test, else it skips gracefully)
-cargo nextest run --profile e2e-repl-nightly -p e2e_test
-# ILM serial lane
-cargo nextest run -j1 --run-ignored ignored-only -p rustfs-scanner -p rustfs \
-  -E 'binary(lifecycle_integration_test) or (package(rustfs) and test(lifecycle_transition_api_test))'
-# s3s-e2e black box
-./scripts/e2e-run.sh ./target/debug/rustfs /tmp/rustfs-e2e-data
-```
-
-**Stale binary.** Tests build the `rustfs` binary once and reuse it. To avoid
-rebuilding while iterating on tests, `common.rs` reuses an existing binary when
-running *inside* the e2e test process even if sources changed
-(`can_reuse_inside_e2e`, [`src/common.rs`](src/common.rs) line 98). Downside: if
-you changed **server** code, force a rebuild with
-`cargo build -p rustfs` (or `touch` a source file outside the reuse window)
-before re-running, or CI's freshly built artifact will diverge from your local
-one.
-
-**Port already in use / orphan processes.** A hard-killed run can leak a
-`rustfs` child holding its port. Find and kill it:
-
-```bash
-pkill -f 'target/debug/rustfs' ; pkill -f 'target/release/rustfs'
-```
-
-The `s3s-e2e` CI job selects a random `RUSTFS_TEST_PORT` (see the `e2e-tests`
-job) to dodge this; local single-node tests already use random ports, so a
-lingering orphan is usually the cause of a spurious bind failure.
-
-**`awscurl` not found.** `awscurl`-dependent tests skip gracefully with a
-visible log line (`awscurl_available()`); install `awscurl` to actually run
-them.
-
-## Related
-
-- Crate rules & environment safety: [`AGENTS.md`](AGENTS.md)
-- Sub-suite guides: [`src/kms/README.md`](src/kms/README.md),
-  [`src/policy/README.md`](src/policy/README.md),
-  [`src/protocols/README.md`](src/protocols/README.md),
-  [`src/reliant/README.md`](src/reliant/README.md)
-- Authoritative per-module counts:
-  [`docs/testing/e2e-suite-inventory.md`](../../docs/testing/e2e-suite-inventory.md)
-- Test pyramid & flake policy: [`docs/testing/README.md`](../../docs/testing/README.md)
-
-## CI smoke subset (`--profile e2e-smoke`)
-
-A subset of this crate runs on every PR via the `e2e-tests` job:
-
-```bash
-cargo nextest run --profile e2e-smoke -p e2e_test
-```
-
-The selection lives in `.config/nextest.toml` under `[profile.e2e-smoke]`
-(`default-filter`). That filter is the **single wiring mechanism** for e2e
-tests in CI — extend it (or add a sibling profile) instead of adding new e2e
-jobs to `ci.yml`.
-
-### Admission criteria for the smoke subset
-
-A test module may join the smoke filter only if every test in it is:
-
-1. **Fast** — single-digit seconds per test; the whole subset must keep the
-   `e2e-tests` job ≤ 20 minutes.
-2. **Single-node** — spawns its own server via
-   `RustFSTestEnvironment`/`start_rustfs_server` on a random port with an
-   isolated temp dir. No `RustFSTestClusterEnvironment`, no fixed ports.
-3. **Dependency-free** — no pre-started server at `localhost:9000`, no Vault,
-   no fixed protocol ports. Tools that may be absent on the runner (e.g.
-   `awscurl`) are acceptable only when the test skips gracefully with a
-   visible log line (see `bucket_policy_check_test.rs`).
-4. **Not `#[ignore]`** — ignored tests are activation work (backlog#1149
-   ci-13 / backlog#1148 ilm-3), not smoke candidates.
-
-Note on `#[serial]`: nextest runs each test in its own process, so
-`serial_test`'s in-process mutex does **not** serialize across tests there
-(see the header of `.config/nextest.toml`). Smoke tests must therefore be
-parallel-safe by construction (random port + isolated temp dir), which the
-current subset is.
-
-### Authoritative test inventory
-
-`docs/testing/e2e-suite-inventory.md` records the per-module test counts as
-listed by `cargo nextest list -p e2e_test`. Regenerate it when adding or
-moving e2e tests so acceptance numbers in the test-strategy issues
-(backlog#1147–#1155) stay auditable.

crates/e2e_test/src/admin_auth_test.rs

@@ -1,319 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Negative / lifecycle e2e coverage for the central admin authorization gate
-//! (`rustfs/src/admin/auth.rs`), tracking rustfs/backlog#1151 sec-4.
-//!
-//! These tests exercise the gate end-to-end against a real server, using raw
-//! SigV4-signed HTTP (no `awscurl` dependency) so the exact HTTP status and S3
-//! error code are asserted:
-//!
-//! 1. A fully authenticated but non-admin credential is rejected with
-//!    `403 AccessDenied` on an admin API (`GET /rustfs/admin/v3/info`), while
-//!    the root credential is accepted — proving the rejection is authorization,
-//!    not a broken endpoint.
-//! 2. Root-credential rotation takes effect across a restart: the new
-//!    credential is accepted and the old credential is rejected, on both the
-//!    S3 data plane and the admin plane.
-//! 3. Starting with the default `rustfsadmin` credentials emits the
-//!    default-credentials startup warning (observable operator signal).
-
-#[cfg(test)]
-mod tests {
-    use crate::common::{RustFSTestEnvironment, init_logging, local_http_client, rustfs_binary_path};
-    use aws_sdk_s3::config::{Credentials, Region};
-    use aws_sdk_s3::{Client, Config};
-    use http::header::HOST;
-    use rustfs_signer::constants::UNSIGNED_PAYLOAD;
-    use rustfs_signer::sign_v4;
-    use s3s::Body;
-    use serial_test::serial;
-    use std::error::Error;
-    use std::io::Read;
-    use std::process::{Command, Stdio};
-    use std::sync::{Arc, Mutex};
-    use std::time::{Duration, Instant};
-
-    const ADMIN_INFO_PATH: &str = "/rustfs/admin/v3/info";
-
-    /// Send a SigV4-signed request to `path` (optionally with a JSON `body`) and
-    /// return `(status, body)`. Uses the `UNSIGNED_PAYLOAD` content hash so a
-    /// request body can be attached without the caller pre-hashing it — the
-    /// server verifies the signature against the same sentinel, exactly as the
-    /// AWS SDKs / MinIO client do for streaming/unsigned payloads.
-    async fn signed_request(
-        base_url: &str,
-        method: http::Method,
-        path: &str,
-        body: Option<&str>,
-        access_key: &str,
-        secret_key: &str,
-    ) -> Result<(reqwest::StatusCode, String), Box<dyn Error + Send + Sync>> {
-        let url = format!("{base_url}{path}");
-        let uri = url.parse::<http::Uri>()?;
-        let authority = uri.authority().ok_or("missing authority")?.to_string();
-        let body_bytes = body.map(|b| b.as_bytes().to_vec()).unwrap_or_default();
-
-        // The signature is computed over `UNSIGNED_PAYLOAD`, so the body bytes do
-        // not participate in the SigV4 hash — sign over an empty body and attach
-        // the real payload to the wire request below.
-        let request = http::Request::builder()
-            .method(method.clone())
-            .uri(uri)
-            .header(HOST, authority)
-            .header("x-amz-content-sha256", UNSIGNED_PAYLOAD);
-        let signed = sign_v4(request.body(Body::empty())?, 0, access_key, secret_key, "", "us-east-1");
-
-        let client = local_http_client();
-        let mut rb = client.request(method, url.as_str());
-        for (name, value) in signed.headers() {
-            rb = rb.header(name, value);
-        }
-        if !body_bytes.is_empty() {
-            rb = rb.body(body_bytes);
-        }
-        let resp = rb.send().await?;
-        let status = resp.status();
-        let text = resp.text().await?;
-        Ok((status, text))
-    }
-
-    /// Build an S3 client bound to explicit credentials (used to exercise the S3
-    /// data plane with rotated / stale root credentials).
-    fn s3_client_with(env: &RustFSTestEnvironment, access_key: &str, secret_key: &str) -> Client {
-        let credentials = Credentials::new(access_key, secret_key, None, None, "sec4-admin-auth");
-        let config = Config::builder()
-            .credentials_provider(credentials)
-            .region(Region::new("us-east-1"))
-            .endpoint_url(&env.url)
-            .force_path_style(true)
-            .behavior_version_latest()
-            .build();
-        Client::from_conf(config)
-    }
-
-    /// Create a non-admin IAM user via the admin `add-user` API using the root
-    /// credentials. The user is created with no policy attached, so it is a
-    /// valid (authenticatable) principal that is nonetheless unauthorized for
-    /// admin actions.
-    async fn create_limited_user(
-        env: &RustFSTestEnvironment,
-        access_key: &str,
-        secret_key: &str,
-    ) -> Result<(), Box<dyn Error + Send + Sync>> {
-        let path = format!("/rustfs/admin/v3/add-user?accessKey={access_key}");
-        let body = serde_json::json!({ "secretKey": secret_key, "status": "enabled" }).to_string();
-        let (status, resp) =
-            signed_request(&env.url, http::Method::PUT, &path, Some(&body), &env.access_key, &env.secret_key).await?;
-        assert!(status.is_success(), "add-user should succeed (status={status}, body={resp})");
-        Ok(())
-    }
-
-    /// A fully authenticated but non-admin credential must be rejected with
-    /// `403 AccessDenied` on an admin API, while the root credential succeeds.
-    #[tokio::test(flavor = "multi_thread")]
-    #[serial]
-    async fn non_admin_credential_denied_on_admin_api() -> Result<(), Box<dyn Error + Send + Sync>> {
-        init_logging();
-        let mut env = RustFSTestEnvironment::new().await?;
-        env.start_rustfs_server(vec![]).await?;
-
-        let user_ak = "sec4limited";
-        let user_sk = "sec4limitedsecret";
-        create_limited_user(&env, user_ak, user_sk).await?;
-
-        // Root credential is authorized: proves the endpoint works.
-        let (root_status, root_body) =
-            signed_request(&env.url, http::Method::GET, ADMIN_INFO_PATH, None, &env.access_key, &env.secret_key).await?;
-        assert_eq!(
-            root_status,
-            reqwest::StatusCode::OK,
-            "root credential must be authorized on the admin API, body: {root_body}"
-        );
-
-        // Non-admin credential is rejected by the admin authorization gate.
-        let (status, body) = signed_request(&env.url, http::Method::GET, ADMIN_INFO_PATH, None, user_ak, user_sk).await?;
-        assert_eq!(
-            status,
-            reqwest::StatusCode::FORBIDDEN,
-            "non-admin credential must get 403 on the admin API, body: {body}"
-        );
-        assert!(
-            body.contains("AccessDenied"),
-            "admin gate rejection must carry the AccessDenied S3 error code, body: {body}"
-        );
-
-        env.stop_server();
-        Ok(())
-    }
-
-    /// Rotating the root credentials (restart with new `--access-key` /
-    /// `--secret-key` on the same data directory) takes effect: the new
-    /// credential is accepted and the old one is rejected, on both the S3 data
-    /// plane and the admin plane.
-    #[tokio::test(flavor = "multi_thread")]
-    #[serial]
-    async fn root_credential_rotation_takes_effect() -> Result<(), Box<dyn Error + Send + Sync>> {
-        init_logging();
-        let mut env = RustFSTestEnvironment::new().await?;
-
-        let old_ak = env.access_key.clone();
-        let old_sk = env.secret_key.clone();
-
-        // --- Boot with the original root credential. ---
-        env.start_rustfs_server(vec![]).await?;
-
-        // Admin plane accepts the original root credential.
-        let (status, body) = signed_request(&env.url, http::Method::GET, ADMIN_INFO_PATH, None, &old_ak, &old_sk).await?;
-        assert_eq!(status, reqwest::StatusCode::OK, "original root must reach admin API, body: {body}");
-        // S3 plane accepts the original root credential.
-        s3_client_with(&env, &old_ak, &old_sk)
-            .list_buckets()
-            .send()
-            .await
-            .expect("original root must be able to list buckets on the S3 plane");
-
-        env.stop_server();
-
-        // --- Rotate: restart on the same data dir with a new root credential. ---
-        let new_ak = "rotatedroot";
-        let new_sk = "rotatedrootsecret";
-        env.access_key = new_ak.to_string();
-        env.secret_key = new_sk.to_string();
-        env.start_rustfs_server(vec![]).await?;
-
-        // New root credential works on both planes.
-        let (status, body) = signed_request(&env.url, http::Method::GET, ADMIN_INFO_PATH, None, new_ak, new_sk).await?;
-        assert_eq!(status, reqwest::StatusCode::OK, "rotated root must reach admin API, body: {body}");
-        s3_client_with(&env, new_ak, new_sk)
-            .list_buckets()
-            .send()
-            .await
-            .expect("rotated root must be able to list buckets on the S3 plane");
-
-        // Old root credential is now rejected on both planes.
-        let (status, body) = signed_request(&env.url, http::Method::GET, ADMIN_INFO_PATH, None, &old_ak, &old_sk).await?;
-        assert_eq!(
-            status,
-            reqwest::StatusCode::FORBIDDEN,
-            "stale root must be rejected on the admin API after rotation, body: {body}"
-        );
-        let s3_old = s3_client_with(&env, &old_ak, &old_sk).list_buckets().send().await;
-        assert!(
-            s3_old.is_err(),
-            "stale root must be rejected on the S3 plane after rotation, got: {s3_old:?}"
-        );
-
-        env.stop_server();
-        Ok(())
-    }
-
-    /// Booting with the default `rustfsadmin` credentials must emit the
-    /// default-credentials startup warning so operators get an observable
-    /// signal. The predicate + message text are unit-tested in
-    /// `rustfs::startup_server`; this pins that the warning actually fires at
-    /// runtime. We capture the child's stdout/stderr directly (the shared
-    /// harness inherits stdio) and poll for the warning until it appears.
-    #[tokio::test(flavor = "multi_thread")]
-    #[serial]
-    async fn default_credentials_emit_startup_warning() -> Result<(), Box<dyn Error + Send + Sync>> {
-        init_logging();
-
-        let port = RustFSTestEnvironment::find_available_port().await?;
-        let address = format!("127.0.0.1:{port}");
-        let temp_dir = std::env::temp_dir().join(format!("rustfs_sec4_defcred_{}_{}", std::process::id(), port));
-        std::fs::create_dir_all(&temp_dir)?;
-        let temp_dir_str = temp_dir.display().to_string();
-
-        let mut child = Command::new(rustfs_binary_path())
-            .env("RUST_LOG", "rustfs=info")
-            .env("RUSTFS_CONSOLE_ENABLE", "false")
-            .args([
-                "--address",
-                &address,
-                // Explicitly the compiled-in defaults, so the warning must fire.
-                "--access-key",
-                "rustfsadmin",
-                "--secret-key",
-                "rustfsadmin",
-                &temp_dir_str,
-            ])
-            .stdout(Stdio::piped())
-            .stderr(Stdio::piped())
-            .spawn()?;
-
-        // Drain stdout + stderr concurrently into a shared buffer so a full
-        // pipe never blocks the child.
-        let captured = Arc::new(Mutex::new(String::new()));
-        let mut readers = Vec::new();
-        for stream in [
-            child.stdout.take().map(|s| Box::new(s) as Box<dyn Read + Send>),
-            child.stderr.take().map(|s| Box::new(s) as Box<dyn Read + Send>),
-        ]
-        .into_iter()
-        .flatten()
-        {
-            let sink = Arc::clone(&captured);
-            readers.push(std::thread::spawn(move || {
-                let mut stream = stream;
-                let mut buf = [0u8; 4096];
-                loop {
-                    match stream.read(&mut buf) {
-                        Ok(0) | Err(_) => break,
-                        Ok(n) => {
-                            if let Ok(mut guard) = sink.lock() {
-                                guard.push_str(&String::from_utf8_lossy(&buf[..n]));
-                            }
-                        }
-                    }
-                }
-            }));
-        }
-
-        const WARNING_NEEDLE: &str = "Detected default root credentials";
-        let deadline = Instant::now() + Duration::from_secs(30);
-        let mut seen = false;
-        while Instant::now() < deadline {
-            if let Ok(Some(status)) = child.try_wait() {
-                // Process exited; make a final check of what we captured.
-                seen = captured.lock().map(|g| g.contains(WARNING_NEEDLE)).unwrap_or(false);
-                assert!(
-                    seen,
-                    "server exited ({status}) before emitting the default-credentials warning; captured: {}",
-                    captured.lock().map(|g| g.clone()).unwrap_or_default()
-                );
-                break;
-            }
-            if captured.lock().map(|g| g.contains(WARNING_NEEDLE)).unwrap_or(false) {
-                seen = true;
-                break;
-            }
-            tokio::time::sleep(Duration::from_millis(200)).await;
-        }
-
-        let _ = child.kill();
-        let _ = child.wait();
-        for r in readers {
-            let _ = r.join();
-        }
-        let _ = std::fs::remove_dir_all(&temp_dir);
-
-        assert!(
-            seen,
-            "starting with default credentials must emit the default-credentials warning; captured: {}",
-            captured.lock().map(|g| g.clone()).unwrap_or_default()
-        );
-        Ok(())
-    }
-}

crates/e2e_test/src/archive_download_integrity_test.rs

@@ -62,9 +62,6 @@ mod tests {
         let binary_path = rustfs_binary_path();
         let mut command = Command::new(&binary_path);
         command.env("RUST_LOG", "rustfs=info,rustfs_notify=debug");
-        // Keep the embedded console off its fixed default port :9001, which may
-        // already be taken by unrelated local services (e.g. Docker Desktop).
-        command.env("RUSTFS_CONSOLE_ENABLE", "false");
         for (key, value) in extra_env {
             command.env(key, value);
         }

crates/e2e_test/src/common.rs

@@ -28,7 +28,7 @@ use reqwest::Client as HttpClient;
 use std::ffi::OsStr;
 use std::fs as stdfs;
 use std::path::{Path, PathBuf};
-use std::process::{Child, Command, Stdio};
+use std::process::{Child, Command};
 use std::sync::Once;
 use std::time::Duration;
 use tokio::fs;
@@ -293,10 +293,6 @@ pub struct RustFSTestEnvironment {
     pub access_key: String,
     pub secret_key: String,
     pub process: Option<Child>,
-    /// When set, the spawned server's stdout+stderr are redirected (append) to
-    /// this file instead of being inherited by the test process. Lets a test
-    /// grep the child's logs (e.g. to confirm which GET reader path ran).
-    pub capture_log_path: Option<String>,
 }
 
 impl RustFSTestEnvironment {
@@ -317,7 +313,6 @@ impl RustFSTestEnvironment {
             access_key: DEFAULT_ACCESS_KEY.to_string(),
             secret_key: DEFAULT_SECRET_KEY.to_string(),
             process: None,
-            capture_log_path: None,
         })
     }
 
@@ -335,7 +330,6 @@ impl RustFSTestEnvironment {
             access_key: DEFAULT_ACCESS_KEY.to_string(),
             secret_key: DEFAULT_SECRET_KEY.to_string(),
             process: None,
-            capture_log_path: None,
         })
     }
 
@@ -397,20 +391,9 @@ impl RustFSTestEnvironment {
         let binary_path = rustfs_binary_path();
         let mut command = Command::new(&binary_path);
         command.env("RUST_LOG", "rustfs=info,rustfs_notify=debug");
-        // The embedded console would bind the fixed default port :9001, which
-        // collides with unrelated local services (e.g. Docker Desktop). Tests
-        // that need the console can still override this via extra_env.
-        command.env("RUSTFS_CONSOLE_ENABLE", "false");
         for (key, value) in extra_env {
             command.env(key, value);
         }
-        // Optionally capture the child's stdout+stderr to a file so the test can
-        // grep server logs (e.g. to confirm which GET reader path was taken).
-        if let Some(log_path) = &self.capture_log_path {
-            let file = stdfs::OpenOptions::new().create(true).append(true).open(log_path)?;
-            let stderr_file = file.try_clone()?;
-            command.stdout(Stdio::from(file)).stderr(Stdio::from(stderr_file));
-        }
         let process = command.args(&args).spawn()?;
 
         self.process = Some(process);
@@ -452,20 +435,11 @@ impl RustFSTestEnvironment {
     /// connections before the S3 stack is fully initialized, which causes
     /// early requests to fail intermittently. Treat readiness as "S3 API
     /// responds successfully" instead.
-    ///
-    /// Fails immediately if the spawned server process has already exited
-    /// (e.g. a port bind failure) instead of polling out the full timeout.
-    pub async fn wait_for_server_ready(&mut self) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    pub async fn wait_for_server_ready(&self) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         info!("Waiting for RustFS server to be ready on {}", self.address);
         let client = self.create_s3_client();
 
         for i in 0..60 {
-            if let Some(process) = self.process.as_mut()
-                && let Ok(Some(status)) = process.try_wait()
-            {
-                return Err(format!("RustFS server process exited before becoming ready: {status}").into());
-            }
-
             if TcpStream::connect(&self.address).await.is_ok() && client.list_buckets().send().await.is_ok() {
                 info!("✅ RustFS server is ready after {} attempts", i + 1);
                 return Ok(());
@@ -527,29 +501,6 @@ impl Drop for RustFSTestEnvironment {
     }
 }
 
-/// Short-interval replication timing overrides for fast e2e runs
-/// (backlog#1147 repl-4).
-///
-/// Returns the full set of replication background-loop env vars with
-/// millisecond values so failure-recovery scenarios converge in seconds
-/// instead of the production defaults (health check 5s, MRF flush 10s,
-/// resync retry poll up to 60s). Pass to
-/// [`RustFSTestEnvironment::start_rustfs_server_with_env`] as
-/// `&replication_fast_env()`. The server reads each value once at background
-/// task startup, so the vars must be set before the process starts. Values
-/// below 10ms are clamped server-side to avoid busy-spin.
-pub fn replication_fast_env() -> Vec<(&'static str, &'static str)> {
-    // Env var names mirror `crates/ecstore/src/bucket/replication/replication_timing.rs`
-    // (this is a process-boundary contract: the vars are passed to the spawned
-    // server, not consumed in-process). The names are pinned by the
-    // `env_var_names_are_a_cross_process_contract` test in that module.
-    vec![
-        ("RUSTFS_REPL_HEALTH_CHECK_INTERVAL_MS", "200"),
-        ("RUSTFS_REPL_MRF_FLUSH_INTERVAL_MS", "100"),
-        ("RUSTFS_REPL_RESYNC_POLL_MAX_MS", "250"),
-    ]
-}
-
 async fn execute_awscurl_with_service(
     url: &str,
     method: &str,

crates/e2e_test/src/compression_test.rs

@@ -56,7 +56,6 @@ async fn start_rustfs_with_compression(env: &mut RustFSTestEnvironment) -> Resul
 
     let binary_path = rustfs_binary_path();
     let process = Command::new(&binary_path)
-        .env("RUSTFS_CONSOLE_ENABLE", "false")
         .env("RUSTFS_COMPRESSION_ENABLED", "true")
         .args([
             "--address",

crates/e2e_test/src/degraded_read_eof_regression_test.rs

@@ -1,443 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! E2E regression net for the large-object degraded-read "unexpected EOF"
-//! silent-truncation bug (dist-13, backlog#1150 / master plan backlog#1155).
-//!
-//! The bug: an EC reconstruct-read that failed *mid-stream* (a later block or
-//! part could not be reconstructed) used to close the HTTP body cleanly under a
-//! full `Content-Length`, so the client saw a `200` with a **truncated body**
-//! and treated it as complete (e.g. `mc mirror` persisted the short file).
-//!
-//! It was fixed in three layers on `main`, each with its own *unit* regression:
-//!   * rustfs#4594 — `GetObjectStreamingReader::poll_read` now returns
-//!     `UnexpectedEof` on a short body instead of a clean `Ok(())`
-//!     (`rustfs/src/app/object_usecase.rs`,
-//!     `app::object_usecase::tests::get_object_streaming_reader_errors_on_short_eof`).
-//!   * rustfs#4560 — the lazy multipart codec reader degrades a later part to
-//!     the legacy per-part decode in place, and surfaces reconstruction errors
-//!     instead of silently truncating
-//!     (`crates/ecstore/src/set_disk/read.rs`, `set_disk::read` tests ~:4379).
-//!   * rustfs#4585 — DARE (`rio-v2`) detects stream truncation at a package
-//!     boundary (`crates/rio-v2/src/encrypt_reader.rs`).
-//!
-//! Those unit tests cover the reader layers in isolation. This suite covers the
-//! layer they do not: the **full HTTP GET path** through a real spawned server,
-//! streaming a real body reconstructed from real on-disk EC shards. The single
-//! load-bearing invariant asserted everywhere below is:
-//!
-//!   **A `2xx` GET response whose body collects without error MUST deliver the
-//!   complete object — its length equals the advertised `Content-Length` and
-//!   its bytes hash to the original. A short-but-clean body is the historical
-//!   bug and fails the test loudly. A degraded read that cannot be served must
-//!   fail (non-2xx, or a mid-stream body error), never truncate silently.**
-//!
-//! Topology: single-node 4-disk EC set (default 2 data + 2 parity), driven by
-//! the in-process `DiskFaultHarness` (see `chaos.rs`). Objects are sized well
-//! above the inline threshold and span multiple 1 MiB EC blocks so every read
-//! exercises real shard reconstruction and a genuine mid-stream failure window.
-
-#[cfg(test)]
-mod tests {
-    use crate::chaos::DiskFaultHarness;
-    use crate::common::init_logging;
-    use aws_sdk_s3::Client;
-    use aws_sdk_s3::primitives::ByteStream;
-    use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
-    use serial_test::serial;
-    use sha2::{Digest, Sha256};
-    use std::error::Error;
-    use tokio::time::{Duration, timeout};
-    use tracing::info;
-
-    type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
-
-    const GET_TIMEOUT: Duration = Duration::from_secs(90);
-    const PUT_TIMEOUT: Duration = Duration::from_secs(90);
-
-    /// 1 MiB EC block (`BLOCK_SIZE_V2`). A "≥2 EC stripes" object is anything
-    /// spanning more than one block; we use 6 MiB single objects and 5 MiB
-    /// multipart parts so every read crosses several block boundaries.
-    const MIB: usize = 1024 * 1024;
-
-    fn sha256_hex(data: &[u8]) -> String {
-        Sha256::digest(data).iter().map(|b| format!("{b:02x}")).collect()
-    }
-
-    /// Deterministic pseudo-random payload so corruption offsets and hashes are
-    /// reproducible across the 3x local flake runs.
-    fn payload(len: usize, seed: u8) -> Vec<u8> {
-        (0..len)
-            .map(|i| (i as u64).wrapping_mul(2654435761).wrapping_add(seed as u64) as u8)
-            .collect()
-    }
-
-    /// Render an error and its full source chain (SDK errors nest the useful
-    /// detail below the terse top-level `Display`).
-    fn render_err(err: &(dyn Error + 'static)) -> String {
-        let mut out = err.to_string();
-        let mut source = err.source();
-        while let Some(cause) = source {
-            out.push_str(" -> ");
-            out.push_str(&cause.to_string());
-            source = cause.source();
-        }
-        out
-    }
-
-    /// Outcome of a truncation-checked GET. The forbidden fourth possibility —
-    /// a `2xx` with a clean but short body — never reaches a variant: it panics
-    /// inside [`get_checked`] so it can never be silently tolerated.
-    #[derive(Debug)]
-    enum GetOutcome {
-        /// `2xx` and the fully collected body matches the expected length + hash.
-        CompleteMatch,
-        /// The transfer failed cleanly: a non-2xx status / send error, or a
-        /// mid-stream body error such as the `UnexpectedEof` from rustfs#4594.
-        /// Acceptable only for reads degraded beyond the read quorum.
-        CleanFailure(String),
-    }
-
-    /// Perform a GET and enforce the no-silent-truncation invariant.
-    ///
-    /// Returns [`GetOutcome::CompleteMatch`] on a full, correct body, or
-    /// [`GetOutcome::CleanFailure`] on any clean failure. It **panics** if the
-    /// server returns a `2xx` whose body collects cleanly but is shorter than
-    /// the advertised `Content-Length` / expected length — that is exactly the
-    /// dist-13 bug.
-    async fn get_checked(
-        client: &Client,
-        bucket: &str,
-        key: &str,
-        expected_len: usize,
-        expected_sha256: &str,
-        phase: &str,
-    ) -> Result<GetOutcome, Box<dyn Error + Send + Sync>> {
-        let send = timeout(GET_TIMEOUT, client.get_object().bucket(bucket).key(key).send())
-            .await
-            .map_err(|_| format!("GET {key} send timed out during {phase}"))?;
-
-        let response = match send {
-            Ok(response) => response,
-            Err(err) => {
-                return Ok(GetOutcome::CleanFailure(format!(
-                    "GET {key} failed before body during {phase}: {}",
-                    render_err(&err)
-                )));
-            }
-        };
-
-        // The advertised body length: this is the `expected` value that the
-        // fixed streaming reader (rustfs#4594) checks its emitted bytes against.
-        let advertised = response.content_length();
-
-        let collected = timeout(GET_TIMEOUT, response.body.collect())
-            .await
-            .map_err(|_| format!("GET {key} body collect timed out during {phase}"))?;
-
-        let bytes = match collected {
-            Ok(aggregated) => aggregated.into_bytes(),
-            Err(err) => {
-                // Fixed behavior for a beyond-quorum read: the stream aborts
-                // mid-body instead of ending cleanly short.
-                return Ok(GetOutcome::CleanFailure(format!(
-                    "GET {key} body errored mid-stream during {phase}: {}",
-                    render_err(&err)
-                )));
-            }
-        };
-
-        // Heart of the regression net: a 2xx body that collected without error
-        // MUST be the whole object. A short clean body under a full
-        // Content-Length is the historical silent truncation — fail loudly.
-        if let Some(advertised_len) = advertised {
-            assert_eq!(
-                bytes.len() as i64,
-                advertised_len,
-                "SILENT TRUNCATION during {phase}: GET {key} returned 2xx advertising \
-                 Content-Length {advertised_len} but the body collected cleanly at {} bytes \
-                 (dist-13 regression: rustfs#4594/#4560/#4585)",
-                bytes.len()
-            );
-        }
-        assert_eq!(
-            bytes.len(),
-            expected_len,
-            "GET {key} during {phase} returned a clean 2xx body of {} bytes, expected {expected_len} \
-             (silent truncation regression)",
-            bytes.len()
-        );
-        assert_eq!(
-            sha256_hex(&bytes),
-            expected_sha256,
-            "GET {key} during {phase} returned a full-length body whose hash does not match the original"
-        );
-
-        Ok(GetOutcome::CompleteMatch)
-    }
-
-    /// PUT a single-part object and return `(expected_len, sha256)`.
-    async fn put_object(
-        client: &Client,
-        bucket: &str,
-        key: &str,
-        body: Vec<u8>,
-    ) -> Result<(usize, String), Box<dyn Error + Send + Sync>> {
-        let len = body.len();
-        let digest = sha256_hex(&body);
-        timeout(
-            PUT_TIMEOUT,
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(key)
-                .body(ByteStream::from(body))
-                .send(),
-        )
-        .await
-        .map_err(|_| format!("PUT {key} timed out"))??;
-        Ok((len, digest))
-    }
-
-    /// Complete a multipart upload from the given parts and return
-    /// `(total_len, sha256_of_concatenation)`.
-    async fn put_multipart(
-        client: &Client,
-        bucket: &str,
-        key: &str,
-        parts: Vec<Vec<u8>>,
-    ) -> Result<(usize, String), Box<dyn Error + Send + Sync>> {
-        let full: Vec<u8> = parts.iter().flatten().copied().collect();
-        let total_len = full.len();
-        let digest = sha256_hex(&full);
-
-        let create = client.create_multipart_upload().bucket(bucket).key(key).send().await?;
-        let upload_id = create.upload_id().ok_or("missing upload id")?.to_string();
-
-        let mut completed = Vec::with_capacity(parts.len());
-        for (index, part_body) in parts.into_iter().enumerate() {
-            let part_number = (index + 1) as i32;
-            let uploaded = timeout(
-                PUT_TIMEOUT,
-                client
-                    .upload_part()
-                    .bucket(bucket)
-                    .key(key)
-                    .upload_id(&upload_id)
-                    .part_number(part_number)
-                    .body(ByteStream::from(part_body))
-                    .send(),
-            )
-            .await
-            .map_err(|_| format!("upload_part {part_number} for {key} timed out"))??;
-            completed.push(
-                CompletedPart::builder()
-                    .part_number(part_number)
-                    .e_tag(uploaded.e_tag().ok_or("missing part etag")?)
-                    .build(),
-            );
-        }
-
-        timeout(
-            PUT_TIMEOUT,
-            client
-                .complete_multipart_upload()
-                .bucket(bucket)
-                .key(key)
-                .upload_id(&upload_id)
-                .multipart_upload(CompletedMultipartUpload::builder().set_parts(Some(completed)).build())
-                .send(),
-        )
-        .await
-        .map_err(|_| format!("complete_multipart_upload {key} timed out"))??;
-
-        Ok((total_len, digest))
-    }
-
-    /// Scenario (a) — one disk offline: a large single-part object (≥2 EC
-    /// stripes) and a multipart object (3 parts × 5 MiB) must GET back as a
-    /// full, byte-identical body with the correct Content-Length. No early EOF.
-    #[tokio::test]
-    #[serial]
-    async fn degraded_read_large_objects_with_one_disk_offline_return_full_body() -> TestResult {
-        init_logging();
-        info!("dist-13 (a): large-object degraded read with one of four disks offline");
-
-        let mut harness = DiskFaultHarness::new(4).await?;
-        harness.start_server().await?;
-        let client = harness.env.create_s3_client();
-
-        let bucket = "dist13-degraded-offline";
-        client.create_bucket().bucket(bucket).send().await?;
-
-        // 6 MiB single object: 6 EC blocks, far above the inline threshold.
-        let single_key = "eof/large-single.bin";
-        let (single_len, single_sha) = put_object(&client, bucket, single_key, payload(6 * MIB, 41)).await?;
-
-        // Multipart: 3 parts × 5 MiB (the S3 minimum part size).
-        let multi_key = "eof/large-multipart.bin";
-        let (multi_len, multi_sha) = put_multipart(
-            &client,
-            bucket,
-            multi_key,
-            vec![payload(5 * MIB, 42), payload(5 * MIB, 43), payload(5 * MIB, 44)],
-        )
-        .await?;
-
-        // Baseline with every disk online.
-        assert!(matches!(
-            get_checked(&client, bucket, single_key, single_len, &single_sha, "baseline single").await?,
-            GetOutcome::CompleteMatch
-        ));
-        assert!(matches!(
-            get_checked(&client, bucket, multi_key, multi_len, &multi_sha, "baseline multipart").await?,
-            GetOutcome::CompleteMatch
-        ));
-
-        // One disk offline: EC 2+2 still has 3 shards, a full read quorum.
-        harness.take_disk_offline(0)?;
-
-        assert!(
-            matches!(
-                get_checked(&client, bucket, single_key, single_len, &single_sha, "degraded single").await?,
-                GetOutcome::CompleteMatch
-            ),
-            "6 MiB single object must reconstruct fully with disk0 offline"
-        );
-        assert!(
-            matches!(
-                get_checked(&client, bucket, multi_key, multi_len, &multi_sha, "degraded multipart").await?,
-                GetOutcome::CompleteMatch
-            ),
-            "15 MiB multipart object must reconstruct fully with disk0 offline"
-        );
-
-        Ok(())
-    }
-
-    /// Scenario (b) — bitrot within the read quorum: two shards of a large
-    /// multipart object are silently corrupted mid-file (2 of 4 in a 2+2 set,
-    /// leaving exactly a quorum). The GET must reconstruct the object and return
-    /// the complete, byte-identical body. Because the corruption sits in the
-    /// middle of the part file, block 0 reads clean, the `200` + full
-    /// Content-Length are already committed, and the bad block is only hit
-    /// mid-stream — the exact window the fixes had to reconstruct through rather
-    /// than truncate.
-    #[tokio::test]
-    #[serial]
-    async fn degraded_read_reconstructs_through_midstream_bitrot_within_quorum() -> TestResult {
-        init_logging();
-        info!("dist-13 (b): mid-stream bitrot within quorum must reconstruct a full body");
-
-        let mut harness = DiskFaultHarness::new(4).await?;
-        harness.start_server().await?;
-        let client = harness.env.create_s3_client();
-
-        let bucket = "dist13-bitrot-quorum";
-        client.create_bucket().bucket(bucket).send().await?;
-
-        let key = "eof/bitrot-multipart.bin";
-        let (len, sha) = put_multipart(
-            &client,
-            bucket,
-            key,
-            vec![payload(5 * MIB, 51), payload(5 * MIB, 52), payload(5 * MIB, 53)],
-        )
-        .await?;
-
-        assert!(matches!(
-            get_checked(&client, bucket, key, len, &sha, "baseline before corruption").await?,
-            GetOutcome::CompleteMatch
-        ));
-
-        // Corrupt two shards (disks 0 and 1). A 2+2 set reconstructs from any
-        // two of four shards, so two clean shards remain and the read must
-        // still deliver the whole object.
-        harness.corrupt_object_shard(0, bucket, key)?;
-        harness.corrupt_object_shard(1, bucket, key)?;
-
-        assert!(
-            matches!(
-                get_checked(&client, bucket, key, len, &sha, "read through 2-shard bitrot").await?,
-                GetOutcome::CompleteMatch
-            ),
-            "2-shard bitrot in a 2+2 set is within quorum and must reconstruct the full body"
-        );
-        // A repeat read confirms reconstruction is stable, not a one-shot.
-        assert!(matches!(
-            get_checked(&client, bucket, key, len, &sha, "second read through bitrot").await?,
-            GetOutcome::CompleteMatch
-        ));
-
-        Ok(())
-    }
-
-    /// Scenario (c) — degraded beyond the read quorum (THE HEART OF THE NET):
-    /// three shards of a large multipart object are corrupted mid-file (3 of 4
-    /// in a 2+2 set → only one clean shard, below the 2-shard quorum). Block 0
-    /// still reads clean, so the server commits `200` + the full Content-Length
-    /// and starts streaming; the corrupted middle block then cannot be
-    /// reconstructed. The read MUST fail — a non-2xx status or a mid-stream body
-    /// error — and MUST NOT close cleanly with a truncated body under the full
-    /// Content-Length. `get_checked` panics on that forbidden outcome, so this
-    /// test fails loudly if the truncation bug ever returns.
-    #[tokio::test]
-    #[serial]
-    async fn beyond_quorum_degraded_read_never_silently_truncates() -> TestResult {
-        init_logging();
-        info!("dist-13 (c): beyond-quorum degraded read must fail, never 200+truncated");
-
-        let mut harness = DiskFaultHarness::new(4).await?;
-        harness.start_server().await?;
-        let client = harness.env.create_s3_client();
-
-        let bucket = "dist13-beyond-quorum";
-        client.create_bucket().bucket(bucket).send().await?;
-
-        let key = "eof/beyond-quorum-multipart.bin";
-        let (len, sha) = put_multipart(
-            &client,
-            bucket,
-            key,
-            vec![payload(5 * MIB, 61), payload(5 * MIB, 62), payload(5 * MIB, 63)],
-        )
-        .await?;
-
-        assert!(matches!(
-            get_checked(&client, bucket, key, len, &sha, "baseline before over-corruption").await?,
-            GetOutcome::CompleteMatch
-        ));
-
-        // Corrupt three of four shards: below the 2-shard read quorum, so the
-        // corrupted block cannot be reconstructed.
-        harness.corrupt_object_shard(0, bucket, key)?;
-        harness.corrupt_object_shard(1, bucket, key)?;
-        harness.corrupt_object_shard(2, bucket, key)?;
-
-        // The invariant is enforced inside get_checked (it panics on a clean
-        // short 2xx body). A CleanFailure here is the correct, fixed behavior;
-        // an (implausible) CompleteMatch would also be acceptable. The only
-        // failing outcome is the silent truncation the net exists to catch.
-        match get_checked(&client, bucket, key, len, &sha, "beyond-quorum read").await? {
-            GetOutcome::CleanFailure(reason) => {
-                info!("beyond-quorum read failed cleanly as required: {reason}");
-            }
-            GetOutcome::CompleteMatch => {
-                info!("beyond-quorum read unexpectedly reconstructed a full body; not a truncation, accepted");
-            }
-        }
-
-        Ok(())
-    }
-}

crates/e2e_test/src/get_codec_streaming_compat_test.rs

@@ -1,535 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! GET codec-streaming fast-path body/header compatibility net (backlog#1183).
-//!
-//! backlog#1183 tracks flipping the default GET data path from the legacy
-//! `tokio::io::duplex` double-copy (`GET_OBJECT_PATH_LEGACY_DUPLEX`) to the
-//! zero-duplex codec-streaming fast path (`GET_OBJECT_PATH_CODEC_STREAMING`,
-//! `crates/ecstore/src/set_disk/ops/object.rs`). That flip is gated behind two
-//! deliberate safety confirmations — `RUSTFS_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED`
-//! and `..._HEADER_COMPAT_CONFIRMED` (`crates/ecstore/src/set_disk/mod.rs`) —
-//! because it rewrites the GET hot path's data flow and any divergence is a
-//! data-availability incident.
-//!
-//! This suite provides the empirical evidence those two gates ask for. It runs
-//! the SAME object matrix twice against the SAME on-disk EC shards, changing
-//! only the codec-streaming env gates between runs, and asserts that the codec
-//! path is **byte-for-byte and header-for-header identical** to the legacy
-//! duplex path:
-//!
-//!   * Phase A (baseline): default env → GETs take `GET_OBJECT_PATH_LEGACY_DUPLEX`.
-//!   * Phase B (codec):    gates opened → GETs take `GET_OBJECT_PATH_CODEC_STREAMING`.
-//!
-//! Path confirmation is not assumed: the legacy path emits a
-//! `"Created duplex pipe for object data transfer"` debug line per full GET, so
-//! the test captures each phase's server log and asserts the baseline phase
-//! created duplex pipes for the large objects while the codec phase created
-//! **zero** — proving the codec path actually ran rather than silently falling
-//! back to the very path it is being compared against.
-//!
-//! Beyond the all-healthy happy path, the suite also drives the codec/legacy
-//! A/B under two conditions the `DiskFaultHarness` makes reachable:
-//!
-//!   * Parity reconstruction: one data disk is taken offline
-//!     (`take_disk_offline`) and the SAME object matrix is GET both ways while
-//!     the EC 2+2 set rebuilds each large object from the surviving shards. The
-//!     codec-streaming reader gate never inspects drive health, so the codec
-//!     fast path is exercised end-to-end through reconstruction; the test
-//!     asserts byte- and header-equality vs the legacy path AND that the codec
-//!     phase never fell back to a duplex pipe while reconstructing.
-//!   * Missing object: a GET for an absent key is compared across both phases
-//!     to prove the error semantics (HTTP status + S3 error code) are identical
-//!     — the codec env must not perturb the NoSuchKey negative path.
-//!
-//! Topology: single-node 4-disk EC set (default 2 data + 2 parity) via the
-//! in-process `DiskFaultHarness` (see `chaos.rs`). Small objects are inlined
-//! into `xl.meta` (served identically on both paths); large objects span one or
-//! more 1 MiB EC blocks so real shard reconstruction runs.
-
-#[cfg(test)]
-mod tests {
-    use crate::chaos::DiskFaultHarness;
-    use crate::common::init_logging;
-    use aws_sdk_s3::Client;
-    use aws_sdk_s3::error::ProvideErrorMetadata;
-    use aws_sdk_s3::primitives::ByteStream;
-    use aws_sdk_s3::types::{CompletedMultipartUpload, CompletedPart};
-    use serial_test::serial;
-    use sha2::{Digest, Sha256};
-    use std::collections::BTreeMap;
-    use std::error::Error;
-    use tokio::time::{Duration, sleep};
-    use tracing::info;
-
-    type TestResult = Result<(), Box<dyn Error + Send + Sync>>;
-
-    const MIB: usize = 1024 * 1024;
-    const BUCKET: &str = "codec-streaming-compat";
-    const CONTENT_TYPE: &str = "application/x-rustfs-compat";
-    /// A key that is never uploaded — used to compare the NoSuchKey negative
-    /// path across the legacy and codec phases.
-    const MISSING_KEY: &str = "does-not-exist/ghost.bin";
-
-    /// Marker the legacy duplex GET path logs once per full-object read
-    /// (`crates/ecstore/src/set_disk/ops/object.rs`). Its presence/absence in a
-    /// phase's captured server log tells us which reader path actually ran.
-    const DUPLEX_MARKER: &str = "Created duplex pipe for object data transfer";
-
-    fn sha256_hex(data: &[u8]) -> String {
-        Sha256::digest(data).iter().map(|b| format!("{b:02x}")).collect()
-    }
-
-    /// Deterministic pseudo-random payload so hashes are reproducible.
-    fn payload(len: usize, seed: u8) -> Vec<u8> {
-        (0..len)
-            .map(|i| (i as u64).wrapping_mul(2654435761).wrapping_add(seed as u64) as u8)
-            .collect()
-    }
-
-    /// A comparable projection of the GET response headers we require the codec
-    /// path to reproduce exactly. Stored in a `BTreeMap` for a stable, readable
-    /// diff on mismatch.
-    #[derive(Debug, Clone, PartialEq, Eq)]
-    struct GetView {
-        sha256: String,
-        len: usize,
-        headers: BTreeMap<String, String>,
-    }
-
-    fn header_projection(resp: &aws_sdk_s3::operation::get_object::GetObjectOutput) -> BTreeMap<String, String> {
-        let mut m = BTreeMap::new();
-        m.insert("content-length".into(), resp.content_length().unwrap_or(-1).to_string());
-        m.insert("etag".into(), resp.e_tag().unwrap_or("<none>").to_string());
-        m.insert("content-type".into(), resp.content_type().unwrap_or("<none>").to_string());
-        m.insert("accept-ranges".into(), resp.accept_ranges().unwrap_or("<none>").to_string());
-        m.insert("content-encoding".into(), resp.content_encoding().unwrap_or("<none>").to_string());
-        m.insert("content-disposition".into(), resp.content_disposition().unwrap_or("<none>").to_string());
-        m.insert("cache-control".into(), resp.cache_control().unwrap_or("<none>").to_string());
-        m.insert("content-range".into(), resp.content_range().unwrap_or("<none>").to_string());
-        m.insert("version-id".into(), resp.version_id().unwrap_or("<none>").to_string());
-        m.insert(
-            "last-modified".into(),
-            resp.last_modified()
-                .map(|t| t.secs().to_string())
-                .unwrap_or_else(|| "<none>".into()),
-        );
-        // User metadata (x-amz-meta-*), order-independent.
-        if let Some(meta) = resp.metadata() {
-            let mut sorted: BTreeMap<&String, &String> = BTreeMap::new();
-            for (k, v) in meta {
-                sorted.insert(k, v);
-            }
-            for (k, v) in sorted {
-                m.insert(format!("meta:{k}"), v.clone());
-            }
-        }
-        m
-    }
-
-    /// Full-object GET, returning body hash/len + the header projection.
-    async fn get_full(client: &Client, key: &str) -> Result<GetView, Box<dyn Error + Send + Sync>> {
-        let resp = client.get_object().bucket(BUCKET).key(key).send().await?;
-        let headers = header_projection(&resp);
-        let body = resp.body.collect().await?.into_bytes();
-        Ok(GetView {
-            sha256: sha256_hex(&body),
-            len: body.len(),
-            headers,
-        })
-    }
-
-    /// Ranged GET, returning body hash/len + header projection.
-    async fn get_range(client: &Client, key: &str, range: &str) -> Result<GetView, Box<dyn Error + Send + Sync>> {
-        let resp = client.get_object().bucket(BUCKET).key(key).range(range).send().await?;
-        let headers = header_projection(&resp);
-        let body = resp.body.collect().await?.into_bytes();
-        Ok(GetView {
-            sha256: sha256_hex(&body),
-            len: body.len(),
-            headers,
-        })
-    }
-
-    /// GET a key expected to be absent, projecting the wire-visible error
-    /// semantics — HTTP status code plus the S3 error code — so the legacy and
-    /// codec phases can be asserted to reject a missing object identically. A
-    /// missing object fails during metadata resolution, before the reader-path
-    /// gate is consulted, so both phases MUST agree; a divergence here would
-    /// mean the codec env perturbed the negative path.
-    async fn missing_key_semantics(client: &Client, key: &str) -> Result<(u16, String), Box<dyn Error + Send + Sync>> {
-        match client.get_object().bucket(BUCKET).key(key).send().await {
-            Ok(_) => Err(format!("GET {key} unexpectedly succeeded; expected a NoSuchKey error").into()),
-            Err(err) => {
-                let status = err.raw_response().map(|r| r.status().as_u16()).unwrap_or(0);
-                let code = err.as_service_error().and_then(|e| e.code()).unwrap_or("<none>").to_string();
-                Ok((status, code))
-            }
-        }
-    }
-
-    /// Env that opens every codec-streaming gate to 100% for the codec phase.
-    /// Mirrors the exact knobs `get_codec_streaming_reader_gate` inspects
-    /// (`crates/ecstore/src/set_disk/mod.rs`).
-    fn codec_env() -> Vec<(&'static str, &'static str)> {
-        vec![
-            ("RUSTFS_GET_CODEC_STREAMING_ENABLE", "true"),
-            ("RUSTFS_GET_CODEC_STREAMING_ROLLOUT", "internal"),
-            ("RUSTFS_GET_CODEC_STREAMING_ROLLOUT_PCT", "100"),
-            ("RUSTFS_GET_CODEC_STREAMING_BODY_COMPAT_CONFIRMED", "true"),
-            ("RUSTFS_GET_CODEC_STREAMING_HEADER_COMPAT_CONFIRMED", "true"),
-            // Lower the min-size floor so every non-inline object below is eligible.
-            ("RUSTFS_GET_CODEC_STREAMING_MIN_SIZE", "4096"),
-            // Route multipart objects through per-part codec streaming too.
-            ("RUSTFS_GET_CODEC_STREAMING_MULTIPART_ENABLE", "true"),
-            // Lock optimization is on by default, but pin it so the gate's
-            // `LockOptimizationDisabled` fallback can never mask the codec path.
-            ("RUSTFS_OBJECT_LOCK_OPTIMIZATION_ENABLE", "true"),
-        ]
-    }
-
-    async fn put_plain(client: &Client, key: &str, data: &[u8]) -> TestResult {
-        client
-            .put_object()
-            .bucket(BUCKET)
-            .key(key)
-            .content_type(CONTENT_TYPE)
-            .metadata("compat", "yes")
-            .metadata("shape", "single-part")
-            .body(ByteStream::from(data.to_vec()))
-            .send()
-            .await?;
-        Ok(())
-    }
-
-    /// Upload a 2-part multipart object; returns the concatenated payload.
-    async fn put_multipart(
-        client: &Client,
-        key: &str,
-        part_len: usize,
-        seed: u8,
-    ) -> Result<Vec<u8>, Box<dyn Error + Send + Sync>> {
-        let create = client
-            .create_multipart_upload()
-            .bucket(BUCKET)
-            .key(key)
-            .content_type(CONTENT_TYPE)
-            .metadata("compat", "yes")
-            .metadata("shape", "multipart")
-            .send()
-            .await?;
-        let upload_id = create.upload_id().ok_or("missing upload id")?.to_string();
-
-        let mut whole = Vec::new();
-        let mut completed = Vec::new();
-        for part_number in 1..=2i32 {
-            let part = payload(part_len, seed.wrapping_add(part_number as u8));
-            whole.extend_from_slice(&part);
-            let resp = client
-                .upload_part()
-                .bucket(BUCKET)
-                .key(key)
-                .upload_id(&upload_id)
-                .part_number(part_number)
-                .body(ByteStream::from(part))
-                .send()
-                .await?;
-            completed.push(
-                CompletedPart::builder()
-                    .part_number(part_number)
-                    .e_tag(resp.e_tag().unwrap_or_default())
-                    .build(),
-            );
-        }
-
-        client
-            .complete_multipart_upload()
-            .bucket(BUCKET)
-            .key(key)
-            .upload_id(&upload_id)
-            .multipart_upload(CompletedMultipartUpload::builder().set_parts(Some(completed)).build())
-            .send()
-            .await?;
-        Ok(whole)
-    }
-
-    fn count_marker(log_path: &str, marker: &str) -> usize {
-        std::fs::read_to_string(log_path)
-            .map(|s| s.lines().filter(|l| l.contains(marker)).count())
-            .unwrap_or(0)
-    }
-
-    /// Object shapes exercised. `expect_large` marks objects that are stored as
-    /// real EC shards (not inlined), i.e. the ones that take the duplex path in
-    /// the baseline phase and must switch to codec streaming in the codec phase.
-    struct Shape {
-        key: &'static str,
-        expect_large: bool,
-    }
-
-    #[tokio::test]
-    #[serial]
-    async fn codec_streaming_matches_legacy_duplex_body_and_headers() -> TestResult {
-        init_logging();
-
-        let scratch = std::env::var("TMPDIR").unwrap_or_else(|_| "/tmp".into());
-        let run_id = uuid::Uuid::new_v4();
-        let base_log = format!("{scratch}/codec_compat_baseline_{run_id}.log");
-        let codec_log = format!("{scratch}/codec_compat_codec_{run_id}.log");
-
-        let mut harness = DiskFaultHarness::new(4).await?;
-        // Capture ecstore debug logs so we can count the legacy duplex marker.
-        harness.set_env("RUST_LOG", "rustfs=info,rustfs_ecstore=debug");
-
-        // ---- Phase A: baseline (default env → legacy duplex) ----
-        harness.env.capture_log_path = Some(base_log.clone());
-        harness.start_server().await?;
-        let client = harness.env.create_s3_client();
-        client.create_bucket().bucket(BUCKET).send().await?;
-
-        // Object matrix: sizes crossing the inline boundary, the codec min-size
-        // and the 1 MiB EC-block boundary, plus a multipart object.
-        let plain: &[(Shape, Vec<u8>)] = &[
-            (
-                Shape {
-                    key: "inline-1kib",
-                    expect_large: false,
-                },
-                payload(1024, 1),
-            ),
-            (
-                Shape {
-                    key: "small-64kib",
-                    expect_large: false,
-                },
-                payload(64 * 1024, 2),
-            ),
-            (
-                Shape {
-                    key: "mid-1_5mib",
-                    expect_large: true,
-                },
-                payload(MIB + MIB / 2, 3),
-            ),
-            (
-                Shape {
-                    key: "large-3mib",
-                    expect_large: true,
-                },
-                payload(3 * MIB, 4),
-            ),
-            (
-                Shape {
-                    key: "large-5mib-plus",
-                    expect_large: true,
-                },
-                payload(5 * MIB + 12345, 5),
-            ),
-        ];
-        for (shape, data) in plain {
-            put_plain(&client, shape.key, data).await?;
-        }
-        let multipart_key = "multipart-2x5mib";
-        let multipart_body = put_multipart(&client, multipart_key, 5 * MIB, 40).await?;
-
-        // Full-object baseline GETs.
-        let mut baseline: BTreeMap<String, GetView> = BTreeMap::new();
-        for (shape, data) in plain {
-            let view = get_full(&client, shape.key).await?;
-            assert_eq!(view.sha256, sha256_hex(data), "baseline body mismatch for {}", shape.key);
-            assert_eq!(view.len, data.len(), "baseline length mismatch for {}", shape.key);
-            baseline.insert(shape.key.to_string(), view);
-        }
-        let mp_view = get_full(&client, multipart_key).await?;
-        assert_eq!(mp_view.sha256, sha256_hex(&multipart_body), "baseline multipart body mismatch");
-        baseline.insert(multipart_key.to_string(), mp_view);
-
-        // Range GET baseline (a range that starts mid-first-block and crosses a
-        // block boundary) on a large object.
-        let range_spec = "bytes=1048570-2097160";
-        let baseline_range = get_range(&client, "large-3mib", range_spec).await?;
-
-        // Flush + snapshot the baseline duplex count.
-        sleep(Duration::from_millis(300)).await;
-        let num_large = plain.iter().filter(|(s, _)| s.expect_large).count() + 1; // + multipart
-        let dup_base = count_marker(&base_log, DUPLEX_MARKER);
-        info!(dup_base, num_large, "baseline duplex marker count");
-        assert!(
-            dup_base >= num_large,
-            "baseline phase should have used the legacy duplex path for the {num_large} large objects, but only saw {dup_base} duplex markers in {base_log}"
-        );
-
-        // Legacy negative path: a GET for an absent key must fail with a
-        // well-formed NoSuchKey (404). Captured now so Phase B can prove the
-        // codec env returns the identical error semantics.
-        let legacy_missing = missing_key_semantics(&client, MISSING_KEY).await?;
-        assert_eq!(
-            legacy_missing,
-            (404, "NoSuchKey".to_string()),
-            "legacy GET of a missing key should be 404/NoSuchKey, got {legacy_missing:?}"
-        );
-
-        // ---- Phase A degraded: pull one data disk, force parity reconstruction ----
-        // With disk0 offline the EC 2+2 set must rebuild every large object's
-        // data from the surviving data+parity shards. Record the legacy-duplex
-        // bytes and headers produced under reconstruction so Phase B can prove
-        // the codec path reconstructs the same bytes and headers. The duplex
-        // marker snapshot (`dup_base`) is already taken, so these extra reads do
-        // not affect the path-confirmation assertion above.
-        harness.take_disk_offline(0)?;
-        let mut baseline_degraded: BTreeMap<String, GetView> = BTreeMap::new();
-        for (shape, data) in plain {
-            let view = get_full(&client, shape.key).await?;
-            assert_eq!(view.sha256, sha256_hex(data), "degraded baseline body mismatch for {}", shape.key);
-            assert_eq!(view.len, data.len(), "degraded baseline length mismatch for {}", shape.key);
-            baseline_degraded.insert(shape.key.to_string(), view);
-        }
-        let mp_view = get_full(&client, multipart_key).await?;
-        assert_eq!(mp_view.sha256, sha256_hex(&multipart_body), "degraded baseline multipart body mismatch");
-        baseline_degraded.insert(multipart_key.to_string(), mp_view);
-        // Restore the disk so Phase B restarts from a clean, complete disk set.
-        harness.bring_disk_online(0)?;
-
-        // ---- Phase B: codec streaming (gates opened) ----
-        harness.kill_server();
-        for (k, v) in codec_env() {
-            harness.set_env(k, v);
-        }
-        harness.env.capture_log_path = Some(codec_log.clone());
-        harness.restart_server().await?;
-        let client = harness.env.create_s3_client();
-
-        // Full-object codec GETs — compare byte-for-byte and header-for-header.
-        let mut codec: BTreeMap<String, GetView> = BTreeMap::new();
-        for (shape, data) in plain {
-            let view = get_full(&client, shape.key).await?;
-            assert_eq!(view.sha256, sha256_hex(data), "codec body mismatch for {}", shape.key);
-            codec.insert(shape.key.to_string(), view);
-        }
-        let mp_view = get_full(&client, multipart_key).await?;
-        assert_eq!(mp_view.sha256, sha256_hex(&multipart_body), "codec multipart body mismatch");
-        codec.insert(multipart_key.to_string(), mp_view);
-
-        // Negative-path equivalence: the codec env must return the exact same
-        // status + error code as the legacy phase for a missing key.
-        let codec_missing = missing_key_semantics(&client, MISSING_KEY).await?;
-        assert_eq!(
-            codec_missing, legacy_missing,
-            "NoSuchKey error semantics diverged between codec and legacy phases: codec={codec_missing:?} legacy={legacy_missing:?}"
-        );
-
-        // Snapshot the codec-phase duplex count BEFORE issuing the ranged GET
-        // (range falls back to the duplex path by design and would pollute it).
-        sleep(Duration::from_millis(300)).await;
-        let dup_codec = count_marker(&codec_log, DUPLEX_MARKER);
-        info!(dup_codec, "codec phase duplex marker count (full GETs only)");
-
-        // Header + body equivalence: codec == baseline for every object.
-        for key in baseline.keys() {
-            let b = &baseline[key];
-            let c = &codec[key];
-            assert_eq!(c.sha256, b.sha256, "body hash diverged for {key}");
-            assert_eq!(c.len, b.len, "body length diverged for {key}");
-            assert_eq!(
-                c.headers, b.headers,
-                "response headers diverged for {key}\nbaseline={:#?}\ncodec={:#?}",
-                b.headers, c.headers
-            );
-        }
-
-        // Path confirmation: the codec phase must NOT have created any duplex
-        // pipe for the full-object GETs — otherwise it silently fell back to the
-        // legacy path and the equivalence above proves nothing.
-        assert_eq!(
-            dup_codec, 0,
-            "codec phase created {dup_codec} duplex pipe(s) for full GETs; the codec-streaming fast path was not exercised (see {codec_log})"
-        );
-
-        // Range GET while codec streaming is enabled. NOTE ON COVERAGE: the
-        // reader gate unconditionally routes every ranged request back to the
-        // legacy duplex path (`GetCodecStreamingFallbackReason::Range`), so both
-        // `baseline_range` and `codec_range` are produced by the SAME legacy
-        // path. This assertion therefore only verifies that ranged GETs keep
-        // working (and keep falling back to legacy) with the codec gates open —
-        // it does NOT exercise or validate a codec-streaming range reader, which
-        // does not exist. It must not be read as codec range-correctness
-        // coverage.
-        let codec_range = get_range(&client, "large-3mib", range_spec).await?;
-        assert_eq!(
-            codec_range.sha256, baseline_range.sha256,
-            "ranged GET body diverged with codec streaming enabled (both served by the legacy range path)"
-        );
-        assert_eq!(
-            codec_range.len, baseline_range.len,
-            "ranged GET length diverged with codec streaming enabled"
-        );
-
-        // ---- Phase B degraded: the same reconstruction, now on the codec path ----
-        // Re-run the reconstruction A/B with the codec-streaming gates still
-        // open. The reader gate decision is independent of drive health (it
-        // never inspects disk state), so the codec fast path is exercised
-        // end-to-end while the EC set rebuilds each large object from the
-        // surviving shards — this is a real codec-vs-legacy reconstruction test,
-        // not legacy-vs-legacy. Snapshot the duplex count first (the range GET
-        // above already used the duplex path) so we can measure only the markers
-        // these degraded codec GETs add.
-        let dup_codec_before_degraded = count_marker(&codec_log, DUPLEX_MARKER);
-        harness.take_disk_offline(0)?;
-        let mut codec_degraded: BTreeMap<String, GetView> = BTreeMap::new();
-        for (shape, data) in plain {
-            let view = get_full(&client, shape.key).await?;
-            assert_eq!(view.sha256, sha256_hex(data), "degraded codec body mismatch for {}", shape.key);
-            codec_degraded.insert(shape.key.to_string(), view);
-        }
-        let mp_view = get_full(&client, multipart_key).await?;
-        assert_eq!(mp_view.sha256, sha256_hex(&multipart_body), "degraded codec multipart body mismatch");
-        codec_degraded.insert(multipart_key.to_string(), mp_view);
-        harness.bring_disk_online(0)?;
-
-        // A/B under parity reconstruction: codec == legacy, byte-for-byte and
-        // header-for-header, for every object in the matrix.
-        for key in baseline_degraded.keys() {
-            let b = &baseline_degraded[key];
-            let c = &codec_degraded[key];
-            assert_eq!(c.sha256, b.sha256, "degraded body hash diverged for {key} (parity reconstruction)");
-            assert_eq!(c.len, b.len, "degraded body length diverged for {key} (parity reconstruction)");
-            assert_eq!(
-                c.headers, b.headers,
-                "degraded response headers diverged for {key} (parity reconstruction)\nbaseline={:#?}\ncodec={:#?}",
-                b.headers, c.headers
-            );
-        }
-
-        // Path confirmation under reconstruction: the codec fast path must have
-        // served the reconstructed large objects without ever falling back to
-        // the legacy duplex pipe. Without this, the equivalence above could be
-        // legacy-vs-legacy and prove nothing about codec reconstruction.
-        sleep(Duration::from_millis(300)).await;
-        let dup_codec_degraded = count_marker(&codec_log, DUPLEX_MARKER).saturating_sub(dup_codec_before_degraded);
-        assert_eq!(
-            dup_codec_degraded, 0,
-            "codec phase created {dup_codec_degraded} duplex pipe(s) while reconstructing large objects with disk0 offline; the codec fast path was not exercised under degraded reads (see {codec_log})"
-        );
-
-        info!(
-            objects = baseline.len(),
-            "codec streaming produced byte- and header-identical GET responses vs legacy duplex (healthy + parity-reconstructed + NoSuchKey)"
-        );
-
-        // Best-effort cleanup of the capture logs.
-        let _ = std::fs::remove_file(&base_log);
-        let _ = std::fs::remove_file(&codec_log);
-        Ok(())
-    }
-}

crates/e2e_test/src/lib.rs

@@ -27,16 +27,6 @@ pub mod chaos;
 #[cfg(test)]
 mod reliability_disk_fault_test;
 
-// dist-13 (backlog#1150/#1155): e2e regression net proving a large-object
-// degraded EC read never returns a silently truncated body (rustfs#4594/#4560/#4585).
-#[cfg(test)]
-mod degraded_read_eof_regression_test;
-
-// backlog#1183: GET codec-streaming fast path must be byte/header identical to
-// the legacy duplex path before its rollout gates can be flipped on by default.
-#[cfg(test)]
-mod get_codec_streaming_compat_test;
-
 #[cfg(test)]
 mod version_id_regression_test;
 
@@ -56,10 +46,6 @@ mod list_objects_duplicates_test;
 #[cfg(test)]
 mod quota_test;
 
-// Harness regression tests: console port isolation + fail-fast startup
-#[cfg(test)]
-mod server_startup_failfast_test;
-
 #[cfg(test)]
 mod bucket_policy_check_test;
 
@@ -67,10 +53,6 @@ mod bucket_policy_check_test;
 #[cfg(test)]
 mod security_boundary_test;
 
-// Admin authorization gate: non-admin denial + root-credential lifecycle (backlog#1151 sec-4)
-#[cfg(test)]
-mod admin_auth_test;
-
 /// IAM / bucket / STS session policy with `s3:ExistingObjectTag` conditions (E2E).
 #[cfg(test)]
 mod existing_object_tag_policy_test;
@@ -174,16 +156,6 @@ mod bucket_logging_test;
 #[cfg(test)]
 mod multipart_auth_test;
 
-// Negative presigned-URL (query-string SigV4) regression suite (backlog#1151
-// sec-2): expired, tampered signature, wrong secret, tampered target.
-#[cfg(test)]
-mod presigned_negative_test;
-
-// Negative header-SigV4 regression suite (backlog#1151 sec-1): tampered
-// signature, wrong secret, skewed date, malformed Authorization.
-#[cfg(test)]
-mod negative_sigv4_test;
-
 #[cfg(test)]
 mod stale_multipart_cleanup_cluster_test;
 

crates/e2e_test/src/list_objects_duplicates_test.rs

@@ -198,121 +198,4 @@ mod tests {
 
         env.stop_server();
     }
-
-    /// Test ensuring that a plain object and a same-named prefix coexist in
-    /// delimiter listings on a single-disk deployment.
-    ///
-    /// Bug Reference: backlog#880 / backlog#1042
-    /// On a single disk, object `a` and its children `a/...` share one backing
-    /// directory, so the non-recursive scan used to classify `a` as an object
-    /// and never produce the prefix entry `a/`. Delimiter="/" listings then
-    /// returned Contents `a` but silently dropped CommonPrefix `a/`.
-    #[tokio::test]
-    #[serial]
-    async fn test_list_objects_v2_object_and_same_named_prefix_coexist() {
-        init_logging();
-        info!("Starting test: ListObjectsV2 should return both object `a` and CommonPrefix `a/`");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("Failed to start RustFS");
-
-        let client = create_s3_client(&env);
-        let bucket = "test-list-object-prefix-coexist";
-
-        create_bucket(&client, bucket).await.expect("Failed to create bucket");
-
-        for (key, body) in [
-            ("a", ByteStream::from_static(b"object body")),
-            ("a/b", ByteStream::from_static(b"child body")),
-            ("plain", ByteStream::from_static(b"no children")),
-        ] {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(key)
-                .body(body)
-                .send()
-                .await
-                .unwrap_or_else(|err| panic!("Failed to create test object {key}: {err}"));
-        }
-
-        let result = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .delimiter("/")
-            .send()
-            .await
-            .expect("Failed to list objects");
-
-        let keys: Vec<&str> = result.contents().iter().filter_map(|object| object.key()).collect();
-        let prefixes: Vec<&str> = result.common_prefixes().iter().filter_map(|prefix| prefix.prefix()).collect();
-
-        info!("Contents: {:?}, CommonPrefixes: {:?}", keys, prefixes);
-
-        assert_eq!(keys, vec!["a", "plain"], "objects `a` and `plain` must both stay in Contents");
-        assert_eq!(
-            prefixes,
-            vec!["a/"],
-            "prefix `a/` must be listed and `plain/` must not appear as a phantom prefix"
-        );
-
-        // Children are still reachable under the prefix.
-        let nested = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .prefix("a/")
-            .delimiter("/")
-            .send()
-            .await
-            .expect("Failed to list objects under prefix");
-        let nested_keys: Vec<&str> = nested.contents().iter().filter_map(|object| object.key()).collect();
-        assert_eq!(nested_keys, vec!["a/b"]);
-
-        // Pagination must keep both entries across page boundaries: `a` sorts
-        // before `a/`, so a one-key page splits them.
-        let page1 = client
-            .list_objects_v2()
-            .bucket(bucket)
-            .delimiter("/")
-            .max_keys(1)
-            .send()
-            .await
-            .expect("Failed to list first page");
-        let page1_keys: Vec<&str> = page1.contents().iter().filter_map(|object| object.key()).collect();
-        assert_eq!(page1_keys, vec!["a"]);
-        assert_eq!(page1.is_truncated(), Some(true), "one-key first page must be truncated");
-
-        let mut token = page1.next_continuation_token().map(ToOwned::to_owned);
-        let mut remaining_keys = Vec::new();
-        let mut remaining_prefixes = Vec::new();
-        while let Some(continuation) = token {
-            let page = client
-                .list_objects_v2()
-                .bucket(bucket)
-                .delimiter("/")
-                .max_keys(1)
-                .continuation_token(continuation)
-                .send()
-                .await
-                .expect("Failed to list continuation page");
-            remaining_keys.extend(
-                page.contents()
-                    .iter()
-                    .filter_map(|object| object.key().map(ToOwned::to_owned)),
-            );
-            remaining_prefixes.extend(
-                page.common_prefixes()
-                    .iter()
-                    .filter_map(|prefix| prefix.prefix().map(ToOwned::to_owned)),
-            );
-            token = page.next_continuation_token().map(ToOwned::to_owned);
-        }
-
-        assert_eq!(remaining_prefixes, vec!["a/".to_string()], "prefix `a/` must survive pagination");
-        assert_eq!(remaining_keys, vec!["plain".to_string()]);
-
-        env.stop_server();
-    }
 }

crates/e2e_test/src/list_objects_v2_pagination_test.rs

@@ -1029,67 +1029,4 @@ mod tests {
 
         env.stop_server();
     }
-
-    /// Test that continuation pages do not skip keys sorting between the marker
-    /// and the marker plus the cursor tag.
-    ///
-    /// Bug Reference: backlog#1047
-    /// The V2 continuation token appends a `[rustfs_cache:...]` cursor tag to the
-    /// last returned key. The orchestration layer passed that raw string to
-    /// `forward_past`, so any key whose byte after the shared stem sorts below
-    /// `[` (0x5B) — `.`, `/`, `-`, digits, uppercase — was silently dropped on
-    /// the next page: with keys `a`, `a.txt`, `zz` and max_keys=1, page 2
-    /// returned `zz` and `a.txt` was never listed.
-    #[tokio::test]
-    #[serial]
-    async fn test_list_objects_v2_continuation_keeps_keys_after_marker_stem() {
-        init_logging();
-        info!("Starting test: continuation must not skip keys sorting below the cursor tag");
-
-        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
-        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "false")])
-            .await
-            .expect("Failed to start RustFS");
-
-        let client = create_s3_client(&env);
-        let bucket = "test-continuation-marker-stem";
-
-        create_bucket(&client, bucket).await.expect("Failed to create bucket");
-
-        let expected_keys = ["a", "a.txt", "zz"];
-        for key in expected_keys {
-            client
-                .put_object()
-                .bucket(bucket)
-                .key(key)
-                .body(ByteStream::from_static(b"content"))
-                .send()
-                .await
-                .unwrap_or_else(|err| panic!("Failed to create test object {key}: {err}"));
-        }
-
-        let mut collected: Vec<String> = Vec::new();
-        let mut token: Option<String> = None;
-        loop {
-            let mut request = client.list_objects_v2().bucket(bucket).max_keys(1);
-            if let Some(continuation) = token.take() {
-                request = request.continuation_token(continuation);
-            }
-            let page = request.send().await.expect("Failed to list objects");
-            collected.extend(
-                page.contents()
-                    .iter()
-                    .filter_map(|object| object.key().map(ToOwned::to_owned)),
-            );
-            token = page.next_continuation_token().map(ToOwned::to_owned);
-            if token.is_none() {
-                break;
-            }
-            assert!(collected.len() <= expected_keys.len() + 1, "pagination did not converge: {collected:?}");
-        }
-
-        assert_eq!(collected, expected_keys, "every key must survive pagination in order");
-
-        env.stop_server();
-    }
 }

crates/e2e_test/src/negative_sigv4_test.rs

@@ -1,382 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Negative header-SigV4 regression suite (backlog#1151 sec-1).
-//!
-//! RustFS delegates SigV4 verification to the `s3s` dependency, so nothing in
-//! this repository pins OUR end-to-end wiring of it: a future dependency swap
-//! or misconfiguration could silently start accepting forged header
-//! signatures. These tests send REJECTED SigV4 header-auth requests against a
-//! live server and assert the HTTP status plus the S3 error code in the
-//! response XML, guarding the rejection contract regardless of who performs
-//! the underlying verification.
-//!
-//! Signatures are hand-built (rather than produced by the AWS SDK, which
-//! cannot emit an invalid signature) by reusing the primitive HMAC/scope
-//! helpers from `rustfs_signer::request_signature_v4`. This gives the test
-//! full control over the timestamp, secret key, signed payload hash, and the
-//! final signature bytes.
-//!
-//! Missing-credential negatives are intentionally NOT duplicated here: those
-//! are already covered by `multipart_auth_test` (anonymous / no-credential
-//! cases) and `anonymous_access_test`. This module covers only PRESENT but
-//! rejected header-SigV4 requests.
-
-use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
-use aws_sdk_s3::primitives::ByteStream;
-use rustfs_signer::constants::UNSIGNED_PAYLOAD;
-use rustfs_signer::request_signature_v4::{SIGN_V4_ALGORITHM, get_scope, get_signature, get_signing_key};
-use serial_test::serial;
-use std::fmt::Write as _;
-use time::macros::format_description;
-use time::{Duration, OffsetDateTime};
-use tracing::info;
-
-const REGION: &str = "us-east-1";
-const BUCKET: &str = "negative-sigv4-bucket";
-
-/// Lowercase hex encoding (matches SigV4 canonical hex format).
-fn hex_lower(bytes: &[u8]) -> String {
-    let mut out = String::with_capacity(bytes.len() * 2);
-    for b in bytes {
-        let _ = write!(out, "{b:02x}");
-    }
-    out
-}
-
-fn sha256_hex(data: &[u8]) -> String {
-    use sha2::{Digest, Sha256};
-    hex_lower(&Sha256::digest(data))
-}
-
-fn amz_datetime(t: OffsetDateTime) -> String {
-    let fmt = format_description!("[year][month][day]T[hour][minute][second]Z");
-    t.format(&fmt).expect("format x-amz-date")
-}
-
-/// A minimal hand-rolled SigV4 header signer with full control over every
-/// input, so tests can deliberately produce forged / stale / mismatched
-/// requests. Always signs exactly `host;x-amz-content-sha256;x-amz-date`.
-struct SigV4 {
-    access_key: String,
-    secret_key: String,
-    host: String,
-    time: OffsetDateTime,
-}
-
-struct SignedHeaders {
-    authorization: String,
-    amz_date: String,
-    content_sha256: String,
-}
-
-impl SigV4 {
-    fn new(env: &RustFSTestEnvironment) -> Self {
-        Self {
-            access_key: env.access_key.clone(),
-            secret_key: env.secret_key.clone(),
-            host: env.address.clone(),
-            time: OffsetDateTime::now_utc(),
-        }
-    }
-
-    /// Build the Authorization header (and the companion `x-amz-date` /
-    /// `x-amz-content-sha256` header values) for a request.
-    ///
-    /// `content_sha256` is the value placed in the `x-amz-content-sha256`
-    /// header AND folded into the canonical request — pass the hash of the
-    /// body you *claim* to send, which may differ from what you actually send.
-    fn sign(&self, method: &str, path: &str, canonical_query: &str, content_sha256: &str) -> SignedHeaders {
-        let amz_date = amz_datetime(self.time);
-        let signed_headers = "host;x-amz-content-sha256;x-amz-date";
-
-        let canonical_headers = format!(
-            "host:{host}\nx-amz-content-sha256:{sha}\nx-amz-date:{date}\n",
-            host = self.host,
-            sha = content_sha256,
-            date = amz_date,
-        );
-        let canonical_request =
-            format!("{method}\n{path}\n{canonical_query}\n{canonical_headers}\n{signed_headers}\n{content_sha256}");
-
-        let scope = get_scope(REGION, self.time, "s3");
-        let string_to_sign = format!("{SIGN_V4_ALGORITHM}\n{amz_date}\n{scope}\n{}", sha256_hex(canonical_request.as_bytes()));
-        let signing_key = get_signing_key(&self.secret_key, REGION, self.time, "s3");
-        let signature = get_signature(signing_key, &string_to_sign);
-
-        let credential = format!("{}/{scope}", self.access_key);
-        let authorization =
-            format!("{SIGN_V4_ALGORITHM} Credential={credential}, SignedHeaders={signed_headers}, Signature={signature}");
-
-        SignedHeaders {
-            authorization,
-            amz_date,
-            content_sha256: content_sha256.to_string(),
-        }
-    }
-}
-
-/// Send a request carrying explicit SigV4 headers. `reqwest` populates `Host`
-/// (matching the signed host) and `Content-Length` automatically.
-async fn send_signed(
-    env: &RustFSTestEnvironment,
-    method: reqwest::Method,
-    path: &str,
-    headers: &SignedHeaders,
-    body: Option<Vec<u8>>,
-) -> reqwest::Result<reqwest::Response> {
-    let url = format!("{}{}", env.url, path);
-    let mut builder = local_http_client()
-        .request(method, &url)
-        .header("x-amz-date", &headers.amz_date)
-        .header("x-amz-content-sha256", &headers.content_sha256)
-        .header("authorization", &headers.authorization);
-    if let Some(body) = body {
-        builder = builder.body(body);
-    }
-    builder.send().await
-}
-
-/// Send a request with a raw (possibly malformed) Authorization header while
-/// keeping the other SigV4 headers well-formed.
-async fn send_raw_authorization(
-    env: &RustFSTestEnvironment,
-    method: reqwest::Method,
-    path: &str,
-    authorization: &str,
-) -> reqwest::Result<reqwest::Response> {
-    let url = format!("{}{}", env.url, path);
-    local_http_client()
-        .request(method, &url)
-        .header("x-amz-date", amz_datetime(OffsetDateTime::now_utc()))
-        .header("x-amz-content-sha256", UNSIGNED_PAYLOAD)
-        .header("authorization", authorization)
-        .send()
-        .await
-}
-
-fn assert_error_code(body: &str, code: &str) {
-    assert!(
-        body.contains(&format!("<Code>{code}</Code>")),
-        "expected S3 error code <Code>{code}</Code> in response body, got:\n{body}"
-    );
-}
-
-async fn setup(env: &mut RustFSTestEnvironment) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    env.start_rustfs_server(vec![]).await?;
-    env.create_test_bucket(BUCKET).await?;
-    Ok(())
-}
-
-/// Positive control: a correctly hand-signed request must succeed. Without
-/// this, every negative assertion below could pass for the wrong reason (a
-/// broken signer that never produces a valid signature).
-#[tokio::test]
-#[serial]
-async fn valid_header_sigv4_request_succeeds() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-    let mut env = RustFSTestEnvironment::new().await?;
-    setup(&mut env).await?;
-
-    let key = "valid-control.txt";
-    let expected = b"valid-sigv4-control-body";
-    env.create_s3_client()
-        .put_object()
-        .bucket(BUCKET)
-        .key(key)
-        .body(ByteStream::from_static(expected))
-        .send()
-        .await?;
-
-    let path = format!("/{BUCKET}/{key}");
-    let signer = SigV4::new(&env);
-    let headers = signer.sign("GET", &path, "", UNSIGNED_PAYLOAD);
-    let resp = send_signed(&env, reqwest::Method::GET, &path, &headers, None).await?;
-
-    assert_eq!(resp.status().as_u16(), 200, "correctly signed GET should succeed");
-    let bytes = resp.bytes().await?;
-    assert_eq!(bytes.as_ref(), expected, "GET body must match stored object");
-    info!("valid header SigV4 control passed");
-    Ok(())
-}
-
-/// (a) Tampering the `Signature=` component must be rejected with
-/// SignatureDoesNotMatch / 403.
-#[tokio::test]
-#[serial]
-async fn tampered_signature_returns_signature_does_not_match() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-    let mut env = RustFSTestEnvironment::new().await?;
-    setup(&mut env).await?;
-
-    let path = format!("/{BUCKET}/any-key.txt");
-    let signer = SigV4::new(&env);
-    let mut headers = signer.sign("GET", &path, "", UNSIGNED_PAYLOAD);
-
-    // Flip bytes inside the Signature= hex without changing its length/shape.
-    let marker = "Signature=";
-    let idx = headers
-        .authorization
-        .find(marker)
-        .expect("authorization must carry Signature=")
-        + marker.len();
-    let (head, sig) = headers.authorization.split_at(idx);
-    let tampered: String = sig
-        .chars()
-        .map(|c| match c {
-            '0' => 'f',
-            'a' => '0',
-            other => other,
-        })
-        .collect();
-    assert_ne!(sig, tampered, "tamper must actually change the signature hex");
-    headers.authorization = format!("{head}{tampered}");
-
-    let resp = send_signed(&env, reqwest::Method::GET, &path, &headers, None).await?;
-    let status = resp.status();
-    let body = resp.text().await?;
-    assert_eq!(status.as_u16(), 403, "tampered signature must be 403, body:\n{body}");
-    assert_error_code(&body, "SignatureDoesNotMatch");
-    Ok(())
-}
-
-/// (b) A valid AccessKeyId paired with the wrong secret key must be rejected
-/// with SignatureDoesNotMatch / 403.
-#[tokio::test]
-#[serial]
-async fn wrong_secret_key_returns_signature_does_not_match() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-    let mut env = RustFSTestEnvironment::new().await?;
-    setup(&mut env).await?;
-
-    let path = format!("/{BUCKET}/any-key.txt");
-    let mut signer = SigV4::new(&env);
-    // Correct, existing access key; wrong (but validly-shaped) secret.
-    signer.secret_key = "totally-wrong-secret-key".to_string();
-    let headers = signer.sign("GET", &path, "", UNSIGNED_PAYLOAD);
-
-    let resp = send_signed(&env, reqwest::Method::GET, &path, &headers, None).await?;
-    let status = resp.status();
-    let body = resp.text().await?;
-    assert_eq!(status.as_u16(), 403, "wrong secret must be 403, body:\n{body}");
-    assert_error_code(&body, "SignatureDoesNotMatch");
-    Ok(())
-}
-
-/// (c) A correctly signed request whose actual body differs from the signed
-/// `x-amz-content-sha256` must NOT be accepted (must not return 200). The
-/// signature itself is valid (it covers the *declared* hash), so the server is
-/// forced to detect the payload/hash mismatch while streaming the body.
-#[tokio::test]
-#[serial]
-async fn tampered_payload_is_rejected() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-    let mut env = RustFSTestEnvironment::new().await?;
-    setup(&mut env).await?;
-
-    let path = format!("/{BUCKET}/tampered-payload.txt");
-    let claimed_body = b"the-body-i-claim-to-send";
-    let actual_body = b"the-body-i-really-send!!";
-    assert_eq!(claimed_body.len(), actual_body.len(), "keep content-length stable for the mismatch");
-
-    let signer = SigV4::new(&env);
-    // Sign over the hash of the CLAIMED body (single-chunk payload hash), then
-    // send a different body of equal length.
-    let headers = signer.sign("PUT", &path, "", &sha256_hex(claimed_body));
-
-    let result = send_signed(&env, reqwest::Method::PUT, &path, &headers, Some(actual_body.to_vec())).await;
-    match result {
-        Ok(resp) => {
-            let status = resp.status();
-            let body = resp.text().await.unwrap_or_default();
-            assert_ne!(status.as_u16(), 200, "payload mismatch must not succeed, body:\n{body}");
-            assert!(
-                status.is_client_error() || status.is_server_error(),
-                "payload mismatch must be an error status, got {status}, body:\n{body}"
-            );
-            info!(%status, "tampered payload rejected with error status");
-        }
-        // A mid-stream hash-mismatch abort surfacing as a transport error is
-        // also a valid rejection (definitely not a 200 success).
-        Err(err) => info!(%err, "tampered payload rejected via transport error"),
-    }
-    Ok(())
-}
-
-/// (e) A request whose `x-amz-date` is skewed beyond the server's tolerance
-/// (s3s default 900s / 15 min) must be rejected with RequestTimeTooSkewed /
-/// 403. The signature is otherwise valid: the credential-scope date and
-/// x-amz-date both derive from the same skewed timestamp, so skew — not a
-/// signature mismatch — is the failure.
-#[tokio::test]
-#[serial]
-async fn skewed_date_returns_request_time_too_skewed() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-    let mut env = RustFSTestEnvironment::new().await?;
-    setup(&mut env).await?;
-
-    let path = format!("/{BUCKET}/any-key.txt");
-    let mut signer = SigV4::new(&env);
-    signer.time = OffsetDateTime::now_utc() - Duration::minutes(20); // > 15 min window
-    let headers = signer.sign("GET", &path, "", UNSIGNED_PAYLOAD);
-
-    let resp = send_signed(&env, reqwest::Method::GET, &path, &headers, None).await?;
-    let status = resp.status();
-    let body = resp.text().await?;
-    assert_eq!(status.as_u16(), 403, "skewed date must be 403, body:\n{body}");
-    assert_error_code(&body, "RequestTimeTooSkewed");
-    Ok(())
-}
-
-/// (d) Malformed (but PRESENT, not missing) Authorization headers must produce
-/// clean 4xx errors — never a 5xx and never a panic/hang. Each variant is a
-/// structurally invalid SigV4 header that must be rejected before any
-/// credential/service handling.
-#[tokio::test]
-#[serial]
-async fn malformed_authorization_header_returns_clean_4xx() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-    let mut env = RustFSTestEnvironment::new().await?;
-    setup(&mut env).await?;
-
-    let path = format!("/{BUCKET}/any-key.txt");
-    let variants = [
-        // Algorithm token only, nothing else.
-        "AWS4-HMAC-SHA256",
-        // Well-formed algorithm but unparseable remainder.
-        "AWS4-HMAC-SHA256 total-garbage-not-sigv4",
-        // Missing the Signature= component entirely.
-        "AWS4-HMAC-SHA256 Credential=rustfsadmin/20240101/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date",
-        // Credential scope is not the required access/date/region/service/aws4_request shape.
-        "AWS4-HMAC-SHA256 Credential=not-a-valid-scope, SignedHeaders=host, Signature=deadbeef",
-        // Empty value.
-        "AWS4-HMAC-SHA256 ",
-    ];
-
-    for variant in variants {
-        let resp = send_raw_authorization(&env, reqwest::Method::GET, &path, variant).await?;
-        let status = resp.status();
-        let body = resp.text().await.unwrap_or_default();
-        assert!(
-            status.is_client_error(),
-            "malformed Authorization {variant:?} must yield a 4xx (got {status}); body:\n{body}"
-        );
-        assert!(
-            !status.is_server_error(),
-            "malformed Authorization {variant:?} must never yield a 5xx (got {status})"
-        );
-        info!(%status, variant, "malformed Authorization rejected with clean 4xx");
-    }
-    Ok(())
-}

crates/e2e_test/src/object_lock/object_lock_test.rs

@@ -470,15 +470,11 @@ async fn test_get_object_retention_returns_configured_values() {
 // Put/Copy/Multipart Legal Hold Tests
 // ============================================================================
 
-// AWS semantics (PR #3179): Object Lock buckets are always versioned, so a
-// plain PUT/copy/multipart write to a held or retained key succeeds by
-// creating a new current version. The lock protects the existing version
-// from deletion; it never blocks new versions.
 #[tokio::test]
 #[serial]
-async fn test_put_object_overwrite_creates_new_version_under_legal_hold() {
+async fn test_put_object_overwrite_blocked_by_legal_hold() {
     init_logging();
-    info!("🧪 Test: PutObject overwrite of a legal-hold version creates a new version");
+    info!("🧪 Test: PutObject overwrite blocked by Legal Hold");
 
     let mut env = ObjectLockTestEnvironment::new().await.unwrap();
     env.start_rustfs().await.unwrap();
@@ -490,73 +486,25 @@ async fn test_put_object_overwrite_creates_new_version_under_legal_hold() {
 
     let client = env.s3_client();
 
-    let held_version_id = put_object_with_legal_hold(&client, bucket, key, b"locked-body", ObjectLockLegalHoldStatus::On)
+    put_object_with_legal_hold(&client, bucket, key, b"locked-body", ObjectLockLegalHoldStatus::On)
         .await
         .unwrap();
 
-    let overwrite_output = client
+    let overwrite_result = client
         .put_object()
         .bucket(bucket)
         .key(key)
         .body(ByteStream::from(b"replacement-body".to_vec()))
         .send()
-        .await
-        .expect("PutObject overwrite should succeed by creating a new version");
+        .await;
 
-    let new_version_id = overwrite_output
-        .version_id()
-        .expect("versioned put should return a version id")
-        .to_string();
-    assert_ne!(new_version_id, held_version_id, "overwrite must create a distinct version");
+    assert!(overwrite_result.is_err(), "PutObject overwrite should fail while legal hold is ON");
 
-    let current_body = client
-        .get_object()
-        .bucket(bucket)
-        .key(key)
-        .send()
-        .await
-        .unwrap()
-        .body
-        .collect()
-        .await
-        .unwrap()
-        .into_bytes();
-    assert_eq!(current_body.as_ref(), b"replacement-body");
-
-    let held_body = client
-        .get_object()
-        .bucket(bucket)
-        .key(key)
-        .version_id(&held_version_id)
-        .send()
-        .await
-        .unwrap()
-        .body
-        .collect()
-        .await
-        .unwrap()
-        .into_bytes();
-    assert_eq!(held_body.as_ref(), b"locked-body", "held version must keep its original content");
-
-    let held_legal_hold = client
-        .get_object_legal_hold()
-        .bucket(bucket)
-        .key(key)
-        .version_id(&held_version_id)
-        .send()
-        .await
-        .unwrap();
-    assert_eq!(
-        held_legal_hold
-            .legal_hold()
-            .and_then(|value| value.status())
-            .map(|value| value.as_str()),
-        Some("ON"),
-        "held version must keep its legal hold after the overwrite"
+    let error_str = format!("{:?}", overwrite_result.unwrap_err());
+    assert!(
+        error_str.to_lowercase().contains("legal") || error_str.to_lowercase().contains("hold"),
+        "overwrite error should mention legal hold, got: {error_str}"
     );
-
-    let delete_result = delete_object_with_bypass(&client, bucket, key, Some(&held_version_id), false).await;
-    assert!(delete_result.is_err(), "held version must stay delete-protected after the overwrite");
 }
 
 #[tokio::test]
@@ -707,9 +655,9 @@ async fn test_copy_object_does_not_inherit_source_legal_hold() {
 
 #[tokio::test]
 #[serial]
-async fn test_copy_object_overwrite_creates_new_version_under_legal_hold() {
+async fn test_copy_object_overwrite_blocked_by_legal_hold() {
     init_logging();
-    info!("🧪 Test: CopyObject overwrite of a legal-hold destination creates a new version");
+    info!("🧪 Test: CopyObject overwrite blocked by Legal Hold");
 
     let mut env = ObjectLockTestEnvironment::new().await.unwrap();
     env.start_rustfs().await.unwrap();
@@ -730,59 +678,28 @@ async fn test_copy_object_overwrite_creates_new_version_under_legal_hold() {
         .await
         .unwrap();
 
-    let held_version_id =
-        put_object_with_legal_hold(&client, bucket, dst_key, b"locked-destination", ObjectLockLegalHoldStatus::On)
-            .await
-            .unwrap();
+    put_object_with_legal_hold(&client, bucket, dst_key, b"locked-destination", ObjectLockLegalHoldStatus::On)
+        .await
+        .unwrap();
 
-    let copy_output = client
+    let copy_result = client
         .copy_object()
         .copy_source(format!("{bucket}/{src_key}"))
         .bucket(bucket)
         .key(dst_key)
         .send()
-        .await
-        .expect("CopyObject overwrite should succeed by creating a new destination version");
+        .await;
 
-    let new_version_id = copy_output
-        .version_id()
-        .expect("versioned copy should return a version id")
-        .to_string();
-    assert_ne!(new_version_id, held_version_id, "copy must create a distinct destination version");
-
-    let current_body = client
-        .get_object()
-        .bucket(bucket)
-        .key(dst_key)
-        .send()
-        .await
-        .unwrap()
-        .body
-        .collect()
-        .await
-        .unwrap()
-        .into_bytes();
-    assert_eq!(current_body.as_ref(), b"copy-source");
-
-    let held_legal_hold = client
-        .get_object_legal_hold()
-        .bucket(bucket)
-        .key(dst_key)
-        .version_id(&held_version_id)
-        .send()
-        .await
-        .unwrap();
-    assert_eq!(
-        held_legal_hold
-            .legal_hold()
-            .and_then(|value| value.status())
-            .map(|value| value.as_str()),
-        Some("ON"),
-        "held destination version must keep its legal hold after the copy"
+    assert!(
+        copy_result.is_err(),
+        "CopyObject overwrite should fail while destination legal hold is ON"
     );
 
-    let delete_result = delete_object_with_bypass(&client, bucket, dst_key, Some(&held_version_id), false).await;
-    assert!(delete_result.is_err(), "held destination version must stay delete-protected");
+    let error_str = format!("{:?}", copy_result.unwrap_err());
+    assert!(
+        error_str.to_lowercase().contains("legal") || error_str.to_lowercase().contains("hold"),
+        "copy overwrite error should mention legal hold, got: {error_str}"
+    );
 }
 
 #[tokio::test]
@@ -853,9 +770,9 @@ async fn test_create_multipart_upload_applies_requested_legal_hold() {
 
 #[tokio::test]
 #[serial]
-async fn test_create_multipart_upload_creates_new_version_under_compliance_retention() {
+async fn test_create_multipart_upload_blocked_by_compliance_retention() {
     init_logging();
-    info!("🧪 Test: CreateMultipartUpload over a COMPLIANCE-retained key creates a new version");
+    info!("🧪 Test: CreateMultipartUpload blocked by COMPLIANCE retention");
 
     let mut env = ObjectLockTestEnvironment::new().await.unwrap();
     env.start_rustfs().await.unwrap();
@@ -866,7 +783,7 @@ async fn test_create_multipart_upload_creates_new_version_under_compliance_reten
     env.create_object_lock_bucket(bucket).await.unwrap();
 
     let client = env.s3_client();
-    let retained_version_id = put_object_with_retention(
+    put_object_with_retention(
         &client,
         bucket,
         key,
@@ -877,57 +794,17 @@ async fn test_create_multipart_upload_creates_new_version_under_compliance_reten
     .await
     .unwrap();
 
-    let create_output = client
-        .create_multipart_upload()
-        .bucket(bucket)
-        .key(key)
-        .send()
-        .await
-        .expect("CreateMultipartUpload should succeed; completion will create a new version");
+    let create_result = client.create_multipart_upload().bucket(bucket).key(key).send().await;
 
-    let upload_id = create_output.upload_id().unwrap();
-    let upload_part_output = client
-        .upload_part()
-        .bucket(bucket)
-        .key(key)
-        .upload_id(upload_id)
-        .part_number(1)
-        .body(ByteStream::from(b"multipart-body".to_vec()))
-        .send()
-        .await
-        .unwrap();
-
-    let completed_upload = CompletedMultipartUpload::builder()
-        .parts(
-            CompletedPart::builder()
-                .part_number(1)
-                .e_tag(upload_part_output.e_tag().unwrap_or_default())
-                .build(),
-        )
-        .build();
-
-    let complete_output = client
-        .complete_multipart_upload()
-        .bucket(bucket)
-        .key(key)
-        .upload_id(upload_id)
-        .multipart_upload(completed_upload)
-        .send()
-        .await
-        .expect("CompleteMultipartUpload should succeed by creating a new version");
-
-    let new_version_id = complete_output
-        .version_id()
-        .expect("versioned multipart completion should return a version id")
-        .to_string();
-    assert_ne!(new_version_id, retained_version_id, "completion must create a distinct version");
-
-    // COMPLIANCE retention on the previous version survives the overwrite and
-    // cannot be bypassed.
-    let delete_result = delete_object_with_bypass(&client, bucket, key, Some(&retained_version_id), true).await;
     assert!(
-        delete_result.is_err(),
-        "retained version must stay delete-protected even with governance bypass"
+        create_result.is_err(),
+        "CreateMultipartUpload should fail while destination is under active COMPLIANCE retention"
+    );
+
+    let error_str = format!("{:?}", create_result.unwrap_err());
+    assert!(
+        error_str.to_lowercase().contains("retention") || error_str.to_lowercase().contains("compliance"),
+        "multipart create error should mention retention, got: {error_str}"
     );
 }
 
@@ -1055,9 +932,9 @@ async fn test_delete_completed_multipart_object_blocked_by_retention() {
 
 #[tokio::test]
 #[serial]
-async fn test_complete_multipart_upload_creates_new_version_under_legal_hold() {
+async fn test_complete_multipart_upload_blocked_when_legal_hold_added_after_create() {
     init_logging();
-    info!("🧪 Test: CompleteMultipartUpload creates a new version when the current version is under Legal Hold");
+    info!("🧪 Test: CompleteMultipartUpload blocked when Legal Hold appears after MPU creation");
 
     let mut env = ObjectLockTestEnvironment::new().await.unwrap();
     env.start_rustfs().await.unwrap();
@@ -1082,10 +959,9 @@ async fn test_complete_multipart_upload_creates_new_version_under_legal_hold() {
         .await
         .unwrap();
 
-    let held_version_id =
-        put_object_with_legal_hold(&client, bucket, key, b"locked-current-version", ObjectLockLegalHoldStatus::On)
-            .await
-            .unwrap();
+    put_object_with_legal_hold(&client, bucket, key, b"locked-current-version", ObjectLockLegalHoldStatus::On)
+        .await
+        .unwrap();
 
     let completed_upload = CompletedMultipartUpload::builder()
         .parts(
@@ -1096,48 +972,29 @@ async fn test_complete_multipart_upload_creates_new_version_under_legal_hold() {
         )
         .build();
 
-    let complete_output = client
+    let complete_result = client
         .complete_multipart_upload()
         .bucket(bucket)
         .key(key)
         .upload_id(upload_id)
         .multipart_upload(completed_upload)
         .send()
-        .await
-        .expect("CompleteMultipartUpload should succeed by creating a new version over the held one");
+        .await;
 
-    let new_version_id = complete_output
-        .version_id()
-        .expect("versioned multipart completion should return a version id")
-        .to_string();
-    assert_ne!(new_version_id, held_version_id, "completion must create a distinct version");
+    assert!(complete_result.is_err(), "CompleteMultipartUpload should fail once legal hold is enabled");
 
-    let held_legal_hold = client
-        .get_object_legal_hold()
-        .bucket(bucket)
-        .key(key)
-        .version_id(&held_version_id)
-        .send()
-        .await
-        .unwrap();
-    assert_eq!(
-        held_legal_hold
-            .legal_hold()
-            .and_then(|value| value.status())
-            .map(|value| value.as_str()),
-        Some("ON"),
-        "held version must keep its legal hold after multipart completion"
+    let error_str = format!("{:?}", complete_result.unwrap_err());
+    assert!(
+        error_str.to_lowercase().contains("legal") || error_str.to_lowercase().contains("hold"),
+        "complete error should mention legal hold, got: {error_str}"
     );
-
-    let delete_result = delete_object_with_bypass(&client, bucket, key, Some(&held_version_id), false).await;
-    assert!(delete_result.is_err(), "held version must stay delete-protected");
 }
 
 #[tokio::test]
 #[serial]
-async fn test_complete_multipart_upload_creates_new_version_under_compliance_retention() {
+async fn test_complete_multipart_upload_blocked_when_compliance_retention_added_after_create() {
     init_logging();
-    info!("🧪 Test: CompleteMultipartUpload creates a new version when the current version is under COMPLIANCE retention");
+    info!("🧪 Test: CompleteMultipartUpload blocked when COMPLIANCE retention appears after MPU creation");
 
     let mut env = ObjectLockTestEnvironment::new().await.unwrap();
     env.start_rustfs().await.unwrap();
@@ -1162,7 +1019,7 @@ async fn test_complete_multipart_upload_creates_new_version_under_compliance_ret
         .await
         .unwrap();
 
-    let retained_version_id = put_object_with_retention(
+    put_object_with_retention(
         &client,
         bucket,
         key,
@@ -1182,28 +1039,24 @@ async fn test_complete_multipart_upload_creates_new_version_under_compliance_ret
         )
         .build();
 
-    let complete_output = client
+    let complete_result = client
         .complete_multipart_upload()
         .bucket(bucket)
         .key(key)
         .upload_id(upload_id)
         .multipart_upload(completed_upload)
         .send()
-        .await
-        .expect("CompleteMultipartUpload should succeed by creating a new version over the retained one");
+        .await;
 
-    let new_version_id = complete_output
-        .version_id()
-        .expect("versioned multipart completion should return a version id")
-        .to_string();
-    assert_ne!(new_version_id, retained_version_id, "completion must create a distinct version");
-
-    // COMPLIANCE retention on the previous version survives the overwrite and
-    // cannot be bypassed.
-    let delete_result = delete_object_with_bypass(&client, bucket, key, Some(&retained_version_id), true).await;
     assert!(
-        delete_result.is_err(),
-        "retained version must stay delete-protected even with governance bypass"
+        complete_result.is_err(),
+        "CompleteMultipartUpload should fail once COMPLIANCE retention is enabled"
+    );
+
+    let error_str = format!("{:?}", complete_result.unwrap_err());
+    assert!(
+        error_str.to_lowercase().contains("retention") || error_str.to_lowercase().contains("compliance"),
+        "complete error should mention retention, got: {error_str}"
     );
 }
 
@@ -2011,7 +1864,7 @@ async fn test_copy_object_retention_uses_destination_policy() {
     assert!(no_default_retention.mode().is_none());
     assert!(no_default_retention.retain_until_date().is_none());
 
-    let retained_version_id = put_object_with_retention(
+    put_object_with_retention(
         &client,
         dst_bucket,
         "locked-destination",
@@ -2022,30 +1875,16 @@ async fn test_copy_object_retention_uses_destination_policy() {
     .await
     .unwrap();
 
-    let overwrite_output = client
+    let overwrite_result = client
         .copy_object()
         .copy_source(format!("{src_bucket}/{src_key}"))
         .bucket(dst_bucket)
         .key("locked-destination")
         .send()
-        .await
-        .expect("CopyObject overwrite should succeed by creating a new destination version");
-    let overwrite_version_id = overwrite_output
-        .version_id()
-        .expect("versioned copy should return a version id")
-        .to_string();
-    assert_ne!(
-        overwrite_version_id, retained_version_id,
-        "copy must create a distinct destination version"
-    );
-
-    // COMPLIANCE retention on the previous destination version survives the
-    // overwrite and cannot be bypassed.
-    let delete_result =
-        delete_object_with_bypass(&client, dst_bucket, "locked-destination", Some(&retained_version_id), true).await;
+        .await;
     assert!(
-        delete_result.is_err(),
-        "retained destination version must stay delete-protected even with governance bypass"
+        overwrite_result.is_err(),
+        "CopyObject overwrite should not bypass active destination retention"
     );
 }
 

crates/e2e_test/src/presigned_negative_test.rs

@@ -1,354 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Negative presigned-URL (query-string SigV4) regression suite (backlog#1151
-//! sec-2).
-//!
-//! Presigned URLs are the query-string SigV4 surface: the signature and its
-//! scope/expiry travel as `X-Amz-*` query parameters rather than in an
-//! `Authorization` header. Until now this repository only exercised presigned
-//! URLs on the *happy* path (e.g. `head_object_consistency_test`), so nothing
-//! pinned OUR end-to-end wiring of expiry enforcement or query-signature
-//! verification — a future dependency swap or misconfiguration could silently
-//! start honouring expired or forged presigned URLs. These tests send REJECTED
-//! presigned requests against a live server and assert the HTTP status plus the
-//! S3 error `<Code>` in the response XML, guarding the rejection contract
-//! regardless of who performs the underlying verification.
-//!
-//! This is the query-string sibling of `negative_sigv4_test` (sec-1, header
-//! SigV4); the two cover distinct attacker-controlled auth surfaces and share
-//! no test cases.
-//!
-//! Expiry is controlled WITHOUT real waiting: the AWS SDK presigner accepts an
-//! explicit `start_time`, so an already-expired URL is produced by signing with
-//! a timestamp far enough in the past that `start_time + X-Amz-Expires` is
-//! already behind the server clock. Forged variants are produced by presigning
-//! a valid URL and then mutating the query (`X-Amz-Signature`) or the signed
-//! target (object key) after the fact, and by presigning with the wrong secret.
-
-use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
-use aws_sdk_s3::config::{Credentials, Region};
-use aws_sdk_s3::presigning::{PresignedRequest, PresigningConfig};
-use aws_sdk_s3::primitives::ByteStream;
-use aws_sdk_s3::{Client, Config};
-use aws_smithy_http_client::Builder as SmithyHttpClientBuilder;
-use serial_test::serial;
-use std::time::{Duration, SystemTime};
-use tracing::info;
-
-const REGION: &str = "us-east-1";
-const BUCKET: &str = "presigned-negative-bucket";
-/// Object that `setup` stores; positive-control GETs read it back.
-const CANONICAL_KEY: &str = "canonical-object.txt";
-const CANONICAL_BODY: &[u8] = b"presigned-negative-canonical-body";
-
-/// Build an S3 client bound to this environment but with a caller-chosen secret
-/// key (mirrors `common::build_test_s3_config`, which is private). Used to
-/// presign with the WRONG secret while keeping the real access key id.
-fn s3_client_with_secret(env: &RustFSTestEnvironment, secret: &str) -> Client {
-    let credentials = Credentials::new(&env.access_key, secret, None, None, "e2e-presigned-negative");
-    let mut config = Config::builder()
-        .credentials_provider(credentials)
-        .region(Region::new(REGION))
-        .endpoint_url(&env.url)
-        .force_path_style(true)
-        .behavior_version_latest();
-    if env.url.starts_with("http://") {
-        config = config.http_client(SmithyHttpClientBuilder::new().build_http());
-    }
-    Client::from_conf(config.build())
-}
-
-/// A presigning config that is ALREADY expired the moment it is produced:
-/// signed as of one hour ago with a 60s validity window, so the server sees a
-/// request whose `X-Amz-Date + X-Amz-Expires` is ~59 minutes in the past. No
-/// real waiting, no flakiness.
-fn expired_config() -> PresigningConfig {
-    PresigningConfig::builder()
-        .start_time(SystemTime::now() - Duration::from_secs(3600))
-        .expires_in(Duration::from_secs(60))
-        .build()
-        .expect("valid presigning config")
-}
-
-/// A generous, valid presigning window for positive controls / pre-tamper URLs.
-fn valid_config() -> PresigningConfig {
-    PresigningConfig::expires_in(Duration::from_secs(300)).expect("valid presigning config")
-}
-
-/// Flip bytes inside the `X-Amz-Signature=` query value without changing its
-/// length, producing a structurally valid but incorrect signature.
-fn tamper_signature(uri: &str) -> String {
-    let marker = "X-Amz-Signature=";
-    let idx = uri.find(marker).expect("presigned uri must carry X-Amz-Signature") + marker.len();
-    let (head, rest) = uri.split_at(idx);
-    let end = rest.find('&').unwrap_or(rest.len());
-    let (sig, tail) = rest.split_at(end);
-    let tampered: String = sig
-        .chars()
-        .map(|c| match c {
-            '0' => 'f',
-            'a' => '0',
-            other => other,
-        })
-        .collect();
-    assert_ne!(sig, tampered, "tamper must actually change the signature hex");
-    format!("{head}{tampered}{tail}")
-}
-
-fn assert_error_code(body: &str, code: &str) {
-    assert!(
-        body.contains(&format!("<Code>{code}</Code>")),
-        "expected S3 error code <Code>{code}</Code> in response body, got:\n{body}"
-    );
-}
-
-/// Replay a `PresignedRequest` faithfully: same method, same URI, forward every
-/// signed header, attach an optional body. `reqwest` derives `Host` from the
-/// URI (matching the signed host).
-async fn send_presigned(pr: &PresignedRequest, body: Option<Vec<u8>>) -> reqwest::Result<reqwest::Response> {
-    send_raw(pr.method(), pr.uri(), pr.headers(), body).await
-}
-
-/// Replay against an ARBITRARY (possibly tampered) URI while keeping the signed
-/// method/headers of the original presigned request.
-async fn send_raw<'a>(
-    method: &str,
-    uri: &str,
-    headers: impl Iterator<Item = (&'a str, &'a str)>,
-    body: Option<Vec<u8>>,
-) -> reqwest::Result<reqwest::Response> {
-    let method = reqwest::Method::from_bytes(method.as_bytes()).expect("valid HTTP method");
-    let mut builder = local_http_client().request(method, uri);
-    for (k, v) in headers {
-        builder = builder.header(k, v);
-    }
-    if let Some(body) = body {
-        builder = builder.body(body);
-    }
-    builder.send().await
-}
-
-async fn setup(env: &mut RustFSTestEnvironment) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    env.start_rustfs_server(vec![]).await?;
-    env.create_test_bucket(BUCKET).await?;
-    env.create_s3_client()
-        .put_object()
-        .bucket(BUCKET)
-        .key(CANONICAL_KEY)
-        .body(ByteStream::from_static(CANONICAL_BODY))
-        .send()
-        .await?;
-    Ok(())
-}
-
-/// Positive control (GET): a valid presigned GET must succeed and return the
-/// stored bytes. Without this, every negative assertion could pass for the
-/// wrong reason (a server that rejects all presigned URLs).
-#[tokio::test]
-#[serial]
-async fn valid_presigned_get_succeeds() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-    let mut env = RustFSTestEnvironment::new().await?;
-    setup(&mut env).await?;
-
-    let pr = env
-        .create_s3_client()
-        .get_object()
-        .bucket(BUCKET)
-        .key(CANONICAL_KEY)
-        .presigned(valid_config())
-        .await?;
-
-    let resp = send_presigned(&pr, None).await?;
-    assert_eq!(resp.status().as_u16(), 200, "valid presigned GET should succeed");
-    let bytes = resp.bytes().await?;
-    assert_eq!(bytes.as_ref(), CANONICAL_BODY, "presigned GET body must match stored object");
-    info!("valid presigned GET control passed");
-    Ok(())
-}
-
-/// Positive control (PUT): a valid presigned PUT must store the object, which we
-/// verify with a follow-up authenticated HEAD.
-#[tokio::test]
-#[serial]
-async fn valid_presigned_put_succeeds() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-    let mut env = RustFSTestEnvironment::new().await?;
-    setup(&mut env).await?;
-
-    let key = "presigned-put-ok.txt";
-    let body = b"stored-via-presigned-put".to_vec();
-    let pr = env
-        .create_s3_client()
-        .put_object()
-        .bucket(BUCKET)
-        .key(key)
-        .presigned(valid_config())
-        .await?;
-
-    let resp = send_presigned(&pr, Some(body.clone())).await?;
-    assert!(resp.status().is_success(), "valid presigned PUT should succeed, got {}", resp.status());
-
-    let head = env.create_s3_client().head_object().bucket(BUCKET).key(key).send().await?;
-    assert_eq!(head.content_length(), Some(body.len() as i64), "stored object length must match");
-    info!("valid presigned PUT control passed");
-    Ok(())
-}
-
-/// (a) An already-expired presigned GET must be rejected with 403 / AccessDenied
-/// ("Request has expired"). s3s checks expiry BEFORE the signature, so the
-/// signature here is otherwise valid — only the elapsed window is at fault.
-#[tokio::test]
-#[serial]
-async fn expired_presigned_get_is_rejected() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-    let mut env = RustFSTestEnvironment::new().await?;
-    setup(&mut env).await?;
-
-    let pr = env
-        .create_s3_client()
-        .get_object()
-        .bucket(BUCKET)
-        .key(CANONICAL_KEY)
-        .presigned(expired_config())
-        .await?;
-
-    let resp = send_presigned(&pr, None).await?;
-    let status = resp.status();
-    let body = resp.text().await?;
-    assert_eq!(status.as_u16(), 403, "expired presigned GET must be 403, body:\n{body}");
-    assert_error_code(&body, "AccessDenied");
-    Ok(())
-}
-
-/// (b) Tampering the `X-Amz-Signature` query value must be rejected with 403 /
-/// SignatureDoesNotMatch.
-#[tokio::test]
-#[serial]
-async fn tampered_signature_returns_signature_does_not_match() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-    let mut env = RustFSTestEnvironment::new().await?;
-    setup(&mut env).await?;
-
-    let pr = env
-        .create_s3_client()
-        .get_object()
-        .bucket(BUCKET)
-        .key(CANONICAL_KEY)
-        .presigned(valid_config())
-        .await?;
-
-    let tampered_uri = tamper_signature(pr.uri());
-    let resp = send_raw(pr.method(), &tampered_uri, pr.headers(), None).await?;
-    let status = resp.status();
-    let body = resp.text().await?;
-    assert_eq!(status.as_u16(), 403, "tampered presigned signature must be 403, body:\n{body}");
-    assert_error_code(&body, "SignatureDoesNotMatch");
-    Ok(())
-}
-
-/// (c) A presigned URL generated with the WRONG secret (but the real access key
-/// id) must be rejected with 403 / SignatureDoesNotMatch.
-#[tokio::test]
-#[serial]
-async fn wrong_secret_key_returns_signature_does_not_match() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-    let mut env = RustFSTestEnvironment::new().await?;
-    setup(&mut env).await?;
-
-    let bad_client = s3_client_with_secret(&env, "totally-wrong-secret-key");
-    let pr = bad_client
-        .get_object()
-        .bucket(BUCKET)
-        .key(CANONICAL_KEY)
-        .presigned(valid_config())
-        .await?;
-
-    let resp = send_presigned(&pr, None).await?;
-    let status = resp.status();
-    let body = resp.text().await?;
-    assert_eq!(status.as_u16(), 403, "wrong-secret presigned URL must be 403, body:\n{body}");
-    assert_error_code(&body, "SignatureDoesNotMatch");
-    Ok(())
-}
-
-/// (d) Changing the signed target (the object key in the path) AFTER signing
-/// must be rejected with 403 / SignatureDoesNotMatch: the presented request no
-/// longer matches the canonical request the signature covers. The signature
-/// check runs during auth, before any object lookup, so the swapped key need
-/// not even exist.
-#[tokio::test]
-#[serial]
-async fn tampered_target_key_returns_signature_does_not_match() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-    let mut env = RustFSTestEnvironment::new().await?;
-    setup(&mut env).await?;
-
-    let signed_key = "signed-target.txt";
-    let served_key = "served-target.txt";
-    let pr = env
-        .create_s3_client()
-        .get_object()
-        .bucket(BUCKET)
-        .key(signed_key)
-        .presigned(valid_config())
-        .await?;
-
-    // Swap the object key in the path while leaving the (now stale) signature
-    // and its scope untouched.
-    let signed_segment = format!("/{signed_key}?");
-    let served_segment = format!("/{served_key}?");
-    let uri = pr.uri();
-    assert!(uri.contains(&signed_segment), "presigned uri must contain the signed key path: {uri}");
-    let tampered_uri = uri.replace(&signed_segment, &served_segment);
-
-    let resp = send_raw(pr.method(), &tampered_uri, pr.headers(), None).await?;
-    let status = resp.status();
-    let body = resp.text().await?;
-    assert_eq!(status.as_u16(), 403, "tampered target key must be 403, body:\n{body}");
-    assert_error_code(&body, "SignatureDoesNotMatch");
-    Ok(())
-}
-
-/// (e / acceptance 4 negative half) Tampering the signature of a presigned PUT
-/// must be rejected with 403 / SignatureDoesNotMatch — the write must not land.
-#[tokio::test]
-#[serial]
-async fn tampered_presigned_put_returns_signature_does_not_match() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-    let mut env = RustFSTestEnvironment::new().await?;
-    setup(&mut env).await?;
-
-    let key = "presigned-put-tampered.txt";
-    let pr = env
-        .create_s3_client()
-        .put_object()
-        .bucket(BUCKET)
-        .key(key)
-        .presigned(valid_config())
-        .await?;
-
-    let tampered_uri = tamper_signature(pr.uri());
-    let resp = send_raw(pr.method(), &tampered_uri, pr.headers(), Some(b"should-not-be-stored".to_vec())).await?;
-    let status = resp.status();
-    let body = resp.text().await?;
-    assert_eq!(status.as_u16(), 403, "tampered presigned PUT must be 403, body:\n{body}");
-    assert_error_code(&body, "SignatureDoesNotMatch");
-
-    // The rejected write must not have created the object.
-    let head = env.create_s3_client().head_object().bucket(BUCKET).key(key).send().await;
-    assert!(head.is_err(), "tampered presigned PUT must not store the object");
-    Ok(())
-}

crates/e2e_test/src/protocols/ftps_core.rs

@@ -13,23 +13,6 @@
 // limitations under the License.
 
 //! Core FTPS tests
-//!
-//! # Security regression coverage
-//!
-//! GHSA-3p3x-734c-h5vx (constant-time secret comparison on the WebDAV/FTPS
-//! password login path, fixed in rustfs/rustfs#4403) is anchored end-to-end
-//! by [`assert_ftps_ghsa_3p3x_wrong_credentials_rejected`], invoked from
-//! [`test_ftps_core_operations`]. It exercises the `ct_eq` rejection branch in
-//! `FtpsAuthenticator::authenticate` (`crates/protocols/src/ftps/server.rs`),
-//! proving that a wrong password and an unknown user are both rejected and are
-//! indistinguishable at the protocol layer (both return FTP `530 Not logged
-//! in`). The sibling WebDAV assertion lives in `webdav_core.rs`; the internode
-//! RPC fail-closed advisory (GHSA-r5qv-rc46-hv8q, rustfs/rustfs#4402) is
-//! anchored by the `ghsa_r5qv_*` unit tests in
-//! `crates/ecstore/src/cluster/rpc/http_auth.rs`. See
-//! `docs/testing/security-regressions.md` for the full advisory -> test map.
-//!
-//! Advisory: <https://github.com/rustfs/rustfs/security/advisories/GHSA-3p3x-734c-h5vx>
 
 use crate::common::rustfs_binary_path_with_features;
 use crate::protocols::test_env::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, ProtocolTestEnvironment};
@@ -41,8 +24,8 @@ use rustls::{ClientConfig, DigitallySignedStruct, Error as RustlsError, Signatur
 use std::io::Cursor;
 use std::path::PathBuf;
 use std::sync::Arc;
-use suppaftp::types::Response;
-use suppaftp::{FtpError, RustlsConnector, RustlsFtpStream, Status};
+use suppaftp::RustlsConnector;
+use suppaftp::RustlsFtpStream;
 use tokio::process::Command;
 use tracing::info;
 
@@ -90,78 +73,6 @@ impl ServerCertVerifier for AcceptAnyServerCertVerifier {
     }
 }
 
-/// Open a fresh, unauthenticated FTPS control connection to the test server
-/// and upgrade it to TLS. The caller is responsible for logging in.
-///
-/// Assumes the process-wide rustls crypto provider has already been installed
-/// (done once at the top of [`test_ftps_core_operations`]).
-fn ftps_connect_secure() -> Result<RustlsFtpStream> {
-    let config = ClientConfig::builder()
-        .dangerous()
-        .with_custom_certificate_verifier(Arc::new(AcceptAnyServerCertVerifier))
-        .with_no_client_auth();
-
-    let tls_connector = RustlsConnector::from(Arc::new(config));
-
-    let ftp_stream = RustlsFtpStream::connect(FTPS_ADDRESS).map_err(|e| anyhow::anyhow!("Failed to connect: {}", e))?;
-
-    ftp_stream
-        .into_secure(tls_connector, "127.0.0.1")
-        .map_err(|e| anyhow::anyhow!("Failed to upgrade to TLS: {}", e))
-}
-
-/// Extract the FTP reply status from a failed login, asserting the failure is a
-/// protocol-level rejection (`FtpError::UnexpectedResponse`) rather than a
-/// transport error.
-fn login_rejection_status(user: &str, password: &str) -> Result<Status> {
-    let mut ftp_stream = ftps_connect_secure()?;
-    match ftp_stream.login(user, password) {
-        Ok(()) => anyhow::bail!("login unexpectedly succeeded for user '{user}'"),
-        Err(FtpError::UnexpectedResponse(Response { status, .. })) => Ok(status),
-        Err(other) => anyhow::bail!("expected a protocol rejection, got transport error: {other}"),
-    }
-}
-
-/// Security regression for GHSA-3p3x-734c-h5vx.
-///
-/// FTPS password verification must reject invalid credentials via the
-/// constant-time `ct_eq` branch in `FtpsAuthenticator::authenticate`
-/// (`crates/protocols/src/ftps/server.rs`, fixed in rustfs/rustfs#4403). This
-/// asserts, purely on behavior (no timing assertions):
-///
-/// 1. A valid user with a wrong password is rejected (`530`), exercising the
-///    secret-mismatch branch that the advisory hardened.
-/// 2. An unknown user is rejected (`530`).
-/// 3. The two failures are indistinguishable at the protocol layer — both
-///    return the same FTP status, so an attacker cannot use the reply code to
-///    tell "user exists, wrong password" from "no such user".
-async fn assert_ftps_ghsa_3p3x_wrong_credentials_rejected() -> Result<()> {
-    info!("Testing FTPS (GHSA-3p3x): wrong password is rejected");
-    let wrong_password_status = login_rejection_status(DEFAULT_ACCESS_KEY, "definitely-not-the-secret")?;
-    assert_eq!(
-        wrong_password_status,
-        Status::NotLoggedIn,
-        "valid user + wrong password must be rejected with 530, got {wrong_password_status:?}"
-    );
-    info!("PASS: FTPS wrong password rejected with 530");
-
-    info!("Testing FTPS (GHSA-3p3x): unknown user is rejected");
-    let unknown_user_status = login_rejection_status("no-such-access-key", DEFAULT_SECRET_KEY)?;
-    assert_eq!(
-        unknown_user_status,
-        Status::NotLoggedIn,
-        "unknown user must be rejected with 530, got {unknown_user_status:?}"
-    );
-    info!("PASS: FTPS unknown user rejected with 530");
-
-    assert_eq!(
-        wrong_password_status, unknown_user_status,
-        "invalid-user and invalid-password failures must be indistinguishable at the protocol layer"
-    );
-    info!("PASS: FTPS invalid-user and invalid-password failures are indistinguishable");
-    Ok(())
-}
-
 /// Test FTPS: put, ls, mkdir, rmdir, delete operations
 pub async fn test_ftps_core_operations() -> Result<()> {
     let env = ProtocolTestEnvironment::new().map_err(|e| anyhow::anyhow!("{}", e))?;
@@ -191,7 +102,6 @@ pub async fn test_ftps_core_operations() -> Result<()> {
     info!("Starting FTPS server on {}", FTPS_ADDRESS);
     let binary_path = rustfs_binary_path_with_features(Some("ftps,webdav"));
     let mut server_process = Command::new(&binary_path)
-        .env("RUSTFS_CONSOLE_ENABLE", "false")
         .env("RUSTFS_FTPS_ENABLE", "true")
         .env("RUSTFS_FTPS_ADDRESS", FTPS_ADDRESS)
         .env("RUSTFS_FTPS_CERTS_DIR", cert_dir.to_str().unwrap())
@@ -205,18 +115,26 @@ pub async fn test_ftps_core_operations() -> Result<()> {
             .await
             .map_err(|e| anyhow::anyhow!("{}", e))?;
 
-        // Install the default crypto provider once for this process before any
-        // TLS handshake. Subsequent connections reuse it via `ftps_connect_secure`.
+        // Build ServerConfig with SNI support
         rustls::crypto::aws_lc_rs::default_provider()
             .install_default()
             .map_err(|e| anyhow::anyhow!("Failed to install crypto provider: {:?}", e))?;
 
-        // Security regression (GHSA-3p3x-734c-h5vx): wrong credentials must be
-        // rejected before any successful login is attempted below.
-        assert_ftps_ghsa_3p3x_wrong_credentials_rejected().await?;
+        let config = ClientConfig::builder()
+            .dangerous()
+            .with_custom_certificate_verifier(Arc::new(AcceptAnyServerCertVerifier))
+            .with_no_client_auth();
 
-        // Connect and log in with valid credentials for the functional flow.
-        let mut ftp_stream = ftps_connect_secure()?;
+        // Wrap in suppaftp's RustlsConnector
+        let tls_connector = RustlsConnector::from(Arc::new(config));
+
+        // Connect to FTPS server
+        let ftp_stream = RustlsFtpStream::connect(FTPS_ADDRESS).map_err(|e| anyhow::anyhow!("Failed to connect: {}", e))?;
+
+        // Upgrade to secure connection
+        let mut ftp_stream = ftp_stream
+            .into_secure(tls_connector, "127.0.0.1")
+            .map_err(|e| anyhow::anyhow!("Failed to upgrade to TLS: {}", e))?;
         ftp_stream.login(DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY)?;
 
         info!("Testing FTPS: mkdir bucket");

crates/e2e_test/src/protocols/sftp_compliance_tests.rs

@@ -192,7 +192,6 @@ pub(crate) async fn spawn_compliance_rustfs(
         .to_str()
         .ok_or_else(|| anyhow!("host key dir path is not utf-8: {}", host_key_dir.display()))?;
     let child = Command::new(&binary_path)
-        .env(ENV_CONSOLE_ENABLE, "false")
         .env(ENV_SFTP_ENABLE, "true")
         .env(ENV_SFTP_ADDRESS, sftp_address)
         .env(ENV_SFTP_HOST_KEY_DIR, host_key_dir_str)

crates/e2e_test/src/protocols/sftp_core.rs

@@ -27,8 +27,8 @@ use russh::client::{self, Handle};
 use russh_sftp::client::SftpSession;
 use russh_sftp::protocol::{FileAttributes, OpenFlags};
 use rustfs_config::{
-    ENV_CONSOLE_ENABLE, ENV_RUSTFS_ADDRESS, ENV_SFTP_ADDRESS, ENV_SFTP_ENABLE, ENV_SFTP_HOST_KEY_DIR, ENV_SFTP_IDLE_TIMEOUT,
-    ENV_SFTP_PART_SIZE, ENV_SFTP_READ_ONLY,
+    ENV_RUSTFS_ADDRESS, ENV_SFTP_ADDRESS, ENV_SFTP_ENABLE, ENV_SFTP_HOST_KEY_DIR, ENV_SFTP_IDLE_TIMEOUT, ENV_SFTP_PART_SIZE,
+    ENV_SFTP_READ_ONLY,
 };
 use sha2::{Digest, Sha256};
 use std::path::PathBuf;
@@ -152,7 +152,6 @@ pub async fn test_sftp_core_operations() -> Result<()> {
         .ok_or_else(|| anyhow!("host key dir path is not utf-8"))?;
     let mut server_process = ServerProcess::new(
         Command::new(&binary_path)
-            .env(ENV_CONSOLE_ENABLE, "false")
             .env(ENV_SFTP_ENABLE, "true")
             .env(ENV_SFTP_ADDRESS, SFTP_ADDRESS)
             .env(ENV_SFTP_HOST_KEY_DIR, host_key_dir_str)
@@ -505,7 +504,6 @@ pub async fn test_sftp_idle_timeout_disconnects() -> Result<()> {
         .ok_or_else(|| anyhow!("host key dir path is not utf-8"))?;
     let mut server_process = ServerProcess::new(
         Command::new(&binary_path)
-            .env(ENV_CONSOLE_ENABLE, "false")
             .env(ENV_SFTP_ENABLE, "true")
             .env(ENV_SFTP_ADDRESS, IDLE_SFTP_ADDRESS)
             .env(ENV_SFTP_HOST_KEY_DIR, host_key_dir_str)

crates/e2e_test/src/protocols/webdav_core.rs

@@ -13,23 +13,6 @@
 // limitations under the License.
 
 //! Core WebDAV tests
-//!
-//! # Security regression coverage
-//!
-//! GHSA-3p3x-734c-h5vx (constant-time secret comparison on the WebDAV/FTPS
-//! password login path, fixed in rustfs/rustfs#4403) is anchored end-to-end by
-//! the authentication-failure block in [`test_webdav_core_operations`], marked
-//! with a `GHSA-3p3x` comment. It drives the `ct_eq` rejection branch in
-//! `WebDavAuth::authenticate` (`crates/protocols/src/webdav/server.rs`): a
-//! valid access key with a wrong secret is rejected (the exact branch the
-//! advisory hardened), and an unknown user is rejected with the same status so
-//! the two failures are indistinguishable. The sibling FTPS assertion lives in
-//! `ftps_core.rs`; the internode RPC fail-closed advisory
-//! (GHSA-r5qv-rc46-hv8q, rustfs/rustfs#4402) is anchored by the `ghsa_r5qv_*`
-//! unit tests in `crates/ecstore/src/cluster/rpc/http_auth.rs`. See
-//! `docs/testing/security-regressions.md` for the full advisory -> test map.
-//!
-//! Advisory: <https://github.com/rustfs/rustfs/security/advisories/GHSA-3p3x-734c-h5vx>
 
 use crate::common::local_http_client;
 use crate::common::rustfs_binary_path_with_features;
@@ -168,7 +151,6 @@ pub async fn test_webdav_core_operations() -> Result<()> {
     let mut server_process = Command::new(&binary_path)
         .arg("--address")
         .arg(S3_TEST_ADDRESS)
-        .env("RUSTFS_CONSOLE_ENABLE", "false")
         .env("RUSTFS_WEBDAV_ENABLE", "true")
         .env("RUSTFS_WEBDAV_ADDRESS", WEBDAV_ADDRESS)
         .env("RUSTFS_WEBDAV_TLS_ENABLED", "false") // No TLS for testing
@@ -670,38 +652,15 @@ pub async fn test_webdav_core_operations() -> Result<()> {
         );
         info!("PASS: DELETE bucket '{}' successful", bucket_name);
 
-        // Security regression (GHSA-3p3x-734c-h5vx): constant-time secret
-        // comparison on the WebDAV password login path (fixed in
-        // rustfs/rustfs#4403). A valid access key with a wrong secret must be
-        // rejected via the `ct_eq` branch in `WebDavAuth::authenticate`, and an
-        // unknown user must be rejected with the same status so the two are
-        // indistinguishable. Behavior-only assertion (no timing checks).
-        info!("Testing WebDAV (GHSA-3p3x): valid access key + wrong secret is rejected");
-        let wrong_secret_resp = client
+        // Test authentication failure
+        info!("Testing WebDAV: Authentication failure");
+        let resp = client
             .request(reqwest::Method::from_bytes(b"PROPFIND").unwrap(), &base_url)
-            .header("Authorization", basic_auth_header_for(DEFAULT_ACCESS_KEY, "definitely-not-the-secret"))
+            .header("Authorization", "Basic aW52YWxpZDppbnZhbGlk") // invalid:invalid
             .send()
             .await?;
-        let wrong_secret_status = wrong_secret_resp.status().as_u16();
-        assert_eq!(
-            wrong_secret_status, 401,
-            "valid access key + wrong secret should return 401, got: {wrong_secret_status}"
-        );
-        info!("PASS: WebDAV wrong secret rejected with 401");
-
-        info!("Testing WebDAV (GHSA-3p3x): unknown user is rejected");
-        let unknown_user_resp = client
-            .request(reqwest::Method::from_bytes(b"PROPFIND").unwrap(), &base_url)
-            .header("Authorization", basic_auth_header_for("no-such-access-key", "definitely-not-the-secret"))
-            .send()
-            .await?;
-        let unknown_user_status = unknown_user_resp.status().as_u16();
-        assert_eq!(unknown_user_status, 401, "unknown user should return 401, got: {unknown_user_status}");
-        assert_eq!(
-            wrong_secret_status, unknown_user_status,
-            "invalid-user and invalid-secret failures must be indistinguishable at the protocol layer"
-        );
-        info!("PASS: WebDAV authentication failure (GHSA-3p3x) test successful");
+        assert_eq!(resp.status().as_u16(), 401, "Invalid auth should return 401, got: {}", resp.status());
+        info!("PASS: Authentication failure test successful");
 
         info!("WebDAV core tests passed");
         Ok(())

crates/e2e_test/src/reliant/lifecycle.rs

@@ -13,302 +13,175 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-//! Self-managed lifecycle *expiry* end-to-end tests (backlog#1148 ilm-3).
-//!
-//! Each test spawns its own `rustfs` binary on a random port with an isolated
-//! temp dir via [`RustFSTestEnvironment`], so there is no dependency on a
-//! pre-started `localhost:9000` server and no `#[ignore]`. Expiry is driven to
-//! completion in seconds using two independent, per-test time-control tools:
-//!
-//! * **`mod_time` back-dating** (see [`put_object_with_backdated_mtime`]): write
-//!   an object whose stored `mod_time` is already in the past, so a `Days`-based
-//!   rule is due immediately without accelerating the day length. Used by
-//!   [`test_lifecycle_expiry_backdated_mtime`].
-//! * **`RUSTFS_ILM_DEBUG_DAY_SECS`** (ilm-5): compress one lifecycle "day" to a
-//!   couple of seconds so a `Days=1` rule fires shortly after a normal PUT. Used
-//!   by [`test_lifecycle_versioned_current_version_expiry_creates_delete_marker`].
-//!
-//! In every case the scanner must actually *run* for expiry to apply, so each
-//! server is started with `RUSTFS_SCANNER_CYCLE=1` (1-second scan cycle) and a
-//! small `RUSTFS_ILM_PROCESS_TIME` rounding boundary. Tests poll for the
-//! terminal state instead of sleeping a fixed wall-clock interval.
-
-use crate::common::RustFSTestEnvironment;
+use aws_config::meta::region::RegionProviderChain;
 use aws_sdk_s3::Client;
-use aws_sdk_s3::primitives::ByteStream;
-use aws_sdk_s3::types::{
-    BucketLifecycleConfiguration, BucketVersioningStatus, ExpirationStatus, LifecycleExpiration, LifecycleRule,
-    LifecycleRuleFilter, VersioningConfiguration,
-};
-use std::time::Duration as StdDuration;
-use time::OffsetDateTime;
-use time::format_description::well_known::Rfc3339;
+use aws_sdk_s3::config::{Credentials, Region};
+use bytes::Bytes;
+use serial_test::serial;
+use std::error::Error;
 
-type TestResult = Result<(), Box<dyn std::error::Error + Send + Sync>>;
+const ENDPOINT: &str = "http://localhost:9000";
+const ACCESS_KEY: &str = "rustfsadmin";
+const SECRET_KEY: &str = "rustfsadmin";
+const BUCKET: &str = "test-basic-bucket";
 
-/// Internal replication headers that back-date a written object's `mod_time`.
-///
-/// These are the RustFS side of the MinIO-compatible replication protocol
-/// (`crates/utils/src/http/header_compat.rs`; consumed in
-/// `rustfs/src/storage/options.rs::put_opts_from_headers`). Sending
-/// `x-rustfs-source-replication-request: true` routes the PUT through the
-/// replica branch and, when `x-rustfs-source-mtime` (RFC3339) is present, forces
-/// the stored `mod_time` to that value.
-const HDR_SOURCE_REPLICATION_REQUEST: &str = "x-rustfs-source-replication-request";
-const HDR_SOURCE_MTIME: &str = "x-rustfs-source-mtime";
+async fn create_aws_s3_client() -> Result<Client, Box<dyn Error>> {
+    let region_provider = RegionProviderChain::default_provider().or_else(Region::new("us-east-1"));
+    let shared_config = aws_config::defaults(aws_config::BehaviorVersion::latest())
+        .region(region_provider)
+        .credentials_provider(Credentials::new(ACCESS_KEY, SECRET_KEY, None, None, "static"))
+        .endpoint_url(ENDPOINT)
+        .load()
+        .await;
 
-/// PUT an object whose stored `mod_time` is back-dated to `mtime`.
-///
-/// # Side effects / caveats
-///
-/// This uses the internal source-replication backdoor, so the write sets
-/// `ObjectOptions::replication_request = true` and takes the replica code path.
-/// That flag by itself does **not** set the object's replication *status* to
-/// `Pending`/`Failed` — only an `x-amz-bucket-replication-status: replica`
-/// header would set `REPLICA` status, and the test bucket has no replication
-/// configuration — so lifecycle expiry is **not** gated
-/// (`crates/lifecycle/src/evaluator.rs::replication_status_blocks_lifecycle`
-/// only blocks on `Pending`/`Failed`). The passing
-/// [`test_lifecycle_expiry_backdated_mtime`] is the end-to-end proof that a
-/// back-dated object is still expired by the scanner.
-async fn put_object_with_backdated_mtime(
-    client: &Client,
-    bucket: &str,
-    key: &str,
-    body: &[u8],
-    mtime: OffsetDateTime,
-) -> TestResult {
-    let mtime_rfc3339 = mtime.format(&Rfc3339)?;
-    client
-        .put_object()
-        .bucket(bucket)
-        .key(key)
-        .body(ByteStream::from(body.to_vec()))
-        .customize()
-        .mutate_request(move |req| {
-            req.headers_mut().insert(HDR_SOURCE_REPLICATION_REQUEST, "true");
-            req.headers_mut().insert(HDR_SOURCE_MTIME, mtime_rfc3339.clone());
-        })
-        .send()
-        .await?;
-    Ok(())
+    let client = Client::from_conf(
+        aws_sdk_s3::Config::from(&shared_config)
+            .to_builder()
+            .force_path_style(true)
+            .build(),
+    );
+    Ok(client)
 }
 
-/// Returns `true` once `GET bucket/key` fails with `NoSuchKey`, `false` while it
-/// still succeeds. Any other error is surfaced.
-async fn object_is_gone(client: &Client, bucket: &str, key: &str) -> Result<bool, Box<dyn std::error::Error + Send + Sync>> {
-    match client.get_object().bucket(bucket).key(key).send().await {
-        Ok(_) => Ok(false),
+async fn setup_test_bucket(client: &Client) -> Result<(), Box<dyn Error>> {
+    match client.create_bucket().bucket(BUCKET).send().await {
+        Ok(_) => {}
         Err(e) => {
-            if let Some(service_error) = e.as_service_error() {
-                if service_error.is_no_such_key() {
-                    return Ok(true);
-                }
-                return Err(format!("expected NoSuchKey, got: {e:?}").into());
+            let error_str = e.to_string();
+            if !error_str.contains("BucketAlreadyOwnedByYou") && !error_str.contains("BucketAlreadyExists") {
+                return Err(e.into());
             }
-            Err(format!("expected a service error, got: {e:?}").into())
         }
     }
-}
-
-/// Poll until `GET bucket/key` returns `NoSuchKey`, or fail after `deadline`.
-///
-/// The scanner cycle is 1s under test, so expiry normally lands within a few
-/// seconds; the deadline is a generous safety net, not the expected wall-clock.
-async fn wait_for_object_expired(client: &Client, bucket: &str, key: &str, deadline: StdDuration) -> TestResult {
-    let start = std::time::Instant::now();
-    loop {
-        if object_is_gone(client, bucket, key).await? {
-            return Ok(());
-        }
-        if start.elapsed() >= deadline {
-            return Err(format!(
-                "object {bucket}/{key} was not expired by the lifecycle scanner within {}s",
-                deadline.as_secs()
-            )
-            .into());
-        }
-        tokio::time::sleep(StdDuration::from_millis(500)).await;
-    }
-}
-
-/// Build a prefix-scoped `Days`-based expiration rule.
-fn expiration_rule(id: &str, prefix: &str, days: i32) -> Result<LifecycleRule, Box<dyn std::error::Error + Send + Sync>> {
-    let rule = LifecycleRule::builder()
-        .id(id)
-        .filter(LifecycleRuleFilter::builder().prefix(prefix).build())
-        .expiration(LifecycleExpiration::builder().days(days).build())
-        .status(ExpirationStatus::Enabled)
-        .build()?;
-    Ok(rule)
-}
-
-async fn put_expiration_config(client: &Client, bucket: &str, rule: LifecycleRule) -> TestResult {
-    let lifecycle = BucketLifecycleConfiguration::builder().rules(rule).build()?;
-    client
-        .put_bucket_lifecycle_configuration()
-        .bucket(bucket)
-        .lifecycle_configuration(lifecycle)
-        .send()
-        .await?;
     Ok(())
 }
 
-/// Env applied to every server in this module: run the scanner every second and
-/// shrink the ILM processing/rounding boundary so `Days`-based deadlines are not
-/// rounded up to the next real day.
-fn fast_lifecycle_env() -> Vec<(&'static str, &'static str)> {
-    vec![("RUSTFS_SCANNER_CYCLE", "1"), ("RUSTFS_ILM_PROCESS_TIME", "1")]
-}
-
-/// `Days=1` expiry driven purely by `mod_time` back-dating (no day-length
-/// acceleration). Proves:
-/// * a matching-prefix object whose `mod_time` is 25h old is expired (the rule's
-///   1-day deadline is already in the past);
-/// * the `put_object_with_backdated_mtime` backdoor does not trip the
-///   replication gate that would otherwise block expiry;
-/// * a non-matching-prefix object with the **same** back-dated `mod_time`
-///   survives, isolating the prefix filter (not recency) as the cause.
 #[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-async fn test_lifecycle_expiry_backdated_mtime() -> TestResult {
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server_with_env(vec![], &fast_lifecycle_env()).await?;
+#[serial]
+#[ignore = "requires running RustFS server at localhost:9000"]
+async fn test_bucket_lifecycle_configuration() -> Result<(), Box<dyn std::error::Error>> {
+    use aws_sdk_s3::types::{BucketLifecycleConfiguration, LifecycleExpiration, LifecycleRule, LifecycleRuleFilter};
+    use chrono::{Duration as ChronoDuration, Utc};
+    use tokio::time::Duration;
 
-    let client = env.create_s3_client();
-    let bucket = "ilm3-backdated";
-    client.create_bucket().bucket(bucket).send().await?;
+    let client = create_aws_s3_client().await?;
+    setup_test_bucket(&client).await?;
 
-    let matched_key = "expire/object.txt";
-    let survivor_key = "keep/object.txt";
-    // 25h in the past: older than the 1-day rule with a normal 86400s day length.
-    let backdated = OffsetDateTime::now_utc() - time::Duration::hours(25);
-
-    put_object_with_backdated_mtime(&client, bucket, matched_key, b"expire me", backdated).await?;
-    put_object_with_backdated_mtime(&client, bucket, survivor_key, b"keep me", backdated).await?;
-
-    // Both objects exist before the lifecycle rule is installed.
-    assert!(!object_is_gone(&client, bucket, matched_key).await?);
-    assert!(!object_is_gone(&client, bucket, survivor_key).await?);
-
-    put_expiration_config(&client, bucket, expiration_rule("expire-backdated", "expire/", 1)?).await?;
-
-    // Matching, back-dated object is expired by the scanner.
-    wait_for_object_expired(&client, bucket, matched_key, StdDuration::from_secs(90)).await?;
-
-    // Negative control: identical back-dating, non-matching prefix -> survives.
-    assert!(
-        !object_is_gone(&client, bucket, survivor_key).await?,
-        "non-matching-prefix object must not be expired by a prefix-scoped rule"
-    );
-
-    Ok(())
-}
-
-/// `Days=1` current-version expiry on a **versioned** bucket, accelerated with
-/// `RUSTFS_ILM_DEBUG_DAY_SECS`. Proves that current-version expiry inserts a
-/// delete marker (rather than permanently removing the object): after expiry a
-/// plain `GET` returns `NoSuchKey`, `ListObjectVersions` shows a latest delete
-/// marker, and the original data version is retained.
-#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-async fn test_lifecycle_versioned_current_version_expiry_creates_delete_marker() -> TestResult {
-    let mut env = RustFSTestEnvironment::new().await?;
-    // One lifecycle "day" == 2s, plus the fast scanner/rounding env.
-    let mut extra_env = fast_lifecycle_env();
-    extra_env.push(("RUSTFS_ILM_DEBUG_DAY_SECS", "2"));
-    env.start_rustfs_server_with_env(vec![], &extra_env).await?;
-
-    let client = env.create_s3_client();
-    let bucket = "ilm3-versioned";
-    client.create_bucket().bucket(bucket).send().await?;
+    // Upload test object first
+    let test_content = "Test object for lifecycle expiration";
+    let lifecycle_object_key = "lifecycle-test-object.txt";
+    let untouched_object_key = "keep-object.txt";
     client
-        .put_bucket_versioning()
-        .bucket(bucket)
-        .versioning_configuration(
-            VersioningConfiguration::builder()
-                .status(BucketVersioningStatus::Enabled)
-                .build(),
-        )
-        .send()
-        .await?;
-
-    let key = "versioned/object.txt";
-    let put = client
         .put_object()
-        .bucket(bucket)
-        .key(key)
-        .body(ByteStream::from_static(b"versioned payload"))
+        .bucket(BUCKET)
+        .key(lifecycle_object_key)
+        .body(Bytes::from(test_content.as_bytes()).into())
         .send()
         .await?;
-    let data_version_id = put
-        .version_id()
-        .map(str::to_string)
-        .expect("versioned PUT returns a version id");
-
-    put_expiration_config(&client, bucket, expiration_rule("expire-versioned", "versioned/", 1)?).await?;
-
-    // Current version expiry adds a delete marker; a plain GET now 404s.
-    wait_for_object_expired(&client, bucket, key, StdDuration::from_secs(90)).await?;
-
-    // ListObjectVersions: original data version retained, latest is a delete marker.
-    let versions = client.list_object_versions().bucket(bucket).prefix(key).send().await?;
-
-    let data_versions = versions.versions();
-    assert!(
-        data_versions.iter().any(|v| v.version_id() == Some(data_version_id.as_str())),
-        "original data version {data_version_id} must be retained, got: {data_versions:?}"
-    );
-
-    let delete_markers = versions.delete_markers();
-    assert!(
-        delete_markers.iter().any(|m| m.is_latest() == Some(true)),
-        "expiry on a versioned bucket must create a latest delete marker, got: {delete_markers:?}"
-    );
-
-    // The retained data version is still directly readable by version id.
-    let by_version = client
-        .get_object()
-        .bucket(bucket)
-        .key(key)
-        .version_id(&data_version_id)
+    client
+        .put_object()
+        .bucket(BUCKET)
+        .key(untouched_object_key)
+        .body(Bytes::from("should-stay".as_bytes()).into())
         .send()
         .await?;
-    assert_eq!(by_version.body.collect().await?.into_bytes().as_ref(), b"versioned payload");
 
-    Ok(())
-}
+    // Verify object exists initially
+    let resp = client.get_object().bucket(BUCKET).key(lifecycle_object_key).send().await?;
+    assert!(resp.content_length().unwrap_or(0) > 0);
+    let untouched_resp = client.get_object().bucket(BUCKET).key(untouched_object_key).send().await?;
+    assert!(untouched_resp.content_length().unwrap_or(0) > 0);
 
-/// `Days=0` expiration is invalid per S3 semantics (`Days` must be a positive
-/// integer >= 1). A `PutBucketLifecycleConfiguration` carrying a zero-day rule
-/// must be rejected with `InvalidArgument` (HTTP 400) — see crates/lifecycle
-/// `validate()` and the PutBucketLifecycleConfiguration handler. This is the
-/// self-managed counterpart of the localhost-only
-/// `test_bucket_lifecycle_rejects_zero_days` unit test.
-#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
-async fn test_lifecycle_rejects_zero_day_rule() -> TestResult {
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server_with_env(vec![], &fast_lifecycle_env()).await?;
-
-    let client = env.create_s3_client();
-    let bucket = "ilm3-zero-day";
-    client.create_bucket().bucket(bucket).send().await?;
-
-    let lifecycle = BucketLifecycleConfiguration::builder()
-        .rules(expiration_rule("expire-zero-days", "zero-days/", 0)?)
+    // Use a past midnight UTC date to trigger immediate lifecycle expiry without requiring days=0.
+    let yesterday_midnight_utc = Utc::now()
+        .date_naive()
+        .and_hms_opt(0, 0, 0)
+        .expect("midnight should always be valid")
+        - ChronoDuration::days(1);
+    let expiration = LifecycleExpiration::builder()
+        .date(aws_sdk_s3::primitives::DateTime::from_secs(yesterday_midnight_utc.and_utc().timestamp()))
+        .build();
+    let filter = LifecycleRuleFilter::builder().prefix(lifecycle_object_key).build();
+    let rule = LifecycleRule::builder()
+        .id("expire-test-object")
+        .filter(filter)
+        .expiration(expiration)
+        .status(aws_sdk_s3::types::ExpirationStatus::Enabled)
         .build()?;
-    let err = client
+    let lifecycle = BucketLifecycleConfiguration::builder().rules(rule).build()?;
+
+    client
         .put_bucket_lifecycle_configuration()
-        .bucket(bucket)
+        .bucket(BUCKET)
         .lifecycle_configuration(lifecycle)
         .send()
-        .await
-        .expect_err("zero-day expiration lifecycle rule should be rejected");
+        .await?;
 
-    let service_error = err.as_service_error().expect("expected an S3 service error");
-    assert_eq!(
-        service_error.meta().code(),
-        Some("InvalidArgument"),
-        "expected InvalidArgument for zero-day expiration, got: {err:?}"
-    );
+    // Verify lifecycle configuration was set
+    let resp = client.get_bucket_lifecycle_configuration().bucket(BUCKET).send().await?;
+    let rules = resp.rules();
+    assert!(rules.iter().any(|r| r.id().unwrap_or("") == "expire-test-object"));
+
+    // Poll for deletion instead of using a fixed sleep to keep the test deterministic.
+    // Default scanner cycle interval is 60s with jitter, so allow enough time for one full cycle.
+    let deadline = tokio::time::Instant::now() + Duration::from_secs(150);
+    loop {
+        let get_result = client.get_object().bucket(BUCKET).key(lifecycle_object_key).send().await;
+        match get_result {
+            Ok(_) => {
+                if tokio::time::Instant::now() >= deadline {
+                    panic!("Expected object to be deleted by lifecycle rule within 150s, but it still exists");
+                }
+                tokio::time::sleep(Duration::from_secs(1)).await;
+            }
+            Err(e) => {
+                if let Some(service_error) = e.as_service_error() {
+                    if service_error.is_no_such_key() {
+                        println!("Lifecycle configuration test completed - object was successfully deleted by lifecycle rule");
+                        break;
+                    }
+                    panic!("Expected NoSuchKey error, but got: {e:?}");
+                } else {
+                    panic!("Expected service error, but got: {e:?}");
+                }
+            }
+        }
+    }
+
+    println!("Lifecycle configuration test completed.");
+
+    // Non-matching prefix object should remain available.
+    let untouched_after = client.get_object().bucket(BUCKET).key(untouched_object_key).send().await?;
+    assert!(untouched_after.content_length().unwrap_or(0) > 0);
+
+    Ok(())
+}
+
+#[tokio::test(flavor = "multi_thread", worker_threads = 2)]
+#[serial]
+#[ignore = "requires running RustFS server at localhost:9000"]
+async fn test_bucket_lifecycle_accepts_zero_days() -> Result<(), Box<dyn std::error::Error>> {
+    use aws_sdk_s3::types::{BucketLifecycleConfiguration, LifecycleExpiration, LifecycleRule, LifecycleRuleFilter};
+
+    let client = create_aws_s3_client().await?;
+    setup_test_bucket(&client).await?;
+
+    let expiration = LifecycleExpiration::builder().days(0).build();
+    let filter = LifecycleRuleFilter::builder().prefix("zero-days/").build();
+    let rule = LifecycleRule::builder()
+        .id("expire-zero-days")
+        .filter(filter)
+        .expiration(expiration)
+        .status(aws_sdk_s3::types::ExpirationStatus::Enabled)
+        .build()?;
+    let lifecycle = BucketLifecycleConfiguration::builder().rules(rule).build()?;
+
+    client
+        .put_bucket_lifecycle_configuration()
+        .bucket(BUCKET)
+        .lifecycle_configuration(lifecycle)
+        .send()
+        .await?;
 
     Ok(())
 }

crates/e2e_test/src/server_startup_failfast_test.rs

@@ -1,60 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Regression tests for e2e harness server-startup behavior.
-//!
-//! The harness spawns servers with the embedded console disabled so they never
-//! contend for its fixed default port :9001 (often held by unrelated local
-//! services such as Docker Desktop), and `wait_for_server_ready` must report a
-//! server that dies during startup immediately instead of polling out the full
-//! readiness timeout.
-
-#[cfg(test)]
-mod tests {
-    use crate::common::{RustFSTestEnvironment, init_logging, rustfs_binary_path};
-    use serial_test::serial;
-    use std::net::TcpListener;
-    use std::time::{Duration, Instant};
-
-    /// Force the console back on (extra_env overrides the harness default)
-    /// while :9001 is occupied: the server exits at startup, and the harness
-    /// must surface that promptly rather than waiting out the 60s timeout.
-    #[tokio::test]
-    #[serial]
-    async fn test_start_fails_fast_when_server_exits_during_startup() {
-        init_logging();
-
-        // Occupy :9001 on both stacks; a failed bind means another local
-        // process already holds the port, which serves equally well.
-        let _guard_v6 = TcpListener::bind(("::", 9001)).ok();
-        let _guard_v4 = TcpListener::bind(("0.0.0.0", 9001)).ok();
-
-        // Resolve (and possibly build) the binary before starting the timer.
-        let _ = rustfs_binary_path();
-
-        let mut env = RustFSTestEnvironment::new().await.expect("Failed to create test environment");
-        let started = Instant::now();
-        let result = env
-            .start_rustfs_server_with_env(vec![], &[("RUSTFS_CONSOLE_ENABLE", "true")])
-            .await;
-        let elapsed = started.elapsed();
-
-        let err = result.expect_err("server start should fail while :9001 is occupied");
-        assert!(err.to_string().contains("exited before becoming ready"), "unexpected start error: {err}");
-        assert!(
-            elapsed < Duration::from_secs(30),
-            "startup failure should be detected fast, took {elapsed:?}"
-        );
-    }
-}

crates/ecstore/Cargo.toml

@@ -34,10 +34,6 @@ workspace = true
 default = []
 rio-v2 = ["dep:rustfs-rio-v2"]
 hotpath = ["dep:hotpath", "hotpath/hotpath", "rustfs-filemeta/hotpath", "rustfs-rio/hotpath"]
-# Exposes shared lifecycle/tier test utilities (MockWarmBackend, fault
-# injection, xl.meta transition assertions) via `api::tier::test_util`.
-# Enable only from `[dev-dependencies]` (rustfs/backlog#1148 ilm-6).
-test-util = []
 
 [dependencies]
 hotpath = { workspace = true, optional = true }
@@ -118,9 +114,7 @@ pin-project-lite.workspace = true
 md-5.workspace = true
 memmap2 = { workspace = true }
 libc.workspace = true
-# "process" adds getrlimit for the io_uring fd-cache RLIMIT_NOFILE gate
-# (backlog#1178); "fs" comes from the workspace default.
-rustix = { workspace = true, features = ["process"] }
+rustix = { workspace = true }
 rustfs-madmin.workspace = true
 reqwest = { workspace = true }
 aes-gcm.workspace = true
@@ -146,12 +140,6 @@ aws-smithy-http-client.workspace = true
 # Observability and Metrics
 metrics = { workspace = true }
 
-# Runtime-probed io_uring read backend (backlog#1104). Linux-only, from
-# crates.io. The guard scripts/check_no_tokio_io_uring.sh allows an explicit
-# io-uring integration; only the tokio "io-uring" runtime feature is banned.
-[target.'cfg(target_os = "linux")'.dependencies]
-rustfs-uring = "0.2.1"
-
 [dev-dependencies]
 tokio = { workspace = true, features = ["rt-multi-thread", "macros", "test-util"] }
 criterion = { workspace = true, features = ["html_reports"] }

crates/ecstore/src/api/mod.rs

@@ -45,8 +45,8 @@ pub mod bucket {
             pub use crate::bucket::lifecycle::bucket_lifecycle_ops::{
                 ExpiryState, LifecycleOps, RestoreRequestOps, TransitionState, TransitionedObject, apply_expiry_rule,
                 apply_transition_rule, enqueue_expiry_for_existing_objects, enqueue_transition_for_existing_objects,
-                enqueue_transition_immediate, expire_transitioned_object, get_global_expiry_state, get_global_transition_state,
-                init_background_expiry, post_restore_opts, run_stale_multipart_upload_cleanup_once, validate_transition_tier,
+                enqueue_transition_immediate, get_global_expiry_state, get_global_transition_state, init_background_expiry,
+                post_restore_opts, run_stale_multipart_upload_cleanup_once, validate_transition_tier,
             };
         }
 
@@ -261,9 +261,9 @@ pub mod data_usage {
         DATA_USAGE_CACHE_NAME, apply_bucket_usage_memory_overlay, init_compression_total_memory_from_backend,
         load_compression_total_from_memory, load_data_usage_from_backend, record_bucket_delete_marker_memory,
         record_bucket_object_delete_memory, record_bucket_object_version_write_memory, record_bucket_object_write_memory,
-        record_bucket_object_write_unknown_previous_memory, record_compression_total_memory,
-        refresh_bucket_usage_from_object_layer, refresh_versioned_bucket_usage_from_object_layer,
-        remove_bucket_usage_from_backend, replace_bucket_usage_memory_from_info, store_compression_total_in_backend,
+        record_compression_total_memory, refresh_bucket_usage_from_object_layer,
+        refresh_versioned_bucket_usage_from_object_layer, remove_bucket_usage_from_backend,
+        replace_bucket_usage_memory_from_info, store_compression_total_in_backend,
     };
 }
 
@@ -273,9 +273,9 @@ pub mod disk {
     pub use crate::disk::{
         BATCH_READ_VERSION_MAX_ITEMS, BUCKET_META_PREFIX, BatchReadVersionItem, BatchReadVersionReq, BatchReadVersionResp,
         CheckPartsResp, DeleteOptions, Disk, DiskAPI, DiskInfo, DiskInfoOptions, DiskLocation, DiskOption, DiskStore,
-        FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, OldCurrentSize, RUSTFS_META_BUCKET, ReadMultipleReq,
-        ReadMultipleResp, ReadOptions, RenameDataResp, STORAGE_FORMAT_FILE, UpdateMetadataOpts, VolumeInfo, WalkDirOptions,
-        new_disk, validate_batch_read_version_item_count,
+        FileInfoVersions, FileReader, FileWriter, HEALING_MARKER_PATH, RUSTFS_META_BUCKET, ReadMultipleReq, ReadMultipleResp,
+        ReadOptions, RenameDataResp, STORAGE_FORMAT_FILE, UpdateMetadataOpts, VolumeInfo, WalkDirOptions, new_disk,
+        validate_batch_read_version_item_count,
     };
     pub use bytes::Bytes;
     pub use endpoint::Endpoint;
@@ -326,7 +326,6 @@ pub mod global {
 }
 
 pub mod runtime {
-    pub use crate::runtime::instance::{InstanceContext, bootstrap_ctx};
     pub use crate::runtime::sources::{
         boot_time, bucket_monitor, deployment_id, endpoint_pools, expiry_state_handle, first_cluster_node_is_local,
         global_lock_client, global_lock_clients, global_tier_config_mgr, local_disk_map_read, object_store_handle, region,
@@ -351,8 +350,8 @@ pub mod notification {
 
 pub mod object {
     pub use crate::object_api::{
-        BLOCK_SIZE_V2, ERASURE_ALGORITHM, GetObjectBodyCacheHook, GetObjectReader, ObjectInfo, ObjectMutationHook, ObjectOptions,
-        PutObjReader, RangedDecompressReader, StreamConsumer, register_get_object_body_cache_hook, register_object_mutation_hook,
+        BLOCK_SIZE_V2, ERASURE_ALGORITHM, GetObjectBodyCacheHook, GetObjectReader, ObjectInfo, ObjectOptions, PutObjReader,
+        RangedDecompressReader, StreamConsumer, register_get_object_body_cache_hook,
     };
 }
 
@@ -388,11 +387,9 @@ pub mod store_list {
 }
 
 pub mod storage {
-    pub use crate::store::HealWalkVersion;
     pub use crate::store::{
-        ECStore, all_local_disk, all_local_disk_path, find_local_disk_by_ref, init_local_disks,
-        init_local_disks_with_instance_ctx, init_lock_clients, prewarm_local_disk_id_map,
-        prewarm_local_disk_id_map_with_instance_ctx,
+        ECStore, all_local_disk, all_local_disk_path, find_local_disk_by_ref, init_local_disks, init_lock_clients,
+        prewarm_local_disk_id_map,
     };
 }
 
@@ -429,13 +426,4 @@ pub mod tier {
             WarmBackend, WarmBackendGetOpts, WarmBackendImpl, build_transition_put_options, check_warm_backend, new_warm_backend,
         };
     }
-
-    #[cfg(feature = "test-util")]
-    pub mod test_util {
-        pub use crate::services::tier::test_util::{
-            FaultConfig, MockStoredObject, MockWarmBackend, MockWarmOp, TransitionMeta, assert_transition_meta_consistent,
-            free_version_count, read_transition_meta, register_mock_tier, register_mock_tier_backend,
-            wait_for_free_version_absence,
-        };
-    }
 }

crates/ecstore/src/bucket/bucket_target_sys.rs

@@ -78,6 +78,7 @@ use tracing::warn;
 use url::Url;
 use uuid::Uuid;
 
+const DEFAULT_HEALTH_CHECK_DURATION: Duration = Duration::from_secs(5);
 const DEFAULT_HEALTH_CHECK_RELOAD_DURATION: Duration = Duration::from_secs(30 * 60);
 const REDACTED_CREDENTIAL: &str = "<redacted>";
 
@@ -118,43 +119,24 @@ pub struct ArnErrs {
     pub bucket: String,
 }
 
-/// A single latency sample tagged with the instant it was recorded.
-///
-/// Only `dur` participates in (de)serialization; `at` is not `Serialize`
-/// (`Instant` isn't) and is reconstructed as `Instant::now()` on deserialize.
-/// A reloaded window therefore simply restarts aging, which is acceptable for
-/// the in-memory endpoint health stats this type backs (see backlog#806-16).
 #[derive(Debug, Clone, Serialize, Deserialize)]
-struct LatencySample {
-    dur: Duration,
+pub struct LastMinuteLatency {
+    times: Vec<Duration>,
     #[serde(skip, default = "instant_now")]
-    at: Instant,
+    start_time: Instant,
 }
 
 fn instant_now() -> Instant {
     Instant::now()
 }
 
-/// A rolling one-minute latency window.
-///
-/// backlog#806-16: the previous implementation stored bare `Vec<Duration>`
-/// samples plus a single, never-updated `start_time`, and its `retain`
-/// predicate ignored the element entirely — it evaluated the constant
-/// `now.duration_since(self.start_time) < 60s`. Once the window had existed
-/// for 60s, that predicate became `false` for every element, so each `add`
-/// dropped ALL samples and the "last minute" average degenerated to the most
-/// recent single sample. Each sample now carries its own timestamp and is
-/// retained by its individual age.
-///
-/// This type is only used for in-memory endpoint health (`EpHealth`) and
-/// admin-API display; it is not persisted or wire-serialized across versions
-/// (the on-the-wire latency shape is `crate::bucket::target::LatencyStat`,
-/// which carries only `curr`/`avg`/`max`). The `Serialize`/`Deserialize`
-/// derives are retained solely so the enclosing `LatencyStat` keeps deriving
-/// them; only the `Duration` part of each sample is serialized.
-#[derive(Debug, Clone, Serialize, Deserialize, Default)]
-pub struct LastMinuteLatency {
-    times: Vec<LatencySample>,
+impl Default for LastMinuteLatency {
+    fn default() -> Self {
+        Self {
+            times: Vec::new(),
+            start_time: Instant::now(),
+        }
+    }
 }
 
 impl LastMinuteLatency {
@@ -163,16 +145,11 @@ impl LastMinuteLatency {
     }
 
     pub fn add(&mut self, duration: Duration) {
-        self.add_at(Instant::now(), duration);
-    }
-
-    /// Records a sample at an explicit instant, dropping samples older than one
-    /// minute relative to `now`. Split out from `add` so the aging logic can be
-    /// exercised with a synthetic clock in tests.
-    fn add_at(&mut self, now: Instant, duration: Duration) {
+        let now = Instant::now();
+        // Remove entries older than 1 minute
         self.times
-            .retain(|sample| now.duration_since(sample.at) < Duration::from_secs(60));
-        self.times.push(LatencySample { dur: duration, at: now });
+            .retain(|_| now.duration_since(self.start_time) < Duration::from_secs(60));
+        self.times.push(duration);
     }
 
     pub fn get_total(&self) -> LatencyAverage {
@@ -181,7 +158,7 @@ impl LastMinuteLatency {
                 avg: Duration::from_secs(0),
             };
         }
-        let total: Duration = self.times.iter().map(|sample| sample.dur).sum();
+        let total: Duration = self.times.iter().sum();
         LatencyAverage {
             avg: total / self.times.len() as u32,
         }
@@ -334,9 +311,7 @@ impl BucketTargetSys {
     }
 
     pub async fn heartbeat(&self) {
-        // Probe interval: `RUSTFS_REPL_HEALTH_CHECK_INTERVAL_MS` (default 5000ms,
-        // clamped to >=10ms), read once when the heartbeat task starts.
-        let mut interval = tokio::time::interval(crate::bucket::replication::replication_timing::health_check_interval());
+        let mut interval = tokio::time::interval(DEFAULT_HEALTH_CHECK_DURATION);
         loop {
             interval.tick().await;
 
@@ -966,37 +941,13 @@ fn has_custom_ca_pem(target: &BucketTarget) -> bool {
     !target.ca_cert_pem.trim().is_empty()
 }
 
-/// Env opt-in that re-enables loopback replication targets. Loopback (`127.0.0.1`,
-/// `::1`, `localhost`) is a classic SSRF vector and stays rejected by default, but
-/// single-host multi-instance dev setups and the e2e harness legitimately replicate
-/// over loopback. Never set this in production.
-const ALLOW_LOOPBACK_REPLICATION_TARGET_ENV: &str = "RUSTFS_REPLICATION_ALLOW_LOOPBACK_TARGET";
-
-fn loopback_replication_targets_allowed() -> bool {
-    std::env::var(ALLOW_LOOPBACK_REPLICATION_TARGET_ENV)
-        .map(|v| v.eq_ignore_ascii_case("true") || v == "1")
-        .unwrap_or(false)
-}
-
 fn validate_replication_target_endpoint(url: &Url) -> Result<(), OutboundUrlError> {
-    validate_replication_target_endpoint_inner(url, loopback_replication_targets_allowed())
-}
-
-fn validate_replication_target_endpoint_inner(url: &Url, allow_loopback: bool) -> Result<(), OutboundUrlError> {
     match validate_outbound_url(url) {
         Ok(()) => Ok(()),
-        // Replication targets are trusted infrastructure the operator configures, and
-        // legitimately live on private networks, so private addresses are always allowed.
         Err(OutboundUrlError::ForbiddenHost {
             reason: "private address",
             ..
         }) => Ok(()),
-        // Loopback is far higher SSRF risk, so it is allowed only under the explicit,
-        // off-by-default opt-in above (single-host multi-instance / the e2e harness).
-        Err(OutboundUrlError::ForbiddenHost {
-            reason: "loopback address" | "loopback host",
-            ..
-        }) if allow_loopback => Ok(()),
         Err(err) => Err(err),
     }
 }
@@ -1930,76 +1881,6 @@ mod tests {
         assert!(!replication_target_versioning_enabled(None));
     }
 
-    fn parse_url(raw: &str) -> Url {
-        Url::parse(raw).expect("test URL should parse")
-    }
-
-    #[test]
-    fn replication_endpoint_always_allows_public_and_private() {
-        // Public hosts and private-network targets are allowed regardless of the
-        // loopback opt-in — replication commonly runs across trusted private infra.
-        for allow_loopback in [false, true] {
-            assert!(validate_replication_target_endpoint_inner(&parse_url("https://s3.example.com"), allow_loopback).is_ok());
-            assert!(validate_replication_target_endpoint_inner(&parse_url("http://10.0.0.5:9000"), allow_loopback).is_ok());
-            assert!(validate_replication_target_endpoint_inner(&parse_url("http://192.168.1.20"), allow_loopback).is_ok());
-        }
-    }
-
-    #[test]
-    fn replication_endpoint_rejects_loopback_without_opt_in() {
-        // Default (production) behaviour: loopback IP and localhost host both rejected.
-        let err = validate_replication_target_endpoint_inner(&parse_url("http://127.0.0.1:9000"), false)
-            .expect_err("loopback IP must be rejected by default");
-        assert!(matches!(
-            err,
-            OutboundUrlError::ForbiddenHost {
-                reason: "loopback address",
-                ..
-            }
-        ));
-        let err = validate_replication_target_endpoint_inner(&parse_url("http://localhost:9000"), false)
-            .expect_err("localhost must be rejected by default");
-        assert!(matches!(
-            err,
-            OutboundUrlError::ForbiddenHost {
-                reason: "loopback host",
-                ..
-            }
-        ));
-    }
-
-    #[test]
-    fn replication_endpoint_allows_loopback_with_opt_in() {
-        // e2e harness / single-host multi-instance: opt-in re-enables loopback in
-        // both IP (127.0.0.1, ::1) and hostname (localhost) forms.
-        assert!(validate_replication_target_endpoint_inner(&parse_url("http://127.0.0.1:9000"), true).is_ok());
-        assert!(validate_replication_target_endpoint_inner(&parse_url("http://[::1]:9000"), true).is_ok());
-        assert!(validate_replication_target_endpoint_inner(&parse_url("http://localhost:9000"), true).is_ok());
-    }
-
-    #[test]
-    fn replication_endpoint_opt_in_does_not_open_other_ssrf_targets() {
-        // The loopback opt-in must not widen into link-local / metadata endpoints.
-        let err = validate_replication_target_endpoint_inner(&parse_url("http://169.254.169.254/latest/meta-data"), true)
-            .expect_err("metadata endpoint must stay rejected even with loopback opt-in");
-        assert!(matches!(
-            err,
-            OutboundUrlError::ForbiddenHost {
-                reason: "metadata endpoint",
-                ..
-            }
-        ));
-        let err = validate_replication_target_endpoint_inner(&parse_url("http://[fe80::1]:9000"), true)
-            .expect_err("link-local must stay rejected even with loopback opt-in");
-        assert!(matches!(
-            err,
-            OutboundUrlError::ForbiddenHost {
-                reason: "link-local address",
-                ..
-            }
-        ));
-    }
-
     #[test]
     fn remote_target_connection_error_display_redacts_access_key() {
         let err = BucketTargetError::RemoteTargetConnectionErr {
@@ -2266,63 +2147,4 @@ mod tests {
 
         assert!(err.to_string().contains("invalid target CA PEM"));
     }
-
-    // backlog#806-16 regression tests for the rolling one-minute latency window.
-
-    #[test]
-    fn last_minute_latency_averages_only_samples_within_window() {
-        let base = Instant::now();
-        let mut window = LastMinuteLatency::new();
-
-        // Two samples that will fall outside the 60s window once the fresh
-        // sample is added, plus one fresh sample.
-        window.add_at(base, Duration::from_millis(100));
-        window.add_at(base + Duration::from_secs(10), Duration::from_millis(200));
-        // 61s after `base`: both earlier samples are now >60s old relative to
-        // the 200ms sample? No — measured against `now`. Add a fresh sample far
-        // in the future so the two old samples age out.
-        window.add_at(base + Duration::from_secs(61), Duration::from_millis(400));
-
-        // Only the fresh sample survives: 100ms (age 61s) and 200ms (age 51s)
-        // vs the fresh one — 51s is still < 60s, so the 200ms sample stays.
-        // Assert the window kept exactly the in-window samples.
-        assert_eq!(window.get_total().avg, Duration::from_millis(300)); // (200 + 400) / 2
-    }
-
-    #[test]
-    fn last_minute_latency_drops_all_stale_samples() {
-        let base = Instant::now();
-        let mut window = LastMinuteLatency::new();
-
-        // Two stale samples, then one sample far enough in the future that both
-        // are strictly older than 60s.
-        window.add_at(base, Duration::from_millis(100));
-        window.add_at(base + Duration::from_secs(5), Duration::from_millis(300));
-        window.add_at(base + Duration::from_secs(120), Duration::from_millis(500));
-
-        // Both old samples aged out (>=60s); only the fresh 500ms remains. Under
-        // the OLD all-or-nothing bug the result would still have been the last
-        // single sample by coincidence, so also verify the mixed-window case below.
-        assert_eq!(window.get_total().avg, Duration::from_millis(500));
-    }
-
-    #[test]
-    fn last_minute_latency_two_fresh_samples_average_both() {
-        let base = Instant::now();
-        let mut window = LastMinuteLatency::new();
-
-        window.add_at(base, Duration::from_millis(100));
-        window.add_at(base + Duration::from_secs(1), Duration::from_millis(300));
-
-        // Both within the window -> average of both. The OLD bug degenerated the
-        // window to a single sample after 60s; here samples are close in time so
-        // the correct behaviour is a genuine two-sample average.
-        assert_eq!(window.get_total().avg, Duration::from_millis(200));
-    }
-
-    #[test]
-    fn last_minute_latency_empty_window_is_zero() {
-        let window = LastMinuteLatency::new();
-        assert_eq!(window.get_total().avg, Duration::from_secs(0));
-    }
 }

crates/ecstore/src/bucket/lifecycle/bucket_lifecycle_ops.rs

@@ -618,7 +618,7 @@ impl ExpiryState {
     async fn worker(rx: &mut Receiver<Option<ExpiryOpType>>, api: Arc<ECStore>, stats: Arc<ExpiryStats>) {
         let cancel_token = runtime_sources::background_services_cancel_token().unwrap_or_else(|| {
             static FALLBACK: std::sync::OnceLock<tokio_util::sync::CancellationToken> = std::sync::OnceLock::new();
-            FALLBACK.get_or_init(tokio_util::sync::CancellationToken::new).clone()
+            FALLBACK.get_or_init(tokio_util::sync::CancellationToken::new)
         });
 
         loop {
@@ -1352,7 +1352,9 @@ fn spawn_tier_free_version_recovery_once(api: Arc<ECStore>) {
     }
 
     tokio::spawn(async move {
-        let cancel_token = runtime_sources::background_services_cancel_token().unwrap_or_default();
+        let cancel_token = runtime_sources::background_services_cancel_token()
+            .cloned()
+            .unwrap_or_else(CancellationToken::new);
         let mut interval = tokio::time::interval(StdDuration::from_secs(60));
         let mut bucket_marker: Option<String> = None;
         let mut object_marker: Option<String> = None;
@@ -1425,7 +1427,9 @@ fn spawn_tier_delete_journal_recovery_once(api: Arc<ECStore>) {
     }
 
     tokio::spawn(async move {
-        let cancel_token = runtime_sources::background_services_cancel_token().unwrap_or_default();
+        let cancel_token = runtime_sources::background_services_cancel_token()
+            .cloned()
+            .unwrap_or_else(CancellationToken::new);
         run_tier_delete_journal_recovery_loop(api, cancel_token).await;
     });
 }
@@ -2261,9 +2265,6 @@ pub async fn expire_transitioned_object(
         opts.transition.expire_restored = true;
         return match api.delete_object(&oi.bucket, &oi.name, opts).await {
             Ok(dobj) => {
-                // Drop any cached restored-copy body so it does not sit resident
-                // until TTL after the copy is expired (ODC-26).
-                crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await;
                 //audit_log_lifecycle(*oi, ILMExpiry, tags, traceFn);
                 Ok(dobj)
             }
@@ -2296,10 +2297,6 @@ pub async fn expire_transitioned_object(
 
     schedule_lifecycle_replication_delete_if_needed(oi, &dobj).await;
 
-    // The transitioned version is gone; evict any cached body for this object
-    // so it does not linger until TTL (ODC-26).
-    crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await;
-
     //audit_log_lifecycle(oi, ILMExpiry, tags);
 
     emit_transitioned_expiration_event(oi, &dobj);
@@ -2312,7 +2309,7 @@ pub fn gen_transition_objname(bucket: &str) -> Result<String, Error> {
     let mut hasher = Sha256::new();
     hasher.update(format!("{}/{}", runtime_sources::deployment_id().unwrap_or_default(), bucket).as_bytes());
     let hash = rustfs_utils::crypto::hex(hasher.finalize().as_slice());
-    let obj = format!("{}/{}/{}/{}", &hash[0..16], &us[0..2], &us[2..4], us);
+    let obj = format!("{}/{}/{}/{}", &hash[0..16], &us[0..2], &us[2..4], &us);
     Ok(obj)
 }
 
@@ -2809,13 +2806,6 @@ pub async fn apply_expiry_on_non_transitioned_objects(
         }
     };
     schedule_lifecycle_replication_delete_if_needed(oi, &dobj).await;
-
-    // The object (or all its versions, for delete_all) was expired; evict any
-    // cached body so dead bytes do not sit resident until TTL (ODC-26). The
-    // cache identity is the decoded object name used by GET, not the
-    // encode_dir_object form passed to delete_object.
-    crate::object_api::notify_object_mutation(&oi.bucket, &oi.name).await;
-
     //debug!("dobj: {:?}", dobj);
     if dobj.name.is_empty() {
         dobj = oi.clone();

crates/ecstore/src/bucket/lifecycle/runtime_boundary.rs

@@ -35,7 +35,7 @@ pub(crate) fn tier_config_mgr_handle() -> Arc<RwLock<TierConfigMgr>> {
     sources::tier_config_mgr_handle()
 }
 
-pub(crate) fn background_services_cancel_token() -> Option<CancellationToken> {
+pub(crate) fn background_services_cancel_token() -> Option<&'static CancellationToken> {
     sources::background_services_cancel_token()
 }
 

crates/ecstore/src/bucket/lifecycle/tier_delete_journal.rs

@@ -63,12 +63,7 @@ impl PersistedTierDeleteJournalEntry {
         if self.version != TIER_DELETE_JOURNAL_VERSION {
             return Err(Error::other(format!("unsupported tier delete journal version {}", self.version)));
         }
-        // Empty `version_id` is a legal sentinel for objects transitioned to an
-        // unversioned remote tier (see CLAUDE.md: a tier version of `None`/`""`
-        // means the tier bucket is unversioned, so the remote delete is issued
-        // without a versionId). Only reject entries missing the object or tier
-        // name, which are always populated for a TRANSITION_COMPLETE object.
-        if self.obj_name.is_empty() || self.tier_name.is_empty() {
+        if self.obj_name.is_empty() || self.version_id.is_empty() || self.tier_name.is_empty() {
             return Err(Error::other("tier delete journal entry is incomplete"));
         }
         Ok(Jentry {
@@ -325,39 +320,4 @@ mod tests {
 
         assert!(err.to_string().contains("incomplete"));
     }
-
-    #[test]
-    fn tier_delete_journal_recovers_unversioned_tier_entry() {
-        // A remote tier that is unversioned records an empty `version_id`. Such a
-        // WAL entry must decode successfully so recovery can drive a versionless
-        // remote delete, otherwise the remote object is orphaned and the journal
-        // file leaks forever.
-        let payload = br#"{"version":1,"obj_name":"remote/object","version_id":"","tier_name":"WARM"}"#;
-
-        let decoded = decode_tier_delete_journal_entry(payload).expect("unversioned tier entry should decode");
-
-        assert_eq!(decoded.obj_name, "remote/object");
-        assert!(decoded.version_id.is_empty());
-        assert_eq!(decoded.tier_name, "WARM");
-    }
-
-    #[test]
-    fn tier_delete_journal_rejects_missing_tier_name() {
-        let payload = br#"{"version":1,"obj_name":"remote/object","version_id":"v1","tier_name":""}"#;
-
-        let err = decode_tier_delete_journal_entry(payload).expect_err("entry missing tier name should be rejected");
-
-        assert!(err.to_string().contains("incomplete"));
-    }
-
-    #[test]
-    fn tier_delete_journal_rejects_truncated_payload() {
-        // A partially written journal file fails at JSON deserialization, so
-        // relaxing the empty-version_id check does not admit truncated records.
-        let payload = br#"{"version":1,"obj_name":"remote/object","version_id":""#;
-
-        let err = decode_tier_delete_journal_entry(payload).expect_err("truncated journal payload should be rejected");
-
-        assert!(err.to_string().contains("decode tier delete journal failed"));
-    }
 }