chore(deps): refresh crates and adopt released aws-sigv4

Run two update/upgrade rounds for the main and standalone fuzz workspaces.
Replace the temporary SigV4a Git override with aws-sigv4 1.6.4 from the
registry and remove its obsolete source permission.

Pin OpenID Connect to e20ddbb01ca68356f2979bc125f5613ab45b4e7e. Upgrade
async-nats to 0.51.0, hotpath to 0.30.0, libunftp to 0.24.0, and the TOML
test dependency to 1.1.8. Retain Arrow/Parquet 59.3.2 for DataFusion 55.2.0
and keep the fuzz PKCS#1 version compatible with pinned RSA rc.18.

Co-Authored-By: heihutu <heihutu@gmail.com>
Co-Authored-By: zhi22915 <qiuzgang@gmail.com>

.agents/references/adversarial-validation.md

@@ -27,7 +27,4 @@ Risk and review shape:
   and list verified and unverified target classes in the PR Impact section.
 
 Available domain lenses are security, concurrency/durability, compatibility,
-and performance. Select `.agents/skills/adversarial-validation/SKILL.md` for an
-explicit adversarial request, a high-risk change, or a substantial PR review;
-then read only its matching role references. A routine standard pass does not
-load the playbook unless the reviewer needs a RustFS-specific probe.
+and performance; select only those matched by the change.

.agents/skills/adversarial-validation/SKILL.md

@@ -1,48 +0,0 @@
----
-name: adversarial-validation
-description: Review RustFS diffs or designs for explicit adversarial requests, high-risk changes under the repository review policy, or substantial PR reviews. Skip ordinary questions, diagnosis, planning, status, routine low-risk implementation, and prose with no execution effect.
----
-
-# RustFS Adversarial Validation
-
-Use the [repository risk tiers and review shape](../../references/adversarial-validation.md). This skill
-routes a review to RustFS-specific probes without loading unrelated domains.
-
-## Select Lenses
-
-Read only the references required by the diff:
-
-| Lens | When to read |
-|---|---|
-| [Correctness](references/correctness.md) | Every non-exempt adversarial review |
-| [Simplicity](references/simplicity.md) | Mechanical/standard changes and production growth |
-| [Test coverage](references/test-coverage.md) | Behavior or test changes |
-| [Security](references/security.md) | Authn/authz, IAM, RPC trust, paths, secrets, parsing, browser, encryption |
-| [Concurrency/durability](references/concurrency-durability.md) | Async shared state, locks, storage commit, cancellation, persisted queues |
-| [Compatibility](references/compatibility.md) | S3 surface, MinIO interop, metadata, wire/disk formats, mixed versions |
-| [Performance](references/performance.md) | Request/object hot paths, allocation, blocking work, fsync, fan-out |
-
-Do not read all references as a precaution. A path name alone is insufficient;
-the changed behavior must touch the lens's domain.
-
-For a dedicated security audit or advisory analysis, use
-`security-advisory-lessons` instead of loading it automatically during every
-adversarial review.
-
-## Review Protocol
-
-1. Freeze the exact final diff/head (or the design under review) and list the
-   selected lenses.
-2. Run the review shape required by the repository risk tier.
-3. For each selected lens, either report a concrete finding or a null verdict
-   naming the attacks performed.
-4. Apply root `AGENTS.md`'s finding standard. Test each candidate against callers,
-   existing coverage, and invariants before accepting it; an adversarial role
-   does not have to produce a defect.
-5. Fix or rebut supported findings with code-path, test, or invariant evidence.
-6. After a non-trivial edit, rerun only lenses affected by that edit against the
-   new exact diff.
-
-Do not turn a null verdict into a long checklist. Record concise evidence that
-the relevant failure classes were attacked, then stop under the root completion
-rule. Keep the required per-lens verdicts for high-risk PRs.

.agents/skills/adversarial-validation/references/compatibility.md

@@ -1,38 +0,0 @@
-# Compatibility Lens
-
-- Internal metadata uses `metadata_compat` helpers for dual RustFS/MinIO keys,
-  including mixed casing and removal of both twins.
-- Binary UUID metadata treats absent, empty, and nil as no value. Unversioned
-  remote tiers receive no `versionId`; versioned purge requests retain the real
-  version ID.
-- `xl.meta` changes preserve supported header/meta versions, recompute
-  signatures, decode legacy fixtures, and remain readable by old RustFS/MinIO.
-- Foreign/corrupt metadata validates parallel array lengths and missing fields;
-  it returns a decode error rather than indexing, panicking, or fabricating data.
-- Do not “correct” byte-for-byte MinIO ports without legacy fixture evidence.
-  Bitrot framing, shard math, distribution, and inline prefixes are contracts.
-- Client-visible metadata/events strip both internal prefixes
-  case-insensitively.
-- Proto fields are appended, never reused/renumbered; FlatBuffers tables extend
-  compatibly and absent new fields fail closed where authorization/quorum is
-  involved.
-- Replay real client request shapes and exact pagination boundaries for S3
-  handler changes.
-- Bucket metadata/IAM/config parsing remains compatible with pinned real MinIO
-  fixtures and encrypted migration data.
-- Compatibility shims use `RUSTFS_COMPAT_TODO(<task-id>)`, have a removal
-  condition, and default toward reading old data safely.
-
-## Outbound targets
-
-- A change to what the replication or migration client sends by default
-  (checksum policy, payload framing, headers, version-id addressing) is judged
-  against every target class, not the one it fixes. Name each target-side rule
-  the current default satisfies — checksum required with Object Lock
-  parameters, `aws-chunked` decoding, version-id adoption, ETag equals content
-  MD5 — and show which cell of
-  `crates/e2e_test/src/replication_target_matrix_test.rs` covers each.
-- A test that asserts the fix ("no trailer header") is not evidence; the
-  matrix cell that asserts the target accepted and stored the object is.
-- Every new environment escape hatch appears in
-  `docs/operations/replication-outbound-transport.md` in the same diff.

.agents/skills/adversarial-validation/references/concurrency-durability.md

@@ -1,27 +0,0 @@
-# Concurrency and Durability Lens
-
-- For every changed lock, enumerate overlapping lock sets and construct the
-  ABBA interleaving. Multiple-lock order must be documented and consistent.
-- Mark guard lifetimes and every `.await`, disk,
-  and RPC call; estimate contention/timeouts. Tokio read/write guards across
-  `.await` need bounded hold time; long-lived reads can wedge writers (#4195).
-  Keep `std::sync::Mutex` holds brief and never across `.await`.
-- Use direct `fetch_*` for unconditional atomic read-modify-write and
-  `compare_exchange` for conditional updates.
-- Object commits remain fenced if the distributed lock is lost after shard
-  writes and before metadata rename.
-- For write/rename changes, trace `write tmp -> sync tmp -> rename -> sync parent
-  -> sync required ancestors`; simulate a crash after each step and honor the
-  configured durability gate.
-- Multi-disk fan-out counts every result. Quorum-minus-one cannot become success;
-  heal remains best-effort per target where that is the established contract.
-- At every new cancellable await between mutation and cleanup/commit, drop the
-  future and inspect leftover files, counters, permits, and replay state.
-- Multipart operations on the same upload ID are serialized where required;
-  abort/complete/list races cannot delete parts before durable commit.
-- Post-commit cleanup is best-effort, retry-safe, and cannot fail an already
-  committed write or delete the last surviving copy.
-- Persisted read-modify-write uses serialization/CAS. Queue replay is crash-safe
-  and duplicate delivery has an idempotency contract.
-- Streaming reconstruction failures after partial output surface as errors, not
-  successful EOF.

.agents/skills/adversarial-validation/references/correctness.md

@@ -1,31 +0,0 @@
-# Correctness Lens
-
-Attack the changed behavior, not every subsystem in the repository.
-
-- Trace new error paths to the caller. Inject the ignored/wildcard variants and
-  verify they cannot become success, not-found, or a plausible default.
-- Exercise zero/empty/missing, maximum, and exact-boundary inputs for every
-  changed count, size, index, page limit, or optional value.
-- For aggregation/quorum changes, test exactly quorum and quorum-minus-one with
-  mixed disk errors and nil/placeholder entries.
-- For listing/pagination, test `n == max`, `n == max + 1`, delimiter folding,
-  continuation markers, and object/prefix name collisions.
-- For EC/read/streaming changes, inject failure after partial output and verify
-  the client receives an error rather than a clean truncated body. Assert exact
-  bytes and length.
-- For multipart/object commits, fail before/after rename and cleanup; committed
-  data must remain readable and pre-commit cleanup must not destroy parts.
-- Bound recursion over untrusted or persisted input; handle corrupted/cyclic
-  tree and cache traversals safely.
-- For version/index ordering, test `len - 1`, `len`, equal timestamps, missing
-  versions, and deterministic tie-breaking.
-- For directory-object behavior, trace `__XLDIR__` at the store layer; branches
-  below the layer that sees trailing slashes are dead.
-- For binary UUID metadata, absent, empty, and nil all mean no value. Never send
-  nil/empty `versionId` to an unversioned tier.
-- For agent rules/skill routers, test a trigger matrix covering ordinary
-  inquiry, low-risk implementation, explicit review, high-risk code, PR
-  creation, release, and post-PR monitoring. Each case must select only the
-  intended workflow and retain required safety/authorization boundaries.
-
-Null verdicts name only the probes relevant to the diff.

.agents/skills/adversarial-validation/references/performance.md

@@ -1,20 +0,0 @@
-# Performance Lens
-
-- For added clones/allocations on request/object/block paths, quantify copied
-  data and frequency. Recommend borrowing, move, `Bytes`/`Arc`, `Cow`, or
-  capacity reservation only for a concrete repeated cost.
-- Route every new sync/flush through the durability-mode and bucket override
-  gates; mode `none` must not pay the new fsync.
-- Keep blocking filesystem/CPU work off async runtime threads, but do not split
-  one small operation into many `spawn_blocking` round trips.
-- Measure lock hold time across I/O and compare acquisition order for ABBA.
-- Keep cleanup, extra stat/rename, and diagnostics out of the PUT commit critical
-  section when they need not be there.
-- Detect per-item serial I/O/RPC in batch APIs and accidental quadratic scans;
-  use a gate or bounded concurrency when the concrete fan-out warrants it.
-- Count buffer growth and byte copies in EC/bitrot paths; preserve pool gauge
-  balance and avoid repeated metadata decode/fetch per object.
-- Repetitive success logs stay at `trace`; metrics/instrumentation on hot paths
-  require an existing gate.
-- Claims of no impact on PUT/GET/commit/erasure paths need relevant benchmark or
-  A/B evidence, especially for 4 KiB objects.

.agents/skills/adversarial-validation/references/security.md

@@ -1,31 +0,0 @@
-# Security Lens
-
-Use `security-advisory-lessons` only for a dedicated advisory/security audit.
-For an ordinary matched diff, attack these boundaries:
-
-- Admin routes: route registration, whitelist, handler authn, and the exact
-  `AdminAction` must agree. Read-only diagnostics still require admin authz.
-- IAM/service accounts: treat parent, claims, keys, groups, status, and policy
-  names as attacker-controlled; prove ownership/root authority before writes.
-- Protocol frontends: every changed/sibling command authorizes the matching S3
-  action before reaching storage.
-- Secrets/signatures: use constant-time comparison, normalize public failures,
-  keep RPC/root/STS keys independent, and fail closed when secrets are absent.
-- RPC: bind signatures to the exact method/path and timestamp; reject replay,
-  stale, malformed, truncated, and invalid-enum payloads without panic.
-- Paths/object/archive entries: reject traversal, absolute/platform escapes,
-  and normalization differences between authz and storage.
-- Copy/multipart/presigned POST: enforce source, destination, version-aware
-  actions, copy-source conditions, and every signed policy condition.
-- Logging/errors: never expose credentials, tokens, expected signatures, raw
-  secret-bearing input, or merged configs—including via `Debug` and parse errors.
-- Untrusted serde: reject unknown fields where compatible and validate
-  security-critical defaults/ranges before numeric conversion.
-- SSE/browser/CORS/trusted proxy: inspect stored ciphertext and wrapper order;
-  isolate user content; never reflect credentialed arbitrary origins or trust
-  forwarded identity from direct clients.
-- Object Lock: unreadable/fabricated/unparsable metadata fails closed across
-  foreground, lifecycle, scanner, and force-delete paths.
-
-Security findings distinguish unauthenticated compromise from a
-low-privileged authenticated bypass.

.agents/skills/adversarial-validation/references/simplicity.md

@@ -1,22 +0,0 @@
-# Simplicity Lens
-
-- Compare the production diff with the smallest equivalent local edit. Fewer
-  lines alone are not evidence; the replacement must preserve correctness,
-  compatibility, readability, and real boundaries.
-- Search the touched crate, domain owner, `crates/utils`, `crates/common`, and
-  relevant dependencies for each new helper, constant, wrapper, or fixture.
-- Reject forced reuse when normalization, error, deadline, or durability
-  semantics differ.
-- Require a concrete trigger for every new defensive branch. Keep boundary
-  checks for disk/RPC/version data and checks immediately before destructive
-  actions.
-- Flag one-caller helpers only when they merely forward or split a short linear
-  flow without adding domain naming, invariant isolation, or useful context.
-- Ensure a replacement removes the superseded in-scope path or keeps one
-  canonical core behind a documented compatibility adapter.
-- Remove narration/change-history comments; preserve concise safety, lock,
-  durability, and compatibility invariants.
-- Treat tests, fixtures, generated code, and documentation separately from
-  production growth. Do not optimize away meaningful regression coverage.
-
-A finding must include a concrete smaller design, not a style preference.

.agents/skills/adversarial-validation/references/test-coverage.md

@@ -1,26 +0,0 @@
-# Test-Coverage Lens
-
-- For every behavior claim, name the focused test/check that fails if the
-  changed hunk is reverted. If none is practical, require the reason and
-  residual risk.
-- Confirm tests exercise the real production path and distinguish the intended
-  behavior from the named regression. A success, `is_err()`, or no-panic check
-  can be sufficient when that is the actual contract; require exact values,
-  bytes, state, or error variants when those distinctions matter to the change.
-- For new flags/modes, verify each branch and ask which test fails if the branch
-  is inverted.
-- For new error propagation, inject the failure and assert the caller observes
-  it; mentally replacing `?`/`return Err` with success must break a test.
-- Streaming GET tests assert the complete body and length under degraded reads.
-- Disk/wire-format tests use pinned foreign/legacy fixtures; same-code
-  round-trips are insufficient for compatibility.
-- Concurrency tests use readiness polling, isolate global state, and avoid fixed
-  sleeps or unrealistically short timeouts. Use nextest groups when process-level
-  serialization is required.
-- Internal metadata tests assert both RustFS and MinIO keys, not only read-back
-  through a helper that prefers one key.
-- Avoid duplicate tests of the same production path and poison-value class.
-  Boundary companions are distinct coverage: `n == max` vs `max + 1`, and
-  absent vs empty vs nil UUID.
-- A focused test proves only the targets/features it builds. Add compilation or
-  Clippy only for uncovered changed targets.

.agents/skills/pr-review/SKILL.md

@@ -13,7 +13,7 @@ description: Review a GitHub PR using its exact base/head and repository risk po
    ```
 
    Record both SHAs; refresh if either moved during fetching. Follow relevant [implementation boundaries](../../references/implementation.md).
-2. Apply the [risk policy](../../references/adversarial-validation.md). Ordinary reviews use the root finding standard directly. For explicit adversarial, substantial, or high-risk reviews, use [the domain probes](../adversarial-validation/SKILL.md). File count does not authorize delegation; use exactly two independent reviewers only when authorized, otherwise two fresh sequential passes for that risk tier. Reviewers do not spawn further agents.
+2. Apply the [risk policy](../../references/adversarial-validation.md). Ordinary reviews use the root finding standard directly. File count does not authorize delegation; use exactly two independent reviewers only when authorized, otherwise two fresh sequential passes for that risk tier. Reviewers do not spawn further agents.
 3. Check `gh pr checks <N> --repo <owner/repo>`. Investigate failures when relevant to a finding or a requested CI/readiness diagnosis. Distinguish pending, unavailable, pre-existing, flaky, and PR-caused states using evidence; a code review does not authorize unrelated repairs.
 4. Report the reviewed SHAs, risk tier, supported findings or `No findings`, material verification limits, check state, and verdict. P0/P1 findings block approval. Severity: P0 = demonstrated data loss, security breach, remote crash, or deadlock; P1 = correctness, compatibility, or material hot-path regression; P2 = a concrete maintainability defect; P3 = optional style, only when requested.
 5. When posting is authorized, follow [Git and PR rules](../../references/pull-requests.md), recheck the remote head, and review any delta before submitting. Use `gh pr review` with `--approve`, `--request-changes`, or `--comment` and `--body-file`; use [the API example](references/posting.md) for authorized inline comments. Reuse existing authorization.

.agents/skills/security-advisory-lessons/SKILL.md

@@ -1,56 +0,0 @@
----
-name: security-advisory-lessons
-description: Perform a dedicated RustFS security/advisory review for authn/authz, IAM, RPC trust, paths, secrets, browser isolation, encryption, Object Lock, or other security boundaries. Use only when the user requests a security/advisory review or an adversarial review explicitly escalates to the full advisory map; do not auto-load solely because code touches a sensitive path.
----
-
-# RustFS Security Advisory Lessons
-
-Use this skill as the deep security lens. For a normal adversarial review with a
-matched security surface, the concise security reference under
-`adversarial-validation` is sufficient.
-
-## Workflow
-
-1. Freeze the exact diff/head and identify the changed trust boundaries.
-2. Inspect the headings in [advisory-patterns.md](references/advisory-patterns.md),
-   then read the matching sections. Read the full map only for a broad security
-   audit. Useful headings are
-   auth/admin, IAM/STS/OIDC, policy/plugins, S3/copy/multipart, protocols, paths,
-   secrets/logging/RPC, browser/CORS/proxy, SSE, Object Lock, and serde.
-3. Trace unauthenticated, low-privilege, wrong-action/owner/bucket, malformed,
-   and default-config cases. Security decisions must fail closed.
-4. Require a focused negative regression test for the bypass/exploit form, not
-   only the intended success path. State residual risk when a test is impractical.
-5. Report proven vulnerabilities separately from defense-in-depth hardening.
-
-When advisory currentness matters, fetch the live inventory instead of treating
-the reference as a status mirror:
-
-```bash
-gh api repos/rustfs/rustfs/security-advisories --paginate \
-  --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
-```
-
-Fetch an individual advisory only when the live summary indicates a new or
-changed lesson.
-
-## Finding Standard
-
-Every candidate names the lower-trust principal, the boundary it crosses, the
-affected resource, and the observable result. A best-practice gap or an impact
-confined to the caller's own principal is hardening, not a candidate.
-
-Report each candidate under exactly one verdict:
-
-- `confirmed`: a complete `file:line` source trace and a bounded observed
-  result. Only confirmed findings carry severity; each also states attacker
-  prerequisites, concrete input/path, impact, smallest safe fix, and a
-  regression check.
-- `needs_validation`: the trace is blocked on one exact unresolved fact, such
-  as deployment or infrastructure behavior. State that fact; assign no severity
-  and do not substitute an assumption.
-- `rejected`: a disproved candidate, with the caller, invariant, or test that
-  disproves it.
-
-Do not exaggerate unauthenticated impact when the actual issue requires
-authenticated low privilege.

.agents/skills/security-advisory-lessons/references/advisory-patterns.md

@@ -1,158 +0,0 @@
-# RustFS Advisory Pattern Map
-
-This file is a lesson map, not an advisory inventory mirror. It keeps durable security patterns distilled from RustFS GitHub Security Advisories.
-
-When current advisory state, severity, URLs, or full text matters, fetch it live:
-
-```bash
-gh api repos/rustfs/rustfs/security-advisories --paginate \
-  --jq '.[] | {ghsa_id,state,severity,summary,updated_at}'
-gh api repos/rustfs/rustfs/security-advisories/<GHSA_ID>
-```
-
-Update this file only when an advisory adds or changes a reusable lesson, affected surface, validation pattern, or regression-test expectation. Do not update it for state-only, URL-only, count-only, or timestamp-only changes.
-
-## Pattern Index
-
-### Admin authorization and route exposure
-
-- `GHSA-pfcq-4gjr-6gjm`: notification target endpoints accepted authenticated users but skipped admin authorization. Lesson: distinguish authn from authz; admin target CRUD must call the operation-specific admin authorization path.
-- `GHSA-mm2q-qcmx-gw4w`: `ListServiceAccount` used `UpdateServiceAccountAdminAction`, while update lacked target ownership checks. Lesson: exact action constants and ownership checks are both required; information disclosure can chain into secret rotation and takeover.
-- `GHSA-vcwh-pff9-64cc`: `ImportIam` checked `ExportIAMAction` for an import/write operation. Lesson: every admin handler must authorize the action it actually performs.
-- `GHSA-jqmc-mg33-v45g` and `GHSA-8784-9m7f-c6p6`: `/profile/cpu` and `/profile/memory` were whitelisted from auth and allowed expensive diagnostics plus path disclosure. Lesson: profiling/debug endpoints need admin auth, opt-in, rate limits, and non-sensitive responses.
-- `GHSA-f5cv-v44x-2xgf`: `/rustfs/admin/v3/metrics` accepted any authenticated IAM user and skipped admin authorization. Lesson: read-only metrics and diagnostic admin endpoints still require an operation-specific admin action check.
-- `GHSA-796f-j7xp-hwf4`: `/rustfs/admin/v3/list-remote-targets` checked only that credentials existed and leaked replication target credentials. Lesson: replication target reads are privileged admin operations, and stored remote credentials require strict authz plus response redaction review.
-- `GHSA-xp32-gxq2-3v52`: console license metadata endpoint was public and exposed subject and expiration fields. Lesson: management metadata endpoints should require admin auth or return only coarse public status.
-
-### IAM import, service accounts, and privilege boundaries
-
-- `GHSA-566f-q62r-wcr8`: `ImportIam` accepted attacker-controlled service account `parent`, `claims`, `accessKey`, and `secretKey`, enabling persistent backdoor accounts under root. Lesson: imported IAM payloads are untrusted data and must be validated against privilege boundaries.
-- `GHSA-3495-h8r9-gfqg`: `ExportIAM` wrote regular-user and service-account secret keys into exported ZIP data. Lesson: IAM export is a credential-disclosure boundary; redact, seal, or strictly justify every exported secret before treating export permission as safe.
-- `GHSA-5354-r3w2-34m8`: `AddServiceAccount` checked `CreateServiceAccountAdminAction` but trusted caller-supplied `target_user`, allowing service accounts under the root parent. Lesson: service-account create paths must validate parent ownership or root/admin authority, not only the create action.
-- `GHSA-xgr5-qc6w-vcg9`: `deny_only=true` skipped allow checks and let restricted service accounts mint unrestricted children. Lesson: deny-only logic must never become implicit allow for privilege creation.
-- `GHSA-mm2q-qcmx-gw4w`: leaked service account access keys plus update-without-ownership formed an escalation chain. Lesson: service-account identifiers are security-sensitive because update APIs consume them.
-
-### STS, OIDC, and federation flows
-
-- `GHSA-5qfg-mf7r-jp3w` and `GHSA-3473-5353-xhwh`: `AssumeRoleWithWebIdentity` was reachable through unauthenticated `POST /` routing and could issue temporary credentials from crafted web identity input. Lesson: every STS route needs explicit SigV4 or trusted identity-provider validation before role assumption, and unauthenticated exemptions must be narrowed to the exact action with uniform failure responses.
-- `GHSA-jxrr-r6pv-h958`: unsigned JWT issuer data was decoded before verification to select an OIDC provider, and distinguishable failures could expose provider configuration. Lesson: web-identity routing may be unauthenticated, but pre-verification claims are untrusted routing hints; bound and rate-limit the request, normalize public errors, and verify signature, issuer, audience, and expiration before issuing credentials.
-- `GHSA-ccrv-v8v9-ch9q`, `GHSA-48rf-7j3q-3hfv`, and `GHSA-xvfh-7c9g-hpw2`: service-account-controlled material could self-sign JWT session tokens with forged policy claims, and missing `exp` was accepted for service-account tokens. Lesson: session tokens must be signed by a trusted issuer/key path, enforce required claims and expiration, and reject self-signed or principal-controlled tokens.
-- `GHSA-9pjf-w3c2-m32r`, `GHSA-4x2q-cpx9-9h26`, and `GHSA-xvpm-p3f7-34c3`: public OIDC authorize/callback flows trusted request `Host` or forwarded scheme when building credential-bearing redirects. Lesson: OIDC redirects must use configured allowlisted origins and trusted-proxy handling; never derive the post-login credential destination from direct client headers.
-- `GHSA-m479-9x88-94w6`, `GHSA-frwq-mfqx-83p8`, `GHSA-q9q8-rf9r-fg9f`, and `GHSA-j5c2-hhf7-6gf5`: OIDC validation accepted attacker-controlled discovery URLs because hostname checks rejected only literal forbidden IPs, allowing DNS rebinding SSRF. Lesson: outbound federation URL validation must resolve and classify hostnames at the connection boundary and reject loopback, private, link-local, and rebound addresses.
-
-### IAM policy conditions and external policy plugins
-
-- `GHSA-6r96-hmgc-726c`: request headers collided with lowercase server-derived condition keys such as `userid`, `groups`, `versionid`, and JWT/LDAP claims. Lesson: never let caller-controlled headers append to or replace server-derived policy context; reserve trusted condition keys and keep intentional request-header keys separate.
-- `GHSA-v9cp-qfw9-9pfp`: quantified negated string conditions applied negation after aggregation, transposing `ForAllValues` and `ForAnyValue` semantics. Lesson: push negation into the per-value predicate for quantified operators and test partially overlapping multi-value sets.
-- `GHSA-5w8r-p896-6vq2`: OPA policy mode skipped `ExistingObjectTag/*` loading, so tagged objects looked untagged to external policies. Lesson: external authorization plugins need the same object-tag and request context as built-in policy evaluation before they decide.
-- `GHSA-9wgh-5r68-49p7`: substituted policy-variable values could recreate `${...}` and loop inside a single resolution pass, never reaching the outer iteration limit. Lesson: bound substitutions within each pass and test cycles formed across otherwise literal resolver values; confirm whether attacker-controlled claims can reach those values before assigning remote impact.
-
-### S3 object actions, copy, multipart, and upload policy validation
-
-- `GHSA-g8w9-qw9q-fghr` and `GHSA-xm99-m3gq-83g8`: a valid presigned `PutObject` accepted extra `x-amz-*` headers omitted from `SignedHeaders`; `x-amz-copy-source` could turn that upload capability into a cross-bucket read performed as the signer. Lesson: a presigned URL is a bounded capability; inspect all received security-sensitive headers and reject unsigned ones before selecting the S3 operation or reaching storage.
-- `GHSA-3ppv-fx5m-m749`: explicit `versionId` reads and copy sources authorized `s3:GetObject` instead of `s3:GetObjectVersion`. Lesson: version-specific object access must select version-specific actions for direct reads, `CopyObject`, and `UploadPartCopy`, with tests proving the backend is not reached on denial.
-- `GHSA-cg33-pxfp-qcj3` and `GHSA-65wq-gjw8-xr9w`: `HeadObject` and object-tagging paths authorized current-object actions while operating on an explicit historical `versionId`. Lesson: select the corresponding version action for metadata and tag reads and writes too; audit every operation that accepts a version ID.
-- `GHSA-gjph-xcgw-4p7w`: `PutObject` and browser POST accepted a caller-supplied `versionId` and replaced an unlocked historical version with only `s3:PutObject`. Lesson: only replication-authorized writes may choose an existing version ID; ordinary uploads must not rewrite version history.
-- `GHSA-x298-9x87-fvjq`: anonymous `ListObjectVersions` fell back to `ListBucket` and returned before public-access-block gates. Lesson: compatibility fallbacks must converge on the same post-authorization checks as direct grants, especially `RestrictPublicBuckets` and anonymous data-plane denies.
-- `GHSA-mx42-j6wv-px98`: `UploadPartCopy` missed source authorization and allowed cross-bucket object exfiltration. Lesson: multipart copy must enforce the same source and destination contract as `CopyObject`.
-- `GHSA-wfxj-ph3v-7mjf`: `UploadPartCopy` checked source and destination independently but missed destination copy-source policy constraints. Lesson: source read and destination write checks are not sufficient when policy constrains allowed copy sources.
-- `GHSA-w5fh-f8xh-5x3p`: presigned POST accepted uploads without enforcing signed policy conditions. Lesson: parse and enforce all POST policy constraints server-side, including size, key prefix, and content type.
-
-### Protocol frontends and IAM parity
-
-- `GHSA-3g29-xff2-92vp`: FTP `RETR` and `SIZE`/`MDTM` read paths authenticated the user but skipped IAM before calling storage. Lesson: non-HTTP protocol frontends must enforce the same per-operation authorization as the S3 API before backend access.
-- `GHSA-g3vq-vv42-f647`: FTPS `MKD` called `create_bucket` without checking `s3:CreateBucket`. Lesson: protocol command handlers need action-specific checks even when sibling handlers already authorize correctly.
-- `GHSA-3p3x-734c-h5vx`: FTPS and WebDAV compared secret keys with early-return string equality, while FTPS also returned distinguishable invalid-user and invalid-password failures. Lesson: password-style protocol auth needs constant-time secret comparison, indistinguishable failures where practical, and rate limiting.
-- `GHSA-rhg6-jrf2-29c4`: WebDAV, FTP/FTPS, and SFTP used identity-policy checks with empty conditions and bypassed S3 bucket-policy explicit denies and condition-key denies. Lesson: alternate listeners must evaluate the same effective policy and trustworthy request context before storage access.
-
-### Filesystem paths and object key traversal
-
-- `GHSA-pq29-69jg-9mxc`: RPC `read_file_stream` joined untrusted paths under a volume directory without canonical boundary checks. Lesson: `PathBuf::join` plus length checks are not path security.
-- `GHSA-8r6f-hmq2-28rg`: object keys containing traversal sequences bypassed bucket/object authorization when mapped to filesystem paths. Lesson: reject traversal at object-key parsing and verify final storage paths remain under the expected bucket/key root.
-- `GHSA-f4vq-9ffr-m8m3`: Snowball auto-extract accepted archive entries such as `../victim-bucket/object`, authorized the raw attacker-bucket path, then storage path cleaning crossed bucket boundaries. Lesson: archive entries become object keys and need traversal rejection plus consistent authz/storage normalization before writes.
-- `GHSA-g7fq-6xjm-mxf4`: versioned `DeleteObjects` accepted dot-segment keys, authorized the literal prefix, then wrote delete markers under a resolved sibling prefix. Lesson: validate each batch key before authorization and storage normalization, using the same object-name rules as single delete.
-
-### Secrets, defaults, and cryptographic misuse
-
-- `GHSA-j59h-h7q5-q348`, `GHSA-3wm5-wpm5-hmfm`, `GHSA-6wc8-xm48-qhmx`, `GHSA-9gf3-jx4p-4xxf`, `GHSA-63xc-c3w3-m2cf`, and `GHSA-ch63-6q4v-hwp5`: RustFS shipped known default root credentials that could authenticate to S3, admin APIs, IAM, KMS, console, and token-signing surfaces. Lesson: root credentials must be operator-provided or generated per install; known defaults and warnings are not acceptable for network-reachable deployments.
-- `GHSA-h956-rh7x-ppgj`: gRPC used the hard-coded token `rustfs rpc` on both client and server. Lesson: source-visible shared tokens are authentication bypasses.
-- `GHSA-r5qv-rc46-hv8q`: internode RPC HMAC secret fell back to the public default `rustfsadmin`. Lesson: RPC/internode auth must fail closed instead of silently using public defaults.
-- `GHSA-75fx-qg6f-8rm7` and `GHSA-68cw-96m3-h2cf`: internode RPC secrets were derivable from known root credentials, making raw storage RPC signatures forgeable when explicit RPC secrets were unset. Lesson: RPC auth keys must be independent random secrets, never derived from S3 root credentials, and raw storage RPC should not share the public S3 listener without an internode-only boundary.
-- `GHSA-m77q-r63m-pj89`: STS JWTs used the root secret key as the shared token signing key, allowing token forgery when the root secret was known. Lesson: STS signing keys need key separation, rotation, and key IDs; do not reuse root credentials for JWT/HMAC signing.
-- `GHSA-923g-jp7v-f97f`: license verification embedded a production RSA private key and used private-key decryption as authenticity. Lesson: ship verifying/public keys only and use real signature verification.
-
-### Sensitive logging and debug output
-
-- `GHSA-r54g-49rx-98cr`: STS credentials were logged at info level. Lesson: generated credentials must never be logged in plaintext.
-- `GHSA-8cm2-h255-v749`: debug logs leaked session tokens, secret keys, JWT claims, and raw STS response bodies. Lesson: redaction must cover custom `Debug` implementations and dependency response-body logging.
-- `GHSA-333v-68xh-8mmq`: invalid RPC signature logging included the shared HMAC secret and expected signature. Lesson: error paths often leak secrets; never log raw secrets or derived authenticators.
-
-### RPC input validation and panic safety
-
-- `GHSA-gw2x-q739-qhcr`: malformed gRPC `GetMetrics` payloads reached `unwrap()` on deserialization and caused remote DoS. Lesson: every network/RPC deserialization failure returns an error, not a panic.
-- `GHSA-h956-rh7x-ppgj` and `GHSA-r5qv-rc46-hv8q`: weak RPC auth increased reachability of otherwise internal handlers. Lesson: panic bugs become more severe when internode auth is weak or defaulted.
-- `GHSA-c667-rgrv-99vj`: NodeService authentication signed the service prefix instead of the concrete generated method path, so valid metadata for one RPC could be replayed to another method during the timestamp window. Lesson: RPC HMAC payloads must bind exact gRPC method path, HTTP method surrogate, timestamp, and secret.
-
-### Browser, CORS, and console isolation
-
-- `GHSA-v9fg-3cr2-277j`: object preview rendered attacker-controlled HTML in a same-origin iframe, exposing console credentials stored in `localStorage`. Lesson: user content must be origin-isolated from the console and protected with `nosniff`, CSP, and strict content-type handling.
-- `GHSA-7gcx-wg4x-q9x6`: an incomplete preview fix reintroduced extension-based PDF detection and bypassed the sandboxed fallback for attacker-controlled content. Lesson: browser-surface fixes need regression tests for alternate viewers and file-type branches, and preview trust must come from validated content type plus sandboxing rather than object names.
-- `GHSA-x5xv-223c-8vm7`: default CORS reflected arbitrary origins with credentials. Lesson: never combine reflected origins with `Access-Control-Allow-Credentials: true`; default should be fail-closed.
-
-### Trusted proxy and source IP conditions
-
-- `GHSA-fc6g-2gcp-2qrq`: `aws:SourceIp` trusted client-supplied `X-Forwarded-For` or `X-Real-IP`. Lesson: forwarded IP headers are valid only behind configured trusted proxies; direct clients use socket peer IP.
-
-### SSE and on-disk storage invariants
-
-- `GHSA-xrrf-67jm-3c2r`: SSE metadata reported encryption while reader composition bypassed `EncryptReader` and stored plaintext. Lesson: test actual bytes on disk and wrapper order, not only API metadata.
-- `GHSA-wqmc-vjgv-jrpw`: SSE-C persisted and listed the plaintext MD5 as the ETag, allowing list-only users to confirm or brute-force low-entropy content and correlate equal plaintext across keys. Lesson: encrypted-object ETags must not expose deterministic plaintext fingerprints; protect them with per-object key material and keep listing, conditional, copy, and multipart semantics consistent.
-
-### Object Lock and retention invariants
-
-- `GHSA-j548-9grx-fh4f`: Object Lock enforcement treated unreadable, fabricated, or unparsable bucket metadata as absent configuration and allowed retained objects to be deleted or expired. Lesson: retention must fail closed unless Object Lock absence is authoritative, and every delete, lifecycle, scanner, force-delete, and default-retention path needs the same state distinction.
-- `GHSA-28w4-rp2p-636w` and `GHSA-c7hx-vr97-45rg`: an authorized `DeleteBucket` with `x-rustfs-force-delete: true` removed active COMPLIANCE-retained versions even though ordinary version deletion was denied. Lesson: bucket force-delete must enforce retention across every version, including against root/owner callers.
-- `GHSA-q23w-fp7p-472v`: `PutObjectRetention` policy conditions read `x-amz-object-lock-*` headers while the handler applied mode and date from the XML body. Lesson: authorize the parsed retention that will be stored, or reject mismatched header/body values; compare retention dates as instants so offsets cannot change the applied bound.
-
-### Serde deserialization and input validation
-
-- Reject unknown fields in untrusted S3 API XML/JSON, lifecycle, policy, and replication input where compatibility permits. Check the current type and supported payload fixtures; do not infer the repository's current coverage from an older audit. Where extra fields are part of the compatibility contract, validate security-critical values explicitly.
-- `#[serde(default)]` on security-critical fields silently accepts missing values as zero/empty. Lesson: when a field has security implications (retention days, permissions, limits), validate the deserialized value explicitly rather than relying on defaults.
-- Integer fields deserialized from user input and cast with `as` (e.g., `i32 as u32`) can wrap negative values to large positives. Lesson: validate ranges before casting; use `try_into()` with a typed error, or clamp only when the domain explicitly requires saturation.
-- XML config typos (e.g., `"NoncurentDays"` instead of `"NoncurrentDays"`) are silently accepted when `deny_unknown_fields` is absent. Lesson: strict deserialization prevents silent misconfiguration that could cause data loss or unexpected retention behavior.
-
-## Useful Search Seeds
-
-Use these targeted searches when a diff touches security-sensitive code:
-
-```bash
-rg -n "validate_admin_request|check_permissions|AdminAction::|deny_only|is_allowed" rustfs crates
-rg -n "authorize_operation|FtpsDriver|SftpDriver|RETR|MKD|SIZE|MDTM|CreateBucket|GetObject|HeadObject" crates/protocols rustfs
-rg -n "UploadPartCopy|upload_part_copy|CompleteMultipart|PostObject|presign|SignedHeaders|x-amz-copy-source|collect_signed_headers|content-length-range|starts-with" rustfs crates
-rg -n "ListBucketVersions|GetObjectVersion|versionId|VersionId|ExistingObjectTag|ForAllValues|ForAnyValue|POLICY_PLUGIN|opa" rustfs crates
-rg -n "normalize_extract_entry_key|Snowball|auto-extract|PathBuf::join|canonicalize|\\.\\.|x-forwarded-for|x-real-ip|SourceIp" rustfs crates
-rg -n "DEFAULT_SECRET|DEFAULT_ACCESS|TEST_PRIVATE_KEY|rustfs rpc|RUSTFS_RPC_SECRET" rustfs crates
-rg -n "TONIC_RPC_PREFIX|verify_rpc_signature|check_auth|NodeServiceServer|x-rustfs-signature" rustfs crates
-rg -n "debug!|trace!|info!|error!|\\?resp|\\?merged_config|session_token|secret_key" rustfs crates
-rg -n "HashReader|EncryptReader|try_resolve_etag|ETag|SSE|server-side encryption|Access-Control-Allow-Credentials|Origin" rustfs crates
-rg -n "ObjectLock|object_lock|retention|COMPLIANCE|GOVERNANCE|delete_prefix|lifecycle|scanner" rustfs crates
-rg -n "deny_unknown_fields|serde.default|as u32|as usize|as i32" rustfs crates
-```
-
-## Minimum Regression Test Expectations
-
-- Authz fixes: include unauthenticated, valid low-privilege, wrong-action, correct-action, owner, non-owner, and root/admin cases as applicable.
-- Protocol frontend authz fixes: include denied `RETR`, `SIZE`/`MDTM`, `MKD`, bucket probe, and sibling allowed-operation cases; test bucket-policy explicit denies and condition-key denies on each enabled listener against an S3 control, and assert denied paths do not reach storage.
-- IAM fixes: include import/update/list service-account cases with attacker-controlled parent, claims, access key, secret key, and policy.
-- Copy/upload fixes: include cross-bucket, cross-user, source-denied, destination-denied, copy-source-condition, and multipart completion cases.
-- Presigned upload fixes: include a valid presign with extra unsigned tagging, redirect, storage-class, and cross-bucket copy-source headers; require rejection before source or destination storage access, and verify explicitly signed equivalents still work.
-- Version-action fixes: include historical UUID, explicit current version, `null`, range, partNumber, presigned, STS/session, service-account, anonymous bucket-policy, copy source, multipart-copy source, `HeadObject`, and tag read/write cases.
-- Version-write fixes: a `PutObject`-only principal must not replace an existing version through `PutObject` or browser POST `?versionId=`; verify replication-authorized writes and versioned, suspended, and unversioned buckets separately.
-- Policy-condition fixes: include reserved-key header collisions, missing keys, partially overlapping multi-value sets, cyclic variable substitutions, plugin mode, and built-in policy mode.
-- Path fixes: include encoded traversal, absolute path, nested traversal, archive entries with `..`, valid object keys that resemble traversal text but should be rejected, and canonical bucket/prefix boundary checks; test per-key `DeleteObjects` errors and prefix-scoped denials on versioned buckets.
-- Logging fixes: assert redacted output for structs and response bodies that may contain credentials.
-- IAM export fixes: assert exported archives omit plaintext user and service-account secrets unless the format deliberately encrypts or seals them.
-- RPC auth fixes: include captured metadata replay across two concrete methods, stale timestamps, wrong path, wrong method surrogate, wrong secret, and valid same-method calls.
-- Browser/CORS fixes: assert no credentials on reflected/default origins, correct behavior for explicit allowlists, and no same-origin script execution for previewed object content.
-- SSE fixes: inspect stored bytes and verify API metadata, read-back behavior, and on-disk ciphertext together; for ETags, upload equal plaintext under different keys and assert list-only callers cannot derive or correlate the plaintext digest, including multipart objects.
-- Object Lock fixes: include unreadable metadata, fabricated metadata defaults, unparsable config, confirmed absent config, COMPLIANCE/GOVERNANCE retention, lifecycle expiry, scanner sweeps, and force-delete paths; an authorized force-delete must preserve active COMPLIANCE versions. For `PutObjectRetention`, test conflicting header/body values and offset dates against policy bounds and read-back retention.

.config/e2e-distributed-selection.txt

@@ -1,2 +1,2 @@
-sha256-linux=14becc458b9eb24add510d94ee6cd04ba1fb19907d598effe1524e5242400d10
-sha256-darwin=14becc458b9eb24add510d94ee6cd04ba1fb19907d598effe1524e5242400d10
+sha256-linux=eb33067d90ee66c0a97e8f3658e46c152e47f897d2fbfdc6cb5ad93cfa55414b
+sha256-darwin=eb33067d90ee66c0a97e8f3658e46c152e47f897d2fbfdc6cb5ad93cfa55414b

.config/e2e-full-selection.txt

@@ -1,2 +1,2 @@
-sha256-darwin=596ca4ce6661a1ccd5c8278a7781fd46922ebe5e23377a254471ec5f4fabfeb0
-sha256-linux=1f3d1d167fbd58a72b46a255edb111c7ee0abdb1ae37ac00400d1239cacdd5e1
+sha256-darwin=67251cb7d9978d38e899268cac4d7898297ff7e5b5c48c9c1546dc760b93568d
+sha256-linux=673da76c2e5710c4622fe2fa32d2b297730895769487fcf1152ee1c2366f32f4

.config/e2e-repl-nightly-selection.txt

@@ -1 +1 @@
-sha256=4eac269f9a59b2f9be4e7a87ea6aa38454f5ebe10ede35c9b8d4218137d2cd35
+sha256=e4ee3898b840a8374cb02c64342cc15f8d7ecd1becaf4b4a2a4657290f9e7ff1

.config/e2e-smoke-selection.txt

@@ -1 +1 @@
-sha256=57ffdd620dec9cba08a411b8ad681f4eb906cf5764ffed3c7c992084f5fc6092
+sha256=e9dc896774122a304e2b8b90f28c82d4d4e0af8ded36a82b147b30945ec994e3

.config/ecstore-required-tests.json

@@ -229,7 +229,7 @@
     {
       "invariant": "list-objects-shared-corpus-orphan-cleanup-requires-authoritative-evidence",
       "suite": "rustfs-ecstore",
-      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_orphan_cleanup_requires_authoritative_evidence"
+      "name": "store::list_objects::differential_tests::list_objects_shared_corpus_orphan_listing_preserves_inputs"
     },
     {
       "invariant": "list-objects-shared-corpus-consumer-drop-joins-real-disk-producer",

.config/make/lint-fmt.mak

@@ -70,11 +70,6 @@ s3s-footprint-check: ## Check the s3s dependency footprint ratchet stays frozen
 	@echo "📦 Checking s3s footprint ratchet..."
 	./scripts/check_s3s_footprint.sh
 
-.PHONY: test-count-ratchet-check
-test-count-ratchet-check: ## Check no workspace crate lost tests or gained #[ignore] against the committed baseline
-	@echo "🧪 Checking per-crate test-count ratchet..."
-	./scripts/check_test_count_ratchet.sh
-
 .PHONY: fips-wording-check
 fips-wording-check: ## Check docs and crates/kms do not over-claim crypto capabilities
 	@echo "📣 Checking cryptographic capability wording guard..."

.config/make/pre-commit.mak

@@ -20,11 +20,11 @@ planning-docs-check: ## Check that no planning-type documents are committed
 	./scripts/check_no_planning_docs.sh
 
 .PHONY: pre-commit
-pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check test-count-ratchet-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
+pre-commit: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check quick-check ## Run fast pre-commit checks without clippy/full tests
 	@echo "✅ All pre-commit checks passed!"
 
 .PHONY: pre-pr
-pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check test-count-ratchet-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
+pre-pr: fmt-check unsafe-code-check architecture-migration-check logging-guardrails-check error-other-ratchet-check tokio-io-uring-check extension-schema-check body-cache-whitelist-check s3s-footprint-check fips-wording-check embedded-secrets-check test-wiring-check doc-paths-check planning-docs-check log-analyzer-rules-check offline-enrollment-e2e-check clippy-check test ## Run full pre-PR checks with clippy and tests
 	@echo "✅ All pre-PR checks passed!"
 
 .PHONY: dev-check

.config/make/tests.mak

@@ -50,10 +50,9 @@ script-tests: ## Run shell script tests
 	./scripts/test_fuzz_runner.sh
 	./scripts/test_python_bin.sh
 	./scripts/check_embedded_secrets.sh --self-test
-	./scripts/check_s3s_footprint.sh --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/check_test_wiring.py --self-test
-	$(RUSTFS_PYTHON_BIN) ./scripts/codemods/s3_error_to_s3_types.py --self-test
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_e2e_binary.py
+	$(RUSTFS_PYTHON_BIN) ./scripts/test_inspect_btf_prototypes.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_release_build_workflow.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_release_glibc.py
 	$(RUSTFS_PYTHON_BIN) ./scripts/test_upgrade_workflow.py

.config/nextest.toml

@@ -128,7 +128,7 @@ setup = 'ecstore-large-stack'
 # path on a real store; measured with RUST_MIN_STACK on 2026-10-15 they need
 # 2016-2040 KiB of the 2 MiB libtest default and overflowed on Linux CI.
 [[profile.default.scripts]]
-filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
+filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::plugin_attempt::v2_tests::delegation_v2_persists_and_vends_with_real_parent_intersection|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
 setup = 'ecstore-base-stack'
 
 [[profile.default.scripts]]
@@ -345,12 +345,6 @@ threads-required = "num-test-threads"
 filter = 'package(rustfs-ecstore) & test(=store::init::tests::dispatch_manifest_rollback_bounded_concurrency_reaches_tail_behind_slow_member)'
 threads-required = "num-test-threads"
 
-# These short-deadline real-disk progress probes need the test run's capacity
-# for scanner and object-lock handoffs; keep their assertions and deadlines.
-[[profile.default.overrides]]
-filter = '(package(rustfs-scanner) & (test(=scanner::tests::cycle_persist_failure::later_cycle_post_commit_failure_reloads_durable_state) | test(=scanner::tests::running_main_loop_catches_up_pause_cleared_after_startup_observe))) | (package(rustfs-ecstore) & test(=store::init::tests::restored_transitioned_delete_causally_enqueues_free_version))'
-threads-required = "num-test-threads"
-
 # ---------------------------------------------------------------------------
 # ci profile — the strict CI gate (ci.yml `cargo nextest run --profile ci`)
 # ---------------------------------------------------------------------------
@@ -419,19 +413,13 @@ threads-required = "num-test-threads"
 filter = 'package(rustfs-scanner) & (test(/^scanner::backlog::tests::native_/) | test(=scanner_folder::tests::mrf_ownership::mrf_ownership_manager_completion_preserves_scanner_pending) | test(/^scanner_io::tests::service_cohort::/))'
 threads-required = "num-test-threads"
 
-# Keep short-deadline real-disk progress probes from competing with unrelated
-# tests for scanner and object-lock capacity. Assertions and deadlines stay on.
-[[profile.ci.overrides]]
-filter = '(package(rustfs-scanner) & (test(=scanner::tests::cycle_persist_failure::later_cycle_post_commit_failure_reloads_durable_state) | test(=scanner::tests::running_main_loop_catches_up_pause_cleared_after_startup_observe))) | (package(rustfs-ecstore) & test(=store::init::tests::restored_transitioned_delete_causally_enqueues_free_version))'
-threads-required = "num-test-threads"
-
 [[profile.ci.scripts]]
 filter = 'package(rustfs-ecstore) & test(/^(bucket::lifecycle::bucket_lifecycle_ops::tests::manual_transition_worker_result_recovery_marks_unknown_for_corrupt_marker|services::rebalance::entry::tests::real_rebalance_run_fence_loss_blocks_multipart_publication|store::init::tests::(batch_transitioned_delete_uses_free_version_per_item|decommission_entry_(allows_free_version_consumed_before_source_lock|rejects_subquorum_free_version_conflict_and_retains_source|skips_cleanup_only_marker_when_free_version_is_present)|dispatched_tier_delete_recovery_(checks_later_pool_then_commits_after_source_removal|finds_directory_source_on_encoded_set|retains_journal_on_source_metadata_error)|force_tier_remove_blocks_on_physical_free_version_hidden_by_other_pool|legacy_unknown_transition_delete_falls_back_for_single_batch_and_blocks_prefix|multi_pool_(recursive_prefix_rejects_legacy_or_hidden_merge_loser_before_delete|same_remote_tuple_(batch|single)_delete_waits_for_all_sources|same_tuple_recursive_prefix_uses_one_journal_owner|transitioned_delete_persists_one_free_version_per_remote_tuple)|recursive_prefix_partial_(pool|set)_failure_keeps_prepared_cleanup_owners|restored_transitioned_delete_uses_free_version_as_cleanup_owner|stable_transitioned_recursive_prefix_delete_uses_journal_owners|suspended_null_transition_delete_uses_free_version_as_sole_owner|tier_mutation_peer_handler_applies_prepare_commit_and_abort_idempotently|transition_response_loss_persists_unknown_outcome_for_provider_recovery|transition_transaction_recovery_(drops_record_after_confirmed_local_commit|keeps_cleanup_pending_local_commit)|transitioned_delete_(free_version_replays_after_store_restart|local_quorum_failure_rolls_back_without_cleanup_owner|uses_free_version_as_cleanup_owner)|versioned_delete_marker_keeps_transitioned_source_and_remote_object|versioned_explicit_transition_delete_preserves_other_version_then_allows_bucket_delete))$/)'
 setup = 'ecstore-large-stack'
 
 # These cross-crate real-storage fixtures need the same bounded stack as ECStore tests.
 [[profile.ci.scripts]]
-filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
+filter = '(package(rustfs) & test(/^(admin::handlers::replication::target_repair_tests::[a-z_]+|admin::handlers::plugin_attempt::v2_tests::delegation_v2_persists_and_vends_with_real_parent_intersection|admin::handlers::site_replication::tests::durable_iam_state_apply_iam_item_rejects_a_stale_recreate_after_a_replicated_delete|admin::service::federated_identity::tests::(binding_keeps_explicit_iam_mapping_strict|markerless_legacy_oidc_session_crosses_auth_and_iam_deletion_boundaries)|storage::scanner_dirty_journal::tests::(clear_callback_keeps_unknown_coverage_until_all_pending_work_is_confirmed|owned_journals_keep_peer_records_after_interleaved_writes_and_clear))$/)) | (package(rustfs-heal) & ((binary(heal_b5_versioned_regression_test) & test(/^serial_tests::(test_heal_unversioned_bucket_e2e|test_historical_null_recursive_heal_restores_physical_versions|test_recursive_deep_heal_reports_stale_delete_marker_repair)$/)) | (binary(usage_observation_read_repair_test) & test(=usage_observation_read_repair_survives_snapshot_cleanup)))) | (package(rustfs-test-utils) & test(=data_usage_snapshot_tests::data_usage_snapshot_recovery_wires_primary_backup_and_negative_cache))'
 setup = 'ecstore-base-stack'
 
 [[profile.ci.scripts]]
@@ -684,14 +672,6 @@ test-group = 'ecstore-serial-flaky'
 # the PR-lane e2e coverage for scanner usage snapshots consumed by quota and
 # admin surfaces. It uses the same single-node, random-port, isolated-temp-dir
 # fixture as the existing smoke modules.
-#
-# Dual-stack differential (rustfs/backlog#2740): `dual_stack_differential::`
-# starts two single-node servers from the same binary, one per
-# RUSTFS_S3_STACK value, on random ports with isolated temp dirs, and runs one
-# SDK script covering every operation of `impl S3 for FS` against both. Its
-# comparator unit tests need no server. It meets the admission criteria
-# unchanged; the module clause admits a new differential test without another
-# filter edit.
 [profile.e2e-smoke]
 default-filter = """
     package(e2e_test) & (
@@ -700,7 +680,6 @@ default-filter = """
         | test(/^reliant::lifecycle::/)
         | test(/^conditional_delete_objects_test::/)
         | test(/^reliant::tiering::/)
-        | test(/^dual_stack_differential::/)
         | test(/^upgrade_compatibility_test::upgrade_write_readiness_tests::/)
         | test(/^on_demand_migration::(get_basic_test::(get_miss_pulls_inline_and_serves_locally_afterwards|head_miss_answers_from_the_source_without_persisting)|interaction_test::test_odm_admin_config_is_redacted_and_status_counts_match_the_source)$/)
     )

.config/s3s-edge-allowlist.txt

@@ -1,37 +0,0 @@
-# Files that may still reference s3s (`s3s::` or `s3s_sigv4`) when Phase 1 of
-# the gateway migration ends (rustfs/backlog#2734 task T0.6, rustfs/backlog#2741).
-# Consumed only by `scripts/check_s3s_footprint.sh --mode allowlist`.
-#
-# One entry per line: a repository-relative file path or a git `:(glob)`
-# pathspec. `**` matches across directories, `*` stays within one path
-# component, and a directory entry matches everything under it. Blank lines
-# and lines starting with `#` are ignored; an entry may not contain whitespace,
-# so a note belongs on its own comment line.
-#
-# Every Rust file listed by `git grep -l '\bs3s\(::\|_sigv4\)' -- '*.rs'` that
-# no entry matches fails the allowlist mode. Task T1.9 switches CI to that
-# mode; until then `--dry-run` prints the files still outside this list, which
-# is the Phase 1 tracking list.
-#
-# Never add crates/** other than crates/e2e_test: the engine crates leave s3s
-# entirely, and the script rejects such an entry. Only the s3s-facing edge of
-# the rustfs binary and the e2e oracle keep it.
-
-# rustfs binary: the s3s-facing edge that survives Phase 1
-rustfs/src/storage/ecfs.rs
-rustfs/src/storage/s3_compat/**
-rustfs/src/storage/access.rs
-rustfs/src/auth.rs
-rustfs/src/error.rs
-rustfs/src/storage_api.rs
-rustfs/src/app/storage_api.rs
-rustfs/src/admin/router.rs
-rustfs/src/app/metadata_route.rs
-rustfs/src/server/http.rs
-rustfs/src/server/layer.rs
-rustfs/src/server/hybrid.rs
-rustfs/src/protocols/client.rs
-
-# e2e oracle: the test target that verifies S3 behaviour against s3s
-crates/e2e_test/src/fake_s3_target/**
-crates/e2e_test/src/**

.config/s3s-footprint-baseline.txt

@@ -1,17 +0,0 @@
-# Exact baseline for the direct s3s footprint, consumed only by
-# scripts/check_s3s_footprint.sh (rustfs/backlog#1677; rustfs/backlog#2734 T0.4).
-#
-# Each value is the exact measurement of the tree, not a ceiling. CI fails when a
-# measured counter differs from its line here in either direction:
-#   - a larger measurement means new code widened the s3s surface: route it
-#     through the gateway abstractions instead; never raise a value here;
-#   - a smaller measurement means the PR shrank the surface: lower the matching
-#     value to the measured number in the same PR.
-#
-# files           Rust files referencing s3s paths, repo-wide minus crates/e2e_test
-# s3_error_lines  lines holding s3_error!, same scope, minus those the script
-#                 header's rule clears (rustfs_s3_types's macro); fails closed
-# ecstore_files   Rust files referencing s3s paths under crates/ecstore/src
-files=65
-s3_error_lines=289
-ecstore_files=0

.config/test-count-baseline.txt

@@ -1,64 +0,0 @@
-# Per-crate test-count baseline consumed by scripts/check_test_count_ratchet.sh
-# (rustfs/backlog#2734 section 1 rule 3; T0.3 in rustfs/backlog#2738).
-#
-# One row per workspace member: <package>, tests, ignored, tab-separated.
-# tests counts #[test], #[tokio::test], #[tokio::test(...)] and #[test_case(...)]
-# attribute lines under the package directory; ignored counts #[ignore] and
-# #[ignore = "..."]. Both columns are exact: CI fails when a crate's tests fall
-# below or rise above its row, or when its ignored count changes. Raise a row in
-# the PR that adds tests; lowering one is a reviewed statement that tests were
-# intentionally removed. Regenerate with:
-#   scripts/check_test_count_ratchet.sh --update
-e2e_test	988	20
-rustfs	5979	64
-rustfs-audit	70	0
-rustfs-checksums	26	0
-rustfs-common	36	0
-rustfs-concurrency	54	0
-rustfs-config	43	0
-rustfs-connect-symbol-catalog	6	0
-rustfs-credentials	24	0
-rustfs-crypto	67	0
-rustfs-data-usage	55	0
-rustfs-ecstore	7312	2
-rustfs-extension-schema	23	0
-rustfs-filemeta	352	0
-rustfs-heal	1001	3
-rustfs-heal-contracts	7	0
-rustfs-iam	394	0
-rustfs-io-core	49	0
-rustfs-io-metrics	151	0
-rustfs-keystone	29	0
-rustfs-kms	715	25
-rustfs-license	2	0
-rustfs-lifecycle	155	0
-rustfs-lock	138	0
-rustfs-log-analyzer	90	0
-rustfs-madmin	210	0
-rustfs-notify	137	0
-rustfs-object-capacity	92	0
-rustfs-object-data-cache	140	0
-rustfs-obs	381	0
-rustfs-policy	733	0
-rustfs-protocols	739	1
-rustfs-protos	73	0
-rustfs-replication	209	0
-rustfs-rio	206	0
-rustfs-rio-v2	34	6
-rustfs-s3-client	69	0
-rustfs-s3-ops	8	0
-rustfs-s3-types	103	0
-rustfs-s3select-api	207	0
-rustfs-s3select-query	189	2
-rustfs-scanner	1213	20
-rustfs-scanner-metrics	90	0
-rustfs-security-governance	20	0
-rustfs-signer	31	0
-rustfs-storage-api	34	0
-rustfs-targets	603	26
-rustfs-test-utils	1	0
-rustfs-tls-runtime	43	0
-rustfs-trusted-proxies	68	0
-rustfs-utils	247	0
-rustfs-zip	30	0
-zstd	11	0

.github/actions/quick-checks/action.yml

@@ -86,10 +86,6 @@ runs:
       shell: bash
       run: ./scripts/check_s3s_footprint.sh
 
-    - name: Check per-crate test-count ratchet
-      shell: bash
-      run: ./scripts/check_test_count_ratchet.sh
-
     - name: Check cryptographic capability wording
       shell: bash
       run: ./scripts/check_fips_wording.sh

.github/actions/schedule-failure-issue/README.md

@@ -74,7 +74,6 @@ Notes:
 - `.github/workflows/e2e-s3tests.yml` (weekly full s3-tests sweep)
 - `.github/workflows/mint.yml` (weekly multi-SDK mint run)
 - `.github/workflows/fuzz.yml` (nightly fuzz corpus jobs)
-- `.github/workflows/performance-ab.yml` (nightly warp A/B gate)
 - **Pending:** the ci-7 e2e nightly-full workflow does not exist yet
   (backlog#1149 ci-7). When it lands, wire it up with the snippet above.
 

.github/scheduled-validations.json

@@ -18,7 +18,6 @@
     "never_ran_grace_until": "2026-09-08T00:00:00Z"
   },
   { "workflow": ".github/workflows/nightly-gnu.yml", "max_age_hours": 36 },
-  { "workflow": ".github/workflows/performance-ab.yml", "max_age_hours": 36 },
   {
     "workflow": ".github/workflows/runner-hygiene.yml",
     "max_age_hours": 792,

.github/workflows/README.md

@@ -7,7 +7,7 @@
 
 This repository is public. For a public repository, GitHub's standard hosted runners (`ubuntu-latest`, `windows-latest`, `macos-latest`) cost nothing and have no minute cap, whatever plan the organisation is on. GitHub bills only two things: larger hosted runners, which have custom labels and are not used here, and standard runners in private repositories.
 
-Every other label in this repository is capacity RustFS pays for by the hour whether or not a job is running well:
+Every other label in this repository is capacity RustFS pays for. ARC pods and lab machines accrue cost while the machine exists. A Ubicloud label is metered for the minutes a job holds its VM:
 
 | Label | Paid by | What it is | Use it for |
 |---|---|---|---|
@@ -15,8 +15,8 @@ Every other label in this repository is capacity RustFS pays for by the hour whe
 | `windows-latest`, `macos-latest` | nobody | GitHub-hosted standard runners | Windows and macOS targets |
 | `sm-standard-2` | RustFS | ARC pod, Ubuntu 24.04, minimal image: no `gh`, no C toolchain; Docker only on the `dind-` label | only a job that a hosted runner cannot run (see the placement rule) |
 | `sm-standard-4` | RustFS | ARC pod, Ubuntu 24.04, 14 CPUs and 28 GB (the label is not the core count), large disk | workspace-scale Cargo builds |
-| `sm-standard-4-arm` | RustFS | ARC pod, aarch64; reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds |
-| `dind-sm-standard-2` | RustFS | ARC pod with Docker-in-Docker | image builds that need `buildx` and a registry push |
+| `ubicloud-standard-16-arm` | RustFS | Ubicloud VM, alias of `ubicloud-standard-16-arm-ubuntu-2404`: Ubuntu 24.04 arm64, 16 vCPU, 48 GB, 300 GB disk. Reached through the `build.yml` matrix, so `actionlint.yaml` does not list it | native aarch64 release builds (`linux-aarch64-musl`, `linux-aarch64-gnu`) |
+| `dind-sm-standard-2` | RustFS | ARC pod with Docker-in-Docker | Docker workloads with a measured resource or environment requirement beyond the hosted runner |
 | `smoke-testing`, `pf-testing` | RustFS | long-lived lab machines | the packaged functional chain and performance suites |
 
 The self-hosted pods and the hosted image are both Ubuntu 24.04, so a binary built on one runs on the other; `ci.yml` builds the debug binary on `sm-standard-4` and runs it on both pools.
@@ -27,11 +27,15 @@ A job runs on `ubuntu-latest` unless it has one of these reasons, and the reason
 
 1. **Disk.** It restores or produces the workspace `target/`: `cargo build` or `cargo test` with `--all-targets`, `cargo nextest run --all`, or any reader of the `ci-dev` / `ci-feat-*` caches. The workspace test build leaves about 64 GB in `target/`; a hosted runner has about 14 GB.
 2. **Critical-path CPU.** It is a build that several downstream lanes wait on and would blow its timeout at 4 vCPU (`build-rustfs-debug-binary` gates the e2e and s3-tests lanes).
-3. **Hardware.** Native aarch64, Docker-in-Docker, or the lab machines.
+3. **Hardware or environment.** Native aarch64 or access to the lab machines. Docker alone is not a reason: hosted Ubuntu ships Docker and supports the QEMU and Buildx setup actions.
 4. **Cache ownership.** It reads a `rust-cache` key that a self-hosted job writes (see the cache rule below).
 
 Jobs that do only the following have no reason to be self-hosted: shell or Python scripts, `cargo fmt`, `cargo deny`, typos, actionlint, API calls (`gh`, dependency review, issue and stale bots, CLA checks), downloading an artifact and running a prebuilt binary, or a `cargo build -p rustfs` with its own cache key. `e2e-distributed.yml` and `e2e-upgrade.yml` build the server and the `e2e_test` harness on `ubuntu-latest` inside their budgets, so "it compiles something" alone is not a reason.
 
+The functional chain's `prepare` and `complete-chain` jobs, the performance runner availability check, and runner hygiene use `ubuntu-latest`: they inspect APIs or artifacts, not the lab hosts. The actual functional and performance suites remain on their lab runners. Functional-chain health collection stays on the operator-managed `smoke-testing` runner as part of the automation environment. Docker image packaging also uses `ubuntu-latest` for both musl and glibc variants; it downloads release binaries rather than compiling the workspace, and still sets up QEMU and Buildx explicitly.
+
+Moving these jobs off AKS reduces capacity demand, not necessarily the Azure bill immediately. VM costs fall only when nodes scale down or smaller capacity replaces them.
+
 A hosted runner is also the only choice for a few things the pods cannot do: `mount -o loop` and `mount -t tmpfs` fail inside the ARC pods, and a container runtime cannot guarantee that `io_uring_setup` is allowed. `uring-integration` and `e2e-distributed` are on `ubuntu-latest` for that reason, not for cost.
 
 ## Cache rule
@@ -40,7 +44,7 @@ A hosted runner is also the only choice for a few things the pods cannot do: `mo
 
 - Every `shared-key` has exactly one writer. `cache-warm.yml` owns `ci-dev`, `ci-feat-rio`, `ci-feat-proto` and `ci-uring`; every other `./.github/actions/setup` call states `cache-save-if` explicitly, and `scripts/security/check_cache_save_if.sh` fails the build when one does not.
 - All readers of a key run on the same image as its writer. Keys that live on hosted runners say so in their name (`ci-uring` is warmed on `ubuntu-latest`; `ci-e2e-distributed-hosted`, `e2e-upgrade-server-hosted`, `e2e-upgrade-tests-hosted`).
-- Moving a job between pools therefore means renaming its key. The new key is empty until a run from `main` (a scheduled run, or a manual dispatch of that workflow from `main`) saves it; until then that lane does a cold build.
+- Moving a job between pools therefore means renaming its key. The new key is empty until a run from `main` (a scheduled run, or a manual dispatch of that workflow from `main`) saves it; until then that lane does a cold build. The `build.yml` aarch64 release legs append `-ubicloud-arm` for this reason. `cache-warm.yml` does not write those keys.
 - `CARGO_BUILD_JOBS` and every other `CARGO*`, `CC*`, `CFLAGS*`, `CXX*`, `CMAKE*` and `RUST*` variable is hashed into the key. `ci.yml` sets `CARGO_BUILD_JOBS` at step level for that reason; promoting it to job level rotates the key for every lane that shares it.
 
 ## Where self-hosted minutes leak
@@ -62,7 +66,7 @@ gh api "repos/rustfs/rustfs/actions/runs/<run_id>/jobs?per_page=100" \
   --jq '.jobs[] | "\(.conclusion)\t\(.labels | join(","))\t\(.started_at)\t\(.completed_at)\t\(.name)"'
 ```
 
-Only rows whose label starts with `sm-`, `dind-`, `smoke-` or `pf-` cost anything. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
+Only rows whose label starts with `sm-`, `dind-`, `ubicloud-`, `smoke-` or `pf-` cost anything. At the October 2026 review, a full pull-request run of `ci.yml` spent about 150 self-hosted minutes, a third of it in jobs that never compiled; treat that as the scale of what a placement mistake costs, not as a current figure.
 
 ## Checklist for a new or moved job
 

.github/workflows/audit.yml

@@ -29,7 +29,6 @@ on:
       - 'scripts/test_package_service_scripts.sh'
       - 'scripts/test_package_versions.sh'
       - 'scripts/test_docker_workflow.py'
-      - 'scripts/security/check_performance_ab_workflow.sh'
       - 'scripts/security/check_preview_release_workflow.sh'
       - 'scripts/security/check_tier_artifact_workflow.sh'
       - 'scripts/security/check_workflow_pins.sh'
@@ -48,7 +47,6 @@ on:
       - 'scripts/test_package_service_scripts.sh'
       - 'scripts/test_package_versions.sh'
       - 'scripts/test_docker_workflow.py'
-      - 'scripts/security/check_performance_ab_workflow.sh'
       - 'scripts/security/check_preview_release_workflow.sh'
       - 'scripts/security/check_tier_artifact_workflow.sh'
       - 'scripts/security/check_workflow_pins.sh'
@@ -155,9 +153,6 @@ jobs:
       - name: Check preview release workflow policy
         run: ./scripts/security/check_preview_release_workflow.sh
 
-      - name: Check performance A/B workflow trust boundary
-        run: ./scripts/security/check_performance_ab_workflow.sh
-
       - name: Check tier evidence workflow isolation
         run: ./scripts/security/check_tier_artifact_workflow.sh
 

.github/workflows/btf-prototype-reference.yml

@@ -0,0 +1,75 @@
+name: BTF prototype reference
+
+on:
+  pull_request:
+    paths:
+      - "scripts/inspect_btf_prototypes.py"
+      - "scripts/test_inspect_btf_prototypes.py"
+      - "scripts/fixtures/btf_prototypes.c"
+      - ".github/workflows/btf-prototype-reference.yml"
+  workflow_dispatch:
+
+permissions:
+  contents: read
+
+concurrency:
+  group: btf-prototype-reference-${{ github.ref }}
+  cancel-in-progress: true
+
+jobs:
+  reference:
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
+        with:
+          persist-credentials: false
+      - uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
+        with:
+          python-version: "3.12"
+      - name: Check prototype and CLI contracts
+        run: python3 -B scripts/test_inspect_btf_prototypes.py
+      - name: Install only missing native reference tools
+        run: |
+          select_bpftool() {
+            local candidate
+            local candidates=()
+            if command -v bpftool >/dev/null; then
+              candidates+=("$(command -v bpftool)")
+            fi
+            for candidate in /usr/lib/linux-tools-*/bpftool /usr/lib/linux-tools/*/bpftool; do
+              if [[ -x "$candidate" ]]; then
+                candidates+=("$candidate")
+              fi
+            done
+            while IFS= read -r candidate; do
+              if [[ -x "$candidate" ]] && file -Lb "$candidate" | grep -q 'ELF' \
+                && "$candidate" version >/dev/null 2>&1; then
+                printf '%s\n' "$candidate"
+                return 0
+              fi
+            done < <(printf '%s\n' "${candidates[@]}" | sort -Vr)
+            return 1
+          }
+          packages=()
+          command -v clang >/dev/null || packages+=(clang)
+          command -v llvm-objcopy >/dev/null || packages+=(llvm)
+          command -v gcc >/dev/null || packages+=(gcc)
+          select_bpftool >/dev/null || packages+=(linux-tools-generic)
+          if (( ${#packages[@]} )); then
+            sudo apt-get update
+            sudo apt-get install --no-install-recommends -y "${packages[@]}"
+          fi
+          # File parsing needs a working binary, not a host-kernel wrapper.
+          selected_bpftool="$(select_bpftool)"
+          "$selected_bpftool" version
+          printf 'BPFTOOL=%s\n' "$selected_bpftool" >> "$GITHUB_ENV"
+      - name: Compare declarations and layouts with independent references
+        run: |
+          clang -target bpf -O2 -g -c scripts/fixtures/btf_prototypes.c -o "$RUNNER_TEMP/prototypes.o"
+          llvm-objcopy --dump-section ".BTF=$RUNNER_TEMP/prototypes.btf" "$RUNNER_TEMP/prototypes.o"
+          gcc -std=c11 -Wall -Wextra -Werror -DRUSTFS_BTF_LAYOUT_REFERENCE \
+            scripts/fixtures/btf_prototypes.c -o "$RUNNER_TEMP/layout-reference"
+          "$RUNNER_TEMP/layout-reference" > "$RUNNER_TEMP/layout-reference.json"
+          python3 -B scripts/test_inspect_btf_prototypes.py --native-reference "$RUNNER_TEMP/prototypes.btf" \
+            --layout-reference "$RUNNER_TEMP/layout-reference.json"

.github/workflows/build.yml

@@ -181,12 +181,18 @@ jobs:
             selected="all"
           fi
 
-
+          # linux-aarch64-* need a real aarch64 host (placement rule: hardware)
+          # so the packaged binary and console smoke execute the artifact.
+          # ubicloud-standard-16-arm is that host: Ubuntu 24.04 arm64, alias of
+          # ubicloud-standard-16-arm-ubuntu-2404 (16 vCPU, 48 GB). linux-x86_64-* stay on the
+          # ARC sm-standard-4 pool. GNU rows keep cross=true so cargo-zigbuild
+          # pins the glibc 2.31 floor on the same architecture; musl stays a
+          # native cargo build.
           all='{"include":[
             {"target_id":"linux-x86_64-musl","os":"sm-standard-4","target":"x86_64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
-            {"target_id":"linux-aarch64-musl","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
+            {"target_id":"linux-aarch64-musl","os":"ubicloud-standard-16-arm","target":"aarch64-unknown-linux-musl","cross":false,"platform":"linux","rustflags":"","features":"","glibc":""},
             {"target_id":"linux-x86_64-gnu","os":"sm-standard-4","target":"x86_64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
-            {"target_id":"linux-aarch64-gnu","os":"sm-standard-4-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
+            {"target_id":"linux-aarch64-gnu","os":"ubicloud-standard-16-arm","target":"aarch64-unknown-linux-gnu","cross":true,"platform":"linux","rustflags":"","features":"pyroscope","glibc":"2.31"},
             {"target_id":"macos-aarch64","os":"macos-latest","target":"aarch64-apple-darwin","cross":false,"platform":"macos","rustflags":"","features":"pyroscope","glibc":""},
             {"target_id":"windows-x86_64","os":"windows-latest","target":"x86_64-pc-windows-msvc","cross":false,"platform":"windows","rustflags":"","features":"","glibc":""}
           ]}'
@@ -244,7 +250,11 @@ jobs:
         with:
           rust-version: stable
           target: ${{ matrix.target }}
-          cache-shared-key: build-${{ matrix.target }}${{ matrix.glibc && format('-glibc-{0}', matrix.glibc) || '' }}
+          # rust-cache keys include OS and architecture but not the runner image.
+          # The Ubicloud aarch64 legs therefore use their own suffix so they
+          # cannot restore a cache written on a different arm64 image.
+          # x86_64, macOS, and Windows keys stay unsuffixed.
+          cache-shared-key: build-${{ matrix.target }}${{ matrix.glibc && format('-glibc-{0}', matrix.glibc) || '' }}${{ matrix.os == 'ubicloud-standard-16-arm' && '-ubicloud-arm' || '' }}
           # main only. A cache saved on refs/tags/X is scoped to that tag: no
           # other tag, no main run and no PR can restore it, so every release
           # cycle wrote up to 12 entries of 1-2GB (preview tag plus final tag,
@@ -764,9 +774,9 @@ jobs:
             exit 0
           fi
 
-          # The self-hosted Linux runners do not ship the aws CLI (GitHub-hosted
-          # images did). Install the official CLI bundle on demand so R2 uploads
-          # survive a fresh runner without relying on a distro package.
+          # ARC pods (the x86_64 legs) do not ship the aws CLI. Ubicloud arm64
+          # images follow the GitHub runner image and often do. Install the
+          # official CLI bundle when it is missing so R2 uploads work on either.
           if ! command -v aws >/dev/null 2>&1; then
             echo "aws CLI not found on runner; installing..."
             if command -v apt-get >/dev/null 2>&1; then

.github/workflows/ci.yml

@@ -16,9 +16,7 @@ name: Continuous Integration
 
 on:
   push:
-    # gateway/integration receives the daily merge of main as a direct push
-    # (rustfs/backlog#2734), so its pushes must prove the result like main's.
-    branches: [ main, gateway/integration ]
+    branches: [ main ]
     paths-ignore:
       - "**.md"
       - "docs/**"
@@ -739,8 +737,6 @@ jobs:
     needs: [ build-rustfs-debug-binary ]
     runs-on: sm-standard-2
     timeout-minutes: 30
-    env:
-      RUSTFS_S3_STACK: legacy
     steps:
       - name: Checkout repository
         uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
@@ -859,105 +855,18 @@ jobs:
           path: ${{ runner.temp }}/rustfs-e2e-*/rustfs.log
           retention-days: 3
 
-  # The same e2e-smoke suite against the same debug binary with the gateway
-  # S3 stack selected (rustfs/backlog#2734). Report-only: it is outside the
-  # Test and Lint aggregate and becomes required at T2.10. The archive and run
-  # steps must stay identical to e2e-tests; scripts/ci_gate.py enforces that.
-  e2e-tests-gateway:
-    name: e2e-smoke (gateway, report-only until T2.10)
-    needs: [ build-rustfs-debug-binary ]
-    runs-on: sm-standard-2
-    timeout-minutes: 30
-    continue-on-error: true
-    env:
-      RUSTFS_S3_STACK: gateway
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
-        with:
-          persist-credentials: false
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: ci-dev
-          cache-save-if: 'false'
-          install-build-packaging-tools: 'false'
-
-      - name: Set up Python
-        uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
-        with:
-          python-version: "3.12"
-
-      - name: Install awscurl
-        run: |
-          python3 -m pip install --user --upgrade pip "awscurl==0.44"
-          echo "AWSCURL_PATH=$HOME/.local/bin/awscurl" >> "$GITHUB_ENV"
-
-      - name: Verify awscurl
-        run: test -x "$AWSCURL_PATH"
-
-      - name: Download debug binary
-        uses: actions/download-artifact@3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c # v8.0.1
-        with:
-          name: rustfs-debug-binary
-          path: target/debug
-
-      - name: Make binary executable
-        run: chmod +x ./target/debug/rustfs
-
-      - name: Archive e2e smoke test binaries
-        env:
-          NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
-          NEXTEST_LISTING: ${{ runner.temp }}/rustfs-e2e-smoke-list.json
-        run: |
-          cargo nextest archive --profile e2e-smoke -p e2e_test --archive-file "${NEXTEST_ARCHIVE}"
-          cargo nextest list --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" --message-format json > "${NEXTEST_LISTING}"
-          python3 ./scripts/check_test_wiring.py --check-profile e2e-smoke "${NEXTEST_LISTING}"
-          ./scripts/check_security_smoke_count.sh check "${NEXTEST_LISTING}"
-
-      - name: Run e2e smoke suite
-        env:
-          NEXTEST_ARCHIVE: ${{ runner.temp }}/rustfs-e2e-smoke.tar.zst
-          RUSTFS_E2E_LOG_DIR: ${{ runner.temp }}/rustfs-e2e-smoke-logs
-          SAMPLER_INTERVAL_SECS: "15"
-        run: |
-          ./scripts/ci/resource_sampler.sh start e2e-smoke
-          trap './scripts/ci/resource_sampler.sh stop' EXIT
-          python3 scripts/e2e_binary.py run --features e2e-test-hooks -- cargo nextest run --profile e2e-smoke --archive-file "${NEXTEST_ARCHIVE}" \
-            --status-level all --final-status-level all --failure-output final
-
-      - name: Upload e2e smoke diagnostics
-        if: always()
-        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
-        with:
-          name: e2e-smoke-gateway-diagnostics-${{ github.run_number }}
-          path: |
-            ${{ runner.temp }}/rustfs-e2e-smoke-logs/
-            ${{ runner.temp }}/rustfs-e2e-smoke-list.json
-          if-no-files-found: warn
-
-      - name: Upload e2e smoke JUnit report
-        if: always()
-        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
-        with:
-          name: e2e-smoke-gateway-junit-${{ github.run_number }}
-          path: target/nextest/e2e-smoke/junit.xml
-          if-no-files-found: warn
-
   e2e-full:
     name: End-to-End Tests (full merge gate)
     # Merge gate only (backlog#1149 ci-5): the never-automated user-visible
     # suites — KMS, object_lock, multipart_auth, quota, checksum, encryption,
     # security-boundary, ... — via the e2e-full nextest profile. Too heavy for
-    # every PR, so it is gated to main and gateway/integration pushes, the merge
-    # queue, and manual dispatch. protocols / the 7 cluster suites / replication
-    # / #[ignore] are owned by other lanes (see .config/nextest.toml profile.e2e-full).
+    # every PR, so it is gated to main pushes, the merge queue, and manual
+    # dispatch. protocols / the 7 cluster suites / replication / #[ignore] are
+    # owned by other lanes (see .config/nextest.toml profile.e2e-full).
     if: >-
       github.event_name == 'workflow_dispatch' ||
       github.event_name == 'merge_group' ||
-      (github.event_name == 'push' && (github.ref == 'refs/heads/main' || github.ref == 'refs/heads/gateway/integration'))
+      (github.event_name == 'push' && github.ref == 'refs/heads/main')
     needs: [ build-rustfs-debug-binary ]
     runs-on: sm-standard-2
     timeout-minutes: 55

.github/workflows/coverage.yml

@@ -45,7 +45,7 @@ on:
     # 07:00 UTC Sunday — staggered clear of the other Sunday crons: ci (00:00),
     # build (01:00), e2e-s3tests (02:00), audit (03:00), nix-flake-update
     # (05:00), mint (06:00), and the daily fuzz (02:00), minio-interop (03:17),
-    # e2e-replication-nightly (04:00) and performance-ab (06:00) lanes.
+    # and e2e-replication-nightly (04:00) lanes.
     - cron: "43 7 * * 0"
 
 concurrency:

.github/workflows/docker.yml

@@ -337,7 +337,7 @@ jobs:
     name: Build Docker Images
     needs: build-check
     if: needs.build-check.outputs.should_build == 'true'
-    runs-on: dind-sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 60
     strategy:
       fail-fast: false

.github/workflows/e2e-replication-nightly.yml

@@ -38,8 +38,8 @@ name: e2e-nightly
 on:
   workflow_dispatch:
   schedule:
-    # 04:00 UTC nightly — staggered clear of fuzz/e2e-s3tests (02:00),
-    # stale (01:30) and performance-ab (06:00).
+    # 04:00 UTC nightly — staggered clear of fuzz/e2e-s3tests (02:00)
+    # and stale (01:30).
     - cron: "29 4 * * *"
 
 # Only alert-on-failure needs more than read access; it declares its own

.github/workflows/performance-ab.yml

@@ -1,384 +0,0 @@
-# Copyright 2024 RustFS Team
-#
-# Licensed under the Apache License, Version 2.0 (the "License");
-# you may not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#     http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-# Warp A/B budget gate for the hotpath series (rustfs/backlog#935 HP-14, item 4).
-#
-# Two entry points, honestly scoped:
-#   * schedule (nightly, on main): post-merge detection — catches a regression
-#     within 24h of landing, not before merge.
-#   * workflow_dispatch: an explicitly selected trusted ref.
-# The dispatch input can run the gate with --allow-regression so a deliberate
-# correctness cost (e.g. the #4221 fsync durability fix) is recorded, not
-# blocked (rustfs/backlog#935 correction 1).
-
-name: Performance A/B
-
-on:
-  schedule:
-    - cron: "31 6 * * *" # 06:31 UTC nightly, against main
-  workflow_dispatch:
-    inputs:
-      duration:
-        description: "warp duration per round"
-        required: false
-        default: "12s"
-        type: string
-      allow_regression:
-        description: "Pass the gate despite a FAIL (deliberate tradeoff)"
-        required: false
-        default: false
-        type: boolean
-permissions:
-  actions: read
-  contents: read
-
-env:
-  CARGO_TERM_COLOR: always
-  RUST_BACKTRACE: 1
-
-jobs:
-  warp-ab:
-    if: github.repository == 'rustfs/rustfs'
-    name: Warp A/B budget gate
-    runs-on: sm-standard-2
-    # A normal nightly restores the last successful binary and builds only the
-    # candidate; daily access keeps that cache warm. A cache miss may build both
-    # and needs room for the A/B run plus artifact and cache publication.
-    timeout-minutes: 180
-    steps:
-      - name: Checkout repository
-        uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
-        with:
-          persist-credentials: false
-          fetch-depth: 0 # baseline may be an earlier successful scheduled head
-
-      - name: Setup Rust environment
-        uses: ./.github/actions/setup
-        with:
-          rust-version: stable
-          cache-shared-key: warp-ab-${{ hashFiles('**/Cargo.lock') }}
-          cache-save-if: ${{ github.ref == 'refs/heads/main' }}
-
-      - name: Install warp
-        run: |
-          set -euo pipefail
-          WARP_VERSION="1.0.0"
-          curl -fsSL "https://github.com/minio/warp/releases/download/v${WARP_VERSION}/warp_Linux_x86_64.tar.gz" \
-            | sudo tar -xz -C /usr/local/bin warp
-          warp --version
-
-      - name: Decide exemption
-        id: exempt
-        env:
-          INPUT_ALLOW_REGRESSION: ${{ github.event.inputs.allow_regression }}
-        run: |
-          allow="false"
-          if [[ "$INPUT_ALLOW_REGRESSION" == "true" ]]; then
-            allow="true"
-          fi
-          echo "allow_regression=$allow" >> "$GITHUB_OUTPUT"
-
-      # A failed regression run must keep comparing against the last known-good
-      # scheduled head. Otherwise the next nightly would absorb the regression
-      # into its baseline and turn green without a fix.
-      - name: Find last successful scheduled baseline
-        id: scheduled_baseline
-        if: github.event_name == 'schedule'
-        uses: actions/github-script@3a2844b7e9c422d3c10d287c895573f7108da1b3 # v9.0.0
-        with:
-          result-encoding: string
-          script: |
-            const { data } = await github.rest.actions.listWorkflowRuns({
-              owner: context.repo.owner,
-              repo: context.repo.repo,
-              workflow_id: "performance-ab.yml",
-              event: "schedule",
-              status: "success",
-              per_page: 1,
-            });
-            return data.workflow_runs[0]?.head_sha ?? "";
-
-      # Manual runs compare a selected ref with current main. Scheduled runs
-      # compare current main with the last successful scheduled head. With no
-      # history, the first run measures the candidate against itself and seeds
-      # that head only if the complete rig succeeds.
-      - name: Resolve baseline / candidate commits
-        id: commits
-        env:
-          SCHEDULED_BASELINE_SHA: ${{ steps.scheduled_baseline.outputs.result }}
-        run: |
-          set -euo pipefail
-          candidate_sha="$(git rev-parse HEAD)"
-          if [[ "${{ github.event_name }}" == "schedule" ]]; then
-            baseline_sha="${SCHEDULED_BASELINE_SHA:-$candidate_sha}"
-          else
-            baseline_sha="$(git rev-parse origin/main)"
-          fi
-          git cat-file -e "${baseline_sha}^{commit}"
-          if ! git merge-base --is-ancestor "$baseline_sha" "$candidate_sha"; then
-            echo "::error::baseline $baseline_sha is not an ancestor of candidate $candidate_sha; update the selected ref before comparing" >&2
-            exit 1
-          fi
-          echo "baseline_sha=$baseline_sha" >> "$GITHUB_OUTPUT"
-          echo "candidate_sha=$candidate_sha" >> "$GITHUB_OUTPUT"
-          echo "baseline commit: $baseline_sha"
-          echo "candidate commit: $candidate_sha"
-
-      # Exact-key restore of the candidate binary saved by its successful
-      # scheduled run. A miss leaves cache-hit unset and falls back to a source
-      # build of that known-good head.
-      - name: Restore cached baseline binary
-        id: baseline_cache
-        uses: actions/cache/restore@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
-        with:
-          path: baseline-bin/rustfs
-          key: rustfs-baseline-${{ steps.commits.outputs.baseline_sha }}
-
-      # Self-heal: on a nightly/dispatch run where the candidate commit IS the
-      # baseline commit, a cache miss would make the rig build the same commit
-      # twice (~65min per side with the post-#4806 LTO profile — no job budget
-      # fits that). Build it once here, reuse it for both phases, and save it
-      # back to the cache so the next run hits.
-      - name: Build baseline on cache miss (same-commit self-heal)
-        id: selfheal
-        if: >-
-          steps.baseline_cache.outputs.cache-hit != 'true' &&
-          steps.commits.outputs.baseline_sha == steps.commits.outputs.candidate_sha
-        run: |
-          set -euo pipefail
-          cargo build --release --bin rustfs
-          mkdir -p baseline-bin
-          cp target/release/rustfs baseline-bin/rustfs
-          echo "built=true" >> "$GITHUB_OUTPUT"
-
-      - name: Build baseline on cache miss (different candidate)
-        id: baseline_build
-        if: >-
-          steps.baseline_cache.outputs.cache-hit != 'true' &&
-          steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha
-        run: |
-          set -euo pipefail
-          baseline_root="$RUNNER_TEMP/rustfs-baseline-${{ github.run_id }}"
-          baseline_target="$RUNNER_TEMP/rustfs-baseline-target-${{ github.run_id }}"
-          git worktree add --detach "$baseline_root" "${{ steps.commits.outputs.baseline_sha }}"
-          cargo build --release --manifest-path "$baseline_root/Cargo.toml" --bin rustfs --target-dir "$baseline_target"
-          mkdir -p baseline-bin
-          cp "$baseline_target/release/rustfs" baseline-bin/rustfs
-          git worktree remove --force "$baseline_root"
-          echo "built=true" >> "$GITHUB_OUTPUT"
-
-      - name: Build candidate binary
-        id: candidate_build
-        if: steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha
-        run: |
-          set -euo pipefail
-          cargo build --release --bin rustfs
-          mkdir -p candidate-bin
-          cp target/release/rustfs candidate-bin/rustfs
-          echo "built=true" >> "$GITHUB_OUTPUT"
-
-      - name: Save self-healed baseline to cache
-        if: steps.selfheal.outputs.built == 'true' || steps.baseline_build.outputs.built == 'true'
-        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
-        with:
-          path: baseline-bin/rustfs
-          key: rustfs-baseline-${{ steps.commits.outputs.baseline_sha }}
-
-      - name: Run warp A/B and gate
-        id: ab
-        env:
-          INPUT_DURATION: ${{ github.event.inputs.duration }}
-        run: |
-          set -euo pipefail
-          # The formal runner executes A1 baseline -> B1 candidate -> B2 candidate
-          # -> A2 baseline for each workload and drive-sync cell. It requires three
-          # rounds per leg to emit tail latency and error-rate evidence.
-          # --health-timeout 180 outlasts the server's own 120s startup-readiness
-          # budget, which the rig's previous 60s health poll undershot (the first
-          # two nightly failures). perf-6 recalibrates these once the noise study
-          # lands.
-          duration="${INPUT_DURATION:-12s}"
-          baseline_sha="${{ steps.commits.outputs.baseline_sha }}"
-          candidate_sha="${{ steps.commits.outputs.candidate_sha }}"
-          baseline_hit="${{ steps.baseline_cache.outputs.cache-hit }}"
-          selfheal_built="${{ steps.selfheal.outputs.built }}"
-          baseline_built="${{ steps.baseline_build.outputs.built }}"
-          candidate_built="${{ steps.candidate_build.outputs.built }}"
-
-          args=(--duration "$duration" --rounds 3 --cooldown 5 --health-timeout 180 --baseline-revision "$baseline_sha" --candidate-revision "$candidate_sha")
-
-          if [[ "$baseline_hit" == "true" || "$selfheal_built" == "true" || "$baseline_built" == "true" ]]; then
-            chmod +x baseline-bin/rustfs
-            base_bin="$PWD/baseline-bin/rustfs"
-            args+=(--baseline-bin "$base_bin")
-            if [[ "$baseline_hit" == "true" ]]; then
-              base_src="actions-cache (rustfs-baseline-$baseline_sha)"
-            elif [[ "$selfheal_built" == "true" ]]; then
-              base_src="source build (cache self-heal, saved as rustfs-baseline-$baseline_sha)"
-            else
-              base_src="isolated baseline source build (saved as rustfs-baseline-$baseline_sha)"
-            fi
-            if [[ "$candidate_sha" == "$baseline_sha" ]]; then
-              # No commits landed since the last successful baseline, so reuse
-              # the one binary for both phases and measure only rig drift.
-              args+=(--candidate-bin "$base_bin")
-              cand_src="same binary as baseline (same commit)"
-            elif [[ "$candidate_built" == "true" ]]; then
-              chmod +x candidate-bin/rustfs
-              args+=(--candidate-bin "$PWD/candidate-bin/rustfs")
-              cand_src="source build of the checked-out ref"
-            else
-              echo "::error::candidate binary was not built" >&2
-              exit 2
-            fi
-          else
-            echo "::error::baseline binary was not restored or built" >&2
-            exit 2
-          fi
-
-          echo "baseline binary: $base_src"
-          echo "candidate binary: $cand_src"
-
-          if [[ "${{ steps.exempt.outputs.allow_regression }}" == "true" ]]; then
-            args+=(--allow-regression --exemption-reason "workflow dispatch override")
-          fi
-          # Do not let a gate FAIL abort the job here; capture status and surface
-          # it after the step summary is written.
-          set +e
-          bash scripts/run_hotpath_warp_abba.sh "${args[@]}"
-          echo "status=$?" >> "$GITHUB_OUTPUT"
-          set -e
-          # Locate the newest run dir + gate.md for the summary/comment/artifact
-          # steps. On a startup failure there is no gate.md, but the run dir still
-          # holds server-logs/ for diagnosis.
-          # Run dirs are UTC-timestamp names (no special chars); ls is safe here.
-          # shellcheck disable=SC2012
-          run_dir="$(ls -td target/hotpath-abba/*/ 2>/dev/null | head -n1 || true)"
-          echo "run_dir=${run_dir%/}" >> "$GITHUB_OUTPUT"
-          # shellcheck disable=SC2012
-          gate_md="$(ls -t target/hotpath-abba/*/candidate_gate.md 2>/dev/null | head -n1 || true)"
-          echo "gate_md=$gate_md" >> "$GITHUB_OUTPUT"
-
-      - name: Upload A/B results
-        if: always()
-        uses: actions/upload-artifact@043fb46d1a93c77aae656e7c1c64a875d1fc6a0a # v7.0.1
-        with:
-          name: hotpath-warp-ab-${{ github.run_number }}
-          # Includes per-cell median_summary.csv / baseline_compare.csv, both gates,
-          # and server-logs/ (rustfs.log + startup env per phase) so a failed run
-          # is diagnosable. Short retention: this is churny nightly debug data.
-          path: target/hotpath-abba/
-          if-no-files-found: warn
-          retention-days: 14
-
-      - name: Write gate summary
-        if: always()
-        run: |
-          set -euo pipefail
-          status="${{ steps.ab.outputs.status }}"
-          gate_md="${{ steps.ab.outputs.gate_md }}"
-          run_dir="${{ steps.ab.outputs.run_dir }}"
-          {
-            echo "## Hotpath warp A/B — run ${{ github.run_number }}"
-            echo
-            if [[ "$status" == "0" ]]; then
-              echo "Rig/gate exit: \`0\` (pass or warn)."
-            else
-              echo "Rig/gate exit: \`${status:-unknown}\` — **FAILED**."
-            fi
-            echo
-            if [[ -n "$gate_md" && -f "$gate_md" ]]; then
-              cat "$gate_md"
-            else
-              echo "No \`gate.md\` produced — the rig failed **before** the gate"
-              echo "(most likely server startup / health). Failing phase(s) below;"
-              echo "full logs in the \`hotpath-warp-ab-${{ github.run_number }}\` artifact."
-              if [[ -n "$run_dir" && -d "$run_dir/server-logs" ]]; then
-                for f in "$run_dir"/server-logs/*.log; do
-                  [[ -f "$f" ]] || continue
-                  echo
-                  echo "<details><summary>$(basename "$f")</summary>"
-                  echo
-                  echo '```'
-                  tail -n 30 "$f"
-                  echo '```'
-                  echo "</details>"
-                done
-              fi
-            fi
-          } >> "$GITHUB_STEP_SUMMARY"
-
-      - name: Stage successful candidate baseline
-        if: >-
-          steps.ab.outputs.status == '0' &&
-          steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha
-        run: |
-          set -euo pipefail
-          cp candidate-bin/rustfs baseline-bin/rustfs
-
-      - name: Cache successful candidate baseline
-        if: >-
-          steps.ab.outputs.status == '0' &&
-          steps.commits.outputs.baseline_sha != steps.commits.outputs.candidate_sha
-        uses: actions/cache/save@55cc8345863c7cc4c66a329aec7e433d2d1c52a9 # v6
-        with:
-          path: baseline-bin/rustfs
-          key: rustfs-baseline-${{ steps.commits.outputs.candidate_sha }}
-
-      # Scheduled failure alerting is handled by the alert-on-failure job below
-      # (perf-2 consuming ci-8's schedule-failure-issue composite action).
-
-      - name: Enforce gate
-        if: always()
-        run: |
-          status="${{ steps.ab.outputs.status }}"
-          if [[ -z "$status" ]]; then
-            echo "::error::warp A/B setup failed before the rig ran. Check the first failed workflow step." >&2
-            exit 1
-          fi
-          if [[ "$status" != "0" ]]; then
-            echo "::error::warp A/B budget gate failed (exit $status). See the step summary / gate.md artifact." >&2
-            exit "$status"
-          fi
-          echo "warp A/B budget gate passed."
-
-  alert-on-failure:
-    name: Alert on scheduled failure
-    needs: [warp-ab]
-    # `always()` is required: without it this job is skipped when a needed
-    # job fails. Alerts only for scheduled (nightly) runs (backlog#1149
-    # ci-8); manual dispatch failures are already watched by a human.
-    # `cancelled` is included alongside `failure` on purpose: a job that hits
-    # timeout-minutes ends as `cancelled`, and the 2026-07-11..07-14 nightly
-    # timeouts went silent precisely because the guard was failure-only. The
-    # composite action already reports cancelled/timed-out jobs in the issue
-    # body.
-    if: >-
-      github.repository == 'rustfs/rustfs' && (
-        always() && github.event_name == 'schedule' &&
-        (contains(needs.*.result, 'failure') || contains(needs.*.result, 'cancelled'))
-      )
-    runs-on: ubuntu-latest
-    timeout-minutes: 10
-    permissions:
-      contents: read
-      issues: write
-    steps:
-      - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7
-        with:
-          persist-credentials: false
-      - name: Open or update failure-tracking issue
-        uses: ./.github/actions/schedule-failure-issue
-        with:
-          github-token: ${{ secrets.BACKLOG_ISSUE_TOKEN }}

.github/workflows/runner-hygiene.yml

@@ -44,7 +44,7 @@ jobs:
   check-ephemerality:
     if: github.repository == 'rustfs/rustfs'
     name: Check runner ephemerality
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 15
     steps:
       - name: Checkout repository

.github/workflows/rustfs-functional-chain.yml

@@ -42,7 +42,7 @@ concurrency:
 jobs:
   prepare:
     if: ${{ github.event_name == 'workflow_dispatch' || github.event.workflow_run.event == 'schedule' }}
-    runs-on: smoke-testing
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     outputs:
       manifest: ${{ steps.candidate.outputs.manifest }}
@@ -188,7 +188,7 @@ jobs:
   complete-chain:
     needs: [prepare, upgrade, s3, kms, tier, storage, heal, pool, security, replication, fault-tolerance, table, performance]
     if: ${{ always() && needs.prepare.result != 'skipped' }}
-    runs-on: smoke-testing
+    runs-on: ubuntu-latest
     timeout-minutes: 10
     steps:
       - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7

.github/workflows/rustfs-performance-test.yml

@@ -90,7 +90,7 @@ env:
 jobs:
   runner-check:
     if: ${{ inputs.chain_manifest != '' || github.event_name == 'workflow_dispatch' || github.event_name == 'repository_dispatch' }}
-    runs-on: sm-standard-2
+    runs-on: ubuntu-latest
     timeout-minutes: 5
     steps:
       - uses: actions/checkout@f548e57e544e1ff5a4c46bf1e1b8685f8e4a348a # v7

.github/workflows/schedule-failure-alert-drill.yml

@@ -16,8 +16,8 @@
 #
 # Forces a job failure and then runs .github/actions/schedule-failure-issue
 # through the exact `needs` + `always() && contains(needs.*.result,
-# 'failure')` wiring used by the real consumers (e2e-s3tests, mint, fuzz,
-# performance-ab). Dispatch it twice to verify both the issue-creation and
+# 'failure')` wiring used by the real consumers (e2e-s3tests, mint, fuzz).
+# Dispatch it twice to verify both the issue-creation and
 # the dedupe-comment paths, then close the resulting
 # "[scheduled-failure] Schedule Failure Alert Drill" issue in rustfs/backlog
 # (the default target repository). Requires the BACKLOG_ISSUE_TOKEN secret

.github/workflows/scheduled-validation-watchdog.yml

@@ -28,7 +28,6 @@ on:
       - "mint"
       - "minio-interop"
       - "Nightly GNU Build"
-      - "Performance A/B"
       - "Runner Hygiene"
     types: [completed]
 

AGENTS.md

@@ -158,9 +158,7 @@ requested adversarial/design reviews, and agent-instruction changes that alter
 execution. Ordinary questions, diagnoses, status reports, non-adversarial code
 reviews, and low-risk planning do not trigger it.
 
-For applicable work and substantial PR reviews, read the [risk tiers and review shape](.agents/references/adversarial-validation.md).
-Load only the matching domain probes; ordinary reviews do not become adversarial
-merely because this reference exists.
+For applicable work and substantial PR reviews, follow the [risk tiers and review shape](.agents/references/adversarial-validation.md) and check changed trust boundaries and relevant correctness, security, concurrency, durability, compatibility, performance, and negative regression coverage.
 
 A review has no finding quota; `No findings` is a complete outcome. A request to
 find problems is not evidence that a defect exists. Before reporting a candidate,

ARCHITECTURE.md

@@ -155,7 +155,7 @@ default build (lifecycle:
    - ⚠️ PARTIALLY VIOLATED: serving-side `s3s` references remain in ecstore
      (bucket metadata/replication/lifecycle DTOs and error mapping). The
      count is ratcheted shrink-only by `scripts/check_s3s_footprint.sh`
-     (the `ecstore_files` line of `.config/s3s-footprint-baseline.txt`; the `object_lock` module was converted to
+     (`S3S_ECSTORE_FILES_BASELINE`; the `object_lock` module was converted to
      storage-level types as the first ratchet step). Target state: the
      baseline reaches zero and ecstore's `Cargo.toml` drops `s3s`.
 

Cargo.toml

@@ -48,7 +48,7 @@ members = [
     "crates/replication", # Replication contracts and wire formats
     "crates/concurrency", # Concurrency management for RustFS - timeout, locking, backpressure, and I/O scheduling
     "crates/s3-client", # S3 client for engine-side consumption of remote S3 endpoints (tiering, transition targets)
-    "crates/s3-types", # S3 event and error type definitions
+    "crates/s3-types", # S3 event type definitions
     "crates/s3-ops", # S3 operation definitions and mapping
     "crates/s3select-api", # S3 Select API interface
     "crates/s3select-query", # S3 Select query engine
@@ -143,20 +143,13 @@ rustfs-tls-runtime = { path = "crates/tls-runtime", version = "1.0.2" }
 rustfs-utils = { path = "crates/utils", version = "1.0.2" }
 rustfs-zip = { path = "./crates/zip", version = "1.0.2" }
 
-# The gateway's S3 types: the DTOs and persisted bucket-configuration codecs the
-# engine crates move onto from s3s (rustfs/backlog#2745). The one pin of the
-# gateway in this workspace; no crate may enable a `compat-s3s*` feature except
-# the s3s-facing edge of the rustfs binary.
-rustfs-gateway-types = { git = "https://github.com/rustfs/gateway.git", rev = "21283e874c2381bab611e9347acbc5b92e9e2b3c", version = "0.32.0", default-features = false }
-rustfs-gateway-stream = { git = "https://github.com/rustfs/gateway.git", rev = "21283e874c2381bab611e9347acbc5b92e9e2b3c", version = "0.2.0", default-features = false }
-
 # Async Runtime and Networking
 async-channel = "2.5.0"
 async_zip = { default-features = false, version = "0.0.19" }
 mysql_async = { default-features = false, version = "0.37.1" }
 async-compression = { version = "0.4.51" }
 async-trait = "0.1.92"
-async-nats = { version = "0.50.0", default-features = false }
+async-nats = { version = "0.51.0", default-features = false }
 axum = "0.8.9"
 futures = "0.3.34"
 futures-core = "0.3.34"
@@ -182,9 +175,9 @@ tokio-rustls = { default-features = false, version = "0.26.6" }
 tokio-stream = { version = "0.1.19" }
 tokio-test = "0.4.6"
 tokio-util = { version = "0.7.20" }
-tonic = { version = "0.14.6" }
-tonic-prost = { version = "0.14.6" }
-tonic-prost-build = { version = "0.14.6" }
+tonic = { version = "0.14.8" }
+tonic-prost = { version = "0.14.8" }
+tonic-prost-build = { version = "0.14.8" }
 tower = { version = "0.5.3" }
 tower-http = { version = "0.7.1" }
 h3 = "0.0.8"
@@ -260,15 +253,15 @@ atoi = "3.1.1"
 atomic_enum = "0.3.0"
 aws-config = { version = "1.13.0" }
 aws-credential-types = { version = "1.3.1" }
-aws-sdk-kms = { default-features = false, version = "1.126.0" }
-aws-sdk-s3 = { default-features = false, version = "1.155.0" }
-aws-sdk-sts = { default-features = false, version = "1.122.0" }
+aws-sdk-kms = { default-features = false, version = "1.128.0" }
+aws-sdk-s3 = { default-features = false, version = "1.156.0" }
+aws-sdk-sts = { default-features = false, version = "1.123.0" }
 aws-smithy-async = { version = "1.3.1" }
-aws-smithy-http-client = { default-features = false, version = "1.5.1" }
-aws-smithy-runtime-api = { version = "1.19.1" }
-aws-smithy-types = { version = "1.9.0" }
+aws-smithy-http-client = { default-features = false, version = "1.5.2" }
+aws-smithy-runtime-api = { version = "1.19.2" }
+aws-smithy-types = { version = "1.9.1" }
 base64-simd = "0.8.0"
-brotli = "9.1.0"
+brotli = "9.1.1"
 clap = { version = "4.6.7" }
 const-str = { version = "1.1.0" }
 convert_case = "0.12.0"
@@ -328,15 +321,11 @@ regex = { version = "1.13.1" }
 rumqttc = { package = "rumqttc-next", version = "0.34.0" }
 redis = { version = "1.8.0" }
 rustify = { version = "0.7", default-features = false }
-rustix = { version = "1.1.5" }
+rustix = { version = "1.1.6" }
 rust-embed = { version = "8.13.0" }
 rustc-hash = { version = "2.1.3" }
 s3s = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
 s3s-sigv4 = { git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0" }
-# The same s3s pin under the name the legacy edge of the gateway migration uses
-# (crates/s3-types, feature `compat-s3s`). Keep `rev` identical to `s3s` above.
-# DELETE BY T4.2 (rustfs/backlog#2784).
-legacy-s3s = { package = "s3s", git = "https://github.com/s3s-project/s3s", rev = "5761ddfe4c509fcac87e3b99fd5bd9e176b442c0", features = ["minio"] }
 serial_test = "4.0.1"
 shadow-rs = { default-features = false, version = "2.0.0" }
 siphasher = "1.0.4"
@@ -345,7 +334,6 @@ compact_str = "0.10.0"
 snap = "1.1.2"
 starshard = { version = "2.4.0" }
 strum = { version = "0.28.0" }
-sync_wrapper = "1.0.2"
 sysinfo = "0.39.6"
 temp-env = "0.3.6"
 tempfile = "3.27.0"
@@ -368,13 +356,13 @@ winapi-util = "0.1.11"
 windows = { version = "0.62.2" }
 windows-sys = "0.61.2"
 xxhash-rust = { version = "0.8.19" }
-zip = "9.1.2"
+zip = "9.1.3"
 zstd = "0.14.3"
 
 # Observability and Metrics
 metrics = "0.24.6"
 metrics-util = "0.20"
-dial9 = { version = "0.5.5", default-features = false }
+dial9 = { version = "0.5.6", default-features = false }
 opentelemetry = { version = "0.33.1" }
 opentelemetry-appender-tracing = { version = "0.33.1" }
 opentelemetry-otlp = { version = "0.33.1" }
@@ -385,7 +373,7 @@ opentelemetry-stdout = { version = "0.33.1" }
 pyroscope = { version = "2.1.5" }
 
 # FTP and SFTP
-libunftp = { version = "0.23.1" }
+libunftp = { version = "0.24.0" }
 unftp-core = "0.1.0"
 suppaftp = { version = "12.2.1" }
 rcgen = { version = "0.14.10", default-features = false, features = ["aws_lc_rs", "crypto", "pem"] }
@@ -398,7 +386,7 @@ dav-server = "0.11.0"
 # Performance Analysis and Memory Profiling
 rustfs-mimalloc = { version = "0.7.2" }
 # Preserve Unicode focus filters until rustfs/backlog#2302 is resolved.
-hotpath = { version = "0.29.2", default-features = false }
+hotpath = { version = "0.30.0", default-features = false }
 # Snapshot testing for output format regression detection
 insta = { version = "1.49" }
 
@@ -441,14 +429,5 @@ strip = "none"
 zstd = { path = "crates/zstd-compat" }
 # Preserve RustFS OIDC flows while migrating the old elliptic-curve dependency chain.
 # Upstream: openidconnect-rs#244 and oauth2-rs#352.
-openidconnect = { git = "https://github.com/ramosbugs/openidconnect-rs", rev = "2f77ee73bc4554e1cb6279adb386005429fb4e41" }
+openidconnect = { git = "https://github.com/ramosbugs/openidconnect-rs", rev = "e20ddbb01ca68356f2979bc125f5613ab45b4e7e" }
 oauth2 = { git = "https://github.com/ramosbugs/oauth2-rs", rev = "89585f4a280afd7b285eaf1c9be0068874e03d22" }
-aws-sigv4 = { git = "https://github.com/smithy-lang/smithy-rs", rev = "5c1c4bca2ea7af97496e077c9f1a718eaac64bf8" }
-
-# Keep the signer's path dependencies on the SDK's existing registry types.
-[patch."https://github.com/smithy-lang/smithy-rs"]
-aws-credential-types = { version = "=1.3.1" }
-aws-smithy-eventstream = { version = "=0.61.6" }
-aws-smithy-http = { version = "=0.64.2" }
-aws-smithy-runtime-api = { version = "=1.19.1" }
-aws-smithy-types = { version = "=1.9.0" }

crates/concurrency/Cargo.toml

@@ -26,7 +26,7 @@ rustfs-io-core = { workspace = true }
 serde = { workspace = true, features = ["derive"] }
 
 # Async runtime
-tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread"] }
+tokio = { workspace = true, features = ["sync", "fs", "rt-multi-thread", "time"] }
 
 # Logging
 tracing = { workspace = true }

crates/config/README.md

@@ -100,6 +100,7 @@ when restoring `/rustfs/console` for a custom OEM build.
 ## S3 API environment variables
 
 - `RUSTFS_API_OBJECT_MAX_VERSIONS` caps the number of retained versions for a single object. It defaults to `9223372036854775807`, matching MinIO's practical-unlimited default. Set a positive integer to enforce a lower per-object metadata bound.
+- `RUSTFS_VEEAM_SOSAPI_ENABLE` serves Veeam SOSAPI `system.xml` and `capacity.xml` for an authenticated GET or HEAD of those exact bucket-root keys. It defaults to `true`. Explicit version reads use the normal object path. Capacity requests return `ServiceUnavailable` until the scanner publishes a complete snapshot certified for the current bucket incarnation and authoritative quota metadata can be read. Legacy or mixed-version snapshots without an incarnation proof cannot serve capacity until a compatible scanner publishes a certified snapshot. Capacity also returns `ServiceUnavailable` when cluster or disk observations are missing, stale, or degraded. Retry after scanner convergence and storage or metadata recovery. Capacity assumes shard paths have independent backing space; shared physical storage can make the estimate optimistic. Set the flag to `false` to restore the normal object path, including deletion of a real object previously stored at that key.
 - `MINIO_API_OBJECT_MAX_VERSIONS` is accepted as a compatibility alias when the canonical RustFS variable is not set.
 
 ## Distributed endpoint locality

crates/config/src/constants/app.rs

@@ -222,6 +222,16 @@ pub const ENV_RUSTFS_TLS_PATH: &str = "RUSTFS_TLS_PATH";
 /// Environment variable for server KMS enable.
 pub const ENV_RUSTFS_KMS_ENABLE: &str = "RUSTFS_KMS_ENABLE";
 
+/// Environment variable for Veeam SOSAPI virtual system objects.
+///
+/// When true (the default), authenticated GET and HEAD of the two Veeam
+/// SOSAPI keys return generated XML. When false, those keys use the normal
+/// object path so an operator can remove a real object stored there earlier.
+pub const ENV_VEEAM_SOSAPI_ENABLE: &str = "RUSTFS_VEEAM_SOSAPI_ENABLE";
+
+/// Default Veeam SOSAPI enablement.
+pub const DEFAULT_VEEAM_SOSAPI_ENABLE: bool = true;
+
 /// Default KMS enable for server-side encryption
 /// This is the default value for enabling KMS encryption for server-side encryption.
 /// Default value: false

crates/data-usage/src/data_usage.rs

@@ -466,6 +466,9 @@ pub struct BucketTargetUsageInfo {
 #[derive(Debug, Default, Clone, Serialize, Deserialize, PartialEq, Eq)]
 pub struct BucketUsageInfo {
     pub size: u64,
+    /// Bucket identity certified by every set contributing to this snapshot.
+    #[serde(default, skip_serializing_if = "Option::is_none")]
+    pub bucket_incarnation: Option<String>,
     // Following five fields suffixed with V1 are here for backward compatibility
     // Total Size for objects that have not yet been replicated
     pub replication_pending_size_v1: u64,
@@ -2316,6 +2319,9 @@ impl BucketUsageInfo {
     /// Add size summary to this bucket usage
     /// Merge another BucketUsageInfo into this one
     pub fn merge(&mut self, other: &BucketUsageInfo) {
+        if self.bucket_incarnation != other.bucket_incarnation {
+            self.bucket_incarnation = None;
+        }
         self.size += other.size;
         self.objects_count += other.objects_count;
         self.versions_count += other.versions_count;
@@ -3299,6 +3305,31 @@ mod tests {
         assert_eq!(usage1.versions_count, 15);
     }
 
+    #[test]
+    fn bucket_usage_merge_discards_incompatible_incarnation_proofs() {
+        let id = "00000000-0000-4000-8000-000000000001";
+        let mut same = BucketUsageInfo {
+            bucket_incarnation: Some(id.into()),
+            ..Default::default()
+        };
+        same.merge(&BucketUsageInfo {
+            bucket_incarnation: Some(id.into()),
+            ..Default::default()
+        });
+        assert_eq!(same.bucket_incarnation.as_deref(), Some(id));
+        same.merge(&BucketUsageInfo::default());
+        assert!(same.bucket_incarnation.is_none());
+        let mut different = BucketUsageInfo {
+            bucket_incarnation: Some(id.into()),
+            ..Default::default()
+        };
+        different.merge(&BucketUsageInfo {
+            bucket_incarnation: Some("00000000-0000-4000-8000-000000000002".into()),
+            ..Default::default()
+        });
+        assert!(different.bucket_incarnation.is_none());
+    }
+
     #[test]
     fn size_summary_add_saturates_instead_of_overflowing() {
         // The scanner folds one summary per object into a per-prefix total, so a

crates/e2e_test/src/anonymous_access_test.rs

@@ -17,8 +17,7 @@
 //! when PublicAccessBlock configuration is missing or explicitly set.
 
 use crate::common::{RustFSTestEnvironment, init_logging, local_http_client};
-use crate::sts_query_compat_test::{OPA_AUTH_TOKEN, OpaMock};
-use aws_sdk_s3::types::{PublicAccessBlockConfiguration, Tag, Tagging};
+use aws_sdk_s3::types::PublicAccessBlockConfiguration;
 use tracing::info;
 
 async fn setup_public_bucket(
@@ -70,98 +69,6 @@ async fn anonymous_get_object(
     local_http_client().get(&url).send().await
 }
 
-#[tokio::test]
-async fn test_plain_http_matches_secure_transport_false_bucket_policy() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server(vec![]).await?;
-    assert!(env.url.starts_with("http://"), "fixture must exercise plaintext HTTP: {}", env.url);
-
-    let bucket = "secure-transport-policy";
-    let key = "plain-http.txt";
-    let admin = env.create_s3_client();
-    admin.create_bucket().bucket(bucket).send().await?;
-    admin
-        .put_object()
-        .bucket(bucket)
-        .key(key)
-        .body(aws_sdk_s3::primitives::ByteStream::from_static(b"plain http"))
-        .send()
-        .await?;
-    admin
-        .put_bucket_policy()
-        .bucket(bucket)
-        .policy(
-            serde_json::json!({
-                "Version": "2012-10-17",
-                "Statement": [{
-                    "Effect": "Allow",
-                    "Principal": "*",
-                    "Action": "s3:GetObject",
-                    "Resource": format!("arn:aws:s3:::{bucket}/*"),
-                    "Condition": {"Bool": {"aws:SecureTransport": "false"}}
-                }]
-            })
-            .to_string(),
-        )
-        .send()
-        .await?;
-
-    let response = anonymous_get_object(&env, bucket, key).await?;
-    assert_eq!(response.status().as_u16(), 200);
-    assert_eq!(response.bytes().await?.as_ref(), b"plain http");
-    Ok(())
-}
-
-/// Root credentials have no attached IAM policy, so these requests depend on
-/// the owner flag reaching policy evaluation unchanged.
-#[tokio::test]
-async fn test_root_requests_keep_policy_owner_flag() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
-    init_logging();
-
-    let opa = OpaMock::start().await?;
-    let mut env = RustFSTestEnvironment::new().await?;
-    env.start_rustfs_server_with_env(
-        vec![],
-        &[
-            ("RUSTFS_POLICY_PLUGIN_URL", opa.url.as_str()),
-            ("RUSTFS_POLICY_PLUGIN_AUTH_TOKEN", OPA_AUTH_TOKEN),
-        ],
-    )
-    .await?;
-
-    let bucket = "root-owner-flag";
-    let key = "owner.txt";
-    let client = env.create_s3_client();
-    client.create_bucket().bucket(bucket).send().await?;
-    client
-        .put_object()
-        .bucket(bucket)
-        .key(key)
-        .body(aws_sdk_s3::primitives::ByteStream::from_static(b"owner access"))
-        .send()
-        .await?;
-
-    // Duplicate stored tags are ambiguous. An owner request must not enter the
-    // OPA tag-loading path, which rejects this metadata before evaluation.
-    client
-        .put_bucket_tagging()
-        .bucket(bucket)
-        .tagging(
-            Tagging::builder()
-                .tag_set(Tag::builder().key("department").value("finance").build()?)
-                .tag_set(Tag::builder().key("department").value("engineering").build()?)
-                .build()?,
-        )
-        .send()
-        .await?;
-    let object = client.get_object().bucket(bucket).key(key).send().await?;
-
-    assert_eq!(object.body.collect().await?.into_bytes().as_ref(), b"owner access");
-    Ok(())
-}
-
 /// Issue #2036: Anonymous GetObject should succeed when bucket policy allows it
 /// and no PublicAccessBlock configuration exists (ConfigNotFound).
 #[tokio::test]

crates/e2e_test/src/distributed/chaos_test.rs

@@ -13,18 +13,20 @@
 // limitations under the License.
 
 use super::harness::{
-    DistCluster, DistLayout, TestResult, assert_object_bytes, payload_for, put_object, retrying_get_equals, unique_bucket,
-    wait_for_ready, wait_until,
+    DistCluster, DistLayout, TestResult, assert_object_bytes, cluster_admin_ok, get_object_bytes, payload_for, put_object,
+    retrying_get_equals, retrying_put, s3_probe_client, s3_probe_error_code, unique_bucket, wait_for_ready, wait_until,
 };
 use crate::chaos::{census_object_version_on_disk, signed_admin_post};
 use crate::common::{build_test_s3_config, init_logging};
 use crate::fault_proxy::FaultMode;
 use aws_sdk_s3::Client;
+use aws_sdk_s3::error::SdkError;
+use aws_sdk_s3::operation::{get_object::GetObjectError, put_object::PutObjectError};
 use std::path::PathBuf;
 use std::sync::Arc;
 use std::time::Duration;
 use tokio::sync::{Barrier, mpsc};
-use tokio::time::timeout;
+use tokio::time::{Instant, timeout};
 
 #[tokio::test]
 async fn kill_and_restart_node_preserves_objects() -> TestResult {
@@ -44,6 +46,133 @@ async fn kill_and_restart_node_preserves_objects() -> TestResult {
     Ok(())
 }
 
+#[tokio::test]
+async fn ec4_partial_node_return_restores_read_write_quorum() -> TestResult {
+    init_logging();
+    let mut dist = DistCluster::start_with_env(
+        DistLayout::FourByFour,
+        &[
+            ("RUSTFS_ERASURE_SET_DRIVE_COUNT", "16"),
+            ("RUSTFS_STORAGE_CLASS_STANDARD", "EC:4"),
+        ],
+    )
+    .await?;
+    let storage = cluster_admin_ok(&dist.cluster, http::Method::GET, "/rustfs/admin/v3/storageinfo", None).await?;
+    let storage: serde_json::Value = serde_json::from_str(&storage)?;
+    let storage: rustfs_madmin::StorageInfo = serde_json::from_value(storage["info"].clone())?;
+    assert_eq!(storage.disks.len(), 16);
+    assert_eq!(storage.backend.total_sets, vec![1]);
+    assert_eq!(storage.backend.drives_per_set, vec![16]);
+    assert_eq!(storage.backend.standard_sc_parity, Some(4));
+
+    let bucket = unique_bucket("partial-ec4");
+    dist.create_bucket(&bucket).await?;
+    let client = s3_probe_client(&dist.client(0)?);
+    let baseline = payload_for("partial-recovery/baseline", 8 * 1024 * 1024);
+    put_object(&client, &bucket, "baseline.bin", baseline.clone()).await?;
+    wait_until(
+        Duration::from_secs(30),
+        || async {
+            for node in &dist.cluster.nodes {
+                for directory in &node.data_dirs {
+                    let census = census_object_version_on_disk(std::path::Path::new(directory), &bucket, "baseline.bin", None)?;
+                    if !census.is_complete() {
+                        return Ok(false);
+                    }
+                    assert_eq!(census.data_blocks, Some(12), "the object must require twelve data shards");
+                    assert_eq!(census.parity_blocks, Some(4), "the object must have four parity shards");
+                    assert!(!census.expected_part_numbers.is_empty(), "the baseline must use physical shards");
+                }
+            }
+            Ok(true)
+        },
+        "the baseline is complete on all sixteen EC:4 drives",
+    )
+    .await?;
+
+    dist.cluster.stop_node(2)?;
+    dist.cluster.stop_node(3)?;
+    let read_error = timeout(Duration::from_secs(60), get_object_bytes(&client, &bucket, "baseline.bin"))
+        .await?
+        .expect_err("eight unavailable drives must prevent the baseline GET");
+    assert!(
+        matches!(
+            s3_probe_error_code(read_error.as_ref()),
+            Some("SlowDown" | "SlowDownRead" | "ServiceUnavailable")
+        ),
+        "quorum loss must produce an availability error: {read_error:?}"
+    );
+    assert_eq!(
+        read_error
+            .downcast_ref::<SdkError<GetObjectError>>()
+            .and_then(SdkError::raw_response)
+            .map(|response| response.status().as_u16()),
+        Some(503),
+        "read quorum loss must remain a service availability failure"
+    );
+    let write_error = timeout(Duration::from_secs(60), put_object(&client, &bucket, "unavailable.bin", baseline.clone()))
+        .await?
+        .expect_err("eight unavailable drives must prevent a successful PUT");
+    assert!(
+        matches!(s3_probe_error_code(write_error.as_ref()), Some("SlowDown" | "ServiceUnavailable")),
+        "quorum loss must produce an availability error: {write_error:?}"
+    );
+    assert_eq!(
+        write_error
+            .downcast_ref::<SdkError<PutObjectError>>()
+            .and_then(SdkError::raw_response)
+            .map(|response| response.status().as_u16()),
+        Some(503),
+        "write quorum loss must remain a service availability failure"
+    );
+
+    // Recover only four drives. The remaining node must stay stopped while
+    // every live coordinator serves old data and acknowledges new writes.
+    dist.cluster.start_node_process(2).await?;
+    let recovery_deadline = Instant::now() + Duration::from_secs(90);
+    let mut recovered_objects = Vec::new();
+    for node in 0..3 {
+        assert!(dist.cluster.nodes[3].process.is_none(), "the fourth node must remain unavailable");
+        let client = s3_probe_client(&dist.client(node)?);
+        retrying_get_equals(
+            &client,
+            &bucket,
+            "baseline.bin",
+            &baseline,
+            recovery_deadline.saturating_duration_since(Instant::now()),
+        )
+        .await?;
+        let key = format!("partial-{node}.bin");
+        let body = payload_for(&key, 1024 * 1024);
+        retrying_put(
+            &client,
+            &bucket,
+            &key,
+            body.clone(),
+            recovery_deadline.saturating_duration_since(Instant::now()),
+        )
+        .await?;
+        recovered_objects.push((key, body));
+    }
+    for node in 0..3 {
+        assert!(dist.cluster.nodes[3].process.is_none(), "reads must complete before all drives return");
+        let client = s3_probe_client(&dist.client(node)?);
+        assert_object_bytes(&client, &bucket, "baseline.bin", &baseline).await?;
+        for (key, body) in &recovered_objects {
+            assert_object_bytes(&client, &bucket, key, body).await?;
+        }
+    }
+
+    dist.cluster.start_node(3).await?;
+    wait_for_ready(&dist.cluster).await?;
+    let client = s3_probe_client(&dist.client(3)?);
+    assert_object_bytes(&client, &bucket, "baseline.bin", &baseline).await?;
+    for (key, body) in &recovered_objects {
+        assert_object_bytes(&client, &bucket, key, body).await?;
+    }
+    Ok(())
+}
+
 #[tokio::test]
 async fn full_cluster_restart_preserves_objects() -> TestResult {
     init_logging();

crates/e2e_test/src/distributed/harness.rs

@@ -1181,11 +1181,16 @@ pub(crate) async fn retrying_get_equals(
                 )
                 .into()),
                 Err(error) => {
-                    if matches!(s3_probe_error_code(error.as_ref()), Some("SlowDown" | "ServiceUnavailable")) {
-                        Ok(false)
-                    } else {
-                        Err(error)
-                    }
+                    let retryable = match s3_probe_error_code(error.as_ref()) {
+                        Some("SlowDown" | "ServiceUnavailable") => true,
+                        // GET preserves this compatibility code for a lost data quorum.
+                        Some("SlowDownRead") => error
+                            .downcast_ref::<SdkError<GetObjectError>>()
+                            .and_then(SdkError::raw_response)
+                            .is_some_and(|response| response.status().as_u16() == 503),
+                        _ => false,
+                    };
+                    if retryable { Ok(false) } else { Err(error) }
                 }
             }
         },
@@ -1324,6 +1329,62 @@ mod retry_tests {
         Ok(())
     }
 
+    #[tokio::test(start_paused = true)]
+    async fn retrying_get_allows_read_quorum_recovery() -> TestResult {
+        use aws_sdk_s3::config::Region;
+        use aws_smithy_http_client::test_util::{ReplayEvent, StaticReplayClient};
+        use aws_smithy_types::body::SdkBody;
+
+        let endpoint = "http://read-quorum.example.invalid";
+        let event = |status: u16, body: &'static str| {
+            ReplayEvent::new(
+                http::Request::builder()
+                    .method(Method::GET)
+                    .uri(format!("{endpoint}/retry-probe/key?x-id=GetObject"))
+                    .body(SdkBody::empty())
+                    .expect("build replay request"),
+                http::Response::builder()
+                    .status(status)
+                    .header("content-type", "application/xml")
+                    .header("content-length", body.len())
+                    .body(SdkBody::from(body))
+                    .expect("build replay response"),
+            )
+        };
+        for status in [503, 403, 500] {
+            let transport = StaticReplayClient::new(vec![
+                event(status, "<Error><Code>SlowDownRead</Code></Error>"),
+                event(200, "expected bytes"),
+            ]);
+            let client = Client::from_conf(
+                aws_sdk_s3::Config::builder()
+                    .behavior_version_latest()
+                    .credentials_provider(Credentials::new(FAKE_ACCESS_KEY, FAKE_SECRET_KEY, None, None, "read-quorum-test"))
+                    .region(Region::new("us-east-1"))
+                    .endpoint_url(endpoint)
+                    .force_path_style(true)
+                    .retry_config(RetryConfig::disabled())
+                    .http_client(transport.clone())
+                    .build(),
+            );
+            let result = retrying_get_equals(&client, "retry-probe", "key", b"expected bytes", Duration::from_secs(5)).await;
+            if status == 503 {
+                result?;
+                assert_eq!(
+                    transport.actual_requests().count(),
+                    2,
+                    "the probe must retry the read-quorum error exactly once"
+                );
+                transport.assert_requests_match(&[]);
+            } else {
+                let error = result.expect_err("the compatibility code must not hide a non-503 failure");
+                assert_eq!(s3_probe_error_code(error.as_ref()), Some("SlowDownRead"));
+                assert_eq!(transport.actual_requests().count(), 1, "non-availability errors must fail immediately");
+            }
+        }
+        Ok(())
+    }
+
     #[tokio::test]
     async fn object_polls_distinguish_committed_and_replicated_keys() -> TestResult {
         let target = FakeS3Target::start().await?;

crates/e2e_test/src/distributed/mod.rs

@@ -40,5 +40,6 @@ mod scanner_heal_topology_fixture;
 mod scanner_heal_topology_test;
 mod scanner_heal_topology_tier_test;
 mod site_replication_test;
+mod table_catalog_test;
 mod upgrade_test;
 mod versioning_test;

crates/e2e_test/src/distributed/multipart_fence_acceptance_test.rs

@@ -1129,9 +1129,198 @@ async fn phase_quorum_matrix(
     Ok(())
 }
 
+async fn list_parts_disk_loss_after_metadata_read(
+    cluster: &RustFSTestClusterEnvironment,
+    proxies: &[RpcFaultProxy],
+    clients: &[Client],
+    bucket: &str,
+) -> TestResult {
+    let key = "list-parts-after-metadata-loss";
+    let body = payload_for("list-parts-acknowledged", 4096);
+    let upload = clients[0].create_multipart_upload().bucket(bucket).key(key).send().await?;
+    let upload_id = upload.upload_id().ok_or("missing upload ID")?.to_owned();
+    let part = clients[0]
+        .upload_part()
+        .bucket(bucket)
+        .key(key)
+        .upload_id(&upload_id)
+        .part_number(1)
+        .body(ByteStream::from(body.clone()))
+        .send()
+        .await?;
+    let etag = part.e_tag().ok_or("missing part ETag")?.to_owned();
+    let path = upload_path_scope(bucket, key, &upload_id)?;
+    let root = Path::new(&cluster.nodes[0].data_dirs[0])
+        .join(RUSTFS_META_MULTIPART_BUCKET)
+        .join(&path);
+    let data_dirs = std::fs::read_dir(&root)?
+        .filter_map(Result::ok)
+        .map(|entry| entry.path())
+        .filter(|entry| entry.join("part.1").is_file() && entry.join("part.1.meta").is_file())
+        .collect::<Vec<_>>();
+    assert_eq!(data_dirs.len(), 1, "the coordinator's actual part directory must be identified");
+    let data_dir = &data_dirs[0];
+    let dir_name = data_dir
+        .file_name()
+        .and_then(|name| name.to_str())
+        .ok_or("non-UTF8 data directory")?;
+    let scoped_path = format!("{path}/{dir_name}");
+    let held_dir = data_dir.with_extension("e2e-held");
+    std::fs::rename(data_dir, &held_dir)?;
+    let pauses = proxies[1..]
+        .iter()
+        .map(|proxy| proxy.pause(&["ListDir"], usize::MAX, Some(&scoped_path)))
+        .collect::<Result<Vec<_>, _>>()?;
+    let request = clients[0].list_parts().bucket(bucket).key(key).upload_id(&upload_id);
+    let task = tokio::spawn(async move { request.send().await });
+    // ListDir is dispatched only after the upload metadata read succeeded.
+    // Require all three scoped remote requests; unrelated failures do not count.
+    wait_for_faults(&pauses, &["ListDir"]).await?;
+    let result = tokio::time::timeout(DEADLINE, task).await??;
+    for pause in &pauses {
+        pause.discard_pending();
+    }
+    drop(pauses);
+    std::fs::rename(&held_dir, data_dir)?;
+    let error = result.expect_err("one missing directory and three unavailable participants cannot prove an empty upload");
+    let response = error.raw_response().expect("ListParts must return an S3 service response");
+    assert_eq!(response.status().as_u16(), 503);
+    assert_eq!(error.as_service_error().and_then(|err| err.meta().code()), Some("SlowDownRead"));
+    assert!(response.headers().get("x-rustfs-error-cause").is_none());
+    assert!(response.headers().get("retry-after").is_none());
+    assert!(
+        response
+            .headers()
+            .get("content-type")
+            .is_some_and(|value| value.starts_with("application/xml"))
+    );
+    assert!(
+        response
+            .headers()
+            .get("x-amz-request-id")
+            .is_some_and(|value| !value.is_empty())
+    );
+    let listed = recover_after_fault("ListParts after directory restoration", || {
+        clients[0].list_parts().bucket(bucket).key(key).upload_id(&upload_id).send()
+    })
+    .await??;
+    assert_eq!(listed.parts().len(), 1);
+    assert_eq!(listed.parts()[0].e_tag(), Some(etag.as_str()));
+    clients[0]
+        .complete_multipart_upload()
+        .bucket(bucket)
+        .key(key)
+        .upload_id(&upload_id)
+        .multipart_upload(
+            CompletedMultipartUpload::builder()
+                .parts(CompletedPart::builder().part_number(1).e_tag(&etag).build())
+                .build(),
+        )
+        .send()
+        .await?;
+    for client in clients {
+        assert_object_bytes_after_fault(client, bucket, key, &body).await?;
+    }
+    eprintln!(
+        "MPU list proof: metadata read completed, three scoped ListDir RPCs held, missing local directory rejected; restored part and all readers verified"
+    );
+    Ok(())
+}
+
+async fn concurrent_capped_parts_respect_limit_on_distinct_nodes(clients: &[Client], bucket: &str) -> TestResult {
+    let key = "concurrent-capped-parts";
+    let upload = clients[0]
+        .create_multipart_upload()
+        .bucket(bucket)
+        .key(key)
+        .customize()
+        .mutate_request(|request| {
+            let separator = if request.uri().contains('?') { "&" } else { "?" };
+            let uri = format!("{}{separator}x-rustfs-max-total-object-size=10", request.uri());
+            request
+                .set_uri(uri)
+                .expect("the signed multipart size query must be a valid URI");
+        })
+        .send()
+        .await?;
+    let upload_id = upload.upload_id().ok_or("missing capped upload ID")?.to_owned();
+    let barrier = std::sync::Arc::new(tokio::sync::Barrier::new(3));
+    let mut tasks = Vec::new();
+    for (index, number, byte) in [(0_usize, 1_i32, 0x41_u8), (1, 2, 0x42)] {
+        let client = clients[index].clone();
+        let bucket = bucket.to_owned();
+        let upload_id = upload_id.clone();
+        let barrier = std::sync::Arc::clone(&barrier);
+        tasks.push(tokio::spawn(async move {
+            barrier.wait().await;
+            let result = client
+                .upload_part()
+                .bucket(&bucket)
+                .key(key)
+                .upload_id(&upload_id)
+                .part_number(number)
+                .body(ByteStream::from(vec![byte; 6]))
+                .send()
+                .await;
+            (number, byte, result)
+        }));
+    }
+    barrier.wait().await;
+    let mut winner = None;
+    let mut rejected = 0;
+    for task in tasks {
+        let (number, byte, result) = tokio::time::timeout(DEADLINE, task).await??;
+        match result {
+            Ok(part) => {
+                assert!(winner.is_none(), "both six-byte parts cannot fit the ten-byte limit");
+                winner = Some((number, byte, part.e_tag().ok_or("missing accepted ETag")?.to_owned()));
+            }
+            Err(error) => {
+                assert_eq!(
+                    error.as_service_error().and_then(|err| err.meta().code()),
+                    Some("EntityTooLarge"),
+                    "{error:?}"
+                );
+                rejected += 1;
+            }
+        }
+    }
+    let (number, byte, etag) = winner.ok_or("no part was admitted")?;
+    assert_eq!(rejected, 1);
+    let listed = clients[0]
+        .list_parts()
+        .bucket(bucket)
+        .key(key)
+        .upload_id(&upload_id)
+        .send()
+        .await?;
+    assert_eq!(listed.parts().len(), 1);
+    assert_eq!(listed.parts()[0].part_number(), Some(number));
+    assert_eq!(listed.parts()[0].e_tag(), Some(etag.as_str()));
+    clients[0]
+        .complete_multipart_upload()
+        .bucket(bucket)
+        .key(key)
+        .upload_id(&upload_id)
+        .multipart_upload(
+            CompletedMultipartUpload::builder()
+                .parts(CompletedPart::builder().part_number(number).e_tag(&etag).build())
+                .build(),
+        )
+        .send()
+        .await?;
+    for client in clients {
+        assert_object_bytes_after_fault(client, bucket, key, &[byte; 6]).await?;
+    }
+    eprintln!("MPU size limit: two coordinators raced; exactly one six-byte part admitted under ten-byte limit");
+    Ok(())
+}
+
 #[tokio::test]
 async fn metadata_delay_preserves_write_quorum_and_rejects_quorum_minus_one() -> TestResult {
     let (cluster, proxies, clients, bucket) = fixture(1, "EC:2").await?;
+    list_parts_disk_loss_after_metadata_read(&cluster, &proxies, &clients, &bucket).await?;
+    concurrent_capped_parts_respect_limit_on_distinct_nodes(&clients, &bucket).await?;
     phase_quorum_matrix(&cluster, &proxies, &clients, &bucket, false, 1, 3).await
 }
 
@@ -1214,6 +1403,22 @@ async fn mixed_version_failed_complete_requires_current_coordinator_for_abort()
     cluster.start_node(0).await?;
     crate::upgrade_compatibility_test::wait_for_upgrade_write_readiness(&clients, &bucket, "mixed-mpu-current-writer", DEADLINE)
         .await?;
+    // Read-only ListParts remains available during the documented mutation pause.
+    // Compare the actual old and current coordinators on the same legacy upload.
+    for (version, client) in [("current", &clients[0]), ("previous", &clients[1])] {
+        let listed = client
+            .list_parts()
+            .bucket(&bucket)
+            .key(key)
+            .upload_id(upload_id)
+            .send()
+            .await?;
+        assert_eq!(listed.parts().len(), 1, "{version} coordinator must list the legacy part");
+        assert_eq!(listed.parts()[0].part_number(), Some(1));
+        assert_eq!(listed.parts()[0].size(), Some(9 * 1024 * 1024));
+        assert_eq!(listed.parts()[0].e_tag(), part.e_tag());
+    }
+    eprintln!("MPU mixed version: old and current coordinators agree on legacy ListParts identity and size");
     let scope = upload_scope(upload_id)?;
     let observers = proxies[1..]
         .iter()
@@ -1344,7 +1549,7 @@ async fn native_certified_partial_abort_retry_reclaims_survivors_below_metadata_
         capability_sha256: Option<[u8; 32]>,
         abort: Option<AbortRef>,
     }
-    let (cluster, proxies, clients, bucket) = fixture(4, "EC:4").await?;
+    let (mut cluster, proxies, clients, bucket) = fixture(4, "EC:4").await?;
     let key = "partial-abort";
     let committed = payload_for("committed-before-abort", 4096);
     clients[0]
@@ -1475,6 +1680,48 @@ async fn native_certified_partial_abort_retry_reclaims_survivors_below_metadata_
             .raw_response()
             .and_then(|response| response.headers().get("x-amz-request-id"))
     );
+    assert!(
+        retained.values().all(|source| source.participant.0 != 3),
+        "the selected offline node must have no surviving root"
+    );
+    cluster.stop_node(3)?;
+    let offline = tokio::time::timeout(
+        DEADLINE,
+        clients[0]
+            .abort_multipart_upload()
+            .bucket(&bucket)
+            .key(key)
+            .upload_id(upload_id)
+            .send(),
+    )
+    .await?
+    .expect_err("an unavailable original participant cannot be replaced by the certificate");
+    assert_unavailable(&offline, "storage_write_quorum", None);
+    assert!(
+        multipart_sources(&cluster, Some(&hash))? == retained,
+        "an unavailable participant must not authorize deletion or change retained evidence"
+    );
+    cluster.start_node(3).await?;
+    wait_for_storage_peer_recovery(&proxies, &clients[0], &bucket, upload_id, 4).await?;
+    eprintln!("MPU certified residual: unavailable original returned storage_write_quorum; all surviving bytes retained");
+
+    // Kill the coordinator and a node owning a real surviving source. The
+    // retained bytes and full-N certificate must survive both process restarts
+    // before a fresh coordinator can authorize the retry.
+    let survivor_node = retained.values().next().ok_or("missing surviving source")?.participant.0;
+    let mut restarted = vec![0, survivor_node];
+    restarted.sort_unstable();
+    restarted.dedup();
+    for node in restarted {
+        cluster.stop_node(node)?;
+        cluster.start_node(node).await?;
+        wait_for_storage_peer_recovery(&proxies, &clients[0], &bucket, upload_id, 4).await?;
+        assert!(
+            multipart_sources(&cluster, Some(&hash))? == retained,
+            "restart must preserve every surviving source and certificate byte"
+        );
+    }
+    eprintln!("MPU certified residual: coordinator and surviving owner restarted; all retained bytes unchanged");
     let delete_observers = proxies
         .iter()
         .map(|proxy| proxy.pause(&["RecoverMultipartSource"], usize::MAX, Some(&path)))
@@ -1638,6 +1885,191 @@ async fn late_complete_packets_cannot_publish_after_abort_and_bucket_recreation(
             "refused late packets must not recreate staging"
         );
     }
+    late_part_packets_preserve_retired_and_fresh_uploads().await?;
+    Ok(())
+}
+
+async fn late_part_packets_preserve_retired_and_fresh_uploads() -> TestResult {
+    for (method, settles) in [("PreparePartTransaction", false), ("SettlePartTransaction", true)] {
+        for recreate_bucket in [false, true] {
+            let (cluster, proxies, clients, bucket) = fixture(1, "EC:2").await?;
+            let key = "late-part-transaction";
+            let previous = payload_for("committed-before-late-part", 4096);
+            clients[0]
+                .put_object()
+                .bucket(&bucket)
+                .key(key)
+                .body(ByteStream::from(previous.clone()))
+                .send()
+                .await?;
+            let upload = clients[0].create_multipart_upload().bucket(&bucket).key(key).send().await?;
+            let upload_id = upload.upload_id().ok_or("missing original upload ID")?;
+            clients[0]
+                .upload_part()
+                .bucket(&bucket)
+                .key(key)
+                .upload_id(upload_id)
+                .part_number(1)
+                .body(ByteStream::from(vec![0x41; 9 * 1024 * 1024]))
+                .send()
+                .await?;
+            let path = upload_path_scope(&bucket, key, upload_id)?;
+            let hash = sha256_hex(format!("{bucket}/{key}").as_bytes());
+            let roots = cluster
+                .nodes
+                .iter()
+                .flat_map(|node| &node.data_dirs)
+                .map(|root| Path::new(root).join(RUSTFS_META_MULTIPART_BUCKET).join(&path))
+                .collect::<Vec<_>>();
+            assert_eq!(roots.len(), 4);
+            let original = multipart_sources(&cluster, Some(&hash))?;
+            assert_eq!(original.len(), 4);
+            assert!(
+                original.values().all(|source| source
+                    .files
+                    .keys()
+                    .any(|path| path.file_name().and_then(|name| name.to_str()) == Some("part.1"))),
+                "all selected peers originally own the acknowledged part"
+            );
+            let scope = upload_scope(upload_id)?;
+            // Retain only the original attempt; recovery of the same upload
+            // must be able to issue fresh settlement requests through the gate.
+            let pauses = proxies[1..3]
+                .iter()
+                .map(|proxy| proxy.pause_late(&[method], 1, &scope))
+                .collect::<Result<Vec<_>, _>>()?;
+            let replacement = clients[0]
+                .upload_part()
+                .bucket(&bucket)
+                .key(key)
+                .upload_id(upload_id)
+                .part_number(1)
+                .body(ByteStream::from(vec![0x62; 8 * 1024 * 1024]));
+            let caller = tokio::spawn(async move { replacement.send().await });
+            wait_for_faults(&pauses, &[method]).await?;
+            let failure = tokio::time::timeout(DEADLINE, caller)
+                .await??
+                .expect_err("two retained peers cannot acknowledge part replacement");
+            assert_unavailable(&failure, "storage_write_quorum", None);
+            assert!(pauses.iter().all(|pause| pause.hits() == 1 && pause.responses() == 0));
+            let expected_actions = if settles { (1, 0) } else { (0, 0) };
+            assert!(
+                pauses.iter().all(|pause| pause.retained_settle_actions() == expected_actions),
+                "settlement capture must be a real Commit, not Rollback"
+            );
+            abort_after_fault(&clients[0], &bucket, key, upload_id, false).await?;
+            assert!(
+                roots.iter().all(|root| !root.exists()),
+                "ordinary Abort physically retires all original roots"
+            );
+            assert!(multipart_sources(&cluster, Some(&hash))?.is_empty());
+            for client in &clients {
+                assert_object_bytes_after_fault(client, &bucket, key, &previous).await?;
+            }
+            let protected = if recreate_bucket {
+                clients[0].delete_object().bucket(&bucket).key(key).send().await?;
+                clients[0].delete_bucket().bucket(&bucket).send().await?;
+                clients[1].create_bucket().bucket(&bucket).send().await?;
+                let fresh = payload_for("recreated-before-late-part", 4096);
+                clients[1]
+                    .put_object()
+                    .bucket(&bucket)
+                    .key(key)
+                    .body(ByteStream::from(fresh.clone()))
+                    .send()
+                    .await?;
+                fresh
+            } else {
+                previous
+            };
+            let fresh = clients[2].create_multipart_upload().bucket(&bucket).key(key).send().await?;
+            let fresh_id = fresh.upload_id().ok_or("missing fresh upload ID")?;
+            let fresh_body = payload_for("fresh-active-part", 4096);
+            let fresh_part = clients[2]
+                .upload_part()
+                .bucket(&bucket)
+                .key(key)
+                .upload_id(fresh_id)
+                .part_number(1)
+                .body(ByteStream::from(fresh_body.clone()))
+                .send()
+                .await?;
+            let before = multipart_sources(&cluster, Some(&hash))?;
+            assert_eq!(before.len(), 4, "the new upload exists on every original participant");
+            for pause in &pauses {
+                pause.release();
+            }
+            wait_until(
+                DEADLINE,
+                || async {
+                    Ok(pauses.iter().all(|pause| {
+                        pause.responses() == pause.hits()
+                            && if settles {
+                                pause.successful_mutation_replies() == pause.hits() && pause.refused_mutations() == 0
+                            } else {
+                                pause.refused_mutations() == pause.hits() && pause.successful_mutation_replies() == 0
+                            }
+                    }))
+                },
+                "every original part packet returns its typed authenticated terminal outcome",
+            )
+            .await?;
+            assert!(roots.iter().all(|root| !root.exists()), "late part packets cannot recreate retired roots");
+            assert!(
+                multipart_sources(&cluster, Some(&hash))? == before,
+                "late packets cannot change a fresh upload's shard or control bytes"
+            );
+            for client in &clients {
+                assert_object_bytes_after_fault(client, &bucket, key, &protected).await?;
+                let listed = client
+                    .list_parts()
+                    .bucket(&bucket)
+                    .key(key)
+                    .upload_id(fresh_id)
+                    .send()
+                    .await?;
+                assert_eq!(listed.parts().len(), 1);
+                assert_eq!(listed.parts()[0].e_tag(), fresh_part.e_tag());
+                let missing = client
+                    .list_parts()
+                    .bucket(&bucket)
+                    .key(key)
+                    .upload_id(upload_id)
+                    .send()
+                    .await
+                    .expect_err("old upload remains retired");
+                assert_eq!(missing.raw_response().expect("old upload service response").status().as_u16(), 404);
+                assert_eq!(missing.as_service_error().and_then(|error| error.meta().code()), Some("NoSuchUpload"));
+            }
+            clients[2]
+                .complete_multipart_upload()
+                .bucket(&bucket)
+                .key(key)
+                .upload_id(fresh_id)
+                .multipart_upload(
+                    CompletedMultipartUpload::builder()
+                        .parts(
+                            CompletedPart::builder()
+                                .part_number(1)
+                                .e_tag(fresh_part.e_tag().ok_or("missing fresh part ETag")?)
+                                .build(),
+                        )
+                        .build(),
+                )
+                .send()
+                .await?;
+            for client in &clients {
+                assert_object_bytes_after_fault(client, &bucket, key, &fresh_body).await?;
+            }
+            assert!(multipart_sources(&cluster, Some(&hash))?.is_empty());
+            assert_no_uploads_after_fault(&clients, &bucket).await?;
+            eprintln!(
+                "MPU late part: method={method}, recreate_bucket={recreate_bucket}, retired_roots=4, delivered=2, refused={}, successful_replies={}, fresh_sources_unchanged=4, verified_readers=4",
+                pauses.iter().map(RpcPause::refused_mutations).sum::<usize>(),
+                pauses.iter().map(RpcPause::successful_mutation_replies).sum::<usize>()
+            );
+        }
+    }
     Ok(())
 }
 

crates/e2e_test/src/distributed/table_catalog_test.rs

@@ -0,0 +1,140 @@
+// Copyright 2026 RustFS Team
+//
+// Licensed under the Apache License, Version 2.0 (the "License");
+// you may not use this file except in compliance with the License.
+// You may obtain a copy of the License at
+//
+//     http://www.apache.org/licenses/LICENSE-2.0
+//
+// Unless required by applicable law or agreed to in writing, software
+// distributed under the License is distributed on an "AS IS" BASIS,
+// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+// See the License for the specific language governing permissions and
+// limitations under the License.
+
+use super::harness::{DistCluster, DistLayout, TestResult, get_object_bytes, put_object, unique_bucket};
+use crate::common::{init_logging, signed_request};
+use http::{Method, StatusCode};
+use serde_json::{Value, json};
+
+async fn catalog_request(
+    dist: &DistCluster,
+    node: usize,
+    method: Method,
+    path: &str,
+    body: Option<Value>,
+    expected: StatusCode,
+) -> TestResult<Value> {
+    let response = signed_request(
+        method.clone(),
+        &format!("{}{path}", dist.cluster.nodes[node].url),
+        &dist.cluster.access_key,
+        &dist.cluster.secret_key,
+        body.map(|value| serde_json::to_vec(&value)).transpose()?,
+        Some("application/json"),
+    )
+    .await?;
+    let status = response.status();
+    let text = response.text().await?;
+    assert_eq!(status, expected, "node {node}: {method} {path}: {text}");
+    if text.is_empty() {
+        Ok(Value::Null)
+    } else {
+        Ok(serde_json::from_str(&text)?)
+    }
+}
+
+async fn assert_bucket_state(dist: &DistCluster, bucket: &str, enabled: bool) -> TestResult {
+    for node in 0..dist.cluster.nodes.len() {
+        let response =
+            catalog_request(dist, node, Method::GET, &format!("/iceberg/v1/buckets/{bucket}"), None, StatusCode::OK).await?;
+        assert_eq!(response["enabled"], enabled, "node {node}: {response}");
+        assert_eq!(response["table-bucket"], bucket);
+        let expected = if enabled { StatusCode::OK } else { StatusCode::BAD_REQUEST };
+        catalog_request(dist, node, Method::GET, &format!("/iceberg/v1/{bucket}/namespaces"), None, expected).await?;
+        assert_eq!(
+            get_object_bytes(&dist.client(node)?, bucket, "ordinary.txt").await?,
+            b"ordinary bucket object",
+            "node {node}: toggling table support changed an ordinary object"
+        );
+    }
+    Ok(())
+}
+
+async fn assert_table_state(dist: &DistCluster, table_path: &str, created: &Value, pointer: &Value) -> TestResult {
+    for node in 0..dist.cluster.nodes.len() {
+        let loaded = catalog_request(dist, node, Method::GET, table_path, None, StatusCode::OK).await?;
+        assert_eq!(loaded["metadata-location"], created["metadata-location"], "node {node}");
+        assert_eq!(loaded["metadata"], created["metadata"], "node {node}");
+        let actual_pointer =
+            catalog_request(dist, node, Method::GET, &format!("{table_path}/metadata-location"), None, StatusCode::OK).await?;
+        assert_eq!(&actual_pointer, pointer, "node {node}: catalog pointer changed");
+    }
+    Ok(())
+}
+
+#[tokio::test]
+async fn table_bucket_toggles_reload_cached_metadata_on_every_peer() -> TestResult {
+    init_logging();
+    let dist = DistCluster::start_with_env(DistLayout::FourByFour, &[("RUSTFS_TABLE_CATALOG_BACKING", "object")]).await?;
+    let bucket = unique_bucket("table-peer-reload");
+    dist.create_bucket(&bucket).await?;
+    put_object(&dist.client(0)?, &bucket, "ordinary.txt", b"ordinary bucket object".to_vec()).await?;
+    let bucket_path = format!("/iceberg/v1/buckets/{bucket}");
+
+    // Warm each node's false cache before changing the marker. Do not poll:
+    // successful toggles must notify peers before their response is returned.
+    assert_bucket_state(&dist, &bucket, false).await?;
+    for (node, method, enabled) in [(0, Method::PUT, true), (1, Method::DELETE, false), (2, Method::PUT, true)] {
+        let response = catalog_request(&dist, node, method, &bucket_path, None, StatusCode::OK).await?;
+        assert_eq!(response["enabled"], enabled);
+        assert_bucket_state(&dist, &bucket, enabled).await?;
+    }
+
+    let namespaces = format!("/iceberg/v1/{bucket}/namespaces");
+    catalog_request(&dist, 3, Method::POST, &namespaces, Some(json!({"namespace": ["lab"]})), StatusCode::OK).await?;
+    let tables = format!("{namespaces}/lab/tables");
+    let created = catalog_request(
+        &dist,
+        0,
+        Method::POST,
+        &tables,
+        Some(json!({
+            "name": "events",
+            "schema": {
+                "type": "struct",
+                "fields": [{"id": 1, "name": "id", "required": false, "type": "long"}]
+            },
+            "stage-create": false
+        })),
+        StatusCode::OK,
+    )
+    .await?;
+    let uuid = created["metadata"]["table-uuid"]
+        .as_str()
+        .ok_or("create response omitted table UUID")?;
+    assert!(!uuid::Uuid::parse_str(uuid)?.is_nil());
+    assert!(
+        created["metadata-location"]
+            .as_str()
+            .is_some_and(|location| location.starts_with(&format!("s3://{bucket}/")))
+    );
+    let table_path = format!("{tables}/events");
+    let pointer =
+        catalog_request(&dist, 3, Method::GET, &format!("{table_path}/metadata-location"), None, StatusCode::OK).await?;
+    assert_eq!(pointer["metadata-location"], created["metadata-location"]);
+    assert!(pointer["version-token"].as_str().is_some_and(|token| !token.is_empty()));
+    assert!(pointer["generation"].as_u64().is_some());
+    assert_eq!(pointer["warehouse-location"], created["metadata"]["location"]);
+    assert_table_state(&dist, &table_path, &created, &pointer).await?;
+
+    // A rejected toggle must preserve both peer protections and catalog state.
+    catalog_request(&dist, 1, Method::DELETE, &bucket_path, None, StatusCode::CONFLICT).await?;
+    assert_bucket_state(&dist, &bucket, true).await?;
+    assert_table_state(&dist, &table_path, &created, &pointer).await?;
+
+    catalog_request(&dist, 2, Method::DELETE, &table_path, None, StatusCode::NO_CONTENT).await?;
+    catalog_request(&dist, 3, Method::DELETE, &format!("{namespaces}/lab"), None, StatusCode::NO_CONTENT).await?;
+    // The cluster fixture owns all remaining catalog objects and data roots.
+    Ok(())
+}

crates/e2e_test/src/distributed/upgrade_test.rs

@@ -47,6 +47,9 @@ struct UpgradeSeed {
     history_bucket: String,
     history_key: &'static str,
     history_body: Vec<u8>,
+    multipart_upload: String,
+    multipart_etag: String,
+    multipart_body: Vec<u8>,
     versioned_bucket: String,
     versioned_key: &'static str,
     version1: String,
@@ -241,6 +244,25 @@ async fn seed_history_and_iam(dist: &DistCluster) -> TestResult<UpgradeSeed> {
     let history_body = b"written by the previous 4-node release".to_vec();
     put_object(&root, &history_bucket, history_key, history_body.clone()).await?;
 
+    let multipart_body = vec![0x5a; 4096];
+    let multipart = root
+        .create_multipart_upload()
+        .bucket(&history_bucket)
+        .key("legacy-multipart.bin")
+        .send()
+        .await?;
+    let multipart_upload = multipart.upload_id().ok_or("legacy multipart upload ID missing")?.to_owned();
+    let part = root
+        .upload_part()
+        .bucket(&history_bucket)
+        .key("legacy-multipart.bin")
+        .upload_id(&multipart_upload)
+        .part_number(1)
+        .body(aws_sdk_s3::primitives::ByteStream::from(multipart_body.clone()))
+        .send()
+        .await?;
+    let multipart_etag = part.e_tag().ok_or("legacy multipart part ETag missing")?.to_owned();
+
     let versioned_key = "versioned-history.txt";
     let version1_body = b"version-one-before-upgrade".to_vec();
     let version1 = root
@@ -278,6 +300,9 @@ async fn seed_history_and_iam(dist: &DistCluster) -> TestResult<UpgradeSeed> {
         history_bucket,
         history_key,
         history_body,
+        multipart_upload,
+        multipart_etag,
+        multipart_body,
         versioned_bucket,
         versioned_key,
         version1,
@@ -304,6 +329,21 @@ async fn assert_history_and_iam(dist: &DistCluster, seed: &UpgradeSeed, context:
     )
     .await?;
     assert_object_bytes(&root_a, &seed.history_bucket, seed.history_key, &seed.history_body).await?;
+    // During mixed-version operation this is read-only; mutation resumes only
+    // after the whole fleet is upgraded.
+    for client in [&root_a, &root_b] {
+        let listed = client
+            .list_parts()
+            .bucket(&seed.history_bucket)
+            .key("legacy-multipart.bin")
+            .upload_id(&seed.multipart_upload)
+            .send()
+            .await?;
+        assert_eq!(listed.parts().len(), 1, "{context}: legacy part listing");
+        assert_eq!(listed.parts()[0].part_number(), Some(1));
+        assert_eq!(listed.parts()[0].size(), Some(4096));
+        assert_eq!(listed.parts()[0].e_tag(), Some(seed.multipart_etag.as_str()));
+    }
 
     let v1 = root_b
         .get_object()
@@ -387,17 +427,146 @@ async fn assert_history_and_iam(dist: &DistCluster, seed: &UpgradeSeed, context:
     Ok(())
 }
 
+async fn compare_degraded_legacy_list_parts(dist: &mut DistCluster, seed: &UpgradeSeed, previous: &Path) -> TestResult {
+    let decoded = String::from_utf8(base64_simd::URL_SAFE_NO_PAD.decode_to_vec(seed.multipart_upload.as_bytes())?)?;
+    let (_, upload_dir) = decoded.split_once('.').ok_or("legacy upload ID lacks deployment delimiter")?;
+    let hash = sha256_hex(format!("{}/legacy-multipart.bin", seed.history_bucket).as_bytes());
+    let root = format!("{hash}/{upload_dir}");
+    let mut held = Vec::new();
+    let mut original_files = Vec::new();
+    for (index, node) in dist.cluster.nodes.iter().enumerate() {
+        let path = Path::new(&node.data_dirs[0])
+            .join(crate::storage_api::multipart_fence_acceptance::RUSTFS_META_MULTIPART_BUCKET)
+            .join(&root);
+        let entries = std::fs::read_dir(path)?.collect::<Result<Vec<_>, _>>()?;
+        let dirs = entries
+            .into_iter()
+            .map(|entry| entry.path())
+            .filter(|entry| entry.join("part.1").is_file() && entry.join("part.1.meta").is_file())
+            .collect::<Vec<_>>();
+        assert_eq!(dirs.len(), 1, "each configured disk must hold the original part before injection");
+        let source_root = dirs[0].parent().ok_or("part directory has no upload root")?;
+        for path in [
+            source_root.join("xl.meta"),
+            dirs[0].join("part.1"),
+            dirs[0].join("part.1.meta"),
+        ] {
+            original_files.push((path.clone(), std::fs::read(path)?));
+        }
+        if index < 3 {
+            held.push((dirs[0].clone(), dirs[0].with_extension("held-for-mixed-test")));
+        }
+    }
+    dist.cluster.stop_node(3)?;
+    for (original, temporary) in &held {
+        std::fs::rename(original, temporary)?;
+    }
+
+    let clients = dist.clients()?;
+    let current = tokio::time::timeout(
+        CREDENTIAL_TIMEOUT,
+        clients[0]
+            .list_parts()
+            .bucket(&seed.history_bucket)
+            .key("legacy-multipart.bin")
+            .upload_id(&seed.multipart_upload)
+            .send(),
+    )
+    .await?;
+    let legacy = tokio::time::timeout(
+        CREDENTIAL_TIMEOUT,
+        clients[1]
+            .list_parts()
+            .bucket(&seed.history_bucket)
+            .key("legacy-multipart.bin")
+            .upload_id(&seed.multipart_upload)
+            .send(),
+    )
+    .await?;
+    for (original, temporary) in &held {
+        std::fs::rename(temporary, original)?;
+    }
+    // Three online disks retain metadata quorum but have no part directory;
+    // the fourth disk is offline. A legacy empty response is not a proof that
+    // the unavailable participant has no part.
+    let legacy = legacy.expect("previous coordinator should expose its legacy empty-list behavior");
+    assert!(legacy.parts().is_empty());
+    let current = current.expect_err("current coordinator must not infer empty from missing directories and offline disks");
+    assert_eq!(current.raw_response().expect("current S3 response").status().as_u16(), 503);
+    assert_eq!(current.as_service_error().and_then(|error| error.meta().code()), Some("SlowDownRead"));
+    dist.cluster.start_node_from_binary(3, previous).await?;
+    wait_for_upgrade_write_readiness(&clients, &seed.history_bucket, "mixed-list-restored", CREDENTIAL_TIMEOUT).await?;
+    for (path, before) in original_files {
+        assert_eq!(
+            std::fs::read(&path)?,
+            before,
+            "the fault must preserve every original source file: {path:?}"
+        );
+    }
+    for client in [&clients[0], &clients[1]] {
+        let listed = client
+            .list_parts()
+            .bucket(&seed.history_bucket)
+            .key("legacy-multipart.bin")
+            .upload_id(&seed.multipart_upload)
+            .send()
+            .await?;
+        assert_eq!(listed.parts().len(), 1);
+        assert_eq!(listed.parts()[0].e_tag(), Some(seed.multipart_etag.as_str()));
+    }
+    eprintln!(
+        "MPU mixed degraded: one offline disk, three missing part directories; previous returned empty, current returned 503 SlowDownRead; original part restored on both"
+    );
+    Ok(())
+}
+
+async fn complete_legacy_multipart(dist: &DistCluster, seed: &UpgradeSeed) -> TestResult {
+    // Native legacy-source upgrade needs every original participant, not just
+    // the ordinary three-of-four write quorum proven by the earlier probes.
+    let clients = dist.clients()?;
+    crate::upgrade_compatibility_test::wait_for_drive_readmission(
+        &clients,
+        &seed.history_bucket,
+        "legacy-multipart-all-drives",
+        CREDENTIAL_TIMEOUT,
+    )
+    .await?;
+    let root = &clients[0];
+    root.complete_multipart_upload()
+        .bucket(&seed.history_bucket)
+        .key("legacy-multipart.bin")
+        .upload_id(&seed.multipart_upload)
+        .multipart_upload(
+            aws_sdk_s3::types::CompletedMultipartUpload::builder()
+                .parts(
+                    aws_sdk_s3::types::CompletedPart::builder()
+                        .part_number(1)
+                        .e_tag(&seed.multipart_etag)
+                        .build(),
+                )
+                .build(),
+        )
+        .send()
+        .await?;
+    for client in &clients {
+        assert_object_bytes(client, &seed.history_bucket, "legacy-multipart.bin", &seed.multipart_body).await?;
+    }
+    Ok(())
+}
+
 #[tokio::test]
 async fn four_node_direct_upgrade_preserves_history_and_iam_credentials() -> TestResult {
     init_logging();
     let previous = source_binary()?;
     let mut dist = DistCluster::new_stopped(DistLayout::FourNodeFourDisk).await?;
+    dist.cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
     capture_upgrade_logs(&mut dist, "direct-upgrade")?;
     dist.start_from_binary(&previous).await?;
 
     let seed = seed_history_and_iam(&dist).await?;
     dist.restart_with_current_binary().await?;
     assert_history_and_iam(&dist, &seed, "direct").await?;
+    complete_legacy_multipart(&dist, &seed).await?;
     Ok(())
 }
 
@@ -419,6 +588,7 @@ async fn four_node_rolling_upgrade_preserves_history_and_iam_credentials() -> Te
     init_logging();
     let previous = source_binary()?;
     let mut dist = DistCluster::new_stopped(DistLayout::FourNodeFourDisk).await?;
+    dist.cluster.set_env("RUSTFS_STORAGE_CLASS_RRS", "EC:0");
     capture_upgrade_logs(&mut dist, "rolling-upgrade")?;
     dist.start_from_binary(&previous).await?;
 
@@ -426,6 +596,7 @@ async fn four_node_rolling_upgrade_preserves_history_and_iam_credentials() -> Te
 
     replace_node_and_wait_for_write_readiness(&mut dist, &seed, 0).await?;
     assert_history_and_iam(&dist, &seed, "one-current-node").await?;
+    compare_degraded_legacy_list_parts(&mut dist, &seed, &previous).await?;
 
     for node_idx in [1, 2] {
         replace_node_and_wait_for_write_readiness(&mut dist, &seed, node_idx).await?;
@@ -434,5 +605,6 @@ async fn four_node_rolling_upgrade_preserves_history_and_iam_credentials() -> Te
 
     replace_node_and_wait_for_write_readiness(&mut dist, &seed, 3).await?;
     assert_history_and_iam(&dist, &seed, "homogeneous-current").await?;
+    complete_legacy_multipart(&dist, &seed).await?;
     Ok(())
 }

crates/e2e_test/src/dual_stack_differential/compare.rs

@@ -1,512 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! What one stack answered, and how two answers are compared.
-//!
-//! An [`Observation`] keeps only what a client acts on: the HTTP status, the S3 error code, the
-//! allowlisted response headers and a few typed fields decoded from the response. The error
-//! message is kept for diagnostics and never compared: message wording is judged by the wire
-//! corpus, not here. The comparator is pure so it can be tested without a server.
-
-use serde::Serialize;
-use std::collections::{BTreeMap, BTreeSet};
-use std::path::{Path, PathBuf};
-
-/// Response headers compared exactly (after [`normalized_header_value`]).
-const HEADER_ALLOWLIST: [&str; 7] = [
-    "content-type",
-    "x-amz-version-id",
-    "etag",
-    "x-amz-delete-marker",
-    "content-range",
-    "accept-ranges",
-    "x-amz-request-charged",
-];
-
-/// Every response header with this prefix is compared as well.
-const HEADER_ALLOWLIST_PREFIX: &str = "x-amz-server-side-encryption";
-
-/// Placeholder for a field one side did not send.
-const ABSENT: &str = "<absent>";
-
-/// The lowercase header name if the comparator looks at this header.
-pub(crate) fn compared_header(name: &str) -> Option<String> {
-    let name = name.to_ascii_lowercase();
-    (HEADER_ALLOWLIST.contains(&name.as_str()) || name.starts_with(HEADER_ALLOWLIST_PREFIX)).then_some(name)
-}
-
-/// A version ID is minted per process, so its value never matches across two servers; only
-/// whether it is the literal `null` (an unversioned write) carries meaning.
-fn normalized_header_value(name: &str, value: &str) -> String {
-    if name == "x-amz-version-id" && value != "null" {
-        "<version-id>".to_owned()
-    } else {
-        value.to_owned()
-    }
-}
-
-/// One request's outcome as a client sees it.
-#[derive(Clone, Debug, Serialize)]
-pub(crate) struct Observation {
-    /// The S3 operation, named as in the `s3s::S3` trait (`GetObject`).
-    pub op: &'static str,
-    /// Which request of the script this was (`normal`, `negative`, or a setup label).
-    pub case: &'static str,
-    /// HTTP status; 0 when no response arrived.
-    pub status: u16,
-    /// The S3 error code, or the SDK's error kind when there was no service error.
-    pub error_code: Option<String>,
-    /// Allowlisted headers only, lowercase names, normalized values.
-    pub headers: BTreeMap<String, String>,
-    /// Typed values decoded from the response body, never free text.
-    pub fields: BTreeMap<String, String>,
-    /// Diagnostic only; never compared.
-    #[serde(skip)]
-    pub message: Option<String>,
-}
-
-impl Observation {
-    /// Builds an observation, keeping only the allowlisted headers.
-    pub(crate) fn new<'h>(
-        op: &'static str,
-        case: &'static str,
-        status: u16,
-        error_code: Option<String>,
-        headers: impl IntoIterator<Item = (&'h str, &'h str)>,
-    ) -> Self {
-        let headers = headers
-            .into_iter()
-            .filter_map(|(name, value)| compared_header(name).map(|name| (name, value)))
-            .map(|(name, value)| {
-                let value = normalized_header_value(&name, value);
-                (name, value)
-            })
-            .collect();
-        Self {
-            op,
-            case,
-            status,
-            error_code,
-            headers,
-            fields: BTreeMap::new(),
-            message: None,
-        }
-    }
-
-    pub(crate) fn with_field(mut self, name: &str, value: impl Into<String>) -> Self {
-        self.fields.insert(name.to_owned(), value.into());
-        self
-    }
-
-    pub(crate) fn is_success(&self) -> bool {
-        (200..300).contains(&self.status)
-    }
-}
-
-/// One field on which the two stacks answered differently.
-#[derive(Clone, Debug, PartialEq, Eq, Serialize)]
-pub(crate) struct Difference {
-    pub op: String,
-    pub case: String,
-    pub field: String,
-    pub a: String,
-    pub b: String,
-}
-
-/// Every difference between two transcripts of the same script, matched by `(op, case)`.
-pub(crate) fn compare(a: &[Observation], b: &[Observation]) -> Vec<Difference> {
-    let mut differences = Vec::new();
-    let (index_a, duplicates_a) = index(a);
-    let (index_b, duplicates_b) = index(b);
-    for (op, case) in duplicates_a.union(&duplicates_b) {
-        let count = |side: &[Observation]| {
-            side.iter()
-                .filter(|observation| (observation.op, observation.case) == (*op, *case))
-                .count()
-                .to_string()
-        };
-        differences.push(difference(op, case, "duplicate_case", count(a), count(b)));
-    }
-    let keys: BTreeSet<_> = index_a.keys().chain(index_b.keys()).copied().collect();
-    for key @ (op, case) in keys {
-        match (index_a.get(&key), index_b.get(&key)) {
-            (Some(a), Some(b)) => compare_one(a, b, &mut differences),
-            (a, b) => {
-                let presence = |side: Option<&&Observation>| if side.is_some() { "present" } else { ABSENT }.to_owned();
-                differences.push(difference(op, case, "observation", presence(a), presence(b)));
-            }
-        }
-    }
-    differences
-}
-
-type Key = (&'static str, &'static str);
-
-fn index(side: &[Observation]) -> (BTreeMap<Key, &Observation>, BTreeSet<Key>) {
-    let mut index = BTreeMap::new();
-    let mut duplicates = BTreeSet::new();
-    for observation in side {
-        let key = (observation.op, observation.case);
-        if index.insert(key, observation).is_some() {
-            duplicates.insert(key);
-        }
-    }
-    (index, duplicates)
-}
-
-fn difference(op: &str, case: &str, field: &str, a: String, b: String) -> Difference {
-    Difference {
-        op: op.to_owned(),
-        case: case.to_owned(),
-        field: field.to_owned(),
-        a,
-        b,
-    }
-}
-
-fn compare_one(a: &Observation, b: &Observation, differences: &mut Vec<Difference>) {
-    let mut field = |name: String, value_a: Option<&str>, value_b: Option<&str>| {
-        if value_a != value_b {
-            let show = |value: Option<&str>| value.unwrap_or(ABSENT).to_owned();
-            differences.push(difference(a.op, a.case, &name, show(value_a), show(value_b)));
-        }
-    };
-    field(
-        "status".to_owned(),
-        Some(a.status.to_string().as_str()),
-        Some(b.status.to_string().as_str()),
-    );
-    field("error_code".to_owned(), a.error_code.as_deref(), b.error_code.as_deref());
-    let headers: BTreeSet<_> = a.headers.keys().chain(b.headers.keys()).collect();
-    for name in headers {
-        field(
-            format!("header.{name}"),
-            a.headers.get(name).map(String::as_str),
-            b.headers.get(name).map(String::as_str),
-        );
-    }
-    let fields: BTreeSet<_> = a.fields.keys().chain(b.fields.keys()).collect();
-    for name in fields {
-        field(
-            format!("body.{name}"),
-            a.fields.get(name).map(String::as_str),
-            b.fields.get(name).map(String::as_str),
-        );
-    }
-}
-
-/// Operations observed on both sides.
-pub(crate) fn compared_ops(a: &[Observation], b: &[Observation]) -> BTreeSet<&'static str> {
-    let ops_b: BTreeSet<_> = b.iter().map(|observation| observation.op).collect();
-    a.iter()
-        .map(|observation| observation.op)
-        .filter(|op| ops_b.contains(op))
-        .collect()
-}
-
-/// `(op, case)` pairs observed on both sides.
-pub(crate) fn compared_cases(a: &[Observation], b: &[Observation]) -> usize {
-    let cases_b: BTreeSet<_> = b.iter().map(|observation| (observation.op, observation.case)).collect();
-    a.iter()
-        .map(|observation| (observation.op, observation.case))
-        .collect::<BTreeSet<_>>()
-        .intersection(&cases_b)
-        .count()
-}
-
-/// The JSON report written for every run.
-#[derive(Debug, Serialize)]
-pub(crate) struct Report {
-    pub schema: u32,
-    pub run: String,
-    pub stack_a: String,
-    pub stack_b: String,
-    pub region_a: Option<String>,
-    pub region_b: Option<String>,
-    pub compared_ops: usize,
-    pub compared_cases: usize,
-    pub differences: Vec<Difference>,
-}
-
-impl Report {
-    /// Writes the report as `<directory>/<run>.json` and returns its path.
-    pub(crate) fn write_to(&self, directory: &Path) -> std::io::Result<PathBuf> {
-        std::fs::create_dir_all(directory)?;
-        let path = directory.join(format!("{}.json", self.run));
-        let json = serde_json::to_vec_pretty(self).map_err(std::io::Error::other)?;
-        std::fs::write(&path, json)?;
-        Ok(path)
-    }
-}
-
-/// One line per difference, for a failing assertion.
-pub(crate) fn render(differences: &[Difference]) -> String {
-    differences
-        .iter()
-        .map(|d| format!("({}, {}, {}, a={:?}, b={:?})", d.op, d.case, d.field, d.a, d.b))
-        .collect::<Vec<_>>()
-        .join("\n")
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    fn ok(op: &'static str) -> Observation {
-        Observation::new(op, "normal", 200, None, [("content-type", "application/xml")])
-    }
-
-    fn fields(differences: &[Difference]) -> Vec<&str> {
-        differences.iter().map(|d| d.field.as_str()).collect()
-    }
-
-    #[test]
-    fn identical_transcripts_have_no_difference() {
-        let a = vec![
-            ok("HeadBucket"),
-            ok("GetBucketLocation").with_field("location_constraint", "us-east-1"),
-        ];
-        let b = a.clone();
-        assert_eq!(compare(&a, &b), Vec::new());
-        assert_eq!(compared_ops(&a, &b).len(), 2);
-    }
-
-    #[test]
-    fn a_status_difference_is_reported() {
-        let a = vec![ok("PutBucketAcl")];
-        let b = vec![Observation::new(
-            "PutBucketAcl",
-            "normal",
-            403,
-            None,
-            [("content-type", "application/xml")],
-        )];
-        assert_eq!(
-            compare(&a, &b),
-            vec![Difference {
-                op: "PutBucketAcl".into(),
-                case: "normal".into(),
-                field: "status".into(),
-                a: "200".into(),
-                b: "403".into(),
-            }]
-        );
-    }
-
-    #[test]
-    fn an_error_code_difference_is_reported() {
-        let a = vec![Observation::new("GetObject", "negative", 404, Some("NoSuchKey".into()), [])];
-        let b = vec![Observation::new(
-            "GetObject",
-            "negative",
-            404,
-            Some("NoSuchBucket".into()),
-            [],
-        )];
-        let differences = compare(&a, &b);
-        assert_eq!(fields(&differences), ["error_code"]);
-        assert_eq!((differences[0].a.as_str(), differences[0].b.as_str()), ("NoSuchKey", "NoSuchBucket"));
-    }
-
-    #[test]
-    fn an_error_code_on_one_side_only_is_reported() {
-        let a = vec![Observation::new("HeadBucket", "negative", 404, Some("NotFound".into()), [])];
-        let b = vec![Observation::new("HeadBucket", "negative", 404, None, [])];
-        let differences = compare(&a, &b);
-        assert_eq!(fields(&differences), ["error_code"]);
-        assert_eq!(differences[0].b, ABSENT);
-    }
-
-    #[test]
-    fn an_allowlisted_header_difference_is_reported() {
-        let a = vec![Observation::new("HeadObject", "normal", 200, None, [("ETag", "\"aa\"")])];
-        let b = vec![Observation::new("HeadObject", "normal", 200, None, [("etag", "\"bb\"")])];
-        assert_eq!(fields(&compare(&a, &b)), ["header.etag"]);
-    }
-
-    #[test]
-    fn a_header_sent_by_one_side_only_is_reported() {
-        let a = vec![Observation::new(
-            "DeleteObject",
-            "normal",
-            204,
-            None,
-            [("x-amz-delete-marker", "true")],
-        )];
-        let b = vec![Observation::new("DeleteObject", "normal", 204, None, [])];
-        let differences = compare(&a, &b);
-        assert_eq!(fields(&differences), ["header.x-amz-delete-marker"]);
-        assert_eq!((differences[0].a.as_str(), differences[0].b.as_str()), ("true", ABSENT));
-    }
-
-    #[test]
-    fn every_server_side_encryption_header_is_compared() {
-        let header = "x-amz-server-side-encryption-customer-key-md5";
-        let a = vec![Observation::new("PutObject", "sse-c", 200, None, [(header, "one")])];
-        let b = vec![Observation::new("PutObject", "sse-c", 200, None, [(header, "two")])];
-        assert_eq!(fields(&compare(&a, &b)), [format!("header.{header}")]);
-    }
-
-    #[test]
-    fn headers_outside_the_allowlist_are_not_compared() {
-        let a = vec![Observation::new(
-            "ListBuckets",
-            "normal",
-            200,
-            None,
-            [
-                ("date", "Mon"),
-                ("x-amz-request-id", "1"),
-                ("server", "a"),
-                ("content-length", "10"),
-            ],
-        )];
-        let b = vec![Observation::new(
-            "ListBuckets",
-            "normal",
-            200,
-            None,
-            [
-                ("date", "Tue"),
-                ("x-amz-request-id", "2"),
-                ("server", "b"),
-                ("content-length", "11"),
-            ],
-        )];
-        assert_eq!(compare(&a, &b), Vec::new());
-        assert!(a[0].headers.is_empty(), "non-allowlisted headers were kept: {:?}", a[0].headers);
-    }
-
-    #[test]
-    fn distinct_version_ids_are_not_a_difference() {
-        let a = vec![Observation::new(
-            "PutObject",
-            "normal",
-            200,
-            None,
-            [("x-amz-version-id", "0b1c")],
-        )];
-        let b = vec![Observation::new(
-            "PutObject",
-            "normal",
-            200,
-            None,
-            [("x-amz-version-id", "9f8e")],
-        )];
-        assert_eq!(compare(&a, &b), Vec::new());
-    }
-
-    #[test]
-    fn a_null_version_id_against_a_real_one_is_reported() {
-        let a = vec![Observation::new(
-            "PutObject",
-            "normal",
-            200,
-            None,
-            [("x-amz-version-id", "null")],
-        )];
-        let b = vec![Observation::new(
-            "PutObject",
-            "normal",
-            200,
-            None,
-            [("x-amz-version-id", "9f8e")],
-        )];
-        assert_eq!(fields(&compare(&a, &b)), ["header.x-amz-version-id"]);
-    }
-
-    #[test]
-    fn a_body_field_difference_is_reported() {
-        let a = vec![ok("GetBucketLocation").with_field("location_constraint", "us-east-1")];
-        let b = vec![ok("GetBucketLocation").with_field("location_constraint", "eu-west-1")];
-        let differences = compare(&a, &b);
-        assert_eq!(fields(&differences), ["body.location_constraint"]);
-        assert_eq!((differences[0].a.as_str(), differences[0].b.as_str()), ("us-east-1", "eu-west-1"));
-    }
-
-    #[test]
-    fn a_body_field_on_one_side_only_is_reported() {
-        let a = vec![ok("ListObjectsV2").with_field("keys", "[]")];
-        let b = vec![ok("ListObjectsV2")];
-        assert_eq!(fields(&compare(&a, &b)), ["body.keys"]);
-    }
-
-    #[test]
-    fn error_messages_are_never_compared() {
-        let mut a = Observation::new("GetObject", "negative", 404, Some("NoSuchKey".into()), []);
-        let mut b = a.clone();
-        a.message = Some("The specified key does not exist.".into());
-        b.message = Some("Object not found".into());
-        assert_eq!(compare(&[a], &[b]), Vec::new());
-    }
-
-    #[test]
-    fn a_case_observed_on_one_side_only_is_reported() {
-        let a = vec![ok("HeadBucket"), ok("ListBuckets")];
-        let b = vec![ok("HeadBucket")];
-        let differences = compare(&a, &b);
-        assert_eq!(fields(&differences), ["observation"]);
-        assert_eq!(differences[0].op, "ListBuckets");
-        assert_eq!(compared_ops(&a, &b).into_iter().collect::<Vec<_>>(), ["HeadBucket"]);
-    }
-
-    #[test]
-    fn a_case_recorded_twice_is_reported() {
-        let a = vec![ok("HeadBucket"), ok("HeadBucket")];
-        let b = vec![ok("HeadBucket")];
-        assert_eq!(fields(&compare(&a, &b)), ["duplicate_case"]);
-    }
-
-    #[test]
-    fn compared_ops_counts_operations_not_cases() {
-        let negative = Observation::new("HeadBucket", "negative", 404, Some("NotFound".into()), []);
-        let a = vec![ok("HeadBucket"), negative, ok("ListBuckets")];
-        let b = a.clone();
-        assert_eq!(compared_ops(&a, &b).len(), 2);
-    }
-
-    #[test]
-    fn the_report_lists_each_difference_as_op_field_a_b() {
-        let directory = std::env::temp_dir().join(format!("dual-stack-report-{}", uuid::Uuid::new_v4()));
-        let report = Report {
-            schema: 1,
-            run: "unit".into(),
-            stack_a: "legacy".into(),
-            stack_b: "legacy".into(),
-            region_a: None,
-            region_b: Some("eu-west-1".into()),
-            compared_ops: 1,
-            compared_cases: 1,
-            differences: compare(
-                &[ok("GetBucketLocation").with_field("location_constraint", "us-east-1")],
-                &[ok("GetBucketLocation").with_field("location_constraint", "eu-west-1")],
-            ),
-        };
-        let path = report.write_to(&directory).expect("write the report");
-        let json: serde_json::Value = serde_json::from_slice(&std::fs::read(&path).expect("read the report")).expect("parse");
-        let _ = std::fs::remove_dir_all(&directory);
-        assert_eq!(path.file_name().and_then(|name| name.to_str()), Some("unit.json"));
-        assert_eq!(json["compared_ops"], 1);
-        assert_eq!(
-            json["differences"],
-            serde_json::json!([{
-                "op": "GetBucketLocation",
-                "case": "normal",
-                "field": "body.location_constraint",
-                "a": "us-east-1",
-                "b": "eu-west-1",
-            }])
-        );
-    }
-}

crates/e2e_test/src/dual_stack_differential/mod.rs

@@ -1,199 +0,0 @@
-// Copyright 2024 RustFS Team
-//
-// Licensed under the Apache License, Version 2.0 (the "License");
-// you may not use this file except in compliance with the License.
-// You may obtain a copy of the License at
-//
-//     http://www.apache.org/licenses/LICENSE-2.0
-//
-// Unless required by applicable law or agreed to in writing, software
-// distributed under the License is distributed on an "AS IS" BASIS,
-// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-// See the License for the specific language governing permissions and
-// limitations under the License.
-
-//! Dual-stack differential: one `rustfs` binary, two processes, one SDK script (rustfs/backlog#2740).
-//!
-//! Stack A always runs `RUSTFS_S3_STACK=legacy`. Stack B runs the value of `RUSTFS_E2E_STACK_B`
-//! (default `legacy` until the gateway stack is assembled), and `RUSTFS_E2E_STACK_B_REGION`, when
-//! set, starts B in another region so that a difference can be injected by hand. Both servers
-//! answer the script in `scenarios`, which covers every operation of `impl S3 for FS`; `compare`
-//! reports each `(op, field, a, b)` on which the answers differ, and every run writes its report
-//! to `<target>/dual-stack/<run>.json`. Two processes, not two embedded servers: bucket metadata
-//! is process-global, so two stacks in one process would read each other's buckets.
-
-mod compare;
-mod scenarios;
-
-use crate::common::{RustFSTestEnvironment, init_logging, workspace_root};
-use compare::{Difference, Observation, Report, compare, compared_cases, compared_ops, render};
-use std::path::PathBuf;
-use std::time::{SystemTime, UNIX_EPOCH};
-
-type BoxError = Box<dyn std::error::Error + Send + Sync>;
-
-const STACK_A: &str = "legacy";
-const ENV_STACK_B: &str = "RUSTFS_E2E_STACK_B";
-const ENV_STACK_B_REGION: &str = "RUSTFS_E2E_STACK_B_REGION";
-/// Becomes `gateway` once rustfs/backlog#2734 T2.5 assembles the gateway stack.
-const DEFAULT_STACK_B: &str = "legacy";
-/// The region the negative control gives stack A explicitly, so an inherited `RUSTFS_REGION`
-/// cannot erase the injected difference. Not `us-east-1`: that region answers GetBucketLocation
-/// with an empty LocationConstraint and re-creating an owned bucket with 200, so it would not
-/// report its own name and would add CreateBucket differences to the control.
-const CONTROL_REGION_A: &str = "us-west-2";
-const CONTROL_REGION_B: &str = "eu-west-1";
-/// `normal` cases the legacy stack must answer with a 2xx. The comparison alone cannot tell a
-/// script that exercises real behavior from one in which both stacks fail the same way. Five of
-/// the 73 fail on legacy by design of this script: PutBucketReplication (no registered remote
-/// target), GetBucketReplication (so no configuration), PutBucketNotificationConfiguration (no
-/// notification target configured), RestoreObject (the object is not archived) and
-/// GetObjectTorrent (legacy answers `NoSuchKey`).
-const MIN_NORMAL_SUCCESSES: usize = 68;
-
-struct PairConfig {
-    stack_b: String,
-    region_a: Option<String>,
-    region_b: Option<String>,
-}
-
-impl PairConfig {
-    fn from_env() -> Self {
-        Self {
-            stack_b: std::env::var(ENV_STACK_B).unwrap_or_else(|_| DEFAULT_STACK_B.to_owned()),
-            region_a: None,
-            region_b: std::env::var(ENV_STACK_B_REGION).ok(),
-        }
-    }
-}
-
-/// Starts one server on `stack`, overriding its region only when asked.
-async fn start_stack(stack: &str, region: Option<&str>) -> Result<RustFSTestEnvironment, BoxError> {
-    let mut env = RustFSTestEnvironment::new().await?;
-    let mut vars = vec![("RUSTFS_S3_STACK", stack)];
-    if let Some(region) = region {
-        vars.push(("RUSTFS_REGION", region));
-    }
-    env.start_rustfs_server_without_cleanup_with_env(&vars).await?;
-    Ok(env)
-}
-
-/// Starts stack A and stack B side by side, each with its own port and data directory.
-async fn spawn_pair(config: &PairConfig) -> Result<(RustFSTestEnvironment, RustFSTestEnvironment), BoxError> {
-    tokio::try_join!(
-        start_stack(STACK_A, config.region_a.as_deref()),
-        start_stack(&config.stack_b, config.region_b.as_deref())
-    )
-}
-
-fn report_directory() -> PathBuf {
-    let target = std::env::var_os("CARGO_TARGET_DIR").map_or_else(|| PathBuf::from("target"), PathBuf::from);
-    let target = if target.is_absolute() {
-        target
-    } else {
-        workspace_root().join(target)
-    };
-    target.join("dual-stack")
-}
-
-struct Run {
-    report: Report,
-    path: PathBuf,
-    a: Vec<Observation>,
-}
-
-/// Runs the script against both stacks concurrently and writes the report.
-async fn run_differential(label: &str, config: &PairConfig) -> Run {
-    let (server_a, server_b) = spawn_pair(config).await.expect("start stack A and stack B");
-    let (a, b) = tokio::join!(Box::pin(scenarios::run(&server_a)), Box::pin(scenarios::run(&server_b)));
-    let millis = SystemTime::now()
-        .duration_since(UNIX_EPOCH)
-        .map_or(0, |elapsed| elapsed.as_millis());
-    let report = Report {
-        schema: 1,
-        run: format!("{label}-{millis}-{}", std::process::id()),
-        stack_a: STACK_A.to_owned(),
-        stack_b: config.stack_b.clone(),
-        region_a: config.region_a.clone(),
-        region_b: config.region_b.clone(),
-        compared_ops: compared_ops(&a, &b).len(),
-        compared_cases: compared_cases(&a, &b),
-        differences: compare(&a, &b),
-    };
-    let path = report.write_to(&report_directory()).expect("write the dual-stack report");
-    Run { report, path, a }
-}
-
-#[cfg(test)]
-mod tests {
-    use super::*;
-
-    #[tokio::test]
-    async fn both_stacks_answer_every_operation_identically() {
-        init_logging();
-        let config = PairConfig::from_env();
-        let run = run_differential("self-comparison", &config).await;
-        let report = &run.report;
-
-        let mut observed: Vec<_> = run.a.iter().map(|observation| observation.op).collect();
-        observed.sort_unstable();
-        observed.dedup();
-        assert_eq!(
-            observed,
-            scenarios::OPERATIONS,
-            "the script must drive exactly the operations of impl S3 for FS"
-        );
-        assert_eq!(report.compared_ops, scenarios::OPERATIONS.len(), "report: {}", run.path.display());
-        assert!(
-            report.differences.is_empty(),
-            "stack {} and stack {} differ on {} of {} cases (report: {}):\n{}",
-            report.stack_a,
-            report.stack_b,
-            report.differences.len(),
-            report.compared_cases,
-            run.path.display(),
-            render(&report.differences)
-        );
-
-        let not_4xx: Vec<_> = run
-            .a
-            .iter()
-            .filter(|observation| observation.case == "negative" && !(400..500).contains(&observation.status))
-            .collect();
-        assert!(not_4xx.is_empty(), "negative cases must fail with a 4xx on {STACK_A}: {not_4xx:#?}");
-        let successes = run
-            .a
-            .iter()
-            .filter(|observation| observation.case == "normal" && observation.is_success())
-            .count();
-        assert!(
-            successes >= MIN_NORMAL_SUCCESSES,
-            "only {successes} normal cases succeeded on {STACK_A}: {:#?}",
-            run.a
-        );
-    }
-
-    #[tokio::test]
-    async fn differential_detects_an_injected_difference() {
-        init_logging();
-        let config = PairConfig {
-            stack_b: STACK_A.to_owned(),
-            region_a: Some(CONTROL_REGION_A.to_owned()),
-            region_b: Some(CONTROL_REGION_B.to_owned()),
-        };
-        let run = run_differential("injected-region", &config).await;
-        let expected = Difference {
-            op: "GetBucketLocation".to_owned(),
-            case: "normal".to_owned(),
-            field: "body.location_constraint".to_owned(),
-            a: CONTROL_REGION_A.to_owned(),
-            b: CONTROL_REGION_B.to_owned(),
-        };
-        assert!(
-            run.report.differences.contains(&expected),
-            "B started in {CONTROL_REGION_B} but GetBucketLocation was not reported (report: {}):\n{}",
-            run.path.display(),
-            render(&run.report.differences)
-        );
-    }
-}

crates/e2e_test/src/heal_erasure_disk_rebuild_test.rs

@@ -1716,6 +1716,15 @@ mod tests {
         cluster.set_env("RUSTFS_HEAL_AUTO_HEAL_ENABLE", background_enabled.to_string());
         cluster.set_env("RUSTFS_HEAL_MRF_ENABLE", background_enabled.to_string());
         cluster.set_env("RUSTFS_SCANNER_ENABLED", background_enabled.to_string());
+        let bucket = "heal-restart-during-rebuild";
+        let outage_key_prefix = "cluster/written-while-node-down-";
+        if !background_enabled {
+            // With MRF disabled, an outage PUT still hands its missing shard to a
+            // best-effort repair on the PUT coordinator. That repair retries until
+            // the target rejoins, so suppress it to keep the explicit root heal
+            // the only owner of the outage object.
+            cluster.set_env("RUSTFS_E2E_SUPPRESS_RENAME_TAIL_HEAL_PREFIX", format!("{bucket}/{outage_key_prefix}"));
+        }
         if background_enabled {
             // Only the scanner cadence is accelerated. Keep normal Heal
             // concurrency and every automatic recovery owner enabled.
@@ -1748,7 +1757,6 @@ mod tests {
         cluster.start_with_binary(&server_binary).await?;
         let clients = cluster.create_all_clients()?;
 
-        let bucket = "heal-restart-during-rebuild";
         clients[0].create_bucket().bucket(bucket).send().await?;
 
         let ReplacementDriveSelection {
@@ -1876,7 +1884,7 @@ mod tests {
         let mut replacement_outage_erasure_index = None;
         let mut rejected_outage_keys = Vec::new();
         for attempt in 0..max_outage_write_attempts {
-            let candidate_key = format!("cluster/written-while-node-down-{attempt:04}.bin");
+            let candidate_key = format!("{outage_key_prefix}{attempt:04}.bin");
             let put_result = timeout(
                 Duration::from_secs(30),
                 clients[2]
@@ -1986,25 +1994,6 @@ mod tests {
         }
 
         let heal_body = r#"{"recursive":true,"dryRun":false,"remove":false,"recreate":true,"scanMode":2,"updateParity":false,"nolock":false}"#;
-        if !background_enabled {
-            // The PUT path may have admitted a direct Internal object repair while
-            // node 1 was offline. Cancel the isolated bucket path before the target
-            // returns; otherwise it could rebuild the outage object and invalidate
-            // the explicit-root ownership assertion below.
-            let cancel_outage_heal_path = format!("/rustfs/admin/v3/heal/{bucket}?forceStop=true");
-            let (cancel_status, cancel_body) = admin_request(
-                &cluster.nodes[0].url,
-                Method::POST,
-                &cancel_outage_heal_path,
-                Some(heal_body.to_string()),
-                &cluster.access_key,
-                &cluster.secret_key,
-            )
-            .await?;
-            if !cancel_status.is_success() {
-                return Err(format!("cancel outage heal failed: {cancel_status} {cancel_body}").into());
-            }
-        }
 
         // Keep the partial-repair checkpoint stable across readiness and admin
         // requests. Endpoint-blackhole tests must prove their own network stall.

crates/e2e_test/src/lib.rs

@@ -31,12 +31,6 @@ pub mod chaos;
 #[cfg(test)]
 pub mod fake_s3_target;
 
-// Dual-stack differential (rustfs/backlog#2740): one binary, two processes on
-// different RUSTFS_S3_STACK values, the same SDK script for every S3 operation,
-// compared on status, error code, allowlisted headers and typed body fields.
-#[cfg(test)]
-mod dual_stack_differential;
-
 // On-demand migration (backlog#2147): shared two-server environment, admin
 // wrappers, and the harness self-test (backlog#2151). Behavior scenarios are
 // added by later ODM tasks.

crates/e2e_test/src/list_objects_v2_pagination_test.rs

@@ -38,6 +38,8 @@ mod tests {
     use tracing::info;
 
     const PAGINATION_TIMEOUT: Duration = Duration::from_secs(120);
+    // Bound queued durable commits on the single fixture disk under CI I/O pressure.
+    const FIXTURE_PUT_CONCURRENCY: usize = 4;
 
     // Fixture population can require thousands of durable PUTs. Bound the
     // listing traversal itself with one deadline shared by every page.
@@ -564,7 +566,7 @@ mod tests {
         let object_count = 1002;
         // Finish every durable PUT before listing, with the same bounded fanout as the delimiter fixtures.
         stream::iter(0..object_count)
-            .for_each_concurrent(16, |i| {
+            .for_each_concurrent(FIXTURE_PUT_CONCURRENCY, |i| {
                 let client = &client;
                 async move {
                     let key = format!("object{i:04}.txt");
@@ -575,7 +577,7 @@ mod tests {
                         .body(ByteStream::from_static(b"test content"))
                         .send()
                         .await
-                        .unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err}"));
+                        .unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err:?}"));
                 }
             })
             .await;
@@ -822,7 +824,7 @@ mod tests {
             .flat_map(|dir| (0..10).map(move |file| format!("dir-{dir:03}/file{file:02}.txt")))
             .collect();
         stream::iter(&all_keys)
-            .for_each_concurrent(16, |key| {
+            .for_each_concurrent(FIXTURE_PUT_CONCURRENCY, |key| {
                 let client = &client;
                 async move {
                     client
@@ -832,7 +834,7 @@ mod tests {
                         .body(ByteStream::from_static(b"x"))
                         .send()
                         .await
-                        .unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err}"));
+                        .unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err:?}"));
                 }
             })
             .await;
@@ -1055,7 +1057,7 @@ mod tests {
         let files_per_dir = 100;
         let fixture_started = std::time::Instant::now();
         stream::iter((0..dir_count).flat_map(|d| (0..files_per_dir).map(move |f| (d, f))))
-            .for_each_concurrent(16, |(d, f)| {
+            .for_each_concurrent(FIXTURE_PUT_CONCURRENCY, |(d, f)| {
                 let client = &client;
                 async move {
                     let key = format!("dir-{d:02}/file{f:03}.txt");
@@ -1066,7 +1068,7 @@ mod tests {
                         .body(ByteStream::from_static(b"x"))
                         .send()
                         .await
-                        .unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err}"));
+                        .unwrap_or_else(|err| panic!("Failed to put fixture object {key}: {err:?}"));
                 }
             })
             .await;

crates/e2e_test/src/protocols/webdav_core.rs

@@ -32,9 +32,7 @@
 //! Advisory: <https://github.com/rustfs/rustfs/security/advisories/GHSA-3p3x-734c-h5vx>
 
 use crate::common::rustfs_binary_path_with_features;
-use crate::common::{
-    AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via, signed_s3_request,
-};
+use crate::common::{AdminTransport, admin_add_canned_policy_via, admin_attach_user_policy_via, admin_create_user_via};
 use crate::common::{FAST_DATA_USAGE_SCANNER_ENV, admin_request};
 use crate::protocols::test_env::{DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY, ProtocolTestEnvironment};
 use anyhow::Result;
@@ -776,81 +774,6 @@ async fn test_webdav_core_operations_direct() -> Result<()> {
     test_webdav_core_operations().await
 }
 
-#[tokio::test]
-async fn test_webdav_content_type_directory_marker() -> Result<()> {
-    let env = ProtocolTestEnvironment::new().map_err(|e| anyhow::anyhow!("{}", e))?;
-    let admin_base_url = format!("http://{}", S3_TEST_ADDRESS);
-    let binary_path = rustfs_binary_path_with_features(Some("webdav"));
-    let mut server_process = Command::new(&binary_path)
-        .arg("--address")
-        .arg(S3_TEST_ADDRESS)
-        .env("RUSTFS_CONSOLE_ENABLE", "false")
-        .env("RUSTFS_WEBDAV_ENABLE", "true")
-        .env("RUSTFS_WEBDAV_ADDRESS", WEBDAV_ADDRESS)
-        .env("RUSTFS_WEBDAV_TLS_ENABLED", "false")
-        .arg(&env.temp_dir)
-        .spawn()?;
-
-    let result = async {
-        ProtocolTestEnvironment::wait_for_port_ready(WEBDAV_PORT, 30)
-            .await
-            .map_err(|e| anyhow::anyhow!("{}", e))?;
-
-        let client = create_client();
-        let bucket_name = "webdav-content-type-bucket";
-        let directory_name = "content-type-directory";
-        let webdav_base_url = format!("http://{}", WEBDAV_ADDRESS);
-        let mkcol_bucket = client
-            .request(
-                reqwest::Method::from_bytes(b"MKCOL").expect("MKCOL is a valid HTTP method"),
-                format!("{}/{}", webdav_base_url, bucket_name),
-            )
-            .header("Authorization", basic_auth_header())
-            .send()
-            .await?;
-        anyhow::ensure!(
-            mkcol_bucket.status().is_success() || mkcol_bucket.status().as_u16() == 201,
-            "MKCOL bucket should succeed, got: {}",
-            mkcol_bucket.status()
-        );
-
-        let mkcol_directory = client
-            .request(
-                reqwest::Method::from_bytes(b"MKCOL").expect("MKCOL is a valid HTTP method"),
-                format!("{}/{}/{}", webdav_base_url, bucket_name, directory_name),
-            )
-            .header("Authorization", basic_auth_header())
-            .send()
-            .await?;
-        anyhow::ensure!(
-            mkcol_directory.status().is_success() || mkcol_directory.status().as_u16() == 201,
-            "MKCOL directory should succeed, got: {}",
-            mkcol_directory.status()
-        );
-
-        let head_url = format!("{}/{}/{}/", admin_base_url, bucket_name, directory_name);
-        let head = signed_s3_request(http::Method::HEAD, &head_url, None, None, DEFAULT_ACCESS_KEY, DEFAULT_SECRET_KEY)
-            .await
-            .map_err(|error| anyhow::anyhow!("{error}"))?;
-        anyhow::ensure!(head.status().is_success(), "HEAD directory marker should succeed, got: {}", head.status());
-        let content_type = head
-            .headers()
-            .get("content-type")
-            .and_then(|value| value.to_str().ok())
-            .unwrap_or_default();
-        anyhow::ensure!(
-            content_type == "application/x-directory",
-            "WebDAV directory marker must preserve its content type, got: {content_type}"
-        );
-        Ok(())
-    }
-    .await;
-
-    let _ = server_process.kill().await;
-    let _ = server_process.wait().await;
-    result
-}
-
 /// Extract the numeric value of the first occurrence of a DAV property in a
 /// PROPFIND multistatus body. Namespace-prefix agnostic; a self-closing
 /// (unreported) property yields None.

crates/e2e_test/src/rpc_fault_proxy.rs

@@ -27,8 +27,8 @@ use hyper_util::{
 };
 use prost::Message as _;
 use rustfs_protos::proto_gen::node_service::{
-    BatchGenerallyLockRequest, DeleteRequest, GenerallyLockRequest, ReadVersionRequest, RecoverMultipartSourceRequest,
-    RenameDataResponse,
+    BatchGenerallyLockRequest, DeleteRequest, GenerallyLockRequest, PreparePartTransactionResponse, ReadVersionRequest,
+    RecoverMultipartSourceRequest, RenameDataResponse, SettlePartTransactionRequest, SettlePartTransactionResponse,
 };
 use std::{
     collections::HashSet,
@@ -56,6 +56,35 @@ enum Disposition {
     Discard,
 }
 
+#[derive(Clone, Copy)]
+enum MutationReply {
+    None,
+    Rename,
+    PreparePart,
+    SettlePart,
+}
+
+impl MutationReply {
+    fn outcome(self, bytes: &[u8]) -> io::Result<Option<(bool, bool)>> {
+        let result = match self {
+            Self::None => return Ok(None),
+            Self::Rename => {
+                let reply = RenameDataResponse::decode(unary_payload(bytes)?).map_err(io::Error::other)?;
+                (reply.success, reply.error.is_some())
+            }
+            Self::PreparePart => {
+                let reply = PreparePartTransactionResponse::decode(unary_payload(bytes)?).map_err(io::Error::other)?;
+                (reply.success, reply.error.is_some())
+            }
+            Self::SettlePart => {
+                let reply = SettlePartTransactionResponse::decode(unary_payload(bytes)?).map_err(io::Error::other)?;
+                (reply.success, reply.error.is_some())
+            }
+        };
+        Ok(Some(result))
+    }
+}
+
 struct PauseState {
     methods: Vec<&'static str>,
     scope: Option<Vec<u8>>,
@@ -64,6 +93,9 @@ struct PauseState {
     limit: usize,
     retain_after_disconnect: bool,
     refused_mutations: AtomicUsize,
+    successful_mutation_replies: AtomicUsize,
+    settle_commits: AtomicUsize,
+    settle_rollbacks: AtomicUsize,
     responses: AtomicUsize,
     matched: AtomicUsize,
     blocked: AtomicUsize,
@@ -122,6 +154,17 @@ impl RpcPause {
         self.state.refused_mutations.load(Ordering::SeqCst)
     }
 
+    pub(crate) fn successful_mutation_replies(&self) -> usize {
+        self.state.successful_mutation_replies.load(Ordering::SeqCst)
+    }
+
+    pub(crate) fn retained_settle_actions(&self) -> (usize, usize) {
+        (
+            self.state.settle_commits.load(Ordering::SeqCst),
+            self.state.settle_rollbacks.load(Ordering::SeqCst),
+        )
+    }
+
     pub(crate) fn unsupported_source_inspection_routes(&self) -> HashSet<String> {
         self.state
             .unsupported_inspection_routes
@@ -394,6 +437,9 @@ impl RpcFaultProxy {
             limit,
             retain_after_disconnect,
             refused_mutations: AtomicUsize::new(0),
+            successful_mutation_replies: AtomicUsize::new(0),
+            settle_commits: AtomicUsize::new(0),
+            settle_rollbacks: AtomicUsize::new(0),
             responses: AtomicUsize::new(0),
             matched: AtomicUsize::new(0),
             blocked: AtomicUsize::new(0),
@@ -509,6 +555,7 @@ async fn forward_request(
         if let Some(method) = method.filter(|name| pause.methods.contains(name)) {
             let mut route = None;
             let mut lock_owners = Vec::new();
+            let mut settle_action = None;
             let matches = if let Some(scope) = pause.scope.as_deref() {
                 // Inspect bounded control routing while preserving the exact
                 // signed bytes and trailers forwarded to the backend.
@@ -522,12 +569,19 @@ async fn forward_request(
                     inspection = Some((pause.clone(), disk));
                 }
                 let mut matches = scope_matches(method, &bytes, scope)?;
+                if matches && method == "SettlePartTransaction" {
+                    settle_action = Some(
+                        SettlePartTransactionRequest::decode(unary_payload(&bytes)?)
+                            .map_err(io::Error::other)?
+                            .rollback,
+                    );
+                }
                 if matches && (pause.record_lock_owners || pause.lock_owner.is_some()) {
                     lock_owners = scoped_lock_owners(method, &bytes, scope)?;
                     if let Some(owner) = &pause.lock_owner {
                         matches = lock_owners.iter().any(|candidate| candidate == owner);
                         if !matches && pause.record_lock_owners {
-                            observed = Some((pause.clone(), None, std::mem::take(&mut lock_owners), false));
+                            observed = Some((pause.clone(), None, std::mem::take(&mut lock_owners), MutationReply::None));
                         }
                     }
                 }
@@ -549,6 +603,14 @@ async fn forward_request(
                 true
             };
             if matches && pause.matched.fetch_add(1, Ordering::SeqCst) < pause.limit {
+                if let Some(rollback) = settle_action {
+                    let counter = if rollback {
+                        &pause.settle_rollbacks
+                    } else {
+                        &pause.settle_commits
+                    };
+                    counter.fetch_add(1, Ordering::SeqCst);
+                }
                 let mut released = pause.release.subscribe();
                 pause.blocked.fetch_add(1, Ordering::SeqCst);
                 pause.reached.notify_one();
@@ -568,10 +630,14 @@ async fn forward_request(
                     pause,
                     route,
                     lock_owners,
-                    matches!(
-                        method,
-                        "RenameData" | "RenameMultipartData" | "RenameDataAtIncarnation" | "RenameDataWithQuotaFence"
-                    ),
+                    match method {
+                        "RenameData" | "RenameMultipartData" | "RenameDataAtIncarnation" | "RenameDataWithQuotaFence" => {
+                            MutationReply::Rename
+                        }
+                        "PreparePartTransaction" => MutationReply::PreparePart,
+                        "SettlePartTransaction" => MutationReply::SettlePart,
+                        _ => MutationReply::None,
+                    },
                 ));
             }
         }
@@ -604,7 +670,7 @@ async fn forward_request(
             .expect("inspection response routes")
             .insert(disk);
     }
-    if let Some((pause, route, lock_owners, rename)) = observed {
+    if let Some((pause, route, lock_owners, mutation_reply)) = observed {
         pause.responses.fetch_add(1, Ordering::SeqCst);
         if response.status().is_success() && response.headers().get("grpc-status").is_none_or(|status| status == "0") {
             pause.forwarded.fetch_add(1, Ordering::SeqCst);
@@ -626,10 +692,14 @@ async fn forward_request(
                 .headers
                 .get("grpc-status")
                 .or_else(|| trailers.as_ref()?.get("grpc-status"));
-            if rename && parts.status.is_success() && status.is_none_or(|status| status == "0") {
-                let result = RenameDataResponse::decode(unary_payload(&bytes)?).map_err(io::Error::other)?;
-                if !result.success && result.error.is_some() {
+            if parts.status.is_success()
+                && status.is_some_and(|status| status == "0")
+                && let Some((success, has_error)) = mutation_reply.outcome(&bytes)?
+            {
+                if !success && has_error {
                     pause.refused_mutations.fetch_add(1, Ordering::SeqCst);
+                } else if success && !has_error {
+                    pause.successful_mutation_replies.fetch_add(1, Ordering::SeqCst);
                 }
             }
             let body = Full::new(bytes)
@@ -676,6 +746,10 @@ mod tests {
     }
 
     async fn backend_with_status(status: Option<&'static str>) -> io::Result<Backend> {
+        backend_with_reply(status, None).await
+    }
+
+    async fn backend_with_reply(status: Option<&'static str>, reply: Option<Bytes>) -> io::Result<Backend> {
         let listener = TcpListener::bind("127.0.0.1:0").await?;
         let address = listener.local_addr()?;
         let received = Arc::new(AtomicUsize::new(0));
@@ -687,9 +761,11 @@ mod tests {
                     accepted = listener.accept() => {
                         let Ok((socket, _)) = accepted else { break };
                         let received = received_task.clone();
+                        let reply = reply.clone();
                         children.spawn(async move {
                             let service = service_fn(move |request: Request<Incoming>| {
                                 let received = received.clone();
+                                let reply = reply.clone();
                                 async move {
                                 received.fetch_add(1, Ordering::SeqCst);
                                 let authority = request.headers().get(http::header::HOST).cloned().or_else(|| {
@@ -698,6 +774,7 @@ mod tests {
                                     })
                                 });
                                 let body = request.into_body().collect().await?.to_bytes();
+                                let body = reply.unwrap_or(body);
                                 let mut trailers = http::HeaderMap::new();
                                 trailers.insert("grpc-status", http::HeaderValue::from_static(status.unwrap_or("0")));
                                 trailers.insert("x-proof", http::HeaderValue::from_static("unchanged-proof"));
@@ -1434,6 +1511,93 @@ mod tests {
         .await
         .expect("scoped discard deadline");
         assert_late_packet_ownership().await;
+        assert_late_part_reply_refusal().await;
+    }
+
+    async fn assert_late_part_reply_refusal() {
+        use rustfs_protos::proto_gen::node_service::{Error, PreparePartTransactionRequest};
+        tokio::time::timeout(Duration::from_secs(10), async {
+            // method, success, error, gRPC status, rollback, expected successes/refusals
+            for (method, success, has_error, status, rollback, expected) in [
+                ("PreparePartTransaction", false, true, None, false, (0, 1)),
+                ("PreparePartTransaction", true, false, None, false, (1, 0)),
+                ("PreparePartTransaction", false, false, None, false, (0, 0)),
+                ("SettlePartTransaction", true, false, None, false, (1, 0)),
+                ("SettlePartTransaction", false, true, None, true, (0, 1)),
+                ("SettlePartTransaction", true, true, None, false, (0, 0)),
+                ("SettlePartTransaction", false, true, Some("16"), false, (0, 0)),
+                ("PreparePartTransaction", false, true, Some("16"), false, (0, 0)),
+                ("RenameMultipartData", false, true, None, false, (0, 1)),
+            ] {
+                let error = has_error.then(|| Error {
+                    code: 1,
+                    error_info: "retired source".to_owned(),
+                });
+                let reply = match method {
+                    "PreparePartTransaction" => unary_frame(&PreparePartTransactionResponse { success, error }),
+                    "SettlePartTransaction" => unary_frame(&SettlePartTransactionResponse { success, error }),
+                    _ => unary_frame(&RenameDataResponse {
+                        success,
+                        error,
+                        ..Default::default()
+                    }),
+                };
+                // An unknown field must survive typed inspection byte-for-byte.
+                let mut framed = reply.to_vec();
+                framed.extend_from_slice(&[0xfa, 0x03, 0x03, b'x', b'y', b'z']);
+                let payload_len = u32::try_from(framed.len() - 5).expect("small reply");
+                framed[1..5].copy_from_slice(&payload_len.to_be_bytes());
+                let reply = Bytes::from(framed);
+                let backend = backend_with_reply(status, Some(reply.clone()))
+                    .await
+                    .expect("typed reply backend");
+                let proxy = RpcFaultProxy::start_on("127.0.0.1:0".parse().expect("loopback"), backend.address)
+                    .await
+                    .expect("proxy");
+                let pause = proxy.pause_late(&[method], 1, "late-upload").expect("part gate");
+                let request = match method {
+                    "PreparePartTransaction" => unary_frame(&PreparePartTransactionRequest {
+                        disk: "disk".to_owned(),
+                        dst_path: "late-upload/part.1".to_owned(),
+                        ..Default::default()
+                    }),
+                    "SettlePartTransaction" => unary_frame(&SettlePartTransactionRequest {
+                        disk: "disk".to_owned(),
+                        path: "late-upload/part.1".to_owned(),
+                        rollback,
+                        ..Default::default()
+                    }),
+                    _ => Bytes::from_static(b"late-upload"),
+                };
+                let address = proxy.local_addr();
+                let caller = tokio::spawn(async move {
+                    let client: ProxyClient = Client::builder(TokioExecutor::new()).http2_only(true).build_http();
+                    let response = call_body(&client, address, method, request).await.expect("retained reply");
+                    response.into_body().collect().await.expect("typed response frames")
+                });
+                pause.wait_for_hits(1).await.expect("actual retained request");
+                pause.release();
+                let response = caller.await.expect("caller task");
+                let trailers = response.trailers().expect("reply trailers");
+                assert_eq!(trailers["x-proof"], "unchanged-proof");
+                assert_eq!(trailers["grpc-status"], status.unwrap_or("0"));
+                assert_eq!(response.to_bytes(), reply);
+                assert_eq!(pause.responses(), 1);
+                assert_eq!(
+                    (pause.successful_mutation_replies(), pause.refused_mutations()),
+                    expected,
+                    "typed {method} outcome"
+                );
+                let actions = if method == "SettlePartTransaction" {
+                    if rollback { (0, 1) } else { (1, 0) }
+                } else {
+                    (0, 0)
+                };
+                assert_eq!(pause.retained_settle_actions(), actions);
+            }
+        })
+        .await
+        .expect("part reply deadline");
     }
     async fn assert_late_packet_ownership() {
         tokio::time::timeout(Duration::from_secs(10), async {

crates/e2e_test/src/snowball_auto_extract_test.rs

@@ -261,6 +261,61 @@ mod tests {
         archive
     }
 
+    #[tokio::test]
+    async fn snowball_auto_extract_rejects_veeam_virtual_members() -> Result<(), Box<dyn Error + Send + Sync>> {
+        init_logging();
+        let mut env = RustFSTestEnvironment::new().await?;
+        env.start_rustfs_server_with_env(vec![], &[("RUSTFS_VEEAM_SOSAPI_ENABLE", "true")])
+            .await?;
+        let client = env.create_s3_client();
+        let bucket = "snowball-veeam-reserved";
+        client.create_bucket().bucket(bucket).send().await?;
+        let prefix = ".system-d26a9498-cb7c-4a87-a44a-8ae204f5ba6c";
+        for (member, destination, use_prefix) in [
+            (format!("{prefix}/system.xml"), format!("{prefix}/system.xml"), false),
+            ("capacity.xml".to_string(), format!("{prefix}/capacity.xml"), true),
+        ] {
+            let mut builder = tokio_tar::Builder::new(Cursor::new(Vec::new()));
+            append_typed_entry(&mut builder, &member, tokio_tar::EntryType::Regular, b"blocked").await?;
+            let archive = builder.into_inner().await?.into_inner();
+            let mut request = client
+                .put_object()
+                .bucket(bucket)
+                .key("fixture.tar")
+                .metadata("Snowball-Auto-Extract", "true")
+                .body(ByteStream::from(archive));
+            if use_prefix {
+                request = request
+                    .metadata("Minio-Snowball-Prefix", prefix)
+                    .metadata("Minio-Snowball-Ignore-Errors", "true");
+            }
+            let error = request.send().await.expect_err("virtual archive members must be rejected");
+            assert_eq!(error.code(), Some("MethodNotAllowed"));
+            let listed = client.list_objects_v2().bucket(bucket).prefix(&destination).send().await?;
+            assert!(listed.contents().is_empty(), "rejected virtual member must not be stored");
+        }
+        let mut builder = tokio_tar::Builder::new(Cursor::new(Vec::new()));
+        append_typed_entry(&mut builder, "ordinary.xml", tokio_tar::EntryType::Regular, b"ordinary").await?;
+        client
+            .put_object()
+            .bucket(bucket)
+            .key("fixture.tar")
+            .metadata("Snowball-Auto-Extract", "true")
+            .metadata("Minio-Snowball-Prefix", prefix)
+            .body(ByteStream::from(builder.into_inner().await?.into_inner()))
+            .send()
+            .await?;
+        let ordinary = client
+            .get_object()
+            .bucket(bucket)
+            .key(format!("{prefix}/ordinary.xml"))
+            .send()
+            .await?;
+        assert_eq!(ordinary.body.collect().await?.into_bytes().as_ref(), b"ordinary");
+        env.stop_server();
+        Ok(())
+    }
+
     #[tokio::test]
     async fn snowball_auto_extract_supports_minio_prefix_and_directory_markers() -> Result<(), Box<dyn Error + Send + Sync>> {
         init_logging();

crates/e2e_test/src/stale_multipart_cleanup_cluster_test.rs

@@ -13,9 +13,13 @@
 // limitations under the License.
 
 use crate::common::{RustFSTestClusterEnvironment, init_logging};
+use crate::storage_api::multipart_fence_acceptance::RUSTFS_META_MULTIPART_BUCKET;
 use aws_sdk_s3::error::SdkError;
 use aws_sdk_s3::primitives::ByteStream;
-use tokio::time::{Duration, sleep};
+use sha2::{Digest, Sha256};
+use std::collections::BTreeMap;
+use std::path::{Path, PathBuf};
+use tokio::time::{Duration, Instant, sleep};
 use tracing::info;
 use uuid::Uuid;
 
@@ -84,6 +88,126 @@ async fn wait_for_cleanup_on_all_nodes(
     Err("stale multipart upload was not cleaned up on all nodes within timeout".into())
 }
 
+type PhysicalTree = BTreeMap<PathBuf, Option<Vec<u8>>>;
+
+fn physical_tree(root: &Path) -> Result<PhysicalTree, Box<dyn std::error::Error + Send + Sync>> {
+    let mut result = BTreeMap::new();
+    let mut pending = vec![root.to_path_buf()];
+    while let Some(directory) = pending.pop() {
+        for entry in std::fs::read_dir(directory)? {
+            let entry = entry?;
+            let path = entry.path();
+            let relative = path.strip_prefix(root)?.to_path_buf();
+            let kind = entry.file_type()?;
+            if kind.is_dir() {
+                result.insert(relative, None);
+                pending.push(path);
+            } else if kind.is_file() {
+                result.insert(relative, Some(std::fs::read(path)?));
+            } else {
+                return Err(format!("unexpected staging entry type: {relative:?}").into());
+            }
+        }
+    }
+    Ok(result)
+}
+
+async fn assert_low_quorum_residuals_are_diagnosed_and_preserved(
+    cluster: &RustFSTestClusterEnvironment,
+    clients: &[aws_sdk_s3::Client],
+    log_paths: &[PathBuf],
+) -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+    let mut residuals = Vec::new();
+    for copies in [1_usize, 2] {
+        let key = format!("multipart/partial-{copies}-{}", Uuid::new_v4().simple());
+        let upload = clients[0]
+            .create_multipart_upload()
+            .bucket(CLEANUP_BUCKET)
+            .key(&key)
+            .send()
+            .await?;
+        let upload_id = upload.upload_id().ok_or("create response missing upload ID")?;
+        clients[0]
+            .upload_part()
+            .bucket(CLEANUP_BUCKET)
+            .key(&key)
+            .upload_id(upload_id)
+            .part_number(1)
+            .body(ByteStream::from(vec![0x41; 4096]))
+            .send()
+            .await?;
+        let decoded = String::from_utf8(base64_simd::URL_SAFE_NO_PAD.decode_to_vec(upload_id.as_bytes())?)?;
+        let (_, upload_dir) = decoded.split_once('.').ok_or("upload ID lacks deployment delimiter")?;
+        let digest = Sha256::digest(format!("{CLEANUP_BUCKET}/{key}").as_bytes());
+        let hash = hex_simd::encode_to_string(digest, hex_simd::AsciiCase::Lower);
+        let relative = format!("{hash}/{upload_dir}");
+        let roots = cluster
+            .nodes
+            .iter()
+            .map(|node| {
+                Path::new(&node.data_dirs[0])
+                    .join(RUSTFS_META_MULTIPART_BUCKET)
+                    .join(&relative)
+            })
+            .collect::<Vec<_>>();
+        assert_eq!(roots.len(), 4);
+        let mut preserved = Vec::new();
+        for root in &roots {
+            assert!(
+                root.join("xl.meta").is_file(),
+                "all four original metadata files must exist before injection"
+            );
+            preserved.push(physical_tree(root)?);
+        }
+        for root in roots.iter().skip(copies) {
+            std::fs::remove_dir_all(root)?;
+        }
+        residuals.push((relative, roots, preserved, copies));
+    }
+
+    let deadline = Instant::now() + Duration::from_secs(45);
+    loop {
+        let logs = log_paths
+            .iter()
+            .filter_map(|path| std::fs::read_to_string(path).ok())
+            .collect::<Vec<_>>();
+        let diagnosed = residuals.iter().all(|(path, _, _, _)| {
+            logs.iter().any(|log| {
+                log.lines()
+                    .any(|line| line.contains(path) && line.contains("multipart_cleanup_lock_or_recheck_failed"))
+            })
+        });
+        if diagnosed {
+            break;
+        }
+        if Instant::now() >= deadline {
+            return Err("stale low-quorum residuals were not diagnosed before the deadline".into());
+        }
+        sleep(Duration::from_millis(200)).await;
+    }
+    for (path, roots, preserved, copies) in &residuals {
+        for slot in 0..*copies {
+            assert_eq!(
+                physical_tree(&roots[slot])?,
+                preserved[slot],
+                "retained source evidence changed: {path}, slot {slot}"
+            );
+        }
+        for root in roots.iter().skip(*copies) {
+            assert!(!root.exists(), "cleanup must not recreate missing source roots: {path}");
+        }
+    }
+    for path in log_paths {
+        let log = std::fs::read_to_string(path)?;
+        assert!(
+            !log.contains("op=\"upload_part\"") && !log.contains("\"op\":\"upload_part\""),
+            "background cleanup must not be logged as a client UploadPart"
+        );
+    }
+    eprintln!("Stale MPU low-quorum: EC:2 one/two-copy roots diagnosed; full surviving file inventories preserved");
+    Ok(())
+}
+
 #[tokio::test]
 async fn test_stale_multipart_cleanup_removes_incomplete_upload_across_cluster()
 -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
@@ -92,6 +216,22 @@ async fn test_stale_multipart_cleanup_removes_incomplete_upload_across_cluster()
     let mut cluster = RustFSTestClusterEnvironment::new(4).await?;
     cluster.set_env("RUSTFS_API_STALE_UPLOADS_EXPIRY", "5s");
     cluster.set_env("RUSTFS_API_STALE_UPLOADS_CLEANUP_INTERVAL", "1s");
+    cluster.set_env("RUSTFS_STORAGE_CLASS_STANDARD", "EC:2");
+    cluster.set_env(
+        "RUST_LOG",
+        concat!("warn,rustfs_ecstore", "::bucket::lifecycle::bucket_lifecycle_ops=debug"),
+    );
+    let log_dir = std::env::var_os("RUSTFS_E2E_LOG_DIR")
+        .map(PathBuf::from)
+        .map(|root| root.join(format!("stale-multipart-{}", Uuid::new_v4().simple())))
+        .unwrap_or_else(|| PathBuf::from(&cluster.temp_dir));
+    std::fs::create_dir_all(&log_dir)?;
+    let log_paths = (0..cluster.nodes.len())
+        .map(|index| log_dir.join(format!("stale-node-{index}.log")))
+        .collect::<Vec<_>>();
+    for (index, path) in log_paths.iter().enumerate() {
+        cluster.set_node_capture_log_path(index, path.to_string_lossy().into_owned())?;
+    }
     cluster.start().await?;
     cluster.create_test_bucket(CLEANUP_BUCKET).await?;
 
@@ -137,6 +277,7 @@ async fn test_stale_multipart_cleanup_removes_incomplete_upload_across_cluster()
     );
 
     wait_for_cleanup_on_all_nodes(&clients, CLEANUP_BUCKET, &key, &upload_id).await?;
+    assert_low_quorum_residuals_are_diagnosed_and_preserved(&cluster, &clients, &log_paths).await?;
 
     Ok(())
 }

crates/e2e_test/src/upgrade_compatibility_test.rs

@@ -1066,7 +1066,7 @@ pub(crate) async fn wait_for_upgrade_write_readiness(
 /// reduced-redundancy class then fails with 503 `ServiceUnavailable` until the
 /// serving node writes through every drive, on the previous release and on
 /// the current build alike.
-async fn wait_for_drive_readmission(clients: &[Client], bucket: &str, context: &str, budget: Duration) -> TestResult {
+pub(crate) async fn wait_for_drive_readmission(clients: &[Client], bucket: &str, context: &str, budget: Duration) -> TestResult {
     wait_for_upgrade_writes(
         clients,
         bucket,

crates/e2e_test/src/version_id_regression_test.rs

@@ -600,6 +600,154 @@ mod tests {
         info!("✅ PASSED: Veeam backup workflow simulation completed successfully");
     }
 
+    /// Veeam B&R discovers SOSAPI by GET/HEAD of two well-known keys.
+    #[tokio::test]
+    async fn test_veeam_sosapi_system_and_capacity() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
+        use crate::common::local_http_client;
+
+        const SYSTEM_KEY: &str = ".system-d26a9498-cb7c-4a87-a44a-8ae204f5ba6c/system.xml";
+        const CAPACITY_KEY: &str = ".system-d26a9498-cb7c-4a87-a44a-8ae204f5ba6c/capacity.xml";
+
+        init_logging();
+        info!("🧪 TEST: Veeam SOSAPI system.xml and capacity.xml");
+
+        let mut env = RustFSTestEnvironment::new().await?;
+        env.start_rustfs_server_with_env(vec![], crate::common::FAST_DATA_USAGE_SCANNER_ENV)
+            .await?;
+        let client = create_s3_client(&env);
+        let bucket = "veeam-sosapi";
+        create_bucket(&client, bucket).await?;
+
+        let system = client.get_object().bucket(bucket).key(SYSTEM_KEY).send().await?;
+        let system_etag = system.e_tag().map(str::to_string);
+        let system_len = system.content_length();
+        assert_eq!(system.content_type(), Some("application/xml"));
+        let system_body = system.body.collect().await?.into_bytes();
+        let system_xml = std::str::from_utf8(&system_body)?;
+        assert!(system_xml.contains("<ProtocolVersion>\"1.0\"</ProtocolVersion>"), "{system_xml}");
+        assert!(
+            system_xml.contains("<ModelName>\"RustFS ") && system_xml.contains("\"</ModelName>"),
+            "{system_xml}"
+        );
+        assert!(system_xml.contains("<CapacityInfo>true</CapacityInfo>"));
+        assert!(system_xml.contains("<UploadSessions>false</UploadSessions>"));
+        assert!(system_xml.contains("<IAMSTS>false</IAMSTS>"));
+        assert!(!system_xml.contains("SystemRecommendations"));
+        assert!(!system_xml.contains("APIEndpoints"));
+        assert!(!system_xml.contains("&quot;"));
+
+        let head = client.head_object().bucket(bucket).key(SYSTEM_KEY).send().await?;
+        assert_eq!(head.content_type(), Some("application/xml"));
+        assert_eq!(head.content_length(), system_len);
+        assert_eq!(head.e_tag().map(str::to_string), system_etag);
+        assert!(head.last_modified().is_some());
+
+        let ranged = client
+            .get_object()
+            .bucket(bucket)
+            .key(SYSTEM_KEY)
+            .range("bytes=0-3")
+            .send()
+            .await?;
+        assert_eq!(ranged.content_length(), Some(4));
+        let ranged_header = ranged.content_range().unwrap_or("");
+        assert!(ranged_header.starts_with("bytes 0-3/"), "{ranged_header}");
+        let ranged_body = ranged.body.collect().await?.into_bytes();
+        assert_eq!(&ranged_body[..], &system_body[..4]);
+
+        let capacity = tokio::time::timeout(std::time::Duration::from_secs(180), async {
+            loop {
+                match client.get_object().bucket(bucket).key(CAPACITY_KEY).send().await {
+                    Ok(capacity) => break Ok(capacity),
+                    Err(err) if err.code() == Some("ServiceUnavailable") => {
+                        tokio::time::sleep(std::time::Duration::from_secs(2)).await;
+                    }
+                    Err(err) => break Err(err),
+                }
+            }
+        })
+        .await??;
+        assert_eq!(capacity.content_type(), Some("application/xml"));
+        let capacity_body = capacity.body.collect().await?.into_bytes();
+        let capacity_xml = std::str::from_utf8(&capacity_body)?;
+        let capacity_value = xml_u64(capacity_xml, "Capacity")?;
+        let available_value = xml_u64(capacity_xml, "Available")?;
+        let used_value = xml_u64(capacity_xml, "Used")?;
+        assert!(used_value <= capacity_value, "{capacity_xml}");
+        assert!(available_value <= capacity_value.saturating_sub(used_value), "{capacity_xml}");
+
+        let put_err = client
+            .put_object()
+            .bucket(bucket)
+            .key(SYSTEM_KEY)
+            .body(ByteStream::from_static(b"overwrite"))
+            .send()
+            .await
+            .expect_err("SOSAPI put must fail");
+        assert_eq!(put_err.code(), Some("MethodNotAllowed"));
+        let delete_err = client
+            .delete_object()
+            .bucket(bucket)
+            .key(CAPACITY_KEY)
+            .send()
+            .await
+            .expect_err("SOSAPI delete must fail");
+        assert_eq!(delete_err.code(), Some("MethodNotAllowed"));
+
+        let normal_key = "Veeam/Backup/job.dat";
+        client
+            .put_object()
+            .bucket(bucket)
+            .key(normal_key)
+            .body(ByteStream::from_static(b"backup-bytes"))
+            .send()
+            .await?;
+        let normal = client.get_object().bucket(bucket).key(normal_key).send().await?;
+        let normal_body = normal.body.collect().await?.into_bytes();
+        assert_eq!(&normal_body[..], b"backup-bytes");
+
+        let anon = local_http_client()
+            .get(format!("{}/{bucket}/{SYSTEM_KEY}", env.url))
+            .send()
+            .await?;
+        assert_eq!(anon.status(), reqwest::StatusCode::FORBIDDEN);
+        let anon_body = anon.text().await?;
+        assert!(!anon_body.contains("ProtocolVersion"), "{anon_body}");
+
+        let now = time::OffsetDateTime::now_utc();
+        let scope_date = format!("{:04}{:02}{:02}", now.year(), u8::from(now.month()), now.day());
+        let amz_date = format!("{scope_date}T{:02}{:02}{:02}Z", now.hour(), now.minute(), now.second());
+        let bad = local_http_client()
+            .get(format!("{}/{bucket}/{SYSTEM_KEY}", env.url))
+            .header("x-amz-date", &amz_date)
+            .header("x-amz-content-sha256", "UNSIGNED-PAYLOAD")
+            .header(
+                "authorization",
+                format!(
+                    "AWS4-HMAC-SHA256 Credential=bad/{scope_date}/us-east-1/s3/aws4_request, SignedHeaders=host;x-amz-content-sha256;x-amz-date, Signature=0000000000000000000000000000000000000000000000000000000000000000"
+                ),
+            )
+            .send()
+            .await?;
+        assert_eq!(bad.status(), reqwest::StatusCode::FORBIDDEN);
+        let bad_body = bad.text().await?;
+        assert!(!bad_body.contains("ProtocolVersion"), "{bad_body}");
+
+        info!("✅ PASSED: Veeam SOSAPI system.xml and capacity.xml");
+        Ok(())
+    }
+
+    fn xml_u64(xml: &str, element: &str) -> Result<u64, Box<dyn std::error::Error + Send + Sync>> {
+        let start = format!("<{element}>");
+        let end = format!("</{element}>");
+        let start_idx = xml.find(&start).ok_or_else(|| format!("missing {element} in {xml}"))? + start.len();
+        let end_idx = xml[start_idx..]
+            .find(&end)
+            .ok_or_else(|| format!("unclosed {element} in {xml}"))?
+            + start_idx;
+        Ok(xml[start_idx..end_idx].parse()?)
+    }
+
     #[tokio::test]
     async fn test_terraform_put_after_delete() -> Result<(), Box<dyn std::error::Error + Send + Sync>> {
         init_logging();

crates/ecstore/AGENTS.md

@@ -12,6 +12,9 @@ Applies to `crates/ecstore/`.
 
 - Be careful with allocations and locking in hot paths.
 - Keep network and disk operations async-friendly; avoid introducing unnecessary blocking.
+- Coalesce related `open`, `read`/`write`, `fsync`, and `close` filesystem steps into a single per-disk blocking closure rather than chaining multiple async or blocking dispatches, and route any closure that performs file or directory sync to the disk's dedicated fsync lane so stalled syncs cannot starve foreground metadata reads.
+- Pin a parent directory once (`openat2(BENEATH | NO_SYMLINKS)` on Linux) and reuse the pinned handle across sibling reads or same-directory renames instead of re-walking ancestor components per file.
+- When the file size is already known from `fstat` metadata, pre-size the destination buffer (`len + 1`) before reading to EOF so small metadata reads complete in a single `read()` + EOF check rather than exponential-growth reallocations and syscalls.
 - Benchmark-sensitive changes should include measurable rationale.
 
 ## Allocation Discipline in Hot Paths

crates/ecstore/Cargo.toml

@@ -137,7 +137,6 @@ rustfs-lifecycle.workspace = true
 rustfs-s3-client = { workspace = true }
 rustfs-s3-types = { workspace = true }
 rustfs-data-usage.workspace = true
-rustfs-gateway-types.workspace = true
 rustfs-object-capacity.workspace = true
 rustfs-object-data-cache = { workspace = true, features = ["runtime-memory"] }
 arc-swap.workspace = true
@@ -156,6 +155,7 @@ serde = { workspace = true, features = ["derive"] }
 time = { workspace = true, features = ["parsing", "formatting", "macros", "serde"] }
 bytesize.workspace = true
 serde_json = { workspace = true, features = ["raw_value"] }
+s3s = { workspace = true, features = ["minio"] }
 http.workspace = true
 opentelemetry.workspace = true
 http-body = { workspace = true }
@@ -244,7 +244,6 @@ windows-sys = { workspace = true, features = [
 windows-sys = { workspace = true, features = ["Win32_System_Ioctl"] }
 
 [dev-dependencies]
-s3s = { workspace = true, features = ["minio"] }
 aws-smithy-async.workspace = true
 rustfs-s3-ops.workspace = true
 tokio = { workspace = true, features = ["rt-multi-thread", "macros", "test-util", "fs"] }

crates/ecstore/src/api/mod.rs

@@ -62,8 +62,6 @@ pub mod bucket {
         }
 
         pub mod bucket_lifecycle_ops {
-            pub use rustfs_lifecycle::{RestoreOutputLocation, RestoreRequest, RestoreS3Location, RestoreTag, RestoreTagging};
-
             pub use crate::bucket::lifecycle::bucket_lifecycle_ops::{
                 ExpiryState, LifecycleOps, ManualTransitionCancelCheck, ManualTransitionProgressSink,
                 ManualTransitionQueueSnapshot, ManualTransitionRunExecution, ManualTransitionRunOptions,
@@ -135,7 +133,7 @@ pub mod bucket {
             pub use crate::bucket::lifecycle::lifecycle::{
                 Event, ExpirationOptions, IlmAction, LIFECYCLE_MALFORMED_XML_ERROR_KIND, Lifecycle, LifecycleCalculate,
                 ObjectOpts, RuleValidate, TRANSITION_COMPLETE, TRANSITION_PENDING, TransitionOptions, expected_expiry_time,
-                format_lifecycle_date, object_opts_from_object_info, parse_lifecycle_date,
+                object_opts_from_object_info,
             };
         }
 
@@ -227,7 +225,7 @@ pub mod bucket {
     }
 
     pub mod object_lock {
-        pub use crate::bucket::object_lock::ObjectLockApi;
+        pub use crate::bucket::object_lock::{ObjectLockApi, ObjectLockStatusExt};
 
         pub mod types {
             pub use crate::bucket::object_lock::types::{
@@ -254,7 +252,7 @@ pub mod bucket {
     pub mod quota {
         pub use crate::bucket::quota::{
             BucketQuota, QUOTA_RESERVATION_PROTOCOL_V1, QUOTA_RESERVATION_PROTOCOL_V2, QuotaCheckResult, QuotaError,
-            QuotaMigrationSkipReason, QuotaOperation, QuotaProtocolMigration,
+            QuotaMigrationSkipReason, QuotaOperation, QuotaProtocolMigration, QuotaType,
         };
 
         pub mod checker {
@@ -322,9 +320,9 @@ pub mod bucket {
 
     pub mod utils {
         pub use crate::bucket::utils::{
-            BucketConfigXml, BucketConfigXmlError, check_bucket_and_object_names, check_list_objs_args,
-            check_object_name_for_length_and_slash, check_valid_bucket_name_strict, deserialize, has_bad_path_component,
-            is_meta_bucketname, is_valid_object_prefix, serialize,
+            check_bucket_and_object_names, check_list_objs_args, check_object_name_for_length_and_slash,
+            check_valid_bucket_name_strict, deserialize, has_bad_path_component, is_meta_bucketname, is_valid_object_prefix,
+            serialize,
         };
     }
 
@@ -343,15 +341,15 @@ pub mod cache {
 
 pub mod capacity {
     pub use crate::core::pools::{
-        DecommissionUnresolvedEntry, PoolDecommissionInfo, PoolMeta, PoolStatus, get_total_usable_capacity,
-        get_total_usable_capacity_free, is_pool_activation_fleet_proof_error, path2_bucket_object,
-        path2_bucket_object_with_base_path,
+        DecommissionUnresolvedEntry, PoolAdminDiagnostic, PoolAvailability, PoolCapacityStatus, PoolDecommissionInfo,
+        PoolDiagnosticStatus, PoolMeta, PoolStatus, get_total_usable_capacity, get_total_usable_capacity_free,
+        is_pool_activation_fleet_proof_error, path2_bucket_object, path2_bucket_object_with_base_path,
     };
     pub use crate::store::utils::is_reserved_or_invalid_bucket;
 }
 
 pub mod object_api_utils {
-    pub use crate::object_api::object_api_utils::{ObjReaderFn, PutObjReader, get_raw_etag, new_getobjectreader};
+    pub use crate::object_api::object_api_utils::{ObjReaderFn, PutObjReader, get_raw_etag, new_getobjectreader, to_s3s_etag};
 }
 
 pub mod cluster {
@@ -373,6 +371,15 @@ pub mod compression {
 
 pub mod config {
     pub mod com {
+        /// Preserve empty existing objects so callers can distinguish corruption from absence.
+        pub async fn read_config_preserve_empty_with_metadata_opts(
+            api: std::sync::Arc<crate::store::ECStore>,
+            file: &str,
+            opts: &crate::object_api::ObjectOptions,
+        ) -> crate::error::Result<(Vec<u8>, crate::object_api::ObjectInfo)> {
+            crate::config::com::read_config_preserve_empty_with_metadata_opts(api, file, opts).await
+        }
+
         pub use crate::config::com::{
             COMMA_SEPARATED_LISTS, CONFIG_PREFIX, ENV_CONFIG_RECOVER_ON_CORRUPTION, STORAGE_CLASS_SUB_SYS,
             ServerConfigCorruptError, ServerConfigSaveResult, ServerConfigSnapshot, delete_config, delete_config_no_lock,
@@ -418,9 +425,10 @@ pub mod data_usage {
         DATA_USAGE_CACHE_NAME, apply_bucket_usage_memory_overlay, compute_bucket_usage,
         init_compression_total_memory_from_backend, invalidate_admin_data_usage_snapshot_cache,
         invalidate_data_usage_snapshot_cache, live_bucket_usage_computations, load_admin_data_usage_from_backend_cached,
-        load_compression_total_from_memory, load_data_usage_from_backend, load_data_usage_from_backend_cached, quota_object_size,
-        record_bucket_delete_marker_memory, record_bucket_object_delete_memory, record_bucket_object_version_write_memory,
-        record_bucket_object_write_memory, record_bucket_object_write_unknown_previous_memory, record_compression_total_memory,
+        load_bucket_usage_from_backend_cached, load_compression_total_from_memory, load_data_usage_from_backend,
+        load_data_usage_from_backend_cached, quota_object_size, record_bucket_delete_marker_memory,
+        record_bucket_object_delete_memory, record_bucket_object_version_write_memory, record_bucket_object_write_memory,
+        record_bucket_object_write_unknown_previous_memory, record_compression_total_memory,
         refresh_bucket_usage_from_object_layer, refresh_versioned_bucket_usage_from_object_layer,
         remove_bucket_usage_from_backend, replace_bucket_usage_memory_from_info, store_compression_total_in_backend,
         store_data_usage_in_backend,

crates/ecstore/src/bucket/lifecycle/core.rs

@@ -17,7 +17,7 @@ use crate::object_api::ObjectInfo;
 pub use rustfs_lifecycle::{
     Event, ExpirationOptions, IlmAction, LIFECYCLE_MALFORMED_XML_ERROR_KIND, Lifecycle, LifecycleCalculate, ObjectOpts,
     RuleValidate, TRANSITION_COMPLETE, TRANSITION_PENDING, TransitionOptions, abort_incomplete_multipart_upload_due,
-    expected_expiry_time, expiration_action_has_valid_target, format_lifecycle_date, parse_lifecycle_date,
+    expected_expiry_time, expiration_action_has_valid_target,
 };
 
 pub fn object_opts_from_object_info(oi: &ObjectInfo) -> ObjectOpts {

crates/ecstore/src/bucket/lifecycle/metadata_boundary.rs

@@ -14,8 +14,7 @@
 
 use std::sync::Arc;
 
-use rustfs_gateway_types::persistence::PersistedLifecycleConfiguration;
-use rustfs_gateway_types::persistence::PersistedObjectLockConfiguration as ObjectLockConfiguration;
+use s3s::dto::{BucketLifecycleConfiguration, ObjectLockConfiguration};
 use time::OffsetDateTime;
 use uuid::Uuid;
 
@@ -25,7 +24,7 @@ use crate::error::{Error, Result};
 
 #[derive(Debug)]
 pub(crate) struct LifecycleExpiryConfigs {
-    pub(crate) lifecycle: Option<Arc<PersistedLifecycleConfiguration>>,
+    pub(crate) lifecycle: Option<Arc<BucketLifecycleConfiguration>>,
     pub(crate) object_lock: Option<Arc<ObjectLockConfiguration>>,
     pub(crate) bucket_incarnation_id: Uuid,
     pub(crate) table_bucket_enabled: bool,
@@ -64,14 +63,7 @@ pub(crate) async fn get_expiry_configs_for_options(
     bucket: &str,
     opts: &crate::object_api::ObjectOptions,
 ) -> Result<LifecycleExpiryConfigs> {
-    let bucket_incarnation_id = match opts
-        .object_lock_config_snapshot
-        .as_ref()
-        .and_then(|snapshot| snapshot.validated_bucket_incarnation_for(api.id, bucket, opts.expected_bucket_incarnation_id))
-    {
-        Some(incarnation_id) => incarnation_id,
-        None => metadata_sys::get_bucket_incarnation_id_for_options_in(&api.ctx, bucket, opts).await?,
-    };
+    let bucket_incarnation_id = metadata_sys::get_bucket_incarnation_id_for_options_in(&api.ctx, bucket, opts).await?;
     let metadata = get_authoritative_metadata(api, bucket, bucket_incarnation_id).await?;
     let table_bucket_enabled = metadata.table_bucket_enabled();
 
@@ -108,7 +100,7 @@ pub(crate) async fn get_expiry_configs_for_options(
     })
 }
 
-pub(crate) async fn get_lifecycle_config(bucket: &str) -> Result<(PersistedLifecycleConfiguration, OffsetDateTime)> {
+pub(crate) async fn get_lifecycle_config(bucket: &str) -> Result<(BucketLifecycleConfiguration, OffsetDateTime)> {
     metadata_sys::get_lifecycle_config(bucket).await
 }
 
@@ -118,15 +110,15 @@ mod tests {
     use crate::bucket::metadata::BucketMetadata;
     use crate::bucket::metadata_sys::{self, test_support::isolated_store_over_temp_disks};
     use crate::storage_api_contracts::bucket::MakeBucketOptions;
-    use rustfs_gateway_types::persistence::{PersistedLifecycleExpiration, PersistedLifecycleRule};
+    use s3s::dto::{ExpirationStatus, LifecycleExpiration, LifecycleRule};
     use serial_test::serial;
 
-    fn lifecycle_config() -> PersistedLifecycleConfiguration {
-        PersistedLifecycleConfiguration {
+    fn lifecycle_config() -> BucketLifecycleConfiguration {
+        BucketLifecycleConfiguration {
             expiry_updated_at: None,
-            rules: vec![PersistedLifecycleRule {
-                status: "Enabled".to_owned(),
-                expiration: Some(PersistedLifecycleExpiration {
+            rules: vec![LifecycleRule {
+                status: ExpirationStatus::from_static(ExpirationStatus::ENABLED),
+                expiration: Some(LifecycleExpiration {
                     days: Some(1),
                     ..Default::default()
                 }),

crates/ecstore/src/bucket/lifecycle/object_lock_boundary.rs

@@ -22,7 +22,7 @@ pub(crate) fn is_object_locked_by_metadata(user_defined: &HashMap<String, String
 }
 
 pub(crate) fn check_object_lock_for_deletion_with_config(
-    config: Option<&rustfs_gateway_types::persistence::PersistedObjectLockConfiguration>,
+    config: Option<&s3s::dto::ObjectLockConfiguration>,
     obj_info: &ObjectInfo,
     bypass_governance: bool,
 ) -> crate::error::Result<Option<ObjectLockBlockReason>> {

crates/ecstore/src/bucket/lifecycle/replication_sink.rs

@@ -82,13 +82,12 @@ mod tests {
     use crate::bucket::replication::{DeleteReplicationConfigSnapshot, ReplicationObjectBridge};
     use crate::object_api::{ObjectInfo, ObjectOptions};
     use crate::storage_api_contracts::object::ObjectToDelete;
-    use rustfs_gateway_types::dto::Status;
-    use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
-    use rustfs_gateway_types::persistence::{
-        PersistedOptionalReplicationStatus, PersistedReplicationConfiguration, PersistedReplicationDestination,
-        PersistedReplicationRule, PersistedReplicationStatus,
-    };
     use rustfs_scanner_metrics::metrics::IlmAction;
+    use s3s::dto::{
+        BucketVersioningStatus, DeleteMarkerReplication, DeleteMarkerReplicationStatus, DeleteReplication,
+        DeleteReplicationStatus, Destination, ReplicationConfiguration, ReplicationRule, ReplicationRuleStatus,
+        VersioningConfiguration,
+    };
     use uuid::Uuid;
 
     use super::*;
@@ -161,27 +160,27 @@ mod tests {
             for purge_enabled in [false, true] {
                 let snapshot = DeleteReplicationConfigSnapshot::from_configs_for_test(
                     VersioningConfiguration {
-                        status: Some("Enabled".to_owned()),
+                        status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
                         ..Default::default()
                     },
-                    Some(PersistedReplicationConfiguration {
+                    Some(ReplicationConfiguration {
                         role: String::new(),
-                        rules: vec![PersistedReplicationRule {
-                            delete_marker_replication: Some(PersistedOptionalReplicationStatus {
+                        rules: vec![ReplicationRule {
+                            delete_marker_replication: Some(DeleteMarkerReplication {
                                 status: Some(if marker_enabled {
-                                    Status::ENABLED.to_string()
+                                    DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)
                                 } else {
-                                    Status::DISABLED.to_string()
+                                    DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::DISABLED)
                                 }),
                             }),
-                            delete_replication: Some(PersistedReplicationStatus {
+                            delete_replication: Some(DeleteReplication {
                                 status: if purge_enabled {
-                                    Status::ENABLED.to_string()
+                                    DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED)
                                 } else {
-                                    Status::DISABLED.to_string()
+                                    DeleteReplicationStatus::from_static(DeleteReplicationStatus::DISABLED)
                                 },
                             }),
-                            destination: PersistedReplicationDestination {
+                            destination: Destination {
                                 bucket: "arn:rustfs:replication:target".to_string(),
                                 ..Default::default()
                             },
@@ -191,7 +190,7 @@ mod tests {
                             prefix: Some(String::new()),
                             priority: Some(1),
                             source_selection_criteria: None,
-                            status: Status::ENABLED.to_string(),
+                            status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
                         }],
                     }),
                 );

crates/ecstore/src/bucket/lifecycle/runtime_boundary.rs

@@ -14,7 +14,7 @@
 
 use std::sync::Arc;
 
-use rustfs_gateway_types::persistence::PersistedLifecycleConfiguration;
+use s3s::dto::BucketLifecycleConfiguration;
 use tokio::sync::RwLock;
 use tokio_util::sync::CancellationToken;
 
@@ -52,7 +52,7 @@ pub(crate) fn deployment_id() -> Option<String> {
     sources::deployment_id()
 }
 
-pub(crate) async fn bucket_lifecycle_config(bucket: &str) -> Option<PersistedLifecycleConfiguration> {
+pub(crate) async fn bucket_lifecycle_config(bucket: &str) -> Option<BucketLifecycleConfiguration> {
     sources::bucket_lifecycle_config(bucket).await
 }
 

crates/ecstore/src/bucket/metadata.rs

@@ -13,7 +13,7 @@
 // limitations under the License.
 
 use super::msgp_decode::{read_msgp_ext8_time, skip_msgp_value, write_msgp_time};
-use super::object_lock::ObjectLockApi;
+use super::object_lock::{ObjectLockApi, ObjectLockStatusExt};
 use super::versioning::VersioningApi;
 use super::{quota::BucketQuota, target::BucketTargets};
 use crate::bucket::replication::invalid_replication_config_status_field;
@@ -25,17 +25,12 @@ use crate::object_api::WriteCompletion;
 use crate::runtime::sources as runtime_sources;
 use crate::store::ECStore;
 use byteorder::{BigEndian, ByteOrder, LittleEndian};
-use rustfs_gateway_types::cors_tagging::{PersistedCorsConfiguration as CORSConfiguration, PersistedTagging as Tagging};
-use rustfs_gateway_types::persistence::{
-    PersistedAccelerateConfiguration as AccelerateConfiguration,
-    PersistedBucketEncryptionConfiguration as ServerSideEncryptionConfiguration,
-    PersistedBucketLoggingStatus as BucketLoggingStatus, PersistedLifecycleConfiguration,
-    PersistedNotificationConfiguration as NotificationConfiguration, PersistedObjectLockConfiguration as ObjectLockConfiguration,
-    PersistedPublicAccessBlockConfiguration as PublicAccessBlockConfiguration, PersistedReplicationConfiguration,
-    PersistedRequestPaymentConfiguration as RequestPaymentConfiguration,
-    PersistedVersioningConfiguration as VersioningConfiguration, PersistedWebsiteConfiguration as WebsiteConfiguration,
-};
 use rustfs_policy::policy::BucketPolicy;
+use s3s::dto::{
+    AccelerateConfiguration, BucketLifecycleConfiguration, BucketLoggingStatus, BucketVersioningStatus, CORSConfiguration,
+    NotificationConfiguration, ObjectLockConfiguration, PublicAccessBlockConfiguration, ReplicationConfiguration,
+    RequestPaymentConfiguration, ServerSideEncryptionConfiguration, Tagging, VersioningConfiguration, WebsiteConfiguration,
+};
 use serde::Serializer;
 use sha2::{Digest, Sha256};
 use std::collections::HashMap;
@@ -45,11 +40,23 @@ use time::{Date, OffsetDateTime, PrimitiveDateTime, Time as CivilTime, UtcOffset
 use tracing::error;
 use uuid::Uuid;
 
-// The persisted Object Lock configuration answers the storage-level trait
-// here because this module owns the persisted `ObjectLockConfiguration`.
+// The serving-layer DTO impls for the storage-level Object Lock traits live
+// here because this module owns the persisted `ObjectLockConfiguration`
+// during the s3s ratchet migration (rustfs/backlog#1842).
 impl ObjectLockApi for ObjectLockConfiguration {
     fn enabled(&self) -> bool {
-        self.object_lock_enabled.as_deref() == Some("Enabled")
+        self.object_lock_enabled
+            .as_ref()
+            .is_some_and(|v| v.as_str() == s3s::dto::ObjectLockEnabled::ENABLED)
+    }
+}
+
+impl ObjectLockStatusExt for s3s::dto::ObjectLockLegalHoldStatus {
+    fn valid(&self) -> bool {
+        matches!(
+            self.as_str(),
+            s3s::dto::ObjectLockLegalHoldStatus::ON | s3s::dto::ObjectLockLegalHoldStatus::OFF
+        )
     }
 }
 
@@ -432,8 +439,6 @@ pub struct BucketMetadata {
     pub bucket_incarnation_id: Uuid,
     pub(crate) bucket_incarnation_sidecar: bool,
     pub(crate) bucket_creation_committed: bool,
-    // A legacy default is not proof of an interrupted creation.
-    pub(crate) bucket_creation_commit_record_present: bool,
     pub policy_config_json: Vec<u8>,
     pub notification_config_xml: Vec<u8>,
     pub lifecycle_config_xml: Vec<u8>,
@@ -482,13 +487,13 @@ pub struct BucketMetadata {
 
     pub policy_config: Option<BucketPolicy>,
     pub notification_config: Option<NotificationConfiguration>,
-    pub lifecycle_config: Option<PersistedLifecycleConfiguration>,
+    pub lifecycle_config: Option<BucketLifecycleConfiguration>,
     pub object_lock_config: Option<ObjectLockConfiguration>,
     pub versioning_config: Option<VersioningConfiguration>,
     pub sse_config: Option<ServerSideEncryptionConfiguration>,
     pub tagging_config: Option<Tagging>,
     pub quota_config: Option<BucketQuota>,
-    pub replication_config: Option<PersistedReplicationConfiguration>,
+    pub replication_config: Option<ReplicationConfiguration>,
     pub bucket_target_config: Option<BucketTargets>,
     pub bucket_target_config_meta: Option<HashMap<String, String>>,
     pub cors_config: Option<CORSConfiguration>,
@@ -509,7 +514,6 @@ impl Default for BucketMetadata {
             bucket_incarnation_id: Uuid::nil(),
             bucket_incarnation_sidecar: false,
             bucket_creation_committed: false,
-            bucket_creation_commit_record_present: false,
             policy_config_json: Default::default(),
             notification_config_xml: Default::default(),
             lifecycle_config_xml: Default::default(),
@@ -748,10 +752,7 @@ impl BucketMetadata {
                 "TableBucketConfigUpdatedAt" => self.table_bucket_config_updated_at = read_msgp_time_value(rd)?,
                 "DurabilityConfigUpdatedAt" => self.durability_config_updated_at = read_msgp_time_value(rd)?,
                 "OnDemandMigrationConfigUpdatedAt" => self.on_demand_migration_config_updated_at = read_msgp_time_value(rd)?,
-                "BucketCreationCommitted" => {
-                    self.bucket_creation_committed = read_msgp_bool(rd)?;
-                    self.bucket_creation_commit_record_present = true;
-                }
+                "BucketCreationCommitted" => self.bucket_creation_committed = read_msgp_bool(rd)?,
                 other => {
                     tracing::debug!(field = %other, "BucketMetadata decode_from: skipping unknown field");
                     skip_msgp_value(rd)?;
@@ -764,10 +765,8 @@ impl BucketMetadata {
 
     /// Encode to msgp bytes. Field order follows MinIO BucketMetadata for compatibility.
     pub fn encode_to<W: Write>(&self, wr: &mut W) -> Result<()> {
-        // Preserve the absence of a legacy commit record across config rewrites.
-        let commit_record_present = self.bucket_creation_commit_record_present || self.bucket_creation_committed;
-        // Map size: MinIO fields (25) + RustFS extensions (21) + optional commit record.
-        let map_len: u32 = 46 + u32::from(commit_record_present);
+        // Map size: MinIO fields (25) + RustFS extensions (22)
+        let map_len: u32 = 47;
         rmp::encode::write_map_len(wr, map_len)?;
 
         // MinIO field order (same as Go struct)
@@ -848,10 +847,8 @@ impl BucketMetadata {
         write_msgp_time(wr, self.durability_config_updated_at)?;
         rmp::encode::write_str(wr, "OnDemandMigrationConfigUpdatedAt")?;
         write_msgp_time(wr, self.on_demand_migration_config_updated_at)?;
-        if commit_record_present {
-            rmp::encode::write_str(wr, "BucketCreationCommitted")?;
-            rmp::encode::write_bool(wr, self.bucket_creation_committed)?;
-        }
+        rmp::encode::write_str(wr, "BucketCreationCommitted")?;
+        rmp::encode::write_bool(wr, self.bucket_creation_committed)?;
 
         Ok(())
     }
@@ -1021,11 +1018,9 @@ impl BucketMetadata {
                     None
                 } else {
                     let config = deserialize::<VersioningConfiguration>(&data)?;
-                    if config
-                        .status
-                        .as_ref()
-                        .is_some_and(|status| !matches!(status.as_str(), "Enabled" | "Suspended"))
-                    {
+                    if config.status.as_ref().is_some_and(|status| {
+                        !matches!(status.as_str(), BucketVersioningStatus::ENABLED | BucketVersioningStatus::SUSPENDED)
+                    }) {
                         return Err(Error::other("bucket versioning configuration has an invalid status"));
                     }
                     Some(config)
@@ -1038,7 +1033,7 @@ impl BucketMetadata {
                 let config = if data.is_empty() {
                     None
                 } else {
-                    let config = deserialize::<PersistedReplicationConfiguration>(&data)?;
+                    let config = deserialize::<ReplicationConfiguration>(&data)?;
                     if let Some(field) = invalid_replication_config_status_field(&config) {
                         return Err(Error::other(format!("replication field {field} has an invalid status")));
                     }
@@ -1232,8 +1227,8 @@ impl BucketMetadata {
         }
         self.lifecycle_config = None;
         if !self.lifecycle_config_xml.is_empty()
-            && let Err(e) = deserialize::<PersistedLifecycleConfiguration>(&self.lifecycle_config_xml)
-                .map(|c| self.lifecycle_config = Some(c))
+            && let Err(e) =
+                deserialize::<BucketLifecycleConfiguration>(&self.lifecycle_config_xml).map(|c| self.lifecycle_config = Some(c))
         {
             tracing::warn!(
                 event = "bucket_metadata_parse_failed",
@@ -1319,8 +1314,8 @@ impl BucketMetadata {
         }
         self.replication_config = None;
         if !self.replication_config_xml.is_empty()
-            && let Err(e) = deserialize::<PersistedReplicationConfiguration>(&self.replication_config_xml)
-                .map(|c| self.replication_config = Some(c))
+            && let Err(e) =
+                deserialize::<ReplicationConfiguration>(&self.replication_config_xml).map(|c| self.replication_config = Some(c))
         {
             tracing::warn!(
                 event = "bucket_metadata_parse_failed",
@@ -1696,13 +1691,6 @@ mod test {
         let mut bm = BucketMetadata::unmarshal(&body[4..]).expect("unmarshal recovered blob");
         assert_eq!(bm.name, "interop");
         assert!(!bm.bucket_creation_committed, "legacy metadata must default to uncommitted");
-        assert!(!bm.bucket_creation_commit_record_present, "legacy metadata has no creation-commit record");
-        let rewritten = BucketMetadata::unmarshal(&bm.marshal_msg().expect("rewrite legacy metadata"))
-            .expect("read rewritten legacy metadata");
-        assert!(
-            !rewritten.bucket_creation_commit_record_present,
-            "rewriting legacy metadata must not fabricate a creation intent"
-        );
         bm.parse_all_configs().expect("parse recovered configs");
         assert!(bm.lifecycle_config.is_some());
     }
@@ -1720,7 +1708,6 @@ mod test {
 
         assert_eq!(bm.name, new.name);
         assert!(new.bucket_creation_committed);
-        assert!(new.bucket_creation_commit_record_present);
         assert!(!bm.bucket_incarnation_id.is_nil());
         assert_eq!(bm.bucket_incarnation_id, new.bucket_incarnation_id);
     }
@@ -1825,15 +1812,17 @@ mod test {
 
     #[test]
     fn object_locking_requires_lock_metadata_not_plain_versioning() {
+        use s3s::dto::ObjectLockEnabled;
+
         let mut bm = BucketMetadata::new("test-bucket");
         bm.versioning_config = Some(VersioningConfiguration {
-            status: Some("Enabled".to_owned()),
+            status: Some(s3s::dto::BucketVersioningStatus::from_static("Enabled")),
             ..Default::default()
         });
         assert!(!bm.object_locking());
 
         bm.object_lock_config = Some(ObjectLockConfiguration {
-            object_lock_enabled: Some("Enabled".to_owned()),
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
             ..Default::default()
         });
         assert!(bm.object_locking());

crates/ecstore/src/bucket/metadata_sys.rs

@@ -30,18 +30,14 @@ use crate::storage_api_contracts::heal::HealOperations as _;
 use crate::storage_api_contracts::namespace::NamespaceLocking as _;
 use crate::store::{ECStore, await_bucket_namespace_operation};
 use futures::future::join_all;
-use rustfs_gateway_types::cors_tagging::{PersistedCorsConfiguration as CORSConfiguration, PersistedTagging as Tagging};
-use rustfs_gateway_types::persistence::{
-    PersistedAccelerateConfiguration as AccelerateConfiguration,
-    PersistedBucketEncryptionConfiguration as ServerSideEncryptionConfiguration,
-    PersistedBucketLoggingStatus as BucketLoggingStatus, PersistedLifecycleConfiguration,
-    PersistedNotificationConfiguration as NotificationConfiguration, PersistedObjectLockConfiguration as ObjectLockConfiguration,
-    PersistedPublicAccessBlockConfiguration as PublicAccessBlockConfiguration, PersistedReplicationConfiguration,
-    PersistedRequestPaymentConfiguration as RequestPaymentConfiguration,
-    PersistedVersioningConfiguration as VersioningConfiguration, PersistedWebsiteConfiguration as WebsiteConfiguration,
-};
 use rustfs_heal_contracts::heal_channel::HealOpts;
 use rustfs_policy::policy::BucketPolicy;
+use s3s::dto::ReplicationConfiguration;
+use s3s::dto::{
+    AccelerateConfiguration, BucketLifecycleConfiguration, BucketLoggingStatus, CORSConfiguration, NotificationConfiguration,
+    ObjectLockConfiguration, ObjectLockEnabled, ObjectLockRetentionMode, PublicAccessBlockConfiguration,
+    RequestPaymentConfiguration, ServerSideEncryptionConfiguration, Tagging, VersioningConfiguration, WebsiteConfiguration,
+};
 use std::collections::HashSet;
 use std::time::Duration;
 use std::{
@@ -172,7 +168,7 @@ pub(crate) fn object_lock_config_state_from_authoritative_metadata(bm: &BucketMe
     if bm.lock_enabled {
         return Ok(ObjectLockConfigState::Configured {
             config: ObjectLockConfiguration {
-                object_lock_enabled: Some("Enabled".to_owned()),
+                object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
                 rule: None,
             },
             updated_at: bm.object_lock_config_updated_at,
@@ -216,10 +212,17 @@ pub(crate) fn configured_object_lock_state_for_tests(
 ) -> ObjectLockConfigState {
     ObjectLockConfigState::Configured {
         config: ObjectLockConfiguration {
-            object_lock_enabled: Some("Enabled".to_owned()),
-            rule: Some(rustfs_gateway_types::persistence::PersistedObjectLockRule {
-                default_retention: Some(rustfs_gateway_types::persistence::PersistedDefaultRetention {
-                    mode: Some(mode.as_str().to_owned()),
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
+            rule: Some(s3s::dto::ObjectLockRule {
+                default_retention: Some(s3s::dto::DefaultRetention {
+                    mode: Some(s3s::dto::ObjectLockRetentionMode::from_static(match mode {
+                        crate::bucket::object_lock::types::RetentionMode::Governance => {
+                            s3s::dto::ObjectLockRetentionMode::GOVERNANCE
+                        }
+                        crate::bucket::object_lock::types::RetentionMode::Compliance => {
+                            s3s::dto::ObjectLockRetentionMode::COMPLIANCE
+                        }
+                    })),
                     days: Some(days),
                     years: None,
                 }),
@@ -230,7 +233,7 @@ pub(crate) fn configured_object_lock_state_for_tests(
 }
 
 fn validate_authoritative_object_lock_config(config: &ObjectLockConfiguration) -> Result<()> {
-    if config.object_lock_enabled.as_deref() != Some("Enabled") {
+    if config.object_lock_enabled.as_ref().map(ObjectLockEnabled::as_str) != Some(ObjectLockEnabled::ENABLED) {
         return Err(Error::other("persisted bucket Object Lock enabled state is invalid"));
     }
 
@@ -243,7 +246,7 @@ fn validate_authoritative_object_lock_config(config: &ObjectLockConfiguration) -
     if !retention
         .mode
         .as_ref()
-        .is_some_and(|mode| matches!(mode.as_str(), "COMPLIANCE" | "GOVERNANCE"))
+        .is_some_and(|mode| matches!(mode.as_str(), ObjectLockRetentionMode::COMPLIANCE | ObjectLockRetentionMode::GOVERNANCE))
     {
         return Err(Error::other("persisted bucket Object Lock retention mode is invalid"));
     }
@@ -561,12 +564,39 @@ pub(crate) async fn validate_bucket_reconstruction_in(
     Ok(())
 }
 
-/// Validate against disk while retaining an already-held Object Lock fence.
+/// Resolve the authoritative bucket incarnation id for `opts`:
+/// 1. When namespace locking is enabled and `opts.object_lock_config_snapshot` is
+///    present, validate and return the snapshot's incarnation under its held
+///    lifecycle and metadata-transaction fences.
+/// 2. When namespace locking is enabled and `opts.bucket_lifecycle_lock_fence` is
+///    present and not lost, reuse the store's `BucketFenceRegistry` memo if a
+///    registered lifecycle read guard is currently held without lock loss.
+/// 3. Otherwise, acquire the bucket metadata transaction read lock (unless
+///    already held on `opts`) and read the incarnation from disk.
 pub(crate) async fn get_bucket_incarnation_id_for_options_in(
     ctx: &crate::runtime::instance::InstanceContext,
     bucket: &str,
     opts: &ObjectOptions,
 ) -> Result<Uuid> {
+    if !ctx.lock_manager().is_disabled() {
+        if let Some(snapshot) = opts.object_lock_config_snapshot.as_ref() {
+            let store = object_store_in(ctx).await?;
+            return snapshot
+                .validated_bucket_incarnation_for(store.id, bucket, opts.expected_bucket_incarnation_id)
+                .ok_or_else(|| {
+                    Error::other("Object Lock snapshot does not hold a valid metadata transaction fence for this bucket")
+                });
+        }
+        if opts
+            .bucket_lifecycle_lock_fence
+            .as_ref()
+            .is_some_and(|fence| !fence.is_lock_lost())
+            && let Some(store) = object_store_if_initialized_in(ctx).await
+            && let Some(incarnation) = store.bucket_fence_registry.validated_incarnation(bucket)
+        {
+            return Ok(incarnation);
+        }
+    }
     let sys = bucket_metadata_sys_of(ctx)?;
     let sys = sys.read().await.clone();
     let guard = acquire_bucket_metadata_transaction_read_lock_for_options_in(ctx, bucket, opts).await?;
@@ -1365,7 +1395,7 @@ pub async fn get_public_access_block_config(bucket: &str) -> Result<(PublicAcces
     bucket_meta_sys.get_public_access_block_config(bucket).await
 }
 
-pub async fn get_lifecycle_config(bucket: &str) -> Result<(PersistedLifecycleConfiguration, OffsetDateTime)> {
+pub async fn get_lifecycle_config(bucket: &str) -> Result<(BucketLifecycleConfiguration, OffsetDateTime)> {
     let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
     let bucket_meta_sys = bucket_meta_sys_lock.read().await;
 
@@ -1584,7 +1614,7 @@ pub(crate) async fn get_cached_quota_config_and_incarnation_in(
     bucket_meta_sys.cached_quota_config_and_incarnation(bucket).await
 }
 
-pub async fn get_replication_config(bucket: &str) -> Result<(PersistedReplicationConfiguration, OffsetDateTime)> {
+pub async fn get_replication_config(bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
     let bucket_meta_sys_lock = get_bucket_metadata_sys()?;
     let bucket_meta_sys = bucket_meta_sys_lock.read().await;
 
@@ -1594,7 +1624,7 @@ pub async fn get_replication_config(bucket: &str) -> Result<(PersistedReplicatio
 pub(crate) async fn get_replication_config_in(
     ctx: &crate::runtime::instance::InstanceContext,
     bucket: &str,
-) -> Result<(PersistedReplicationConfiguration, OffsetDateTime)> {
+) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
     let bucket_meta_sys_lock = bucket_metadata_sys_of(ctx)?;
     let bucket_meta_sys = bucket_meta_sys_lock.read().await;
 
@@ -1720,27 +1750,9 @@ struct MetadataPublishGuard {
     _guard: tokio::sync::OwnedMutexGuard<MetadataPublishLockState>,
 }
 
-#[derive(Debug, Clone)]
-pub(crate) struct PendingBucketCreationProof {
-    pub(crate) generation: Uuid,
-    pub(crate) intent: Vec<u8>,
-    pub(crate) versioning_requested: bool,
-    pub(crate) witnesses: Vec<crate::disk::os::NativeBucketCreationWitness>,
-}
-
-impl PendingBucketCreationProof {
-    pub(crate) fn matches_metadata(&self, metadata: &BucketMetadata) -> Result<bool> {
-        Ok(!metadata.bucket_creation_committed
-            && metadata.bucket_incarnation_sidecar
-            && metadata.bucket_incarnation_id == self.generation
-            && metadata.marshal_msg()? == self.intent)
-    }
-}
-
 #[derive(Debug, Clone)]
 pub struct BucketMetadataSys {
     metadata_map: Arc<RwLock<HashMap<String, Arc<BucketMetadata>>>>,
-    pending_bucket_creations: Arc<RwLock<HashMap<String, PendingBucketCreationProof>>>,
     /// Serializes metadata-map commits and their derived cache updates for one
     /// bucket. Namespace locks, when present, are acquired before this lock.
     metadata_publish_locks: Arc<MetadataPublishLockRegistry>,
@@ -1784,7 +1796,6 @@ impl BucketMetadataSys {
     pub fn new(api: Arc<ECStore>) -> Self {
         Self {
             metadata_map: Arc::new(RwLock::new(HashMap::new())),
-            pending_bucket_creations: Arc::new(RwLock::new(HashMap::new())),
             metadata_publish_locks: Arc::new(MetadataPublishLockRegistry {
                 locks: StdMutex::new(HashMap::new()),
             }),
@@ -1815,81 +1826,6 @@ impl BucketMetadataSys {
             .expect("bucket metadata object store should still be live")
     }
 
-    pub(crate) async fn pending_bucket_creation(&self, bucket: &str) -> Option<PendingBucketCreationProof> {
-        self.pending_bucket_creations.read().await.get(bucket).cloned()
-    }
-
-    pub(crate) async fn grant_pending_bucket_creation(&self, bucket: &str, proof: PendingBucketCreationProof) {
-        self.pending_bucket_creations.write().await.insert(bucket.to_owned(), proof);
-    }
-
-    pub(crate) async fn revoke_pending_bucket_creation(&self, bucket: &str) {
-        self.pending_bucket_creations.write().await.remove(bucket);
-    }
-
-    pub(crate) async fn consume_pending_bucket_creation(
-        &self,
-        bucket: &str,
-        metadata: &BucketMetadata,
-        witnesses: &[crate::disk::os::NativeBucketCreationWitness],
-    ) -> Result<bool> {
-        let mut pending = self.pending_bucket_creations.write().await;
-        let Some(proof) = pending.get(bucket) else {
-            return Ok(false);
-        };
-        if !proof.matches_metadata(metadata)?
-            || proof.witnesses.len() != witnesses.len()
-            || !proof
-                .witnesses
-                .iter()
-                .zip(witnesses)
-                .all(|(old, current)| old.matches(current))
-            || !crate::disk::os::consume_native_creation_witnesses(witnesses)
-        {
-            return Ok(false);
-        }
-        pending.remove(bucket);
-        Ok(true)
-    }
-
-    #[cfg(test)]
-    pub(crate) async fn refresh_bucket_for_test(&self, bucket: &str) -> Result<()> {
-        let expected = self.metadata_map.read().await.get(bucket).cloned();
-        let lock = self.object_store().new_ns_lock(bucket, bucket).await?;
-        let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
-        self.load_bucket_under_namespace(bucket, MetadataLoadMode::Refresh, expected.as_ref(), &guard)
-            .await
-    }
-
-    #[cfg(test)]
-    pub(crate) async fn publish_persisted_metadata_for_test(&self, bucket: &str) -> Result<()> {
-        let expected = self.metadata_map.read().await.get(bucket).cloned();
-        let lock = self.object_store().new_ns_lock(bucket, bucket).await?;
-        let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
-        let (metadata, persisted) = await_bucket_namespace_operation(
-            Some(&guard),
-            bucket,
-            "test cached metadata load",
-            load_bucket_metadata_parse_with_presence(self.object_store(), bucket, true),
-        )
-        .await?;
-        self.publish_if_unchanged(bucket, expected.as_ref(), metadata, persisted, &guard)
-            .await
-    }
-
-    #[cfg(test)]
-    pub(crate) async fn migrate_bucket_creation_commit_for_test(&self, bucket: &str) -> Result<()> {
-        self.migrate_bucket_creation_commit(bucket).await.map(|_| ())
-    }
-
-    #[cfg(test)]
-    pub(crate) async fn legacy_migration_reports_missing_bucket_for_test(&self, bucket: &str) -> Result<bool> {
-        Ok(matches!(
-            self.migrate_legacy_metadata(bucket).await?,
-            BucketMetadataAuthority::MissingBucket
-        ))
-    }
-
     fn object_store_if_live(&self) -> Option<Arc<ECStore>> {
         self.api.upgrade()
     }
@@ -2080,7 +2016,6 @@ impl BucketMetadataSys {
         expected: Option<&Arc<BucketMetadata>>,
         namespace_guard: &rustfs_lock::NamespaceLockGuard,
     ) -> Result<()> {
-        self.revoke_pending_bucket_creation(bucket).await;
         if !await_bucket_namespace_operation(
             Some(namespace_guard),
             bucket,
@@ -2164,7 +2099,6 @@ impl BucketMetadataSys {
         persisted: bool,
         namespace_guard: &rustfs_lock::NamespaceLockGuard,
     ) -> Result<()> {
-        self.revoke_pending_bucket_creation(bucket).await;
         if !persisted {
             let _publish_guard = self
                 .lock_metadata_publish(bucket, namespace_guard, "refreshed bucket metadata absence publish")
@@ -2226,7 +2160,6 @@ impl BucketMetadataSys {
     }
 
     pub async fn set(&self, bucket: String, bm: Arc<BucketMetadata>) {
-        self.revoke_pending_bucket_creation(&bucket).await;
         if !is_meta_bucketname(&bucket) {
             let publish_lock = self.metadata_publish_lock(&bucket);
             let _publish_guard = publish_lock.lock().await;
@@ -2248,7 +2181,6 @@ impl BucketMetadataSys {
         if is_meta_bucketname(bucket) {
             return false;
         }
-        self.revoke_pending_bucket_creation(bucket).await;
         let publish_lock = self.metadata_publish_lock(bucket);
         let _publish_guard = publish_lock.lock().await;
         let mut map = self.metadata_map.write().await;
@@ -2266,7 +2198,6 @@ impl BucketMetadataSys {
     }
 
     async fn _reset(&mut self) {
-        self.pending_bucket_creations.write().await.clear();
         let mut map = self.metadata_map.write().await;
         map.clear();
         drop(map);
@@ -2405,7 +2336,6 @@ impl BucketMetadataSys {
     /// server's metadata never leaks into the ambient (first) instance.
     pub(crate) async fn persist_and_set(&self, bm: BucketMetadata) -> Result<()> {
         let mut bm = bm;
-        self.revoke_pending_bucket_creation(&bm.name).await;
         bm.save_with_store(self.object_store()).await?;
 
         self.set(bm.name.clone(), Arc::new(bm)).await;
@@ -2414,7 +2344,6 @@ impl BucketMetadataSys {
     }
 
     async fn persist_new_and_set(&self, mut bm: BucketMetadata) -> Result<()> {
-        self.revoke_pending_bucket_creation(&bm.name).await;
         bm.bucket_creation_committed = true;
         bm.save_with_store_committed(self.object_store()).await?;
         save_bucket_incarnation(self.object_store(), &bm.name, bm.bucket_incarnation_id).await?;
@@ -2424,9 +2353,7 @@ impl BucketMetadataSys {
     }
 
     async fn persist_new_bucket_metadata_intent(&self, mut bm: BucketMetadata) -> Result<()> {
-        self.revoke_pending_bucket_creation(&bm.name).await;
         bm.bucket_creation_committed = false;
-        bm.bucket_creation_commit_record_present = true;
         bm.save_with_store(self.object_store()).await?;
         save_bucket_incarnation(self.object_store(), &bm.name, bm.bucket_incarnation_id).await?;
         bm.bucket_incarnation_sidecar = true;
@@ -2436,7 +2363,6 @@ impl BucketMetadataSys {
     }
 
     async fn commit_bucket_metadata(&self, mut bm: BucketMetadata) -> Result<()> {
-        self.revoke_pending_bucket_creation(&bm.name).await;
         bm.parse_all_configs()?;
         bm.bucket_incarnation_sidecar = true;
         // The caller holds the full bucket creation fence (lifecycle, metadata
@@ -2485,7 +2411,6 @@ impl BucketMetadataSys {
         bucket: &str,
         namespace_guard: &rustfs_lock::NamespaceLockGuard,
     ) -> Result<()> {
-        self.revoke_pending_bucket_creation(bucket).await;
         let expected = self.metadata_map.read().await.get(bucket).cloned();
         if !self
             .bucket_exists(bucket, namespace_guard, "peer bucket metadata existence check")
@@ -2542,7 +2467,6 @@ impl BucketMetadataSys {
             #[cfg(test)]
             self.lazy_disk_loads.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
 
-            self.revoke_pending_bucket_creation(bucket).await;
             let lock = self.object_store().new_ns_lock(bucket, bucket).await?;
             let guard = lock.get_read_lock(crate::set_disk::get_lock_acquire_timeout()).await?;
             #[cfg(test)]
@@ -2868,7 +2792,6 @@ impl BucketMetadataSys {
     /// revalidated at write quorum, so a stale snapshot can neither revert an
     /// acknowledged configuration update nor outlive a delete/recreate.
     async fn migrate_bucket_creation_commit(&self, bucket: &str) -> Result<BucketMetadataAuthority> {
-        self.revoke_pending_bucket_creation(bucket).await;
         let transaction_lock = self
             .object_store()
             .new_ns_lock(RUSTFS_META_BUCKET, &bucket_metadata_transaction_lock_key(bucket))
@@ -2946,7 +2869,6 @@ impl BucketMetadataSys {
         &self,
         bucket: &str,
     ) -> Result<BucketMetadataAuthority> {
-        self.revoke_pending_bucket_creation(bucket).await;
         #[cfg(test)]
         if self.object_lock_disk_read_errors.write().await.remove(bucket) {
             return Err(Error::other(format!("injected Object Lock metadata disk read failure: {bucket}")));
@@ -3122,7 +3044,7 @@ impl BucketMetadataSys {
         }
     }
 
-    pub async fn get_lifecycle_config(&self, bucket: &str) -> Result<(PersistedLifecycleConfiguration, OffsetDateTime)> {
+    pub async fn get_lifecycle_config(&self, bucket: &str) -> Result<(BucketLifecycleConfiguration, OffsetDateTime)> {
         let (bm, _) = self.get_config(bucket).await?;
 
         match ConfigState::of(&bm.lifecycle_config_xml, &bm.lifecycle_config).require(bucket, BUCKET_LIFECYCLE_CONFIG)? {
@@ -3245,7 +3167,7 @@ impl BucketMetadataSys {
         }))
     }
 
-    pub async fn get_replication_config(&self, bucket: &str) -> Result<(PersistedReplicationConfiguration, OffsetDateTime)> {
+    pub async fn get_replication_config(&self, bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
         let (bm, _) = self.get_config(bucket).await?;
 
         if !bm.replication_config_xml.is_empty() && bm.replication_config.is_none() {
@@ -3339,7 +3261,6 @@ mod tests {
     };
     use crate::bucket::target::{BucketTarget, BucketTargetType, Credentials};
     use crate::config::com::read_config;
-    use crate::disk::{DiskAPI as _, RUSTFS_META_TMP_BUCKET};
     use crate::storage_api_contracts::bucket::{BucketOperations as _, DeleteBucketOptions, MakeBucketOptions};
     use byteorder::{ByteOrder as _, LittleEndian};
     use serial_test::serial;
@@ -3394,180 +3315,6 @@ mod tests {
         ),
     ];
 
-    #[tokio::test]
-    #[serial(storage_class_env)]
-    async fn persistence_entry_failures_revoke_pending_creation_before_cache_publish() {
-        let (dirs, store) = isolated_store_over_temp_disks().await;
-        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
-        let sys = bucket_metadata_sys_of(&store.ctx).expect("instance metadata system");
-        let sys = sys.read().await.clone();
-        let options = MakeBucketOptions {
-            lock_enabled: true,
-            ..Default::default()
-        };
-        for entry in ["save", "persist", "persist-new", "commit"] {
-            let bucket = format!("revoke-{entry}-{}", Uuid::new_v4());
-            store.fail_next_bucket_creation_after_intent_for_test(&bucket);
-            store
-                .make_bucket(&bucket, &options)
-                .await
-                .expect_err("stop the real producer before physical publication");
-            sys.pending_bucket_creation(&bucket)
-                .await
-                .expect("the producer must leave a real live creation permission");
-            let (intent, persisted) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
-                .await
-                .expect("read the original producer intent");
-            assert!(persisted, "{entry}: the complete original intent must be persisted");
-            let original_intent = intent.marshal_msg().expect("encode the original intent");
-
-            // Keep the creation fence order used by the real producer.
-            let lifecycle_guard = store
-                .acquire_bucket_lifecycle_write_lock(&bucket)
-                .await
-                .expect("hold the bucket lifecycle fence");
-            let transaction_guard = acquire_bucket_metadata_transaction_lock_in(&store.ctx, &bucket)
-                .await
-                .expect("hold the metadata transaction fence");
-            let namespace_lock = store
-                .new_ns_lock(&bucket, &bucket)
-                .await
-                .expect("create the exact bucket namespace lock");
-            let namespace_guard = namespace_lock
-                .get_write_lock(crate::set_disk::get_lock_acquire_timeout())
-                .await
-                .expect("hold the exact bucket namespace fence");
-
-            // A file at each native staging root causes actual IO to fail before
-            // metadata can reach the cache-publishing set call. Preserve the
-            // original staging directories so no fixture data is discarded.
-            for dir in &dirs {
-                let staging = dir.path().join(RUSTFS_META_TMP_BUCKET);
-                let saved = dir.path().join(".rustfs.sys/revocation-saved-tmp");
-                std::fs::rename(&staging, &saved).expect("preserve the native staging directory");
-                std::fs::write(&staging, b"block metadata staging with a non-directory")
-                    .expect("install an actual non-directory IO obstruction");
-            }
-            let result = match entry {
-                "save" => sys.save(intent).await,
-                "persist" => sys.persist_and_set(intent).await,
-                "persist-new" => sys.persist_new_and_set(intent).await,
-                "commit" => sys.commit_bucket_metadata(intent).await,
-                _ => unreachable!(),
-            };
-            for dir in &dirs {
-                let staging = dir.path().join(RUSTFS_META_TMP_BUCKET);
-                let saved = dir.path().join(".rustfs.sys/revocation-saved-tmp");
-                std::fs::remove_file(&staging).expect("remove only the fixture's staging obstruction");
-                std::fs::rename(&saved, &staging).expect("restore the complete original staging directory");
-            }
-            drop((namespace_guard, transaction_guard, lifecycle_guard));
-            result.expect_err("native persistence must fail before its downstream cache set");
-            assert!(
-                sys.pending_bucket_creation(&bucket).await.is_none(),
-                "{entry}: the entry revoker must retire live permission even when persistence fails"
-            );
-            let (after_failure, present) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
-                .await
-                .expect("read the intent after the failed persistence entry");
-            assert!(present, "{entry}: a failed persistence entry must retain the original metadata");
-            assert_eq!(
-                after_failure
-                    .marshal_msg()
-                    .expect("encode the original intent after persistence failure"),
-                original_intent,
-                "{entry}: native IO failure must not change the original generation or complete intent"
-            );
-            let error = store
-                .make_bucket(&bucket, &options)
-                .await
-                .expect_err("a failed ordinary persistence entry cannot preserve creation permission");
-            assert!(error.to_string().contains("administrator recovery"), "{entry}: {error}");
-            let (after_retry, _) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
-                .await
-                .expect("read the original intent after the rejected retry");
-            assert_eq!(
-                after_retry.marshal_msg().expect("encode the intent after the rejected retry"),
-                original_intent,
-                "{entry}: the rejected retry must preserve the complete original intent"
-            );
-            for disk in store.pools[0].disk_set[0].disks.read().await.iter() {
-                assert!(
-                    matches!(
-                        disk.as_ref().expect("native disk").stat_volume(&bucket).await,
-                        Err(crate::disk::error::DiskError::VolumeNotFound)
-                    ),
-                    "{entry}: failed persistence and the rejected retry must not publish a physical volume"
-                );
-            }
-        }
-    }
-
-    #[tokio::test]
-    #[serial(storage_class_env)]
-    async fn metadata_reset_revokes_real_pending_creation_without_changing_persisted_intent() {
-        let (_dirs, store) = isolated_store_over_temp_disks().await;
-        init_bucket_metadata_sys(store.clone(), Vec::new()).await;
-        let bucket = format!("revoke-reset-{}", Uuid::new_v4());
-        let options = MakeBucketOptions {
-            lock_enabled: true,
-            ..Default::default()
-        };
-        store.fail_next_bucket_creation_after_intent_for_test(&bucket);
-        store
-            .make_bucket(&bucket, &options)
-            .await
-            .expect_err("stop the real producer before physical publication");
-        let sys = bucket_metadata_sys_of(&store.ctx).expect("instance metadata system");
-        let mut sys = sys.read().await.clone();
-        sys.pending_bucket_creation(&bucket)
-            .await
-            .expect("the real producer must leave live creation permission before reset");
-        let (intent, persisted) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
-            .await
-            .expect("read the original intent before cache reset");
-        assert!(persisted, "cache reset starts from a complete real persisted intent");
-        let original_intent = intent.marshal_msg().expect("encode the original reset intent");
-        sys._reset().await;
-        assert!(
-            sys.pending_bucket_creation(&bucket).await.is_none(),
-            "reset must revoke the producer-issued permission shared with the instance metadata system"
-        );
-        let (after_reset, present) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
-            .await
-            .expect("read the original intent after cache reset");
-        assert!(present, "cache reset must retain the persisted intent");
-        assert_eq!(
-            after_reset.marshal_msg().expect("encode the intent after cache reset"),
-            original_intent,
-            "cache reset must leave the original generation and complete persisted intent unchanged"
-        );
-        let error = store
-            .make_bucket(&bucket, &options)
-            .await
-            .expect_err("reset cannot regrant the producer's old creation permission");
-        assert!(error.to_string().contains("administrator recovery"), "{error}");
-        let (after_retry, _) = get_config_from_disk_with_presence_in(&store.ctx, &bucket)
-            .await
-            .expect("read the intent after the reset-induced retry rejection");
-        assert_eq!(
-            after_retry
-                .marshal_msg()
-                .expect("encode the intent after the rejected reset retry"),
-            original_intent,
-            "the reset-induced rejection must preserve the original complete intent"
-        );
-        for disk in store.pools[0].disk_set[0].disks.read().await.iter() {
-            assert!(
-                matches!(
-                    disk.as_ref().expect("native disk").stat_volume(&bucket).await,
-                    Err(crate::disk::error::DiskError::VolumeNotFound)
-                ),
-                "cache reset and the rejected retry must not publish a physical volume"
-            );
-        }
-    }
-
     #[tokio::test]
     async fn g_d3_003_new_writer_replication_loads_without_fail_closed_state() {
         let (dirs, store) = isolated_store_over_temp_disks().await;
@@ -3756,7 +3503,7 @@ mod tests {
         let mut versioned = BucketMetadata::new("authoritative-versioned");
         versioned.versioning_config_xml = b"<VersioningConfiguration><Status>Enabled</Status></VersioningConfiguration>".to_vec();
         versioned.versioning_config = Some(VersioningConfiguration {
-            status: Some("Enabled".to_owned()),
+            status: Some(s3s::dto::BucketVersioningStatus::from_static(s3s::dto::BucketVersioningStatus::ENABLED)),
             ..Default::default()
         });
         sys.set("authoritative-versioned".to_string(), Arc::new(versioned)).await;
@@ -3879,24 +3626,25 @@ mod tests {
         else {
             panic!("legacy lock-enabled metadata must remain configured");
         };
-        assert_eq!(config.object_lock_enabled.as_deref(), Some("Enabled"));
+        assert_eq!(
+            config.object_lock_enabled.as_ref().map(|value| value.as_str()),
+            Some(ObjectLockEnabled::ENABLED)
+        );
         assert!(config.rule.is_none());
     }
 
     #[test]
     fn authoritative_object_lock_state_rejects_invalid_retention_periods() {
-        use rustfs_gateway_types::persistence::{
-            PersistedDefaultRetention as DefaultRetention, PersistedObjectLockRule as ObjectLockRule,
-        };
+        use s3s::dto::{DefaultRetention, ObjectLockRule};
 
         for (days, years) in [(Some(0), None), (Some(-1), None), (Some(1), Some(1)), (None, None)] {
             let mut metadata = BucketMetadata::new("invalid-object-lock");
             metadata.object_lock_config = Some(ObjectLockConfiguration {
-                object_lock_enabled: Some("Enabled".to_owned()),
+                object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
                 rule: Some(ObjectLockRule {
                     default_retention: Some(DefaultRetention {
                         days,
-                        mode: Some("COMPLIANCE".to_owned()),
+                        mode: Some(ObjectLockRetentionMode::from_static(ObjectLockRetentionMode::COMPLIANCE)),
                         years,
                     }),
                 }),
@@ -3911,11 +3659,9 @@ mod tests {
 
     #[test]
     fn authoritative_object_lock_state_rejects_invalid_enabled_mode_and_rule() {
-        use rustfs_gateway_types::persistence::{
-            PersistedDefaultRetention as DefaultRetention, PersistedObjectLockRule as ObjectLockRule,
-        };
+        use s3s::dto::{DefaultRetention, ObjectLockRule};
 
-        for enabled in [None, Some("Disabled".to_string())] {
+        for enabled in [None, Some(ObjectLockEnabled::from("Disabled".to_string()))] {
             let mut metadata = BucketMetadata::new("invalid-object-lock-enabled");
             metadata.lock_enabled = true;
             metadata.object_lock_config = Some(ObjectLockConfiguration {
@@ -3927,11 +3673,11 @@ mod tests {
 
         let mut invalid_mode = BucketMetadata::new("invalid-object-lock-mode");
         invalid_mode.object_lock_config = Some(ObjectLockConfiguration {
-            object_lock_enabled: Some("Enabled".to_owned()),
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
             rule: Some(ObjectLockRule {
                 default_retention: Some(DefaultRetention {
                     days: Some(1),
-                    mode: Some("invalid".to_string()),
+                    mode: Some(ObjectLockRetentionMode::from("invalid".to_string())),
                     years: None,
                 }),
             }),
@@ -3940,7 +3686,7 @@ mod tests {
 
         let mut missing_retention = BucketMetadata::new("missing-object-lock-retention");
         missing_retention.object_lock_config = Some(ObjectLockConfiguration {
-            object_lock_enabled: Some("Enabled".to_owned()),
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
             rule: Some(ObjectLockRule { default_retention: None }),
         });
         assert!(object_lock_config_state_from_authoritative_metadata(&missing_retention).is_err());
@@ -3950,7 +3696,7 @@ mod tests {
     fn object_lock_ever_enabled_marker_survives_config_removal() {
         let mut metadata = BucketMetadata::new("lock-config-removed");
         let config = ObjectLockConfiguration {
-            object_lock_enabled: Some("Enabled".to_owned()),
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
             rule: None,
         };
         metadata
@@ -3968,7 +3714,10 @@ mod tests {
         else {
             panic!("an ever-enabled bucket must remain Object Lock configured");
         };
-        assert_eq!(config.object_lock_enabled.as_deref(), Some("Enabled"));
+        assert_eq!(
+            config.object_lock_enabled.as_ref().map(ObjectLockEnabled::as_str),
+            Some(ObjectLockEnabled::ENABLED)
+        );
     }
 
     #[tokio::test]
@@ -3984,7 +3733,7 @@ mod tests {
 
         let node = Arc::new(RwLock::new(BucketMetadataSys::new(ecstore.clone())));
         let config = ObjectLockConfiguration {
-            object_lock_enabled: Some("Enabled".to_owned()),
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
             rule: None,
         };
         update_with_sys(
@@ -4965,7 +4714,7 @@ mod tests {
     #[tokio::test]
     async fn update_config_with_persists_tagging_rewrite_across_disk_reload() {
         use crate::bucket::metadata::BUCKET_TAGGING_CONFIG;
-        use rustfs_gateway_types::cors_tagging::PersistedTag as Tag;
+        use s3s::dto::Tag;
 
         let (dirs, ecstore) = isolated_store_over_temp_disks().await;
 
@@ -5129,7 +4878,7 @@ mod tests {
     #[tokio::test]
     async fn concurrent_update_config_with_calls_do_not_lose_writes() {
         use crate::bucket::metadata::BUCKET_TAGGING_CONFIG;
-        use rustfs_gateway_types::cors_tagging::PersistedTag as Tag;
+        use s3s::dto::Tag;
 
         let (dirs, ecstore) = isolated_store_over_temp_disks().await;
         let sys = Arc::new(RwLock::new(BucketMetadataSys::new(ecstore)));
@@ -5229,7 +4978,7 @@ mod tests {
         let mut persisted = BucketMetadata::new("reload-bucket");
         persisted.policy_config_json = b"persisted-marker".to_vec();
         let lock_config = ObjectLockConfiguration {
-            object_lock_enabled: Some("Enabled".to_owned()),
+            object_lock_enabled: Some(ObjectLockEnabled::from_static(ObjectLockEnabled::ENABLED)),
             rule: None,
         };
         persisted
@@ -5508,7 +5257,7 @@ mod tests {
     #[serial]
     async fn bucket_delete_waits_for_config_mutation_fence() {
         use crate::bucket::metadata::BUCKET_TAGGING_CONFIG;
-        use rustfs_gateway_types::cors_tagging::PersistedTag as Tag;
+        use s3s::dto::Tag;
 
         let (_dirs, store) = isolated_store_over_temp_disks().await;
         init_bucket_metadata_sys(store.clone(), Vec::new()).await;

crates/ecstore/src/bucket/object_lock/mod.rs

@@ -17,8 +17,15 @@ pub mod objectlock_sys;
 pub mod types;
 
 /// Whether a bucket Object Lock configuration has locking enabled. The
-/// persisted `ObjectLockConfiguration` implements this in the bucket-metadata
-/// module, which owns the persisted configuration type.
+/// serving-layer `ObjectLockConfiguration` DTO implements this in
+/// the bucket-metadata module, which owns the persisted configuration type
+/// during the s3s ratchet migration (rustfs/backlog#1842).
 pub trait ObjectLockApi {
     fn enabled(&self) -> bool;
 }
+
+/// Whether a legal-hold status value is one of the two valid wire values.
+/// Implemented for the serving-layer DTO in the bucket-metadata module.
+pub trait ObjectLockStatusExt {
+    fn valid(&self) -> bool;
+}

crates/ecstore/src/bucket/replication/replication_config_boundary.rs

@@ -12,7 +12,6 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-pub use rustfs_replication::config_cache_serde;
 pub use rustfs_replication::{
     ObjectOpts, OperatorRuleContract, REMOTE_TARGET_CAPABILITY_CONTRACT_VERSION, REMOTE_TARGET_READ_ONLY_HISTORICAL_FIELDS,
     REMOTE_TARGET_UNSUPPORTED_FIELDS, REMOTE_TARGET_WRITABLE_FIELDS, REPLICATION_CAPABILITY_CONTRACT_VERSION,

crates/ecstore/src/bucket/replication/replication_lifecycle_bridge.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
+use s3s::dto::ReplicationConfiguration;
 
 use super::replication_config_boundary::ReplicationConfigurationExt as _;
 use super::replication_filemeta_boundary::{
@@ -42,7 +42,7 @@ impl ReplicationLifecycleBridge {
         dead_code,
         reason = "declared boundary surface for the ECStore replication split plan; no caller in this port (backlog#1823)"
     )]
-    pub(crate) fn new_config(config: PersistedReplicationConfiguration) -> ReplicationLifecycleConfig {
+    pub(crate) fn new_config(config: ReplicationConfiguration) -> ReplicationLifecycleConfig {
         ReplicationConfig::new(Some(config), None)
     }
 
@@ -118,17 +118,16 @@ impl ReplicationLifecycleBridge {
 
 #[cfg(test)]
 mod tests {
-    use rustfs_gateway_types::dto::Status;
-    use rustfs_gateway_types::persistence::{PersistedReplicationDestination, PersistedReplicationRule};
+    use s3s::dto::{Destination, ReplicationRule, ReplicationRuleStatus};
 
     use super::super::replication_filemeta_boundary::{ReplicateTargetDecision, VersionPurgeStatusType};
     use super::*;
 
-    fn replication_rule() -> PersistedReplicationRule {
-        PersistedReplicationRule {
+    fn replication_rule() -> ReplicationRule {
+        ReplicationRule {
             delete_marker_replication: None,
             delete_replication: None,
-            destination: PersistedReplicationDestination {
+            destination: Destination {
                 bucket: "arn:aws:s3:::target-bucket".to_string(),
                 ..Default::default()
             },
@@ -138,13 +137,13 @@ mod tests {
             prefix: Some(String::new()),
             priority: Some(1),
             source_selection_criteria: None,
-            status: Status::ENABLED.to_string(),
+            status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
         }
     }
 
     #[test]
     fn has_pending_version_purge_preserves_replication_active_rule_behavior() {
-        let config = ReplicationLifecycleBridge::new_config(PersistedReplicationConfiguration {
+        let config = ReplicationLifecycleBridge::new_config(ReplicationConfiguration {
             role: String::new(),
             rules: vec![replication_rule()],
         });

crates/ecstore/src/bucket/replication/replication_metadata_boundary.rs

@@ -17,8 +17,8 @@ use std::sync::Arc;
 use crate::bucket::{metadata::BucketMetadata, metadata_sys};
 use crate::disk::{BUCKET_META_PREFIX, RUSTFS_META_BUCKET};
 use crate::runtime::instance::InstanceContext;
-use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
 use rustfs_utils::path::path_join_buf;
+use s3s::dto::ReplicationConfiguration;
 use time::OffsetDateTime;
 
 use super::replication_error_boundary::{Error, Result};
@@ -36,11 +36,11 @@ impl ReplicationMetadataStore {
     pub(crate) const FORCE_DELETE_REPLICATION_FILE: &'static str = "config/replication/force-delete.bin";
     pub(crate) const FORCE_DELETE_REPLICATION_TRANSACTION_LOCK: &'static str = "config/replication/force-delete.bin.transaction";
 
-    pub(crate) async fn replication_config(bucket: &str) -> Result<(PersistedReplicationConfiguration, OffsetDateTime)> {
+    pub(crate) async fn replication_config(bucket: &str) -> Result<(ReplicationConfiguration, OffsetDateTime)> {
         metadata_sys::get_replication_config(bucket).await
     }
 
-    pub(crate) async fn optional_replication_config(bucket: &str) -> Result<Option<PersistedReplicationConfiguration>> {
+    pub(crate) async fn optional_replication_config(bucket: &str) -> Result<Option<ReplicationConfiguration>> {
         let config = match Self::replication_config(bucket).await {
             Ok((config, _)) => Some(config),
             Err(err) => {

crates/ecstore/src/bucket/replication/replication_object_config.rs

@@ -16,9 +16,7 @@ use std::{collections::HashMap, fmt, sync::Arc};
 
 use super::replication_filemeta_boundary::metadata_keys;
 use crate::bucket::metadata::BucketMetadata;
-use rustfs_gateway_types::dto::Status;
-use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
-use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
+use s3s::dto::{BucketVersioningStatus, ReplicationConfiguration, ReplicationRuleStatus, VersioningConfiguration};
 use serde::{Deserialize, Serialize};
 use time::OffsetDateTime;
 use tracing::error;
@@ -41,7 +39,7 @@ use super::replication_target_boundary::{BucketTargets, ReplicationTargetStore};
 use super::replication_versioning_boundary::ReplicationVersioningStore;
 use super::runtime_boundary as runtime_sources;
 
-pub(crate) async fn get_replication_config(bucket: &str) -> Result<Option<PersistedReplicationConfiguration>> {
+pub(crate) async fn get_replication_config(bucket: &str) -> Result<Option<ReplicationConfiguration>> {
     let config = ReplicationMetadataStore::optional_replication_config(bucket).await?;
     validate_delete_replication_config(&VersioningConfiguration::default(), config.as_ref())?;
     Ok(config)
@@ -66,7 +64,7 @@ impl DeleteReplicationConfigSnapshot {
     #[cfg(test)]
     pub(crate) fn from_configs_for_test(
         versioning: VersioningConfiguration,
-        replication: Option<PersistedReplicationConfiguration>,
+        replication: Option<ReplicationConfiguration>,
     ) -> Self {
         let metadata = replication.map(|config| {
             let mut metadata = BucketMetadata::new("test-bucket");
@@ -80,7 +78,7 @@ impl DeleteReplicationConfigSnapshot {
         &self.versioning
     }
 
-    pub fn replication_config(&self) -> Option<&PersistedReplicationConfiguration> {
+    pub fn replication_config(&self) -> Option<&ReplicationConfiguration> {
         self.metadata
             .as_ref()
             .and_then(|metadata| metadata.replication_config.as_ref())
@@ -103,7 +101,7 @@ impl DeleteReplicationConfigSnapshot {
     pub(crate) fn active_delete_marker_rules_require_tags(&self, object: &str) -> bool {
         self.replication_config().is_some_and(|config| {
             config.rules.iter().any(|rule| {
-                if rule.status == Status::DISABLED.as_str() {
+                if rule.status == ReplicationRuleStatus::from_static(ReplicationRuleStatus::DISABLED) {
                     return false;
                 }
                 if !object.starts_with(rule.prefix()) {
@@ -124,12 +122,12 @@ impl DeleteReplicationConfigSnapshot {
 
 fn validate_delete_replication_config(
     versioning: &VersioningConfiguration,
-    config: Option<&PersistedReplicationConfiguration>,
+    config: Option<&ReplicationConfiguration>,
 ) -> Result<()> {
     if versioning
         .status
         .as_ref()
-        .is_some_and(|status| !matches!(status.as_str(), "Enabled" | "Suspended"))
+        .is_some_and(|status| !matches!(status.as_str(), BucketVersioningStatus::ENABLED | BucketVersioningStatus::SUSPENDED))
     {
         return Err(super::replication_error_boundary::Error::other(
             "bucket versioning configuration has an invalid status",
@@ -146,7 +144,7 @@ fn validate_delete_replication_config(
         let role = config.role.trim();
         let mut role_destination = None;
         for rule in &config.rules {
-            if rule.status.as_str() == Status::ENABLED.as_str() {
+            if rule.status.as_str() == ReplicationRuleStatus::ENABLED {
                 let destination = rule.destination.bucket.trim();
                 if role.is_empty() && destination.is_empty() {
                     return Err(super::replication_error_boundary::Error::other(
@@ -171,7 +169,7 @@ fn validate_delete_replication_config(
     Ok(())
 }
 
-fn replication_config_from_metadata(metadata: &BucketMetadata) -> Result<Option<&PersistedReplicationConfiguration>> {
+fn replication_config_from_metadata(metadata: &BucketMetadata) -> Result<Option<&ReplicationConfiguration>> {
     if !metadata.replication_config_xml.is_empty() && metadata.replication_config.is_none() {
         return Err(super::replication_error_boundary::Error::other(
             "persisted bucket replication configuration is invalid",
@@ -252,15 +250,12 @@ pub(crate) async fn load_delete_replication_config_in(
 
 #[derive(Debug, Clone, Default, Serialize, Deserialize)]
 pub struct ReplicationConfig {
-    /// Carried in the scanner's persisted data usage cache, so its serde form
-    /// is the one the s3s DTO wrote there (see `config_cache_serde` in the replication crate).
-    #[serde(with = "super::replication_config_boundary::config_cache_serde")]
-    pub config: Option<PersistedReplicationConfiguration>,
+    pub config: Option<ReplicationConfiguration>,
     pub remotes: Option<BucketTargets>,
 }
 
 impl ReplicationConfig {
-    pub fn new(config: Option<PersistedReplicationConfiguration>, remotes: Option<BucketTargets>) -> Self {
+    pub fn new(config: Option<ReplicationConfiguration>, remotes: Option<BucketTargets>) -> Self {
         Self { config, remotes }
     }
 
@@ -473,7 +468,7 @@ fn check_replicate_delete_with_config(
     oi: &ObjectInfo,
     del_opts: &ObjectOptions,
     source_error: bool,
-    config: Option<&PersistedReplicationConfiguration>,
+    config: Option<&ReplicationConfiguration>,
     existing_delete_marker: bool,
     trust_persisted_replica_status: bool,
 ) -> ReplicateDecision {
@@ -592,7 +587,7 @@ pub(crate) async fn must_replicate(bucket: &str, object: &str, mopts: MustReplic
 }
 
 fn metadata_target_should_replicate(
-    cfg: &PersistedReplicationConfiguration,
+    cfg: &ReplicationConfiguration,
     opts: &ObjectOpts,
     mopts: &MustReplicateOptions,
     arn: &str,
@@ -602,21 +597,20 @@ fn metadata_target_should_replicate(
 
 #[cfg(test)]
 mod tests {
-    use rustfs_gateway_types::dto::Status;
-    use rustfs_gateway_types::persistence::{
-        PersistedOptionalReplicationStatus, PersistedReplicationDestination, PersistedReplicationFilter,
-        PersistedReplicationRule, PersistedReplicationStatus, PersistedReplicationTag, PersistedSourceSelectionCriteria,
+    use s3s::dto::{
+        DeleteMarkerReplication, DeleteMarkerReplicationStatus, DeleteReplication, DeleteReplicationStatus, Destination,
+        ReplicaModifications, ReplicationRule, ReplicationRuleFilter, ReplicationRuleStatus, SourceSelectionCriteria, Tag,
     };
 
     use super::super::replication_filemeta_boundary::VersionPurgeStatusType;
     use super::super::replication_target_boundary::BucketTarget;
     use super::*;
 
-    fn replication_rule() -> PersistedReplicationRule {
-        PersistedReplicationRule {
+    fn replication_rule() -> ReplicationRule {
+        ReplicationRule {
             delete_marker_replication: None,
             delete_replication: None,
-            destination: PersistedReplicationDestination {
+            destination: Destination {
                 bucket: "arn:aws:s3:::target-bucket".to_string(),
                 ..Default::default()
             },
@@ -626,7 +620,7 @@ mod tests {
             prefix: Some(String::new()),
             priority: Some(1),
             source_selection_criteria: None,
-            status: Status::ENABLED.to_string(),
+            status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
         }
     }
 
@@ -635,11 +629,11 @@ mod tests {
         let arn = "arn:rustfs:replication:us-east-1:target:bucket";
         let mut rule = replication_rule();
         rule.destination.bucket = arn.to_string();
-        rule.filter = Some(PersistedReplicationFilter {
+        rule.filter = Some(ReplicationRuleFilter {
             prefix: Some("admitted/".to_string()),
             ..Default::default()
         });
-        let cfg = PersistedReplicationConfiguration {
+        let cfg = ReplicationConfiguration {
             role: String::new(),
             rules: vec![rule],
         };
@@ -677,7 +671,7 @@ mod tests {
         assert!(!empty.replicate(&ObjectOpts::default()));
 
         let config = ReplicationConfig::new(
-            Some(PersistedReplicationConfiguration {
+            Some(ReplicationConfiguration {
                 role: String::new(),
                 rules: vec![replication_rule()],
             }),
@@ -731,7 +725,7 @@ mod tests {
     fn delete_snapshot_rejects_enabled_rules_without_a_destination() {
         let mut rule = replication_rule();
         rule.destination.bucket.clear();
-        let config = PersistedReplicationConfiguration {
+        let config = ReplicationConfiguration {
             role: String::new(),
             rules: vec![rule],
         };
@@ -747,7 +741,7 @@ mod tests {
         let first = replication_rule();
         let mut second = replication_rule();
         second.destination.bucket = "arn:aws:s3:::other-target".to_string();
-        let config = PersistedReplicationConfiguration {
+        let config = ReplicationConfiguration {
             role: "arn:aws:s3:::role-target".to_string(),
             rules: vec![first, second],
         };
@@ -758,47 +752,47 @@ mod tests {
     #[test]
     fn delete_snapshot_rejects_unknown_status_values() {
         let invalid_versioning = VersioningConfiguration {
-            status: Some("Enabld".to_string()),
+            status: Some("Enabld".to_string().into()),
             ..Default::default()
         };
         assert!(validate_delete_replication_config(&invalid_versioning, None).is_err());
 
         let mut invalid_rule = replication_rule();
-        invalid_rule.status = "Enabld".to_string();
-        let config = PersistedReplicationConfiguration {
+        invalid_rule.status = "Enabld".to_string().into();
+        let config = ReplicationConfiguration {
             role: String::new(),
             rules: vec![invalid_rule],
         };
         assert!(validate_delete_replication_config(&VersioningConfiguration::default(), Some(&config)).is_err());
 
         let mut invalid_delete = replication_rule();
-        invalid_delete.delete_replication = Some(PersistedReplicationStatus {
-            status: "Enabld".to_string(),
+        invalid_delete.delete_replication = Some(DeleteReplication {
+            status: "Enabld".to_string().into(),
         });
-        let config = PersistedReplicationConfiguration {
+        let config = ReplicationConfiguration {
             role: String::new(),
             rules: vec![invalid_delete],
         };
         assert!(validate_delete_replication_config(&VersioningConfiguration::default(), Some(&config)).is_err());
 
         let mut invalid_delete_marker = replication_rule();
-        invalid_delete_marker.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
-            status: Some("Enabld".to_string()),
+        invalid_delete_marker.delete_marker_replication = Some(DeleteMarkerReplication {
+            status: Some("Enabld".to_string().into()),
         });
-        let config = PersistedReplicationConfiguration {
+        let config = ReplicationConfiguration {
             role: String::new(),
             rules: vec![invalid_delete_marker],
         };
         assert!(validate_delete_replication_config(&VersioningConfiguration::default(), Some(&config)).is_err());
 
         let mut invalid_replica_modifications = replication_rule();
-        invalid_replica_modifications.source_selection_criteria = Some(PersistedSourceSelectionCriteria {
-            replica_modifications: Some(PersistedReplicationStatus {
-                status: "Enabld".to_string(),
+        invalid_replica_modifications.source_selection_criteria = Some(SourceSelectionCriteria {
+            replica_modifications: Some(ReplicaModifications {
+                status: "Enabld".to_string().into(),
             }),
             sse_kms_encrypted_objects: None,
         });
-        let config = PersistedReplicationConfiguration {
+        let config = ReplicationConfiguration {
             role: String::new(),
             rules: vec![invalid_replica_modifications],
         };
@@ -810,11 +804,11 @@ mod tests {
         let mut metadata = BucketMetadata::new("bucket");
         metadata.versioning_config_xml = b"configured".to_vec();
         metadata.versioning_config = Some(VersioningConfiguration {
-            status: Some("Enabled".to_owned()),
+            status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
             ..Default::default()
         });
         metadata.replication_config_xml = b"configured".to_vec();
-        metadata.replication_config = Some(PersistedReplicationConfiguration {
+        metadata.replication_config = Some(ReplicationConfiguration {
             role: String::new(),
             rules: vec![replication_rule()],
         });
@@ -854,7 +848,7 @@ mod tests {
         );
 
         let mut inconsistent = BucketMetadata::new("bucket");
-        inconsistent.replication_config = Some(PersistedReplicationConfiguration {
+        inconsistent.replication_config = Some(ReplicationConfiguration {
             role: String::new(),
             rules: vec![replication_rule()],
         });
@@ -869,11 +863,11 @@ mod tests {
         let arn = "arn:rustfs:replication:us-east-1:target:bucket";
         let mut rule = replication_rule();
         rule.destination.bucket = arn.to_string();
-        rule.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
-            status: Some(Status::ENABLED.to_string()),
+        rule.delete_marker_replication = Some(DeleteMarkerReplication {
+            status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
         });
         let mut metadata = BucketMetadata::new("bucket");
-        metadata.replication_config = Some(PersistedReplicationConfiguration {
+        metadata.replication_config = Some(ReplicationConfiguration {
             role: String::new(),
             rules: vec![rule],
         });
@@ -904,27 +898,27 @@ mod tests {
     fn delete_marker_source_read_is_required_only_for_tag_filtered_rules() {
         let mut prefix_rule = replication_rule();
         prefix_rule.prefix = Some("logs/".to_string());
-        prefix_rule.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
-            status: Some(Status::ENABLED.to_string()),
+        prefix_rule.delete_marker_replication = Some(DeleteMarkerReplication {
+            status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
         });
 
         let prefix_snapshot = DeleteReplicationConfigSnapshot::from_configs_for_test(
             VersioningConfiguration {
-                status: Some("Enabled".to_owned()),
+                status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
                 ..Default::default()
             },
-            Some(PersistedReplicationConfiguration {
+            Some(ReplicationConfiguration {
                 role: String::new(),
                 rules: vec![prefix_rule],
             }),
         );
 
         let mut tag_rule = replication_rule();
-        tag_rule.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
-            status: Some(Status::ENABLED.to_string()),
+        tag_rule.delete_marker_replication = Some(DeleteMarkerReplication {
+            status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::ENABLED)),
         });
-        tag_rule.filter = Some(PersistedReplicationFilter {
-            tag: Some(PersistedReplicationTag {
+        tag_rule.filter = Some(ReplicationRuleFilter {
+            tag: Some(Tag {
                 key: Some("class".to_string()),
                 value: Some("audit".to_string()),
             }),
@@ -932,10 +926,10 @@ mod tests {
         });
         let tag_snapshot = DeleteReplicationConfigSnapshot::from_configs_for_test(
             VersioningConfiguration {
-                status: Some("Enabled".to_owned()),
+                status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
                 ..Default::default()
             },
-            Some(PersistedReplicationConfiguration {
+            Some(ReplicationConfiguration {
                 role: String::new(),
                 rules: vec![tag_rule],
             }),
@@ -950,14 +944,14 @@ mod tests {
         let arn = "arn:rustfs:replication:us-east-1:target:bucket";
         let mut rule = replication_rule();
         rule.destination.bucket = arn.to_string();
-        rule.delete_replication = Some(PersistedReplicationStatus {
-            status: Status::ENABLED.to_string(),
+        rule.delete_replication = Some(DeleteReplication {
+            status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
         });
-        rule.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
-            status: Some(Status::DISABLED.to_string()),
+        rule.delete_marker_replication = Some(DeleteMarkerReplication {
+            status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::DISABLED)),
         });
         let config = ReplicationConfig::new(
-            Some(PersistedReplicationConfiguration {
+            Some(ReplicationConfiguration {
                 role: String::new(),
                 rules: vec![rule],
             }),
@@ -991,22 +985,22 @@ mod tests {
     #[test]
     fn live_delete_does_not_trust_persisted_replica_status() {
         let mut rule = replication_rule();
-        rule.delete_replication = Some(PersistedReplicationStatus {
-            status: Status::ENABLED.to_string(),
+        rule.delete_replication = Some(DeleteReplication {
+            status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
         });
-        rule.source_selection_criteria = Some(PersistedSourceSelectionCriteria {
-            replica_modifications: Some(PersistedReplicationStatus {
-                status: Status::DISABLED.to_string(),
+        rule.source_selection_criteria = Some(SourceSelectionCriteria {
+            replica_modifications: Some(ReplicaModifications {
+                status: s3s::dto::ReplicaModificationsStatus::from_static(s3s::dto::ReplicaModificationsStatus::DISABLED),
             }),
             sse_kms_encrypted_objects: None,
         });
-        let replication = PersistedReplicationConfiguration {
+        let replication = ReplicationConfiguration {
             role: String::new(),
             rules: vec![rule],
         };
         let snapshot = DeleteReplicationConfigSnapshot::from_configs_for_test(
             VersioningConfiguration {
-                status: Some("Enabled".to_owned()),
+                status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
                 ..Default::default()
             },
             Some(replication.clone()),
@@ -1042,14 +1036,14 @@ mod tests {
         let arn = "arn:rustfs:replication:us-east-1:target:bucket";
         let mut rule = replication_rule();
         rule.destination.bucket = arn.to_string();
-        rule.delete_replication = Some(PersistedReplicationStatus {
-            status: Status::ENABLED.to_string(),
+        rule.delete_replication = Some(DeleteReplication {
+            status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
         });
-        rule.delete_marker_replication = Some(PersistedOptionalReplicationStatus {
-            status: Some(Status::DISABLED.to_string()),
+        rule.delete_marker_replication = Some(DeleteMarkerReplication {
+            status: Some(DeleteMarkerReplicationStatus::from_static(DeleteMarkerReplicationStatus::DISABLED)),
         });
         let config = ReplicationConfig::new(
-            Some(PersistedReplicationConfiguration {
+            Some(ReplicationConfiguration {
                 role: String::new(),
                 rules: vec![rule],
             }),

crates/ecstore/src/bucket/replication/replication_pool.rs

@@ -4794,10 +4794,9 @@ mod tests {
     #[tokio::test]
     async fn heal_queue_marks_missing_versioning_state_as_missed() {
         use super::super::replication_target_boundary::BucketTargets;
-        use rustfs_gateway_types::dto::Status;
-        use rustfs_gateway_types::persistence::{
-            PersistedReplicationConfiguration, PersistedReplicationDestination, PersistedReplicationRule,
-            PersistedReplicationStatus,
+        use s3s::dto::{
+            DeleteReplication, DeleteReplicationStatus, Destination, ReplicationConfiguration, ReplicationRule,
+            ReplicationRuleStatus,
         };
 
         let arn = "arn:rustfs:replication:us-east-1:target:bucket";
@@ -4812,14 +4811,14 @@ mod tests {
                 ..Default::default()
             },
             ReplicationConfig::new(
-                Some(PersistedReplicationConfiguration {
+                Some(ReplicationConfiguration {
                     role: String::new(),
-                    rules: vec![PersistedReplicationRule {
+                    rules: vec![ReplicationRule {
                         delete_marker_replication: None,
-                        delete_replication: Some(PersistedReplicationStatus {
-                            status: Status::ENABLED.to_string(),
+                        delete_replication: Some(DeleteReplication {
+                            status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
                         }),
-                        destination: PersistedReplicationDestination {
+                        destination: Destination {
                             bucket: arn.to_string(),
                             ..Default::default()
                         },
@@ -4829,7 +4828,7 @@ mod tests {
                         prefix: Some(String::new()),
                         priority: Some(1),
                         source_selection_criteria: None,
-                        status: Status::ENABLED.to_string(),
+                        status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
                     }],
                 }),
                 Some(BucketTargets::default()),

crates/ecstore/src/bucket/replication/replication_resyncer.rs

@@ -85,8 +85,6 @@ use http_body_util::StreamBody;
 use metrics::counter;
 #[cfg(test)]
 use rmp_serde;
-#[cfg(test)]
-use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
 use rustfs_s3_types::EventName;
 use rustfs_utils::http::{
     AMZ_OBJECT_LOCK_LEGAL_HOLD, AMZ_OBJECT_LOCK_MODE, AMZ_OBJECT_LOCK_RETAIN_UNTIL_DATE, AMZ_TAGGING_DIRECTIVE,
@@ -94,6 +92,8 @@ use rustfs_utils::http::{
     insert_str, replication_target_versions,
 };
 use rustfs_utils::{DEFAULT_SIP_HASH_KEY, get_env_usize, sip_hash};
+#[cfg(test)]
+use s3s::dto::ReplicationConfiguration;
 use std::collections::{HashMap, HashSet};
 use std::fmt::Display;
 use std::pin::Pin;
@@ -6500,10 +6500,9 @@ mod tests {
 
     use super::super::replication_target_boundary::{BucketTarget, BucketTargets};
     use super::*;
-    use rustfs_gateway_types::dto::Status;
-    use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
-    use rustfs_gateway_types::persistence::{
-        PersistedReplicationDestination, PersistedReplicationRule, PersistedReplicationStatus,
+    use s3s::dto::{
+        BucketVersioningStatus, DeleteReplication, DeleteReplicationStatus, Destination, ExcludedPrefix, ReplicationRule,
+        ReplicationRuleStatus, VersioningConfiguration,
     };
     use std::collections::HashMap;
     use time::OffsetDateTime;
@@ -8569,7 +8568,7 @@ mod tests {
             ..Default::default()
         };
         let rcfg = ReplicationConfig::new(
-            Some(PersistedReplicationConfiguration {
+            Some(ReplicationConfiguration {
                 role: role.to_string(),
                 rules: vec![],
             }),
@@ -8595,7 +8594,7 @@ mod tests {
         ReplicationVersioningStore::install_prefix_state_test_config(
             &bucket,
             VersioningConfiguration {
-                status: Some("Enabled".to_owned()),
+                status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
                 ..Default::default()
             },
         );
@@ -8628,20 +8627,22 @@ mod tests {
         ReplicationVersioningStore::install_prefix_state_test_config(
             &bucket,
             VersioningConfiguration {
-                status: Some("Enabled".to_owned()),
-                excluded_prefixes: Some(vec![Some("archive/".to_string())]),
+                status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
+                excluded_prefixes: Some(vec![ExcludedPrefix {
+                    prefix: Some("archive/".to_string()),
+                }]),
                 ..Default::default()
             },
         );
         let rcfg = ReplicationConfig::new(
-            Some(PersistedReplicationConfiguration {
+            Some(ReplicationConfiguration {
                 role: String::new(),
-                rules: vec![PersistedReplicationRule {
+                rules: vec![ReplicationRule {
                     delete_marker_replication: None,
-                    delete_replication: Some(PersistedReplicationStatus {
-                        status: Status::ENABLED.to_string(),
+                    delete_replication: Some(DeleteReplication {
+                        status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::ENABLED),
                     }),
-                    destination: PersistedReplicationDestination {
+                    destination: Destination {
                         bucket: arn.to_string(),
                         ..Default::default()
                     },
@@ -8651,7 +8652,7 @@ mod tests {
                     prefix: Some(String::new()),
                     priority: Some(1),
                     source_selection_criteria: None,
-                    status: Status::ENABLED.to_string(),
+                    status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
                 }],
             }),
             Some(BucketTargets {
@@ -8687,14 +8688,14 @@ mod tests {
         let admitted_arn = "arn:rustfs:replication:us-east-1:target:admitted";
         let current_role = "arn:rustfs:replication:us-east-1:target:current";
         let rcfg = ReplicationConfig::new(
-            Some(PersistedReplicationConfiguration {
+            Some(ReplicationConfiguration {
                 role: current_role.to_string(),
-                rules: vec![PersistedReplicationRule {
+                rules: vec![ReplicationRule {
                     delete_marker_replication: None,
-                    delete_replication: Some(PersistedReplicationStatus {
-                        status: Status::DISABLED.to_string(),
+                    delete_replication: Some(DeleteReplication {
+                        status: DeleteReplicationStatus::from_static(DeleteReplicationStatus::DISABLED),
                     }),
-                    destination: PersistedReplicationDestination {
+                    destination: Destination {
                         bucket: current_role.to_string(),
                         ..Default::default()
                     },
@@ -8704,7 +8705,7 @@ mod tests {
                     prefix: Some(String::new()),
                     priority: Some(1),
                     source_selection_criteria: None,
-                    status: Status::ENABLED.to_string(),
+                    status: ReplicationRuleStatus::from_static(ReplicationRuleStatus::ENABLED),
                 }],
             }),
             Some(BucketTargets {
@@ -8752,12 +8753,12 @@ mod tests {
     #[tokio::test]
     async fn heal_rejects_semantically_invalid_replication_config() {
         let rcfg = ReplicationConfig::new(
-            Some(PersistedReplicationConfiguration {
+            Some(ReplicationConfiguration {
                 role: String::new(),
-                rules: vec![PersistedReplicationRule {
+                rules: vec![ReplicationRule {
                     delete_marker_replication: None,
                     delete_replication: None,
-                    destination: PersistedReplicationDestination {
+                    destination: Destination {
                         bucket: "arn:rustfs:replication:us-east-1:target:bucket".to_string(),
                         ..Default::default()
                     },
@@ -8767,7 +8768,7 @@ mod tests {
                     prefix: Some(String::new()),
                     priority: Some(1),
                     source_selection_criteria: None,
-                    status: "Enabld".to_owned(),
+                    status: ReplicationRuleStatus::from_static("Enabld"),
                 }],
             }),
             Some(BucketTargets::default()),

crates/ecstore/src/bucket/replication/replication_target_config_bridge.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use rustfs_gateway_types::persistence::PersistedReplicationConfiguration;
+use s3s::dto::ReplicationConfiguration;
 
 use super::replication_config_boundary::{ObjectOpts, ReplicationConfigurationExt};
 use super::replication_filemeta_boundary::ReplicationType;
@@ -20,7 +20,7 @@ use super::replication_filemeta_boundary::ReplicationType;
 pub(crate) struct ReplicationTargetConfigBridge;
 
 impl ReplicationTargetConfigBridge {
-    pub(crate) fn target_is_used_by_rules(config: &PersistedReplicationConfiguration, arn: &str) -> bool {
+    pub(crate) fn target_is_used_by_rules(config: &ReplicationConfiguration, arn: &str) -> bool {
         config
             .filter_target_arns(&ObjectOpts {
                 op_type: ReplicationType::All,
@@ -37,7 +37,7 @@ mod tests {
 
     #[test]
     fn target_config_bridge_matches_role_target() {
-        let config = PersistedReplicationConfiguration {
+        let config = ReplicationConfiguration {
             role: " arn:target ".to_string(),
             rules: Vec::new(),
         };

crates/ecstore/src/bucket/replication/replication_versioning_boundary.rs

@@ -15,7 +15,7 @@
 use super::replication_error_boundary::Result;
 use crate::bucket::{versioning::VersioningApi as _, versioning_sys::BucketVersioningSys};
 #[cfg(test)]
-use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
+use s3s::dto::VersioningConfiguration;
 #[cfg(test)]
 use std::{collections::HashMap, sync::LazyLock, sync::Mutex};
 

crates/ecstore/src/bucket/tagging/mod.rs

@@ -12,7 +12,7 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use rustfs_gateway_types::cors_tagging::PersistedTag as Tag;
+use s3s::dto::Tag;
 use std::cmp::Ordering;
 use std::collections::HashMap;
 use url::form_urlencoded;

crates/ecstore/src/bucket/utils.rs

@@ -15,23 +15,9 @@
 use crate::disk::{MIGRATING_META_BUCKET, RUSTFS_META_BUCKET};
 use crate::error::{Error, Result, StorageError};
 use regex::Regex;
-use rustfs_gateway_types::cors_tagging::{
-    CorsTaggingCodecError, PersistedCorsConfiguration, PersistedTagging, parse_cors, parse_tagging, serialize_cors,
-    serialize_tagging,
-};
-use rustfs_gateway_types::persistence::{
-    PersistedAccelerateConfiguration, PersistedBucketEncryptionConfiguration, PersistedBucketLoggingStatus,
-    PersistedLifecycleConfiguration, PersistedNotificationConfiguration, PersistedObjectLockConfiguration,
-    PersistedPublicAccessBlockConfiguration, PersistedReplicationConfiguration, PersistedRequestPaymentConfiguration,
-    PersistedVersioningConfiguration, PersistedWebsiteConfiguration, PersistenceCodecError, parse_accelerate,
-    parse_bucket_encryption, parse_bucket_logging, parse_lifecycle_legacy_dates, parse_notification, parse_object_lock,
-    parse_public_access_block, parse_replication, parse_request_payment, parse_versioning, parse_website, serialize_accelerate,
-    serialize_bucket_encryption, serialize_bucket_logging, serialize_lifecycle_legacy_dates, serialize_notification,
-    serialize_object_lock, serialize_public_access_block, serialize_replication, serialize_request_payment, serialize_versioning,
-    serialize_website,
-};
 use rustfs_utils::WindowsFileNameError;
 use rustfs_utils::path::SLASH_SEPARATOR;
+use s3s::xml;
 use tracing::instrument;
 
 pub fn is_meta_bucketname(name: &str) -> bool {
@@ -88,173 +74,37 @@ pub fn check_valid_bucket_name_strict(bucket_name: &str) -> Result<()> {
     check_bucket_name_common(bucket_name, true)
 }
 
-/// A bucket configuration persisted as XML in the bucket metadata, together
-/// with the codec that reads and writes its stored bytes.
-///
-/// Every family reads and writes its stored bytes through the gateway
-/// persistence codecs (`rustfs_gateway_types::persistence` and
-/// `rustfs_gateway_types::cors_tagging`, rustfs/backlog#2745). The codecs keep
-/// the persisted bytes the `bucket_config_goldens` pin, so a document the old
-/// s3s codec wrote reads back and re-encodes unchanged.
-pub trait BucketConfigXml: Sized {
-    /// Decodes the stored bytes of one configuration document.
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError>;
-
-    /// Encodes the value as RustFS stores it.
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError>;
+// RUSTFS_COMPAT_TODO(s3gate-metadata-xml): the s3s codec reads persisted XML during migration. Remove after every supported writer uses the gateway codec and every retained metadata object and backup archive is verified or rewritten.
+pub fn deserialize<T>(input: &[u8]) -> xml::DeResult<T>
+where
+    T: for<'xml> xml::Deserialize<'xml>,
+{
+    let mut d = xml::Deserializer::new(input);
+    let ans = T::deserialize(&mut d)?;
+    d.expect_eof()?;
+    Ok(ans)
 }
 
-/// Why a persisted bucket configuration could not be decoded or encoded.
-#[derive(Debug, thiserror::Error)]
-pub enum BucketConfigXmlError {
-    /// A gateway persistence codec refused the bytes or the value.
-    #[error(transparent)]
-    Persisted(#[from] PersistenceCodecError),
-    /// The gateway CORS or Tagging codec refused the bytes or the value.
-    #[error(transparent)]
-    CorsTagging(#[from] CorsTaggingCodecError),
+#[allow(
+    dead_code,
+    reason = "xml serialize helper with no caller in this port; the live sibling is deserialize (backlog#1823)"
+)]
+pub fn serialize_content<T: xml::SerializeContent>(val: &T) -> xml::SerResult<String> {
+    let mut buf = Vec::with_capacity(256);
+    {
+        let mut ser = xml::Serializer::new(&mut buf);
+        val.serialize_content(&mut ser)?;
+    }
+    Ok(String::from_utf8(buf).unwrap())
 }
 
-/// Decodes one persisted bucket configuration document.
-pub fn deserialize<T: BucketConfigXml>(input: &[u8]) -> std::result::Result<T, BucketConfigXmlError> {
-    T::decode_config_xml(input)
-}
-
-/// Encodes one bucket configuration as RustFS persists it.
-pub fn serialize<T: BucketConfigXml>(val: &T) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-    val.encode_config_xml()
-}
-
-impl BucketConfigXml for PersistedReplicationConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_replication(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_replication(self)?)
-    }
-}
-
-impl BucketConfigXml for PersistedLifecycleConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_lifecycle_legacy_dates(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_lifecycle_legacy_dates(self)?)
-    }
-}
-
-// The gateway writers of these families are infallible: a value that exists is
-// always encodable, so only the readers can refuse bytes.
-impl BucketConfigXml for PersistedAccelerateConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_accelerate(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_accelerate(self))
-    }
-}
-
-impl BucketConfigXml for PersistedBucketLoggingStatus {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_bucket_logging(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_bucket_logging(self))
-    }
-}
-
-impl BucketConfigXml for PersistedCorsConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_cors(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_cors(self))
-    }
-}
-
-impl BucketConfigXml for PersistedNotificationConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_notification(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_notification(self))
-    }
-}
-
-impl BucketConfigXml for PersistedObjectLockConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_object_lock(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_object_lock(self))
-    }
-}
-
-impl BucketConfigXml for PersistedPublicAccessBlockConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_public_access_block(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_public_access_block(self))
-    }
-}
-
-impl BucketConfigXml for PersistedRequestPaymentConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_request_payment(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_request_payment(self))
-    }
-}
-
-impl BucketConfigXml for PersistedBucketEncryptionConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_bucket_encryption(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_bucket_encryption(self))
-    }
-}
-
-impl BucketConfigXml for PersistedTagging {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_tagging(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_tagging(self))
-    }
-}
-
-impl BucketConfigXml for PersistedVersioningConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_versioning(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_versioning(self))
-    }
-}
-
-impl BucketConfigXml for PersistedWebsiteConfiguration {
-    fn decode_config_xml(input: &[u8]) -> std::result::Result<Self, BucketConfigXmlError> {
-        Ok(parse_website(input)?)
-    }
-
-    fn encode_config_xml(&self) -> std::result::Result<Vec<u8>, BucketConfigXmlError> {
-        Ok(serialize_website(self))
+pub fn serialize<T: xml::Serialize>(val: &T) -> xml::SerResult<Vec<u8>> {
+    let mut buf = Vec::with_capacity(256);
+    {
+        let mut ser = xml::Serializer::new(&mut buf);
+        val.serialize(&mut ser)?;
     }
+    Ok(buf)
 }
 
 pub fn has_bad_path_component(path: &str) -> bool {
@@ -584,40 +434,6 @@ mod tests {
     use super::*;
     use proptest::prelude::*;
 
-    /// Every bucket-configuration family reads and writes through its own
-    /// gateway codec (rustfs/backlog#2745): the family's golden document
-    /// decodes and re-encodes to the same bytes, and a document of another
-    /// family is refused instead of decoding into an empty configuration.
-    #[test]
-    fn every_bucket_config_family_uses_its_own_gateway_codec() {
-        fn assert_family<T: BucketConfigXml>(family: &str, own: &[u8], foreign: &[u8]) {
-            let value = deserialize::<T>(own).unwrap_or_else(|err| panic!("{family}: own document must decode: {err}"));
-            let encoded = serialize(&value).unwrap_or_else(|err| panic!("{family}: value must encode: {err}"));
-            assert_eq!(encoded, own, "{family}: re-encoding must keep the persisted bytes");
-            assert!(deserialize::<T>(foreign).is_err(), "{family}: a foreign root element must be refused");
-        }
-
-        macro_rules! golden {
-            ($family:literal) => {
-                include_bytes!(concat!("../../tests/fixtures/bucket-config-goldens/", $family, "/minio.out.xml"))
-            };
-        }
-        let tagging = golden!("tagging");
-        let versioning = golden!("versioning");
-
-        assert_family::<PersistedAccelerateConfiguration>("accelerate", golden!("accelerate"), tagging);
-        assert_family::<PersistedBucketLoggingStatus>("logging", golden!("logging"), tagging);
-        assert_family::<PersistedCorsConfiguration>("cors", golden!("cors"), tagging);
-        assert_family::<PersistedNotificationConfiguration>("notification", golden!("notification"), tagging);
-        assert_family::<PersistedObjectLockConfiguration>("object_lock", golden!("object_lock"), tagging);
-        assert_family::<PersistedPublicAccessBlockConfiguration>("public_access_block", golden!("public_access_block"), tagging);
-        assert_family::<PersistedRequestPaymentConfiguration>("request_payment", golden!("request_payment"), tagging);
-        assert_family::<PersistedBucketEncryptionConfiguration>("encryption", golden!("encryption"), tagging);
-        assert_family::<PersistedTagging>("tagging", tagging, versioning);
-        assert_family::<PersistedVersioningConfiguration>("versioning", versioning, tagging);
-        assert_family::<PersistedWebsiteConfiguration>("website", golden!("website"), tagging);
-    }
-
     /// rustfs#5785: keys whose path segments exceed the on-disk NAME_MAX
     /// budget must be rejected up front as ObjectNameInvalid (4xx), not leak
     /// ENAMETOOLONG as InternalError 500 from the disk layer.

crates/ecstore/src/bucket/versioning/mod.rs

@@ -12,8 +12,8 @@
 // See the License for the specific language governing permissions and
 // limitations under the License.
 
-use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
 use rustfs_utils::string::match_simple;
+use s3s::dto::{BucketVersioningStatus, VersioningConfiguration};
 
 pub trait VersioningApi {
     fn enabled(&self) -> bool;
@@ -28,10 +28,10 @@ pub trait VersioningApi {
 
 impl VersioningApi for VersioningConfiguration {
     fn enabled(&self) -> bool {
-        self.status.as_deref() == Some("Enabled")
+        self.status == Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED))
     }
     fn prefix_enabled(&self, prefix: &str) -> bool {
-        if self.status.as_deref() != Some("Enabled") {
+        if self.status != Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)) {
             return false;
         }
 
@@ -44,10 +44,12 @@ impl VersioningApi for VersioningConfiguration {
         }
 
         if let Some(ref excluded_prefixes) = self.excluded_prefixes {
-            for sprefix in excluded_prefixes.iter().flatten() {
-                let pattern = format!("{sprefix}*");
-                if match_simple(&pattern, prefix) {
-                    return false;
+            for p in excluded_prefixes.iter() {
+                if let Some(ref sprefix) = p.prefix {
+                    let pattern = format!("{sprefix}*");
+                    if match_simple(&pattern, prefix) {
+                        return false;
+                    }
                 }
             }
         }
@@ -56,11 +58,11 @@ impl VersioningApi for VersioningConfiguration {
     }
 
     fn prefix_suspended(&self, prefix: &str) -> bool {
-        if self.status.as_deref() == Some("Suspended") {
+        if self.status == Some(BucketVersioningStatus::from_static(BucketVersioningStatus::SUSPENDED)) {
             return true;
         }
 
-        if self.status.as_deref() == Some("Enabled") {
+        if self.status == Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)) {
             if prefix.is_empty() {
                 return false;
             }
@@ -73,10 +75,12 @@ impl VersioningApi for VersioningConfiguration {
             }
 
             if let Some(ref excluded_prefixes) = self.excluded_prefixes {
-                for sprefix in excluded_prefixes.iter().flatten() {
-                    let pattern = format!("{sprefix}*");
-                    if match_simple(&pattern, prefix) {
-                        return true;
+                for p in excluded_prefixes.iter() {
+                    if let Some(ref sprefix) = p.prefix {
+                        let pattern = format!("{sprefix}*");
+                        if match_simple(&pattern, prefix) {
+                            return true;
+                        }
                     }
                 }
             }
@@ -88,12 +92,14 @@ impl VersioningApi for VersioningConfiguration {
         self.prefix_enabled(prefix) || self.prefix_suspended(prefix)
     }
     fn suspended(&self) -> bool {
-        self.status.as_deref() == Some("Suspended")
+        self.status == Some(BucketVersioningStatus::from_static(BucketVersioningStatus::SUSPENDED))
     }
 }
 
 #[cfg(test)]
 mod tests {
+    use s3s::dto::{BucketVersioningStatus, ExcludedPrefix};
+
     use super::*;
 
     struct LegacyVersioning;
@@ -128,8 +134,10 @@ mod tests {
     #[test]
     fn delete_state_treats_excluded_prefixes_as_unversioned() {
         let config = VersioningConfiguration {
-            status: Some("Enabled".to_owned()),
-            excluded_prefixes: Some(vec![Some("archive/".to_string())]),
+            status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::ENABLED)),
+            excluded_prefixes: Some(vec![ExcludedPrefix {
+                prefix: Some("archive/".to_string()),
+            }]),
             ..Default::default()
         };
 
@@ -138,7 +146,7 @@ mod tests {
         assert_eq!(config.delete_state("live/object"), (true, false));
 
         let suspended = VersioningConfiguration {
-            status: Some("Suspended".to_owned()),
+            status: Some(BucketVersioningStatus::from_static(BucketVersioningStatus::SUSPENDED)),
             ..Default::default()
         };
         assert_eq!(suspended.delete_state("archive/object"), (false, true));

crates/ecstore/src/bucket/versioning_sys.rs

@@ -17,7 +17,7 @@ use super::metadata::unreadable_config_refusal;
 use super::{metadata_sys::get_bucket_metadata_sys, versioning::VersioningApi};
 use crate::disk::RUSTFS_META_BUCKET;
 use crate::error::Result;
-use rustfs_gateway_types::persistence::PersistedVersioningConfiguration as VersioningConfiguration;
+use s3s::dto::VersioningConfiguration;
 use tracing::{error, warn};
 
 pub struct BucketVersioningSys {}
@@ -163,7 +163,7 @@ mod tests {
 
     fn enabled() -> VersioningConfiguration {
         VersioningConfiguration {
-            status: Some("Enabled".to_owned()),
+            status: Some(s3s::dto::BucketVersioningStatus::from_static(s3s::dto::BucketVersioningStatus::ENABLED)),
             ..Default::default()
         }
     }

crates/ecstore/src/cluster/rpc/peer_rest_client.rs

@@ -1965,15 +1965,6 @@ impl PeerRestClient {
         Ok((self.topology_member.clone(), epoch))
     }
 
-    pub async fn probe_restore_copyback_capability(&self, topology_fingerprint: String) -> Result<(String, Uuid)> {
-        let probe = rustfs_protos::restore_copyback_capability_probe(Uuid::new_v4().as_bytes());
-        let result = self
-            .heal_control(rustfs_protos::HEAL_CONTROL_PROTOCOL_VERSION, topology_fingerprint, probe)
-            .await?;
-        let epoch = decode_remote_version_state_capability(&self.topology_member, &result)?;
-        Ok((self.topology_member.clone(), epoch))
-    }
-
     pub async fn probe_cross_pool_fence(&self, topology_fingerprint: String) -> Result<(String, u32, Uuid)> {
         let mut probe = rustfs_protos::CROSS_POOL_FENCE_CAPABILITY_PROBE_PREFIX.to_vec();
         probe.extend_from_slice(Uuid::new_v4().as_bytes());

crates/ecstore/src/cluster/rpc/remote_locker.rs

@@ -41,6 +41,10 @@ use tonic::service::interceptor::InterceptedService;
 use tonic::{Request, Response};
 use tracing::{debug, info, warn};
 
+const EVENT_NAMESPACE_LOCK_RELEASE: &str = "namespace_lock_release";
+const LOG_COMPONENT_ECSTORE: &str = "ecstore";
+const LOG_SUBSYSTEM_REMOTE_LOCK: &str = "remote_lock";
+
 fn attach_lock_mutation_body_digest<T: rustfs_protos::CanonicalMutationBody>(request: &mut Request<T>) -> std::io::Result<()> {
     set_tonic_rolling_mutation_body_digest(request)
 }
@@ -233,6 +237,20 @@ pub struct RemoteClient {
 impl RemoteClient {
     const ONLINE_CHECK_RESOURCE: &'static str = "health-lock-online";
 
+    fn record_release_ack(&self, lock_id: &LockId) {
+        tracing::trace!(
+            target: "rustfs_ecstore::object_lock_diag",
+            event = EVENT_NAMESPACE_LOCK_RELEASE,
+            component = LOG_COMPONENT_ECSTORE,
+            subsystem = LOG_SUBSYSTEM_REMOTE_LOCK,
+            state = "acknowledged",
+            peer_kind = "remote",
+            lock_id = %lock_id,
+            peer = %self.addr,
+            "Remote namespace lock release acknowledged"
+        );
+    }
+
     pub fn new(endpoint: String) -> Self {
         Self { addr: endpoint }
     }
@@ -1204,8 +1222,6 @@ impl LockClient for RemoteClient {
     }
 
     async fn release(&self, lock_id: &LockId) -> Result<bool> {
-        info!("remote release for {}", lock_id);
-
         let unlock_request = Self::create_unlock_request(lock_id);
         let request_string = serde_json::to_string(&unlock_request)
             .map_err(|e| LockError::internal(format!("Failed to serialize request: {e}")))?;
@@ -1226,6 +1242,9 @@ impl LockClient for RemoteClient {
         if let Some(error_info) = resp.error_info {
             return Err(LockError::internal(error_info));
         }
+        if resp.success {
+            self.record_release_ack(lock_id);
+        }
         Ok(resp.success)
     }
 
@@ -1261,7 +1280,13 @@ impl LockClient for RemoteClient {
         Ok(lock_ids
             .iter()
             .enumerate()
-            .map(|(idx, _)| resp.results.get(idx).map(|result| result.success).unwrap_or(false))
+            .map(|(idx, lock_id)| {
+                let result = resp.results.get(idx);
+                if result.is_some_and(|result| result.success && result.error_info.is_none()) {
+                    self.record_release_ack(lock_id);
+                }
+                result.map(|result| result.success).unwrap_or(false)
+            })
             .collect())
     }
 

crates/ecstore/src/config/com.rs

@@ -488,6 +488,18 @@ where
     Ok(data)
 }
 
+/// Read config bytes and metadata while preserving an existing empty payload.
+pub(crate) async fn read_config_preserve_empty_with_metadata_opts<S>(
+    api: Arc<S>,
+    file: &str,
+    opts: &ObjectOptions,
+) -> Result<(Vec<u8>, ObjectInfo)>
+where
+    S: EcstoreObjectIO,
+{
+    read_config_with_metadata_inner(api, file, opts, true, None).await
+}
+
 pub async fn read_config_no_lock<S>(api: Arc<S>, file: &str) -> Result<Vec<u8>>
 where
     S: ObjectIO<

crates/ecstore/src/core/pools.rs

@@ -84,13 +84,12 @@ use rand::RngExt as _;
 use rmp_serde::Deserializer;
 use rmp_serde::Serializer;
 use rustfs_filemeta::{FileInfo, FileInfoVersions, MetaCacheEntries, MetaCacheEntry, MetadataResolutionParams};
-use rustfs_gateway_types::persistence::PersistedObjectLockConfiguration as ObjectLockConfiguration;
-use rustfs_gateway_types::persistence::{PersistedLifecycleConfiguration, PersistedReplicationConfiguration};
 use rustfs_heal_contracts::heal_channel::HealOpts;
 use rustfs_utils::crypto::{hex_sha256, is_sha256_checksum};
 use rustfs_utils::path::{
     decode_dir_object, encode_dir_object, path_join, path_to_bucket_object, path_to_bucket_object_with_base_path,
 };
+use s3s::dto::{BucketLifecycleConfiguration, ObjectLockConfiguration, ReplicationConfiguration};
 use serde::{Deserialize, Serialize};
 use sha2::{Digest, Sha256};
 use std::collections::{HashMap, HashSet};
@@ -3721,6 +3720,46 @@ fn is_decommission_copy_quorum_deferral(err: &Error) -> bool {
     data_movement::data_movement_stage_source_as::<DecommissionCopyQuorumFailure>(err).is_some()
 }
 
+/// The storage error of a version copy that kept timing out on a namespace
+/// lock through its in-place retries, kept typed so the entry failure policy
+/// defers it.
+#[derive(Debug, thiserror::Error)]
+#[error("{0}")]
+struct DecommissionCopyLockTimeout(#[source] Error);
+
+fn is_decommission_copy_lock_timeout_error(err: &Error) -> bool {
+    let err = data_movement::data_movement_stage_source(err).unwrap_or(err);
+    matches!(err, Error::Lock(rustfs_lock::LockError::Timeout { .. }))
+}
+
+/// A version copy that timed out acquiring a namespace lock leaves the source
+/// in place; the next attempt copies the version again or finds an equivalent
+/// committed target. The lock is held by another mutation of the key or, when
+/// this node restarted mid-copy, by its previous incarnation until that lock's
+/// lease expires on the peers. Either clears by waiting, so the entry is
+/// deferred instead of being counted as a permanent per-object failure.
+fn decommission_copy_lock_timeout_deferral(stage: &str, bucket: &str, object: &str, err: Error) -> Error {
+    data_movement::data_movement_context_error(
+        format!("decommission {stage} timed out on a namespace lock for {bucket}/{object}: {err}"),
+        DecommissionCopyLockTimeout(err),
+    )
+}
+
+fn is_decommission_copy_lock_timeout_deferral(err: &Error) -> bool {
+    data_movement::data_movement_stage_source_as::<DecommissionCopyLockTimeout>(err).is_some()
+}
+
+/// The deferral of a version copy whose in-place retries ended on `error`, if
+/// that failure belongs to the pools at that moment rather than to the object.
+fn take_decommission_copy_deferral(stage: &str, bucket: &str, object: &str, error: &mut Option<Error>) -> Option<Error> {
+    if let Some(err) = error.take_if(|err| is_decommission_copy_quorum_error(err)) {
+        return Some(decommission_copy_quorum_deferral(stage, bucket, object, err));
+    }
+    error
+        .take_if(|err| is_decommission_copy_lock_timeout_error(err))
+        .map(|err| decommission_copy_lock_timeout_deferral(stage, bucket, object, err))
+}
+
 fn resolve_decommission_entry_reload_result(result: Result<()>, bucket: &str, object_name: &str) -> Result<()> {
     result.map_err(|err| Error::other(format!("decommission reload_pool_meta failed for {bucket}/{object_name}: {err}")))
 }
@@ -5783,11 +5822,14 @@ enum DecommissionEntryOutcome {
 /// entry is done and must not be counted as a failure or escalated. A quorum
 /// failure of the source cleanup is deferred too: the copied source is still
 /// in place and the next attempt cleans it up again. So is a version copy that
-/// lost drive quorum through its in-place retries: the source still holds the
-/// version and the next attempt copies it again. Every other failure stays
-/// terminal.
+/// lost drive quorum or timed out on a namespace lock through its in-place
+/// retries: the source still holds the version and the next attempt copies it
+/// again. Every other failure stays terminal.
 fn resolve_decommission_entry_failure(err: Error) -> Result<DecommissionEntryOutcome> {
-    if is_decommission_source_cleanup_retryable_error(&err) || is_decommission_copy_quorum_deferral(&err) {
+    if is_decommission_source_cleanup_retryable_error(&err)
+        || is_decommission_copy_quorum_deferral(&err)
+        || is_decommission_copy_lock_timeout_deferral(&err)
+    {
         return Ok(DecommissionEntryOutcome::Deferred(err));
     }
     match classify_decommission_capacity_failure(&err) {
@@ -11703,6 +11745,61 @@ pub struct PoolSpaceInfo {
     pub used: usize,
 }
 
+/// Read-only observations used by the admin pool diagnostic surface. The HTTP
+/// layer converts these stable values to strings without changing the pool
+/// lifecycle status contract.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum PoolAvailability {
+    Online,
+    Degraded,
+    Offline,
+    Unknown,
+}
+
+impl PoolAvailability {
+    pub fn as_str(self) -> &'static str {
+        match self {
+            Self::Online => "online",
+            Self::Degraded => "degraded",
+            Self::Offline => "offline",
+            Self::Unknown => "unknown",
+        }
+    }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub enum PoolCapacityStatus {
+    Live,
+    Cached,
+    Partial,
+    Unknown,
+}
+
+impl PoolCapacityStatus {
+    pub fn as_str(self) -> &'static str {
+        match self {
+            Self::Live => "live",
+            Self::Cached => "cached",
+            Self::Partial => "partial",
+            Self::Unknown => "unknown",
+        }
+    }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PoolDiagnosticStatus {
+    pub availability: PoolAvailability,
+    pub capacity_status: PoolCapacityStatus,
+}
+
+#[derive(Debug, Clone)]
+pub struct PoolAdminDiagnostic {
+    pub id: usize,
+    pub cmd_line: String,
+    pub pool: Option<PoolStatus>,
+    pub diagnostic: PoolDiagnosticStatus,
+}
+
 #[derive(Debug, Clone, Copy, PartialEq, Eq)]
 pub(crate) struct DecommissionPoolCapacityInfo {
     pool_index: usize,
@@ -11711,6 +11808,7 @@ pub(crate) struct DecommissionPoolCapacityInfo {
     physical_free: usize,
     physical_total: usize,
     physical_used: usize,
+    diagnostic: PoolDiagnosticStatus,
 }
 
 impl DecommissionPoolCapacityInfo {
@@ -11733,6 +11831,10 @@ impl DecommissionPoolCapacityInfo {
             physical_free,
             physical_total,
             physical_used,
+            diagnostic: PoolDiagnosticStatus {
+                availability: PoolAvailability::Unknown,
+                capacity_status: PoolCapacityStatus::Unknown,
+            },
         }
     }
 }
@@ -11754,6 +11856,121 @@ where
     Ok(capacities)
 }
 
+fn pool_diagnostic_status(
+    disks: &[rustfs_madmin::Disk],
+    pool_index: usize,
+    set_count: usize,
+    set_drive_count: usize,
+    read_quorum: usize,
+) -> PoolDiagnosticStatus {
+    if set_count == 0 || set_drive_count == 0 || read_quorum == 0 || read_quorum > set_drive_count {
+        return PoolDiagnosticStatus {
+            availability: PoolAvailability::Unknown,
+            capacity_status: PoolCapacityStatus::Unknown,
+        };
+    }
+
+    let mut live_capacity = true;
+    let mut cached_capacity = false;
+    let mut partial_capacity = false;
+    let mut online_by_set = vec![0usize; set_count];
+    let mut observed_disks = 0usize;
+    let mut seen_slots = HashSet::new();
+
+    for disk in disks.iter().filter(|disk| disk.pool_index == pool_index as i32) {
+        observed_disks = observed_disks.saturating_add(1);
+        match disk.capacity_observation_source.as_deref() {
+            Some("live_probe") => {}
+            Some("snapshot") => {
+                live_capacity = false;
+                cached_capacity = true;
+            }
+            Some("missing") | None => {
+                live_capacity = false;
+                partial_capacity = true;
+            }
+            Some(_) => {
+                live_capacity = false;
+                partial_capacity = true;
+            }
+        }
+
+        let Some(set_index) = usize::try_from(disk.set_index).ok() else {
+            continue;
+        };
+        let Some(disk_index) = usize::try_from(disk.disk_index).ok() else {
+            continue;
+        };
+        if set_index >= set_count
+            || disk_index >= set_drive_count
+            || !seen_slots.insert((set_index, disk_index))
+            || !disk_is_available_for_pool_diagnostic(disk)
+        {
+            continue;
+        }
+        online_by_set[set_index] = online_by_set[set_index].saturating_add(1);
+    }
+
+    let availability = if online_by_set.iter().any(|&online| online < read_quorum) {
+        PoolAvailability::Offline
+    } else if online_by_set.iter().all(|&online| online == set_drive_count) {
+        PoolAvailability::Online
+    } else {
+        PoolAvailability::Degraded
+    };
+    let capacity_status = if observed_disks == 0 {
+        PoolCapacityStatus::Unknown
+    } else if partial_capacity {
+        PoolCapacityStatus::Partial
+    } else if cached_capacity {
+        PoolCapacityStatus::Cached
+    } else if live_capacity {
+        PoolCapacityStatus::Live
+    } else {
+        PoolCapacityStatus::Unknown
+    };
+
+    PoolDiagnosticStatus {
+        availability,
+        capacity_status,
+    }
+}
+
+fn disk_is_available_for_pool_diagnostic(disk: &rustfs_madmin::Disk) -> bool {
+    let state_ok = disk.state.eq_ignore_ascii_case("ok") || disk.state.eq_ignore_ascii_case(rustfs_madmin::ITEM_ONLINE);
+    let runtime_ok = disk
+        .runtime_state
+        .as_deref()
+        .map(|state| state.eq_ignore_ascii_case(rustfs_madmin::ITEM_ONLINE))
+        .unwrap_or(state_ok);
+    state_ok && runtime_ok
+}
+
+fn pool_diagnostic_metadata_status(err: &Error) -> &'static str {
+    if let Some(failure) = err.pool_metadata_failure() {
+        return match failure.kind {
+            crate::error::PoolMetadataFailure::ReadUnavailable => "stale",
+            crate::error::PoolMetadataFailure::RecoveryRequired
+            | crate::error::PoolMetadataFailure::TransactionUnknown
+            | crate::error::PoolMetadataFailure::FenceLost => "recovery_required",
+        };
+    }
+
+    match err {
+        Error::Timeout
+        | Error::Lock(_)
+        | Error::DiskNotFound
+        | Error::FaultyDisk
+        | Error::FaultyRemoteDisk
+        | Error::RemoteClientUnavailable(_)
+        | Error::RemoteNotInitialized
+        | Error::ErasureReadQuorum
+        | Error::InsufficientReadQuorum(_, _)
+        | Error::NamespaceLockQuorumUnavailable { .. } => "stale",
+        _ => "recovery_required",
+    }
+}
+
 #[cfg(test)]
 static DECOMMISSION_START_GATE_WAIT_OVERRIDES: std::sync::OnceLock<std::sync::Mutex<HashMap<uuid::Uuid, std::time::Duration>>> =
     std::sync::OnceLock::new();
@@ -12666,7 +12883,7 @@ pub(crate) async fn should_skip_lifecycle_for_data_movement(
     store: Arc<ECStore>,
     bucket: &str,
     version: &rustfs_filemeta::FileInfo,
-    lifecycle_config: Option<&PersistedLifecycleConfiguration>,
+    lifecycle_config: Option<&BucketLifecycleConfiguration>,
     object_lock_config: Option<&ObjectLockConfiguration>,
     apply_actions: bool,
     event_source: &LcEventSrc,
@@ -15126,6 +15343,70 @@ impl ECStore {
         Ok(snapshot)
     }
 
+    /// Admin-only diagnostic snapshot. A transient metadata read failure is
+    /// reported as stale without publishing the fallback or relaxing any
+    /// mutation/recovery path.
+    pub async fn admin_pool_diagnostic_snapshot(
+        &self,
+        target_pool_index: Option<usize>,
+    ) -> (Vec<PoolAdminDiagnostic>, &'static str) {
+        let fallback = self.pool_meta.read().await.clone();
+        let (pool_meta, mut metadata_status) = match self.pool_status_meta_snapshot().await {
+            Ok(pool_meta) => (pool_meta, "fresh"),
+            Err(err) => (fallback, pool_diagnostic_metadata_status(&err)),
+        };
+        if metadata_status == "stale" && pool_meta.pools.len() != self.pools.len() {
+            metadata_status = "unknown";
+        }
+
+        let backend = rustfs_madmin::StorageInfo {
+            backend: StorageAdminApi::backend_info(self).await,
+            ..Default::default()
+        };
+        let indices = target_pool_index.map_or_else(|| (0..self.pools.len()).collect(), |idx| vec![idx]);
+        let backend = &backend;
+        let samples = futures::future::join_all(indices.into_iter().map(|idx| async move {
+            (
+                idx,
+                self.get_decommission_pool_capacity_info_with_backend(idx, false, backend)
+                    .await,
+            )
+        }))
+        .await;
+
+        let mut diagnostics = Vec::with_capacity(samples.len());
+        for (idx, sample) in samples {
+            let Some(sets) = self.pools.get(idx) else {
+                continue;
+            };
+            let (pool, diagnostic) = match sample {
+                Ok(capacity) => (
+                    pool_meta
+                        .pools
+                        .get(idx)
+                        .cloned()
+                        .map(|pool| apply_decommission_status_space_info(pool, capacity.space)),
+                    capacity.diagnostic,
+                ),
+                Err(_) => (
+                    pool_meta.pools.get(idx).cloned(),
+                    PoolDiagnosticStatus {
+                        availability: PoolAvailability::Unknown,
+                        capacity_status: PoolCapacityStatus::Unknown,
+                    },
+                ),
+            };
+            diagnostics.push(PoolAdminDiagnostic {
+                id: idx,
+                cmd_line: sets.endpoints.cmd_line.clone(),
+                pool,
+                diagnostic,
+            });
+        }
+
+        (diagnostics, metadata_status)
+    }
+
     /// Load and publish under the movement writer, so no movement read sees
     /// the transition mid-operation and no transition interleaves the load.
     async fn publish_pool_status_refresh(&self, _movement_guard: tokio::sync::OwnedRwLockWriteGuard<()>) -> Result<()> {
@@ -15202,6 +15483,7 @@ impl ECStore {
             };
             let (physical_total, physical_free, physical_used) =
                 decommission_physical_pool_capacity(&info.disks, idx, layout, space, owned_physical_used.as_ref());
+            let diagnostic = pool_diagnostic_status(&info.disks, idx, sets.set_count, sets.set_drive_count, layout.data);
 
             Ok(DecommissionPoolCapacityInfo {
                 pool_index: idx,
@@ -15210,6 +15492,7 @@ impl ECStore {
                 physical_free,
                 physical_total,
                 physical_used,
+                diagnostic,
             })
         } else {
             Err(invalid_decommission_pool_index_error(self.pools.len(), idx))
@@ -16529,9 +16812,9 @@ impl ECStore {
         generation: OffsetDateTime,
         bucket: String,
         set: Arc<SetDisks>,
-        lifecycle_config: Option<PersistedLifecycleConfiguration>,
+        lifecycle_config: Option<BucketLifecycleConfiguration>,
         object_lock_config: Option<ObjectLockConfiguration>,
-        replication_config: Option<(PersistedReplicationConfiguration, OffsetDateTime)>,
+        replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
         expected_bucket_incarnation_id: Option<uuid::Uuid>,
         source_changed_exhaustions: Arc<AtomicUsize>,
         entry_budget: Arc<Semaphore>,
@@ -16703,9 +16986,9 @@ impl ECStore {
         mut deferred_entry: DeferredDecommissionEntry,
         bucket: String,
         set: Arc<SetDisks>,
-        lifecycle_config: Option<PersistedLifecycleConfiguration>,
+        lifecycle_config: Option<BucketLifecycleConfiguration>,
         object_lock_config: Option<ObjectLockConfiguration>,
-        replication_config: Option<(PersistedReplicationConfiguration, OffsetDateTime)>,
+        replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
         expected_bucket_incarnation_id: Option<uuid::Uuid>,
         source_changed_exhaustions: Arc<AtomicUsize>,
         entry_budget: Arc<Semaphore>,
@@ -16812,9 +17095,9 @@ impl ECStore {
         generation: OffsetDateTime,
         set: Arc<SetDisks>,
         bi: DecomBucketInfo,
-        lifecycle_config: Option<PersistedLifecycleConfiguration>,
+        lifecycle_config: Option<BucketLifecycleConfiguration>,
         object_lock_config: Option<ObjectLockConfiguration>,
-        replication_config: Option<(PersistedReplicationConfiguration, OffsetDateTime)>,
+        replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
         expected_bucket_incarnation_id: Option<uuid::Uuid>,
         source_changed_exhaustions: Arc<AtomicUsize>,
         entry_budget: Arc<Semaphore>,
@@ -17489,9 +17772,9 @@ impl ECStore {
         entry: MetaCacheEntry,
         bucket: String,
         set: Arc<SetDisks>,
-        lifecycle_config: Option<PersistedLifecycleConfiguration>,
+        lifecycle_config: Option<BucketLifecycleConfiguration>,
         object_lock_config: Option<ObjectLockConfiguration>,
-        replication_config: Option<(PersistedReplicationConfiguration, OffsetDateTime)>,
+        replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
         expected_bucket_incarnation_id: Option<uuid::Uuid>,
         source_changed_exhaustions: Arc<AtomicUsize>,
         attempts: &mut usize,
@@ -17675,9 +17958,9 @@ impl ECStore {
         entry: MetaCacheEntry,
         bucket: String,
         set: Arc<SetDisks>,
-        lifecycle_config: Option<PersistedLifecycleConfiguration>,
+        lifecycle_config: Option<BucketLifecycleConfiguration>,
         object_lock_config: Option<ObjectLockConfiguration>,
-        replication_config: Option<(PersistedReplicationConfiguration, OffsetDateTime)>,
+        replication_config: Option<(ReplicationConfiguration, OffsetDateTime)>,
         expected_bucket_incarnation_id: Option<uuid::Uuid>,
         entry_attempt: usize,
         scanner_backlog_handoff_attempt: usize,
@@ -18151,13 +18434,11 @@ impl ECStore {
                     }
                 }
 
-                if failure && let Some(err) = error.take_if(|err| is_decommission_copy_quorum_error(err)) {
-                    return Err(decommission_copy_quorum_deferral(
-                        "delete_marker_copy",
-                        bucket.as_str(),
-                        version.name.as_str(),
-                        err,
-                    ));
+                if failure
+                    && let Some(err) =
+                        take_decommission_copy_deferral("delete_marker_copy", bucket.as_str(), version.name.as_str(), &mut error)
+                {
+                    return Err(err);
                 }
 
                 if ignore {
@@ -18543,14 +18824,14 @@ impl ECStore {
 
             if failure
                 && !source_read_failed
-                && let Some(err) = error.take_if(|err| is_decommission_copy_quorum_error(err))
-            {
-                return Err(decommission_copy_quorum_deferral(
+                && let Some(err) = take_decommission_copy_deferral(
                     DECOMMISSION_STAGE_MIGRATE_OBJECT,
                     bucket.as_str(),
                     version.name.as_str(),
-                    err,
-                ));
+                    &mut error,
+                )
+            {
+                return Err(err);
             }
 
             if ignore {
@@ -18965,7 +19246,7 @@ impl ECStore {
         entry: MetaCacheEntry,
         bucket: String,
         set: Arc<SetDisks>,
-        lifecycle_config: Option<PersistedLifecycleConfiguration>,
+        lifecycle_config: Option<BucketLifecycleConfiguration>,
     ) -> Result<()> {
         let expected_bucket_incarnation_id = if is_meta_bucketname(&bucket) {
             None
@@ -21804,6 +22085,85 @@ mod tests {
     use crate::storage_api_contracts::multipart::MultipartOperations as _;
     use serde::Serialize;
 
+    fn diagnostic_disk(set_index: i32, disk_index: i32, online: bool) -> rustfs_madmin::Disk {
+        rustfs_madmin::Disk {
+            pool_index: 0,
+            set_index,
+            disk_index,
+            state: if online { "ok" } else { "offline" }.to_string(),
+            runtime_state: Some(if online { "online" } else { "offline" }.to_string()),
+            capacity_observation_source: Some("live_probe".to_string()),
+            ..Default::default()
+        }
+    }
+
+    #[test]
+    fn pool_diagnostic_status_tracks_read_quorum_and_capacity_source() {
+        let online = vec![
+            diagnostic_disk(0, 0, true),
+            diagnostic_disk(0, 1, true),
+            diagnostic_disk(1, 0, true),
+            diagnostic_disk(1, 1, true),
+        ];
+        assert_eq!(
+            pool_diagnostic_status(&online, 0, 2, 2, 1),
+            PoolDiagnosticStatus {
+                availability: PoolAvailability::Online,
+                capacity_status: PoolCapacityStatus::Live,
+            }
+        );
+
+        let mut degraded = online.clone();
+        degraded[0] = diagnostic_disk(0, 0, false);
+        degraded[1].capacity_observation_source = Some("snapshot".to_string());
+        assert_eq!(
+            pool_diagnostic_status(&degraded, 0, 2, 2, 1),
+            PoolDiagnosticStatus {
+                availability: PoolAvailability::Degraded,
+                capacity_status: PoolCapacityStatus::Cached,
+            }
+        );
+
+        degraded[1] = diagnostic_disk(0, 1, false);
+        assert_eq!(pool_diagnostic_status(&degraded, 0, 2, 2, 1).availability, PoolAvailability::Offline);
+        assert_eq!(
+            pool_diagnostic_status(&[], 0, 2, 2, 1),
+            PoolDiagnosticStatus {
+                availability: PoolAvailability::Offline,
+                capacity_status: PoolCapacityStatus::Unknown,
+            }
+        );
+        assert_eq!(pool_diagnostic_status(&online, 0, 0, 2, 1).availability, PoolAvailability::Unknown);
+    }
+
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn admin_pool_diagnostic_snapshot_lists_an_unavailable_nonfirst_pool() {
+        let (_temp_dirs, store, _other_store) = Box::pin(crate::services::rebalance::test_two_pool_stores(None)).await;
+        for set in &store.pools[1].disk_set {
+            let mut disks = set.disks.write().await;
+            let disk_count = disks.len();
+            *disks = vec![None; disk_count];
+        }
+
+        let (snapshot, metadata_status) =
+            tokio::time::timeout(std::time::Duration::from_secs(10), store.admin_pool_diagnostic_snapshot(None))
+                .await
+                .expect("pool diagnostics should remain bounded");
+
+        assert_eq!(metadata_status, "stale");
+        assert_eq!(snapshot.len(), 2, "configured topology must remain visible");
+        assert_eq!(snapshot[0].id, 0);
+        assert_eq!(snapshot[1].id, 1);
+        assert_eq!(snapshot[1].diagnostic.availability, PoolAvailability::Offline);
+        store
+            .pool_meta_save_gate
+            .lock()
+            .await
+            .ensure_write_safe("ordinary object write")
+            .expect("a failed diagnostic read must not latch the writer gate");
+    }
+
     #[test]
     fn decommission_set_local_usage_cache_classification_is_exact() {
         for object in [
@@ -27940,6 +28300,167 @@ mod tests {
         );
     }
 
+    /// One version commits, then every in-place attempt to copy the next one
+    /// times out on the target namespace lock. The entry is deferred, not
+    /// failed, with both source versions retained; the resumed entry finds the
+    /// committed version and moves both with their exact bytes.
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn decommission_defers_target_lookup_lock_timeout_and_retries_without_losing_source() {
+        use tokio::io::AsyncReadExt as _;
+
+        let fixture = NullVersionDecommission::new(NullVersionTop::NewerVersion).await;
+        let object = fixture.object;
+        let injected = Arc::new(AtomicUsize::new(0));
+        let attempted = Arc::new(AtomicUsize::new(0));
+        let fault = DecommissionTestFaultGuard::install_with_error(
+            Arc::new({
+                let injected = Arc::clone(&injected);
+                let attempted = Arc::clone(&attempted);
+                move |stage, _, faulted, _, success| {
+                    let target = stage == "before_target_commit" && faulted == object && success;
+                    let should_inject = target && attempted.fetch_add(1, Ordering::SeqCst) > 0;
+                    if should_inject {
+                        injected.fetch_add(1, Ordering::SeqCst);
+                    }
+                    should_inject
+                }
+            }),
+            Some(Arc::new(|_, bucket, object| {
+                data_movement::data_movement_stage_error_for_test(
+                    "decommission_object",
+                    "prepare_put_object",
+                    bucket,
+                    object,
+                    Error::Lock(rustfs_lock::LockError::timeout(
+                        format!("{bucket}/{object}@latest"),
+                        std::time::Duration::from_secs(5),
+                    )),
+                )
+            })),
+        );
+        let (outcome, _) = fixture.run_entry().await;
+        assert_eq!(injected.load(Ordering::SeqCst), DECOMMISSION_VERSION_COPY_ATTEMPTS);
+        fixture.assert_source_retained().await;
+        assert_eq!(
+            fixture
+                .target_versions()
+                .await
+                .expect("the first version was copied")
+                .versions
+                .len(),
+            1,
+            "one version must commit before the next version times out"
+        );
+        assert!(
+            matches!(&outcome, DecommissionEntryOutcome::Deferred(err) if is_decommission_copy_lock_timeout_deferral(err)),
+            "{outcome:?}"
+        );
+        assert_eq!(
+            fixture.store.pool_meta.read().await.pools[0]
+                .decommission
+                .as_ref()
+                .expect("active decommission")
+                .items_decommission_failed,
+            0,
+            "a deferred entry must not be counted as a permanent failure"
+        );
+        drop(fault);
+        let (outcome, _) = fixture.run_entry().await;
+        assert!(matches!(outcome, DecommissionEntryOutcome::Complete), "{outcome:?}");
+        fixture.assert_moved().await;
+        let mut reader = fixture.store.pools[1]
+            .get_object_reader(
+                fixture.bucket,
+                fixture.object,
+                None,
+                HeaderMap::new(),
+                &ObjectOptions {
+                    version_id: Some(fixture.seeded.versions[0].version_id.expect("newer version").to_string()),
+                    ..Default::default()
+                },
+            )
+            .await
+            .expect("the newer version must remain readable");
+        let mut actual = Vec::new();
+        reader.stream.read_to_end(&mut actual).await.expect("read the newer version");
+        assert_eq!(actual, b"written after versioning was enabled");
+    }
+
+    /// A namespace lock still held on the target key, as by a copy whose node
+    /// was killed mid-write, makes target selection fail with a typed
+    /// object-lock timeout. A real held lock, not an injected error, must defer
+    /// the entry rather than fail it; once the lock is released selection
+    /// recovers and the resumed entry moves both versions.
+    #[tokio::test]
+    #[serial_test::serial]
+    async fn decommission_target_selection_returns_timeout_for_a_held_object_lock() {
+        temp_env::async_with_vars([(rustfs_config::ENV_OBJECT_LOCK_ACQUIRE_TIMEOUT, Some("1"))], async {
+            let fixture = NullVersionDecommission::new(NullVersionTop::NewerVersion).await;
+            let target_set = fixture.store.pools[1].get_disks_by_key(fixture.object);
+            let lock = target_set
+                .new_ns_lock(fixture.bucket, fixture.object)
+                .await
+                .expect("target lock");
+            let guard = lock
+                .get_write_lock(std::time::Duration::from_secs(30))
+                .await
+                .expect("hold target lock");
+            let opts = ObjectOptions {
+                data_movement: true,
+                src_pool_idx: 0,
+                version_id: Some(uuid::Uuid::nil().to_string()),
+                ..Default::default()
+            };
+            let select = || {
+                fixture
+                    .store
+                    .select_data_movement_pool_idx(fixture.bucket, fixture.object, 1, &opts, false)
+            };
+            let result = tokio::time::timeout(get_lock_acquire_timeout() + std::time::Duration::from_secs(15), select())
+                .await
+                .expect("target selection must terminate");
+            match result {
+                Err(Error::Lock(rustfs_lock::LockError::Timeout { resource, .. })) => {
+                    assert!(resource.contains(fixture.bucket) && resource.contains(fixture.object), "{resource}");
+                }
+                other => panic!("expected an object lock timeout, got {other:?}"),
+            }
+
+            let (outcome, _) = fixture.run_entry().await;
+            assert!(
+                matches!(&outcome, DecommissionEntryOutcome::Deferred(err) if is_decommission_copy_lock_timeout_deferral(err)),
+                "{outcome:?}"
+            );
+            assert_eq!(
+                fixture.store.pool_meta.read().await.pools[0]
+                    .decommission
+                    .as_ref()
+                    .expect("active decommission")
+                    .items_decommission_failed,
+                0,
+                "a held target lock must not be counted as a permanent failure"
+            );
+            fixture.assert_source_retained().await;
+            assert!(
+                fixture.target_versions().await.is_none(),
+                "no version may commit while the target lock is held"
+            );
+
+            drop(guard);
+            assert_eq!(
+                select()
+                    .await
+                    .expect("target selection must recover once the lock is released"),
+                1
+            );
+            let (outcome, _) = fixture.run_entry().await;
+            assert!(matches!(outcome, DecommissionEntryOutcome::Complete), "{outcome:?}");
+            fixture.assert_moved().await;
+        })
+        .await;
+    }
+
     #[test]
     fn decommission_object_attempt_metric_keeps_the_peak() {
         let mut attempts = 0;
@@ -28297,6 +28818,61 @@ mod tests {
         ));
     }
 
+    /// A restarted node's previous incarnation keeps its namespace locks on the
+    /// peers until their lease expires, so a resumed copy of a key that was in
+    /// flight at the crash times out acquiring the lock. That copy never
+    /// committed: it is deferred, never counted as a failed object.
+    #[test]
+    fn decommission_copy_lock_timeout_is_deferred_not_failed() {
+        let timeout = || {
+            Error::Lock(rustfs_lock::LockError::timeout(
+                "bucket-a/object-a@latest",
+                std::time::Duration::from_secs(5),
+            ))
+        };
+        let prepare = data_movement::data_movement_stage_error_for_test(
+            "decommission_object",
+            "prepare_put_object",
+            "bucket-a",
+            "object-a",
+            timeout(),
+        );
+        for err in [timeout(), prepare] {
+            assert!(is_decommission_copy_lock_timeout_error(&err), "{err}");
+            assert!(!is_decommission_copy_quorum_error(&err), "{err}");
+            let mut error = Some(err);
+            let deferred = take_decommission_copy_deferral("migrate_object", "bucket-a", "object-a", &mut error)
+                .expect("a lock timeout copy must be deferred");
+            assert!(error.is_none());
+            assert!(is_decommission_copy_lock_timeout_deferral(&deferred));
+            assert!(!is_decommission_copy_quorum_deferral(&deferred));
+            let failure = data_movement::data_movement_stage_source_as::<DecommissionCopyLockTimeout>(&deferred)
+                .expect("the deferred error must keep its lock-timeout origin");
+            assert!(is_decommission_copy_lock_timeout_error(&failure.0));
+            assert!(matches!(
+                resolve_decommission_entry_failure(deferred),
+                Ok(DecommissionEntryOutcome::Deferred(_))
+            ));
+        }
+
+        let mut quorum = Some(Error::ErasureWriteQuorum);
+        let deferred = take_decommission_copy_deferral("migrate_object", "bucket-a", "object-a", &mut quorum)
+            .expect("a lost drive quorum copy stays deferred");
+        assert!(is_decommission_copy_quorum_deferral(&deferred));
+
+        for other in [
+            Error::Lock(rustfs_lock::LockError::internal("lock backend failed")),
+            Error::other("Lock acquisition timeout for resource 'bucket-a/object-a@latest' after 5s"),
+        ] {
+            let mut error = Some(other);
+            assert!(
+                take_decommission_copy_deferral("migrate_object", "bucket-a", "object-a", &mut error).is_none(),
+                "{error:?}"
+            );
+            assert!(error.is_some(), "a per-object failure keeps its error");
+        }
+    }
+
     #[test]
     fn decommission_target_capacity_error_accepts_wrapped_capacity_errors() {
         let disk_full = Error::other(format!("decommission_object: put_object failed for bucket/object: {}", Error::DiskFull));

crates/ecstore/src/core/pools_test.rs

@@ -2480,6 +2480,93 @@ mod decommission_lock_order_tests {
         });
     }
 
+    /// A node killed mid-copy leaves its namespace lock on the peers until the
+    /// lock lease expires, so the resumed decommission times out acquiring the
+    /// lock of the key it was copying through every in-place retry. The entry
+    /// is deferred and copied once the lease is gone; it is never counted as a
+    /// failed object, which would fail the whole decommission.
+    #[test]
+    #[serial_test::serial]
+    fn decommission_defers_a_copy_that_timed_out_on_a_stale_namespace_lock() {
+        run_large_stack_async_test("decommission-copy-lock-timeout", || async {
+            let (_temp_dirs, store, _other_store) = test_three_pool_stores_with_isolated_node_contexts(None).await;
+            let bucket = test_bucket("copy-lock-timeout");
+            let object = "external/005.bin";
+            let payload = vec![0x3c; 2048];
+            store
+                .make_bucket(&bucket, &MakeBucketOptions::default())
+                .await
+                .expect("create the bucket");
+            put_source_object(&store, &bucket, object, payload.clone()).await;
+
+            // The stale lease outlives every in-place attempt of the first
+            // pass and has expired by the deferred replay.
+            let stale_lease_calls = crate::core::pools::DECOMMISSION_VERSION_COPY_ATTEMPTS;
+            let copy_calls = Arc::new(AtomicUsize::new(0));
+            let _fault = DecommissionTestFaultGuard::install_with_error(
+                Arc::new({
+                    let copy_calls = Arc::clone(&copy_calls);
+                    let bucket = bucket.clone();
+                    move |stage, entry_bucket, entry_object, _, _| {
+                        stage == "before_target_commit"
+                            && entry_bucket == bucket
+                            && entry_object == object
+                            && copy_calls.fetch_add(1, Ordering::AcqRel) < stale_lease_calls
+                    }
+                }),
+                Some(Arc::new(|_, bucket, object| {
+                    // The shape the target write reports when the lock domain
+                    // still holds the previous incarnation's lock.
+                    data_movement::data_movement_stage_error_for_test(
+                        "decommission_object",
+                        "prepare_put_object",
+                        bucket,
+                        object,
+                        crate::error::Error::Lock(rustfs_lock::LockError::timeout(
+                            format!("{bucket}/{object}@latest"),
+                            Duration::from_secs(5),
+                        )),
+                    )
+                })),
+            );
+
+            start_transitioned_entry_decommission(&store, &bucket).await;
+            spawn_decommission_routine(&store, 1)
+                .await
+                .expect("the decommission routine task should join")
+                .expect("the decommission routine should return");
+
+            assert_eq!(
+                copy_calls.load(Ordering::Acquire),
+                stale_lease_calls + 1,
+                "the first pass exhausts its in-place retries and the replay copies the object once"
+            );
+            let local = store.pool_meta.read().await;
+            let info = local.pools[0]
+                .decommission
+                .as_ref()
+                .expect("the decommission state should remain present");
+            assert_eq!(
+                info.items_decommission_failed, 0,
+                "a lock held until its lease expires is not a permanent failure: {info:?}"
+            );
+            assert!(
+                info.complete && !info.failed && !info.canceled && info.capacity_blocked_reason.is_none(),
+                "the replayed entry must complete the decommission: {info:?}"
+            );
+            drop(local);
+
+            assert!(exact_versions(&store, 0, &bucket, object).await.is_empty(), "the source is cleaned up");
+            let mut reader = store
+                .get_object_reader(&bucket, object, None, HeaderMap::new(), &ObjectOptions::default())
+                .await
+                .expect("the migrated object stays readable");
+            let mut body = Vec::new();
+            reader.stream.read_to_end(&mut body).await.expect("read the migrated object");
+            assert_eq!(body, payload);
+        });
+    }
+
     /// Seed `object` in pool 0 with three 2 KiB data versions and a newest
     /// delete marker; returns the data versions oldest first.
     async fn seed_versioned_entry_with_marker(

crates/ecstore/src/core/sets.rs

@@ -65,38 +65,6 @@ use uuid::Uuid;
 
 const LIST_MULTIPART_SETS_CONCURRENCY: usize = 4;
 
-#[cfg(test)]
-type TestSetLockers = HashMap<usize, Vec<Arc<dyn rustfs_lock::LockClient>>>;
-
-#[cfg(test)]
-static TEST_SET_LOCKERS: std::sync::LazyLock<std::sync::Mutex<TestSetLockers>> =
-    std::sync::LazyLock::new(|| std::sync::Mutex::new(HashMap::new()));
-
-/// Keep a fixture's real locker domain scoped to its owning instance.
-#[cfg(test)]
-pub(crate) struct ScopedSetLockers(Arc<InstanceContext>);
-
-#[cfg(test)]
-impl ScopedSetLockers {
-    pub(crate) fn install(ctx: Arc<InstanceContext>, lockers: Vec<Arc<dyn rustfs_lock::LockClient>>) -> Self {
-        let key = Arc::as_ptr(&ctx) as usize;
-        let mut domains = TEST_SET_LOCKERS.lock().expect("test set lockers mutex");
-        assert!(!domains.contains_key(&key), "the instance already owns a test locker domain");
-        domains.insert(key, lockers);
-        Self(ctx)
-    }
-}
-
-#[cfg(test)]
-impl Drop for ScopedSetLockers {
-    fn drop(&mut self) {
-        TEST_SET_LOCKERS
-            .lock()
-            .expect("test set lockers mutex")
-            .remove(&(Arc::as_ptr(&self.0) as usize));
-    }
-}
-
 fn is_idempotent_delete_prefix_error(err: &Error) -> bool {
     is_err_object_not_found(err) || is_err_strict_volume_not_found(err)
 }
@@ -298,14 +266,6 @@ impl Sets {
                 }
             }
 
-            #[cfg(test)]
-            let pool_lockers = TEST_SET_LOCKERS
-                .lock()
-                .expect("test set lockers mutex")
-                .get(&(Arc::as_ptr(&instance_ctx) as usize))
-                .cloned()
-                .unwrap_or_else(|| pool_lockers.clone());
-
             let set_disks = SetDisks::new_with_instance_ctx(
                 runtime_sources::local_node_name().await,
                 Arc::new(RwLock::new(set_drive)),

crates/ecstore/src/data_usage/mod.rs

@@ -28,7 +28,7 @@ use crate::{
     error::{Error, classify_system_path_failure_reason},
     object_api::{ObjectInfo, ObjectOptions, PutObjReader},
     runtime::sources as runtime_sources,
-    store::{ECStore, list_objects::list_marker_key},
+    store::ECStore,
 };
 pub use local_snapshot::{LocalUsageSnapshot, read_snapshot as read_local_snapshot, snapshot_path};
 use rustfs_data_usage::{
@@ -1493,6 +1493,48 @@ pub async fn load_data_usage_from_backend_cached(store: Arc<ECStore>) -> Result<
     project_data_usage_from_backend_cached(store, DataUsageInfo::clone).await
 }
 
+/// Read one bucket's persisted usage without cloning the complete snapshot.
+pub async fn load_bucket_usage_from_backend_cached(
+    store: Arc<ECStore>,
+    bucket: &str,
+    incarnation: uuid::Uuid,
+) -> Result<Option<u64>, Error> {
+    project_data_usage_from_backend_cached(store, |info| certified_bucket_usage(info, bucket, incarnation)).await
+}
+
+fn certified_bucket_usage(info: &DataUsageInfo, bucket: &str, incarnation: uuid::Uuid) -> Option<u64> {
+    if incarnation.is_nil()
+        || !info.is_complete_bucket_usage_snapshot()
+        || info.usage_snapshot_observational
+        || info.scanner_cycle.is_none()
+        || info.scanner_epoch.is_none()
+    {
+        return None;
+    }
+    let pending_count = u64::try_from(info.usage_snapshot_pending_buckets.len()).ok()?;
+    if pending_count != info.usage_snapshot_pending_bucket_count {
+        return None;
+    }
+    match info.usage_snapshot_converged {
+        Some(true) if pending_count == 0 => {}
+        Some(false)
+            if pending_count > 0
+                && info.usage_snapshot_pending_buckets.windows(2).all(|pair| pair[0] < pair[1])
+                && info
+                    .usage_snapshot_pending_buckets
+                    .binary_search_by(|pending| pending.as_str().cmp(bucket))
+                    .is_err() => {}
+        _ => return None,
+    }
+    let usage = info.buckets_usage.get(bucket)?;
+    let certified = usage
+        .bucket_incarnation
+        .as_deref()
+        .and_then(|id| uuid::Uuid::parse_str(id).ok())
+        .filter(|id| !id.is_nil())?;
+    (certified == incarnation).then_some(usage.size)
+}
+
 // Project under the cache guard so scalar readers never copy the snapshot's
 // bucket maps. The owned API shares the same TTL, refresh and invalidation path.
 async fn project_data_usage_from_backend_cached<T>(
@@ -1966,8 +2008,8 @@ fn advance_version_listing_cursor(
     let next_version_marker = next_version_marker
         .filter(|next_version_marker| !next_version_marker.is_empty())
         .ok_or_else(|| Error::other(format!("bucket {bucket} version listing was truncated without a version marker")))?;
-    let current_key_marker = marker.as_deref().map(list_marker_key);
-    let next_key_marker = list_marker_key(&next_marker);
+    let current_key_marker = marker.as_deref();
+    let next_key_marker = next_marker.as_str();
     if current_key_marker == Some(next_key_marker) && version_marker.as_deref() == Some(next_version_marker.as_str()) {
         return Err(Error::other(format!(
             "bucket {bucket} version listing returned a repeated continuation marker"
@@ -5167,20 +5209,20 @@ mod tests {
         );
         assert!(ensure_truncated_version_page_has_entries("bucket-a", 0).is_err());
         ensure_truncated_version_page_has_entries("bucket-a", 1).expect("a truncated page with an entry can advance");
+    }
 
-        let mut tagged_marker = Some("object-a[rustfs_cache:v2,id:old]".to_string());
-        let mut tagged_version_marker = Some("version-a".to_string());
-        assert!(
-            advance_version_listing_cursor(
-                "bucket-a",
-                &mut tagged_marker,
-                &mut tagged_version_marker,
-                Some("object-a[rustfs_cache:v2,id:new]".to_string()),
-                Some("version-a".to_string()),
-            )
-            .is_err(),
-            "different cache tags for the same logical marker must not count as progress"
-        );
+    #[test]
+    fn version_listing_cursor_compares_literal_cache_shaped_keys() {
+        let mut marker = Some("object-a".to_string());
+        let mut version_marker = Some("version-a".to_string());
+        advance_version_listing_cursor(
+            "bucket-a",
+            &mut marker,
+            &mut version_marker,
+            Some("object-a[rustfs_cache:v2,id:old]".to_string()),
+            Some("version-a".to_string()),
+        )
+        .expect("a distinct literal key advances the cursor");
     }
 
     #[test]

crates/ecstore/src/data_usage/tests/snapshot_projection.rs

@@ -129,6 +129,41 @@ fn projected_publication_moves_bucket_maps_into_cache() {
     );
 }
 
+#[test]
+fn certified_bucket_usage_rejects_unproven_and_pending_data() {
+    let id = uuid::Uuid::from_u128(7);
+    let mut info = data_usage_info_for_test("known", 1, 42, SystemTime::UNIX_EPOCH);
+    info.usage_snapshot_converged = Some(true);
+    info.scanner_cycle = Some(1);
+    info.scanner_epoch = Some(1);
+    info.buckets_usage.get_mut("known").unwrap().bucket_incarnation = Some(id.to_string());
+    assert_eq!(certified_bucket_usage(&info, "known", id), Some(42));
+    assert_eq!(certified_bucket_usage(&info, "known", uuid::Uuid::from_u128(8)), None);
+    assert_eq!(certified_bucket_usage(&info, "known", uuid::Uuid::nil()), None);
+    let mut legacy = info.clone();
+    legacy.buckets_usage.get_mut("known").unwrap().bucket_incarnation = None;
+    assert_eq!(certified_bucket_usage(&legacy, "known", id), None);
+    legacy.buckets_usage.get_mut("known").unwrap().bucket_incarnation = Some("corrupt".into());
+    assert_eq!(certified_bucket_usage(&legacy, "known", id), None);
+    let mut pending = info.clone();
+    pending.usage_snapshot_converged = Some(false);
+    pending.usage_snapshot_pending_buckets = vec!["other".into()];
+    pending.usage_snapshot_pending_bucket_count = 1;
+    assert_eq!(certified_bucket_usage(&pending, "known", id), Some(42));
+    pending.usage_snapshot_pending_buckets = vec!["known".into()];
+    assert_eq!(certified_bucket_usage(&pending, "known", id), None);
+    pending.usage_snapshot_pending_bucket_count = 0;
+    assert_eq!(certified_bucket_usage(&pending, "known", id), None);
+    pending.usage_snapshot_pending_buckets = vec!["z".into(), "a".into()];
+    pending.usage_snapshot_pending_bucket_count = 2;
+    assert_eq!(certified_bucket_usage(&pending, "known", id), None);
+    let mut observational = info.clone();
+    observational.usage_snapshot_observational = true;
+    assert_eq!(certified_bucket_usage(&observational, "known", id), None);
+    info.usage_snapshot_converged = None;
+    assert_eq!(certified_bucket_usage(&info, "known", id), None);
+}
+
 #[tokio::test]
 #[serial]
 async fn persisted_scalar_projection_shares_cache_refresh_and_invalidation() {
@@ -140,6 +175,13 @@ async fn persisted_scalar_projection_shares_cache_refresh_and_invalidation() {
     info.buckets_usage.insert("empty".to_string(), BucketUsageInfo::default());
     info.bucket_sizes.insert("empty".to_string(), 0);
     info.buckets_count = 2;
+    info.usage_snapshot_converged = Some(true);
+    info.scanner_cycle = Some(1);
+    info.scanner_epoch = Some(1);
+    let incarnation = uuid::Uuid::from_u128(7);
+    for usage in info.buckets_usage.values_mut() {
+        usage.bucket_incarnation = Some(incarnation.to_string());
+    }
     save_config(
         store.clone(),
         DATA_USAGE_OBJ_NAME_PATH.as_str(),
@@ -151,11 +193,29 @@ async fn persisted_scalar_projection_shares_cache_refresh_and_invalidation() {
 
     for (bucket, expected) in [("known", Some(42)), ("empty", Some(0)), ("missing", None)] {
         assert_eq!(persisted_bucket_usage(store.clone(), bucket).await, expected);
+        assert_eq!(
+            load_bucket_usage_from_backend_cached(store.clone(), bucket, incarnation)
+                .await
+                .expect("project bucket usage"),
+            expected
+        );
         let owned = load_data_usage_from_backend_cached(store.clone())
             .await
             .expect("owned cached snapshot");
         assert_eq!(expected, owned.buckets_usage.get(bucket).map(|usage| usage.size));
     }
+    assert_eq!(
+        load_bucket_usage_from_backend_cached(store.clone(), "known", uuid::Uuid::from_u128(8))
+            .await
+            .unwrap(),
+        None
+    );
+    assert_eq!(
+        load_bucket_usage_from_backend_cached(store.clone(), "known", uuid::Uuid::nil())
+            .await
+            .unwrap(),
+        None
+    );
     info.buckets_usage.get_mut("known").expect("known bucket").size = 84;
     info.bucket_sizes.insert("known".to_string(), 84);
     save_config(